Possible "Types" for API Return Values

[jiyer2016]

The official documentation states the following:

API Function Return Value
The output of an API function can be any of the follow types:

pandas.DataFrame
pandas.Series
numpy.ndarray
tensorflow.Tensor

# List of JSON Serializable
# JSON = t.Union[str, int, float, bool, None, t.Mapping[str, 'JSON'], t.List['JSON']]
List[JSON]

The recommendation is for the a list of JSON to be returned (and NOT a single JSON object).
This is presumably related to the Micro-Batching feature.

However, in one of our recent discussions on Shadow Model Inference - #1051 - @yubozhao added the following code - where a single JSON response object was returned.

class MyService(BentoService):
    def prediction_current(self, df):
          return self.models.current_model.predict(df)

    def prediction_shadow(self, df):
          return self.models.shadow_model.predict(df)

    @api(input=DataframeInput())
    def predict(self, df):
          current_prediction = self.prediction_current(df)
          shadow_prediction = self.predict_shadow(df)
          return {result: current_prediction, shadow_result: shadow_prediction}

Is the correct thing to do actually the below:

return [{result: current_prediction, shadow_result: shadow_prediction}, ]

[parano]

Hi @jiyer2016 - you are absolutely right on this. The example @yubozhao provided will not work with the current version of BentoML. It should be returning a list as you pointed out.

Btw, with the revamped adapters implementation coming out next release #1056, the possible return value of API function is more clearly defined and gives users more flexibility to customize the HTTP response, while working with micro-batching mechanism in BentoML.

[yubozhao]

Yes, I missed the [] around return value. I will update the comment in that discussion. Thank you for spotting it.