Quickstart Deploy a Transformer model with BentoML 500 Error TypeError: forward() got an unexpected keyword argument 'max_length'

[naidrr]

I am trying to do the Quickstart https://docs.bentoml.org/en/latest/quickstarts/deploy-a-transformer-model-with-bentoml.html on my Mac.

I am trying the http://0.0.0.0:3000/summarize endpoint but getting a 500 error with the following stack trace:

2023-11-23T16:58:21-0500 [ERROR] [api_server:6] Exception on /summarize [POST] (trace=b1b81e844a57d2ef282e588ff34b2b45,span=f41cdbaaeb697260,sampled=0,service.name=summarization)
Traceback (most recent call last):
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/server/http_app.py", line 341, in api_func
    output = await api.func(*args)
  File "/Users/n/aidasher/bentoml/service.py", line 11, in summarize
    generated = await summarizer_runner.async_run(text, max_length=3000)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/runner/runner.py", line 55, in async_run
    return await self.runner._runner_handle.async_run_method(self, *args, **kwargs)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/runner/runner_handle/remote.py", line 242, in async_run_method
    raise RemoteException(
bentoml.exceptions.RemoteException: An unexpected exception occurred in remote runner summarization: [500] Traceback (most recent call last):
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/server/http/traffic.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 596, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/server/http/instruments.py", line 252, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/server/http/access.py", line 126, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/server/runner_app.py", line 306, in _request_handler
    payload = await infer(params)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/marshal/dispatcher.py", line 182, in _func
    raise r
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/marshal/dispatcher.py", line 382, in outbound_call
    outputs = await self.callback(
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/server/runner_app.py", line 281, in infer_single
    ret = await runner_method.async_run(
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/runner/runner.py", line 55, in async_run
    return await self.runner._runner_handle.async_run_method(self, *args, **kwargs)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 62, in async_run_method
    return await anyio.to_thread.run_sync(
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/anyio/_backends/_asyncio.py", line 2106, in run_sync_in_worker_thread
    return await future
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/anyio/_backends/_asyncio.py", line 833, in run
    result = context.run(func, *args)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/runner/runnable.py", line 143, in method
    return self.func(obj, *args, **kwargs)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/bentoml/_internal/frameworks/transformers.py", line 1201, in _run
    return self.predict_fns[method_name](*args, **kwargs)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/n/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'max_length'

I am running Python 3.9.6, pip 23.3.1, torch version 2.1.1, transformers v 4.35.2, and bentoml version 1.1.10. What could be the issue? (my spidey sense tells me could be my version of torch breaks something in bentoml but I'm totally new to bentoml so I wouldn't know).

[aarnphm]

You will need to change max_length to max_new_tokens. I will update the repo to use the newer arguments. This is from transformers.