通过fastchat部署本地模型遇到的问题

[YinSonglin1997]

您好，我使用fastchat进行加载chatglm3-6b模型，
step1 python3 -m fastchat.serve.controller
step2 python3 -m fastchat.serve.model_worker --model-path /ldata/llms/chatglm3-6b
step3 python3 -m fastchat.serve.openai_api_server --host 10.0.1.227 --port 30008
启动服务后，我修改了fs_agents.yaml文件，内容为

default:
  module: "src.client.agents.FastChatAgent"
  parameters:
    name: "FastChat"
    controller_address: "http://10.0.1.227:30008"
    max_new_tokens: 512
    temperature: 0

chatglm3-6b:
  parameters:
    model_name: "chatglm3-6b"

然后运行 python -m src.client.agent_test --config configs/agents/fs_agents.yaml --agent chatglm3-6b
接着新开了一个终端运行命令 python -m src.start_task -a，此时报错
mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:16018' (111)
请问这个问题怎么解决？还有请问我的步骤以及yaml文件有问题吗？期待您的回复！

[lambda7xx]

你这个问题解决了吗

[YinSonglin1997]

controller_address地址写错了，改为python3 -m fastchat.serve.controller的地址；
将configs/assignments/default.yaml中agent名字修改。

[lambda7xx]

那这个python3 -m fastchat.serve.controller的地址，是多少

[lambda7xx]

这个control-address我这里应该是http://localhost:21002

那python3 -m fastchat.serve.openai_api_server --host 10.0.1.227 --port 30008这个应该怎么搞,host和port不变吗

[YinSonglin1997]

这个control-address我这里应该是http://localhost:21002 那python3 -m fastchat.serve.openai_api_server --host 10.0.1.227 --port 30008这个应该怎么搞,host和port不变吗

host是你服务器的地址，port你可以自定义

[lambda7xx]

我试了下llama3-8b

1 terminal 1: python3 -m fastchat.serve.controller
2 terminal 2:python3 -m fastchat.serve.model_worker --model-path /data/llama3/Meta-Llama-3-8B-hf
3 termain 3: python3 -m fastchat.serve.openai_api_server --host 127.0.0.1 --port 30008
4 fs_agent.yaml

default:
  module: "src.client.agents.FastChatAgent"
  parameters:
    name: "FastChat"
    controller_address: "http://localhost:21002"
    max_new_tokens: 512
    temperature: 0

vicuna-33b:
  parameters:
    model_name: "vicuna-33b-v1.3"

wizard-30b:
  parameters:
    model_name: "WizardLM-30B-V1.0-merged"

vicuna-13b:
  parameters:
    model_name: "vicuna-13b-v1.5"

vicuna-7b:
  parameters:
    model_name: "vicuna-7b-v1.5"

llama3-8b:
  parameters:
    model_name: "llama3-8b"

5 terminal 4: python -m src.client.agent_test --config configs/agents/fs_agent.yaml --agent llama3-8b

6 terminal 5: python -m src.start_task -a

7 termina 6:python -m src.assigner

报错

/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
Traceback (most recent call last):
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/xxxx/AgentBench/src/assigner.py", line 422, in <module>
    value = AssignmentConfig.post_validate(value)
  File "/home/xxxx/AgentBench/src/typings/config.py", line 79, in post_validate
    assert (
AssertionError: Agent llam3-8b is not defined.

[YinSonglin1997]

configs/assignments/default.yaml中agent名字改掉了吗？

[lambda7xx]

configs/assignments/default.yaml是要改成这样吗

import: definition.yaml

concurrency:
  task:
    dbbench-std: 5
    os-std: 5
  agent:
    llama3-8b: 5 #todo(xiao):this can changed to other models 

assignments: # List[Assignment] | Assignment
  - agent: # "task": List[str] | str ,  "agent": List[str] | str
      - llama3-8b #todo(xiao):this can changed to other models 
    task:
      - dbbench-std
      - os-std

output: "outputs/{TIMESTAMP}"

[YinSonglin1997]

configs/assignments/default.yaml是要改成这样吗

import: definition.yaml

concurrency:
  task:
    dbbench-std: 5
    os-std: 5
  agent:
    llama3-8b: 5 #todo(xiao):this can changed to other models 

assignments: # List[Assignment] | Assignment
  - agent: # "task": List[str] | str ,  "agent": List[str] | str
      - llama3-8b #todo(xiao):this can changed to other models 
    task:
      - dbbench-std
      - os-std

output: "outputs/{TIMESTAMP}"

是的，能跑起来了吗？

[lambda7xx]

我在 terminal 5: python -m src.start_task -a报的错是

mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:13563' (111)
Traceback (most recent call last):
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 291, in _open_connection
    self._cmysql.connect(**cnx_kwargs)
_mysql_connector.MySQLInterfaceError: Can't connect to MySQL server on '127.0.0.1:14585' (111)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/xxxx/AgentBench/src/server/task_worker.py", line 256, in <module>
    asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create()
  File "/home/xxxx/AgentBench/src/typings/general.py", line 35, in create
    return getattr(mod, self.module.split(".")[-1])(**self.parameters)
  File "/home/xxxx/AgentBench/src/server/tasks/dbbench/__init__.py", line 82, in __init__
    self.container = Container()
  File "/home/xxxx/AgentBench/src/server/tasks/dbbench/Interaction.py", line 40, in __init__
    self.conn = mysql.connector.connect(
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 280, in connect
    return _get_pooled_connection(**kwargs)
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 90, in _get_pooled_connection
    _CONNECTION_POOLS[pool_name] = MySQLConnectionPool(**kwargs)
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 430, in __init__
    self.add_connection()
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 538, in add_connection
    cnx = connect(**self._cnx_config)  # type: ignore[assignment]
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 293, in connect
    return CMySQLConnection(*args, **kwargs)
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 120, in __init__
    self.connect(**kwargs)
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/abstracts.py", line 1181, in connect
    self._open_connection()
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 296, in _open_connection
    raise get_mysql_exception(
mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:14585' (111)

[lambda7xx]

在 termina 6:python -m src.assigner报的错是

python -m src.assigner
/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
Warning: 8 agent(s) and 16 task(s) are defined but not used, they will be ignored.
    Agent: {'llama3-70b', 'gpt-3.5-turbo-0613', 'text-davinci-003', 'vicuna-33b', 'text-davinci-002', 'vicuna-13b', 'vicuna-7b', 'wizard-30b'}
    Task: {'cg-dev', 'alfworld-dev', 'avalon-dev-naive', 'ltp-dev', 'kg-dev', 'avalon-dev-single', 'webshop-dev', 'm2w-std', 'cg-std', 'kg-std', 'm2w-dev', 'alfworld-std', 'dbbench-dev', 'webshop-std', 'ltp-std', 'os-dev'}
creating os-std client...
TaskClient created: os-std (http://localhost:5000/api)
creating dbbench-std client...
TaskClient created: dbbench-std (http://localhost:5000/api)
Traceback (most recent call last):
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/xxxx/anaconda3/envs/agent-bench/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/xxxx/AgentBench/src/assigner.py", line 425, in <module>
    Assigner(value, args.retry).start(tqdm_out=orig_stdout)
  File "/home/xxxx/AgentBench/src/assigner.py", line 94, in __init__
    self.task_indices[task] = self.tasks[task].get_indices()
  File "/home/xxxx/AgentBench/src/client/task.py", line 31, in get_indices
    raise AgentBenchException(result.text, result.status_code, self.name)
src.typings.exception.AgentBenchException: ('{"detail":"Error: Task does not exist"}', 400, 'dbbench-std')

[YinSonglin1997]

terminal 5: python -m src.start_task -a报了Can't connect to MySQL server on '127.0.0.1:14585' (111)，后续没有INFO: 127.0.0.1:39502 - "POST /api/receive_heartbeat HTTP/1.1" 200 OK这个吗？你后台能看到起了mysql容器吗？
建议将你的5、6、7步全部结束关闭，然后重新尝试下看看。