Runpodgives me an error #650

8bit-boom · 2025-01-21T19:21:50Z

8bit-boom
Jan 21, 2025

I try to run cloud trainig, but I get this error

"activating venv D:\SD\Onetrainer\OneTrainer\venv
Using Python "D:\SD\Onetrainer\OneTrainer\venv\Scripts\python.exe"
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
waiting for public IP... Status: https://www.runpod.io/console/pods/8qf0wmgwr0kk1f
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 485, in connect
self._auth(
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 818, in _auth
raise saved_exception
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 794, in _auth
self._transport.auth_publickey(username, key)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 1709, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 485, in connect
self._auth(
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 818, in _auth
raise saved_exception
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 794, in _auth
self._transport.auth_publickey(username, key)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 1709, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 485, in connect
self._auth(
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 818, in _auth
raise saved_exception
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 794, in _auth
self._transport.auth_publickey(username, key)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 1709, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 485, in connect
self._auth(
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 818, in _auth
raise saved_exception
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 794, in _auth
self._transport.auth_publickey(username, key)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 1709, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 485, in connect
self._auth(
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 818, in _auth
raise saved_exception
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 794, in _auth
self._transport.auth_publickey(username, key)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 1709, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 485, in connect
self._auth(
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 818, in _auth
raise saved_exception
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 794, in _auth
self._transport.auth_publickey(username, key)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 1709, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\auth_handler.py", line 263, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed."

dxqbYD · 2025-01-21T20:21:36Z

dxqbYD
Jan 21, 2025
Collaborator

User is empty
the default and the correct value for runpods is root. Only change it if you use other linux servers where you have a different username.

3 replies

8bit-boom Jan 22, 2025
Author

Thanks, added "root" but now I get this errror

"activating venv D:\SD\Onetrainer\OneTrainer\venv
Using Python "D:\SD\Onetrainer\OneTrainer\venv\Scripts\python.exe"
transport.py :1944 2025-01-22 06:39:13,081 Exception (client): Error reading SSH protocol banner
transport.py :1942 2025-01-22 06:39:13,105 Traceback (most recent call last):
transport.py :1942 2025-01-22 06:39:13,105 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2369, in _check_banner
transport.py :1942 2025-01-22 06:39:13,105 buf = self.packetizer.readline(timeout)
transport.py :1942 2025-01-22 06:39:13,105 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 395, in readline
transport.py :1942 2025-01-22 06:39:13,105 buf += self._read_timeout(timeout)
transport.py :1942 2025-01-22 06:39:13,105 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 673, in _read_timeout
transport.py :1942 2025-01-22 06:39:13,105 raise socket.timeout()
transport.py :1942 2025-01-22 06:39:13,105 TimeoutError
transport.py :1942 2025-01-22 06:39:13,106
transport.py :1942 2025-01-22 06:39:13,106 During handling of the above exception, another exception occurred:
transport.py :1942 2025-01-22 06:39:13,106
transport.py :1942 2025-01-22 06:39:13,106 Traceback (most recent call last):
transport.py :1942 2025-01-22 06:39:13,106 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2185, in run
transport.py :1942 2025-01-22 06:39:13,106 self._check_banner()
transport.py :1942 2025-01-22 06:39:13,106 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2373, in _check_banner
transport.py :1942 2025-01-22 06:39:13,106 raise SSHException(
transport.py :1942 2025-01-22 06:39:13,106 paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
transport.py :1942 2025-01-22 06:39:13,106
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2369, in _check_banner
buf = self.packetizer.readline(timeout)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 395, in readline
buf += self._read_timeout(timeout)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 673, in _read_timeout
raise socket.timeout()
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 451, in connect
t.start_client(timeout=timeout)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 773, in start_client
raise e
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2185, in run
self._check_banner()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2373, in _check_banner
raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
transport.py :1944 2025-01-22 06:39:39,188 Exception (client): Error reading SSH protocol banner
transport.py :1942 2025-01-22 06:39:39,188 Traceback (most recent call last):
transport.py :1942 2025-01-22 06:39:39,189 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2369, in _check_banner
transport.py :1942 2025-01-22 06:39:39,190 buf = self.packetizer.readline(timeout)
transport.py :1942 2025-01-22 06:39:39,190 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 395, in readline
transport.py :1942 2025-01-22 06:39:39,190 buf += self._read_timeout(timeout)
transport.py :1942 2025-01-22 06:39:39,190 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 673, in _read_timeout
transport.py :1942 2025-01-22 06:39:39,190 raise socket.timeout()
transport.py :1942 2025-01-22 06:39:39,190 TimeoutError
transport.py :1942 2025-01-22 06:39:39,190
transport.py :1942 2025-01-22 06:39:39,190 During handling of the above exception, another exception occurred:
transport.py :1942 2025-01-22 06:39:39,190
transport.py :1942 2025-01-22 06:39:39,190 Traceback (most recent call last):
transport.py :1942 2025-01-22 06:39:39,190 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2185, in run
transport.py :1942 2025-01-22 06:39:39,190 self._check_banner()
transport.py :1942 2025-01-22 06:39:39,190 File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2373, in _check_banner
transport.py :1942 2025-01-22 06:39:39,190 raise SSHException(
transport.py :1942 2025-01-22 06:39:39,190 paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
transport.py :1942 2025-01-22 06:39:39,191
Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2369, in _check_banner
buf = self.packetizer.readline(timeout)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 395, in readline
buf += self._read_timeout(timeout)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\packet.py", line 673, in _read_timeout
raise socket.timeout()
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\SD\Onetrainer\OneTrainer\modules\ui\TrainUI.py", line 569, in __training_thread_function
trainer.start()
File "D:\SD\Onetrainer\OneTrainer\modules\trainer\CloudTrainer.py", line 45, in start
self.cloud.setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 73, in setup
super().setup()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\BaseCloud.py", line 20, in setup
self._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\RunpodCloud.py", line 63, in _connect
super()._connect()
File "D:\SD\Onetrainer\OneTrainer\modules\cloud\LinuxCloud.py", line 45, in _connect
self.connection.open()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\fabric\connection.py", line 665, in open
result = self.client.connect(**kwargs)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\client.py", line 451, in connect
t.start_client(timeout=timeout)
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 773, in start_client
raise e
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2185, in run
self._check_banner()
File "D:\SD\Onetrainer\OneTrainer\venv\lib\site-packages\paramiko\transport.py", line 2373, in _check_banner
raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner"

dxqbYD Jan 22, 2025
Collaborator

Haven't seen this one before. Have you created your SSH key pair and added your key on the runpod website?
How to do that: https://github.com/Nerogar/OneTrainer/wiki/Cloud-Training

8bit-boom Jan 22, 2025
Author

Yes I did. I restarted Pod and it start seemingly working but now it get to certain point and just stops, nothing happens. I try to finetune flux model by the way.

"activating venv D:\SD\Onetrainer\OneTrainer\venv
Using Python "D:\SD\Onetrainer\OneTrainer\venv\Scripts\python.exe"
[OneTrainer] Updating OneTrainer to latest version from Git repository...
Already up to date.
[OneTrainer] Using Python Venv environment in "venv"...
[OneTrainer] + python scripts/util/version_check.py 3.10 3.13
[OneTrainer] Installing requirements in active environment...
[OneTrainer] + python -m pip install --upgrade --upgrade-strategy eager pip setuptools
Requirement already satisfied: pip in ./venv/lib/python3.11/site-packages (24.3.1)
Requirement already satisfied: setuptools in ./venv/lib/python3.11/site-packages (75.8.0)
[OneTrainer] + python -m pip install --upgrade --upgrade-strategy eager -r requirements-global.txt -r requirements-cuda.txt
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu124
Obtaining diffusers from git+https://github.com/huggingface/diffusers.git@c944f06#egg=diffusers (from -r requirements-global.txt (line 19))
Updating ./venv/src/diffusers clone (to revision c944f06)
Running command git fetch -q --tags
WARNING: Did not find branch or tag 'c944f06', assuming revision or ref.
Running command git reset --hard -q c944f06
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Checking if build backend supports build_editable: started
Checking if build backend supports build_editable: finished with status 'done'
Getting requirements to build editable: started
Getting requirements to build editable: finished with status 'done'
Preparing editable metadata (pyproject.toml): started
Preparing editable metadata (pyproject.toml): finished with status 'done'
Obtaining mgds from git+https://github.com/Nerogar/mgds.git@fcaec25#egg=mgds (from -r requirements-global.txt (line 30))
Updating ./venv/src/mgds clone (to revision fcaec25)
Running command git fetch -q --tags
WARNING: Did not find branch or tag 'fcaec25', assuming revision or ref.
Running command git reset --hard -q fcaec25
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: numpy==1.26.4 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 2)) (1.26.4)
Requirement already satisfied: opencv-python==4.10.0.84 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 3)) (4.10.0.84)
Requirement already satisfied: pillow==11.0.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 4)) (11.0.0)
Requirement already satisfied: tqdm==4.66.6 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 5)) (4.66.6)
Requirement already satisfied: PyYAML==6.0.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 6)) (6.0.2)
Requirement already satisfied: huggingface-hub==0.27.1 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 7)) (0.27.1)
Requirement already satisfied: scipy==1.14.1 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 8)) (1.14.1)
Requirement already satisfied: matplotlib==3.9.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 9)) (3.9.2)
Requirement already satisfied: av==13.1.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 10)) (13.1.0)
Requirement already satisfied: accelerate==1.0.1 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 13)) (1.0.1)
Requirement already satisfied: safetensors==0.4.5 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 14)) (0.4.5)
Requirement already satisfied: tensorboard==2.18.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 15)) (2.18.0)
Requirement already satisfied: pytorch-lightning==2.4.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 16)) (2.4.0)
Requirement already satisfied: transformers==4.47.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 20)) (4.47.0)
Requirement already satisfied: sentencepiece==0.2.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 21)) (0.2.0)
Requirement already satisfied: omegaconf==2.3.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 22)) (2.3.0)
Requirement already satisfied: invisible-watermark==0.2.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 23)) (0.2.0)
Requirement already satisfied: pooch==1.8.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 26)) (1.8.2)
Requirement already satisfied: open-clip-torch==2.28.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 27)) (2.28.0)
Requirement already satisfied: dadaptation==3.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 33)) (3.2)
Requirement already satisfied: lion-pytorch==0.2.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 34)) (0.2.2)
Requirement already satisfied: prodigyopt==1.1.1 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 35)) (1.1.1)
Requirement already satisfied: schedulefree==1.3.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 36)) (1.3)
Requirement already satisfied: pytorch_optimizer==3.3.0 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 37)) (3.3.0)
Requirement already satisfied: scalene==1.5.45 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 40)) (1.5.45)
Requirement already satisfied: customtkinter==5.2.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 43)) (5.2.2)
Requirement already satisfied: runpod==1.7.4 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 46)) (1.7.4)
Requirement already satisfied: fabric==3.2.2 in ./venv/lib/python3.11/site-packages (from -r requirements-global.txt (line 47)) (3.2.2)
Requirement already satisfied: torch==2.5.1+cu124 in ./venv/lib/python3.11/site-packages (from -r requirements-cuda.txt (line 3)) (2.5.1+cu124)
Requirement already satisfied: torchvision==0.20.1+cu124 in ./venv/lib/python3.11/site-packages (from -r requirements-cuda.txt (line 4)) (0.20.1+cu124)
Requirement already satisfied: onnxruntime-gpu==1.19.2 in ./venv/lib/python3.11/site-packages (from -r requirements-cuda.txt (line 5)) (1.19.2)
Requirement already satisfied: xformers==0.0.28.post3 in ./venv/lib/python3.11/site-packages (from -r requirements-cuda.txt (line 8)) (0.0.28.post3)
Requirement already satisfied: bitsandbytes==0.44.1 in ./venv/lib/python3.11/site-packages (from -r requirements-cuda.txt (line 11)) (0.44.1)
Requirement already satisfied: filelock in ./venv/lib/python3.11/site-packages (from huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (3.17.0)
Requirement already satisfied: fsspec>=2023.5.0 in ./venv/lib/python3.11/site-packages (from huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (2024.12.0)
Requirement already satisfied: packaging>=20.9 in ./venv/lib/python3.11/site-packages (from huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (24.2)
Requirement already satisfied: requests in ./venv/lib/python3.11/site-packages (from huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (2.32.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./venv/lib/python3.11/site-packages (from huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (4.12.2)
Requirement already satisfied: contourpy>=1.0.1 in ./venv/lib/python3.11/site-packages (from matplotlib==3.9.2->-r requirements-global.txt (line 9)) (1.3.1)
Requirement already satisfied: cycler>=0.10 in ./venv/lib/python3.11/site-packages (from matplotlib==3.9.2->-r requirements-global.txt (line 9)) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in ./venv/lib/python3.11/site-packages (from matplotlib==3.9.2->-r requirements-global.txt (line 9)) (4.55.4)
Requirement already satisfied: kiwisolver>=1.3.1 in ./venv/lib/python3.11/site-packages (from matplotlib==3.9.2->-r requirements-global.txt (line 9)) (1.4.8)
Requirement already satisfied: pyparsing>=2.3.1 in ./venv/lib/python3.11/site-packages (from matplotlib==3.9.2->-r requirements-global.txt (line 9)) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in ./venv/lib/python3.11/site-packages (from matplotlib==3.9.2->-r requirements-global.txt (line 9)) (2.9.0.post0)
Requirement already satisfied: psutil in ./venv/lib/python3.11/site-packages (from accelerate==1.0.1->-r requirements-global.txt (line 13)) (6.1.1)
Requirement already satisfied: absl-py>=0.4 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (2.1.0)
Requirement already satisfied: grpcio>=1.48.2 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (1.69.0)
Requirement already satisfied: markdown>=2.6.8 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (3.7)
Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (5.29.3)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (75.8.0)
Requirement already satisfied: six>1.9 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (1.17.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in ./venv/lib/python3.11/site-packages (from tensorboard==2.18.0->-r requirements-global.txt (line 15)) (3.1.3)
Requirement already satisfied: torchmetrics>=0.7.0 in ./venv/lib/python3.11/site-packages (from pytorch-lightning==2.4.0->-r requirements-global.txt (line 16)) (1.6.1)
Requirement already satisfied: lightning-utilities>=0.10.0 in ./venv/lib/python3.11/site-packages (from pytorch-lightning==2.4.0->-r requirements-global.txt (line 16)) (0.11.9)
Requirement already satisfied: regex!=2019.12.17 in ./venv/lib/python3.11/site-packages (from transformers==4.47.0->-r requirements-global.txt (line 20)) (2024.11.6)
Requirement already satisfied: tokenizers<0.22,>=0.21 in ./venv/lib/python3.11/site-packages (from transformers==4.47.0->-r requirements-global.txt (line 20)) (0.21.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in ./venv/lib/python3.11/site-packages (from omegaconf==2.3.0->-r requirements-global.txt (line 22)) (4.9.3)
Requirement already satisfied: PyWavelets>=1.1.1 in ./venv/lib/python3.11/site-packages (from invisible-watermark==0.2.0->-r requirements-global.txt (line 23)) (1.8.0)
Requirement already satisfied: platformdirs>=2.5.0 in ./venv/lib/python3.11/site-packages (from pooch==1.8.2->-r requirements-global.txt (line 26)) (4.3.6)
Requirement already satisfied: ftfy in ./venv/lib/python3.11/site-packages (from open-clip-torch==2.28.0->-r requirements-global.txt (line 27)) (6.3.1)
Requirement already satisfied: timm in ./venv/lib/python3.11/site-packages (from open-clip-torch==2.28.0->-r requirements-global.txt (line 27)) (1.0.14)
Requirement already satisfied: wheel>=0.36.1 in ./venv/lib/python3.11/site-packages (from scalene==1.5.45->-r requirements-global.txt (line 40)) (0.45.1)
Requirement already satisfied: rich>=10.7.0 in ./venv/lib/python3.11/site-packages (from scalene==1.5.45->-r requirements-global.txt (line 40)) (13.9.4)
Requirement already satisfied: cloudpickle>=2.2.1 in ./venv/lib/python3.11/site-packages (from scalene==1.5.45->-r requirements-global.txt (line 40)) (3.1.1)
Requirement already satisfied: Jinja2>=3.0.3 in ./venv/lib/python3.11/site-packages (from scalene==1.5.45->-r requirements-global.txt (line 40)) (3.1.5)
Requirement already satisfied: pydantic>=2.6 in ./venv/lib/python3.11/site-packages (from scalene==1.5.45->-r requirements-global.txt (line 40)) (2.10.5)
Requirement already satisfied: nvidia-ml-py>=12.555.43 in ./venv/lib/python3.11/site-packages (from scalene==1.5.45->-r requirements-global.txt (line 40)) (12.560.30)
Requirement already satisfied: darkdetect in ./venv/lib/python3.11/site-packages (from customtkinter==5.2.2->-r requirements-global.txt (line 43)) (0.8.0)
Requirement already satisfied: aiohttp>=3.9.3 in ./venv/lib/python3.11/site-packages (from aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (3.11.11)
Requirement already satisfied: aiohttp-retry>=2.8.3 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (2.9.1)
Requirement already satisfied: backoff>=2.2.1 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (2.2.1)
Requirement already satisfied: boto3>=1.26.165 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (1.36.3)
Requirement already satisfied: click>=8.1.7 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (8.1.8)
Requirement already satisfied: colorama<0.4.7,>=0.2.5 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (0.4.6)
Requirement already satisfied: cryptography<44.0.0 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (43.0.3)
Requirement already satisfied: fastapi>=0.94.0 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.115.6)
Requirement already satisfied: paramiko>=3.3.1 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (3.5.0)
Requirement already satisfied: prettytable>=3.9.0 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (3.12.0)
Requirement already satisfied: py-cpuinfo>=9.0.0 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (9.0.0)
Requirement already satisfied: inquirerpy==0.3.4 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (0.3.4)
Requirement already satisfied: tomli>=2.0.1 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (2.2.1)
Requirement already satisfied: tomlkit>=0.12.2 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (0.13.2)
Requirement already satisfied: tqdm-loggable>=0.1.4 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (0.2)
Requirement already satisfied: urllib3>=1.26.6 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (2.3.0)
Requirement already satisfied: watchdog>=3.0.0 in ./venv/lib/python3.11/site-packages (from runpod==1.7.4->-r requirements-global.txt (line 46)) (6.0.0)
Requirement already satisfied: invoke>=2.0 in ./venv/lib/python3.11/site-packages (from fabric==3.2.2->-r requirements-global.txt (line 47)) (2.2.0)
Requirement already satisfied: decorator>=5 in ./venv/lib/python3.11/site-packages (from fabric==3.2.2->-r requirements-global.txt (line 47)) (5.1.1)
Requirement already satisfied: deprecated>=1.2 in ./venv/lib/python3.11/site-packages (from fabric==3.2.2->-r requirements-global.txt (line 47)) (1.2.15)
Requirement already satisfied: networkx in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (3.4.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.3.1.170)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (12.4.127)
Requirement already satisfied: triton==3.1.0 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (3.1.0)
Requirement already satisfied: sympy==1.13.1 in ./venv/lib/python3.11/site-packages (from torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (1.13.1)
Requirement already satisfied: coloredlogs in ./venv/lib/python3.11/site-packages (from onnxruntime-gpu==1.19.2->-r requirements-cuda.txt (line 5)) (15.0.1)
Requirement already satisfied: flatbuffers in ./venv/lib/python3.11/site-packages (from onnxruntime-gpu==1.19.2->-r requirements-cuda.txt (line 5)) (25.1.21)
Requirement already satisfied: pfzy<0.4.0,>=0.3.1 in ./venv/lib/python3.11/site-packages (from inquirerpy==0.3.4->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.3.4)
Requirement already satisfied: prompt-toolkit<4.0.0,>=3.0.1 in ./venv/lib/python3.11/site-packages (from inquirerpy==0.3.4->runpod==1.7.4->-r requirements-global.txt (line 46)) (3.0.50)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./venv/lib/python3.11/site-packages (from sympy==1.13.1->torch==2.5.1+cu124->-r requirements-cuda.txt (line 3)) (1.3.0)
Requirement already satisfied: importlib_metadata in ./venv/lib/python3.11/site-packages (from diffusers->-r requirements-global.txt (line 19)) (8.6.1)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.4.4)
Requirement already satisfied: aiosignal>=1.1.2 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.3.2)
Requirement already satisfied: attrs>=17.3.0 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (24.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.9.3->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.18.3)
Requirement already satisfied: aiodns>=3.2.0 in ./venv/lib/python3.11/site-packages (from aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (3.2.0)
Requirement already satisfied: Brotli in ./venv/lib/python3.11/site-packages (from aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.1.0)
Requirement already satisfied: botocore<1.37.0,>=1.36.3 in ./venv/lib/python3.11/site-packages (from boto3>=1.26.165->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.36.3)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./venv/lib/python3.11/site-packages (from boto3>=1.26.165->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.0.1)
Requirement already satisfied: s3transfer<0.12.0,>=0.11.0 in ./venv/lib/python3.11/site-packages (from boto3>=1.26.165->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.11.1)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.11/site-packages (from cryptography<44.0.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.17.1)
Requirement already satisfied: wrapt<2,>=1.10 in ./venv/lib/python3.11/site-packages (from deprecated>=1.2->fabric==3.2.2->-r requirements-global.txt (line 47)) (1.17.2)
Requirement already satisfied: starlette<0.42.0,>=0.40.0 in ./venv/lib/python3.11/site-packages (from fastapi>=0.94.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.41.3)
Requirement already satisfied: fastapi-cli>=0.0.5 in ./venv/lib/python3.11/site-packages (from fastapi-cli[standard]>=0.0.5; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.0.7)
Requirement already satisfied: httpx>=0.23.0 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.28.1)
Requirement already satisfied: python-multipart>=0.0.7 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.0.20)
Requirement already satisfied: itsdangerous>=1.1.0 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.2.0)
Requirement already satisfied: ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (5.10.0)
Requirement already satisfied: orjson>=3.2.1 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (3.10.15)
Requirement already satisfied: email-validator>=2.0.0 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.2.0)
Requirement already satisfied: uvicorn>=0.12.0 in ./venv/lib/python3.11/site-packages (from uvicorn[standard]>=0.12.0; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.34.0)
Requirement already satisfied: pydantic-settings>=2.0.0 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.7.1)
Requirement already satisfied: pydantic-extra-types>=2.0.0 in ./venv/lib/python3.11/site-packages (from fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.10.2)
Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.11/site-packages (from Jinja2>=3.0.3->scalene==1.5.45->-r requirements-global.txt (line 40)) (3.0.2)
Requirement already satisfied: bcrypt>=3.2 in ./venv/lib/python3.11/site-packages (from paramiko>=3.3.1->runpod==1.7.4->-r requirements-global.txt (line 46)) (4.2.1)
Requirement already satisfied: pynacl>=1.5 in ./venv/lib/python3.11/site-packages (from paramiko>=3.3.1->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.5.0)
Requirement already satisfied: wcwidth in ./venv/lib/python3.11/site-packages (from prettytable>=3.9.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.2.13)
Requirement already satisfied: annotated-types>=0.6.0 in ./venv/lib/python3.11/site-packages (from pydantic>=2.6->scalene==1.5.45->-r requirements-global.txt (line 40)) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in ./venv/lib/python3.11/site-packages (from pydantic>=2.6->scalene==1.5.45->-r requirements-global.txt (line 40)) (2.27.2)
Requirement already satisfied: charset-normalizer<4,>=2 in ./venv/lib/python3.11/site-packages (from requests->huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.11/site-packages (from requests->huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.11/site-packages (from requests->huggingface-hub==0.27.1->-r requirements-global.txt (line 7)) (2024.12.14)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./venv/lib/python3.11/site-packages (from rich>=10.7.0->scalene==1.5.45->-r requirements-global.txt (line 40)) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./venv/lib/python3.11/site-packages (from rich>=10.7.0->scalene==1.5.45->-r requirements-global.txt (line 40)) (2.19.1)
Requirement already satisfied: humanfriendly>=9.1 in ./venv/lib/python3.11/site-packages (from coloredlogs->onnxruntime-gpu==1.19.2->-r requirements-cuda.txt (line 5)) (10.0)
Requirement already satisfied: zipp>=3.20 in ./venv/lib/python3.11/site-packages (from importlib_metadata->diffusers->-r requirements-global.txt (line 19)) (3.21.0)
Requirement already satisfied: pycares>=4.0.0 in ./venv/lib/python3.11/site-packages (from aiodns>=3.2.0->aiohttp[speedups]>=3.9.3->runpod==1.7.4->-r requirements-global.txt (line 46)) (4.5.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.11/site-packages (from cffi>=1.12->cryptography<44.0.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.22)
Requirement already satisfied: dnspython>=2.0.0 in ./venv/lib/python3.11/site-packages (from email-validator>=2.0.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (2.7.0)
Requirement already satisfied: typer>=0.12.3 in ./venv/lib/python3.11/site-packages (from fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.15.1)
Requirement already satisfied: rich-toolkit>=0.11.1 in ./venv/lib/python3.11/site-packages (from fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.13.2)
Requirement already satisfied: anyio in ./venv/lib/python3.11/site-packages (from httpx>=0.23.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (4.8.0)
Requirement already satisfied: httpcore==1.* in ./venv/lib/python3.11/site-packages (from httpx>=0.23.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.0.7)
Requirement already satisfied: h11<0.15,>=0.13 in ./venv/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.23.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.14.0)
Requirement already satisfied: mdurl~=0.1 in ./venv/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=10.7.0->scalene==1.5.45->-r requirements-global.txt (line 40)) (0.1.2)
Requirement already satisfied: python-dotenv>=0.21.0 in ./venv/lib/python3.11/site-packages (from pydantic-settings>=2.0.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.0.1)
Requirement already satisfied: httptools>=0.6.3 in ./venv/lib/python3.11/site-packages (from uvicorn[standard]>=0.12.0; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.6.4)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in ./venv/lib/python3.11/site-packages (from uvicorn[standard]>=0.12.0; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (0.21.0)
Requirement already satisfied: watchfiles>=0.13 in ./venv/lib/python3.11/site-packages (from uvicorn[standard]>=0.12.0; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.0.4)
Requirement already satisfied: websockets>=10.4 in ./venv/lib/python3.11/site-packages (from uvicorn[standard]>=0.12.0; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (14.2)
Requirement already satisfied: sniffio>=1.1 in ./venv/lib/python3.11/site-packages (from anyio->httpx>=0.23.0->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.3.1)
Requirement already satisfied: shellingham>=1.3.0 in ./venv/lib/python3.11/site-packages (from typer>=0.12.3->fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "all"->fastapi[all]>=0.94.0->runpod==1.7.4->-r requirements-global.txt (line 46)) (1.5.4)
Building wheels for collected packages: diffusers
Building editable for diffusers (pyproject.toml): started
Building editable for diffusers (pyproject.toml): finished with status 'done'
Created wheel for diffusers: filename=diffusers-0.33.0.dev0-0.editable-py3-none-any.whl size=11232 sha256=a7424f18aa956d1b4bd251343ea66d8bd33423ff75133835aaf9d38729a9f693
Stored in directory: /tmp/pip-ephem-wheel-cache-n1x7cp9v/wheels/8b/de/6d/3a0c11d0e1dd9ebea74a1e0e5f8614065d3cf6d23bebd6d2ca
Successfully built diffusers
Installing collected packages: mgds, diffusers
Attempting uninstall: mgds
Found existing installation: mgds 0.1
Uninstalling mgds-0.1:
Successfully uninstalled mgds-0.1
DEPRECATION: Legacy editable install of mgds from git+https://github.com/Nerogar/mgds.git@fcaec25#egg=mgds (from -r requirements-global.txt (line 30)) (setup.py develop) is deprecated. pip 25.0 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at pypa/pip#11457
Running setup.py develop for mgds
Attempting uninstall: diffusers
Found existing installation: diffusers 0.33.0.dev0
Uninstalling diffusers-0.33.0.dev0:
Successfully uninstalled diffusers-0.33.0.dev0
Successfully installed diffusers-0.33.0.dev0 mgds-0.1
remote_config-2025-01-22_06-48-49.json 100% 28KB 130.4KB/s 00:00
uploading concept 1...
uploading concept 2...
uploading concept 3...
uploading concept 4...
uploading concept 5...
uploading concept 6...
[OneTrainer] Using Python Venv environment in "venv"...
[OneTrainer] + python scripts/util/version_check.py 3.10 3.13
[OneTrainer] + python scripts/train_remote.py --config-path=/workspace/2025-01-22_06-48-15.json --callback-path=/workspace/2025-01-22_06-48-15.callback --command-path=/workspace/2025-01-22_06-48-15.command
Clearing cache directory /workspace/remote/workspace-cache/run! You can disable this if you want to continue using the same cache.
TensorFlow installation not found - running with reduced feature set.

NOTE: Using experimental fast data loading logic. To disable, pass
"--load_fast=false" and report issues on GitHub. More details:
tensorflow/tensorboard#4784

Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.18.0 at http://localhost:6009/ (Press CTRL+C to quit)
2025-01-22_05-49-04.json 100% 27KB 85.1KB/s 00:00"

My model setting

8bit-boom · 2025-01-22T05:55:39Z

8bit-boom
Jan 22, 2025
Author

Nevermind, it start working, just took long time for it.

1 reply

dxqbYD Jan 22, 2025
Collaborator

first time takes a while because it downloads 30 GB Flux model, and there is no progress bar for the huggingface download

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runpodgives me an error #650

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Runpodgives me an error #650

8bit-boom Jan 21, 2025

Replies: 2 comments · 4 replies

dxqbYD Jan 21, 2025 Collaborator

8bit-boom Jan 22, 2025 Author

dxqbYD Jan 22, 2025 Collaborator

8bit-boom Jan 22, 2025 Author

8bit-boom Jan 22, 2025 Author

dxqbYD Jan 22, 2025 Collaborator

8bit-boom
Jan 21, 2025

Replies: 2 comments 4 replies

dxqbYD
Jan 21, 2025
Collaborator

8bit-boom Jan 22, 2025
Author

dxqbYD Jan 22, 2025
Collaborator

8bit-boom Jan 22, 2025
Author

8bit-boom
Jan 22, 2025
Author

dxqbYD Jan 22, 2025
Collaborator