CUDA error: out of memory

[zarkua]

RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Interrogate Request!
Network: [interrogator_network: "BLIP"
]
File size: 20132
Unloading interrogator blip2-opt-6.7b
Exception processing item!
Exception string: 'CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
'
Traceback (most recent call last):
File "E:\Downloads\BDTM_V2.0.3_NetCore60_AllInclude\interrogator_rpc\main.py", line 142, in InterrogateImage
tag_ret = interrogate_image(network_conf.interrogator_network, image_obj, skip_online=request.skip_internet_requests)
File "E:\Downloads\BDTM_V2.0.3_NetCore60_AllInclude\interrogator_rpc\main.py", line 57, in interrogate_image
interg.stop()
File "E:\Downloads\BDTM_V2.0.3_NetCore60_AllInclude\interrogator_rpc\ext_kohya\captioning.py", line 55, in stop
self.interrogator.unload()
File "E:\Downloads\BDTM_V2.0.3_NetCore60_AllInclude\interrogator_rpc\ext_kohya\interrogators\blip2_captioning.py", line 30, in unload
devices.torch_gc()
File "E:\Downloads\BDTM_V2.0.3_NetCore60_AllInclude\interrogator_rpc\ext_kohya\devices.py", line 67, in torch_gc
torch.cuda.empty_cache()
File "C:\Users\George\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\cuda\memory.py", line 159, in empty_cache
torch._C._cuda_emptyCache()
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.