Race condition in downloads when running multiple jobs (manifests as 'file not found')

[ianandersonlol]

Hi! Truly truly amazing release. Congratulations!

I've been running into some issues when running multiple folds in parallel some of them fail with this error. Any ideas what I might try? it seems to not do this when running multipel

Traceback (most recent call last):
  File "/aian/scripts/Chai/run_chai.py", line 25, in <module>
    output_paths = run_inference(
                   ^^^^^^^^^^^^^^
  File "/software/envs/chai/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/software/envs/chai/lib/python3.11/site-packages/chai_lab/chai1.py", line 271, in run_inference
    chains = load_chains_from_raw(fasta_inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/envs/chai/lib/python3.11/site-packages/chai_lab/data/dataset/inference_dataset.py", line 164, in load_chains_from_raw
    conformer_generator = RefConformerGenerator()
                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/envs/chai/lib/python3.11/site-packages/chai_lab/data/sources/rdkit.py", line 58, in __init__
    conformers_cache_file = paths.cached_conformers.get_path().as_posix()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/envs/chai/lib/python3.11/site-packages/chai_lab/utils/paths.py", line 51, in get_path
    download(self.url, path=self.path)
  File "/software/envs/chai/lib/python3.11/site-packages/chai_lab/utils/paths.py", line 39, in download
    tmp_path.rename(path)
  File "/software/envs/chai/lib/python3.11/pathlib.py", line 1175, in rename
    os.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/software/envs/chai/lib/python3.11/site-packages/downloads/conformers_v1.download_tmp' -> '/software/envs/chai/lib/python3.11/site-packages/downloads/conformers_v1.apkl'

[arogozhnikov]

hi Ian,

thx for reporting, this part of code is not protected from race conditions, will fix

[arogozhnikov]

fixed in #114 (install new version from github!)