torch-loader(example): use prefetch and try to run example in linux #691
Conversation
Deploying datachain-documentation with
|
| Latest commit: |
59c7aa6
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://4ce804fc.datachain-documentation.pages.dev |
| Branch Preview URL: | https://run-torch-loader.datachain-documentation.pages.dev |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #691 +/- ##
==========================================
- Coverage 87.34% 87.31% -0.03%
==========================================
Files 113 113
Lines 10791 10791
Branches 1479 1479
==========================================
- Hits 9425 9422 -3
- Misses 989 991 +2
- Partials 377 378 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
skshetry
force-pushed
the
run-torch-loader
branch
from
067f991 to
37403b9
Compare
skshetry
force-pushed
the
run-torch-loader
branch
from
af02fff to
840a841
Compare
skshetry
force-pushed
the
run-torch-loader
branch
from
840a841 to
59c7aa6
Compare
| num_workers=2, | ||
| batch_size=25, | ||
| num_workers=4, | ||
| multiprocessing_context=multiprocessing.get_context("spawn"), |
There was a problem hiding this comment.
fsspec's loop is not fork-safe. Even though we create a new loop for each forked processes, s3fs or other filesystems may not be fork-safe.
See https://s3fs.readthedocs.io/en/latest/#multiprocessing.
This causes future from a run_coroutine_threadsafe to hang forever on prefetch.
I could contribute a fix to the first problem, but the second problem still remains.
Also, Python 3.14 is changing default start method for posix systems (except macOS which uses 'spawn') to 'forkserver'. See python/cpython#84559.
So, I think it's better to recommend to use a different start method.
No description provided.