Import tqdm to support jupyter better

[shcheklein]

Partially addresses #780

Screen.Recording.2025-01-12.at.7.30.04.PM.mov

What I don't know how to fix yet is progress bars when we use parallel and we run the whole thing in the subprocess. In this case child process doesn't have access to Ipython widgets (or whatever the mechanics is behind these nice progress bars).

[cloudflare-workers-and-pages]

Deploying datachain-documentation with    Cloudflare Pages

Latest commit:	3c2e0cf
Status:	✅  Deploy successful!
Preview URL:	https://ded407f0.datachain-documentation.pages.dev
Branch Preview URL:	https://fix-780-progress-bars-jupyte.datachain-documentation.pages.dev

View logs

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.65%. Comparing base (3767173) to head (3c2e0cf).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #812      +/-   ##
==========================================
+ Coverage   87.42%   87.65%   +0.22%     
==========================================
  Files         128      128              
  Lines       11373    11333      -40     
  Branches     1537     1530       -7     
==========================================
- Hits         9943     9934       -9     
+ Misses       1049     1018      -31     
  Partials      381      381              

Flag	Coverage Δ
datachain	87.59% <100.00%> (+0.22%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[skshetry]

What I don't know how to fix yet is progress bars when we use parallel and we run the whole thing in the subprocess. In this case child process doesn't have access to Ipython widgets (or whatever the mechanics is behind these nice progress bars).

We should not use progress bars inside udf at all. This just does not work with Tqdm, and it will make progress bars jump constantly, and leave gaps in between. The problem gets worse when the user's code creates progress bars in the main thread. tqdm has position value that does not get shared in multiple processes - it is essentially copied to different processes once, and one will try to overwrite another.

[shcheklein]

We should not use progress bars inside udf at all. This just does not work with Tqdm, and it will make progress bars jump constantly, and leave gaps in between. The problem gets worse when the user's code creates progress bars in the main thread. tqdm has position value that does not get shared in multiple processes - it is essentially copied to different processes once, and one will try to overwrite another.

we have a single progress bar there (per udf), we don't pass it to all the subprocesses - so it works fine in CLI. We run a wrapper process though to avoid main, that process controls progress (not processes that actually run UDFs) and that process doesn't have access to JUpyter.

[skshetry]

we have a single progress bar there (per udf), we don't pass it to all the subprocesses - so it works fine in CLI. We run a wrapper process though to avoid main, that process controls progress (not processes that actually run UDFs) and that process doesn't have access to JUpyter.

We do have some UDFs that create progress bars - such as ArrowGenerator and maybe Listing?

[shcheklein]

We do have some UDFs that create progress bars - such as ArrowGenerator and maybe Listing?

yep, may be. But even in that case this PR is most likely an improvement. We can decide and review and remove some internal progress bars. I'm not sure why we use / need them.

[skshetry]

Let's remove this too when we can.

Maybe too much detail, but tqdm does locking by default which is multiprocess-safe. But since we overwrite this, this is only thread-safe.

dvc-data does reset it, so we'll be overriding it anyway as we are importing it someplace. But I hope we can remove it someday there. I have no idea why this was done in dvc.

[skshetry]

(I have a suspicion that this is what causes progressbars to jump in dvc)

[shcheklein]

yep, makes sense

[skshetry]

dvc-data fixed in iterative/dvc-data#592.