Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce DALI proxy #5726

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

Introduce DALI proxy #5726

wants to merge 22 commits into from

Conversation

jantonguirao
Copy link
Contributor

@jantonguirao jantonguirao commented Nov 27, 2024 •
edited
Loading

Category:

New feature

Description:

  • DALI proxy is a new way to integrate DALI pipelines with existing torch data loading pipelines
  • The idea is that torch data processes send data to the main process, where it is processed by DALI before handling it over to the training loop.
  • The solution allows for mixing data loading from Pytorch with partial processing on DALI

Co-author: @mdabek-nvidia

Additional information:

Affected modules and functionalities:

  • Torch plugin

Key points relevant for the review:

dali/python/nvidia/dali/plugin/pytorch/init.py

Tests:

  • Existing tests apply
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A
    Added options to run with DALI proxy to RN50 and EfficientNet examples

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • [?] RST
    • [?] Jupyter
    • Other
  • N/A
    TODO

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: N/A

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [20865442]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [20865442]: BUILD PASSED

@jantonguirao jantonguirao changed the title Dali proxy2 Introduce DALI proxy Nov 28, 2024
@jantonguirao jantonguirao marked this pull request as ready for review November 28, 2024 09:21
szkarpinski
szkarpinski requested changes Nov 29, 2024
View reviewed changes
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
@szkarpinski
Copy link
Collaborator

I see there are no tests except for the resnet50 example. I believe we should have normal TL0 tests as well.

@szkarpinski
Copy link
Collaborator

Did you test error propagation between the processes? Multiprocessing doesn't automagically propagate exceptions afaik. Maybe we should have tests to check how are the errors in particular processes reported?

@jantonguirao jantonguirao force-pushed the dali_proxy2 branch 2 times, most recently from caf2d0b to 0472490 Compare November 29, 2024 10:28
klecki
klecki reviewed Nov 29, 2024
View reviewed changes
Copy link
Contributor

@klecki klecki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posting few comments as the code started moving.
I also feel like we should limit the scope of the API and hide most of the implementation.

dali/python/setup.py.in Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
dali/python/nvidia/dali/plugin/pytorch/__init__.py Outdated Show resolved Hide resolved
@jantonguirao jantonguirao force-pushed the dali_proxy2 branch 7 times, most recently from c4a7f74 to 1a50983 Compare December 3, 2024 17:52
@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21050826]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21050826]: BUILD FAILED

@jantonguirao jantonguirao force-pushed the dali_proxy2 branch 2 times, most recently from 3bdfd37 to 014032d Compare December 3, 2024 18:32
@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21052236]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21052236]: BUILD FAILED

@jantonguirao
Introduce DALI proxy
1739c82 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Code review changes
70c69a1 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Remove nvtx import
165e7bb 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Move to torch_utils
c34db8f 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Add tests
d931d75 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Code review fixes
31a5a12 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Invoke test_dali_proxy.py
75b274c 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Fix
984980b 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Code review fixes
255b61b 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Add deterministic behavior
e17e033 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
documentation
f976bc5 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Code review fixes
33e0d0a 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Moving DALI proxy to experimental
b509dc0 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Fixes
0af7fb3 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Update tutorial
facf120 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Add documentation
de847be 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Enhance tests
511436b 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Allow PIL.Image as an input to the pipeline (can be converted to np.a…
926b3bc 
…rray)

Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Apply code review suggestions
826f463 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Add TL3_RN50_benchmark/test_pytorch.sh
659d0f1 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao jantonguirao force-pushed the dali_proxy2 branch 3 times, most recently from 75f5921 to 32983fe Compare December 24, 2024 12:16
@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21819002]: BUILD STARTED

@jantonguirao
Code review fixes
18e0781 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@jantonguirao
Simplify example code
fb43cf3 
Signed-off-by: Joaquin Anton Guirao <[email protected]>
@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21823100]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [21823100]: BUILD PASSED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mzient mzient mzient left review comments

@klecki klecki klecki requested changes

@szkarpinski szkarpinski Awaiting requested review from szkarpinski

Assignees

@klecki klecki

@mzient mzient

@szkarpinski szkarpinski

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jantonguirao @dali-automaton @szkarpinski @klecki @mzient