Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Init cudagraph #424

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Init cudagraph #424

wants to merge 4 commits into from

Conversation

nimlgen
Copy link

@nimlgen nimlgen commented Sep 12, 2023

This is one more implementation of cuda graphs baed on 2 previous MR. This is a first MR, so has some basic functionality to create CUDAgraphs + setting some dynamic parameters to nodes.

I'd love your comments and feedback.

@mitkotak
Copy link
Contributor

mitkotak commented Sep 12, 2023

This is one more implementation of cuda graphs baed on 2 previous MR. This is a first MR, so has some basic functionality to create CUDAgraphs + setting some dynamic parameters to nodes.

I'd love your comments and feedback.

Thank you so much for restarting this thread ! Can you add some basic tests similar to https://gitlab.tiker.net/kaushikcfd/pycuda/-/merge_requests/2/diffs#653379dd0e00d91ae08b155f35494fd427d5c80e
so that we can start testing all of the functions piece by piece ? Page 17 contains some snippets of how the API looks in my branch

Co-authored-by: Mit Kotak <[email protected]>
Co-authored-by: Gerlof Fokkema <[email protected]>
@mitkotak
Copy link
Contributor

@nimlgen Getting a segfault for test_dynamic_params. I am trying to debug but also wanted to keep you in the loop

(ceesd) mitak2@bock:~/cudagraph_test/pycuda$ pytest test/test_graph.py
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/mitak2/cudagraph_test/pycuda
plugins: pudb-0.7.0, cov-4.0.0
collected 3 items

test/test_graph.py .Fatal Python error: Segmentation fault

Current thread 0x00007fd8ede96740 (most recent call first):
  File "/home/mitak2/cudagraph_test/pycuda/test/test_graph.py", line 71 in test_dynamic_params
  File "/home/mitak2/cudagraph_test/pycuda/pycuda/tools.py", line 510 in f
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/python.py", line 1799 in runtest
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/main.py", line 323 in _main
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/config/__init__.py", line 166 in main
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/lib/python3.11/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "/home/mitak2/emirge/miniforge3/envs/ceesd/bin/pytest", line 33 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pycuda._driver, pycuda._pvt_struct, numpy.linalg.lapack_lite (total: 16)
Segmentation fault (core dumped)

@nimlgen
Copy link
Author

nimlgen commented Oct 20, 2023

Thanks for testing. Should be fixed now (would appreciate if you check on your end as well). LMK if any other refactors are needed.

@geohot
Copy link

geohot commented Nov 16, 2023

Hey @inducer, anything I can do to help get this upstreamed?

@mitkotak
Copy link
Contributor

mitkotak commented Nov 16, 2023

Hey @inducer, anything I can do to help get this upstreamed?

Thanks for jumping in ! Here's a non-exhaustive list of node calls that I am working on porting from my PR to this one:

  • add_memalloc_node
  • add_memfree_node
  • add_memcpy_htod_node
  • add_memcpy_dtoh_node
  • add_memcpy_dtod_node
  • add_memset_node
  • add_empty_node

Along with the add_node calls, it would be good to have the corresponding get_params and set_params calls. For some context, the eventual goal is to implement all of the node calls that we would need to generate pycuda-cudagraph code for pytato which can then be called by an array program workalike.

Feel free to pick your favorite node :) Hopefully that helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants