-
Notifications
You must be signed in to change notification settings - Fork 350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨[Feature] A hook function after each graph change or partition #3233
Comments
@peri044 did we implement LOG_GRAPH for dynamo? |
@sean-xiang-applovin We use a PassManager to manage the lowering.
There are two sets of passes: You can inject custom passes that can do whatever (print out graphs, dump or save graphs etc.) and place them at a particular index in the pass pipeline. Its not a particularly ergonomic API but if you want to submit improvements you find useful in a PR we would be interested |
@narendasan Sure, I think that pass manager could help with the graph before partition, what about I want to access the partitioned graph? |
We do have the dry_run system which can provide insight into post partitioning structure but I think we would be unlikely to add a callback or something to save partitioned but not compiled graphs |
Got it, thanks for your reply @narendasan |
Is your feature request related to a problem? Please describe.
I am trying to compile our model with tensorrt these days with exported program, our model is not very big, the original graph contains like 2k~ of nodes.
I am running into different problems, and there are problems even before we start to convert the partitioned graph, like during decomposition or post lowering phrase. Sometimes the debug log indicates some error of some node, and I cannot find that node in my original graph.
I find it helpful to dump the graph after each "change". Even though there is log printed, but
Describe the solution you'd like
A hook function to be called after each intermediate graph, like graph after each pass, and each partitioned graph.
The function takes a gm object, and some metadata of this gm object, like a name, or a number k indicating it's the kth change to the original graph.
The most common implementation of this hook function is probably, to print the graph, or save it somewhere on the disk. So that I can check each graph and find which is the first graph contains the error node described in the error log.
I can also see this can be helpful if people want to visualize each graph and graph change during compilation.
Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered: