Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Unity][MSC][M1.3] Add translate && codegen for tensorrt #15950

Merged
merged 4 commits into from
Oct 26, 2023

Conversation

Archermmt
Copy link
Contributor

@Archermmt Archermmt commented Oct 19, 2023

This is a pull request for MSC(Multi-System Compile)
RFC: https://discuss.tvm.apache.org/t/rfc-unity-msc-introduction-to-multi-system-compiler/15251/5
Tracking issue: #15233

This is the 3rd part of Milestone 1: Finish RuntimeManager for frameworks

To limit each PR in reviewable size, the Milestone 1 will be split into some steps:
M1.1 Add translate && codegen for torch
M1.2 Add translate && codegen for tensorflow
M1.3 Add codegen for tensorrt
M1.4 Add Runner and test with relax
M1.5 Add Runner and test with torch
M1.6 Add Runner and test with tensorflow
M1.7 Add Runner and test with tensorrt

This PR support tensorrt in a codegen way. Same strategy is used in Nvidia Nackthon 2021: https://github.com/Archermmt/yolov3_dcn_nv_hackthon2021 (special price for latency and bug detection).

2 passes are added to support BYOC
1.BindShape: Works like FuseOpsByPattern(bind_constant=True). This pass bind the ShapeExpr of extern func to Expr.
2.FuseTuple: This pass fuse Tuple and TupleGetitem to byoc func.

Why use codegen flavor to support TensorRT:

  1. To enable deploy tensorrt solution in different runtime environment without re-compile the model.
  2. To enable the usage of MSC tools in TensorRT. Including pruning, debugging and so on.

cc @Hzfengsy @junrushao

@Hzfengsy Hzfengsy merged commit 5d62e9b into apache:unity Oct 26, 2023
4 checks passed
@Archermmt Archermmt deleted the msc branch October 26, 2023 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants