Skip to content

Latest commit

 

History

History
98 lines (67 loc) · 3.39 KB

DEPLOY.md

File metadata and controls

98 lines (67 loc) · 3.39 KB
Raw

Deployment Guide

  • PyTorch -> ONNX
  • ONNX -> TensorRT
  • ONNX inference
  • TensorRT inference
  • ONNX visualization
  • TensorRT visualization

Installation

A separate Python virtual environment is recommended here to avoid effects to your training & testing environment.

Install all deployment packages (our tested conda version) by:

conda install cudatoolkit=10.2 -c pytorch
conda install cudnn==8.0.4 -c nvidia
pip install onnx==1.10.2 onnxruntime-gpu==1.6.0
python3 -m pip install --upgrade nvidia-tensorrt==8.2.1.8  // you may need to add --extra-index-url https://pypi.ngc.nvidia.com

In this version, TensorRT may use CUDA runtime >= 11, you might avoid using conda if you have sudo access on your device.

Or you can incrementally install Extra Dependencies through the tutorial.

Important Note

Note that we only convert the model forward() function, post-processing (i.e., inference()) is not included. Typical post-processing includes: segmentation map interpolation, line NMS for anchor-based lane detection, sigmoid/softmax activations, etc.

PyTorch -> ONNX:

PyTorch version >= 1.6.0 is recommended for this feature.

Extra Dependencies:

pip install onnx==1.10.2 onnxruntime-gpu==<version>

<version> depends on your CUDA/CuDNN version, see here, if you met version issues, try install exact cudatoolkit and cudnn from conda. Or you can just install the CPU onnxruntime for this functionality.

Conversion:

The conversion is based on Torch JIT's tracing on random input. To convert a checkpoint (e.g., ckpt.pt) to ONNX, simply run this command:

python tools/to_onnx.py --config=<config file path>  --height=<input height> --width=<input width> --checkpoint=ckpt.pt

You'll then see the saved ckpt.onnx file and a report on the conversion quality.

Same config mechanism and commandline overwrite by --cfg-options as in training/testing.

For detailed instructions and commandline shortcuts available, run:

python tools/to_onnx.py --help

Currently Unsupported Models:

  • ENet (segmentation)
  • ENet backbone (lane detection)
  • DCNv2 in BézierLaneNet (lane detection)
  • Swin backbone (supported if pytorch >= 1.10.0)
  • LaneATT (supported if pytorch >= 1.8.0)

ONNX -> TensorRT:

Extra Dependencies:

python3 -m pip install --upgrade nvidia-tensorrt==<version>

TensorRT <version> is recommended to be at least 7.2, you can also install it via other means than pip. To work better with onnxruntime (for checking of conversion quality), you best checkout the compatibility.

Conversion:

The conversion is mainly a building of TensorRT engine. To convert a checkpoint (e.g., ckpt.onnx) to TensorRT, simply run this command:

python tools/to_tensorrt.py --height=<input height> --width=<input width> --onnx-path=<ckpt.onnx>

You'll then see the saved ckpt.engine file and a report on the conversion quality.

Currently Unsupported Models:

  • ENet (segmentation)
  • ENet backbone (lane detection)
  • SCNN (lane detection)
  • Swin backbone (lane detection)
  • DCNv2 in BézierLaneNet (lane detection, could support if built custom op from mmcv and directly convert from PyTorch to TensorRT)
  • LaneATT (supported if TensorRT >= 8.4.1.5)