ML based Occupancy

[lexavtanke]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I've agreed with the maintainers that I can plan this task.

Description

Integrate occupancy prediction to the autoware perception stack to increase reliability of occupancy grid map. Because now only heuristic method used in this task in autoware as https://autowarefoundation.github.io/autoware.universe/main/perception/ground_segmentation/. This method is lack of semantic information and also It is tedious to adjust to environment where vehicle operates.

With ML based occupancy prediction it is possible to apply for sophisticated logic for obstacle detection and use additional information in planning to increase reliability and comfort. Also 3d occupancy information allow to cope with situations then shape of detected objects can't be incorporated to the simple form as bounding box (unusual vehicle or just simple open door)

Purpose

Enhance Autoware with ML based occupancy prediction method.

Possible approaches

• OpenOccupancy
• TPVFormer
• VoxFormer
• BEVFormer

Definition of done

• Evaluation of different methods and choose one for implementation
• Implementation of chosen method
• Integration implementation with autoware occupancy grid map (without semantic information)
• Proposal new occupancy grid format for autoware (BEV with semantics, 3d volumetric with semantic)

[BonoloAWF]

@lexavtanke could you create a Discussion here https://github.com/orgs/autowarefoundation/discussions/categories/design outlining your proposed approach and how this will integrate with the Autoware architecture?

[lexavtanke]

@BonoloAWF I've started discussion in design section link

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[lexavtanke]

Model which will be used as a baseline for the integration in to Autoware is BEVDet (it has occupancy head and it is realtime). In the future it is possible to move to a multimodal model (lidar + cams) like BEVfusion.
Current actual code for the model and its conversion into TensorRT is in the repo
Current status is to make a code to convert BEVDet occ model to TensorRT. The autors of the original repo provide tensorrt conversion only for base BEVDet model but alll occ versions based on BEVDet4D with mods (stereo, depth).

[lexavtanke]

Latest progress report can be acceded in Perception and Sensing WG Meeting notes

[lexavtanke]

Latest progress report can be accessed here.

And current brief status:
Code for exporting BEVDet4D to TensorRT is done and in the repo.

Inference speed test results on RTX 3080 Mobile:

• approximately 76,5 fps
• 13 ms per frame
• GPU memory consumption 2658 Mb

Now I'm working on conversion BEVDet4DStereoOCC to TensorRT.

[lexavtanke]

Currently I it possible to export BEVDet4dStereoOcc model to ONNX with the use of the contrib operators because it supports inverse operator. But after it is impossible to convert this model to TensorRT because it doesn't support inverse operator.

Inverse operator is used in method gen_grid of the DepthNet part of the view_transformer net. So I start to investigate how I can work with this.

After some investigation I found that It is possible to export a pytorch model as several separate engines (as it was done with the center point model in Autoware) but I didn't find any documentation in Autoware about how it was exported after training. There is a mention that center_point was trained with mmdetection3d. But I didn't find any export code for ONNX and TensorRT export there.

@miursh Can you please provide the information about export center_point to ONNX and TensorRT or tell who can help with this question?

After it will be possible to separate the model in 2 parts and write code for gen_grid in the node which will call the engines and process the data.

Also possible way which need a bit more investigation is TPAT. They tell that it is possible to autogenerate plugins for unsupported operators but I'm not sure that It will works with contrib operators.

And the last way is to write own custom plugin for TensorRT.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[vividf]

I will close this one since it has been stale for a year.

[xmfcx]

This was sort of superseeded by:

• Retrain BEVDet Using TIER4 Dataset #8036