Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

Open
wants to merge 87 commits into
base: main
Choose a base branch
from

Conversation

cyn-liu
Copy link
Contributor

@cyn-liu cyn-liu commented Jul 11, 2024

Description

Integrating BEVDet into Autoware for 3D object detection based on multi-view images.

Related links

Issue Link

How was this PR tested?

This PR has been tested on local environment.
env1:

RTX 3080
docker image: `ghcr.io/autowarefoundation/autoware-universe:humble-20231001-cuda`
CUDA11.6
TensorRT8.4.2

env2:

RTXA1000 `Dell notebook computer`
docker image: ghcr.io/autowarefoundation/autoware:20240612-devel-cuda-amd64
CUDA==12.3
libnvinfer==8.6.1.6

Notes for reviewers

Test using a bag containing Nuscenes data.
Note: The frame_id of /lidar_top in this bag is map
onnx model file
bag

Interface changes

Add a new perception_mode, If set perception_mode = camera, the detector will launch tesnsorrt_bevdet node.

Topic changes

Additions and removals

Change type Topic Type Topic Name Message Type Description
Added Sub ~/input/topic_cloud sensor_msgs::msg::PointCloud2 used for time alignment and display
Added Sub ~/input/topic_img_fl sensor_msgs::msg::Image input front_left camera image
Added Sub ~/input/topic_img_f sensor_msgs::msg::Image input front camera image
Added Sub ~/input/topic_img_fr sensor_msgs::msg::Image input front_right camera image
Added Sub ~/input/topic_img_bl sensor_msgs::msg::Image input back_left camera image
Added Sub ~/input/topic_img_b sensor_msgs::msg::Image input back camera image
Added Sub ~/input/topic_img_br sensor_msgs::msg::Image input back_right camera image
Added Pub ~/output/painting_cloud sensor_msgs::msg::PointCloud2 output pointcloud just used for display
Added Pub ~/output/boxes autoware_perception_msgs::msg::DetectedObjects detected objects

ROS Parameter Changes

Additions and removals

Change type Parameter Name Type Default Value Description
Added bev_detection_model string bevdet Set BEV detection model type

Modifications

Version Parameter Name Type Default Value Description
Old perception_mode string lidar Set perception mode
New perception_mode string lidar Add a new perception mode of camera

Effects on system behavior

None.

@github-actions github-actions bot added type:documentation Creating or refining documentation. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) component:launch Launch files, scripts and initialization tools. (auto-assigned) labels Jul 11, 2024
Copy link

github-actions bot commented Jul 11, 2024

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

@kminoda kminoda added the tag:run-build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Jul 11, 2024
Copy link
Contributor

@kminoda kminoda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool PR 🚀

Let me provide some minor comments first. Also, please make sure that all the CIs are passing before opening the PR 🙏

perception/tensorrt_bevdet/package.xml Outdated Show resolved Hide resolved
perception/tensorrt_bevdet/include/bevdet_node.hpp Outdated Show resolved Hide resolved
@cyn-liu cyn-liu linked an issue Jul 16, 2024 that may be closed by this pull request
7 tasks
@kminoda
Copy link
Contributor

kminoda commented Jul 22, 2024

(Friendly ping 🙏 ) Would you mention us once all the CIs are fixed?

@technolojin
Copy link
Contributor

I could see two required checks are not passed.

  1. cppcheck-differential > Details > Show cppcheck-report result
Run cat cppcheck-report.txt
Error: /home/runner/work/autoware.universe/autoware.universe/perception/tensorrt_bevdet/src/bevdet.cpp:167:13: performance: Range variable 'task_factors' should be declared as const reference. [iterateByValue]
  for (auto task_factors : nms_factor_temp) {
            ^
Error: /home/runner/work/autoware.universe/autoware.universe/perception/tensorrt_bevdet/src/data.cpp:37:20: performance: Range variable 'name' should be declared as const reference. [iterateByValue]
  for (std::string name : cams_name) {
                   ^
  1. pre-commit.ci -pr > Details
fix include guard........................................................Failed
- hook id: ros-include-guard
- exit code: 1

No include guard in perception/tensorrt_bevdet/include/gatherbev_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/bevpool_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/preprocess_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/bevdet_node.hpp
No include guard in perception/tensorrt_bevdet/include/alignbev_plugin.hpp

sort package.xml.........................................................Passed
shellcheck...............................................................Passed
shfmt....................................................................Passed
isort....................................................................Passed
black....................................................................Passed
clang-format.............................................................Passed
cpplint..................................................................Failed
- hook id: cpplint
- exit code: 1

perception/tensorrt_bevdet/include/bevdet_node.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
perception/tensorrt_bevdet/include/bevdet_node.hpp:86:  Add #include <vector> for vector<>  [build/include_what_you_use] [4]
perception/tensorrt_bevdet/include/bevdet_node.hpp:113:  Add #include <memory> for shared_ptr<>  [build/include_what_you_use] [4]
Done processing perception/tensorrt_bevdet/include/bevdet_node.hpp
perception/tensorrt_bevdet/include/cpu_jpegdecoder.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
Done processing perception/tensorrt_bevdet/include/cpu_jpegdecoder.hpp
perception/tensorrt_bevdet/include/gatherbev_plugin.hpp:45:  Single-parameter constructors should be marked explicit.  [runtime/explicit] [5]
Done processing perception/tensorrt_bevdet/include/gatherbev_plugin.hpp
perception/tensorrt_bevdet/include/preprocess.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
Done processing perception/tensorrt_bevdet/include/preprocess.hpp
perception/tensorrt_bevdet/src/bevdet.cpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
perception/tensorrt_bevdet/src/bevdet.cpp:190:  Using C-style cast.  Use static_cast<float>(...) instead  [readability/casting] [4]

...

@technolojin technolojin self-assigned this Jul 26, 2024
@kminoda
Copy link
Contributor

kminoda commented Jul 29, 2024

@cyn-liu As this PR is stale for a week, from the maintenance perspective, let me make this PR a draft for now. Feel free to re-open the PR once all the CIs are fixed and ready to be reviewed. Thank you for your understanding, and we are looking forward to reviewing your PR.

@kminoda kminoda marked this pull request as draft July 29, 2024 02:27
Copy link
Contributor

@kminoda kminoda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mojomex mojomex marked this pull request as draft September 6, 2024 06:47
@cyn-liu cyn-liu marked this pull request as ready for review September 18, 2024 09:30
void BEVDet::mallocDeviceMemory()
{
trt_buffer_sizes_.resize(trt_engine_->getNbBindings());
trt_buffer_dev_ = reinterpret_cast<void **>(new void *[trt_engine_->getNbBindings()]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyn-liu

Please apply the changes not only to the point we pointed out, but to all points related to the comment.

As we said in #7956 (comment), don't use new.
Please check our coding guidelines:

https://autowarefoundation.github.io/autoware-documentation/main/contributing/coding-guidelines/languages/cpp/
Our coding guideline follows CppCoreGuidelines

Copy link
Contributor Author

@cyn-liu cyn-liu Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactoring the entire open source project to fully meet the specifications is a significant workload, but I have to do so.
I found that the refactored code often encounters unknown problems during launch, and the cause of this problem has not been found. After restarting the node multiple times, it can be resolved.
Although the previous code did not meet your coding standards, it was simple clear, and can ran stably.

Copy link
Contributor

@Shin-kyoto Shin-kyoto Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyn-liu
Is this what you meant in your comment?

  • autoware_tensorrt_bevdet/src/bevdet.cpp was copied from another open source project.
  • The content written in the other open source project is being used as is.

If that's the case, instead of modifying the code, the following actions are also possible:

  • Clearly indicate where the code was copied from, like in this example.
  • Store all copied code in a lib directory.

For code outside the lib, you have to follow Autoware's coding guidelines. For example, Autoware’s coding guidelines recommend avoiding the use of new.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyn-liu (cc. @Shin-kyoto)
With all due respect, but

Although the previous code did not meet your coding standards

All PR authors are asked to read and accept the coding standards in the contribution guidelines, so this is a requirement that was clear even before the work on this PR started, so we expect this to be followed. A large open source project cannot be managed otherwise and this is not a concept unique to Autoware.

it was simple clear

I disagree, and so do our coding standards and the C++ core guidelines. Also, this PR is almost 6000 lines which makes even perfectly simple and clear code hard to review thoroughly.

and can ran stably.

As a reviewer, this was impossible to verify due to all the non-standard and almost C-style C++. There is a reason that core guidelines and tools like Clang-Tidy are enforcing the use of smart pointers, disallowing reinterpret_cast etc.
If it does not run stably after the refactoring, that hints at race conditions and memory problems that still need to be addressed. Consider using AddressSanitizer and ThreadSanitizer to pinpoint the issues.

As this review is taking quite some hours to review for our employees, I would strongly suggest that the coding style issues are addressed, and that the PR is split up before occupying people with the same discussion over and over again.

If you do not want to comply with our coding standards, I am afraid we cannot merge this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyn-liu

friendly ping

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyn-liu

friendly ping

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyn-liu

I have talked with @Shin-kyoto on this.Like @mojomex mentioned, we have a coding standard in AWF which encourages the use of smart pointers instead of normal pointers. Therefore, we would like to comply to the guideline as much as possible.

However, I also understand the effort of refactoring the whole code, especially if the code was taken from another open source. If it is difficult to refactor, alternate option is to wrap the original BEVDet code as is as a vendor package and maintain it outside Autoware with different CIs settings and code standards. Then we can added through autoware.repos or through ROS Buildfarm and just call the libraries from Autoware Nodes. This might also make it easier for us to update the BEVDet implementation if any updates are made upstream.

Do you think the above approach is feasible? It might be difficult if you have already made a lot of modification to the BEVDet implementation so it would be nice if you can provide us how much modification you made from the original code.

I'm also okay to set up an online call if that makes you easier to discuss on this topic.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good Afternoon, Mits-san @mitsudome-r . thank you for the coordination.

  1. Second way is a better option for Continuously introduce great open source project into autoware
  2. I will tell Cynthia @cyn-liu to make a list of the modification point here
  3. it will be great if we can have a online call for this, especially for the topic “call the libraries from Autoware Nodes”, we are not familiar with this part

have a nice day!
Lucas

Copy link
Contributor Author

@cyn-liu cyn-liu Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main modifications made to tensorrt-bevdet-cpp library are as follows:

  1. Standardize log output by changing print ,std:: cout etc to ros2 rclcpp logger
  2. Standardize the names of member functions and variables
  3. Modify multiple pointers to cpp std::vector
  4. If the engine model does not exist, export engine before inference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:launch Launch files, scripts and initialization tools. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) tag:require-cuda-build-and-test tag:run-build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) type:documentation Creating or refining documentation. (auto-assigned)
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

Implementing BEVDet in Autoware
9 participants