feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

cyn-liu · 2024-07-11T03:52:38Z

Description

Integrating BEVDet into Autoware for 3D object detection based on multi-view images.

NEWS: Refactoring the integration method of bevdet-tensorrt-cpp to Autoware:

We wrap the original bevdet-tensorrt-cpp code as a vendor package and maintain it outside Autoware with different CIs settings and code standards. Then we added through autoware.repos and just call the libraries in autoware_tensorrt_bevdet node.

NOTE: This PR depends on another PR fix(autoware.repos): add bevdet_vendor external package #5456

How was this PR tested?

This PR has been tested on local environment.
env1:

RTX 3080
docker image: `ghcr.io/autowarefoundation/autoware-universe:humble-20231001-cuda`
CUDA11.6
TensorRT8.4.2

env2:

RTXA1000 `Dell notebook computer`
docker image: ghcr.io/autowarefoundation/autoware:20240612-devel-cuda-amd64
CUDA==12.3
libnvinfer==8.6.1.6

Notes for reviewers

Test using a bag containing Nuscenes data.
Note: The frame_id of /lidar_top in this bag is map
onnx model file
bag

Interface changes

Add a new perception_mode, If set perception_mode = camera, the detector will launch tesnsorrt_bevdet node.

Topic changes

Additions and removals

Change type	Topic Type	Topic Name	Message Type	Description
Added	Sub	`~/input/topic_cloud`	`sensor_msgs::msg::PointCloud2`	used for time alignment and display
Added	Sub	`~/input/topic_img_fl`	`sensor_msgs::msg::Image`	input front_left camera image
Added	Sub	`~/input/topic_img_f`	`sensor_msgs::msg::Image`	input front camera image
Added	Sub	`~/input/topic_img_fr`	`sensor_msgs::msg::Image`	input front_right camera image
Added	Sub	`~/input/topic_img_bl`	`sensor_msgs::msg::Image`	input back_left camera image
Added	Sub	`~/input/topic_img_b`	`sensor_msgs::msg::Image`	input back camera image
Added	Sub	`~/input/topic_img_br`	`sensor_msgs::msg::Image`	input back_right camera image
Added	Pub	`~/output/painting_cloud`	`sensor_msgs::msg::PointCloud2`	output pointcloud just used for display
Added	Pub	`~/output/boxes`	`autoware_perception_msgs::msg::DetectedObjects`	detected objects

ROS Parameter Changes

Additions and removals

Change type	Parameter Name	Type	Default Value	Description
Added	`bev_detection_model`	`string`	`bevdet`	Set BEV detection model type

Modifications

Version	Parameter Name	Type	Default Value	Description
Old	`perception_mode`	`string`	`lidar`	Set perception mode
New	`perception_mode`	`string`	`lidar`	Add a new perception mode of `camera`

Effects on system behavior

None.

Signed-off-by: liu cui <[email protected]>

github-actions · 2024-07-11T03:52:54Z

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

You've checked our contribution guidelines.
Your PR follows our pull request guidelines.
All required CI checks pass before marking the PR ready for review.

kminoda

Cool PR 🚀

Let me provide some minor comments first. Also, please make sure that all the CIs are passing before opening the PR 🙏

perception/tensorrt_bevdet/package.xml

perception/tensorrt_bevdet/include/bevdet_node.hpp

kminoda · 2024-07-11T04:34:20Z

@cyn-liu Please make sure that the node follows coding guildline: https://autowarefoundation.github.io/autoware-documentation/main/contributing/coding-guidelines/ros-nodes/class-design/

such as

Signed-off-by: liu cui <[email protected]>

…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>

Signed-off-by: liu cui <[email protected]>

…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>

kminoda · 2024-07-22T02:41:04Z

(Friendly ping 🙏 ) Would you mention us once all the CIs are fixed?

technolojin · 2024-07-23T00:27:40Z

I could see two required checks are not passed.

cppcheck-differential > Details > Show cppcheck-report result

Run cat cppcheck-report.txt
Error: /home/runner/work/autoware.universe/autoware.universe/perception/tensorrt_bevdet/src/bevdet.cpp:167:13: performance: Range variable 'task_factors' should be declared as const reference. [iterateByValue]
  for (auto task_factors : nms_factor_temp) {
            ^
Error: /home/runner/work/autoware.universe/autoware.universe/perception/tensorrt_bevdet/src/data.cpp:37:20: performance: Range variable 'name' should be declared as const reference. [iterateByValue]
  for (std::string name : cams_name) {
                   ^

pre-commit.ci -pr > Details

fix include guard........................................................Failed
- hook id: ros-include-guard
- exit code: 1

No include guard in perception/tensorrt_bevdet/include/gatherbev_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/bevpool_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/preprocess_plugin.hpp
No include guard in perception/tensorrt_bevdet/include/bevdet_node.hpp
No include guard in perception/tensorrt_bevdet/include/alignbev_plugin.hpp

sort package.xml.........................................................Passed
shellcheck...............................................................Passed
shfmt....................................................................Passed
isort....................................................................Passed
black....................................................................Passed
clang-format.............................................................Passed
cpplint..................................................................Failed
- hook id: cpplint
- exit code: 1

perception/tensorrt_bevdet/include/bevdet_node.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
perception/tensorrt_bevdet/include/bevdet_node.hpp:86:  Add #include <vector> for vector<>  [build/include_what_you_use] [4]
perception/tensorrt_bevdet/include/bevdet_node.hpp:113:  Add #include <memory> for shared_ptr<>  [build/include_what_you_use] [4]
Done processing perception/tensorrt_bevdet/include/bevdet_node.hpp
perception/tensorrt_bevdet/include/cpu_jpegdecoder.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
Done processing perception/tensorrt_bevdet/include/cpu_jpegdecoder.hpp
perception/tensorrt_bevdet/include/gatherbev_plugin.hpp:45:  Single-parameter constructors should be marked explicit.  [runtime/explicit] [5]
Done processing perception/tensorrt_bevdet/include/gatherbev_plugin.hpp
perception/tensorrt_bevdet/include/preprocess.hpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
Done processing perception/tensorrt_bevdet/include/preprocess.hpp
perception/tensorrt_bevdet/src/bevdet.cpp:0:  No copyright message found.  You should have a line: "Copyright [year] <Copyright Owner>"  [legal/copyright] [5]
perception/tensorrt_bevdet/src/bevdet.cpp:190:  Using C-style cast.  Use static_cast<float>(...) instead  [readability/casting] [4]

...

Signed-off-by: liu cui <[email protected]>

…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>

kminoda · 2024-07-29T02:26:57Z

@cyn-liu As this PR is stale for a week, from the maintenance perspective, let me make this PR a draft for now. Feel free to re-open the PR once all the CIs are fixed and ready to be reviewed. Thank you for your understanding, and we are looking forward to reviewing your PR.

Signed-off-by: liu cui <[email protected]>

perception/autoware_tensorrt_bevdet/src/bevdet.cpp

Signed-off-by: liu cui <[email protected]>

perception/autoware_tensorrt_bevdet/src/bevdet.cpp

…ernal bevdet_vendor library Signed-off-by: liu cui <[email protected]>

Signed-off-by: liu cui <[email protected]>

…ck error Signed-off-by: liu cui <[email protected]>

mitsudome-r · 2024-11-20T07:39:18Z

You should also update build_depends.repos which is used for the build-and-test CI.
https://github.com/autowarefoundation/autoware.universe/blob/main/build_depends.repos

Signed-off-by: liu cui <[email protected]>

liuXinGangChina · 2024-11-25T08:52:39Z

@xmfcx Hi， Mr.Fatih

we found a error rasied by ci, we are not familiar with this one,

Will you help us with this, for now we don't know how to resolve it

Best Regards

Lucas

perception/autoware_tensorrt_bevdet/CMakeLists.txt

Signed-off-by: liu cui <[email protected]>

mitsudome-r · 2024-12-04T06:11:07Z

perception/autoware_tensorrt_bevdet/src/ros_utils.cpp

+void imageTransport(std::vector<cv::Mat> imgs, uchar * out_imgs, size_t width, size_t height)
+{
+  uchar * temp = new uchar[width * height * 3];
+  uchar * temp_gpu = nullptr;
+  CHECK_CUDA(cudaMalloc(&temp_gpu, width * height * 3));
+
+  for (size_t i = 0; i < imgs.size(); i++) {
+    cv::cvtColor(imgs[i], imgs[i], cv::COLOR_BGR2RGB);
+    CHECK_CUDA(cudaMemcpy(temp_gpu, imgs[i].data, width * height * 3, cudaMemcpyHostToDevice));
+    convert_RGBHWC_to_BGRCHW(temp_gpu, out_imgs + i * width * height * 3, 3, height, width);
+  }
+  delete[] temp;
+  CHECK_CUDA(cudaFree(temp_gpu));
+}


@Shin-kyoto AutoCore has moved out the copied code as the vendor package from the suggestions you made here: #7956 (comment).

Now most of the code is following the coding guideline I think except for this code that actually converts the image format into a input format that is expected by the library.

Do you think this would be acceptable?

my suggestion was applied ✅

feat(tensorrt_bevdet): add new 3d object detection method

e6d39e6

Signed-off-by: liu cui <[email protected]>

cyn-liu requested review from miursh, YoshiRi and yukkysaito as code owners July 11, 2024 03:52

github-actions bot added type:documentation Creating or refining documentation. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) component:launch Launch files, scripts and initialization tools. (auto-assigned) labels Jul 11, 2024

style(pre-commit): autofix

1e1fd44

kminoda added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Jul 11, 2024

kminoda reviewed Jul 11, 2024

View reviewed changes

perception/tensorrt_bevdet/package.xml Outdated Show resolved Hide resolved

perception/tensorrt_bevdet/include/bevdet_node.hpp Outdated Show resolved Hide resolved

cyn-liu and others added 3 commits July 12, 2024 10:15

feat(tensorrt_bevdet): add new 3d object detection method

f17c6b0

Signed-off-by: liu cui <[email protected]>

Merge branch 'feat/add_tensorrt_bevdet' of github.com:cyn-liu/autowar…

9806852

…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

eca8ed0

cyn-liu linked an issue Jul 16, 2024 that may be closed by this pull request

Implementing BEVDet in Autoware autowarefoundation/autoware#4635

Open

7 tasks

cyn-liu added 2 commits July 19, 2024 13:48

feat(tensorrt_bevdet): add new 3d object detection method

acb33f0

Signed-off-by: liu cui <[email protected]>

Merge branch 'feat/add_tensorrt_bevdet' of github.com:cyn-liu/autowar…

741e917

…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>

cyn-liu requested a review from technolojin as a code owner July 19, 2024 06:06

style(pre-commit): autofix

3baff54

cyn-liu and others added 3 commits July 23, 2024 17:45

feat(tensorrt_bevdet): add new 3d object detection method

0dac361

Signed-off-by: liu cui <[email protected]>

Merge branch 'feat/add_tensorrt_bevdet' of github.com:cyn-liu/autowar…

67f1d92

…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

3149fd3

technolojin self-assigned this Jul 26, 2024

kminoda marked this pull request as draft July 29, 2024 02:27

cyn-liu and others added 2 commits July 31, 2024 18:29

feat(tensorrt_bevdet): add new 3d object detection method

956fd28

Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

35561b9

cyn-liu and others added 6 commits September 10, 2024 20:05

fix: add how to use in readme.md

cea3f9e

Signed-off-by: liu cui <[email protected]>

fix: replace ptr with std::vector

8a07984

Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

b6b1f91

trying fix pre-commit.ci error

7dc00d7

Signed-off-by: liu cui <[email protected]>

trying fix pre-commit.ci error

dbc0337

Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

dc6fd9b

cyn-liu marked this pull request as ready for review September 18, 2024 09:30

Shin-kyoto requested changes Sep 20, 2024

View reviewed changes

perception/autoware_tensorrt_bevdet/src/bevdet.cpp Outdated Show resolved Hide resolved

cyn-liu and others added 2 commits September 23, 2024 19:56

replace shared_ptr with std::vector

13b4c5b

Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

93556de

Shin-kyoto requested changes Sep 25, 2024

View reviewed changes

perception/autoware_tensorrt_bevdet/src/bevdet.cpp Outdated Show resolved Hide resolved

cyn-liu and others added 6 commits November 19, 2024 14:13

Merge branch 'main' into feat/add_tensorrt_bevdet

57f2c1f

refactor: migration of open-source tensorrt-bevdet source code to ext…

04aa368

…ernal bevdet_vendor library Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

b154357

fix: add missing links in the readme.md

02f0178

Signed-off-by: liu cui <[email protected]>

fix: mark TRTBEVDetNode constructor to explicit and try fix spell-che…

d3608e7

…ck error Signed-off-by: liu cui <[email protected]>

style(pre-commit): autofix

960e6b7

cyn-liu mentioned this pull request Nov 19, 2024

fix(autoware.repos): add bevdet_vendor external package autowarefoundation/autoware#5456

Open

4 tasks

cyn-liu added 3 commits November 20, 2024 17:57

Merge branch 'main' into feat/add_tensorrt_bevdet

c4c5cbc

fix: add bevdet_vendor dependency in build_depends.repos

110e975

Signed-off-by: liu cui <[email protected]>

fix: simplify the import check of bevdet_vendor package

eeb670a

Signed-off-by: liu cui <[email protected]>

cyn-liu force-pushed the feat/add_tensorrt_bevdet branch from bc7d287 to 1a5038b Compare November 21, 2024 09:30

fix: modify README.md

4583652

Signed-off-by: liu cui <[email protected]>

cyn-liu force-pushed the feat/add_tensorrt_bevdet branch from 1a5038b to 4583652 Compare November 21, 2024 09:47

xmfcx previously requested changes Nov 25, 2024

View reviewed changes

perception/autoware_tensorrt_bevdet/CMakeLists.txt Outdated Show resolved Hide resolved

fix: delete unnecessary spaces in CMakeLists.txt

5611257

Signed-off-by: liu cui <[email protected]>

mitsudome-r reviewed Dec 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

cyn-liu commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

kminoda left a comment

kminoda commented Jul 11, 2024

kminoda commented Jul 22, 2024 •

edited

Loading

technolojin commented Jul 23, 2024

kminoda commented Jul 29, 2024

mitsudome-r commented Nov 20, 2024

liuXinGangChina commented Nov 25, 2024

mitsudome-r Dec 4, 2024

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

Are you sure you want to change the base?

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

Conversation

cyn-liu commented Jul 11, 2024 • edited Loading

Description

Related links

How was this PR tested?

Notes for reviewers

Interface changes

Topic changes

Additions and removals

ROS Parameter Changes

Additions and removals

Modifications

Effects on system behavior

github-actions bot commented Jul 11, 2024 • edited Loading

kminoda left a comment

Choose a reason for hiding this comment

kminoda commented Jul 11, 2024

kminoda commented Jul 22, 2024 • edited Loading

technolojin commented Jul 23, 2024

kminoda commented Jul 29, 2024

mitsudome-r commented Nov 20, 2024

liuXinGangChina commented Nov 25, 2024

mitsudome-r Dec 4, 2024

Choose a reason for hiding this comment

cyn-liu commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

kminoda commented Jul 22, 2024 •

edited

Loading