zed camera+nvblox won't work on Jetson Orin

[AndreV84]

Hi
Have anyone had success running the nvblox with zed on Jetson orin?
which topics from zed need to be subscribed to? what exactly are the color topics?
Thanks
AV

[swapnesh-wani-nvidia]

You can match Zed topics to the topic description listed here.

[AndreV84]

@swapnesh-wani-nvidia
did you test it?
Stereolabs doesn't plan Humble support until 2023, as it seems
Foxy topics do not match Humble nvblox deployment
Foxy deployment neither works on orin, but works on AGX

[AndreV84]

on orin it was not possible to visualize on foxy nvblox node from foxy zed topics at all,
neither on nvblox humble from zed foxy topics
but it was possible to visualize on on Jetson AGX[not orin], but without color[ just grey] also with some corruption as it seems

[nakai-omer]

@AndreV84 We were having the same issue for integrating with a ZED mini camera. The main issue was that this lib doesn't support the Zed's BGRA8 image encoding, only RGB8. We have create a PR to overcome this issue: #32.
This is how our topic mapping are:

   remappings=[('depth/image', '/zedm/zed_node/depth/depth_registered'),
                        ('depth/camera_info', '/zedm/zed_node/depth/camera_info'),
                        ('color/image', '/zedm/zed_node/rgb/image_rect_color'),
                        ('color/camera_info', '/zedm/zed_node/rgb/camera_info'), ]),

The mesh in RVIZ doesn't look good yet, we are still investigating to why. If you get a chance to check it out, please let us know your results.

[AndreV84]

@hemalshahNV
Has the issue been closed as resolved or as insoluble?

[podomong]

Because zed_wrapper publishes color images encoded in BGRA8, nvblox cannot consume them. So you need to encode the color image to RGB8. The zed-ros2-examples repository contains the zed_rgb_convert package. This package converts images encoded in BGRA8 to BGR8. Change BGR8 to RGB8 and then build the package.

And depth images published by zed may have black pixels to distinguish boudaries from overlapping objects. Note depth settings document. I think that nvblox tend to be difficult for localizing voxels when it use black pixels from depth image.

If you set the depth option following bellow in common.yaml of the zed sdk, it can naturally fill in those black pixels.

option 1. ------------------------------------

depth:
quality: 1 #PERFORMANCE
sensing_mode: 1 #FILL
depth_confidence: 100

option 2. ------------------------------------

depth:
quality: 4 #NEURAL
sensing_mode: 0 #STANDARD
depth_confidence: 100

I've tested these settings on Xavier NX, Jetpack 4.6.2 and ZED2.

[nakai-omer]

@podomong I have created a PR that adds support for all main color encoding inside this package: #32.

[AndreV84]

@podomong so the nvblox is intact in your design, in order to get zed working with nvbolx on Jetson - I just clone the https://github.com/stereolabs/zed-ros2-examples/ repository, then change " BGR8 to RGB8 ", then build the package? right? how do I have to run the example build? concurrently with nvblox? OR you suggest to edit the BGR8 to RGB8 in nvblox code?

@nakai-omer could you extend how to use your suggested submodule to verity on Jetson with zed camera, please? soes it have to be running concurrently with nvblox in order to do conversion?

[AndreV84]

if to run the convert like that it will simply crash not starting the node at all

ros2 run zed_rgb_convert zed_rgb_convert

[nakai-omer]

@AndreV84 It is this repo, just with an added library, you will need to checkout the PR, and build it from source (notice there is a submodule so you will need to run git submodule update --init --recursive

[AndreV84]

@nakai-omer thank you for your reply
Could you extend what you mean by PR, please?

[gulerburak]

Has anyone managed to get this working?