Unable to use GPU via docker(Open AD Kit)

[KTKTKT0401]

I'm having trouble with the performance of rosbag simulation using docker 'devel' image. And I think the this happens because my GPU is not working correctly.

Through the following workflow, I arrived at the hypothesis that the GPU is not functioning properly.

Does anyone have the same issue, or could you please give me some advice?

---Workflow1 -planning simulation---
I tried the planning simulation in the docker 'devel' image.
The simulation looks fine, but below are the results of nvidia-smi and nvcc -V.
It looks like the GPU is not working.

---Workflow2 -rosbag simulation---
When trying to use 'rosbag simulation' at docker 'devel' image, an error occured that '~/autoware_data' does not exists.
I checked inside the docker container, and I could check 'autoware_map' is mounted but 'autoware_data' is not mounted.

So inside the /docker/run.sh, I added a script at docker run command to mount 'autoware_data'(${AUTOWARE_DATA}is the script I added(Path to my autoware_data in my host.)).

The simulation has large latency visualizing lidar pointcloud.
The vehicles and pedestrians are not recognized.
Below are the results of nvidia-smi and nvcc -V.
It looks like the GPU is not working.

PS)
I could activate rosbag simulation without mounting 'autoware_data' when using the launch command discussed in the URL below.
https://github.com/orgs/autowarefoundation/discussions/4775#discussion-6731288

[sasakisasaki]

@KTKTKT0401 Hello! I had the similar issue before: not only the pointcloud topic has huge delay and also the other topics are the same. Perhaps following the Cyclonedds configuration might be needed inside of the docker container.

export ROS_LOCALHOST_ONLY=1
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
sysctl -w net.core.rmem_max=2147483647
ip link set lo multicast on

Ref. https://autowarefoundation.github.io/autoware-documentation/main/installation/additional-settings-for-developers/network-configuration/dds-settings/

Personally the most effective one was export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp which enables the CycloneDDS. Hopefully this provides some hints for your debug.

[sasakisasaki]

Sorry, there was one missing information in my message.

My proposal would might be effective for the following issue, but not be effective for the issue why the CUDA is not used.

The simulation has large latency visualizing lidar pointcloud.

[KTKTKT0401]

@sasakisasaki
Hello!
Thank you for giving me advice.
I've checked the dds-settings, but looks like its fine.
I think I need to look further about the cuda related issues to fix my environment.

[liuXinGangChina]

hi @KTKTKT0401

if you pull the right docker ，and pull up with right “docker run command”，docker will work normally by calling your nv-gpu， please follow autoware official document link

happy hacking！

Xingang

[KTKTKT0401]

Hi, @liuXinGangChina

I tried the rocker command(https://autowarefoundation.github.io/autoware-documentation/release-v1.0_beta/installation/autoware/docker-installation/), and failed to run autoware.
It looks like 'autoware-universe:latest-cuda' no more exists on github.

So, I tried to find out the docker image in github. Although, there are so many tagged version of autoware with differnet names(https://github.com/autowarefoundation/autoware/pkgs/container/autoware/versions?filters%5Bversion_type%5D=tagged).
Does anyone know which version of autoware is a fixed version?

[liuXinGangChina]

Sorry for late reply

Please try this one docker link

since docker name pattern change a little bit and the document has not catch up yet

Happy haciking！

Xingang

[KTKTKT0401]

Hi @liuXinGangChina!

I've tried the docker installation for quick start, and used the image in the link provided.
(docker link)

The results are below.
1)Cannot use 'autoware:universe-devel-cuda-241125'
Since my PC is amd environment, I couldn't use the image provided.
So, I tried the image which had the similar image name(universe-devel-cuda-241122).
This image was available in amd environment.

2)Cannot open rviz2
I couldn't open rviz2 with the command below.
rocker --nvidia --x11 --user --volume $HOME/autoware --volume $HOME/autoware_map -- ghcr.io/autowarefoundation/autoware:universe-devel-cuda-20241122

By adding the '--device /dev/dri' to the rocker command, I was able to do 'Planning simulation' inside the container. Command is below.
rocker --nvidia --x11 --user --device /dev/dri --volume $HOME/autoware --volume $HOME/autoware_map -- ghcr.io/autowarefoundation/autoware:universe-devel-cuda-20241122

3)Cannot use rosbag simulation
When I tried to use rosbag simulation inside the container, the autoware didn't start with error. The error says, no path to 'autoware_data'.
So I added '--volume $HOME/autoware_data' in the rocker command.
Command is below.
rocker --nvidia --x11 --device /dev/dri --user --volume $HOME/autoware_map --volume $HOME/autoware_data -- ghcr.io/autowarefoundation/autoware:universe-devel-cuda-20241122

Although, there are still errors occuring when I start playing rosbag data.
The localization doesn't succeed.

Does anyone has the same issue?

[liuXinGangChina]

hi ktktkt，
the original docker gpu issue has been solved by this discussion，i will close this one and mark it answered
feel free to reach us when you have further question

happy hacking

心刚