Error when launching RAIManipulationDemo

[theMiriam]

Describe the bug
I followed the README to download RAI and the README to use the robotic arm demo (https://github.com/RobotecAI/rai/blob/main/docs/demos/manipulation.md).
When I run the command to launch it "ros2 launch examples/manipulation-demo.launch.py game_launcher:=path/to/RAIManipulationDemo.GameLauncher" I get some errors. I extracted the project into: /home/rai/src/examples/rai-manipulation-demo/ros2_ws/src.

To Reproduce
Follow the step at https://github.com/RobotecAI/rai/blob/main/docs/demos/manipulation.md

Expected behavior
Being able to perform simple tasks like moving the cubes.

Screenshots

Platform

• OS: Ubuntu 24.04
• ROS 2 Version Jazzy

Version
Main tag 1.1.0

Additional context
In /home//rai/src/examples/rai-manipulation-demo/ros2_ws/src/RAIManipulationDemoGamePackage/ i don't have Assets/Engine/exclude.filetag, I don't understand how to create it. Also, inside RAI there is a .urdf.txt file but it is empty

[maciejmajek]

Hi @theMiriam, thank you for the issue. @knicked could you please take a look?

[knicked]

Hi @theMiriam, these errors you highlighted should not have an impact on running the binary launcher, this is how my terminal outputs look when I launch it:

Can you try to launch the binary directly, rather than by using the ros2 launch command? Navigate to the directory where you unpacked the binary and run ./RAIManipulationDemo.GameLauncher, see if the scene loads correctly, it should look exactly like in the screenshot above. If your terminal outputs anything else than what is shown here, please let us know and we will try to resolve the issues.

[theMiriam]

Thanks for the quick response!
Launching the binary directly, the scene loads correctly, and my terminal looks exactly like yours.
But when I open the interface to send commands using "streamlit run examples/manipulation-demo-streamlit.py", and i write a simple commands like "pick up the red cube and drop it on the blue cub" it doesn't work, and I get errors. I thought it wasn't working because of the errors I mentioned earlier.

Additionally, I don't have the topic camera/image_raw.

(Given this new information, do I need to change the issue title?)

[knicked]

It looks like you stumbled upon one of the issues that happens to us from time to time, which we fixed internally already but haven't released it yet. It is related to the simulation trying to force the manipulator's gripper position to 0.0, which, when done imperfectly, sets the position to a small negative value and then the moveit package complains that it's an invalid state. Try downloading this package instead and see if this fixes the issue:
https://robotecai.sharepoint.com/:u:/g/EemJ_CqFGd5JqFVYxJhFDx0B-CD4bKrwHGrUrG6Q5KozIQ

As to the camera related topics, the correct topic that should work is /camera_image5, which you can see is showing up when you do ros2 topic list. The fact that the get_camera_image tool is trying to listen to other topics is because it's entirely controlled by the LLM agent, which is supposed to first call a tool to get the available topics, and it sometimes refuses to do it, but it only confirms that the agent is (in some sense) working correctly :)

[theMiriam]

Ok, with the new package, I no longer get errors, and the arm moves. However, it doesn’t execute the command I give it. Is there anything I can do?

Could this issue be related to the camera problem? Because the arm always performs the same movement to the right and closes the gripper, no matter what command I give it.

Is this a default movement when it doesn’t detect the camera?
Thank you for your time and support. 😊

[maciejmajek]

Interesting behavior. Could you please check if any of the "Completed tool calling" expanders show "get_object_positions" or "GetObjectPositionsTool"? If so, what are the centroid values? The requested values of 10.0, 20.0, and 30.0 are far outside the reach of the arm.

I can see that the model struggles with tool calls. For example, the tool "end_effector_interaction_status" does not exist (implying hallucination problems), and there's a validation error in the move_to_point tool, which suggests the model is having difficulty.

What model are you using?

[theMiriam]

"get_object_positions" or "GetObjectPositionsTool" are not present. I'm using the LLaMA 3.2 model. However, I was wrong to say that it always moves to the right—the movement changes, but once it moves to a position, it no longer moves even if I give another command.