feat: simulation and benchmark suite

[maciejmajek]

Purpose

Until now, RAI has been tested and evaluated manually which limited our capabilities to track performance along versions and different models.

Proposed Changes

Introducing rai_sim and rai_bench packages.

Issues

Testing

Instructions for running a single benchmark:

1. Setup the repository

poetry install --with openset
colcon build --symlink-install
source setup_shell.sh

2. Download GameLauncher binary: humble or jazzy

3. Populate src/rai_bench/rai_bench/o3de_test_bench/configs/o3de_config.yaml with:

binary_path: /path/to/your/GameLauncher
robotic_stack_command: ros2 launch examples/manipulation-demo-no-binary.launch.py
required_simulation_ros2_interfaces:
  services:
    - /spawn_entity
    - /delete_entity
  topics:
    - /color_image5
    - /depth_image5
    - /color_camera_info5
  actions: []
required_robotic_ros2_interfaces:
  services:
    - /grounding_dino_classify
    - /grounded_sam_segment
    - /manipulator_move_to
  topics: []
  actions: []

4. Run the benchmark

python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py

[jmatejcz]

requested changes proposed here: #438

[MagdalenaKotynia]

I noticed that the simulation stops responding and quits after the window of the simulations is manually resized.

[knicked]

I noticed that the simulation stops responding and quits after the window of the simulations is manually resized.

I tried to investigate this problem but its root probably lies deep in the o3de rendering pipeline, it happens even in an empty o3de project. I opened an issue here. I also opened a PR that allows to disable the resizability of a window using a command-line parameter. I can add it to our binaries as a workaround.

[boczekbartek]

@maciejmajek
I've tested on ubuntu22.04 ~ ros2 humble. In general it worked as expected, good job! However, I noticed some issues, could you have a look?

1. I noticed failed get_objects_positions tool call in the logs. I think rate of images is much higher than 1.0s, so it's unexpected. Also, this error occurred only in the first scenario, which started after some time, because grounded_dino model had to be downloaded. I am wondering if long wait might have affected the first scenario. Once benchmark was ran second time (gdino model was loaded from cache), error didn't happen.

2025-03-05 23:12:33 robo-pc-005 Agent logger[1422059] INFO Error in "get_object_positions", error: No message received from topic: /depth_image5 within 1.0 seconds

2. In general I got inconsistent scores between runs, which I think can be expected due to llm agents nature, is that correct @maciejmajek?
   log.log: first run
   log2.log: second run

(rai-framework-py3.10) robo-pc-005 ➜  rai git:(feat/simbench) ✗ grep 'TASK SCORE' log*.log
log2.log:2025-03-05 23:34:52 robo-pc-005 Benchmark logger[1435873] INFO TASK SCORE: 1.0, TOTAL TIME: 55.830, NUM_OF_TOOL_CALLS: 5
log2.log:2025-03-05 23:37:29 robo-pc-005 Benchmark logger[1435873] INFO TASK SCORE: 0.0, TOTAL TIME: 154.256, NUM_OF_TOOL_CALLS: 7
log2.log:2025-03-05 23:38:43 robo-pc-005 Benchmark logger[1435873] INFO TASK SCORE: 0.0, TOTAL TIME: 70.599, NUM_OF_TOOL_CALLS: 5
log2.log:2025-03-05 23:39:38 robo-pc-005 Benchmark logger[1435873] INFO TASK SCORE: 0.0, TOTAL TIME: 52.363, NUM_OF_TOOL_CALLS: 7
log.log:2025-03-05 23:14:20 robo-pc-005 Benchmark logger[1422059] INFO TASK SCORE: 0.0, TOTAL TIME: 128.217, NUM_OF_TOOL_CALLS: 5
log.log:2025-03-05 23:15:46 robo-pc-005 Benchmark logger[1422059] INFO TASK SCORE: 0.5, TOTAL TIME: 83.618, NUM_OF_TOOL_CALLS: 7
log.log:2025-03-05 23:17:03 robo-pc-005 Benchmark logger[1422059] INFO TASK SCORE: 0.0, TOTAL TIME: 73.732, NUM_OF_TOOL_CALLS: 6
log.log:2025-03-05 23:17:30 robo-pc-005 Benchmark logger[1422059] INFO TASK SCORE: 1.0, TOTAL TIME: 25.040, NUM_OF_TOOL_CALLS: 6

4. Is there currently a way of seeing the aggregated result of benchmark?

5. Cleanup failed for grounding_dino and moveit - this might be misleading for the user, because seeing these errors one might think that. I think it's not blocking to merge this PR, but leaving an issue would be nice.

`grounding_dino`

[grounding_dino-1] Traceback (most recent call last):
[grounding_dino-1]   File "/home/bboczek/projects/01_internal/01_repos/rai/install/rai_open_set_vision/lib/rai_open_set_vision/grounding_dino", line 33, in <module>
[grounding_dino-1]     sys.exit(load_entry_point('rai-open-set-vision', 'console_scripts', 'grounding_dino')())
[grounding_dino-1]   File "/u24/home/bboczek/projects_u24/01_internal/01_repos/rai/build/rai_open_set_vision/rai_open_set_vision/services/grounding_dino.py", line 136, in main
[grounding_dino-1]     rclpy.shutdown()
[grounding_dino-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/__init__.py", line 130, in shutdown
[grounding_dino-1]     _shutdown(context=context)
[grounding_dino-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/utilities.py", line 58, in shutdown
[grounding_dino-1]     return context.shutdown()
[grounding_dino-1]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/context.py", line 102, in shutdown
[grounding_dino-1]     self.__context.shutdown()
[grounding_dino-1] rclpy._rclpy_pybind11.RCLError: failed to shutdown: rcl_shutdown already called on the given context, at ./src/rcl/init.c:241

`moveit`

[move_group-5]          at line 127 in ./src/class_loader.cpp
[move_group-5] Stack trace (most recent call last):
[move_group-5] #16   Object "", at 0xffffffffffffffff, in
[move_group-5] #15   Object "/opt/ros/humble/lib/moveit_ros_move_group/move_group", at 0x622b04626724, in
[move_group-5] #14   Source "../csu/libc-start.c", line 392, in __libc_start_main_impl [0x7d459ce29e3f]
[move_group-5] #13   Source "../sysdeps/nptl/libc_start_call_main.h", line 58, in __libc_start_call_main [0x7d459ce29d8f]
[move_group-5] #12   Object "/opt/ros/humble/lib/moveit_ros_move_group/move_group", at 0x622b0462565b, in
[move_group-5] #11   Object "/opt/ros/humble/lib/moveit_ros_move_group/move_group", at 0x622b04627709, in
[move_group-5] #10   Object "/opt/ros/humble/lib/libmoveit_cpp.so.2.5.7", at 0x7d459da84486, in moveit_cpp::MoveItCpp::~MoveItCpp()
[move_group-5] #9    Object "/opt/ros/humble/lib/libmoveit_cpp.so.2.5.7", at 0x7d459da827b9, in
[move_group-5] #8    Object "/opt/ros/humble/lib/libmoveit_trajectory_execution_manager.so.2.5.7", at 0x7d459cd8d395, in trajectory_execution_manager::TrajectoryExecutionManager::~TrajectoryExecutionManager()
[move_group-5] #7    Object "/opt/ros/humble/lib/libmoveit_trajectory_execution_manager.so.2.5.7", at 0x7d459cd9ffa9, in
[move_group-5] #6    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6e49dc, in rclcpp::Node::~Node()
[move_group-5] #5    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6e497e, in rclcpp::Node::~Node()
[move_group-5] #4    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6bfc59, in
[move_group-5] #3    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6ee029, in
[move_group-5] #2    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6edf70, in rclcpp::node_interfaces::NodeBase::~NodeBase()
[move_group-5] #1    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6bfc59, in
[move_group-5] #0    Object "/opt/ros/humble/lib/librclcpp.so", at 0x7d459d6c49d1, in rclcpp::CallbackGroup::~CallbackGroup()
[move_group-5] Segmentation fault (Address not mapped to object [0x7d4595dcc798])

[maciejmajek]

@boczekbartek thank you for the review. I will submit the issues regarding these problems. Let's merge this PR and work on the minor problems in the next week