Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue Persist - libtritonserver.so missing error , loadExtensions Error: GXF_FAILURE #31

Open
HappySamuel opened this issue Oct 1, 2024 · 0 comments

Comments

@HappySamuel
Copy link

agx-orin@agx-orin:~$ ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py

Run the above command and get the following error

[INFO] [launch]: All log files can be found below /home/agx-orin/.ros/log/2024-09-25-11-02-51-783634-agx-orin-72726
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [72739]
[component_container_mt-1] [INFO] [1727233372.330269030] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/lib/libresize_node.so
[component_container_mt-1] [INFO] [1727233372.379226314] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1727233372.379363181] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1727233372.384657529] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1727233372.384977375] [image_resize]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.386433629] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1727233372.394073112] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1727233372.398385072] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1727233372.403409622] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1727233372.408950055] [NitrosContext]: [NitrosContext] Loading application: '/home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1727233372.410260290] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233372.414354709] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2024-09-25 11:02:52.414 WARN  gxf/std/program.cpp@514: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1727233372.416194170] [image_resize]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.416252508] [image_resize]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.419034356] [image_resize]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.419152343] [image_resize]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.424359489] [image_resize]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.424474883] [image_resize]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.424957933] [image_resize]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.425791422] [image_resize]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1727233372.428377970] [image_resize]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1727233372.430388443] [image_resize]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1727233372.438524065] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1727233372.440148578] [image_resize]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1727233372.443976848] [image_resize]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1727233372.493788197] [image_resize]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1727233372.496638303] [image_resize]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1727233372.512293822] [image_resize]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/image_resize' in container '/dope_container'
[component_container_mt-1] [INFO] [1727233372.519357837] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/lib/libdnn_image_encoder_node.so
[component_container_mt-1] [INFO] [1727233372.524754427] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode>
[component_container_mt-1] [INFO] [1727233372.524862845] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode>
[component_container_mt-1] [INFO] [1727233372.530387950] [dope_encoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.532631835] [dope_encoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.532706237] [dope_encoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.534896425] [dope_encoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.534977419] [dope_encoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.537223257] [dope_encoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.537330619] [dope_encoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.537810341] [dope_encoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.539977521] [dope_encoder]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1727233372.547792400] [dope_encoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1727233372.553611430] [dope_encoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1727233372.772957781] [dope_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1727233372.779016592] [dope_encoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1727233372.796975933] [dope_encoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dope_encoder' in container '/dope_container'
[component_container_mt-1] [INFO] [1727233372.798867172] [image_resize]: Negotiating
[component_container_mt-1] [INFO] [1727233372.801147026] [dope_encoder]: Negotiating
[component_container_mt-1] [INFO] [1727233372.801253172] [dope_encoder]: Could not negotiate
[component_container_mt-1] [INFO] [1727233372.802279849] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_triton/lib/libtriton_node.so
[component_container_mt-1] [INFO] [1727233372.805071298] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1727233372.805131715] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1727233372.813943222] [dope_inference]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.817003157] [dope_inference]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1727233372.817070742] [dope_inference]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1727233372.817202073] [dope_inference]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.817225817] [dope_inference]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.822337857] [dope_inference]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.822447907] [dope_inference]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.827121122] [dope_inference]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.827204676] [dope_inference]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.827795760] [dope_inference]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.832051815] [dope_inference]: [NitrosContext] Loading extension: gxf/triton/libgxf_triton_ext.so
[component_container_mt-1] [ERROR] [1727233372.834654684] [dope_inference]: [NitrosContext] dlopen failed when opening "/home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_triton/share/isaac_ros_triton/gxf/triton/libgxf_triton_ext.so": libtritonserver.so: cannot open shared object file: No such file or directory
[component_container_mt-1] [ERROR] [1727233372.834745054] [dope_inference]: [NitrosNode] loadExtensions Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1727233372.834981954] [dope_inference]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1727233372.835018435] [dope_inference]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] 2024-09-25 11:02:52.835 ERROR gxf/std/program.cpp@533: Attempted interrupting when not running (state=0).
[component_container_mt-1] 2024-09-25 11:02:52.835 ERROR gxf/core/runtime.cpp@1400: Graph interrupt failed with error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [ERROR] [1727233372.835097797] [dope_inference]: [NitrosContext] GxfGraphInterrupt Error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [INFO] [1727233372.835118213] [dope_inference]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1727233372.835150214] [dope_inference]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1727233372.835184231] [dope_inference]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1727233372.835225895] [dope_inference]: [NitrosNode] Application termination done
[component_container_mt-1] [ERROR] [1727233372.842207317] [dope_container]: Component constructor threw an exception: [NitrosNode] loadExtensions Error: GXF_FAILURE
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'dope_inference' of type 'nvidia::isaac_ros::dnn_inference::TritonNode' in container '/dope_container': Component constructor threw an exception: [NitrosNode] loadExtensions Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1727233372.847150170] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_dope/lib/libdope_decoder_node.so
[component_container_mt-1] [INFO] [1727233372.854090055] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dope::DopeDecoderNode>
[component_container_mt-1] [INFO] [1727233372.854179241] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dope::DopeDecoderNode>
[component_container_mt-1] [INFO] [1727233372.863758924] [dope_decoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.868125733] [dope_decoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.868309608] [dope_decoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.873930971] [dope_decoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.874028733] [dope_decoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.874800076] [dope_decoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.874841933] [dope_decoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.874869390] [dope_decoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.880137177] [dope_decoder]: [NitrosContext] Loading extension: gxf/lib/dope/libgxf_dope.so
[component_container_mt-1] [INFO] [1727233372.890965365] [dope_decoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1727233372.892724281] [dope_decoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1727233372.896013852] [dope_decoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1727233372.896348963] [dope_decoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1727233372.900684539] [dope_decoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dope_decoder' in container '/dope_container'
[component_container_mt-1] [INFO] [1727233373.515063117] [image_resize]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1727233373.515239856] [image_resize]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1727233373.515301585] [image_resize]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1727233373.515341874] [image_resize]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.515365875] [image_resize]: [NitrosPublisher] Use only the compatible publisher: topic_name="/resize/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1727233373.515390931] [image_resize]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.515412308] [image_resize]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1727233373.515434068] [image_resize]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.515485621] [image_resize]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1727233373.515593079] [image_resize]: [NitrosPublisherSubscriberGroup] Adjusted the compatible format of the component "input_compositor/image_in" (type="nvidia::gxf::DoubleBufferReceiver") from "nitros_image_bgr8" to "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1727233373.516048961] [image_resize]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1727233373.532879383] [image_resize]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/VIQFKZGMPC/VIQFKZGMPC.yaml"
[component_container_mt-1] [INFO] [1727233373.533028058] [image_resize]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1727233373.533128764] [image_resize]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1727233373.533163965] [image_resize]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/VIQFKZGMPC/VIQFKZGMPC.yaml'
[component_container_mt-1] [INFO] [1727233373.542136243] [image_resize]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1727233373.542612765] [image_resize]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1727233373.542671614] [image_resize]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1727233373.542866114] [image_resize]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233373.797827684] [dope_encoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1727233373.797980999] [dope_encoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1727233373.798019816] [dope_encoder]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.798045001] [dope_encoder]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_pub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1727233373.798073609] [dope_encoder]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1727233373.799566088] [dope_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1727233373.868288414] [image_resize]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1727233373.868732007] [image_resize]: [NitrosNode] Starting a heartbeat timer (eid=17)
[component_container_mt-1] [INFO] [1727233373.901425792] [dope_decoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1727233373.901550850] [dope_decoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1727233373.901570723] [dope_decoder]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.901588835] [dope_decoder]: [NitrosPublisher] Use only the compatible publisher: topic_name="/poses", data_format="nitros_pose_array"
[component_container_mt-1] [INFO] [1727233373.901600419] [dope_decoder]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.901609827] [dope_decoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/tensor_sub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1727233373.901848424] [dope_decoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1727233373.904317242] [dope_decoder]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/QAOLSFIFQD/QAOLSFIFQD.yaml"
[component_container_mt-1] [INFO] [1727233373.904385756] [dope_decoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1727233373.904398428] [dope_decoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1727233373.904412412] [dope_decoder]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/QAOLSFIFQD/QAOLSFIFQD.yaml'
[component_container_mt-1] [INFO] [1727233373.906357892] [dope_decoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1727233373.906511079] [dope_decoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1727233373.906601385] [dope_decoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233373.908811798] [dope_decoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1727233373.909311424] [dope_decoder]: [NitrosNode] Starting a heartbeat timer (eid=67)
[component_container_mt-1] [INFO] [1727233373.911403179] [dope_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/WFJKNUFLZT/WFJKNUFLZT.yaml"
[component_container_mt-1] [INFO] [1727233373.911514989] [dope_encoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1727233373.911545134] [dope_encoder]: In DNN Image Encoder Node preLoadGraphCallback().
[component_container_mt-1] [INFO] [1727233373.911627119] [dope_encoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1727233373.911648016] [dope_encoder]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/WFJKNUFLZT/WFJKNUFLZT.yaml'
[component_container_mt-1] [INFO] [1727233373.917195840] [dope_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1727233373.917432837] [dope_encoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1727233373.917458534] [dope_encoder]: In DNN Image Encoder Node postLoadGraphCallback().
[component_container_mt-1] [INFO] [1727233373.917694859] [dope_encoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233373.944590862] [dope_encoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1727233373.944967477] [dope_encoder]: [NitrosNode] Starting a heartbeat timer (eid=98)

Have tried the solution suggested in #19 , but the issue still persist. Any solution for this ?

Best
Samuel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant