Skip to content

Commit

Permalink
Merge pull request #13 from NVIDIA-ISAAC-ROS/release-dp3.1
Browse files Browse the repository at this point in the history
Isaac ROS 0.31.0 (DP3.1)
  • Loading branch information
jaiveersinghNV authored May 26, 2023
2 parents 169cdba + e5f327c commit fd902ff
Show file tree
Hide file tree
Showing 14 changed files with 82 additions and 62 deletions.
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ This package is powered by [NVIDIA Isaac Transport for ROS (NITROS)](https://dev

The following table summarizes the per-platform performance statistics of sample graphs that use this package, with links included to the full benchmark output. These benchmark configurations are taken from the [Isaac ROS Benchmark](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark#list-of-isaac-ros-benchmarks) collection, based on the [`ros2_benchmark`](https://github.com/NVIDIA-ISAAC-ROS/ros2_benchmark) framework.

| Sample Graph | Input Size | AGX Orin | Orin NX | x86_64 w/ RTX 3060 Ti |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Proximity Segmentation Node](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/scripts//isaac_ros_bi3d_node.py) | 576p | [45.9 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_bi3d_node-agx_orin.json)<br>32 ms | [26.3 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_bi3d_node-orin_nx.json)<br>65 ms | [148 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_bi3d_node-x86_64_rtx_3060Ti.json)<br>22 ms |
| Sample Graph | Input Size | AGX Orin | Orin NX | x86_64 w/ RTX 4060 Ti |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| [Proximity Segmentation Node](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/scripts//isaac_ros_bi3d_node.py) | 576p | [49.3 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_bi3d_node-agx_orin.json)<br>32 ms | [24.2 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_bi3d_node-orin_nx.json)<br>65 ms | [159 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_bi3d_node-nuc_4060ti.json)<br>23 ms |

## Table of Contents

Expand Down Expand Up @@ -66,7 +66,7 @@ The following table summarizes the per-platform performance statistics of sample

## Latest Update

Update 2023-04-05: Source available GXF extensions
Update 2023-05-25: Performance improvements.

## Supported Platforms

Expand Down Expand Up @@ -328,6 +328,7 @@ Check [here](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference/blob/ma

| Date | Changes |
| ---------- | ------------------------------------------------------------------ |
| 2023-05-25 | Performance improvements |
| 2023-04-05 | Source available GXF extensions |
| 2022-08-31 | Update to use latest model and to be compatible with JetPack 5.0.2 |
| 2022-06-30 | Initial release |
21 changes: 9 additions & 12 deletions isaac_ros_bi3d/config/bi3d_node.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
%YAML 1.2
# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
# Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -252,7 +252,7 @@ components:
featnet_engine_file_path: "featnet_engine_file_path"
featnet_input_layers_name: ["input.1"]
featnet_output_layers_name: ["97"]

segnet_engine_file_path: "segnet_engine_file_path"
segnet_input_layers_name: ["input.1"]
segnet_output_layers_name: ["278"]
Expand Down Expand Up @@ -306,7 +306,7 @@ components:
type: nvidia::gxf::BlockMemoryPool
parameters:
storage_type: 1
block_size: 2211840
block_size: 2211840
num_blocks: 12
- name: forward_pool
type: nvidia::gxf::BlockMemoryPool
Expand Down Expand Up @@ -348,7 +348,7 @@ components:
featnet_engine_file_path: "featnet_engine_file_path"
featnet_input_layers_name: ["input.1"]
featnet_output_layers_name: ["97"]

segnet_engine_file_path: "segnet_engine_file_path"
segnet_input_layers_name: ["input.1"]
segnet_output_layers_name: ["278"]
Expand Down Expand Up @@ -408,7 +408,7 @@ components:
parameters:
rx: rx
tx: tx
---
---
name: bi3d_postprocess
components:
- name: pool
Expand Down Expand Up @@ -442,7 +442,7 @@ components:
disparity_tensor_name: "disparity"
disparity_values_tensor_name: "bi3d_inference_disparities"
---
name: bi3d_output_vault
name: bi3d_output_sink
components:
- name: input
type: nvidia::gxf::DoubleBufferReceiver
Expand All @@ -453,12 +453,10 @@ components:
parameters:
receiver: input
min_size: 1
- name: vault
type: nvidia::gxf::Vault
- name: sink
type: nvidia::isaac_ros::MessageRelay
parameters:
source: input
max_waiting_count: 1
drop_waiting: false
---
components:
- name: connection
Expand Down Expand Up @@ -556,7 +554,7 @@ components:
type: nvidia::gxf::Connection
parameters:
source: bi3d_postprocess/data_transmitter
target: bi3d_output_vault/input
target: bi3d_output_sink/input
---
name: scheduler
components:
Expand All @@ -567,4 +565,3 @@ components:
clock: clock
worker_thread_number: 8
stop_on_deadlock: false

8 changes: 5 additions & 3 deletions isaac_ros_bi3d/gxf/bi3d/3dv/src/Bi3DPreProcessor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@

namespace cvcore { namespace bi3d {

struct Bi3DPreProcessor::Bi3DPreProcessorImpl
struct Bi3DPreProcessor::Bi3DPreProcessorImpl
{
mutable detail::InputImage m_leftImageDevice;
mutable detail::InputImage m_rightImageDevice;
Expand All @@ -46,7 +46,7 @@ struct Bi3DPreProcessor::Bi3DPreProcessorImpl
ImagePreProcessingParams m_preProcessorParams;
ModelInputParams m_modelParams;

Bi3DPreProcessorImpl(const ImagePreProcessingParams & preProcessorParams,
Bi3DPreProcessorImpl(const ImagePreProcessingParams & preProcessorParams,
const ModelInputParams & modelInputParams,
cudaStream_t stream)
: m_preProcessorParams{preProcessorParams},
Expand Down Expand Up @@ -82,7 +82,7 @@ struct Bi3DPreProcessor::Bi3DPreProcessorImpl
// Bi3D Frontend
// =============================================================================

Bi3DPreProcessor::Bi3DPreProcessor(const ImagePreProcessingParams & preProcessorParams,
Bi3DPreProcessor::Bi3DPreProcessor(const ImagePreProcessingParams & preProcessorParams,
const ModelInputParams & modelInputParams,
cudaStream_t stream)
: m_pImpl{new Bi3DPreProcessorImpl(preProcessorParams, modelInputParams, stream)} {}
Expand All @@ -107,6 +107,7 @@ void Bi3DPreProcessor::execute(detail::PreprocessedImage & preprocessedLeftImage
m_pImpl->resizeBuffers(leftImage.getWidth(), leftImage.getHeight());

// Ensuring data is on the GPU
// FIXME: Rename so this is not confused with cvcore::Copy
Copy(m_pImpl->m_leftImageDevice, leftImage, stream);
Copy(m_pImpl->m_rightImageDevice, rightImage, stream);

Expand Down Expand Up @@ -144,6 +145,7 @@ void Bi3DPreProcessor::execute(detail::PreprocessedImage & preprocessedLeftImage
// Ensuring the buffers are appropreately allocated
m_pImpl->resizeBuffers(leftImage.getWidth(), leftImage.getHeight());

// FIXME: Rename so this is not confused with cvcore::Copy
Copy(m_pImpl->m_preLeftImageDevice, leftImage, stream);
Copy(m_pImpl->m_preRightImageDevice, rightImage, stream);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

namespace cvcore { namespace tensor_ops {

// TODO: Add doxygen style comments for each of the error codes
enum class TensorOpsErrorCode : std::int32_t
{
SUCCESS = 0,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ std::string GetCPUName()
#else
std::string GetCPUName()
{
// TODO: this will only work on linux platform
std::ifstream cpuInfo("/proc/cpuinfo");
if (!cpuInfo.good())
{
Expand Down
2 changes: 1 addition & 1 deletion isaac_ros_bi3d/gxf/bi3d/cvcore/src/tensor_ops/DBScan.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ void DBScan::doClusterAndMerge(Array<BBox> &input, Array<BBox> &output, BBoxMerg
doCluster(input, clusters);
output.setSize(m_numClusters);

// merge bboxes based on different modes
// merge bboxes based on different modes (TODO: might add mininum/average in the future)
if (type == MAXIMUM)
{
MergeMaximumBBoxes(input, clusters, output);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ std::error_code CreateVPIImageWrapperImpl(VPIImage &vpiImg, VPIImageData &imgdat
return make_error_code(vpiStatus);
}

// helper function to wrap VPI image for planar image types
// helper function to wrap VPI image for planar image types (TODO: not supported in vpi so far)
template<ImageType T, typename std::enable_if<IsPlanarImage<T>::value>::type * = nullptr>
std::error_code CreateVPIImageWrapperImpl(VPIImage &vpiImg, VPIImageData &imgdata, const Image<T> &cvcoreImage, VPIBackend backend)
{
Expand Down Expand Up @@ -224,6 +224,7 @@ std::error_code DestroyVPIImageWrapper(VPIImage &image, VPIImageData &imageWrap)
}
std::error_code VPITensorStream::Status() noexcept
{
// TODO Needs to be updated for supporting non blocking function calls.
return ErrorCode::SUCCESS;
}

Expand Down
2 changes: 2 additions & 0 deletions isaac_ros_bi3d/gxf/bi3d/cvcore/src/trtbackend/TRTBackend.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,7 @@ void TRTImpl::loadFromMemoryPointer(void *engine)

void TRTImpl::setupIO(int batchSize)
{
// @TODO: use getBindingDimensions to avoid re-setting the IO.
m_bindingsCount = m_inferenceEngine->getNbBindings();
for (int i = 0; i < m_bindingsCount; i++)
{
Expand Down Expand Up @@ -542,6 +543,7 @@ void TRTBackend::infer(void **buffer)

void TRTBackend::infer(void **buffer, int batchSize, cudaStream_t stream)
{
//@TODO: fix kMin, kOpt, kMax batch size in SetupProfile() call and then add a check here.
m_pImpl->setupIO(batchSize);

bool success = true;
Expand Down
74 changes: 45 additions & 29 deletions isaac_ros_bi3d/gxf/bi3d/extensions/tensor_ops/TensorOps.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,46 +30,62 @@
#include "gxf/std/extension_factory_helper.hpp"

GXF_EXT_FACTORY_BEGIN()
GXF_EXT_FACTORY_SET_INFO(0x6eae64ff97a94d9b, 0xb324f85e6a98a75a, "NvCvTensorOpsExtension",
"Generic CVCORE tensor_ops interfaces", "Nvidia_Gxf", "3.1.0", "LICENSE");
GXF_EXT_FACTORY_SET_INFO(
0x6eae64ff97a94d9b, 0xb324f85e6a98a75a, "NvCvTensorOpsExtension",
"Generic CVCORE tensor_ops interfaces", "Nvidia_Gxf", "3.1.0", "LICENSE");

GXF_EXT_FACTORY_ADD(0xd073a92344ba4b81, 0xbd0f18f4996048e9, nvidia::cvcore::tensor_ops::CameraModel,
nvidia::gxf::Component,
"Construct Camera distortion model / Camera intrinsic compatible with CVCORE");
GXF_EXT_FACTORY_ADD(
0xd073a92344ba4b82, 0xbd0f18f4996048e8, nvidia::cvcore::tensor_ops::CameraModel,
nvidia::gxf::Component,
"Construct Camera distortion model / Camera intrinsic compatible with CVCORE");

GXF_EXT_FACTORY_ADD(0x6c9419223e4b4c2b, 0x899a4d65279c6507, nvidia::cvcore::tensor_ops::Frame3D, nvidia::gxf::Component,
"Construct Camera extrinsic compatible with CVCORE");
GXF_EXT_FACTORY_ADD(
0x6c9419223e4b4c2c, 0x899a4d65279c6508, nvidia::cvcore::tensor_ops::Frame3D,
nvidia::gxf::Component,
"Construct Camera extrinsic compatible with CVCORE");

GXF_EXT_FACTORY_ADD(0xd94385e5b35b4634, 0x9adb0d214a3865f6, nvidia::cvcore::tensor_ops::TensorStream,
nvidia::gxf::Component, "Wrapper of CVCORE ITensorOperatorStream/ITensorOperatorContext");
GXF_EXT_FACTORY_ADD(
0xd94385e5b35b4635, 0x9adb0d214a3865f7, nvidia::cvcore::tensor_ops::TensorStream,
nvidia::gxf::Component, "Wrapper of CVCORE ITensorOperatorStream/ITensorOperatorContext");

GXF_EXT_FACTORY_ADD(0xd0c4ddad486a4a91, 0xb69c8a5304b205ef, nvidia::cvcore::tensor_ops::ImageAdapter,
nvidia::gxf::Component, "Utility component for conversion between message and cvcore image type");
GXF_EXT_FACTORY_ADD(
0xd0c4ddad486a4a92, 0xb69c8a5304b205ea, nvidia::cvcore::tensor_ops::ImageAdapter,
nvidia::gxf::Component, "Utility component for conversion between message and cvcore image type");

GXF_EXT_FACTORY_ADD(0xadebc792bd0b4a56, 0x99c1405fd2ea0727, nvidia::cvcore::tensor_ops::StreamUndistort,
nvidia::gxf::Codelet, "Codelet for stream image undistortion in tensor_ops");
GXF_EXT_FACTORY_ADD(
0xadebc792bd0b4a57, 0x99c1405fd2ea0728, nvidia::cvcore::tensor_ops::StreamUndistort,
nvidia::gxf::Codelet, "Codelet for stream image undistortion in tensor_ops");

GXF_EXT_FACTORY_ADD(0xa58141ac7eca4ea5, 0x9b545446fe379a11, nvidia::cvcore::tensor_ops::Resize, nvidia::gxf::Codelet,
"Codelet for image resizing in tensor_ops");
GXF_EXT_FACTORY_ADD(
0xa58141ac7eca4ea6, 0x9b545446fe379a12, nvidia::cvcore::tensor_ops::Resize, nvidia::gxf::Codelet,
"Codelet for image resizing in tensor_ops");

GXF_EXT_FACTORY_ADD(0xeb8b5f5b36d44b48, 0x81f959fd28e6f677, nvidia::cvcore::tensor_ops::StreamResize,
nvidia::gxf::Codelet, "Codelet for stream image resizing in tensor_ops");
GXF_EXT_FACTORY_ADD(
0xeb8b5f5b36d44b49, 0x81f959fd28e6f678, nvidia::cvcore::tensor_ops::StreamResize,
nvidia::gxf::Codelet, "Codelet for stream image resizing in tensor_ops");

GXF_EXT_FACTORY_ADD(0x4a7ff422de3841bc, 0x9e743ac10d9294b6, nvidia::cvcore::tensor_ops::CropAndResize,
nvidia::gxf::Codelet, "Codelet for crop and resizing operation in tensor_ops");
GXF_EXT_FACTORY_ADD(
0x4a7ff422de3841bd, 0x9e743ac10d9294b7, nvidia::cvcore::tensor_ops::CropAndResize,
nvidia::gxf::Codelet, "Codelet for crop and resizing operation in tensor_ops");

GXF_EXT_FACTORY_ADD(0x7018f0b9034c462b, 0xa9fbaf7ee012974f, nvidia::cvcore::tensor_ops::Normalize, nvidia::gxf::Codelet,
"Codelet for image normalization in tensor_ops");
GXF_EXT_FACTORY_ADD(
0x7018f0b9034c462c, 0xa9fbaf7ee012974a, nvidia::cvcore::tensor_ops::Normalize,
nvidia::gxf::Codelet,
"Codelet for image normalization in tensor_ops");

GXF_EXT_FACTORY_ADD(0x269d4237f3c3479d, 0xbcca9ecc44c71a70, nvidia::cvcore::tensor_ops::InterleavedToPlanar,
nvidia::gxf::Codelet, "Codelet for convert interleaved image to planar image in tensor_ops");
GXF_EXT_FACTORY_ADD(
0x269d4237f3c3479e, 0xbcca9ecc44c71a71, nvidia::cvcore::tensor_ops::InterleavedToPlanar,
nvidia::gxf::Codelet, "Codelet for convert interleaved image to planar image in tensor_ops");

GXF_EXT_FACTORY_ADD(0xfc4d7b4d8fcc4daa, 0xa286056e0fcafa78, nvidia::cvcore::tensor_ops::ConvertColorFormat,
nvidia::gxf::Codelet, "Codelet for image color conversion in tensor_ops");
GXF_EXT_FACTORY_ADD(
0xfc4d7b4d8fcc4dab, 0xa286056e0fcafa79, nvidia::cvcore::tensor_ops::ConvertColorFormat,
nvidia::gxf::Codelet, "Codelet for image color conversion in tensor_ops");

GXF_EXT_FACTORY_ADD(0x5ab4a4d8f7a34552, 0xa90be52660b076fd, nvidia::cvcore::tensor_ops::StreamConvertColorFormat,
nvidia::gxf::Codelet, "Codelet for stream image color conversion in tensor_ops");
GXF_EXT_FACTORY_ADD(
0x5ab4a4d8f7a34553, 0xa90be52660b076fe, nvidia::cvcore::tensor_ops::StreamConvertColorFormat,
nvidia::gxf::Codelet, "Codelet for stream image color conversion in tensor_ops");

GXF_EXT_FACTORY_ADD(0x26789b7d5a8d4e84, 0x86b845ec5f4cd12a, nvidia::cvcore::tensor_ops::Reshape, nvidia::gxf::Codelet,
"Codelet for image reshape in tensor_ops");
GXF_EXT_FACTORY_ADD(
0x26789b7d5a8d4e85, 0x86b845ec5f4cd12b, nvidia::cvcore::tensor_ops::Reshape, nvidia::gxf::Codelet,
"Codelet for image reshape in tensor_ops");
GXF_EXT_FACTORY_END()
2 changes: 1 addition & 1 deletion isaac_ros_bi3d/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>isaac_ros_bi3d</name>
<version>0.30.0</version>
<version>0.31.0</version>
<description>Bi3D inference network for ROS</description>

<maintainer email="[email protected]">Hemal Shah</maintainer>
Expand Down
2 changes: 1 addition & 1 deletion isaac_ros_bi3d/src/bi3d_node.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ constexpr char INPUT_RIGHT_IMAGE_TOPIC_NAME[] = "right_image_bi3d";

constexpr char INPUT_DISPARITY_COMPONENT_KEY[] = "disparity_roundrobin/data_receiver";

constexpr char OUTPUT_BI3D_KEY[] = "bi3d_output_vault/vault";
constexpr char OUTPUT_BI3D_KEY[] = "bi3d_output_sink/sink";
constexpr char OUTPUT_BI3D_DEFAULT_TENSOR_FORMAT[] = "nitros_disparity_image_32FC1";
constexpr char OUTPUT_BI3D_TOPIC_NAME[] = "bi3d_node/bi3d_output";

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>isaac_ros_nitros_bi3d_inference_param_array_type</name>
<version>0.30.0</version>
<version>0.31.0</version>
<description>Isaac ROS Nitros Bi3D Inferece Param Array Type</description>

<maintainer email="[email protected]">Hemal Shah</maintainer>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
%YAML 1.2
# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
# Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -33,7 +33,7 @@ components:
receiver: input
min_size: 1
---
name: vault
name: sink
components:
- name: input
type: nvidia::gxf::DoubleBufferReceiver
Expand All @@ -43,23 +43,22 @@ components:
parameters:
receiver: input
min_size: 1
- name: vault
type: nvidia::gxf::Vault
- name: sink
type: nvidia::isaac_ros::MessageRelay
parameters:
source: input
max_waiting_count: 1
drop_waiting: false
---
components:
- type: nvidia::gxf::Connection
parameters:
source: forward/output
target: vault/input
target: sink/input
---
components:
- type: nvidia::gxf::GreedyScheduler
parameters:
clock: clock
stop_on_deadlock: false
check_recession_period_us: 100
- name: clock
type: nvidia::gxf::RealtimeClock
Loading

0 comments on commit fd902ff

Please sign in to comment.