From abef2178f66da48cfcdc1180c49b7083f6854be5 Mon Sep 17 00:00:00 2001 From: Jacob Perron Date: Thu, 9 Jun 2022 10:12:36 -0700 Subject: [PATCH 1/8] Add draft for REP about service intropsection * Add content for motivation, specification, and rationale. * Feature progress is left as TODO as we've just been prototyping * Add "Other" section will examples of tooling leveraging the proposed feature Co-authored-by: Aditya Pande Co-authored-by: Brian Chen Co-authored-by: Deepanshu Bansal Signed-off-by: Jacob Perron --- rep-2012.rst | 466 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 466 insertions(+) create mode 100644 rep-2012.rst diff --git a/rep-2012.rst b/rep-2012.rst new file mode 100644 index 000000000..985f8bff9 --- /dev/null +++ b/rep-2012.rst @@ -0,0 +1,466 @@ +REP: 2012 +Title: Service Introspection +Author: Aditya Pande , Brian Chen , Deepanshu Bansal , Jacob Perron +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 07-Jun-2022 +Post-History: + +Abstract +======== + +This REP proposes a feature to introspect ROS services during runtime. +The feature allows users to remotely monitor service requests and responses. + + +Terminology +=========== + +:Service: + A ROS service. + Accepts requests from clients and sends responses. +:Client: + A ROS service client. + Sends requests to services and receives responses. +:Request: + A ROS service request. +:Response: + A ROS service response. + + +Motivation +========== + +The primary motivation for this proposal is to make it easier for users to externally validate that services are operating as expected. +Drawing an analogy to ROS topics, there exists tools and libraries for "echoing" and recording messages sent over a topic, and we would like to see the same kind of capabilities for services. +Specifically, we want the capability to introspect requests and responses that are sent to and from services and clients. +Being able to remotely monitor services allows users to more effectively troubleshoot issues in a ROS system. +For example, we could verify requests are being received by a service by employing a command-line tool during runtime. +Or, we could post-process recorded requests and responses to validate their content. + +There are additional features that could leverage this proposal, such as: + +- Playback of recorded services (for example, from a rosbag [1]_) +- Introspection of ROS actions, which are built on services +- Validation of a live ROS system by referencing a recording from a previous session + +Though this proposal focuses on the core feature of introspecting requests and responses, we want to keep the design flexible so additional features like those listed above can be implemented in the future. + + +Specification +============= + +Publishing Service Events +------------------------- + +Whenever a request or response is sent or received, a *service event* message will be published to a topic. +Services are responsible for publishing a message when they receive a request and when they send a response. +Likewise, clients are responsible for publishing a message when they send a request and when they receive a response. +Therefore, we have a total of four possible events: + +:Request Sent: + Emitted from a client after sending a request to a service. +:Request Received: + Emitted from a service after receiving a request from a client. +:Response Sent: + Emitted from a service after sending a response to a client. +:Response Received: + Emitted from a client after receiving a response from a service. + +Request event messages shall be published to the hidden topic ``/SERVICE_NAME/_request_event``, where ``SERVICE_NAME`` is the fully-qualified name of the service. +Similarly, response event messages shall be published to the hidden topic ``/SERVICE_NAME/_response_event``. +Note that this implies that services must have unique names. + +By publishing service event messages to predetermined topics, tools and libraries are able to subscribe to these topics to inspect the flow of data between services. + +Service Event Definitions +------------------------- + +For each service definition, ``my/srv/Foo``, we define two new ROS message types using the ROS IDL specification [2]_: + +:my/msg/Foo_RequestEvent: + This type communicates a request sent or request received event. + + .. code-block:: + + # Indicates this request event was emitted from a client + uint8 CLIENT_SENT = 0 + + # Indicates this request event was emitted from a service + uint8 SERVICE_RECEIVED = 1 + + # Whether this is a CLIENT_SENT or SERVICE_RECEIVED event. + uint8 request_type + + # Timestamp for when the event occurred (sent or received time) + builtin_interfaces/msg/Time stamp + + # Unique identifier for the client that sent the request + # Note, this is only unique for the current session + unique_identifier_msgs/msg/UUID client_id + + # Sequence number for the request + # Combined with the client ID, this creates a unique ID for the request + int64 sequence_number + + # The actual request content sent or received + # Setting this field is optional + my/srv/Foo_Request request + +:my/msg/Foo_ResponseEvent: + This type communicates a response sent or response received event. + + .. code-block:: + + # Indicates this response event was emitted from a service + uint8 SERVICE_SENT = 0 + + # Indicates this response event was emitted from a client + uint8 CLIENT_RECEIVED = 1 + + # Whether this is a SERVICE_SENT or CLIENT_RECEIVED event. + uint8 response_type + + # Timestamp for when the event occurred (sent or received time) + builtin_interfaces/msg/Time stamp + + # Unique identifier for the client that sent the request associated with this response + # Note, this is only unique for the current session + unique_identifier_msgs/msg/UUID client_id + + # Sequence number for the request associated with this response + # Combined with the client ID, this creates a unique ID for the request + int64 sequence_number + + # The actual response content sent or received + # Setting this field is optional + my/srv/Foo_Response response + +Note, we put the service event definitions into the ``msg`` namespace, as this is expected for message interface types in many libraries and tools in the ROS ecosystem. + +The service event definitions are generated as part of the ``rosidl`` pipeline [3]_. + +Timestamp +^^^^^^^^^ +Timestamps represent the time at which the event occurred. +I.e. they are set to the time directly after a request or response is sent or received. + +Timestamps shall respect ROS time [4]_. +This means by default they will be set with wall-time. +If simulation time is enabled by the node implementing the service or client, then timestamps will get their time from the ``/clock`` topic. + +Client ID and sequence number +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Both the client ID and sequence number are provided by the ROS middleware [5]_. +They can be accessed from ``rcl`` [6]_ when taking a request or response for a service or client respectively. + +Example +^^^^^^^ + +For example, consider a service ``example_interfaces/srv/AddTwoInts`` defined as follows: + +.. code-block:: + + int64 a + int64 b + --- + int64 sum + +The following service event message definitions are generated when building the ``example_interfaces`` package (comments elided for brevity): + +:example_interfaces/msg/AddTwoInts_RequestEvent: + +.. code-block:: + + uint8 CLIENT_SENT = 0 + uint8 SERVICE_RECEIVED = 1 + + uint8 request_type + + builtin_interfaces/msg/Time stamp + + unique_identifier_msgs/msg/UUID client_id + + int64 sequence_number + + example_interfaces/srv/AddTwoInts_Request request + +:example_interfaces/msg/AddTwoInts_ResponseEvent: + +.. code-block:: + + uint8 SERVICE_SENT = 0 + uint8 CLIENT_RECEIVED = 1 + + uint8 response_type + + builtin_interfaces/msg/Time stamp + + unique_identifier_msgs/msg/UUID client_id + + int64 sequence_number + + example_interfaces/srv/AddTwoInts_Response response + +The definition for ``example_interfaces/srv/AddTwoInts_Request`` is, + +.. code-block:: + + int64 a + int64 b + +And the definitions for ``example_interfaces/srv/AddTwoInts_Reponse`` is, + +.. code-block:: + + int64 sum + +Configuration +------------- + +Configuration of service introspection features will be done through ROS parameters. +Each node may offer the following four Boolean parameters: + +:publish_client_events: + If true, then client "request sent" and "response received" events will be published for all clients created by this node. +:publish_service_events: + If true, then service "request received" and "response sent" events will be published for all services created by this node. +:client_event_content: + If true, then client event messages will include the request and response content. + Only applies if ``publish_client_events`` is ``true``. +:service_event_content: + If true, then service event messages will include the request and response content. + Only applies if ``publish_service_events`` is ``true``. + +By default, all parameters shall be ``false`` so users do not pay for a feature they do not plan to use. +Furthermore, node authors may opt-in by default or disable the service introspection feature altogether as they see fit. + +As an example, users can enable service introspection for all services by providing the following parameters YAML file to nodes [7]_: + +.. code-block:: yaml + + /**: + ros__parameters: + publish_service_events: true + # Also include request and response content in event messages + service_event_content: true + + +Since it is possible to set parameters with a parameter service [8]_, one or more service events *may* be published when setting a service introspection parameter. +The behavior depends on the order of operations and is defined by the implementation. +For example, if ``publish_service_events`` was previously set to ``true``, then any service call to change one of the parameters defined above may cause a service request event and/or a response event to be published for the parameter service. + +Quality of Service +------------------ + +The service event topics proposed in this REP shall use the default quality of service settings [9]_. + +Security +-------- + +Enabling service introspection creates more attack surface for an existing ROS system by adding 2*N more topics (where N is the number of services with the feature enabled). +These topics are vulnerable to undesired actors listening in on service communication or even interfering with parts of the system that may be relying on service events. + +Luckily, we can leverage the existing security feature for topics in ROS 2 (see SROS 2 [10]_). +Any existing tooling for aiding users in setting up ROS security should consider the new service event topics introduced by this REP (e.g. NoDL [11]_). + + +Rationale +========= + + +Using ROS Parameters for configuration +-------------------------------------- + +ROS parameters are the canonical way to configure a node at runtime, and so it seems to be a natural choice for configuring the service introspection feature. +We can benefit from existing tools for interacting with parameters such as the parameter services API in ``rclcpp`` or ``rclpy`` and ROS launch files [12]_. +Parameters also offer the convenience of being reconfigurable during runtime, so by extension service introspection may be toggled on and off while the node is running. + +Environment variables were considered as an alternative method for configuring service introspection, however they are not reconfigurable during runtime and do not naturally map to nodes the same way parameters do. + +Configuration options +--------------------- + +It would be nice to define a single enumeration type to reduce the number of configuration points, however since ROS parameters do not support enumeration types multiple Boolean parameters are defined instead. + +The number of parameters was chosen as a compromise between flexibility and complexity. +At one extreme, service introspection could be configured per-service with nodes offering S * 4 parameters, where S is the number of services and each has 4 parameters as described in `Configuration `_. +This runs the risk of overwhelming users with the sheer number of parameters offered by a node. +At the other extreme, there could be a single option to enable service introspection for all services in a ROS system, which is not very flexible. + +It seems likely that there will by a large number of users that will want to introspect (or record) all services in system, +therefore it should be easy to turn on the feature for all services. +However, since it is difficult to predict how the feature will ultimately be used, we want to provide some flexibility. + +Boolean parameters are chosen as an easy way to toggle the feature on and off per node (because ROS parameters are stored per node). +Recognizing that both client events and service events contain redundant information (besides timestamps), we provide a separate parameter to toggle the feature for clients. +Similarly, extra overhead may arise in cases where service requests or responses are *very* large. +So, additional parameters are offered to avoid sending content for client or service events. +This way users who are concerned with extra overhead incurred by enabling service introspection have mitigation options. + + +Parameter Services +------------------ + +There already exists a set of default services for interacting with ROS parameters [8]_. +There is also the ``/parameter_events`` topic where changes to ROS parameters for all nodes are published. +We considered leveraging this REP to implement (or replace) the existing ``/parameter_events`` topic, however parameter events may be triggered by local changes in a node (and not necessarily through a parameter service), so we cannot capture all possible parameter events from service events. + + +Only supporting one service per name +------------------------------------ + +It is technically possible to create more than one service with the same name (though not recommended). +However, this is generally not recommended and may be forbidden in the future. +Therefore, as far as this REP is concerned, creating multiple services with the same name is undefined behavior. + + +Separate request and response events instead of single service event +-------------------------------------------------------------------- + +This REP defines two event types for requests and responses. +Publishing separate events from client and services makes it possible to detect the situations such as: + +* a request was sent by a client, but not received by a service +* a request was received by a service, but a response was not sent + +Alternatively, a single event type could have been defined containing both the request and response. +While this would be convenient for tools to match requests and responses, it would result in duplicate or unused message content. + +A second alternative is to define unique request and response event types for clients and services (for a total of four event types and four topics per service). +However, it's not clear that there is much benefit in the additional types considering the definition of a client request type and service request type would be identical (the same applying to response types). + + +Backwards Compatibility +======================= + +The addition of service introspection should not impact existing logic. +As an opt-in feature, users should not incur additional overhead by default. + +Feature Progress +================ + +TODO: development of a prototype is underway. + + +Other +===== + + +Tooling +------- + +``ros2 service`` +^^^^^^^^^^^^^^^^ +The existing ``ros2 service`` tool can be extended using an ``echo`` keyword to monitor service events. +Internally, it would subscribe to the `hidden topics `_ and echo them. +The existing command line parameters for topics can be extended to be used with this ``echo`` verb, along with new +arguments on to filter message content and analyze delays. + +Building off the example with AddTwoInts discussed earlier, an example ``ros2 service echo`` call may look like the following: + +.. code-block:: + + $ ros2 service echo /add_two_ints + ----------------------- + request_type: REQUEST_SENT + stamp: 1.00 + client_id: 1234 + sequence_number: 1 + request: + a: 1 + b: 2 + ----------------------- + request_type: REQUEST_RECEIVED + stamp: 1.10 + client_id: 1235 + sequence_number: 1 + request: + a: 1 + b: 2 + ----------------------- + request_type: RESPONSE_SENT + stamp: 1.20 + client_id: 1235 + sequence_number: 2 + request: + sum: 3 + ----------------------- + request_type: RESPONSE_RECEIVED + stamp: 1.30 + client_id: 1234 + sequence_number: 2 + request: + sum: 3 + ----------------------- + + +``ros2 bag`` +^^^^^^^^^^^^ + +``rosbag2`` integration for service introspection will come more or less for free since the request/response events are simply being echoed through ROS 2 publishers. +Syntactic sugar may be included to enable service introspection and record, e.g. ``ros2 bag record --enable-services``. + +Replaying service and client events +----------------------------------- + +The design should support implementation of a tool for "replaying" service and client events. +For example, tooling may be developed to take the recorded event stream and replay requests and responses back into the ROS network. + + +References +========== + +.. [1] rosbag2 + (https://github.com/ros2/rosbag2) + +.. [2] ROS 2 interfaces + (https://docs.ros.org/en/rolling/Concepts/About-ROS-Interfaces.html) + +.. [3] ROS IDL pipeline + (https://github.com/ros2/rosidl) + +.. [4] ROS Time + (https://design.ros2.org/articles/clock_and_time.html) + +.. [5] RMW + (https://github.com/ros2/rmw) + +.. [6] rcl + (https://github.com/ros2/rcl) + +.. [7] YAML parameter file wildcard + (https://docs.ros.org/en/rolling/Tutorials/Launch/Using-ROS2-Launch-For-Large-Projects.html#using-wildcards-in-yaml-files) + +.. [8] ROS Parameters + (https://docs.ros.org/en/foxy/Concepts/About-ROS-2-Parameters.html) + +.. [9] Quality of Service Settings + (https://docs.ros.org/en/rolling/Concepts/About-Quality-of-Service-Settings.html) + +.. [10] SROS 2 + (https://github.com/ros2/sros2) + +.. [11] NoDL + (https://github.com/ubuntu-robotics/nodl) + +.. [12] Launch ROS + (https://github.com/ros2/launch_ros) + + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 5816be4c08ab8a6c1bf17d0f1fe6ca820ecba0c7 Mon Sep 17 00:00:00 2001 From: Jacob Perron Date: Tue, 19 Jul 2022 09:51:30 -0700 Subject: [PATCH 2/8] Use one message definition per service * Move metadata into common ServiceEventInfo message * Generate one event message per service containing an optional request or response * Minor rewording Signed-off-by: Jacob Perron --- rep-2012.rst | 163 ++++++++++++++++++++------------------------------- 1 file changed, 65 insertions(+), 98 deletions(-) diff --git a/rep-2012.rst b/rep-2012.rst index 985f8bff9..dd7fa8769 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -17,17 +17,16 @@ The feature allows users to remotely monitor service requests and responses. Terminology =========== +:Request: + A ROS service request. +:Response: + A ROS service response. :Service: A ROS service. Accepts requests from clients and sends responses. :Client: A ROS service client. Sends requests to services and receives responses. -:Request: - A ROS service request. -:Response: - A ROS service response. - Motivation ========== @@ -68,81 +67,70 @@ Therefore, we have a total of four possible events: :Response Received: Emitted from a client after receiving a response from a service. -Request event messages shall be published to the hidden topic ``/SERVICE_NAME/_request_event``, where ``SERVICE_NAME`` is the fully-qualified name of the service. -Similarly, response event messages shall be published to the hidden topic ``/SERVICE_NAME/_response_event``. +Event messages shall be published to the hidden topic ``/SERVICE_NAME/_service_event``, where ``SERVICE_NAME`` is the fully-qualified name of the service. +Both services and clients will publish events to the same topic. Note that this implies that services must have unique names. By publishing service event messages to predetermined topics, tools and libraries are able to subscribe to these topics to inspect the flow of data between services. -Service Event Definitions -------------------------- - -For each service definition, ``my/srv/Foo``, we define two new ROS message types using the ROS IDL specification [2]_: +Service Event Definition +------------------------ -:my/msg/Foo_RequestEvent: - This type communicates a request sent or request received event. +For each service definition, ``my/srv/Foo.srv``, we define a new ROS message type, ``my/srv/Foo_Event.msg``, with the ROS IDL specification [2]_: .. code-block:: - # Indicates this request event was emitted from a client - uint8 CLIENT_SENT = 0 - - # Indicates this request event was emitted from a service - uint8 SERVICE_RECEIVED = 1 - - # Whether this is a CLIENT_SENT or SERVICE_RECEIVED event. - uint8 request_type - - # Timestamp for when the event occurred (sent or received time) - builtin_interfaces/msg/Time stamp - - # Unique identifier for the client that sent the request - # Note, this is only unique for the current session - unique_identifier_msgs/msg/UUID client_id + # Event info + # Contains event type, timestamp, and request ID + rcl_interfaces/msg/ServiceEventInfo info - # Sequence number for the request - # Combined with the client ID, this creates a unique ID for the request - int64 sequence_number + # The actual request content sent or received + # This field is only set if the event type is REQUEST_SENT or REQUEST_RECEIVED, + # and the introspection feauture is configured to include payload data. + my/srv/Foo_Request[<=1] request - # The actual request content sent or received - # Setting this field is optional - my/srv/Foo_Request request + # The actual response content sent or received + # This field is only set if the event type is RESPONSE_SENT or RESPONSE_RECEIVED, + # and the introspection feauture is configured to include payload data. + my/srv/Foo_Response[<=1] response -:my/msg/Foo_ResponseEvent: - This type communicates a response sent or response received event. +where ``rcl_interfaces/msg/ServiceEventInfo.msg`` is defined as, .. code-block:: - # Indicates this response event was emitted from a service - uint8 SERVICE_SENT = 0 + # Indicates this is a request sent event emitted from a client + uint8 REQUEST_SENT = 0 - # Indicates this response event was emitted from a client - uint8 CLIENT_RECEIVED = 1 + # Indicates this is a request received event emitted from a service + uint8 REQUEST_RECEIVED = 1 - # Whether this is a SERVICE_SENT or CLIENT_RECEIVED event. - uint8 response_type + # Indicates this is a response sent event emitted from a service + uint8 RESPONSE_SENT = 2 - # Timestamp for when the event occurred (sent or received time) - builtin_interfaces/msg/Time stamp + # Indicates this is a response received event emitted from a client + uint8 RESPONSE_RECEIVED = 3 - # Unique identifier for the client that sent the request associated with this response - # Note, this is only unique for the current session - unique_identifier_msgs/msg/UUID client_id + # The kind of event this message represents + uint8 event_type - # Sequence number for the request associated with this response - # Combined with the client ID, this creates a unique ID for the request - int64 sequence_number + # Timestamp for when the event occurred (sent or received time) + builtin_interfaces/msg/Time stamp - # The actual response content sent or received - # Setting this field is optional - my/srv/Foo_Response response + # Unique identifier for the client that sent the service request + # Note, this is only unique for the current session + unique_identifier_msgs/msg/UUID client_id -Note, we put the service event definitions into the ``msg`` namespace, as this is expected for message interface types in many libraries and tools in the ROS ecosystem. + # Sequence number for the request + # Combined with the client ID, this creates a unique ID for the service transaction + int64 sequence_number -The service event definitions are generated as part of the ``rosidl`` pipeline [3]_. +The reserved underscore character is used in the generated type name to avoid potential collisions with user-defined types. + +Service event definitions are generated as part of the ``rosidl`` pipeline [3]_. Timestamp ^^^^^^^^^ + Timestamps represent the time at which the event occurred. I.e. they are set to the time directly after a request or response is sent or received. @@ -152,6 +140,7 @@ If simulation time is enabled by the node implementing the service or client, th Client ID and sequence number ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Both the client ID and sequence number are provided by the ROS middleware [5]_. They can be accessed from ``rcl`` [6]_ when taking a request or response for a service or client respectively. @@ -167,41 +156,17 @@ For example, consider a service ``example_interfaces/srv/AddTwoInts`` defined as --- int64 sum -The following service event message definitions are generated when building the ``example_interfaces`` package (comments elided for brevity): - -:example_interfaces/msg/AddTwoInts_RequestEvent: - -.. code-block:: - - uint8 CLIENT_SENT = 0 - uint8 SERVICE_RECEIVED = 1 - - uint8 request_type - - builtin_interfaces/msg/Time stamp - - unique_identifier_msgs/msg/UUID client_id - - int64 sequence_number - - example_interfaces/srv/AddTwoInts_Request request +The following service event message definition is generated when building the ``example_interfaces`` package (comments elided for brevity): -:example_interfaces/msg/AddTwoInts_ResponseEvent: +:example_interfaces/srv/AddTwoInts_Event: .. code-block:: - uint8 SERVICE_SENT = 0 - uint8 CLIENT_RECEIVED = 1 - - uint8 response_type - - builtin_interfaces/msg/Time stamp + rcl_interfaces/msg/ServiceEventInfo info - unique_identifier_msgs/msg/UUID client_id + example_interfaces/srv/AddTwoInts_Request[<=1] request - int64 sequence_number - - example_interfaces/srv/AddTwoInts_Response response + example_interfaces/srv/AddTwoInts_Response[<=1] response The definition for ``example_interfaces/srv/AddTwoInts_Request`` is, @@ -223,29 +188,31 @@ Configuration of service introspection features will be done through ROS paramet Each node may offer the following four Boolean parameters: :publish_client_events: - If true, then client "request sent" and "response received" events will be published for all clients created by this node. + If ``true``, then client "request sent" and "response received" events will be published for all clients created by this node. + The default value is ``false``. :publish_service_events: - If true, then service "request received" and "response sent" events will be published for all services created by this node. -:client_event_content: - If true, then client event messages will include the request and response content. + If ``true``, then service "request received" and "response sent" events will be published for all services created by this node. + The default value is ``false``. +:client_event_payload: + If ``true``, then client event messages will have the ``request`` and ``response`` members set. Only applies if ``publish_client_events`` is ``true``. -:service_event_content: - If true, then service event messages will include the request and response content. + The default value is ``true``. +:service_event_payload: + If ``true``, then service event messages will have the ``request`` and ``response`` members set. Only applies if ``publish_service_events`` is ``true``. + The default value is ``true``. -By default, all parameters shall be ``false`` so users do not pay for a feature they do not plan to use. +By default, the event publishing feautre is off so users do not pay for a feature they do not plan to use. Furthermore, node authors may opt-in by default or disable the service introspection feature altogether as they see fit. -As an example, users can enable service introspection for all services by providing the following parameters YAML file to nodes [7]_: +As an example, service introspection can be enabled for all services and clients by providing the following parameters YAML file to ROS nodes [7]_: .. code-block:: yaml /**: ros__parameters: publish_service_events: true - # Also include request and response content in event messages - service_event_content: true - + publish_client_events: true Since it is possible to set parameters with a parameter service [8]_, one or more service events *may* be published when setting a service introspection parameter. The behavior depends on the order of operations and is defined by the implementation. @@ -262,7 +229,7 @@ Security Enabling service introspection creates more attack surface for an existing ROS system by adding 2*N more topics (where N is the number of services with the feature enabled). These topics are vulnerable to undesired actors listening in on service communication or even interfering with parts of the system that may be relying on service events. -Luckily, we can leverage the existing security feature for topics in ROS 2 (see SROS 2 [10]_). +Luckily, we can leverage the existing security feature for topics and services in ROS 2 (see SROS 2 [10]_). Any existing tooling for aiding users in setting up ROS security should consider the new service event topics introduced by this REP (e.g. NoDL [11]_). @@ -319,13 +286,13 @@ Therefore, as far as this REP is concerned, creating multiple services with the Separate request and response events instead of single service event -------------------------------------------------------------------- -This REP defines two event types for requests and responses. +This REP defines four event types for requests and responses. Publishing separate events from client and services makes it possible to detect the situations such as: * a request was sent by a client, but not received by a service * a request was received by a service, but a response was not sent -Alternatively, a single event type could have been defined containing both the request and response. +Alternatively, a single event could have been defined containing both the request and response. While this would be convenient for tools to match requests and responses, it would result in duplicate or unused message content. A second alternative is to define unique request and response event types for clients and services (for a total of four event types and four topics per service). From ffb13c80ae4c23e7f86e7fe5a4a971ab38d7d9d6 Mon Sep 17 00:00:00 2001 From: Jacob Perron Date: Mon, 25 Jul 2022 14:03:11 -0700 Subject: [PATCH 3/8] Replace SROS 2 reference with link to paper Signed-off-by: Jacob Perron --- rep-2012.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-2012.rst b/rep-2012.rst index dd7fa8769..28625eb7e 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -407,7 +407,7 @@ References (https://docs.ros.org/en/rolling/Concepts/About-Quality-of-Service-Settings.html) .. [10] SROS 2 - (https://github.com/ros2/sros2) + (https://aliasrobotics.com/files/SROS2.pdf) .. [11] NoDL (https://github.com/ubuntu-robotics/nodl) From 9c6a95e8c6f62ad40e2654fd43b530513eac3262 Mon Sep 17 00:00:00 2001 From: Jacob Perron Date: Mon, 25 Jul 2022 14:24:27 -0700 Subject: [PATCH 4/8] Use the term 'server' instead of 'service' Trying to make clear the distinction between 'services' as the larger concept and service 'servers' as one half of a service interaction. Signed-off-by: Jacob Perron --- rep-2012.rst | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/rep-2012.rst b/rep-2012.rst index 28625eb7e..8eda9b9dc 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -21,21 +21,21 @@ Terminology A ROS service request. :Response: A ROS service response. -:Service: - A ROS service. +:Server: + A ROS service server. Accepts requests from clients and sends responses. :Client: A ROS service client. - Sends requests to services and receives responses. + Sends requests to servers and receives responses. Motivation ========== The primary motivation for this proposal is to make it easier for users to externally validate that services are operating as expected. Drawing an analogy to ROS topics, there exists tools and libraries for "echoing" and recording messages sent over a topic, and we would like to see the same kind of capabilities for services. -Specifically, we want the capability to introspect requests and responses that are sent to and from services and clients. +Specifically, we want the capability to introspect requests and responses that are sent to and from service servers and clients. Being able to remotely monitor services allows users to more effectively troubleshoot issues in a ROS system. -For example, we could verify requests are being received by a service by employing a command-line tool during runtime. +For example, we could verify requests are being received by a server by employing a command-line tool during runtime. Or, we could post-process recorded requests and responses to validate their content. There are additional features that could leverage this proposal, such as: @@ -54,21 +54,21 @@ Publishing Service Events ------------------------- Whenever a request or response is sent or received, a *service event* message will be published to a topic. -Services are responsible for publishing a message when they receive a request and when they send a response. +Servers are responsible for publishing a message when they receive a request and when they send a response. Likewise, clients are responsible for publishing a message when they send a request and when they receive a response. Therefore, we have a total of four possible events: :Request Sent: - Emitted from a client after sending a request to a service. + Emitted from a client after sending a request to a server. :Request Received: - Emitted from a service after receiving a request from a client. + Emitted from a server after receiving a request from a client. :Response Sent: - Emitted from a service after sending a response to a client. + Emitted from a server after sending a response to a client. :Response Received: - Emitted from a client after receiving a response from a service. + Emitted from a client after receiving a response from a server. Event messages shall be published to the hidden topic ``/SERVICE_NAME/_service_event``, where ``SERVICE_NAME`` is the fully-qualified name of the service. -Both services and clients will publish events to the same topic. +Both servers and clients will publish events to the same topic. Note that this implies that services must have unique names. By publishing service event messages to predetermined topics, tools and libraries are able to subscribe to these topics to inspect the flow of data between services. @@ -101,10 +101,10 @@ where ``rcl_interfaces/msg/ServiceEventInfo.msg`` is defined as, # Indicates this is a request sent event emitted from a client uint8 REQUEST_SENT = 0 - # Indicates this is a request received event emitted from a service + # Indicates this is a request received event emitted from a server uint8 REQUEST_RECEIVED = 1 - # Indicates this is a response sent event emitted from a service + # Indicates this is a response sent event emitted from a server uint8 RESPONSE_SENT = 2 # Indicates this is a response received event emitted from a client @@ -136,13 +136,13 @@ I.e. they are set to the time directly after a request or response is sent or re Timestamps shall respect ROS time [4]_. This means by default they will be set with wall-time. -If simulation time is enabled by the node implementing the service or client, then timestamps will get their time from the ``/clock`` topic. +If simulation time is enabled by the node implementing the server or client, then timestamps will get their time from the ``/clock`` topic. Client ID and sequence number ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Both the client ID and sequence number are provided by the ROS middleware [5]_. -They can be accessed from ``rcl`` [6]_ when taking a request or response for a service or client respectively. +They can be accessed from ``rcl`` [6]_ when taking a request or response for a service server or client respectively. Example ^^^^^^^ @@ -191,21 +191,21 @@ Each node may offer the following four Boolean parameters: If ``true``, then client "request sent" and "response received" events will be published for all clients created by this node. The default value is ``false``. :publish_service_events: - If ``true``, then service "request received" and "response sent" events will be published for all services created by this node. + If ``true``, then service "request received" and "response sent" events will be published for all service servers created by this node. The default value is ``false``. :client_event_payload: If ``true``, then client event messages will have the ``request`` and ``response`` members set. Only applies if ``publish_client_events`` is ``true``. The default value is ``true``. :service_event_payload: - If ``true``, then service event messages will have the ``request`` and ``response`` members set. + If ``true``, then server event messages will have the ``request`` and ``response`` members set. Only applies if ``publish_service_events`` is ``true``. The default value is ``true``. -By default, the event publishing feautre is off so users do not pay for a feature they do not plan to use. +By default, the event publishing feature is off so users do not pay for a feature they do not plan to use. Furthermore, node authors may opt-in by default or disable the service introspection feature altogether as they see fit. -As an example, service introspection can be enabled for all services and clients by providing the following parameters YAML file to ROS nodes [7]_: +As an example, service introspection can be enabled for all servers and clients by providing the following parameters YAML file to ROS nodes [7]_: .. code-block:: yaml @@ -261,9 +261,9 @@ therefore it should be easy to turn on the feature for all services. However, since it is difficult to predict how the feature will ultimately be used, we want to provide some flexibility. Boolean parameters are chosen as an easy way to toggle the feature on and off per node (because ROS parameters are stored per node). -Recognizing that both client events and service events contain redundant information (besides timestamps), we provide a separate parameter to toggle the feature for clients. +Recognizing that both client events and server events contain redundant information (besides timestamps), we provide a separate parameter to toggle the feature for clients. Similarly, extra overhead may arise in cases where service requests or responses are *very* large. -So, additional parameters are offered to avoid sending content for client or service events. +So, additional parameters are offered to avoid sending content for client or server events. This way users who are concerned with extra overhead incurred by enabling service introspection have mitigation options. @@ -287,10 +287,10 @@ Separate request and response events instead of single service event -------------------------------------------------------------------- This REP defines four event types for requests and responses. -Publishing separate events from client and services makes it possible to detect the situations such as: +Publishing separate events from client and servers makes it possible to detect situations such as: -* a request was sent by a client, but not received by a service -* a request was received by a service, but a response was not sent +* a request was sent by a client, but not received by a server +* a request was received by a server, but a response was not sent Alternatively, a single event could have been defined containing both the request and response. While this would be convenient for tools to match requests and responses, it would result in duplicate or unused message content. From 81b190b63cf06dfa6f4157f507e60264c0338e1c Mon Sep 17 00:00:00 2001 From: Jacob Perron Date: Tue, 13 Sep 2022 15:44:15 -0700 Subject: [PATCH 5/8] Minor additions + single event type discussion Signed-off-by: Jacob Perron --- rep-2012.rst | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/rep-2012.rst b/rep-2012.rst index 8eda9b9dc..1bea52df9 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -94,7 +94,9 @@ For each service definition, ``my/srv/Foo.srv``, we define a new ROS message typ # and the introspection feauture is configured to include payload data. my/srv/Foo_Response[<=1] response -where ``rcl_interfaces/msg/ServiceEventInfo.msg`` is defined as, +The reserved underscore character is used in the generated type name to avoid potential collisions with user-defined types. + +``rcl_interfaces/msg/ServiceEventInfo.msg`` is defined as, .. code-block:: @@ -124,8 +126,6 @@ where ``rcl_interfaces/msg/ServiceEventInfo.msg`` is defined as, # Combined with the client ID, this creates a unique ID for the service transaction int64 sequence_number -The reserved underscore character is used in the generated type name to avoid potential collisions with user-defined types. - Service event definitions are generated as part of the ``rosidl`` pipeline [3]_. Timestamp @@ -143,6 +143,7 @@ Client ID and sequence number Both the client ID and sequence number are provided by the ROS middleware [5]_. They can be accessed from ``rcl`` [6]_ when taking a request or response for a service server or client respectively. +Together they are used to uniquely identify the service transaction (i.e. they uniquely identify a single request-reponse pair). Example ^^^^^^^ @@ -236,6 +237,7 @@ Any existing tooling for aiding users in setting up ROS security should consider Rationale ========= +The following sections summarize *why* certain design decisions were made and some of the alternatives considered. Using ROS Parameters for configuration -------------------------------------- @@ -298,6 +300,29 @@ While this would be convenient for tools to match requests and responses, it wou A second alternative is to define unique request and response event types for clients and services (for a total of four event types and four topics per service). However, it's not clear that there is much benefit in the additional types considering the definition of a client request type and service request type would be identical (the same applying to response types). +Define a single event type with serialized data +----------------------------------------------- + +Rather than generating event types in ``rosidl``, we considered defining a single type with type-erased data for the request and/or response, for example, + + .. code-block:: + + rcl_interfaces/msg/ServiceEventInfo info + + # The request/response type + # e.g. my/srv/Foo_Request + string idl_type_name + + # Serialized data + byte[] request_or_response + +This has the benefit of avoiding additional code generation for each service type and gives us the option to put all service events on one common topic. + +The downsides include extra overhead from serializing/deserializing the data and tools having to filter out messages based on the service type or name. + +Ultimately, it was decided that having separate event topics per service name would be more useful for tooling and debugging. +For example, it makes it easier to selectively introspection a subset of services by name. + Backwards Compatibility ======================= @@ -325,7 +350,7 @@ Internally, it would subscribe to the `hidden topics Date: Tue, 13 Sep 2022 15:58:15 -0700 Subject: [PATCH 6/8] Add post date and link to discussion Signed-off-by: Jacob Perron --- rep-2012.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/rep-2012.rst b/rep-2012.rst index 1bea52df9..2a9ce73dc 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -5,7 +5,7 @@ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 07-Jun-2022 -Post-History: +Post-History: 20-Jun-2022 Abstract ======== @@ -441,6 +441,12 @@ References (https://github.com/ros2/launch_ros) +Discussions +----------- + +* Review of first draft review on GitHub + (https://github.com/ros-infrastructure/rep/pull/360) + Copyright ========= From 655236c5fd361b0520ef2db7a6fba5d650212a83 Mon Sep 17 00:00:00 2001 From: Jacob Perron Date: Tue, 13 Sep 2022 16:22:11 -0700 Subject: [PATCH 7/8] Update feature progress Signed-off-by: Jacob Perron --- rep-2012.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rep-2012.rst b/rep-2012.rst index 2a9ce73dc..d19e94502 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -333,8 +333,9 @@ As an opt-in feature, users should not incur additional overhead by default. Feature Progress ================ -TODO: development of a prototype is underway. +Most elements of this proposal have been implemented and are currently under review. +Progress on the implementation is being tracked on GitHub at `ros2/ros2#1285 `_. Other ===== From 336d3b66135ca71a82d5e903de1e865e4ceddd2d Mon Sep 17 00:00:00 2001 From: Chris Lalancette Date: Tue, 28 Feb 2023 14:13:57 -0500 Subject: [PATCH 8/8] Revamp of REP-2012 according to the merged implementation. Signed-off-by: Chris Lalancette --- rep-2012.rst | 110 ++++++++++++++++----------------------------------- 1 file changed, 35 insertions(+), 75 deletions(-) diff --git a/rep-2012.rst b/rep-2012.rst index d19e94502..70b781d9e 100644 --- a/rep-2012.rst +++ b/rep-2012.rst @@ -32,11 +32,11 @@ Motivation ========== The primary motivation for this proposal is to make it easier for users to externally validate that services are operating as expected. -Drawing an analogy to ROS topics, there exists tools and libraries for "echoing" and recording messages sent over a topic, and we would like to see the same kind of capabilities for services. -Specifically, we want the capability to introspect requests and responses that are sent to and from service servers and clients. +Drawing an analogy to ROS topics, there exist tools and libraries for "echoing" and recording messages sent over a topic, and this REP proposes the same kind of capabilities for services. +Specifically, the capability to introspect requests and responses that are sent to and from service servers and clients. Being able to remotely monitor services allows users to more effectively troubleshoot issues in a ROS system. -For example, we could verify requests are being received by a server by employing a command-line tool during runtime. -Or, we could post-process recorded requests and responses to validate their content. +For example, users could verify requests are being received by a server by employing a command-line tool during runtime. +Or the user could post-process recorded requests and responses to validate their content. There are additional features that could leverage this proposal, such as: @@ -44,7 +44,7 @@ There are additional features that could leverage this proposal, such as: - Introspection of ROS actions, which are built on services - Validation of a live ROS system by referencing a recording from a previous session -Though this proposal focuses on the core feature of introspecting requests and responses, we want to keep the design flexible so additional features like those listed above can be implemented in the future. +Though this proposal focuses on the core feature of introspecting requests and responses, the design is purposely flexible so additional features like those listed above can be implemented in the future. Specification @@ -76,13 +76,13 @@ By publishing service event messages to predetermined topics, tools and librarie Service Event Definition ------------------------ -For each service definition, ``my/srv/Foo.srv``, we define a new ROS message type, ``my/srv/Foo_Event.msg``, with the ROS IDL specification [2]_: +For each service definition, ``my/srv/Foo.srv``, a new (hidden) ROS message type is defined, ``my/srv/Foo_Event.msg``, with the ROS IDL specification [2]_: .. code-block:: # Event info # Contains event type, timestamp, and request ID - rcl_interfaces/msg/ServiceEventInfo info + service_msgs/msg/ServiceEventInfo info # The actual request content sent or received # This field is only set if the event type is REQUEST_SENT or REQUEST_RECEIVED, @@ -96,7 +96,7 @@ For each service definition, ``my/srv/Foo.srv``, we define a new ROS message typ The reserved underscore character is used in the generated type name to avoid potential collisions with user-defined types. -``rcl_interfaces/msg/ServiceEventInfo.msg`` is defined as, +``service_msgs/msg/ServiceEventInfo.msg`` is defined as, .. code-block:: @@ -112,15 +112,18 @@ The reserved underscore character is used in the generated type name to avoid po # Indicates this is a response received event emitted from a client uint8 RESPONSE_RECEIVED = 3 - # The kind of event this message represents + # The type of event this message represents uint8 event_type # Timestamp for when the event occurred (sent or received time) builtin_interfaces/msg/Time stamp # Unique identifier for the client that sent the service request - # Note, this is only unique for the current session - unique_identifier_msgs/msg/UUID client_id + # Note, this is only unique for the current session. + # The size here has to match the size of rmw_dds_common/msg/Gid, + # but unfortunately we cannot use that message directly due to a + # circular dependency. + char[16] client_gid # Sequence number for the request # Combined with the client ID, this creates a unique ID for the service transaction @@ -132,7 +135,7 @@ Timestamp ^^^^^^^^^ Timestamps represent the time at which the event occurred. -I.e. they are set to the time directly after a request or response is sent or received. +That is, they are set to the time directly after a request or response is sent or received. Timestamps shall respect ROS time [4]_. This means by default they will be set with wall-time. @@ -157,7 +160,7 @@ For example, consider a service ``example_interfaces/srv/AddTwoInts`` defined as --- int64 sum -The following service event message definition is generated when building the ``example_interfaces`` package (comments elided for brevity): +The following (hidden) service event message definition is generated when building the ``example_interfaces`` package: :example_interfaces/srv/AddTwoInts_Event: @@ -185,39 +188,16 @@ And the definitions for ``example_interfaces/srv/AddTwoInts_Reponse`` is, Configuration ------------- -Configuration of service introspection features will be done through ROS parameters. -Each node may offer the following four Boolean parameters: - -:publish_client_events: - If ``true``, then client "request sent" and "response received" events will be published for all clients created by this node. - The default value is ``false``. -:publish_service_events: - If ``true``, then service "request received" and "response sent" events will be published for all service servers created by this node. - The default value is ``false``. -:client_event_payload: - If ``true``, then client event messages will have the ``request`` and ``response`` members set. - Only applies if ``publish_client_events`` is ``true``. - The default value is ``true``. -:service_event_payload: - If ``true``, then server event messages will have the ``request`` and ``response`` members set. - Only applies if ``publish_service_events`` is ``true``. - The default value is ``true``. - -By default, the event publishing feature is off so users do not pay for a feature they do not plan to use. -Furthermore, node authors may opt-in by default or disable the service introspection feature altogether as they see fit. - -As an example, service introspection can be enabled for all servers and clients by providing the following parameters YAML file to ROS nodes [7]_: +Configuration of service introspection features will be done through API calls on a per-client or per-server basis. -.. code-block:: yaml +The API will allow users to: - /**: - ros__parameters: - publish_service_events: true - publish_client_events: true +- Disable introspection completely +- Enable the sending of only metadata +- Enable the sending of metadata and service contents -Since it is possible to set parameters with a parameter service [8]_, one or more service events *may* be published when setting a service introspection parameter. -The behavior depends on the order of operations and is defined by the implementation. -For example, if ``publish_service_events`` was previously set to ``true``, then any service call to change one of the parameters defined above may cause a service request event and/or a response event to be published for the parameter service. +By default, the event publishing feature is off for all clients and all services so users do not pay for a feature they do not plan to use. +Furthermore, node authors may opt-in by default or disable the service introspection feature altogether as they see fit. Quality of Service ------------------ @@ -239,43 +219,23 @@ Rationale The following sections summarize *why* certain design decisions were made and some of the alternatives considered. -Using ROS Parameters for configuration --------------------------------------- - -ROS parameters are the canonical way to configure a node at runtime, and so it seems to be a natural choice for configuring the service introspection feature. -We can benefit from existing tools for interacting with parameters such as the parameter services API in ``rclcpp`` or ``rclpy`` and ROS launch files [12]_. -Parameters also offer the convenience of being reconfigurable during runtime, so by extension service introspection may be toggled on and off while the node is running. - -Environment variables were considered as an alternative method for configuring service introspection, however they are not reconfigurable during runtime and do not naturally map to nodes the same way parameters do. +Configuring service introspection through API calls +--------------------------------------------------- -Configuration options ---------------------- +There are a few reasons to configure introspection through API calls. -It would be nice to define a single enumeration type to reduce the number of configuration points, however since ROS parameters do not support enumeration types multiple Boolean parameters are defined instead. +First, enabling or disabling introspection is fundamentally a per-client or per-service action. +In most scenarios, users probably will not want to enable service introspection on all clients and services at once, as this will greatly increase network traffic. -The number of parameters was chosen as a compromise between flexibility and complexity. -At one extreme, service introspection could be configured per-service with nodes offering S * 4 parameters, where S is the number of services and each has 4 parameters as described in `Configuration `_. -This runs the risk of overwhelming users with the sheer number of parameters offered by a node. -At the other extreme, there could be a single option to enable service introspection for all services in a ROS system, which is not very flexible. - -It seems likely that there will by a large number of users that will want to introspect (or record) all services in system, -therefore it should be easy to turn on the feature for all services. -However, since it is difficult to predict how the feature will ultimately be used, we want to provide some flexibility. - -Boolean parameters are chosen as an easy way to toggle the feature on and off per node (because ROS parameters are stored per node). -Recognizing that both client events and server events contain redundant information (besides timestamps), we provide a separate parameter to toggle the feature for clients. -Similarly, extra overhead may arise in cases where service requests or responses are *very* large. -So, additional parameters are offered to avoid sending content for client or server events. -This way users who are concerned with extra overhead incurred by enabling service introspection have mitigation options. - - -Parameter Services ------------------- +Next, it makes sense to have users of the rcl client/server API not have to specified arguments that will never be used if they don't enable introspection. +By having a separate API for this, only API users concerned with enabling the introspection feature need to provide the feature. -There already exists a set of default services for interacting with ROS parameters [8]_. -There is also the ``/parameter_events`` topic where changes to ROS parameters for all nodes are published. -We considered leveraging this REP to implement (or replace) the existing ``/parameter_events`` topic, however parameter events may be triggered by local changes in a node (and not necessarily through a parameter service), so we cannot capture all possible parameter events from service events. +Finally, by having a separate API call for introspection, the API behavior ends up being completely orthogonal. +That is, users can cycle between having introspection off, metadata only, or contents sent, and the system will do the correct thing. +One downside of using APIs for configuration is that there is no obvious way to configure the introspection feature at runtime. +However, it is easy to hook up the API call to a ROS parameter (say), and control it through that. +If that turns out to be a popular feature, the implementation can be extended to automatically expose this per-service introspection as a parameter. Only supporting one service per name ------------------------------------