Add explicit support for request/response communication pattern (device to application)

[calohmn]

This issue is about a use case where a device sends a request message and expects a corresponding response from a northbound application.

Such a scenario can already be implemented with Hono. The device can send a telemetry/event message with a correlation id property/parameter and with a ttd parameter (signaling its readiness to receive a command message). The northbound application can send a one-way command message as response, including the given correlation id. Upon receiving the command message, the device can correlate the message to its initial telemetry/event message using the correlation id.

However, such a solution is not very intuitive and requires manual correlation of the messages.

Therefore it would be good to explore the feasibility of adding explicit support for such a communication pattern, including correlation support and matching endpoint names.

[calohmn]

Initial idea:

Device side: Sending the request message

• to the MQTT adapter:
  • device chooses a correlation id
  • device subscribes to a topic req_resp/${tenant-id}/${device-id}/resp/${correlation-id}/#
  • device publishes request to req_resp/${tenant-id}/${device-id}/req/${correlation-id}
• to the HTTP adapter:
  • device chooses a correlation id (could also be described as request id)
  • device sends request to /req_resp/${correlation-id} and receives corresponding response

---

Handling request and response messages in terms of AMQP messages could either be done with correlation-id based links (ie. with the overhead of extra link creation):

Shared Protocol Adapter logic

• protocol adapter forwards a request message on a sender link with address req_resp/${tenant-id}, sets req_resp/${tenant-id}/${correlation-id} as reply-to address
• protocol adapter opens a receiver link for the response on req_resp/${tenant-id}/${correlation-id}

Application side

• application opens receiver link on req_resp/${tenant-id}
• application receives a request and sends the response message on a sender link with address req_resp/${tenant-id}/${device-id}/${correlation-id} (according to reply-to of request message)

---

Or, AMQP messages could be handled using a fixed number of reusable links (along with mapping incoming responses to the right protocol adapter instance):

Shared Protocol Adapter logic

• protocol adapter forwards a request message on a sender link with address req_resp/${tenant-id}, sets req_resp/${tenant-id}/${correlation-id} as reply-to address
• protocol adapter opens a receiver link on req_resp/${tenant-id}/${protocolAdapterUUID} (see Reduce number of Command & Control receiver links #1272)
• protocol adapter opens a receiver link on req_resp/${tenant-id} - incoming response messages will be forwarded to the right req_resp/${tenant-id}/${protocolAdapterUUID} address if necessary (see Reduce number of Command & Control receiver links #1272)

Application side

• application opens receiver link on req_resp/${tenant-id}
• application opens sender link on req_resp/${tenant-id}
• application receives a request and sends the response message on the sender link, setting the to property of the message to req_resp/${tenant-id}/${device-id}/${correlation-id}

[wkl3nk]

The need for such a feature even becomes even more urgent if Hono is to be used in combination with Ditto: Currently, without adding a piece of software in the north of Ditto, it is not possible to fetch the current state of the Digital Twin from the Cloud. While one can send a Ditto "retrieve" message via the Hub's telemetry/event channel, there is no way to receive the response. Neither synchronously nor asynchronously. This makes it impossible to implement the desired/reported pattern for configuration values, as it is possible for instance in AWS IoT.

[sophokles73]

This makes it impossible to implement the desired/reported pattern for configuration values, as it is possible for instance in AWS IoT.

FMPOV Ditto would need to be responsible to push the desired state to a device while the device is responsible to report its current state. However, Ditto needs to support the desired/reported states in order for that to work. I do not think that a device should need to pull the desired state from the back end. We deliberately provided means for devices to signal their intention to receive messages from the back end applications. Ditto would simply need to use it as a trigger to push the desired state to the device using a command ...

[avgustinmm]

I'm not sure if the Ditto shall implement push state. This could be tricky.
In order to push state, Ditto shall do that via commands (the only way Hono supports).

• What command will Ditto call? Shall there be a specified push (e.g. Vorto feature with push operation)?
• How shall Ditto know if the device support push command?
• Shall that be bulk command (e.g. push all changes) or one by one (order of apply may matter).
• MOST important: Is the device always able to execute a push? If device could request push info it could apply it whenever it likes. Could try few times if needed. If Ditto pushes info and device isn't ready, error may occur. Shall Ditto retry?

[avgustinmm]

I remember times when Hono was supposed (?) to have C&C in both directions (https://www.slideshare.net/dejanb/scaling-out-eclipse-hono 4th slide).

Also I thing AWS IoT provides restricted support for request/response from device.
Looking the https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-mqtt.html it seems that using publish to /get and reading from /get/accepted the devices could (and I think is expected to) request its state (especially desired) from the Device Shadow service. This in fact is kind of, restricted, access to the Message Broker (Hono analog/competitor) backend apps - Device Shadow.
AWS IoT also provide request/response for jobs - https://docs.aws.amazon.com/iot/latest/developerguide/jobs-api.html.
So AWS IoT provides read/write access to Device Shadow but Hono hides Ditto. It also provides means for querying pending jobs (which Hono tries to do with ttd + job executor that will send commands).

[sophokles73]

Some thoughts regarding the resource structure ...

HTTP adapter:
clients post requests to request[/${tenant}/${deviceId}]
clients receive response (if any) in response body. FMPOV there is no need for a client to supply a correlation ID, this can be done by the adapter.

MQTT adapter:
clients publish to request/[${tenant}]/[${deviceId}]/${correlationId}
clients subscribe to response/[${tenant}]/[${deviceId}]/#

I wonder if we need dedicated link addresses for sending the requests downstream and receiving a response from a downstream application. For sending the request message downstream I see no reason why we cannot simply re-use the telemetry or event link. The message would then have its correlation-id property set to ${tenant}/${deviceId}/${correlationId} and its reply-to property set to e.g. response/${tenant}.
We could also use the standard command/${tenant} link for receiving the response from the application. In that case the application would set the response message's correlation-id property but would not set the reply-to property. This way the protocol adapter could determine that this is not the request message of a req/resp command but instead the response to a request issued by a device.
That way we could prevent adding more and more links which would need to be handled and monitored individually.
WDYT @calohmn ?

[calohmn]

HTTP adapter:
[..]
clients receive response (if any) in response body. FMPOV there is no need for a client to supply a correlation ID, this can be done by the adapter.

Right, the correlation ID can be set by the adapter (if the client hasn't explicitly provided one).

I wonder if we need dedicated link addresses for sending the requests downstream and receiving a response from a downstream application. For sending the request message downstream I see no reason why we cannot simply re-use the telemetry or event link.

I agree that we should try to prevent adding new links to the AMQP messaging network. However, a reason for not re-using the telemetry or event link here would simply be that the resource names are not so straigthforward anymore then. Currently we use event and telemetry as resource names on the southbound as well as the northbound sides. Now for an application to receive device request messages on an event (or telemetry) link looks somewhat inconsistent. The same applies for the application to send the reply on the command link and the device getting it on a response resource.

Rather than introducing request/response resource names only on the device side (and keeping event/telemetry/command on the business application side to prevent the need for new links), I think we could stick to the existing names and only extend the existing resources with extra path parts/parameters.

HTTP adapter:
A request message from a device would be represented by a event/telemetry message with a hono-ttd header/parameter and an additional reply-expected boolean header/parameter. That reply-expected header/parameter would mean that not just any command message (possibly triggered by a previous hono-ttd request) is returned in the HTTP response, but that the actual reply message with matching correlation id is returned.

The downstream AMQP message would get a corresponding reply-expected application property then (along with the correlation-id property) so that the business application can distinguish such request messages. This applies to all kinds of protocol adapters then.

MQTT adapter:
The device can send a request message by publishing on the event/[${tenant}]/[${deviceId}]/${correlationId}?reply-expected=true topic (or maybe better event/[${tenant}]/[${deviceId}]?correlation-id=${correlationId}&reply-expected=true instead).

The device would subscribe to command///reply/# messages (in contrast to command///req/# for uncorrelated commands) and would receive the reply message with a command///reply/${correlationId} topic.
(The fact that we already have a similar topic command///res/${req-id}/${status} with res for response, used for the downstream direction for a command response, is somewhat unfortunate. We might therefore choose cor for correlation instead of reply above, in order not to use similar terms. I'm not sure on that one yet.)

_
The above would be the least invasive implementation to support request/response correlation, while keeping our limited set of resource names for its simplicity. Going for the extra request/response resource names on the device and the application side would possibly make it more intuitive for the user at first glance but maybe would bring its own questions (will a request message have event or telemetry semantics?).

[sophokles73]

Right, the correlation ID can be set by the adapter (if the client hasn't explicitly provided one).

If we don't need the client to specify it, then IMHO we should not allow him to specify one so that there is no chance of using a non-unique correlation ID.

A request message from a device would be represented by a event/telemetry message with a hono-ttd header/parameter and an additional reply-expected boolean header/parameter. That reply-expected header/parameter would mean that not just any command message (possibly triggered by a previous hono-ttd request) is returned in the HTTP response, but that the actual reply message with matching correlation id is returned.

The downstream AMQP message would get a corresponding reply-expected application property then (along with the correlation-id property) so that the business application can distinguish such request messages. This applies to all kinds of protocol adapters then.

This sounds good in general. However, what if the HTTP adapter receives a command for the device instead of a response to a request? Would the HTTP adapter then release the command?

[calohmn]

Right, the correlation ID can be set by the adapter (if the client hasn't explicitly provided one).

If we don't need the client to specify it, then IMHO we should not allow him to specify one so that there is no chance of using a non-unique correlation ID.

Since we would allow the client to specify it for MQTT messages, my intuition would be to also allow it for HTTP, letting it be the responsibility of the device to ensure uniqueness if it chooses to supply a correlation ID. But since for HTTP the usual case should be the adapter-generated ID we can omit that feature in the initial implementation and add it later, if there is a need.

This sounds good in general. However, what if the HTTP adapter receives a command for the device instead of a response to a request? Would the HTTP adapter then release the command?

Yes, provided the device hasn't sent another still-ongoing request without a reply-expected header/parameter. In general, a HTTP device sending a request with a reply-expected header/parameter means that only a command with the matching correlation id will be sent in the response to the device.
An event/telemetry message with reply-expected from the device would associate tenant + deviceId + correlationId with the protocol adapter instance in the Hono device connection service.

Matching incoming commands

That means that an incoming command would first be checked for its correlation id, to find a matching protocol adapter instance and command handler. Only if no matching protocol adapter instance is found, the check would be repeated without the correlation id (just with tenant + device id).
This 2nd check means that a command with an unmatched correlation id would be taken on by a device with a command subscription without the reply-expected header/parameter.
(Alternatively, we could say that a command message with a correlation id would always have to be matched with a subscription with a correlation id/reply-expected header/parameter, and would otherwise be released. But that would break existing scenarios where commands get send with a correlation id already. For another alternative, see last paragraph below.)

For an MQTT device, this means that a command with a matched correlation id (ie. an id the device used in a reply-expected event/telemetry message) will get sent to the device on the command///reply/${correlationId} topic (provided there is a command///reply/# subscription).
A command with an unmatched correlation id will get handled as before by sending it to the device on the command///req/${req-id}/${command} topic (provided there is a command///req/# subscription).

Matching incoming commands - revised

Thinking more about commands with unmatched correlation ids:
The approach above would handle them as "normal" commands (not as responses to requests), provided the same device has opened a command subscription for such "normal" commands as well.
If the same device indeed wants to receive "normal" commands as well as request-response command messages, this might be a problem. A reply-expected HTTP request might have timed out already so that the corresponding request-response command message could potentially end up as a response to a later, "normal" hono-ttd HTTP request (without reply-expected) - which might not be the desired place for the application message to end up.

This means we probably should require the actual request-response command messages to have a boolean is-reply application property (or something like that) so that they can be distinguished from "normal" commands.
With that, we could prevent them from getting sent to a device on channels meant for "normal" commands.
(This would also make routing the command messages easier since then only one lookup in the device connection service is needed instead of 2 (no need to check either deviceId or deviceId+correlationId).)

[avgustinmm]

Hi all,

looking at the proposals, I personally like the first approach Carsten proposed - dedicated links for request/response to backend application.
I think that request/response is exactly the same feature like command/command_response just in the opposite direction (I would even call request/response to backend app -> command to backend). So, it seems that the most natural way to do it is to follow the same approach but on the opposite direction.
For instance for commands (MQTT) we have:

• device receives a command at command/[${tenant-id}]/[${device-id}]/req/${command} => so now the adapter should receive a request at req_resp///req/${subject} to receive requests (yes, why not request subject in order to make things similar?)
• device sends command response to command///res/${req-id}/${status} -> so now the adapter should send request's response to req_res/[${tenant-id}]/[${device-id}]/res/${req-id}/${status} (yes, why not a status of request?)

IMO, this looks very natural from the user's perspective and keep things consistent.

I believe that approach could be also very easy to implement:

• The implementation that now sends commands to device could not, also, send request's responses to device - just to different link.
• The implementation that receives command responses will now, also, receive requests - just on different links.

Now, we will just have couple of instances for of both types just using different links.
The command router could also be reused the same way.

What do you think?

[wkl3nk]

I now longer care.

[sophokles73]

@avgustinmm how would the device correlate the response received from the adapter to the request message that it has sent to the adapter?

[avgustinmm]

@sophokles73
my bad, I've started to convert the request topic from the one way command topic - that's why I ended with "adapter should receive a request at req_resp///req/${subject}". Indeed this would be the case when a device sends an one way command to the backend. Which, IMO, is a valid use case but not really a request/response.

So, let device send request to req_resp///req/${req-id}/${subject} (same way it sends response in the two way command case - command/[${tenant-id}]/[${device-id}]/req/${req-id}/${command}).
Then device will receive response on req_res/[${tenant-id}]/[${device-id}]/res/${req-id}/${status}.
And will correlate response by ${req-id} - the same way, I suppose, application client. correlates command and its response.
Wouldn't that work?