Eclipse Hono on Microsoft Azure

[erryB]

Hello Hono community,

I would like to have your feedback about Eclipse Hono on Microsoft Azure.
We are currently working on the deployment of Hono on Azure Kubernetes Service and we already opened a PR, but we would also like to improve the experience of Hono on Azure by using managed services.
Below you can find some topics we consider valuable to explore, what do you think?

Thanks,
Erica

Overview

The main goal is to provide the best experience of Eclipse Hono on Microsoft Azure, and to do so we should take care of different aspects:

• Eclipse Hono deployment on Microsoft Azure
• Integration of Eclipse Hono with Azure PaaS Services, to take advantage of some features like scalability and security
• Guidelines and samples to support the community

For each of these main topics we propose some ideas and steps to be followed, but before going deeper into technical details, we would like to get you feedback and thoughts in order to understand what features can be consider more valuable and interesting for Hono community. Please keep in mind that at the moment we are sharing ideas and this should not be considered as a promise for contribution.

Eclipse Hono deployment on Microsoft Azure

• Azure Kubernetes Service (AKS) deployment support
  As mentioned, we already started working on this, following the standard Kubernetes approach. We created this PR including:

  • step by step instructions to describe how to set up an AKS cluster and deploy Eclipse Hono as is
  • bash scripts and yaml files necessary to follow these guidelines

• Provide deployment guidelines using Helm, following similar approach.

• Provide guidelines to set up and leverage Azure Pipelines for CI/CD

Integration of Eclipse Hono with Azure PaaS Services

• Provide a reference for AMQP Messaging Network implementation
  The idea is to leverage Azure Event Hubs for scaling and performance capabilities. To do so, it's important to:

  • Define the most appropriate Partition Key to be used in order to have the best performances in Event Hubs.
  • Create a component (microservice) called Event Processor in order to read data off Azure Event Hubs. Details on its structure need to be defined.
  • Handle Consumer Groups and forward the messages to the correct Endpoint

• Provide a sample to leverage Azure CosmosDB in the Device Registry
  The goal is to provide the best experience in terms of scalability, security and georedundancy. Requirements and aspects we need to take into account include:

  • Compatibility with Hono Device Registry APIs
  • Partitioning
  • Performance
  • Security
  • Indexing and optimal storage strategy and API selection

Guidelines and samples to support the community

• Scripts and guidelines to deploy Hono on Azure Kubernetes Services: PR opened
• Sample and guidelines to use Azure Event Hubs
  This will include an additional microservice (Event Processor component)
• Sample and guidelines to use Azure CosmosDB in the Device Registry
  This will include an additional microservice (Azure Device Registry component)
• Guidelines to create a secure Network topology
  The main goal is to explain how to control access and limit data exposure, creating different subnets with different levels of trust.
• Guidelines for Monitoring
  The goal is to provide an overview of some Azure tools that can be used for monitoring purpose

[sophokles73]

Hi @erryB,

thanks a lot for your interest in bringing Hono to Microsoft Azure :-)
IMHO it would be wonderful if we could support users to leverage native Azure services to get the best possible experience when deploying Hono on Azure. I will be glad to engage the necessary discussions around this endeavor.

FMPOV all the topics you have pointed out are valid and deserve further discussion. In the past, the Hono project has mostly focused on providing the technical means to run Hono on container orchestration platforms (kubernetes, OpenShift) and allow integration with other (existing) services via standardized interfaces. That said, I think we can (and should) split up the proposed topics in two groups, a first one dealing with the technical implications of integrating Hono with native Azure services, and a second one, dealing with guides and and samples that promote and support the actual setup of those native services and describe the deployment (and potentially operation) of Hono on Azure.

Based on this distinction we might be able to create follow up issues scoped to specific aspects of the overall effort which we can then use to have a more focused discussion of these aspects.

WDYT? Does this make sense to you?

I have seen that you also have posted the content of this issue to the mailing list. In my experience, people find it easier to follow up on GitHub issues so I propose that we continue the discussion here. I will post a corresponding reply on the mailing list as well.

[erryB]

Hello @sophokles73

Sounds great to me! Here I described the general overview of our ideas, but the intention is indeed to go deeper into technical details for the integration with Azure services, and I completely agree with you that it will require separate and focused conversations.
We are currently working on the architectural structure of our proposal and we'll be happy to share more details soon.
In the meantime, any additional feedback is much appreciated :)

[dejanb]

Hi @erryB,

just to follow up on Kai's comment and welcome you to the community. To me, the proposed two aspects of future collaboration sounds good and I'm looking forward to discussing all the new ideas in separate threads.

[sophokles73]

@erryB

sorry for leaving you in the dark. FMPOV the most important topic would be the usage of one of Azure's native messaging services as a substitute for Apache Qpid and ActiveMQ Artemis.

Hono conceptually distinguishes between Telemetry data flowing downstream (from devices to applications) and Command & Control messages flowing upstream. Both have different characteristics regarding scalability and QoS and thus may be implemented using different services as well, each of them supporting the specific characteristics of one of those directions.

What about starting with the Telemetry direction? I could imagine that Azure EventHubs might be a good choice for it? WDYT?

[sophokles73]

BTW just a small remark on the picture you have included: devices only ever communicate with the protocol adapters. There shouldn't be any reason for them to invoke any of the Tenant, Device Registration or Credentials services nor would they normally be authorized to do so.

[mhemmeter]

Hi @erryB,
in your picture the business application is connecting to a component called Default AMQP 1.0 Messaging Network for interacting with Hono. The default implementation of the messaging network in Hono consists of a single Apache Qpid Dispatch Router instance connected to a single Apache Artemis broker instance. This setup is not meant for productive use (at least if we are talking about a scalable cloud service) and I'm wondering if you already have an idea if this component will be in place in the target architecture and if yes how it can be implemented?

[erryB]

Thanks a lot for your additional feedback! I'm going to fix the picture and to create a new issue describing the architecture we propose for the messaging network, including both Device to Cloud and Cloud to Device messages.