Add support for the NGINX MQTT module

[brianehlert]

With the release of NGINX Plus R29 came the addition of a new MQTT module
https://www.nginx.com/blog/nginx-plus-r29-released/#_Support_for_MQTT

• MQTT broker load balancing
• Session persistence (reconnecting clients to the same broker)
• TLS termination
• Client certificate authentication
• CONNECT message parsing and rewriting

This should be supported as a first class protocol with the ingress controller.
I would like to raise a conversation around this.

MQTT is a bit of a interesting protocol. It is a bit of layer 4 and a bit of layer 7. But is definitely its own thing.
And the NGINX module exposes capabilities similar to header manipulation: "CONNECT message parsing and rewriting"

For the NGINX Ingress Controller does this manifest as an enhancement to TransportServer (L4 / stream under the hood), an enhancement to VirtualServer (L7 / http under the hood), or should a new object be created (such as would be an MQTTRoute with Gateway API) to express the unique capabilities and characteristics of MQTT.
To stay true to the 'server' naming, this could be an MQTTServer or IoTServer

I am torn between a new object or a new kind for TransportServer.
I would like to hear from those of you out there, and specifically from customers that would immediately want to implement MQTT support.

[jasonwilliams14]

MQTT released in NGINX R29.

NGINX Plus R29 release

Initial release provides support for MQTT versions 3.1.1 and 5.
Two new modules are introduced:

• ngx_stream_mqtt_preread_module
• ngx_stream_mqtt_filter_module

ngx_stream_mqtt_preread_module allows for the parsing of MQTT CONNECT messages to extract MQTT client ID and username, fields, expose them as NGINX variables before proxying the connection to the backend servers.

These variables include:

• $mqtt_preread_clientid
• $mqtt_preread_username

These new variables can be used by NGINX Ingress controller for hash load balancing, connecting to the same clients and users to the same MQTT backends.

This new release has a mqtt_buffers directive that can be used to tune as needed specifically for MQTT.

To enable ngx_stream_mqtt_preread_module you need to enable it:

mqtt_preread on;
Context: stream,server

mqtt_preread on is only for the ngx_stream_mqtt_preread_module.

Simple stream example:

stream {
  upstream mqtt_example {
    upstream 192.168.1.5:8881;
    upstream 192.168.1.6:8881;
    hash $mqtt_preread_clientid consistent;
    }

server {
    listen 127.0.0.1:8881;

    mqtt_preread on;

    proxy_pass mqtt_example;
    }

ngx_stream_mqtt_filter_module can parse any MQTT message and allows for rewriting MQTT CONNECT messages while proxying them to the backend servers.
In this simple example, we rewrite MQTT connect client ID field with the subject DN string of the client certificate for an established SSL connection.

mqtt on; this option is only for the mqtt_filter_module and turns on the mqtt_filter_module

stream {
  upstream mqtt_example {
  server 192.168.1.5:8881;
  server 192.168.1.6:8881;

  hash $mqtt_preread_client_id consistent;
  }

server {
    listen 127.0.0.1:8881;

    mqtt_preread on;
    mqtt on;
    mqtt_set_connect clientid $client;

    proxy_pass mqtt_example;
    }

Specific items to consider for NGINX Ingress controller.

Since MQTT is layer 4, and will be using stream. Therefore, we need to configure both globalConfiguration and transportServer.

If filtering needs to be done on MQTT for rewrites, then mqtt_set_connect needs to be enabled for clientid and username.

globalConfiguration items to consider to add.

apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
  name: nginx-configuration

spec:
  listeners:
  - name: mqtt-udp
    port: 8881
    protocol: TCP
    mqtt: true
  - name: mqtt-tcp
    port: 8881
    protocol: TCP
    mqtt: true

Sample transportServer config:

apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: mqtt-tcp
spec:
  listener:
    name: mqtt-tcp
    protocol: TCP
  upstreams:
  - name: mqtt-app
    service: mqtt-app
    port: 8881
  action:
    pass: mqtt-app

In order to use the mqtt_filter_module, we need to expand action in transportServer like we do for virtualserver.