Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
Most of my applications have focused on telemetry but I had been thinking about local control for solutions that have to run disconnected. In “real-world” deployments connectivity to Azure EventGrid MQTT Broker isn’t 100% reliable (also delay and jitter issues) which are an issue for control at the edge.
Over Christmas I read an article about the Internet of Vehicles(IoV) which got me thinking about “edge brokers”. In “real-world” deployments connectivity to Azure EventGrid MQTT Broker would not 100% reliable so I have been looking at lightweight edge brokers.
NanoMQ is a Linux Foundation Edge project. It is ultra-lightweight with a footprint of less than 200Kb when deployed with the minimum feature set, is cross platform (POSIX), has MQTT V5 support, bridging, and message persistence. The support for secure deployment is very limited.
Eclipse Mosquitto is another open-source message broker that implements the MQTT versions 5.0, 3.1.1 and 3.1. It is lightweight and is suitable for use on all devices from low power single board computers to full servers.
HiveMQ Edge is an MQTT V5, V3.1.1 and V3.1 compliant, with protocol adapters which connect various industrial communication protocols and integrate several database engines, bidirectional connections to enterprise brokers. For disconnected scenarios messages are stored on disk and published once online. There are opensource(community support) and commercial versions (additional features and support). The open-source version does not include offline persistence.
I couldn’t figure out why my code was failing so I gave up. Then the following day the application worked which was really odd. I then fired up my original Arduino PubSubClient library based application to do some testing and went to make a coffee. The application was failing to connect as XiaoTandHandCO2A and even when the bug was fixed (last will setup configuration commented out) it still wouldn’t connect.
Accidentally the configuration was changed XiaoTandHandCO2B and the application worked which was unexpected. I still couldn’t figure out why my code was failing XiaoTandHandCO2A so I gave up.
The following morning the XiaoTandHandCO2A configuration worked which was a bit odd. My best guess is that after a number of failed attempts the device is “disabled” for a while (but this is not displayed in the client configuration) and the following morning enough time had passed for the device to be “re-enabled”.
For testing I usually split my constants & secrets into two files, constants has the certificate chains for the Azure Event Grid MQTT Broker and secrets has certificate chains for devices and the Azure EventGrid MQTT Broker. I did consider have a secrets.h file for each device or splitting the device intermediate certificates, but as I was debugging and testing, it was just easier to duplicate them.
// constants.h
#pragma once
const uint16_t MQTT_PORT = 8883;
// This certificate is used to authenticate the host
static const char CA_ROOT_PEM[] PROGMEM = R"PEM(
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
)PEM";
To make deployments easier I have some software which stores the certificates in an optional “secure element” like the Microchip ATECC608C(The Seeedstudio Edgebox100 documentation refers to an optional Microchip ATECC680A but these are “not recommended for new designs”). This isn’t a great solution but deployments easier and is “better” than a string constant in the binary.
Still couldn’t figure out why my code was failing so I turned up logging to 11 and noticed a couple of messages which didn’t make sense. The device was connecting than disconnecting which indicated a another problem. As part of the Message Queue Telemetry Transport(MQTT) specification there is a “feature” Last Will and Testament(LWT) which a client can configure so that the MQTT broker sends a message to a topic if the device disconnects unexpectedly.
I was looking at the code and noticed that LWT was being used and that the topic didn’t exist in my Azure Event Grid MQTT Broker namespace. When the LWT configuration was commented out the application worked.
Paying close attention to the logging I noticed the “Subscribing to ssl/mqtts” followed by “Subscribe request sent”
I checked the sample application and found that if the connect was successful the application would then try and subscribe to a topic that didn’t exist.
The library doesn’t support client authentication with certificates, so I added two methods setClientCert and setClientKey to the esp-mqtt-arduino.h and esp-mqtt-arduino.cpp files
I tried increasing the log levels to get more debugging information, adding delays on startup to make it easier to see what was going on, trying different options of protocol support.
The PEM encoded root CA certificate chain that is used to validate the server
public const string CA_ROOT_PEM = @"-----BEGIN CERTIFICATE-----
CN: CN = Microsoft Azure ECC TLS Issuing CA 03
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
CN: CN = DigiCert Global Root G3
-----END CERTIFICATE-----";
The PEM encoded certificate chain that is used to authenticate the device
public const string CLIENT_CERT_PEM_A = @"-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----
CN=Self signed device certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
CN=Self signed Intermediate certificate
-----END CERTIFICATE-----";
The PEM encoded private key of device
public const string CLIENT_KEY_PEM_A = @"-----BEGIN EC PRIVATE KEY-----
-----END EC PRIVATE KEY-----";
For a non-trivial system there should be a number of intermediate certificates. I have tried creating intermediate certificates for a device type, geography, application, customer and combinations of these. The first couple of times got it wrong so start with a field trial so that it isn’t so painful to go back and fix. (beware the sunk cost fallacy)
I found creating an intermediate certificate that could sign device certificates required a conf file for the basicConstraints and keyUsage configuration.
critical-The extension must be understood and processed by any application validating the certificate. If the application does not understand it, the certificate must be rejected.
CA:TRUE-This certificate is allowed to act as a Certificate Authority (CA), meaning it can sign other certificates.
pathlen:0-This CA can only issue end-entity (leaf) certificates and cannot issue further intermediate CA certificates.
keyCertSig- The certificate can be used to sign other certificates (i.e., it’s a CA certificate).
For production systems putting some thought into the Common name(CN), Organizational unit name(OU), Organization name(O), locality name(L), state or province name(S) and Country name(C)
Establishing a connection to the Azure Event Grid MQTT broker often failed which surprised me. Initially I didn’t have any retry logic which meant I wasted quite a bit of time trying to debug failed connections
The Myriota Connector only supports Direct Methods which provide immediate confirmation of the result being queued by the Myriota Cloud API. The Myriota (API) control message send method responds with 400 Bad Request if there is already a message being sent to a device.
Myriota Azure Function Environment Variable configuration
Sense and Locate Azure IoT Central Template Fan Speed Enumeration
The fan speed value in the message payload is configured in the fan speed enumeration.
Sense and Locate Azure IoT Central Command Fan Speed Selection
The FanSpeed.cs payload formatter extracts the FanSpeed value from the Javascript Object Notation(JSON) payload and returns a two-byte array containing the message type and speed of the fan.
using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public class FormatterDownlink : PayloadFormatter.IFormatterDownlink
{
public byte[] Evaluate(string terminalId, string methodName, JObject payloadJson, byte[] payloadBytes)
{
byte? status = payloadJson.Value<byte?>("FanSpeed");
if (!status.HasValue)
{
return new byte[] { };
}
return new byte[] { 1, status.Value };
}
}
Sense and Locate Azure IoT Central Command Fan Speed History
Each Azure Application Insights log entry starts with the TerminalID (to simplify searching for all the messages related to device) and the requestId a Globally Unique Identifier (GUID) to simplify searching for all the “steps” associated with sending/receiving a message) with the rest of the logging message containing “step” specific diagnostic information.
Sense and Locate Azure IoT Central Command Fan Speed Application Insights
In the Myriota Device Manager the status of Control Messages can be tracked and they can be cancelled if in the “pending” state.
Myriota Control Message status Pending
A Control Message can take up to 24hrs to be delivered and confirmation of delivery has to be implemented by the application developer.