Tag Archives: mqtt

Azure Event Grid Arduino Client – Publisher

Posted on by

The Arduino application for testing Azure Event Grid MQTT Broker connectivity worked on my Seeedstudio EdgeBox ESP100 and Seeedstudio Xiao ESP32S3 devices, so the next step was to modify it to publish some messages.

The first version generated the JSON payload using an snprintf which was a bit “nasty”

static uint32_t sequenceNumber = 0;

void loop() {
  mqttClient.loop();

  Serial.println("MQTT Publish start");

  char payloadBuffer[64];

  snprintf(payloadBuffer, sizeof(payloadBuffer), "{\"ClientID\":\"%s\", \"Sequence\": %i}", MQTT_CLIENTID, sequenceNumber++);

  Serial.println(payloadBuffer);

  if (!mqttClient.publish(MQTT_TOPIC_PUBLISH, payloadBuffer, strlen(payloadBuffer))) {
    Serial.print("\nMQTT publish failed:");        
    Serial.println(mqttClient.state());    
  }
  Serial.println("MQTT Publish finish");

  delay(60000);
}

I then configured my client (Edgebox100A) and updated the “secrets.h” file

Azure Event Grid MQTT Broker Clients

The application connected to the Azure Event Grid MQTT broker and started publishing the JSON payload with the incrementing sequence number.

Arduino IDE serial monitor output of JSON payload publishing

The second version generated the JSON payload using ArduinoJson library.

static uint32_t sequenceNumber = 0;

void loop() {
  mqttClient.loop();

  Serial.println("MQTT Publish start");

  // Create a static JSON document with fixed size
  StaticJsonDocument<64> doc;

  doc["Sequence"] = counter++;
  doc["ClientID"] = MQTT_CLIENTID;

  // Serialize JSON to a buffer
  char jsonBuffer[64];
  size_t n = serializeJson(doc, jsonBuffer);

  Serial.println(jsonBuffer);

  if(!mqttClient.publish(MQTT_TOPIC_PUBLISH, jsonBuffer, n))
  {
    Serial.println(mqttClient.state());    
  }

  Serial.println("MQTT Publish finish");

  delay(2000);
}

I could see the application was working in the Azure Event Grid MQTT broker metrics because the number of messages published was increasing.

Azure Event Grid MQTT Broker metrics with messages published selected

The published messages were “routed” to an Azure Storage Queue where they can be inspected with a tool like Azure Storage Explorer.

Azure Storage Explorer displaying a message’s payload

The message payload is in Base64 encoded so I used copilot convert it to text.

Microsoft copilot decoding the Base64 payload

In this post I have assumed that the reader is familiar with configuring Azure Event Grid clients, client groups, topic spaces, permission bindings and routing.

Bonus also managed to slip in a reference to copilot.

Azure Event Grid Arduino Client – The joy of certs

Posted on by

“Lets start at the very beginning, A very good place to start”

The Azure Event Grid MQTT Broker server X509 certificate chain can be copy ‘n’ paste from the output of the openssl command

openssl s_client -connect YourNamespace.newzealandnorth-1.ts.eventgrid.azure.net:8883 -showcerts

A self-signed X509 root certificate which can sign intermediate X509 certificates and key file can be generated with a single openssl command.

openssl req -x509 -newkey rsa:4096 -keyout rootCA.key -out rootCA.crt -days 3650 -nodes -subj "/CN=devMobile  /O=devMobile.co.nz /C=NZ" -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign"

For a non-trivial system there should be a number of intermediate certificates. I have tried creating intermediate certificates for a device type, geography, application, customer and combinations of these. The first couple of times got it wrong so start with a field trial so that it isn’t so painful to go back and fix. (beware the sunk cost fallacy)

openssl genrsa -out intermediate.key 4096

openssl req -new -key intermediate.key -out intermediate.csr -subj "/CN=intermediate  /O=devMobile.co.nz /C=NZ"

I found creating an intermediate certificate that could sign device certificates required a conf file for the basicConstraints and keyUsage configuration.

[ v3_intermediate_ca ]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, keyCertSign
  • critical-The extension must be understood and processed by any application validating the certificate. If the application does not understand it, the certificate must be rejected.
  • CA:TRUE-This certificate is allowed to act as a Certificate Authority (CA), meaning it can sign other certificates.
  • pathlen:0-This CA can only issue end-entity (leaf) certificates and cannot issue further intermediate CA certificates.
  • keyCertSig- The certificate can be used to sign other certificates (i.e., it’s a CA certificate).
openssl x509 -req -in intermediate.csr  -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out intermediate.crt -days 1825 -extfile intermediate_ext.cnf -extensions v3_intermediate_ca

Creating a device certificate is similar to the process for the intermediate certificate but doesn’t need to be able to sign certificates.

openssl genrsa -out EdgeBox100A.key 4096

openssl req -new -key EdgeBox100A.key -out EdgeBox100A.csr -subj "/CN=EdgeBox100A"

openssl x509 -req -in EdgeBox100A.csr -CA intermediate.crt -CAkey intermediate.key -CAcreateserial -out EdgeBox100A.crt -days 365

For production systems putting some thought into the Common name(CN), Organizational unit name(OU), Organization name(O), locality name(L), state or province name(S) and Country name(C)

// Minimalist ESP32 + Event Grid MQTT (mTLS) with PubSubClient
// Copyright (c) November 2025, devMobile Software
#include <PubSubClient.h>
#include <WiFi.h>
#include <WiFiClientSecure.h>

#include "constants.h"
#include "secrets.h"

// --- Wi-Fi ---
//const char* WIFI_SSID     = "";
//const char* WIFI_PASSWORD = "";

// --- Event Grid MQTT ---
//const char* MQTT_SERVER = "";
const uint16_t MQTT_PORT = 8883;

//const char* MQTT_CLIENTID = "";
//const char* MQTT_USERNAME = "";
//const char* MQTT_PASSWORD = "";
//const char* MQTT_TOPIC_PUBLISH = "devices/";
//const char* MQTT_TOPIC_SUBSCRIBE = "devices/";

/*
// The certificate that is used to authenticate the MQTT Broker
const char CA_ROOT_PEM[] PROGMEM = R"PEM(
-----BEGIN CERTIFICATE-----
      Thumbprint: 56D955C849887874AA1767810366D90ADF6C8536
      CN: CN=Microsoft Azure ECC TLS Issuing CA 03
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
      Thumbprint: 7E04DE896A3E666D00E687D33FFAD93BE83D349E
      CN: CN=DigiCert Global Root G3
-----END CERTIFICATE-----
)PEM";

The certificate that is used to authenticate the device
static const char CLIENT_CERT_PEM[] PROGMEM = R"PEM(
-----BEGIN CERTIFICATE-----
 CN=Self signed device certificate
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
 CN=Self signed Intermediate certificate
-----END CERTIFICATE-----
)PEM";

 The PEM encoded private key of device
static const char CLIENT_KEY_PEM[] PROGMEM = R"PEM(
-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----
)PEM";
*/

WiFiClientSecure secureClient;
PubSubClient mqttClient(secureClient);

void setup() {
  Serial.begin(9600);
  delay(5000);
  Serial.println();

  // Connect to WiFi
  Serial.println("WiFi connecting");
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("*");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print("*");
  }
  Serial.println("\nWiFi connected");

  // Sync time for TLS
  Serial.println("\nTime synchronising");
  configTime(0, 0, "pool.ntp.org", "time.nist.gov");
  Serial.print("*");
  while (time(nullptr) < 100000) {
    delay(500);
    Serial.print("*");
  }
  Serial.println("\nTime synchronised");

  Serial.println("\nValidating ServerFQDN-Certificate combination");
  secureClient.setCACert(CA_ROOT_PEM);

  Serial.println("TCP connecting");
  if (secureClient.connect(MQTT_SERVER, MQTT_PORT)) {
    Serial.println("\nTCP connected");
  } else {
    Serial.println("\nTCP connection failed");
    return;
  }

  secureClient.setCertificate(CLIENT_CERT_A_PEM);
  secureClient.setPrivateKey(CLIENT_KEY_A_PEM);

  mqttClient.setServer(MQTT_SERVER, MQTT_PORT);

  Serial.println("\nMQTT connecting");
  Serial.print("*");
  while (!mqttClient.connect(MQTT_CLIENTID, MQTT_USERNAME, MQTT_PASSWORD)) {
    Serial.println(mqttClient.state());
    delay(5000);
    Serial.print("*");
  }
  Serial.println("\nMQTT connected");
}

static uint32_t sequenceNumber = 0;

void loop() {
  mqttClient.loop();

  Serial.print("'.");
  delay(10000);
}

My Arduino Xiao ESP32S3 and EdgeBox-ESP-100-Industrial Edge Controller devices could connect to the local Wi-Fi, get the time and date using the network time protocol(NTP), and validate the Azure Event Grid MQTT broker certificate. Then connect to the Azure Event Grid MQTT broker with the client name specified in the subject name of its X509 certificate.

Establishing a connection to the Azure Event Grid MQTT broker often failed which surprised me. Initially I didn’t have any retry logic which meant I wasted quite a bit of time trying to debug failed connections

Azure Event Grid MQTT-With HiveMQ & MQTTnet Clients

Posted on by

Most of the examples of connecting to Azure Event Grid’s MQTT broker use MQTTnet so for a bit of variety I started with a hivemq-mqtt-client-dotnet based client. (A customer had been evaluating HiveMQ for a project which was later cancelled)

BEWARE – ClientID parameter is case sensitive.

The HiveMQ client was “inspired” by the How to Guides > Custom Client Certificates documentation.

class Program
{
   private static Model.ApplicationSettings _applicationSettings;
   private static HiveMQClient _client;
   private static bool _publisherBusy = false;

   static async Task Main()
   {
      Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Hive MQ client starting");

      try
      {
         // load the app settings into configuration
         var configuration = new ConfigurationBuilder()
               .AddJsonFile("appsettings.json", false, true)
               .AddUserSecrets<Program>()
         .Build();

         _applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();

         var optionsBuilder = new HiveMQClientOptionsBuilder();

         optionsBuilder
            .WithClientId(_applicationSettings.ClientId)
            .WithBroker(_applicationSettings.Host)
            .WithPort(_applicationSettings.Port)
            .WithUserName(_applicationSettings.UserName)
            .WithCleanStart(_applicationSettings.CleanStart)
            .WithClientCertificate(_applicationSettings.ClientCertificateFileName, _applicationSettings.ClientCertificatePassword)
            .WithUseTls(true);

         using (_client = new HiveMQClient(optionsBuilder.Build()))
         {
            _client.OnMessageReceived += OnMessageReceived;

            var connectResult = await _client.ConnectAsync();
            if (connectResult.ReasonCode != ConnAckReasonCode.Success)
            {
               throw new Exception($"Failed to connect: {connectResult.ReasonString}");
            }

            Console.WriteLine($"Subscribed to Topic");
            foreach (string topic in _applicationSettings.SubscribeTopics.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
            {
               var subscribeResult = await _client.SubscribeAsync(topic, _applicationSettings.SubscribeQualityOfService);

               Console.WriteLine($" Topic:{topic} Result:{subscribeResult.Subscriptions[0].SubscribeReasonCode}");
            }
   }
//...
}
HiveMQ Client console application output

The MQTTnet client was “inspired” by the Azure MQTT .NET Application sample

class Program
{
   private static Model.ApplicationSettings _applicationSettings;
   private static IMqttClient _client;
   private static bool _publisherBusy = false;

   static async Task Main()
   {
      Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} MQTTNet client starting");

      try
      {
         // load the app settings into configuration
         var configuration = new ConfigurationBuilder()
               .AddJsonFile("appsettings.json", false, true)
               .AddUserSecrets<Program>()
         .Build();

         _applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();

         var mqttFactory = new MqttFactory();

         using (_client = mqttFactory.CreateMqttClient())
         {
            // Certificate based authentication
            List<X509Certificate2> certificates = new List<X509Certificate2>
            {
               new X509Certificate2(_applicationSettings.ClientCertificateFileName, _applicationSettings.ClientCertificatePassword)
            };

            var tlsOptions = new MqttClientTlsOptionsBuilder()
                  .WithClientCertificates(certificates)
                  .WithSslProtocols(System.Security.Authentication.SslProtocols.Tls12)
                  .UseTls(true)
                  .Build();

            MqttClientOptions mqttClientOptions = new MqttClientOptionsBuilder()
                     .WithClientId(_applicationSettings.ClientId)
                     .WithTcpServer(_applicationSettings.Host, _applicationSettings.Port)
                     .WithCredentials(_applicationSettings.UserName, _applicationSettings.Password)
                     .WithCleanStart(_applicationSettings.CleanStart)
                     .WithTlsOptions(tlsOptions)
                     .Build();

            var connectResult = await _client.ConnectAsync(mqttClientOptions);
            if (connectResult.ResultCode != MqttClientConnectResultCode.Success)
            {
               throw new Exception($"Failed to connect: {connectResult.ReasonString}");
            }

            _client.ApplicationMessageReceivedAsync += OnApplicationMessageReceivedAsync;

            Console.WriteLine($"Subscribed to Topic");
            foreach (string topic in _applicationSettings.SubscribeTopics.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
            {
               var subscribeResult = await _client.SubscribeAsync(topic, _applicationSettings.SubscribeQualityOfService);

               Console.WriteLine($" {topic} Result:{subscribeResult.Items.First().ResultCode}");
            }
      }
//...
}
MQTTnet client console application output

The design of the MQTT protocol means that the hivemq-mqtt-client-dotnet and MQTTnet implementations are similar. Having used both I personally prefer the HiveMQ client library.

Azure Event Grid MQTT-Certificates

Posted on by

When configuring Root, Intermediate and Device certificates for my Azure Event Grid Devices using the smallstep step-ca or OpenSSL I made mistakes/typos which broke my deployment and in the end I was copying and pasting commands from Windows Notepad.

For one test deployment it took me an hour to generate the Root, Intermediate and a number of Devices certificates which was a waste of time. At this point I decided investigate writing some applications to simplify the process.

After some searching I stumbled across CREATING CERTIFICATES FOR X.509 SECURITY IN AZURE IOT HUB USING .NET CORE by Damien Bod which showed how to generate certificates for Azure IoT Hub solutions using his NuGet package Certificate Manager.

The AzureIoTHubDps repository had a sample showing how to generate the certificate chain for Azure IoT Hub devices. It worked really well but I accidentally overwrote my Root and Intermediate certificates, there were some “magic numbers” and hard coded passwords (it was a sample) so I decided to “chop” the sample up into three command line applications. 

static void Main(string[] args)
{
   var serviceProvider = new ServiceCollection()
         .AddCertificateManager()
         .BuildServiceProvider();

   // load the app settings into configuration
   var configuration = new ConfigurationBuilder()
         .AddJsonFile("appsettings.json", false, true)
         .AddUserSecrets<Program>()
   .Build();

   _applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
//------
   Console.WriteLine($"validFrom:{validFrom} ValidTo:{validTo}");

   var serverRootCertificate = serviceProvider.GetService<CreateCertificatesClientServerAuth>();

   var root = serverRootCertificate.NewRootCertificate(
         new DistinguishedName { 
               CommonName = _applicationSettings.CommonName,
               Organisation = _applicationSettings.Organisation,
               OrganisationUnit = _applicationSettings.OrganisationUnit,
               Locality = _applicationSettings.Locality,
               StateProvince  = _applicationSettings.StateProvince,
               Country = _applicationSettings.Country
         },
         new ValidityPeriod { 
         ValidFrom = validFrom,
         ValidTo = validTo,
         },
         _applicationSettings.PathLengthConstraint,
         _applicationSettings.DnsName);
   root.FriendlyName = _applicationSettings.FriendlyName;

   Console.Write("PFX Password:");
   string password = Console.ReadLine();
   if ( String.IsNullOrEmpty(password))
   {
      Console.WriteLine("PFX Password invalid");
      return;
   }

   var exportCertificate = serviceProvider.GetService<ImportExportCertificate>();

   var rootCertificatePfxBytes = exportCertificate.ExportRootPfx(password, root);
   File.WriteAllBytes(_applicationSettings.RootCertificateFilePath, rootCertificatePfxBytes);

   Console.WriteLine($"Root certificate file:{_applicationSettings.RootCertificateFilePath}");
   Console.WriteLine("press enter to exit");
   Console.ReadLine();
}

The application’s configuration was split between application settings file(certificate file paths, validity periods, Organisation etc.) or entered at runtime ( certificate filenames, passwords etc.) The first application generates a Root Certificate using the distinguished name information from the application settings, plus file names and passwords entered by the user.

Root Certificate generation application output

The second application generates an Intermediate Certificate using the Root Certificate, the distinguished name information from the application settings, plus file names and passwords entered by the user.

static void Main(string[] args)
{
   var serviceProvider = new ServiceCollection()
         .AddCertificateManager()
         .BuildServiceProvider();

   // load the app settings into configuration
   var configuration = new ConfigurationBuilder()
         .AddJsonFile("appsettings.json", false, true)
         .AddUserSecrets<Program>()
   .Build();

   _applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
//------
   Console.WriteLine($"validFrom:{validFrom} be after ValidTo:{validTo}");

   Console.WriteLine($"Root Certificate file:{_applicationSettings.RootCertificateFilePath}");

   Console.Write("Root Certificate Password:");
   string rootPassword = Console.ReadLine();
   if (String.IsNullOrEmpty(rootPassword))
   {
      Console.WriteLine("Fail");
      return;
   }
   var rootCertificate = new X509Certificate2(_applicationSettings.RootCertificateFilePath, rootPassword);

   var intermediateCertificateCreate = serviceProvider.GetService<CreateCertificatesClientServerAuth>();

   var intermediateCertificate = intermediateCertificateCreate.NewIntermediateChainedCertificate(
         new DistinguishedName
         {
            CommonName = _applicationSettings.CommonName,
            Organisation = _applicationSettings.Organisation,
            OrganisationUnit = _applicationSettings.OrganisationUnit,
            Locality = _applicationSettings.Locality,
            StateProvince = _applicationSettings.StateProvince,
            Country = _applicationSettings.Country
         },
      new ValidityPeriod
      {
         ValidFrom = validFrom,
         ValidTo = validTo,
      },
            _applicationSettings.PathLengthConstraint,
            _applicationSettings.DnsName, rootCertificate);
      intermediateCertificate.FriendlyName = _applicationSettings.FriendlyName;

   Console.Write("Intermediate certificate Password:");
   string intermediatePassword = Console.ReadLine();
   if (String.IsNullOrEmpty(intermediatePassword))
   {
      Console.WriteLine("Fail");
      return;
   }

   var importExportCertificate = serviceProvider.GetService<ImportExportCertificate>();

   Console.WriteLine($"Intermediate PFX file:{_applicationSettings.IntermediateCertificatePfxFilePath}");
   var intermediateCertificatePfxBtyes = importExportCertificate.ExportChainedCertificatePfx(intermediatePassword, intermediateCertificate, rootCertificate);
   File.WriteAllBytes(_applicationSettings.IntermediateCertificatePfxFilePath, intermediateCertificatePfxBtyes);

   Console.WriteLine($"Intermediate CER file:{_applicationSettings.IntermediateCertificateCerFilePath}");
   var intermediateCertificatePemText = importExportCertificate.PemExportPublicKeyCertificate(intermediateCertificate);
   File.WriteAllText(_applicationSettings.IntermediateCertificateCerFilePath, intermediateCertificatePemText);

   Console.WriteLine("press enter to exit");
   Console.ReadLine();
}
Intermediate Certificate generation application output
Uploading the Intermediate certificate to Azure Event Grid

The third application generates Device Certificates using the Intermediate Certificate, distinguished name information from the application settings, plus device id, file names and passwords entered by the user.

static void Main(string[] args)
{
   var serviceProvider = new ServiceCollection()
         .AddCertificateManager()
         .BuildServiceProvider();

   // load the app settings into configuration
   var configuration = new ConfigurationBuilder()
         .AddJsonFile("appsettings.json", false, true)
         .AddUserSecrets<Program>()
   .Build();

   _applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
//------
   Console.WriteLine($"validFrom:{validFrom} ValidTo:{validTo}");

   Console.WriteLine($"Intermediate PFX file:{_applicationSettings.IntermediateCertificateFilePath}");

   Console.Write("Intermediate PFX Password:");
   string intermediatePassword = Console.ReadLine();
   if (String.IsNullOrEmpty(intermediatePassword))
   {
      Console.WriteLine("Intermediate PFX Password invalid");
      return;
   }
   var intermediate = new X509Certificate2(_applicationSettings.IntermediateCertificateFilePath, intermediatePassword);

   Console.Write("Device ID:");
   string deviceId = Console.ReadLine();
   if (String.IsNullOrEmpty(deviceId))
   {
      Console.WriteLine("Device ID invalid");
      return;
   }

   var createClientServerAuthCerts = serviceProvider.GetService<CreateCertificatesClientServerAuth>();

   var device = createClientServerAuthCerts.NewDeviceChainedCertificate(
         new DistinguishedName
         {
            CommonName = deviceId,
            Organisation = _applicationSettings.Organisation,
            OrganisationUnit = _applicationSettings.OrganisationUnit,
            Locality = _applicationSettings.Locality,
            StateProvince = _applicationSettings.StateProvince,
            Country = _applicationSettings.Country
         },
      new ValidityPeriod
      {
         ValidFrom = validFrom,
         ValidTo = validTo,
      },
      deviceId, intermediate);
   device.FriendlyName = deviceId;

   Console.Write("Device PFX Password:");
   string devicePassword = Console.ReadLine();
   if (String.IsNullOrEmpty(devicePassword))
   {
      Console.WriteLine("Fail");
      return;
   }

   var importExportCertificate = serviceProvider.GetService<ImportExportCertificate>();

   string devicePfxPath = string.Format(_applicationSettings.DeviceCertificatePfxFilePath, deviceId);

   Console.WriteLine($"Device PFX file:{devicePfxPath}");
   var deviceCertificatePath = importExportCertificate.ExportChainedCertificatePfx(devicePassword, device, intermediate);
   File.WriteAllBytes(devicePfxPath,  deviceCertificatePath);

   Console.WriteLine("press enter to exit");
   Console.ReadLine();
}
Device Certificate generation application output
Uploading the Intermediate certificate to Azure Event Grid

These applications wouldn’t have been possible without Damien Bod’s CREATING CERTIFICATES FOR X.509 SECURITY IN AZURE IOT HUB USING .NET CORE blog post, and his Certificate Manager NuGet package.

RAK7258 Local server and Message Queuing Telemetry Transport(MQTT)

Posted on by

This post was originally about getting the built in Network Server of my RAKWireless RAK7258 WisGate Edge Lite to connect to an Azure IoT Hub or Azure IoT Central. The RAK7258 had been connected to The Things Industries(TTI) network so I updated the firmware and checked the “mode” in the LoRaWAN Network settings.

RAK 7258 LoRaWAN Network settings

Azure IoT Hub is not a fully featured MQTT broker so I initially looked at running Eclipse Mosquitto or HiveMQ locally but this seemed like a lot of effort for a Proof of Concept(PoC).

RAK 7258 Network Server Global Integration settings

I have used MQTTNet in a few other projects (The Things Network(TTN) V3 Azure IoT Connector, The Things Network V2 MQTT SQL Connector, Windows 10 IoT Core MQTT Field gateway etc.) and there was a sample application which showed ho to build a simple server so that became my preferred approach.

I then started exploring how applications and devices are provisioned in the RAK Network Server.

RAK 7258 Network Server applications list

The network server software has “unified” and “separate” “Device authentication mode”s and will “auto Add LoRa Device”s if enabled.

RAK 7258 Network Server Separate Application basic setup
RAK 7258 Network Server Separate Application device basic setup
RAK 7258 Network Server Unified Application device basic setup

Applications also have configurable payload formats(raw & LPP) and integrations (uplink messages plus join, ack, and device notifications etc.)

RAK7258 live device data display

In the sample server I could see how ValidatingConnectionAsync was used to check the clientID, username and password when a device connected. I just wanted to display messages and payloads without having to use an MQTT client and it looked like InterceptingPublishAsync was a possible solution.

But the search results were a bit sparse…

InterceptingPublishAsync + MQTTNet search results

After some reading the MQTTNet documentation and some experimentation I could display the message payload (same as in the live device data display) in a “nasty” console application.

namespace devMobile.IoT.RAKWisgate.ServerBasic
{
   using System;
	using System.Threading.Tasks;

   using MQTTnet;
   using MQTTnet.Protocol;
   using MQTTnet.Server;

   public static class Program
   {
      static async Task Main(string[] args)
      {
         var mqttFactory = new MqttFactory();

         var mqttServerOptions = new MqttServerOptionsBuilder()
             .WithDefaultEndpoint()
             .Build();

         using (var mqttServer = mqttFactory.CreateMqttServer(mqttServerOptions))
         {
            mqttServer.InterceptingPublishAsync += e =>
            {
               Console.WriteLine($"Client:{e.ClientId} Topic:{e.ApplicationMessage.Topic} {e.ApplicationMessage.ConvertPayloadToString()}");

               return Task.CompletedTask;
            };

            mqttServer.ValidatingConnectionAsync += e =>
            {
               if (e.ClientId != "RAK Wisgate7258")
               {
                  e.ReasonCode = MqttConnectReasonCode.ClientIdentifierNotValid;
               }

               if (e.Username != "ValidUser")
               {
                  e.ReasonCode = MqttConnectReasonCode.BadUserNameOrPassword;
               }

               if (e.Password != "TopSecretPassword")
               {
                  e.ReasonCode = MqttConnectReasonCode.BadUserNameOrPassword;
               }

               return Task.CompletedTask;
            };

            await mqttServer.StartAsync();

            Console.WriteLine("Press Enter to exit.");
            Console.ReadLine();

            await mqttServer.StopAsync();
         }
      }
   }
}
MQTTNet based console application displaying device payloads

The process of provisioning Applications and Devices is quite different (The use of the AppEUI/JoinEUI is odd) to The Things Network(TTN) and other platforms I have used so I will explore this some more in future post(s).

TTN V3 Connector Revisited

Posted on by

Earlier in the year I built Things Network(TTN) V2 and V3 connectors and after using these in production applications I have learnt a lot about what I had got wrong, less wrong and what I had got right.

Using a TTN V3 MQTT Application integration wasn’t a great idea. The management of state was very complex. The storage of application keys in a app.settings file made configuration easy but was bad for security.

The use of Azure Key Vault in the TTNV2 connector was a good approach, but the process of creation and updating of the settings needs to be easier.

Using TTN device registry as the “single source of truth” was a good decision as managing the amount of LoRaWAN network, application and device specific configuration in an Azure IoT Hub would be non-trivial.

Using a Webhooks Application Integration like the TTNV2 connector is my preferred approach.

The TTNV2 Connector’s use of Azure Storage Queues was a good idea as they it provide an elastic buffer between the different parts of the application.

The use of Azure Functions to securely ingest webhook calls and write them to Azure Storage Queues with output bindingts should simplify configuration and deployment. The use of Azure Storage Queue input bindings to process messages is the preferred approach.

The TTN V3 processing of JSON uplink messages into a structure that Azure IoT Central could ingest is a required feature

The TTN V2 and V3 support for the Azure Device Provisioning Service(DPS) is a required feature (mandated by Azure IoT Central). The TTN V3 connector support for DTDLV2 is a desirable feature. The DPS implementation worked with Azure IoT Central but I was unable to get the DeviceClient based version working.

Using DPS to pre-provision devices in Azure IoT Hubs and Azure IoT Central by using the TTN Application Registry API then enumerating the TTN applications, then devices needs to be revisited as it was initially slow then became quite complex.

The support for Azure IoT Hub connection strings was a useful feature, but added some complexity. This plus basic Azure IoT Hub DPS support(No Azure IoT Central support) could be implemented in a standalone application which connects via Azure Storage Queue messages.

The processing of Azure IoT Central Basic, and Request commands then translating the payloads so they work with TTN V3 is a required feature. The management of Azure IoT Hub command delivery confirmations (abandon, complete and Reject) is a required feature.

I’m considering building a new TTN V3 connector but is it worth the effort as TTN has one now?

TTI V3 Gateway Device Provisioning Service(DPS) Performance

Posted on by

My The Things Industries(TTI) V3 connector is an Identity Translation Cloud Gateway, it maps LoRaWAN devices to Azure IoT Hub devices. The connector creates a DeviceClient for each TTI LoRaWAN device and can use an Azure Device Connection string or the Azure Device Provisioning Service(DPS).

While debugging the connector on my desktop I had noticed that using a connection string was quite a bit faster than using DPS and I had assumed this was just happenstance. While doing some testing in the Azure North Europe data-center (Closer to TTI European servers) I grabbed some screen shots of the trace messages in Azure Application Insights as the TTI Connector Application was starting.

I only have six LoRaWAN devices configured in my TTI dev instance, but I repeated each test several times and the results were consistent so the request durations are reasonable. My TTI Connector application, IoT Hub, DPS and Application insights instances are all in the same Azure Region and Azure Resource Group so networking overheads shouldn’t be significant.

Azure IoT Hub Connection device connection string

Using an Azure IoT Hub Device Shared Access policy connection string establishing a connection took less than a second.

My Azure DPS Instance

Using my own DPS instance to provide the connection string and then establishing a connection took between 3 and 7 seconds.

Azure IoT Central DPS

For my Azure IoT Central instance getting a connection string and establishing a connection took between 4 and 7 seconds.

The Azure DPS client code was copied from one of the sample applications so I have assumed it is “correct”.

using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
{
	ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create( 
		Constants.AzureDpsGlobalDeviceEndpoint,
		deviceProvisiongServiceSettings.IdScope,
		securityProvider,
		transport);

	DeviceRegistrationResult result;

	if (!string.IsNullOrEmpty(modelId))
	{
		ProvisioningRegistrationAdditionalData provisioningRegistrationAdditionalData = new ProvisioningRegistrationAdditionalData()
		{
			JsonData = $"{{"modelId": "{modelId}"}}"
		};

		result = await provClient.RegisterAsync(provisioningRegistrationAdditionalData, stoppingToken);
	}
	else
    {
		result = await provClient.RegisterAsync(stoppingToken);
	}

	if (result.Status != ProvisioningRegistrationStatusType.Assigned)
	{
		_logger.LogError("Config-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);

		return false;
	}

	IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

	deviceClient = DeviceClient.Create(result.AssignedHub, authentication, transportSettings);
}

I need to investigate why getting a connection string from the DPS then connecting take significantly longer (I appreciate that “behind the scenes” service calls maybe required). This wouldn’t be an issue for individual devices connecting from different locations but for my Identity Translation Cloud gateway which currently open connections sequentially this could be a problem when there are a large number of devices.

If the individual requests duration can’t be reduced (using connection pooling etc.) I may have to spin up multiple threads so multiple devices can be connecting concurrently.

TTI V3 Gateway Azure IoT Central Digital Twin Definition Language(DTDL) support

Posted on by

Over the last couple of days I have added limited Digital Twin Definition Language(DTDLV2) support to my The Things Industries(TTI) V3 connector so that Azure IoT Central devices can be “zero touch” provisioned. For this blog post I used five Seeeduino LoRaWAN devices left over from another abandoned project.

The first step was to configure and Azure IoT Central enrollment group (ensure “Automatically connect devices in this group” is on) and copy the IDScope and Group Enrollment key to the appsettings.json file (see sample file below for more detail)

Azure IoT Central Enrollment Group configuration

Then I created an Azure IoT Central template for the seeeduino LoRAWAN devices which are running software (developed with the Arduino tooling) that read values from a Grove – Temperature&Humidity sensor. The naming of telemetry properties in specified by the Low Power Protocol(LPP) encoder/decoder (I check the decoded payload in TTI EndDevice “Live Data” tab).

Configuring Seeeduino LoRaWAN device template

Then I mapped the Azure IoT Central Device Group to my Azure IoT Central Enrollment Group

Associating Device Group with Group Enrollment configuration

The Device Template @Id can be configured as the “default” template for all the devices in a TTI application in the app.settings.json file.

{
...
   "ProgramSettings": {
      "Applications": {
...
      "seeeduinolorawan": {
        "AzureSettings": {
           "DeviceProvisioningServiceSettings": {
              "IdScope": "...",
              "GroupEnrollmentKey": "..."
            }
         },
         "DTDLModelId": "dtmi:ttnv3connectorclient:SeeeduinoLoRaWAN4cz;1",
         "MQTTAccessKey": "...",
         "DeviceIntegrationDefault": true,
         "DevicePageSize": 10
      }
   }.
...

The Device Template @Id can also be set using a dtdlmodelid attribute in a TTI end device settings so devices can be individually configured.

TTI Application EndDevice dtdlmodelid attribute usage

At startup the TTI Gateway enumerates through the devices in each application configured in the app.settings.json. The Azure Device Provisioning Service(DPS) is used to retrieve each device’s connection string and configure it in Azure IoT Central if required.

Azure IoT Central Device Group with no provisioned Devices
TTI Connector application connecting and provisioning EndDevices
Azure IoT Central devices mapped to an Azure IoT Central Template via the modelID

The ProvisioningRegistrationAdditionalData optional parameter of the DPS RegisterAsync method has a JSON property which is used to the specify the device ModelID.

using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
{
	ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create( 
		Constants.AzureDpsGlobalDeviceEndpoint,
		deviceProvisiongServiceSettings.IdScope,
		securityProvider,
		transport);

	DeviceRegistrationResult result;

	if (!string.IsNullOrEmpty(modelId))
	{
		ProvisioningRegistrationAdditionalData provisioningRegistrationAdditionalData = new ProvisioningRegistrationAdditionalData()
		{
			JsonData = $"{{\"modelId\": \"{modelId}\"}}"
		};

		result = await provClient.RegisterAsync(provisioningRegistrationAdditionalData, stoppingToken);
	}
	else
    {
		result = await provClient.RegisterAsync(stoppingToken);
	}

	if (result.Status != ProvisioningRegistrationStatusType.Assigned)
	{
		_logger.LogError("Config-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);

		return false;
	}

	IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

	deviceClient = DeviceClient.Create(result.AssignedHub, authentication, transportSettings);
}

My implementation was “inspired” by TemperatureController project in the PnP Device Samples.

Azure IoT Central Dashboard with Seeeduino LoRaWAN devices around my house that were “automagically” provisioned

I need to do some testing to confirm my code works reliably with both DPS and user provided connection strings. The RegisterAsync call is currently taking about four seconds which could be an issue for TTI applications with many devices.

TTI V3 Gateway Azure IoT Hub Support

Posted on by

After a couple of weeks work my The Things Industries(TTI) V3 gateway is in beta testing. For this blog post I have configured five Seeeduino LoRaWAN devices. My sensor nodes connect to an Azure IoT Hub with a Shared Access Signature(SAS) device policy connection string. I’m using Device Twin Explorer to display Telemetry from and send messages to the sensor nodes. I have also configured Azure Stream Analytics and PowerBI to graph telemetry from the sensor nodes.

Device Twin Explorer displaying telemetry from one of the Seeeduino devices

My integration uses only queued messages as often they won’t be delivered to the sensor node immediately, especially if the sensor node only sends an uplink message every 30 minutes/hour/day.

The confirmed flag should be used with care as the Azure IoT Hub messages may expire before a delivery Ack/Nack/Failed is received from the TTI.

PowerBI graph of temperature and humidity in my garage over 24 hours

To send a downlink message, TTI needs a LoRaWAN port number (plus optional queue, confirmed and priority values) which is specified in the Azure IoT Hub message custom properties.

Device explorer displaying a raw payload message which has been confirmed delivered
TTI device live data tab displaying raw payload in downlink message information tab
Azure IoT Connector console application sending raw payload to sensor node with confirmation ack
Arduino monitor displaying received raw payload from TTI

If the Azure IoT Hub message payload is valid JSON it is copied into the payload decoded downlink message property. and if it is not valid JSON it assumed to be a Base64 encoded value and copied into the payload raw downlink message property.

try
{
	// Split over multiple lines in an attempt to improve readability. A valid JSON string should start/end with {/} for an object or [/] for an array
	if (!(payloadText.StartsWith("{") && payloadText.EndsWith("}"))
										&&
		(!(payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
	{
		throw new JsonReaderException();
	}

	downlink.PayloadDecoded = JToken.Parse(payloadText);
}
catch (JsonReaderException)
{
	downlink.PayloadRaw = payloadText;
}

Like the Azure IoT Central JSON validation I had to add a check that the string started with a “{” and finished with a “}” (a JSON object) or started with a “[” and finished with a “]” (a JSON array) as part of the validation process.

Device explorer displaying a JSON payload message which has been confirmed delivered

I normally wouldn’t use exceptions for flow control but I can’t see a better way of doing this.

TTI device live data tab displaying JSON payload in downlink message information tab
Azure IoT Connector console application sending JSON payload to sensor node with confirmation ack
Arduino monitor displaying received JSON payload from TTI

The build in TTI decoder only supports downlink decoded payloads with property names “value_0” through “value_x” custom encoders may support other property names.

TTI V3 Gateway Azure IoT Central Support

Posted on by

After a couple of weeks work my The Things Industries(TTI) V3 gateway is in beta testing. For this blog post the client is a GHI Electronics Fezduino with a RAK811 LPWAN Evaluation Board(EVB). My test device was configured in Azure IoT Central by the Device Provisioning Service(DPS) and I then manually migrated the device to each of the four templates used in this post.

The first step was to display the temperature and barometric pressure values from the Seeedstudio Grove BMP180 attached to my sensor node.

Sensor node displaying temperature and barometric pressure values
Azure IoT Central temperature and barometric pressure telemetry configuration
Azure IoT Central Telemetry Dashboard displaying temperature and barometric pressure values

The next step was to configure a simple Azure IoT Central command to send to the sensor node. This was a queued request with no payload. An example of this sort of command would be a request for a sensor node to reboot or turn on an actuator.

My integration uses only offline queued commands as often messages won’t be delivered to the sensor node immediately, especially if the sensor node only sends a message every half hour/hour/day. The confirmed flag should be used with care as the Azure IoT Hub messages may expire before a delivery Ack/Nack/Failed is received from the TTI and it consumes downlink bandwidth.

if (message.Properties.ContainsKey("method-name"))
{
}

I determine an Azure IoT Hub message is an Azure IoT Central command by the presence of the “method-name” property. If the Azure IoT Central command does not have a request payload the Azure IoT Hub message payload will contain a single “@” character so the Azure IoT Connector sends a TTI downlink message with an empty raw payload via the TTI Data API(MQTT). 

if (payloadText.CompareTo("@") != 0)
{
   .
}
else
{
   downlink.PayloadRaw = "";
}
Azure IoT Central command with out a request payload value command configuration

To send a downlink message, TTI needs a LoRaWAN port number (plus optional queue, confirmed and priority values) which can’t be provided via the Azure IoT Central command setup so these values are configured in the app.settings file.

Each TTI application has zero or more Azure IoT Central command configurations which supply the port, confirmed, priority and queue settings. 

  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          ...
          }
        },
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
        }
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
...
   }
}
Azure IoT Central simple command dashboard
Azure IoT Central simple command initiation
Azure IoT TTI connector application sending a simple command to my sensor node
Sensor node display simple command information. The note message payload is empty

The next step was to configure a more complex Azure IoT Central command to send to the sensor node. This was a queued request with a single value payload. An example of this sort of command could be setting the speed of a fan or the maximum temperature of a freezer for an out of band (OOB) notification to be sent.

Azure IoT Central single value command configuration
  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          ...
          }
        },
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_0": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_1": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
        }
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
...
   }
}

The value_0 settings are for the minimum temperature the value_1 settings are for the maximum temperature value.

Azure IoT Central single value command initiation
Azure IoT TTI connector application sending a single value command to my sensor node
Sensor node displaying single value command information. There are two downlink messages and each payload contains a single value

The single value command payload contains the textual representation of the value e.g. “true”/”false” or “1.23” which are also valid JSON. This initially caused issues as I was trying to splice a single value into the decoded payload.

I had to add a check that the string started with a “{” and finished with a “}” (a JSON object) or started with a “[” and finished with a “]” (a JSON array) as part of the validation process.

For a single value command the payload decoded has a single property with the method-name value as the name and the payload as the value. For a command with a JSON payload the message payload is copied into the PayloadDecoded.

I normally wouldn’t use exceptions for flow control but I can’t see a better way of doing this.

	try
	{
		// Split over multiple lines to improve readability
		if (!(payloadText.StartsWith("{") && payloadText.EndsWith("}"))
									&&
			(!(payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
		{
			throw new JsonReaderException();
		}

		downlink.PayloadDecoded = JToken.Parse(payloadText);
	}
	catch (JsonReaderException)
	{
		try
		{
			JToken value = JToken.Parse(payloadText);

			downlink.PayloadDecoded = new JObject(new JProperty(methodName, value));
		}
		catch (JsonReaderException)
		{
			downlink.PayloadDecoded = new JObject(new JProperty(methodName, payloadText));
		}
	}

The final step was to configure an another Azure IoT Central command with a JSON payload to send to the sensor node. A “real-world” example of this sort of command would be setting the minimum and maximum temperatures of a freezer in a single downlink message.

Azure IoT Central JSON payload command setup
Azure IoT Central JSON payload command payload configuration
  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          ...
          }
        },
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_0": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_1": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "TemperatureOOBAlertMinimumAndMaximum": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          }
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
        }
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
...
   }
}
Azure IoT Central JSON payload command initiation

Azure IoT TTI connector application sending a JSON payload command to my sensor node
Sensor node displaying JSON command information. There is a single payload which contains a two values

The build in TTI decoder only supports downlink decoded payloads with property names “value_0” through “value_x” which results in some odd command names and JSON payload property names. (Custom encoders may support other property names). Case sensitivity of some configuration values also tripped me up.