Azure IoT Hub MQTT LoRa Field Gateway

Back in April I started working on an MQTT LoRa Field gateway which was going to support a selection of different Software as a service(SaaS) Internet of Things IoT) platforms.

I now have a working Azure IoT Hub plug-in (Azure IoT Central support as planned as well) with the first iteration focused on Device to Cloud (D2C) messaging. In a future iteration I will add Cloud to Device messaging(C2D).

My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)

I have a number of sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield etc. clients. These work with both my platform specific (Adafruit.IO, Azure IoT Hub/Central) gateways and protocol specific field gateways.

Azure IoT Hub Device Explorer Data Display

When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.

{
  "MQTTUserName": "YourIoTHubHub.azure-devices.net/MQTTLoRa915MHz/api-version=2018-06-30",
  "MQTTPassword": "SharedAccessSignature sr=YourIoTHubHub.azure-devices.net%2Fdevices%2FMQTTLoRa915MHz&sig=123456789012345678901234567890123456789012345%3D&se=1574673583",
  "MQTTClientID": "MQTTLoRa915MHz",
  "MQTTServer": "YourIoTHubHub.azure-devices.net",
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0,
  "MessageHandlerAssembly": "Mqtt.IoTCore.FieldGateway.LoRa.AzureIoTHub",
  "PlatformSpecificConfiguration": ""
}

The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel

MQTT LoRa Gateway with Azure IoT Hub plug-in

The message handler uploads all values in an inbound messages in one MQTT message.

namespace devMobile.Mqtt.IoTCore.FieldGateway
{
   using System;
   using System.Diagnostics;
   using System.Text;
   using Windows.Foundation.Diagnostics;

   using devMobile.IoT.Rfm9x;
   using MQTTnet;
   using MQTTnet.Client;
   using Newtonsoft.Json.Linq;
   using Newtonsoft.Json;

   public class MessageHandler : IMessageHandler
   {
      private LoggingChannel Logging { get; set; }
      private IMqttClient MqttClient { get; set; }
      private Rfm9XDevice Rfm9XDevice { get; set; }
      private string PlatformSpecificConfiguration { get; set; }

      void IMessageHandler.Initialise(LoggingChannel logging, IMqttClient mqttClient, Rfm9XDevice rfm9XDevice, string platformSpecificConfiguration)
      {
         LoggingFields processInitialiseLoggingFields = new LoggingFields();

         this.Logging = logging;
         this.MqttClient = mqttClient;
         this.Rfm9XDevice = rfm9XDevice;
         this.PlatformSpecificConfiguration = platformSpecificConfiguration;
      }

      async void IMessageHandler.Rfm9XOnReceive(Rfm9XDevice.OnDataReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();
         char[] sensorReadingSeparators = { ',' };
         char[] sensorIdAndValueSeparators = { ' ' };

         processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
         processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
         processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

         string addressBcdText = BitConverter.ToString(e.Address);
         processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
         processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

         string messageText;
         try
         {
            messageText = UTF8Encoding.UTF8.GetString(e.Data);
            processReceiveLoggingFields.AddString("MessageText", messageText);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         // Chop up the CSV text
         string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
         if (sensorReadings.Length < 1)
         {
            this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         JObject payloadJObject = new JObject();

         JObject feeds = new JObject();

         // Chop up each sensor read into an ID & value
         foreach (string sensorReading in sensorReadings)
         {
            string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);

            // Check that there is an id & value
            if (sensorIdAndValue.Length != 2)
            {
               this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
               return;
            }

            string sensorId = string.Concat(addressBcdText, sensorIdAndValue[0]);
            string value = sensorIdAndValue[1];

            feeds.Add(sensorId.ToLower(), value);
         }
         payloadJObject.Add("feeds", feeds);

         string topic = $"devices/{MqttClient.Options.ClientId}/messages/events/";

         try
         {
            var message = new MqttApplicationMessageBuilder()
               .WithTopic(topic)
               .WithPayload(JsonConvert.SerializeObject(payloadJObject))
               .WithAtLeastOnceQoS()
               .Build();
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
            await MqttClient.PublishAsync(message);
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);

            this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Information);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Error);
         }
      }

      void IMessageHandler.MqttApplicationMessageReceived(MqttApplicationMessageReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();

         processReceiveLoggingFields.AddString("ClientId", e.ClientId);
#if DEBUG
         processReceiveLoggingFields.AddString("Payload", e.ApplicationMessage.ConvertPayloadToString());
#endif
         processReceiveLoggingFields.AddString("QualityOfServiceLevel", e.ApplicationMessage.QualityOfServiceLevel.ToString());
         processReceiveLoggingFields.AddBoolean("Retain", e.ApplicationMessage.Retain);
         processReceiveLoggingFields.AddString("Topic", e.ApplicationMessage.Topic);

         this.Logging.LogEvent("MqttApplicationMessageReceived topic not processed", processReceiveLoggingFields, LoggingLevel.Error);
      }

      void IMessageHandler.Rfm9xOnTransmit(Rfm9XDevice.OnDataTransmitedEventArgs e)
      {
      }
   }
}

The formatting of the username and generation of password are password are a bit awkward and will be fixed in a future refactoring. Along with regenerating the SAS connection token just before it is due to expire.

ubidots MQTT LoRa Field Gateway

Back in April I started working on an MQTT LoRa Field gateway which was going to support a selection of different Software as a service(SaaS) Internet of Things(IoT) platforms.

After a long pause in development I have a working ubidots client and have 3 proof of concept (PoC) integrations for Adafruit.IO, AskSensors, and Losant. I am also working on Azure IoT Hub, Azure IoT Central and MyDevice Cayenne clients. The first iteration is focused on Device to Cloud (D2C) messaging in the next iteration I will add Cloud to Device where viable(C2D).

My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)

I have a number of sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield etc. clients. These work with both my platform specific (Adafruit.IO, Azure IoT Central) gateways and protocol specific field gateways.

Ubidots dashboard

When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.

{
  "MQTTUserName": "Ubidots generated usname here",
  "MQTTPassword": "NotVerySecure",
  "MQTTClientID": "MQTTLoRaGateway",
  "MQTTServer": "industrial.api.ubidots.com",
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0,
  "MessageHandlerAssembly": "Mqtt.IoTCore.FieldGateway.LoRa.Ubidots",
  "PlatformSpecificConfiguration": ""
}

The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel

MQTT LoRa Field Gateway with ubidots plugin generated telemetry
ubidots device management
ubidot managment

The message handler uploads all values in an inbound messages in one MQTT message using the ubidots MQTT message format

async void IMessageHandler.Rfm9XOnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e)
{
	LoggingFields processReceiveLoggingFields = new LoggingFields();
	JObject telemetryDataPoint = new JObject();
	char[] sensorReadingSeparators = { ',' };
	char[] sensorIdAndValueSeparators = { ' ' };

	processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
	processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
	processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

	string addressBcdText = BitConverter.ToString(e.Address);
	processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
	processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

	string messageText;
	try
	{
		messageText = UTF8Encoding.UTF8.GetString(e.Data);
		processReceiveLoggingFields.AddString("MessageText", messageText);
	}
	catch (Exception ex)
	{
		processReceiveLoggingFields.AddString("Exception", ex.ToString());
		this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
		return;
	}

	// Chop up the CSV text
	string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
	if (sensorReadings.Length < 1)
	{
		this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
		return;
	}

	// Chop up each sensor read into an ID & value
	foreach (string sensorReading in sensorReadings)
	{
		string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);
		// Check that there is an id & value
		if (sensorIdAndValue.Length != 2)
		{
			this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
			return;
		}

		string sensorId = sensorIdAndValue[0];
		string value = sensorIdAndValue[1];

		telemetryDataPoint.Add(addressBcdText + sensorId, Convert.ToDouble(value));
	}
	processReceiveLoggingFields.AddString("MQTTClientId", MqttClient.Options.ClientId);

	string stateTopic = string.Format(stateTopicFormat, MqttClient.Options.ClientId);

	try
	{
		var message = new MqttApplicationMessageBuilder()
			.WithTopic(stateTopic)
			.WithPayload(JsonConvert.SerializeObject(telemetryDataPoint))
			.WithAtLeastOnceQoS()
			.Build();
		Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
		await MqttClient.PublishAsync(message);
		Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);

		this.Logging.LogEvent("PublishAsync Ubidots payload", processReceiveLoggingFields, LoggingLevel.Information);
	}
	catch (Exception ex)
	{
		processReceiveLoggingFields.AddString("Exception", ex.ToString());
		this.Logging.LogEvent("PublishAsync Ubidots payload", processReceiveLoggingFields, LoggingLevel.Error);
	}
}

The “automagic” provisioning of feeds does make setting up small scale systems easier, though I’m not certain how well it would scale.

Some of the fields weren’t obviously editable e.g.”ÄPI Label” in device configuration which I only discovered by clicking on them..

The limitations of the free account meant I couldn’t evaluate ubidots in much depth but what was available appeared to be robust and reliable (Nov 2019).

Adafruit.IO MQTT LoRa Field Gateway

Back in April I started working on an MQTT LoRa Field gateway which was going to support a selection of different Software as a service(SaaS) Internet of Things(IoT) platforms.

After a long pause in development I have a working AdaFruit.IO client and have 3 proof of concept (PoC) integrations for AskSensors, Losant and Ubidots. I am also working on Azure IoT Hub, Azure IoT Central and MyDevice Cayenne clients. The first iteration is focused on Device to Cloud (D2C) messaging in the next iteration I will add Cloud to Device where viable(C2D).

My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)

I have a number of sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield etc. clients. These work with both my platform specific (Adafruit.IO, Azure IoT Central) gateways and protocol specific field gateways.

Maduino client dashboard

When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.

{
  "MQTTUserName": "AdaFruitIO user",
  "MQTTPassword": "AIO Key",
  "MQTTClientID": "MQTTLoRaGateway",
  "MQTTServer": "io.adafruit.com",
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0,
  "MessageHandlerAssembly": "Mqtt.IoTCore.FieldGateway.LoRa.Adafruit",
  "PlatformSpecificConfiguration": "mqttloragateway"
}

The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel

MQTT LoRa Gateway with Adafruit.IO plug-in

The SaaS platform specific interface has gained an additional parameter for platform specific configuration.

namespace devMobile.Mqtt.IoTCore.FieldGateway
{
	using System;
	using Windows.Foundation.Diagnostics;

	using devMobile.IoT.Rfm9x;
	using MQTTnet;
	using MQTTnet.Client;

	public interface IMessageHandler
	{
		void Initialise(LoggingChannel logging, IMqttClient mqttClient, Rfm9XDevice rfm9XDevice,string platformSpecificConfiguration);

		void Rfm9XOnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e);

		void MqttApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e);

		void Rfm9xOnTransmit(object sender, Rfm9XDevice.OnDataTransmitedEventArgs e);
	}
}

This is used for the AdaFruit.IO GroupName so Adafruit.IO feed values are not all in a single group.

public class MessageHandler : IMessageHandler
	{
		private LoggingChannel Logging { get; set; }
		private IMqttClient MqttClient { get; set; }
		private Rfm9XDevice Rfm9XDevice { get; set; }
      private string PlatformSpecificConfiguration { get; set; }


      void IMessageHandler.Initialise(LoggingChannel logging, IMqttClient mqttClient, Rfm9XDevice rfm9XDevice, string platformSpecificConfiguration)
		{
			LoggingFields processInitialiseLoggingFields = new LoggingFields();

			this.Logging = logging;
			this.MqttClient = mqttClient;
			this.Rfm9XDevice = rfm9XDevice;
			this.PlatformSpecificConfiguration = platformSpecificConfiguration;
		}

		async void IMessageHandler.Rfm9XOnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e)
		{
			LoggingFields processReceiveLoggingFields = new LoggingFields();

			processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
			processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
			processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

			string addressBcdText = BitConverter.ToString(e.Address);
			processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
			processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

			string payloadBcdText = BitConverter.ToString(e.Data);
			processReceiveLoggingFields.AddInt32("PayloadLength", e.Data.Length);
			processReceiveLoggingFields.AddString("DeviceAddressBCD", payloadBcdText);

			this.Logging.LogEvent("Rfm9XOnReceive", processReceiveLoggingFields, LoggingLevel.Information);
		}

		void IMessageHandler.MqttApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e)
		{
			LoggingFields processReceiveLoggingFields = new LoggingFields();

			processReceiveLoggingFields.AddString("ClientId", e.ClientId);
#if DEBUG
			processReceiveLoggingFields.AddString("Payload", e.ApplicationMessage.ConvertPayloadToString());
#endif
			processReceiveLoggingFields.AddString("QualityOfServiceLevel", e.ApplicationMessage.QualityOfServiceLevel.ToString());
			processReceiveLoggingFields.AddBoolean("Retain", e.ApplicationMessage.Retain);
			processReceiveLoggingFields.AddString("Topic", e.ApplicationMessage.Topic);

			this.Logging.LogEvent("MqttApplicationMessageReceived topic not processed", processReceiveLoggingFields, LoggingLevel.Error);
		}

		void IMessageHandler.Rfm9xOnTransmit(object sender, Rfm9XDevice.OnDataTransmitedEventArgs e)
		{
			this.Logging.LogMessage("Rfm9xOnTransmit", LoggingLevel.Information);
		}
	}
Adafruit.IO Group for a single field gateway
Group Setup

The message handler uploads all values in an inbound messages in one MQTT message using the AdaFruit.IO Group Feed format.

      async void IMessageHandler.Rfm9XOnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();
         char[] sensorReadingSeparators = { ',' };
         char[] sensorIdAndValueSeparators = { ' ' };

         processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
         processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
         processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

         string addressBcdText = BitConverter.ToString(e.Address);
         processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
         processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

         string messageText;
         try
         {
            messageText = UTF8Encoding.UTF8.GetString(e.Data);
            processReceiveLoggingFields.AddString("MessageText", messageText);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         // Chop up the CSV text
         string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
         if (sensorReadings.Length < 1)
         {
            this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         JObject payloadJObject = new JObject();

         JObject feeds = new JObject();

         // Chop up each sensor read into an ID & value
         foreach (string sensorReading in sensorReadings)
         {
            string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);

            // Check that there is an id & value
            if (sensorIdAndValue.Length != 2)
            {
               this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
               return;
            }

            string sensorId = string.Concat(addressBcdText, sensorIdAndValue[0]);
            string value = sensorIdAndValue[1];

            feeds.Add(sensorId.ToLower(), value);
         }
         payloadJObject.Add("feeds", feeds);

         string topic = $"{MqttClient.Options.Credentials.Username}/groups/{PlatformSpecificConfiguration}";

         try
         {
            var message = new MqttApplicationMessageBuilder()
               .WithTopic(topic)
               .WithPayload(JsonConvert.SerializeObject(payloadJObject))
               .WithAtLeastOnceQoS()
               .Build();
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
            await MqttClient.PublishAsync(message);
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);

            this.Logging.LogEvent("PublishAsync Adafruit payload", processReceiveLoggingFields, LoggingLevel.Information);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PublishAsync Adafruit payload", processReceiveLoggingFields, LoggingLevel.Error);
         }
      }

The casing of User names (Must match exactly) and Group/Feed names (must be lower case) tripped me up yet again. The “automagic” provisioning of feeds does make setting up small scale systems easier, though I’m not certain how well it would scale.

Re-reading the SX1231 datasheet

While trying to debug a deadlock in my RFM69 library I noticed in the logging that I was getting a receive interrupt while putting bytes in the FIFO for transmission.

Register 0x49 - Value 0X00 - Bits 00000000
Register 0x4a - Value 0X00 - Bits 00000000
Register 0x4b - Value 0X00 - Bits 00000000
Register 0x4c - Value 0X00 - Bits 00000000
Register 0x4d - Value 0X00 - Bits 00000000
...
22:58:47.192 Received To 0X22 a 33 byte message hello world RFM69-915-02 10-58-47 CRC Ok True
The thread 0x6a8 has exited with code 0 (0x0).
22:58:48.334 Send-hello world RFM69-915-01 10-58-48
22:58:48.351 Send-Done
22:58:48.388 Received To 0X22 a 33 byte message hello world RFM69-915-02 10-58-52 CRC Ok True
22:58:48.462 Transmit-Done
The thread 0xde4 has exited with code 0 (0x0).
22:58:53.427 Send-hello world RFM69-915-01 10-58-53
22:58:53.445 Send-Done
22:58:53.556 Transmit-Done
22:58:57.382 Received To 0X22 a 33 byte message hello world RFM69-915-02 10-58-57 CRC Ok True
The thread 0x17c has exited with code 0 (0x0).

After re-reading the RFM69CW/RFM69HCW module datasheet (based on the Semtech SX1231/SX1231H) I realised my code for loading the FIFO could be more efficient.

SX1231 Register map FIFO

Based on the timing diagram I could remove the loop which loads bytes in the FIFO for transmission.

The changes in my RFM69 library weren’t huge but I think they are also applicable to the RFM9X LoRa library as well

public void SendMessage(byte address, byte[] messageBytes)
{
#region Guard conditions
#endregion

	lock (Rfm9XRegFifoLock)
	{
		SetMode(RegOpModeMode.StandBy);

		if (PacketFormat == RegPacketConfig1PacketFormat.VariableLength)
		{
			RegisterManager.WriteByte((byte)Registers.RegFifo, (byte)(messageBytes.Length + 1)); // Additional byte for address 
		}

		RegisterManager.WriteByte((byte)Registers.RegFifo, address);

		foreach (byte b in messageBytes)
		{
				this.RegisterManager.WriteByte((byte)Registers.RegFifo, b);
		}

		SetMode(RegOpModeMode.Transmit);
	}
}

After modifications

public void SendMessage(byte address, byte[] messageBytes)
{
#region Guard conditions
...
#endregion

	SetDioPinMapping(dio0Mapping: Dio0Mapping.TransmitPacketSent);

	lock (Rfm9XRegFifoLock)
	{
		SetMode(RegOpModeMode.StandBy);

		if (PacketFormat == RegPacketConfig1PacketFormat.VariableLength)
		{
			RegisterManager.WriteByte((byte)Registers.RegFifo, (byte)(messageBytes.Length + 1)); // Additional byte for address 
		}

		RegisterManager.WriteByte((byte)Registers.RegFifo, address);

		this.RegisterManager.Write((byte)Registers.RegFifo, messageBytes);
	}
	SetMode(RegOpModeMode.Transmit);
}

After stress testing with several client devices this appears to have reduced the scope of a receive interrupt to occur while a packet was being loaded into the FIFO for transmission. I now need to dig deeper into the timing around entering different operational modes and selectively enabling event interrupts.

iwanders/plainRFM69 revisited

After problems with interleaved interrupt handling in my Windows 10 IoT Core client I figured the AutoMode used by the plainRFM69 library might be worth investigation. My first Arduino client was based on the plainRFM69 library but had Interoperability issues.

For this attempt I also started with the minimal sample and modified the code to send and receive text messages.

/*
    Copyright (c) 2014, Ivor Wanders, Bryn Lewis 2019
    MIT License, see the LICENSE.md file in the root folder.
*/

#include <SPI.h>
#include <plainRFM69.h>

// slave select pin.
#define SLAVE_SELECT_PIN 10

// connected to the reset pin of the RFM69.
#define RESET_PIN 9

// tie this pin down on the receiver.
#define SENDER_DETECT_PIN A0

const uint8_t tx_buffer[] = "ABCDEFGHIJKLMNOPQRSTURWXYZ1234567890";
//const uint8_t tx_buffer[] = "abcdefghijklmnopqrstuvwxyz1234567890";
uint8_t rx_buffer[sizeof(tx_buffer)] = "";

plainRFM69 rfm = plainRFM69(SLAVE_SELECT_PIN);


void sender() {

  uint32_t start_time = millis();

  uint32_t counter = 1; // the counter which we are going to send.

  while (true) {
    rfm.poll(); // run poll as often as possible.

    if (!rfm.canSend()) {
      continue; // sending is not possible, already sending.
    }

    if ((millis() - start_time) > 1000) { // every 500 ms.
      start_time = millis();

      // be a little bit verbose.
      Serial.print("Send:"); Serial.println(counter);

      // send the number of bytes equal to that set with setPacketLength.
      // read those bytes from memory where counter starts.
      rfm.sendVariable(tx_buffer, counter);

      counter++; // increase the counter.

      if ( counter > strlen(tx_buffer))
      {
        counter = 1;
      }
    }
  }
}

void receiver() {
  uint32_t counter = 0; // to count the messages.

  while (true) {

    rfm.poll(); // poll as often as possible.

    while (rfm.available())
    {
      uint8_t len = rfm.read(rx_buffer); // read the packet into the new_counter.

      // print verbose output.
      Serial.print("Packet Len:");
      Serial.print( len );
      Serial.print(" : ");
      Serial.println((char*)rx_buffer);
    }
  }
}

void setup() {
  Serial.begin(9600);
  SPI.begin();

  bareRFM69::reset(RESET_PIN); // sent the RFM69 a hard-reset.

  //rfm.setRecommended(); // set recommended paramters in RFM69.
  rfm.setPacketType(true, false); // set the used packet type.

  rfm.setBufferSize(2);   // set the internal buffer size.
  rfm.setPacketLength(sizeof(rx_buffer)); // set the packet length.

  rfm.setFrequency((uint32_t)909560000); // set the frequency.

  rfm.setLNA(RFM69_LNA_IMP_200OHM, RFM69_LNA_GAIN_AGC_LOOP);

  // p71, 3 preamble bytes.
  rfm.setPreambleSize(16);

  // p71, 4 bytes sync of 0x01, only start listening when sync is matched.
  //uint8_t syncthing[] = {0xaa, 0x2d, 0xd4};
  uint8_t syncthing[] = {0xd4, 0x2d, 0xaa};
  rfm.setSyncConfig(true, false, sizeof(syncthing), 0);
  rfm.setSyncValue(&syncthing, sizeof(syncthing));

  rfm.dumpRegisters(Serial);

  // baudrate is default, 4800 bps now.

  rfm.receive();
  // set it to receiving mode.

  pinMode(SENDER_DETECT_PIN, INPUT_PULLUP);
  delay(5);
}

void loop() {
  if (digitalRead(SENDER_DETECT_PIN) == LOW) {
    Serial.println("Going Receiver!");
    receiver();
    // this function never returns and contains an infinite loop.
  } else {
    Serial.println("Going sender!");
    sender();
    // idem.
  }
}

I took the list register values and loaded them into a Excel spreadsheet alongside the values from my Windows 10 IoT Core application

17:35:03.044 -> 0x0: 0x0
17:35:03.078 -> 0x1: 0x4
17:35:03.078 -> 0x2: 0x0
17:35:03.078 -> 0x3: 0x1A
17:35:03.112 -> 0x4: 0xB
17:35:03.112 -> 0x5: 0x0
17:35:03.112 -> 0x6: 0x52
17:35:03.146 -> 0x7: 0xE3
17:35:03.146 -> 0x8: 0x63
17:35:03.146 -> 0x9: 0xD7
17:35:03.180 -> 0xA: 0x41
17:35:03.180 -> 0xB: 0x40
17:35:03.180 -> 0xC: 0x2
17:35:03.215 -> 0xD: 0x92
17:35:03.215 -> 0xE: 0xF5
17:35:03.249 -> 0xF: 0x20
17:35:03.249 -> 0x10: 0x24
17:35:03.249 -> 0x11: 0x9F
17:35:03.282 -> 0x12: 0x9
17:35:03.282 -> 0x13: 0x1A
17:35:03.282 -> 0x14: 0x40
17:35:03.317 -> 0x15: 0xB0
17:35:03.317 -> 0x16: 0x7B
17:35:03.317 -> 0x17: 0x9B
17:35:03.317 -> 0x18: 0x88
17:35:03.351 -> 0x19: 0x86
17:35:03.351 -> 0x1A: 0x8A
17:35:03.384 -> 0x1B: 0x40
17:35:03.384 -> 0x1C: 0x80
17:35:03.384 -> 0x1D: 0x6
17:35:03.418 -> 0x1E: 0x10
17:35:03.418 -> 0x1F: 0x0
17:35:03.452 -> 0x20: 0x0
17:35:03.452 -> 0x21: 0x0
17:35:03.452 -> 0x22: 0x0
17:35:03.487 -> 0x23: 0x2
17:35:03.487 -> 0x24: 0xFF
17:35:03.487 -> 0x25: 0x0
17:35:03.521 -> 0x26: 0x5
17:35:03.521 -> 0x27: 0x80
17:35:03.521 -> 0x28: 0x0
17:35:03.556 -> 0x29: 0xFF
17:35:03.556 -> 0x2A: 0x0
17:35:03.556 -> 0x2B: 0x0
17:35:03.556 -> 0x2C: 0x0
17:35:03.590 -> 0x2D: 0x10
17:35:03.590 -> 0x2E: 0x90
17:35:03.624 -> 0x2F: 0xAA
17:35:03.624 -> 0x30: 0x2D
17:35:03.624 -> 0x31: 0xD4
17:35:03.659 -> 0x32: 0x0
17:35:03.659 -> 0x33: 0x0
17:35:03.659 -> 0x34: 0x0
17:35:03.693 -> 0x35: 0x0
17:35:03.693 -> 0x36: 0x0
17:35:03.728 -> 0x37: 0xD0
17:35:03.728 -> 0x38: 0x25
17:35:03.728 -> 0x39: 0x0
17:35:03.761 -> 0x3A: 0x0
17:35:03.761 -> 0x3B: 0x0
17:35:03.761 -> 0x3C: 0x1
17:35:03.795 -> 0x3D: 0x0
17:35:03.795 -> Going sender!
17:35:04.725 -> Send:1

Arduino RFM69HCW Client in receive mode

First thing I noticed was the order of the three sync byes (Registers 0x2F, 0x30, 0x31) was reversed. I then modified the run method in the Windows 10 code so the registers settings on both devices matched. (I removed the PlainRFM69 SetRecommended call so as many of the default options as possible were used).

public void Run(IBackgroundTaskInstance taskInstance)
{
	byte[] syncValues = { 0xAA, 0x2D, 0xD4 };
	byte[] aesKeyValues = { 0x0, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0X0E, 0X0F };

	try
	{
		rfm69Device.Initialise(Rfm69HcwDevice.RegOpModeMode.StandBy
										,frequency: 909560000.0 
										,dio0Mapping: Rfm69HcwDevice.Dio0Mapping.ReceiveCrcOk
										,preambleSize: 16												
										,syncValues: syncValues
										,packetFormat: Rfm69HcwDevice.RegPacketConfig1PacketFormat.VariableLength
										,packetDcFree: Rfm69HcwDevice.RegPacketConfig1DcFree.Whitening
										,autoRestartRx: false
										//,addressNode: 0x22
										//,addressbroadcast: 0x99
										//,aesKey: aesKeyValues
										);

		rfm69Device.OnReceive += Rfm69Device_OnReceive;
		rfm69Device.OnTransmit += Rfm69Device_OnTransmit;

		rfm69Device.RegisterDump();
		rfm69Device.SetMode(Rfm69HcwDevice.RegOpModeMode.Receive);


		while (true)
		{
			if (true)
			{
				string message = $"hello world {Environment.MachineName} {DateTime.Now:hh-mm-ss}";

				byte[] messageBuffer = UTF8Encoding.UTF8.GetBytes(message);

				Debug.WriteLine("{0:HH:mm:ss.fff} Send-{1}", DateTime.Now, message);
				//rfm69Device.SendMessage( 0x11, messageBuffer);
				rfm69Device.SendMessage(messageBuffer);

				Debug.WriteLine("{0:HH:mm:ss.fff} Send-Done", DateTime.Now);

				Task.Delay(5000).Wait();
			}
			else
			{
				Debug.Write(".");
				Task.Delay(1000).Wait();
			}
		}
	}
	catch (Exception ex)
	{
		Debug.WriteLine(ex.Message);
	}
}

I also found an error with the declaration of the RegPacketConfig1DcFree enumeration (Whitening = 0b0100000 vs. Whitening = 0b01000000) which wouldn’t have helped.

public enum RegPacketConfig1DcFree : byte
{
	None = 0b00000000,
	Manchester = 0b00100000,
	Whitening = 0b01000000,
	Reserved = 0b01100000,
}
const RegPacketConfig1DcFree RegPacketConfig1DcFreeDefault = RegPacketConfig1DcFree.None;

I could then reliably sent messages to and receive messages from my Arduino Nano Radio Shield RFM69/95 device

Register 0x4c - Value 0X00 - Bits 00000000
Register 0x4d - Value 0X00 - Bits 00000000
...
17:55:53.559 Received 1 byte message A CRC Ok True
.17:55:54.441 Received 2 byte message AB CRC Ok True
.17:55:55.444 Received 3 byte message ABC CRC Ok True
.17:55:56.447 Received 4 byte message ABCD CRC Ok True
.17:55:57.449 Received 5 byte message ABCDE CRC Ok True
.17:55:58.453 Received 6 byte message ABCDEF CRC Ok True
The thread 0x578 has exited with code 0 (0x0).
.17:55:59.622 Received 7 byte message ABCDEFG CRC Ok True
.17:56:00.457 Received 8 byte message ABCDEFGH CRC Ok True
.17:56:01.460 Received 9 byte message ABCDEFGHI CRC Ok True
.17:56:02.463 Received 10 byte message ABCDEFGHIJ CRC Ok True
..17:56:03.955 Received 11 byte message ABCDEFGHIJK CRC Ok True
17:56:04.583 Received 12 byte message ABCDEFGHIJKL CRC Ok True

I did some investigation into that the plainRMF69 code and found the ReadMultiple and WriteMuliple methods reverse the byte order

void bareRFM69::writeMultiple(uint8_t reg, void* data, uint8_t len){
    SPI.beginTransaction(SPISettings(10000000, MSBFIRST, SPI_MODE0));  // gain control of SPI bus
    this->chipSelect(true); // assert chip select
    SPI.transfer(RFM69_WRITE_REG_MASK | (reg & RFM69_READ_REG_MASK)); 
    uint8_t* r = reinterpret_cast<uint8_t*>(data);
    for (uint8_t i=0; i < len ; i++){
        SPI.transfer(r[len - i - 1]);
    }
    this->chipSelect(false);// deassert chip select
    SPI.endTransaction();    // release the SPI bus
}

void bareRFM69::readMultiple(uint8_t reg, void* data, uint8_t len){
    SPI.beginTransaction(SPISettings(10000000, MSBFIRST, SPI_MODE0));  // gain control of SPI bus
    this->chipSelect(true); // assert chip select
    
    SPI.transfer((reg % RFM69_READ_REG_MASK));
    uint8_t* r = reinterpret_cast<uint8_t*>(data);
    for (uint8_t i=0; i < len ; i++){
        r[len - i - 1] = SPI.transfer(0);
    }
    this->chipSelect(false);// deassert chip select
    SPI.endTransaction();    // release the SPI bus
}

I won’t be able to use interrupt AutoMode clients with the EasySensors shields as the DIO2 pin is not connected but on the AdaFruit RFM69HCW Radio Bonnet 433MHz or 915MHz it is connected to GPIO24.

RFM69 hat library lockups and corruptions

While doing yet more stress testing I noticed a couple of odd message go past and a long pause every so often when sending a message in the Visual Studio output window.

I have two Arduino devices sending addressed messages every (both individual and broadcast) to the Adafruit RFM69 HCW Radio Bonnet, on my two Windows 10 IoT Core devices every 100mSec. At the same time the windows 10 devices are sending each other a message every 5 seconds.

To help spot the pauses I added some code to mark any events where there was a significant gap. In this case ” is ASCII character for 0x22 the device address

21:10:30.746 Received To 34 a 23 byte message Hello World ---0x22:236 CRC Ok True
21:10:30.918 Received To 153 a 23 byte message Hello World ---0x99:236 CRC Ok True
21:10:31.399 Received To 34 a 23 byte message Hello World ---0x22:237 CRC Ok True
21:10:31.568 Send-hello world RFM69-915-01 09-10-31
21:10:31.580 Send-Done
21:10:31.592 Received To 34 a 33 byte message """"""""""""""""""""""""""""""""" CRC Ok True
RC-------------------------------------------
21:10:32.052 Received To 34 a 23 byte message Hello World ---0x22:238 CRC Ok True
21:10:32.225 Received To 153 a 23 byte message Hello World ---0x99:238 CRC Ok True
21:10:32.705 Received To 34 a 23 byte message Hello World ---0x22:239 CRC Ok True

There were also still some corrupted messages

21:10:30.746 Received To 34 a 23 byte message Hello World ---0x22:236 CRC Ok True
21:10:30.918 Received To 153 a 23 byte message Hello World ---0x99:236 CRC Ok True
21:10:31.399 Received To 34 a 23 byte message Hello World ---0x22:237 CRC Ok True
21:10:31.568 Send-hello world RFM69-915-01 09-10-31
21:10:31.580 Send-Done
21:10:31.592 Received To 34 a 33 byte message """"""""""""""""""""""""""""""""" CRC Ok True
RC-------------------------------------------
21:10:32.052 Received To 34 a 23 byte message Hello World ---0x22:238 CRC Ok True
21:10:32.225 Received To 153 a 23 byte message Hello World ---0x99:238 CRC Ok True
21:10:32.705 Received To 34 a 23 byte message Hello World ---0x22:239 CRC Ok True

It looks like if the base station receives a message as it is about to send a message the Rfm69Device_OnTransmit never gets called.

It also looks like every so often the transmitter gets stuck on one of Windows 10 devices effectively jamming the frequency.

Transmit stuck on
16:12:10.193 Received To 34 a 22 byte message Hello World ---0x22:65 CRC Ok True
16:12:10.360 Received To 153 a 22 byte message Hello World ---0x99:65 CRC Ok True
16:12:10.831 Received To 34 a 22 byte message Hello World ---0x22:66 CRC Ok True
16:12:10.998 Received To 153 a 22 byte message Hello World ---0x99:66 CRC Ok True
The thread 0x570 has exited with code 0 (0x0).
16:12:11.484 Send-hello world RFM69-915-01 04-12-11
16:12:11.494 Received To 34 a 22 byte message Hello World ---0x22:67 CRC Ok True
16:12:11.504 Send-Done
The thread 0x3a8 has exited with code 0 (0x0).
16:12:16.554 Send-hello world RFM69-915-01 04-12-16
16:12:16.566 Send-Done
16:12:16.660 Transmit-Done
T--------------------------------------------
16:12:16.736 Received To 153 a 22 byte message Hello World ---0x99:75 CRC Ok True
16:12:17.206 Received To 34 a 22 byte message Hello World ---0x22:76 CRC Ok True
16:12:17.374 Received To 153 a 22 byte message Hello World ---0x99:76 CRC Ok True
16:12:18.011 Received To 153 a 22 byte message Hello World ---0x99:77 CRC Ok True


Transmit stuck 
16:12:07.591 Transmit-Done
16:12:07.880 Received To 153 a 23 byte message Hello World ---0x99:137 CRC Ok True
16:12:08.533 Received To 153 a 23 byte message Hello World ---0x99:138 CRC Ok True
16:12:08.839 Received To 17 a 24 byte message Hello World ----0x11:139 CRC Ok True
16:12:09.186 Received To 153 a 23 byte message Hello World ---0x99:139 CRC Ok True
16:12:09.493 Received To 17 a 24 byte message Hello World ----0x11:140 CRC Ok True
16:12:10.799 Received To 17 a 24 byte message Hello World ----0x11:142 CRC Ok True
The thread 0xc8 has exited with code 0 (0x0).
16:12:12.567 Send-hello world RFM69-915-02 04-12-12
16:12:12.589 Send-Done
16:12:12.681 Transmit-Done
16:12:16.510 Received To 17 a 33 byte message hello world RFM69-915-01 04-12-16 CRC Ok True
16:12:16.576 Received To 153 a 22 byte message Hello World ---0x99:75 CRC Ok True
16:12:17.025 Received To 153 a 23 byte message Hello World ---0x99:151 CRC Ok True
16:12:17.214 Received To 153 a 22 byte message Hello World ---0x99:76 CRC Ok True
16:12:17.331 Received To 17 a 24 byte message Hello World ----0x11:152 CRC Ok True
The thread 0xfa0 has exited with code 0 (0x0).
16:12:17.661 Send-hello world RFM69-915-02 04-12-17
16:12:17.680 Send-Done
16:12:17.772 Transmit-Done
16:12:17.851 Received To 153 a 22 byte message Hello World ---0x99:77 CRC Ok True
16:12:18.331 Received To 153 a 23 byte message Hello World ---0x99:153 CRC Ok True
16:12:18.489 Received To 153 a 22 byte message Hello World ---0x99:78 CRC Ok True
16:12:18.638 Received To 17 a 24 byte message Hello World ----0x11:154 CRC Ok True
16:12:18.985 Received To 153 a 23 byte message Hello World ---0x99:154 CRC Ok True
16:12:19.291 Received To 17 a 24 byte message Hello World ----0x11:155 CRC Ok True
16:12:19.638 Received To 153 a 23 byte message Hello World ---0x99:155 CRC Ok True
16:12:19.944 Received To 17 a 24 byte message Hello World ----0x11:156 CRC Ok True
16:12:20.291 Received To 153 a 23 byte message Hello World ---0x99:156 CRC Ok True
16:12:20.597 Received To 17 a 24 byte message Hello World ----0x11:157 CRC Ok True

Then as rfm69Device.SetMode(Rfm69HcwDevice.RegOpModeMode.Receive) hasn’t been called no messages are received until another message is sent.

It looks like a timing issue around access to the message fifo (I have that in a critical section) so I need todo some more debugging. Maybe purging the receive buffer

byte regPacketConfig2 = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegPacketConfig2);
			regPacketConfig2 |= 0b00000100;
			RegisterManager.WriteByte((byte)Rfm69HcwDevice.Registers.RegPacketConfig2, regPacketConfig2);

The adfruit.io RFM69 shield has DIO2 which can be used for automode operation which might remove some of the synchronisation issues I am encountering.

RFM69 hat library h WWWWWWWWoo

Again, while doing some stress testing I noticed an odd message go past in the Visual Studio output window. I had multiple devices sending addressed messages (both individual and broadcast) to the Adafruit RFM69 HCW Radio Bonnet, on my Windows 10 IoT Core device while it was sending a message every 5 seconds.

Received From 102 a 15 byte message Hello World:161
23:42:33.343 RegIrqFlags2 01100110
23:42:33.356 RegIrqFlags1 11011001
23:42:33.374 Address 0X99 10011001
Received From 153 a 15 byte message Hello World:106
23:42:33.761 RegIrqFlags2 01100110
23:42:33.774 RegIrqFlags1 11011001
23:42:33.791 Address 0X66 01100110
Received From 102 a 15 byte message Hello World:162
The thread 0xd20 has exited with code 0 (0x0).
23:42:34.500 RegIrqFlags2 01100110
23:42:34.501 Send-hello world 11:42:34 PM
23:42:34.520 RegIrqFlags1 11011001
23:42:34.545 Send-Done
23:42:34.551 Address 0X10 00010000
Received From 16 a 15 byte message h    WWWWWWWWoo
23:42:34.686 RegIrqFlags2 00001000
23:42:34.701 RegIrqFlags1 10110000
23:42:34.715 Transmit-Done
Transmit-Done
23:42:34.902 RegIrqFlags2 01100110
23:42:34.915 RegIrqFlags1 11011001
23:42:34.931 Address 0X66 01100110
Received From 102 a 15 byte message Hello World:163
23:42:35.626 RegIrqFlags2 01100110
23:42:35.640 RegIrqFlags1 11011001
23:42:35.659 Address 0X99 10011001
Received From 153 a 15 byte message Hello World:108
23:42:36.042 RegIrqFlags2 01100110
23:42:36.055 RegIrqFlags1 11011001
23:42:36.073 Address 0X66 01100110

The RegIrqFlags2 CrcOk (bit 1) was set and the message was corrupt.

RegIrqFlags2 bit flags from SX1231 datasheet

I have added code to check the CRC on inbound messages if this functionality is enabled. So the library can be used with CRCs disabled I have added a flag to the OnDataReceivedEventArgs class to indicate whether the CRC on the inbound message was OK.

private readonly Object Rfm9XRegFifoLock = new object();
...
private void ProcessPayloadReady(RegIrqFlags1 irqFlags1, RegIrqFlags2 irqFlags2)
{
	byte? address = null;
	byte numberOfBytes;
	byte[] messageBytes;

	lock (Rfm9XRegFifoLock)
	{
		// Read the length of the buffer if variable length packets
		if (PacketFormat == RegPacketConfig1PacketFormat.VariableLength)
		{
			numberOfBytes = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegFifo);
		}
		else
		{
			numberOfBytes = PayloadLength;
		}

		// Remove the address from start of the payload
		if (AddressingEnabled)
		{
			address = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegFifo);

			Debug.WriteLine("{0:HH:mm:ss.fff} Address 0X{1:X2} {2}", DateTime.Now, address, Convert.ToString((byte)address, 2).PadLeft(8, '0'));
			numberOfBytes--;
		}

		// Allocate a buffer for the payload and read characters from the Fifo
		messageBytes = new byte[numberOfBytes];

		for (int i = 0; i < numberOfBytes; i++)
		{
			messageBytes[i] = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegFifo);
		}
	}
...
public void SendMessage(byte[] messageBytes)
{
#region Guard conditions
#endregion

	lock (Rfm9XRegFifoLock)
	{
		SetMode(RegOpModeMode.StandBy);

		if (PacketFormat == RegPacketConfig1PacketFormat.VariableLength)
		{
			RegisterManager.WriteByte((byte)Registers.RegFifo, (byte)messageBytes.Length);
		}

		foreach (byte b in messageBytes)
		{
			this.RegisterManager.WriteByte((byte)Registers.RegFifo, b);
		}

		SetMode(RegOpModeMode.Transmit);
	}
}

I can most probably reduce the duration which I hold the lock for but that will require some more stress testing.

RFM69 hat library Hello Woooooooo

While doing some stress testing I noticed an odd message go past in the Visual Studio output window. I had multiple devices sending addressed messages (both individual and broadcast) to the Adafruit RFM69 HCW Radio Bonnet, on my Windows 10 IoT Core device while it was sending a message every 5 seconds.

Received From 153 a 13 byte message Hello World:7
18:43:56.544 RegIrqFlags2 01100110
18:43:56.558 RegIrqFlags1 11011001
18:43:56.575 Address 0X66 01100110
Received From 102 a 15 byte message Hello World:162
The thread 0x254 has exited with code 0 (0x0).
18:43:57.699 Send-hello world 6:43:57 PM
18:43:57.699 RegIrqFlags2 01100110
18:43:57.731 RegIrqFlags1 10000000
18:43:57.747 Address 0X66 01100110
18:43:57.765 Send-Done
Received From 102 a 15 byte message Hello Woooooooo
18:43:57.987 RegIrqFlags2 00001000
18:43:58.003 RegIrqFlags1 10110000
18:43:58.017 Transmit-Done
Transmit-Done
18:43:58.825 RegIrqFlags2 01100110
18:43:58.838 RegIrqFlags1 11011001
18:43:58.857 Address 0X66 01100110
Received From 102 a 15 byte message Hello World:164
18:43:59.966 RegIrqFlags2 01100110
18:43:59.979 RegIrqFlags1 11011001
18:43:59.998 Address 0X66 01100110

The odd thing was that the RegIrqFlags2 CrcOk (bit 1) was set but the message was still corrupt.

RegIrqFlags2 bit flags from SX1231 datasheet

After looking at the code I think the problem was the reading of the received message bytes from the device FIFO and the writing of bytes of message to be transmitted into the device FIFO overlapped. To stop this occurring again I have added code to synchronise access (using a Lock) to the FIFO.

private readonly Object Rfm9XRegFifoLock = new object();
...
private void ProcessPayloadReady(RegIrqFlags1 irqFlags1, RegIrqFlags2 irqFlags2)
{
	byte? address = null;
	byte numberOfBytes;
	byte[] messageBytes;

	lock (Rfm9XRegFifoLock)
	{
		// Read the length of the buffer if variable length packets
		if (PacketFormat == RegPacketConfig1PacketFormat.VariableLength)
		{
			numberOfBytes = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegFifo);
		}
		else
		{
			numberOfBytes = PayloadLength;
		}

		// Remove the address from start of the payload
		if (AddressingEnabled)
		{
			address = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegFifo);

			Debug.WriteLine("{0:HH:mm:ss.fff} Address 0X{1:X2} {2}", DateTime.Now, address, Convert.ToString((byte)address, 2).PadLeft(8, '0'));
			numberOfBytes--;
		}

		// Allocate a buffer for the payload and read characters from the Fifo
		messageBytes = new byte[numberOfBytes];

		for (int i = 0; i < numberOfBytes; i++)
		{
			messageBytes[i] = RegisterManager.ReadByte((byte)Rfm69HcwDevice.Registers.RegFifo);
		}
	}
...
public void SendMessage(byte[] messageBytes)
{
#region Guard conditions
#endregion

	lock (Rfm9XRegFifoLock)
	{
		SetMode(RegOpModeMode.StandBy);

		if (PacketFormat == RegPacketConfig1PacketFormat.VariableLength)
		{
			RegisterManager.WriteByte((byte)Registers.RegFifo, (byte)messageBytes.Length);
		}

		foreach (byte b in messageBytes)
		{
			this.RegisterManager.WriteByte((byte)Registers.RegFifo, b);
		}

		SetMode(RegOpModeMode.Transmit);
	}
}

The code has been running for a day without any corrupted messages so the lock appears to be working. I can most probably reduce the duration which I hold the lock for but that will require some more stress testing.

RFM69 hat library Part12E

Enums and Masks – RegDIOMapping1 & RegDIOMapping2

The RFM69CW/RFM69HCW module (based on the Semtech SX1231/SX1231H) has configurable digital outputs (RegDIOMapping1 & RegDIOMapping2) . Which I use to trigger interrupts on my Windows 10 IoT Core or Arduino devices. Currently (Sep 2019) the library only supports the mapping of the digital outputs D0 & D1 when the RFM69 is in Packet Mode.

RegiDIOMapping0 & RegDIOMapping2 settings for DIO thru DIO5

I added some additional constants and enumerations for the other settings configured in RegDioMapping1 & RegDioMapping2.

// RegDioMapping1 & RegDioMapping2 Packet Mode Table 22 pg48
// DIO 0 Bits 7&6 of RegDioMapping1
[Flags]
public enum Dio0Mapping
{
	// Sleep
	// Standby
	// Frequency Synthesis
	// Reserved 00-10
	FrequencySynthesisPllLock = 0b11000000,
	ReceiveCrcOk = 0b00000000,
	ReceivePayloadReady = 0b01000000,
	ReceiveSyncAddress = 0b10000000,
	ReceiveRssi = 0b11000000,
	TransmitPacketSent = 0b00000000,
	TransmitTxReady = 0b01000000,
	// Reserved 10
	PllLock = 0b11000000
}
const Dio0Mapping Dio0MappingDefault = 0x00;

// DIO 1 Bits 5&4 of RegDioMapping1
public enum Dio1Mapping
{
	SleepFifoLevel = 0b00000000,
	SleepFifoFull = 0b00010000,
	SleepFifoNotEmpty = 0b00100000,
	// Reserved 11
	StandByFifoLevel = 0b00000000,
	StandByFifoFull = 0b00010000,
	StandByFifoNotEmpty = 0b00100000,
	FrequencySynthesisFifoLevel = 0b00000000,
	FrequencySynthesisFifoFull = 0b00010000,
	FrequencySynthesisFifoNotEmpty = 0b00100000,
	FrequencySynthesisPllLock = 0b00110000,
	ReceiveFifoLevel = 0b00000000,
	ReceiveFifoFull = 0b00010000,
	ReceiveFifoNotEmpty = 0b00100000,
	ReceiveTimeout = 0b00110000,
	TransmitFifoLevel = 0b00000000,
	TransmitFifoFull = 0b00010000,
	TransmitFifoNotEmpty = 0b00100000,
	TransmitPllLock = 0b00110000,
}
const Dio1Mapping Dio1MappingDefault = 0x00;

// DIO 2 Bits 3&2 of RegDioMapping1
public enum Dio2Mapping
{
}
const Dio2Mapping Dio2MappingDefault = 0x00;

// DIO 2 Bits 1&0 of RegDioMapping1
public enum Dio3Mapping
{
}
const Dio3Mapping Dio3MappingDefault = 0x00;

// DIO 2 Bits 7&6 of RegDioMapping2
public enum Dio4Mapping
{
}
const Dio4Mapping Dio4MappingDefault = 0x00;

// DIO 2 Bits 5&4 of RegDioMapping2
public enum Dio5Mapping
{
}
const Dio5Mapping Dio5MappingDefault = 0x00;

// RegDioMapping2 Bits 2-0
public enum ClockOutDioMapping : byte
{
	FXOsc = 0b00000000,
	FXOscDiv2 = 0b00000001,
	FXOscDiv4 = 0b00000010,
	FXOscDiv8 = 0b00000011,
	FXOscDiv16 = 0b00000100,
	FXOscDiv32 = 0b00000101,
	RC = 0b00000110,
	Off = 0b00000111,
}
public const ClockOutDioMapping ClockOutDioMappingDefault = ClockOutDioMapping.Off;

I also added some code to the initialise method to set the RegDioMapping1 & RegDioMapping1 only if the values were not the defaults.

public void Initialise(RegOpModeMode modeAfterInitialise,
	bool sequencer = RegOpModeSequencerDefault,
	bool listen = RegOpModeListenDefault,
	BitRate bitRate = BitRateDefault,
	ushort frequencyDeviation = frequencyDeviationDefault,
	double frequency = FrequencyDefault,
	AfcLowBeta afcLowBeta = AfcLowBetaDefault,
	ListenModeIdleResolution listenModeIdleResolution = ListenModeIdleResolutionDefault, ListenModeRXTime listenModeRXTime = ListenModeRXTimeDefault, ListenModeCriteria listenModeCrieria = ListenModeCriteriaDefault, ListenModeEnd listenModeEnd = ListenModeEndDefault,
	byte listenCoefficientIdle = ListenCoefficientIdleDefault,
	byte listenCoefficientReceive = ListenCoefficientReceiveDefault,
	bool pa0On = pa0OnDefault, bool pa1On = pa1OnDefaut, bool pa2On = pa2OnDefault, byte outputpower = OutputpowerDefault,
	PaRamp paRamp = PaRampDefault,
	bool ocpOn = OcpOnDefault, byte ocpTrim = OcpTrimDefault,
	LnaZin lnaZin = LnaZinDefault, LnaCurrentGain lnaCurrentGain = LnaCurrentGainDefault, LnaGainSelect lnaGainSelect = LnaGainSelectDefault,
	byte dccFrequency = DccFrequencyDefault, RxBwMant rxBwMant = RxBwMantDefault, byte RxBwExp = RxBwExpDefault,
	byte dccFreqAfc = DccFreqAfcDefault, byte rxBwMantAfc = RxBwMantAfcDefault, byte bxBwExpAfc = RxBwExpAfcDefault,
	Dio0Mapping dio0Mapping = Dio0MappingDefault,
	Dio1Mapping dio1Mapping = Dio1MappingDefault,
	Dio2Mapping dio2Mapping = Dio2MappingDefault,
	Dio3Mapping dio3Mapping = Dio3MappingDefault,
	Dio4Mapping dio4Mapping = Dio4MappingDefault,
	Dio5Mapping dio5Mapping = Dio5MappingDefault,
	ClockOutDioMapping clockOutDioMapping = ClockOutDioMappingDefault,
	ushort preambleSize = PreambleSizeDefault,
	RegSyncConfigFifoFileCondition? syncFifoFileCondition = null, byte? syncTolerance = null, byte[] syncValues = null,
	RegPacketConfig1PacketFormat packetFormat = RegPacketConfig1PacketFormat.FixedLength,
	RegPacketConfig1DcFree packetDcFree = RegPacketConfig1DcFreeDefault,
	bool packetCrc = PacketCrcOnDefault,
	bool packetCrcAutoClear = PacketCrcAutoClearDefault,
	byte payloadLength = PayloadLengthDefault,
	byte? addressNode = null, byte? addressbroadcast = null,
	TxStartCondition txStartCondition = TxStartConditionDefault, byte fifoThreshold = FifoThresholdDefault,
	byte interPacketRxDelay = InterPacketRxDelayDefault, bool autoRestartRx = AutoRestartRxDefault,
	byte[] aesKey = null
	)
...
	// RegDioMapping1
	if ((dio0Mapping != Dio0MappingDefault) ||
	    (dio1Mapping != Dio1MappingDefault) ||
	    (dio2Mapping != Dio2MappingDefault) ||
	    (dio3Mapping != Dio3MappingDefault))
	{
		byte regDioMapping1Value = (byte)dio0Mapping;

		regDioMapping1Value |= (byte)dio1Mapping;
		regDioMapping1Value |= (byte)dio2Mapping;
		regDioMapping1Value |= (byte)dio3Mapping;

		RegisterManager.WriteByte((byte)Registers.RegDioMapping1, regDioMapping1Value);
	}

	// RegDioMapping2
	if ((dio4Mapping != Dio4MappingDefault) ||
		 (dio5Mapping != Dio5MappingDefault) ||
		 (clockOutDioMapping != ClockOutDioMappingDefault ))
	{
		byte regDioMapping2Value = (byte)dio4Mapping;

		regDioMapping2Value |= (byte)dio5Mapping;
		regDioMapping2Value |= (byte)clockOutDioMapping;

		RegisterManager.WriteByte((byte)Registers.RegDioMapping2, regDioMapping2Value);
	}

I had several failed attempts at defining suitable enumerations for configuring the RegDioMapping1 & RegDioMapping2 registers. I initially started with an enumeration for each Mode (Sleep, StandBy etc.) but the implementation was quite complex. The initial version only supports DIO0 & DIO1 as most of the shields I have, only DIO0 adn/or DIO1 are connected.

Windows 10 IoT Core Cognitive Services Azure IoT Hub Client

This application builds on Windows 10 IoT Core Cognitive Services Vision API client. It uses my Lego brick classifier model and a new m&m object detection model.

m&m counter test rig

I created a new Visual Studio 2017 Windows IoT Core project and copied across the Windows 10 IoT Core Cognitive Services Custom Vision API code, (changing the namespace and manifest details) and added the Azure Devices Client NuGet package.

Azure Devices Client NuGet

In the start up code I added code to initialise the Azure IoT Hub client, retrieve the device twin settings, and update the device twin properties.

try
{
	this.azureIoTHubClient = DeviceClient.CreateFromConnectionString(this.azureIoTHubConnectionString, this.transportType);
}
catch (Exception ex)
{
	this.logging.LogMessage("AzureIOT Hub DeviceClient.CreateFromConnectionString failed " + ex.Message, LoggingLevel.Error);
	return;
}

try
{
	TwinCollection reportedProperties = new TwinCollection();

	// This is from the OS
	reportedProperties["Timezone"] = TimeZoneSettings.CurrentTimeZoneDisplayName;
	reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
	reportedProperties["MachineName"] = Environment.MachineName;

	reportedProperties["ApplicationDisplayName"] = package.DisplayName;
	reportedProperties["ApplicationName"] = packageId.Name;
	reportedProperties["ApplicationVersion"] = string.Format($"{version.Major}.{version.Minor}.{version.Build}.{version.Revision}");

	// Unique identifier from the hardware
	SystemIdentificationInfo systemIdentificationInfo = SystemIdentification.GetSystemIdForPublisher();
	using (DataReader reader = DataReader.FromBuffer(systemIdentificationInfo.Id))
	{
		byte[] bytes = new byte[systemIdentificationInfo.Id.Length];
		reader.ReadBytes(bytes);
		reportedProperties["SystemId"] = BitConverter.ToString(bytes);
	}
	this.azureIoTHubClient.UpdateReportedPropertiesAsync(reportedProperties).Wait();
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure IoT Hub client UpdateReportedPropertiesAsync failed " + ex.Message, LoggingLevel.Error);
	return;
}

try
{
	LoggingFields configurationInformation = new LoggingFields();

	Twin deviceTwin = this.azureIoTHubClient.GetTwinAsync().GetAwaiter().GetResult();

	if (!deviceTwin.Properties.Desired.Contains("ImageUpdateDue") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdateDue"].value.ToString(), out imageUpdateDue))
	{
		this.logging.LogMessage("DeviceTwin.Properties ImageUpdateDue setting missing or invalid format", LoggingLevel.Warning);
		return;
	}
	configurationInformation.AddTimeSpan("ImageUpdateDue", imageUpdateDue);

	if (!deviceTwin.Properties.Desired.Contains("ImageUpdatePeriod") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdatePeriod"].value.ToString(), out imageUpdatePeriod))
	{
		this.logging.LogMessage("DeviceTwin.Properties ImageUpdatePeriod setting missing or invalid format", LoggingLevel.Warning);
		return;
	}
…
	if (!deviceTwin.Properties.Desired.Contains("DebounceTimeout") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["DebounceTimeout"].value.ToString(), out debounceTimeout))
	{
		this.logging.LogMessage("DeviceTwin.Properties DebounceTimeout setting missing or invalid format", LoggingLevel.Warning);
		return;
	}
				configurationInformation.AddTimeSpan("DebounceTimeout", debounceTimeout);

	this.logging.LogEvent("Configuration settings", configurationInformation);
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure IoT Hub client GetTwinAsync failed or property missing/invalid" + ex.Message, LoggingLevel.Error);
	return;
}

When the digital input (configured in the app.settings file) is strobed or the timer fires (configured in the device properties) an image is captured, uploaded to Azure Cognitive Services Custom Vision for processing.

The returned results are then post processed to make them Azure IoT Central friendly, and finally uploaded to an Azure IoT Hub.

For testing I have used a simple object detection model.

I trained the model with images of 6 different colours of m&m’s.

For my first dataset I tagged the location of a single m&m of each of the colour in 15 images.

Testing the training of the model

I then trained the model multiple times adding additional images where the model was having trouble distiguishing colours.

The published name comes from the training performance tab

Project settings

The projectID, AzureCognitiveServicesSubscriptionKey (PredictionKey) and PublishedName (From the Performance tab in project) are from the custom vision project properties.

All of the Custom Vision model settings are configured in the Azure IoT Hub device properties.

The app.settings file contains only the hardware configuration settings and the Azure IoT Hub connection string.

{
  "InterruptPinNumber": 24,
  "interruptTriggerOn": "RisingEdge",
  "DisplayPinNumber": 35,
  "AzureIoTHubConnectionString": "",
  "TransportType": "Mqtt"
} 

The LED connected to the display pin is illuminated while an image is being processed or briefly flashed if the insufficient time between image captures has passed.

The image data is post processed differently based on the model.

// Post process the predictions based on the type of model
switch (modelType)
{
	case ModelType.Classification:
		// Use only the tags above the specified minimum probability
		foreach (var prediction in imagePrediction.Predictions)
		{
			if (prediction.Probability >= probabilityThreshold)
			{
				// Display and log the individual tag probabilities
				Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Probability:0.00}");
				imageInformation.AddDouble($"Tag valid:{prediction.TagName}", prediction.Probability);
					telemetryDataPoint.Add(prediction.TagName, prediction.Probability);
			}
		}
		break;

	case ModelType.Detection:
		// Group the tags to get the count, include only the predictions above the specified minimum probability
		var groupedPredictions = from prediction in imagePrediction.Predictions
										 where prediction.Probability >= probabilityThreshold
										 group prediction by new { prediction.TagName }
				into newGroup
										 select new
										 {
											 TagName = newGroup.Key.TagName,
											 Count = newGroup.Count(),
										 };

		// Display and log the agregated predictions
		foreach (var prediction in groupedPredictions)
		{
			Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Count}");
			imageInformation.AddInt32($"Tag valid:{prediction.TagName}", prediction.Count);
			telemetryDataPoint.Add(prediction.TagName, prediction.Count);
		}
		break;
	default:
		throw new ArgumentException("ModelType Invalid");
}

For a classifier only the tags with a probability greater than or equal the specified threshold are uploaded.

For a detection model the instances of each tag are counted. Only the tags with a prediction value greater than the specified threshold are included in the count.

19-08-14 05:26:14 Timer triggered
Prediction count 33
 Tag:Blue 0.0146500813
 Tag:Blue 0.61186564
 Tag:Blue 0.0923164859
 Tag:Blue 0.7813785
 Tag:Brown 0.0100603029
 Tag:Brown 0.128318727
 Tag:Brown 0.0135991769
 Tag:Brown 0.687322736
 Tag:Brown 0.846672833
 Tag:Brown 0.1826635
 Tag:Brown 0.0183384717
 Tag:Green 0.0200069249
 Tag:Green 0.367765248
 Tag:Green 0.011428359
 Tag:Orange 0.678825438
 Tag:Orange 0.03718319
 Tag:Orange 0.8643157
 Tag:Orange 0.0296728313
 Tag:Red 0.02141669
 Tag:Red 0.7183208
 Tag:Red 0.0183610674
 Tag:Red 0.0130951973
 Tag:Red 0.82097
 Tag:Red 0.0618815944
 Tag:Red 0.0130757084
 Tag:Yellow 0.04150853
 Tag:Yellow 0.0106579047
 Tag:Yellow 0.0210028365
 Tag:Yellow 0.03392527
 Tag:Yellow 0.129197285
 Tag:Yellow 0.8089519
 Tag:Yellow 0.03723789
 Tag:Yellow 0.74729687
 Tag valid:Blue 2
 Tag valid:Brown 2
 Tag valid:Orange 2
 Tag valid:Red 2
 Tag valid:Yellow 2
 05:26:17 AzureIoTHubClient SendEventAsync start
 05:26:18 AzureIoTHubClient SendEventAsync finish

The debugging output of the application includes the different categories identified in the captured image.

I found my small model was pretty good at detection of individual m&m as long as the ambient lighting was consistent, and the background fairly plain.

Sample image from test rig

Every so often the camera contrast setting went bad and could only be restored by restarting the device which needs further investigation.

Image with contrast problem

This application could be the basis for projects which need to run an Azure Cognitive Services model to count or classify then upload the results to an Azure IoT Hub or Azure IoT Central for presentation.

With a suitable model this application could be used to count the number of people in a room, which could be displayed along with the ambient temperature, humidity, CO2, and noise levels in Azure IoT Central.

The code for this application is available In on GitHub.