Azure IoT Hub MQTT/AMQP oddness

This is a long post which covers some oddness I noticed when changing the protocol used by an Azure IoT Hub client from Message Queuing Telemetry Transport(MQTT) to Advanced Message Queuing Protocol (AMQP). I want to build a console application to test the pooling of AMQP connections so I started with an MQTT client written for another post.

class Program
{
   private static string payload;

   static async Task Main(string[] args)
   {
      string filename;
      string azureIoTHubconnectionString;
      DeviceClient azureIoTHubClient;

      if (args.Length != 2)
      {
         Console.WriteLine("[JOSN file] [AzureIoTHubConnectionString]");
         Console.WriteLine("Press <enter> to exit");
         Console.ReadLine();
         return;
      }

      filename = args[0];
      azureIoTHubconnectionString = args[1];

      try
      {
         payload = File.ReadAllText(filename);

         // Open up the connection
         azureIoTHubClient = DeviceClient.CreateFromConnectionString(azureIoTHubconnectionString, TransportType.Mqtt);
         //azureIoTHubClient = DeviceClient.CreateFromConnectionString(azureIoTHubconnectionString, TransportType.Mqtt_Tcp_Only);
         //azureIoTHubClient = DeviceClient.CreateFromConnectionString(azureIoTHubconnectionString, TransportType.Mqtt_WebSocket_Only);

         await azureIoTHubClient.OpenAsync();

         await azureIoTHubClient.SetMethodDefaultHandlerAsync(MethodCallbackDefault, null);

         Timer MessageSender = new Timer(TimerCallback, azureIoTHubClient, new TimeSpan(0, 0, 10), new TimeSpan(0, 0, 10));


         Console.WriteLine("Press <enter> to exit");
         Console.ReadLine();
      }
      catch (Exception ex)
      {
         Console.WriteLine(ex.Message);
         Console.WriteLine("Press <enter> to exit");
         Console.ReadLine();
      }
   }

   public static async void TimerCallback(object state)
   {
      DeviceClient azureIoTHubClient = (DeviceClient)state;

      try
      {
         // I know having the payload as a global is a bit nasty but this is a demo..
         using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(payload))))
         {
            Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync start", DateTime.UtcNow);
            await azureIoTHubClient.SendEventAsync(message);
            Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync finish", DateTime.UtcNow);
         }
      }
      catch (Exception ex)
      {
         Console.WriteLine(ex.Message);
      }
   }

   private static async Task<MethodResponse> MethodCallbackDefault(MethodRequest methodRequest, object userContext)
   {
      Console.WriteLine($"Default handler method {methodRequest.Name} was called.");

      return new MethodResponse(200);
   }
}

I configured an Azure IoT hub then used Azure IoT explorer to create a device and get the connections string for my application. After fixing up the application’s command line parameters I could see the timer code was successfully sending telemetry messages to my Azure IoT Hub. I also explored the different MQTT connections options TransportType.Mqtt, TransportType.Mqtt_Tcp_Only, and TransportType.Mqtt_WebSocket_Only which worked as expected.

MQTT Console application displaying sent telemetry
Azure IoT Hub displaying received telemetry

I could also initiate Direct Method calls to my console application from Azure IoT explorer.

Azure IoT Explorer initiating a Direct Method
MQTT console application displaying direct method call.

I then changed the protocol to AMQP

class Program
{
   private static string payload;

   static async Task Main(string[] args)
   {
      string filename;
      string azureIoTHubconnectionString;
      DeviceClient azureIoTHubClient;
      Timer MessageSender;

      if (args.Length != 2)
      {
         Console.WriteLine("[JOSN file] [AzureIoTHubConnectionString]");
         Console.WriteLine("Press <enter> to exit");
         Console.ReadLine();
         return;
      }

      filename = args[0];
      azureIoTHubconnectionString = args[1];

      try
      {
         payload = File.ReadAllText(filename);

         // Open up the connection
         azureIoTHubClient = DeviceClient.CreateFromConnectionString(azureIoTHubconnectionString, TransportType.Amqp);
         //azureIoTHubClient = DeviceClient.CreateFromConnectionString(azureIoTHubconnectionString, TransportType.Amqp_Tcp_Only);
         //azureIoTHubClient = DeviceClient.CreateFromConnectionString(azureIoTHubconnectionString, TransportType.Amqp_WebSocket_Only);

         await azureIoTHubClient.OpenAsync();

         await azureIoTHubClient.SetMethodDefaultHandlerAsync(MethodCallbackDefault, null);

         //MessageSender = new Timer(TimerCallbackAsync, azureIoTHubClient, new TimeSpan(0, 0, 10), new TimeSpan(0, 0, 10));
         MessageSender = new Timer(TimerCallbackSync, azureIoTHubClient, new TimeSpan(0, 0, 10), new TimeSpan(0, 0, 10));

#if MESSAGE_PUMP
         Console.WriteLine("Press any key to exit");
         while (!Console.KeyAvailable)
         {
            await Task.Delay(100);
         }
#else
         Console.WriteLine("Press <enter> to exit");
         Console.ReadLine();
#endif
      }
      catch (Exception ex)
      {
         Console.WriteLine(ex.Message);
         Console.WriteLine("Press <enter> to exit");
         Console.ReadLine();
      }
   }

   public static async void TimerCallbackAsync(object state)
   {
      DeviceClient azureIoTHubClient = (DeviceClient)state;

      try
      {
         // I know having the payload as a global is a bit nasty but this is a demo..
         using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(payload))))
         {
            Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync start", DateTime.UtcNow);
            await azureIoTHubClient.SendEventAsync(message);
            Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync finish", DateTime.UtcNow);
         }
      }
      catch (Exception ex)
      {
         Console.WriteLine(ex.Message);
      }
   }

   public static void TimerCallbackSync(object state)
   {
      DeviceClient azureIoTHubClient = (DeviceClient)state;

      try
      {
         // I know having the payload as a global is a bit nasty but this is a demo..
         using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(payload))))
         {
            Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync start", DateTime.UtcNow);
            azureIoTHubClient.SendEventAsync(message).GetAwaiter();
            Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync finish", DateTime.UtcNow);
         }
      }
      catch (Exception ex)
      {
         Console.WriteLine(ex.Message);
      }
   }


   private static async Task<MethodResponse> MethodCallbackDefault(MethodRequest methodRequest, object userContext)
   {
      Console.WriteLine($"Default handler method {methodRequest.Name} was called.");

      return new MethodResponse(200);
   }
}

In the first version of my console application I could see the SendEventAsync method was getting called but was not returning

AMQP Console application displaying sent telemetry failure

Even though the SendEventAsync call was not returning the telemetry messages were making it to my Azure IoT Hub.

Azure IoT Hub displaying AMQP telemetry

When I tried to initiate a Direct Method call from Azure IoT Explorer it failed after a while with a timeout.

Azure IoT Explorer initiating a Direct Method

The first successful approach I tried was to change the Console.Readline to a “message pump” (flashbacks to Win32 API programming).

Console.WriteLine("Press any key to exit");
while (!Console.KeyAvailable)
{
   await Task.Delay(100);
}

After some more experimentation I found that changing the timer method from asynchronous to synchronous also worked.

public static void TimerCallbackSync(object state)
{
   DeviceClient azureIoTHubClient = (DeviceClient)state;

   try
   {
      // I know having the payload as a global is a bit nasty but this is a demo..
      using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(payload))))
      {
         Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync start", DateTime.UtcNow);
         azureIoTHubClient.SendEventAsync(message).GetAwaiter();
         Console.WriteLine(" {0:HH:mm:ss} AzureIoTHubDeviceClient SendEventAsync finish", DateTime.UtcNow);
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine(ex.Message);
   }
}

I also had to change the method declaration and modify the SendEventAsync call to use a GetAwaiter.

AMQP Console application displaying sent telemetry
Azure IoT Hub displaying received telemetry
Azure IoT Explorer initiating a Direct Method
MQTT console application displaying direct method call.

It took a while to figure out enough about what was going on so I could do a search with the right keywords (DeviceClient AMQP async await SendEventAsync) to confirm my suspicion that MQTT and AMQP clients did behave differently.

For anyone who reads this post, I think this Github issue about task handling and blocking calls is most probably the answer (October 2020).

Cayenne Low Power Payload (LPP) Encoder

Reducing the size of message payloads is important for LoRa/LoRaWAN communications, as it reduces power consumption and bandwidth usage. One of the more common formats is myDevices Cayenne Low Power Payload(LPP) which is based on the IPSO Alliance Smart Objects Guidelines and is natively supported by The Things Network(TTN).

 private enum DataType : byte
{
   DigitalInput = 0, // 1 byte
   DigitialOutput = 1, // 1 byte
   AnalogInput = 2, // 2 bytes, 0.01 signed
   AnalogOutput = 3, // 2 bytes, 0.01 signed
   Luminosity = 101, // 2 bytes, 1 lux unsigned
   Presence = 102, // 1 byte, 1
   Temperature = 103, // 2 bytes, 0.1°C signed
   RelativeHumidity = 104, // 1 byte, 0.5% unsigned
   Accelerometer = 113, // 2 bytes per axis, 0.001G
   BarometricPressure = 115, // 2 bytes 0.1 hPa Unsigned
   Gyrometer = 134, // 2 bytes per axis, 0.01 °/s
   Gps = 136, // 3 byte lon/lat 0.0001 °, 3 bytes alt 0.01m
}

My implementation was “inspired” by the myDevices C/C++ sample code. The first step was to allocate a buffer to store the byte encoded values. I pre allocated the buffer to try and reduce the impacts of garbage collection. The code uses a manually incremented index into the buffer for performance reasons, plus the inconsistent support of System.Collections.Generic and Language Integrated Query(LINQ) on my three embedded platforms. The maximum length message that can be sent is limited by coding rate, duty cycle and bandwidth of the LoRa channel.

public Encoder(byte bufferSize)
{
   if ((bufferSize < BufferSizeMinimum) || ( bufferSize > BufferSizeMaximum))
   {
      throw new ArgumentException($"BufferSize must be between {BufferSizeMinimum} and {BufferSizeMaximum}", "bufferSize");
   }

   buffer = new byte[bufferSize];
}

For a simple data types like a digital input a single byte (True or False ) is used. The channel parameter is included so that multiple values of the same data type can be included in a message.

public void DigitalInputAdd(byte channel, bool value)
{
   if ((index + DigitalInputSize) > buffer.Length)
   {
     throw new ApplicationException("DigitalInputAdd insufficent buffer capacity");
   }

   buffer[index++] = channel;
   buffer[index++] = (byte)DataType.DigitalInput;
   // I know this is fugly but it works on all platforms
   if (value)
   {
      buffer[index++] = 1;
   }
   else
   {
      buffer[index++] = 0;
   }
}

For more complex data types like a Global Positioning System(GPS) location (Latitude, Longitude and Altitude) the values are converted to 32bit signed integers and only 3 of the 4 bytes are used.

public void GpsAdd(byte channel, float latitude, float longitude, float meters)
{
   if ((index + GpsSize) > buffer.Length)
   {
     throw new ApplicationException("GpsAdd insufficent buffer capacity");
   }

   int lat = (int)(latitude * 10000);
   int lon = (int)(longitude * 10000);
   int alt = (int)(meters * 100);

   buffer[index++] = channel;
   buffer[index++] = (byte)DataType.Gps;

   buffer[index++] = (byte)(lat >> 16);
   buffer[index++] = (byte)(lat >> 8);
   buffer[index++] = (byte)lat;
   buffer[index++] = (byte)(lon >> 16);
   buffer[index++] = (byte)(lon >> 8);
   buffer[index++] = (byte)lon;
   buffer[index++] = (byte)(alt >> 16);
   buffer[index++] = (byte)(alt >> 8);
   buffer[index++] = (byte)alt;
}
Azure IoT Central map position granularity

Before the message can be sent it needs to be converted to its Binary Coded Decimal(BCD) representation and all formatting characters removed.

public string Bcd()
{
   StringBuilder payloadBcd = new StringBuilder(BitConverter.ToString(buffer, 0, index));

   payloadBcd = payloadBcd.Replace("-", "");

   return payloadBcd.ToString();
}

TTN Device Data Display
Visual Studio 2019 Debug output

The implementation had to be revised a couple of times so It would work with desktop and GHI Electronics TinyCLRV2 powered devices. There maybe some modifications required as I port it to nanoFramework and Wilderness Labs Meadow devices.

The Things Network HTTP Azure IoT Integration Soak Testing

I wanted to do some testing to make sure the application would reliably process messages from 1000’s of devices…

The first thing I learnt was “don’t forget to restart your Azure Function after deleting all the devices from the Azure IoT Hub” as the DeviceClients are cached. Also make sure you delete the devices from both your Azure Device Provisioning service(DPS) and Azure IoT Hub instances.

Applications Insights provisioning event tracking

The next “learning” was that if you forget to enable “always on” the caching won’t work and your application will call the DPS way more often than expected.

Azure Application “always on configuration

The next “learning” was if your soak test sends 24000 messages it will start to fail just after you go out to get a coffee because of the 8000 msgs/day limit on the free version of IoT Hub.

Azure IoT Hub Free tier 8000 messages/day limit

After these “learnings” the application appeared to be working and every so often a message would briefly appear in Azure Storage Explorer queue view.

Azure storage explorer view of uplink messages queue

The console test application simulated 1000 devices sending 24 messages every so often and took roughly 8 hours to complete.

Message generator finished

In the Azure IoT Hub telemetry 24000 messages had been received after roughly 8 hours confirming the test rig was working as expected.

The notch was another “learning”, if you go and do some gardening then after roughly 40 minutes of inactivity your desktop PC will go into power save mode and the test client will stop sending messages.

The caching of settings appeared to be work as there were only a couple of requests to my Azure Key Vault where sensitive information like connection strings, symmetric keys etc. are stored.

Memory consumption did look to bad and topped out at roughly 120M.

In the application logging you can see the 1000 calls to DPS at the beginning (the yellow dependency events) then the regular processing of messages.

Application Insights logging

Even with the “learnings” the testing went pretty well overall. I do need to run the test rig for longer and with even more simulated devices.

I think this should do

48K Telemetry messages

If you get lots of errors in the logs “Host thresholds exceeded: [Connections]…. might need to bump your plan to something a bit larger

nRF24L01-TinyCLR V2 RC2 on Github

The source code of RC2 of my port GHI Electronics TinyCLR-0SV2RC1 nRF24L01 library is live on GitHub. The sample application now supports Fezduino (with embeddedcoolness.com or other Arduino shield), Fezportal and the SC2010 Dev board (with mikroe nrf24C Click, mikroe nRF24S Click or mikroenRF24T Click) .

Fezduino with Embedded Coolness shield
Fezportal with Mikroe nRF24 C Click
SC20100 Dev board

The application has gained four compile time configuration options

  • TINYCLR_V2_SC20100DEV_MIKROBUS_1
  • TINYCLR_V2_SC20100DEV_MIKROBUS_2
  • TINYCLR_V2_FEZDUINO
  • TINYCLR_V2_FEZPORTAL

These options configure the chip enable, chip selected and interrupt pins.

//---------------------------------------------------------------------------------
// Copyright (c) May 2020, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// Need one of TINYCLR_V2_SC20100DEV_MIKROBUS_1/TINYCLR_V2_SC20100DEV_MIKROBUS_2/TINYCLR_V2_FEZDUINO/TINYCLR_V2_FEZPORTAL defined
//---------------------------------------------------------------------------------
namespace devMobile.IoT.FieldGateway.TinyCLRV2nRF24Client
{
   using System;
   using System.Diagnostics;
   using System.Text;
   using System.Threading;

   using GHIElectronics.TinyCLR.Pins;

   using Radios.RF24;

   class Program
   {
      private const string BaseStationAddress = "Base1";
      private const string DeviceAddress = "Dev01";

      static void Main()
      {
         RF24 radio = new RF24();
         byte messageCount = System.Byte.MaxValue;

         try
         {
            radio.OnDataReceived += Radio_OnDataReceived;
            radio.OnTransmitFailed += Radio_OnTransmitFailed;
            radio.OnTransmitSuccess += Radio_OnTransmitSuccess;

#if TINYCLR_V2_SC20100DEV_MIKROBUS_1
            radio.Initialize(SC20100.SpiBus.Spi3, SC20100.GpioPin.PD4, SC20100.GpioPin.PD3, SC20100.GpioPin.PC5);
#endif
#if TINYCLR_V2_SC20100DEV_MIKROBUS_2
            radio.Initialize(SC20100.SpiBus.Spi3, SC20100.GpioPin.PD15, SC20100.GpioPin.PD14, SC20100.GpioPin.PA8);
#endif
#if TINYCLR_V2_FEZDUINO
            radio.Initialize(SC20100.SpiBus.Spi6, SC20100.GpioPin.PE11, SC20100.GpioPin.PC4, SC20100.GpioPin.PA1);
#endif
#if TINYCLR_V2_FEZPORTAL
            radio.Initialize(SC20100.SpiBus.Spi3, SC20100.GpioPin.PD4, SC20100.GpioPin.PC13, SC20100.GpioPin.PC2);
#endif
            radio.Address = Encoding.UTF8.GetBytes(DeviceAddress);

            radio.Channel = 15;
            radio.PowerLevel = PowerLevel.Minimum;
            radio.DataRate = DataRate.DR250Kbps;
            radio.IsEnabled = true;

            radio.IsAutoAcknowledge = true;
            radio.IsDyanmicAcknowledge = false;
            radio.IsDynamicPayload = true;

            Debug.WriteLine($"Address: {Encoding.UTF8.GetString(radio.Address)}");
            Debug.WriteLine($"PowerLevel: {radio.PowerLevel}");
            Debug.WriteLine($"IsAutoAcknowledge: {radio.IsAutoAcknowledge}");
            Debug.WriteLine($"Channel: {radio.Channel}");
            Debug.WriteLine($"DataRate: {radio.DataRate}");
            Debug.WriteLine($"IsDynamicAcknowledge: {radio.IsDyanmicAcknowledge}");
            Debug.WriteLine($"IsDynamicPayload: {radio.IsDynamicPayload}");
            Debug.WriteLine($"IsEnabled: {radio.IsEnabled}");
            Debug.WriteLine($"Frequency: {radio.Frequency}");
            Debug.WriteLine($"IsInitialized: {radio.IsInitialized}");
            Debug.WriteLine($"IsPowered: {radio.IsPowered}");

            while (true)
            {
               string payload = $"hello {messageCount}";
               messageCount -= 1;

               Debug.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-TX {payload.Length} byte message {payload}");
               radio.SendTo(Encoding.UTF8.GetBytes(BaseStationAddress), Encoding.UTF8.GetBytes(payload));

               Thread.Sleep(30000);
            }
         }
         catch (Exception ex)
         {
            Debug.WriteLine(ex.Message);
         }
      }

      private static void Radio_OnDataReceived(byte[] data)
      {
         // display as hex
         Debug.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-RX Hex Length {data.Length} Payload {BitConverter.ToString(data)}");

         // Display as Unicode
         string unicodeText = Encoding.UTF8.GetString(data);
         Debug.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-RX Unicode Length {unicodeText.Length} Unicode text {unicodeText}");
      }

      private static void Radio_OnTransmitSuccess()
      {
         Debug.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-TX Succeeded!");
      }

      private static void Radio_OnTransmitFailed()
      {
         Debug.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-TX failed!");
      }
   }
}

RAK811 LPWAN EVB Part3

Invalidating the warranty…

I wanted the RAK811 LPWAN Evaluation Board(EVB) -AS923 to work with selection of my Arduino and nanoFramework devices. The first decision was which of the hardware serial port (D0,D1) or the software serial port (D10,D11) should be connected to P1?

To use the EVB with my STM32F691DISCOVERY board running the nanoFramework (COM5 on the hardware serial port pins D0,D1) I removed R17&R19.

After some tinkering, I found that R8 which is connected to the RAK811 module reset had to be cut as well for the shield to work on my Arduino Uno R3 and STM32F691DISCOVERY devices.

RAK811 EVB with R17,R19 & R8 cut

I can still run the Arduino Uno R3 and RAK811 EVB in the original configuration with a couple of jumper leads

RAK811 on Arduino with Serial connected to D10,D1 a SoftwareSerial port

For devices where I needed D10,D11 for a  Serial Peripheral Interface(SPI) I could use an FTDI board and a couple of other pins (in this case D2,D3) for serial logging.

RAK811 on Arduino with Serial connected to D2,D2 a SoftwareSerial port

After debugging some code I also replaced the small jumpers on P1 with a couple of jumper leads so it was less fiddly to swap from downloading to debugging.

RAK811 LPWAN EVB Part2

How can I use this…

Just over a week ago I purchased a RAK811 LPWAN Evaluation Board -AS923 and now I want to trial it with selection of devices and configurations.

Initially I didn’t want to modify the shield by removing resistors as I only have one, and I’m not certain what device(s) it will be used with. The initial hardware configuration required jumpers for the serial port, ground and 5V power.

Arduino Uno R3 and RAK811 LPWAN Evaluation board 5V config

After looking at the schematic it should be possible to use the shield with a 3v3 device.

RAK 811 EVB schematic pg1
RAK 811 EVB schematic pg2

I confirmed this with a Seeeduino V4.2 devices set to 3v3, by putting a jumper on J1 and shifting the jumper wire from the 5V to the 3V3 pin.

Seeeduino V4 and RAK811 LPWAN Evaluation board 3V3 config

The next step was to see how I could get the RAK shield working on other devices without jumpers. On Arduino Uno R3 devices D0&D1 are the hardware(HW) serial port which are used for uploading sketches, and diagnostic logging.

The shield also connects the module serial port to D0&D1 to D10&D11, so by removing R17&R19 the shield should work on a device This would also allow the use of the Serial Peripheral Interface(SPI) port for other applications.

Using the HW Serial port but without any logging.

Unplugging the jumpers to upload was painful but the lack of logging made it really hard to debug my code.

To get around this I configured a SoftwareSerial port on D2&D3 for logging.

/********************************************************
 * This demo is only supported after RUI firmware version 3.0.0.13.X on RAK811
 * Master Board Uart Receive buffer size at least 128 bytes. 
 ********************************************************/
//#define SERIAL_BUFFER_SIZE 128
//#define SERIAL_TX_BUFFER_SIZE 64
//#define SERIAL_RX_BUFFER_SIZE 128
//#define _SS_MAX_RX_BUFF 128
#include "RAK811.h"
#include "SoftwareSerial.h"
#define WORK_MODE LoRaWAN   //  LoRaWAN or LoRaP2P
#define JOIN_MODE OTAA    //  OTAA or ABP
#if JOIN_MODE == OTAA
String DevEui = "..."; // From TTN
String AppEui = "...";
String AppKey = "...";
#else JOIN_MODE == ABP
String NwkSKey = "...";
String AppSKey = "...";
String DevAddr = "...";
#endif

#define TXpin 3   // Set the virtual serial port pins
#define RXpin 2

SoftwareSerial DebugSerial(RXpin,TXpin); // Declare a virtual serial port for debugging
#define ATSerial Serial

char buffer[]= "48656C6C6F20776F726C6435";

bool InitLoRaWAN(void);
RAK811 RAKLoRa(ATSerial,DebugSerial);

void setup() {
  DebugSerial.begin(19200);
  DebugSerial.println(F("Starting"));
  while(DebugSerial.available())
  {
    DebugSerial.read(); 
  }
  
  ATSerial.begin(9600); //set ATSerial baudrate:This baud rate has to be consistent with  the baud rate of the WisNode device.
  while(ATSerial.available())
  {
    ATSerial.read(); 
  }

  if(!RAKLoRa.rk_setWorkingMode(0))  //set WisNode work_mode to LoRaWAN.
  {
    DebugSerial.println(F("set work_mode failed, please reset module."));
    while(1);
  }
  
  RAKLoRa.rk_getVersion();  //get RAK811 firmware version
  DebugSerial.println(RAKLoRa.rk_recvData());  //print version number

  DebugSerial.println(F("Start init RAK811 parameters..."));
 
  if (!InitLoRaWAN())  //init LoRaWAN
  {
    DebugSerial.println(F("Init error,please reset module.")); 
    while(1);
  }

  DebugSerial.println(F("Start to join LoRaWAN..."));
  while(!RAKLoRa.rk_joinLoRaNetwork(60))  //Joining LoRaNetwork timeout 60s
  {
    DebugSerial.println();
    DebugSerial.println(F("Rejoin again after 5s..."));
    delay(5000);
  }
  DebugSerial.println(F("Join LoRaWAN success"));

  if(!RAKLoRa.rk_isConfirm(0))  //set LoRa data send package type:0->unconfirm, 1->confirm
  {
    DebugSerial.println(F("LoRa data send package set error,please reset module.")); 
    while(1);    
  }
}

bool InitLoRaWAN(void)
{
  if(RAKLoRa.rk_setJoinMode(JOIN_MODE))  //set join_mode:OTAA
  {
    if(RAKLoRa.rk_setRegion(0))  //set region EU868
    {
      if (RAKLoRa.rk_initOTAA(DevEui, AppEui, AppKey))
      {
        DebugSerial.println(F("RAK811 init OK!"));  
        return true;    
      }
    }
  }
  return false;
}

void loop() 
{
  DebugSerial.println(F("Start send data..."));
  if (RAKLoRa.rk_sendData(1, buffer))
  {    
    //for (unsigned long start = millis(); millis() - start < 300000L;)
    for (unsigned long start = millis(); millis() - start < 10000L;)
    {
      String ret = RAKLoRa.rk_recvData();
      if(ret != NULL)
      { 
        DebugSerial.println("ret != NULL");
        DebugSerial.println(ret);
      }
      if((ret.indexOf("OK")>0)||(ret.indexOf("ERROR")>0))
      {
        DebugSerial.println(F("Go to Sleep."));
        RAKLoRa.rk_sleep(1);  //Set RAK811 enter sleep mode
        delay(10000);  //delay 10s
        RAKLoRa.rk_sleep(0);  //Wakeup RAK811 from sleep mode
        break;
      }
    }
  }
}

I used an FTDI module I had lying around to connect the diagnostic logging serial port on the test rig to my development box.

Using the HW Serial port but with logging.

Now I only had to unplug the jumpers for D0&D1 and change ports in the Arduino IDE. One port for debugging the other for downloading.

Depending on the application I may remove R8 so I can manually reset the shield.

RAK811 LPWAN EVB Part1

I followed the instructions…

Just over a week ago I purchased some gear from RAK Wireless, the shipping was reasonable, it arrived promptly, and was well packaged. I had ordered

I figured a good first project would be to get the evaluation board going with one of my older Arduino Uno R3 devices following the Interfacing your RAK811 LPWAN Evaluation Board with Arduino Boards instructions.

The evaluation board was in its own box along with a USB cable, some spare PCB jumpers, some jumper leads and an antenna labeled with the frequency band which was thoughtful.

Arduino Uno R3 and RAK811 LPWAN Evaluation board 5V config

I downloaded the specified library from the RAK Wireless Github repository extracted the contents and copied the V1.3 directory into the libraries folder of my Arduino IDE install.

I updated the module software to the latest using the tools provided in the github download and checked this with the RAK Serial Port tool over the Universal Serial Bus(USB) connection (make sure the jumpers next to the antenna connection are set correctly)

Version number check with RAK Serial Port tool

I created a new project based on JoinNetworkOTAA (Over the Air Activation) example.

/********************************************************
 * This demo is only supported after RUI firmware version 3.0.0.13.X on RAK811
 * Master Board Uart Receive buffer size at least 128 bytes. 
 ********************************************************/

#include "RAK811.h"
#include "SoftwareSerial.h"
#define WORK_MODE LoRaWAN   //  LoRaWAN or LoRaP2P
#define JOIN_MODE OTAA    //  OTAA or ABP
#if JOIN_MODE == OTAA
String DevEui = "8680000000000001";
String AppEui = "70B3D57ED00285A7";
String AppKey = "DDDFB1023885FBFF74D3A55202EDF2B1";
#else JOIN_MODE == ABP
String NwkSKey = "69AF20AEA26C01B243945A28C9172B42";
String AppSKey = "841986913ACD00BBC2BE2479D70F3228";
String DevAddr = "260125D7";
#endif
#define TXpin 11   // Set the virtual serial port pins
#define RXpin 10
#define DebugSerial Serial
SoftwareSerial ATSerial(RXpin,TXpin);    // Declare a virtual serial port
char buffer[]= "72616B776972656C657373";

bool InitLoRaWAN(void);
RAK811 RAKLoRa(ATSerial,DebugSerial);


void setup() {
  DebugSerial.begin(115200);
  while(DebugSerial.available())
  {
    DebugSerial.read(); 
  }
  
  ATSerial.begin(9600); //set ATSerial baudrate:This baud rate has to be consistent with  the baud rate of the WisNode device.
  while(ATSerial.available())
  {
    ATSerial.read(); 
  }

  if(!RAKLoRa.rk_setWorkingMode(0))  //set WisNode work_mode to LoRaWAN.
  {
    DebugSerial.println(F("set work_mode failed, please reset module."));
    while(1);
  }
  
  RAKLoRa.rk_getVersion();  //get RAK811 firmware version
  DebugSerial.println(RAKLoRa.rk_recvData());  //print version number

  DebugSerial.println(F("Start init RAK811 parameters..."));
 
  if (!InitLoRaWAN())  //init LoRaWAN
  {
    DebugSerial.println(F("Init error,please reset module.")); 
    while(1);
  }

  DebugSerial.println(F("Start to join LoRaWAN..."));
  while(!RAKLoRa.rk_joinLoRaNetwork(60))  //Joining LoRaNetwork timeout 60s
  {
    DebugSerial.println();
    DebugSerial.println(F("Rejoin again after 5s..."));
    delay(5000);
  }
  DebugSerial.println(F("Join LoRaWAN success"));

  if(!RAKLoRa.rk_isConfirm(0))  //set LoRa data send package type:0->unconfirm, 1->confirm
  {
    DebugSerial.println(F("LoRa data send package set error,please reset module.")); 
    while(1);    
  }
}

bool InitLoRaWAN(void)
{
  if(RAKLoRa.rk_setJoinMode(JOIN_MODE))  //set join_mode:OTAA
  {
    if(RAKLoRa.rk_setRegion(0))  //set region EU868
    {
      if (RAKLoRa.rk_initOTAA(DevEui, AppEui, AppKey))
      {
        DebugSerial.println(F("RAK811 init OK!"));  
        return true;    
      }
    }
  }
  return false;
}

void loop() {
  DebugSerial.println(F("Start send data..."));
  if (RAKLoRa.rk_sendData(1, buffer))
  {    
    for (unsigned long start = millis(); millis() - start < 90000L;)
    {
      String ret = RAKLoRa.rk_recvData();
      if(ret != NULL)
      { 
        DebugSerial.println(ret);
      }
      if((ret.indexOf("OK")>0)||(ret.indexOf("ERROR")>0))
      {
        DebugSerial.println(F("Go to Sleep."));
        RAKLoRa.rk_sleep(1);  //Set RAK811 enter sleep mode
        delay(10000);  //delay 10s
        RAKLoRa.rk_sleep(0);  //Wakeup RAK811 from sleep mode
        break;
      }
    }
  }
}

I had to look at the library code to work out the value the rk_setRegion call needed for the AS932 band used in my region

bool RAK811::rk_setRegion(int region)
{
  if (region > 9)
  {
    _serial1.println(F("Parameter error"));
    return false;
  }
  String REGION;
  switch (region)
  {
    case 0:REGION="AS923";
      break;
    case 1:REGION="AU915";
      break;
    case 2:REGION="CN470";
      break;
    case 3:REGION="CN779";
      break;
    case 4:REGION="EU433";
      break;
    case 5:REGION="EU868";
      break;
    case 6:REGION="KR920";
      break;
    case 7:REGION="IN865";
      break;
    case 8:REGION="US915";
      break;
    case 9:REGION="US915_Hybrid";
      break;
  }
  _serial1.println("Current work region: "+REGION);
  sendRawCommand("at+set_config=lora:region:" + REGION);
  ret = rk_recvData();
#if defined DEBUG_MODE
  _serial1.println(ret);
#endif
  if (ret.indexOf("OK") >= 0)
  {
    return true;
  }
  else
  {
    return false;
  }
}

I compiled the code, uploaded it to my device and it didn’t work…

Arduino monitor output showing rk_setWorkingMode failing

I then had a look at the Arduino library code and enabled some of the commented out diagnostic println statements. At the time it did seem odd there were no responses from the module.

Arduino monitor output showing rk_setWorkingMode failing with debugging

I had noticed some odd characters in the RAK Serial Port Tool while checking version numbers etc.

Setting work Mode with RAK Serial Port Tool

It looked like maybe the serial port was having some issues, so I double checked my modification of the HardwareSerial.h file and began to wonder (as the binary size wasn’t changing) if I had the right file. After some research I found there are several copies of that file and I wasn’t modifying the correct one.

Multiple locations of HardwareSerial.h

Then I realised that the port sending AT Commands to the module was actually a SoftwareSerial port not a hardware one. I then tried changing the size of the software serial buffers but still was having problems.

Arduino tool with default buffer sizes (833 bytes)
Arduino tool with non-default buffer sizes (961 bytes)

I then tried recompiling with different settings to see if the serial port issues would stop. The global variables size changed which showed I had the right files/settings but the code still didn’t work.

Going back over my settings I tried the command used in the rk_setWorkingMode call in the RAK Serial Port Tool and it worked.

I then then went for a walk and when I came back I realised the module speed was set to 115200 baud by default (which it is). I then used at+set_config=device:uart:1:9600 (don’t forget to press <enter> at end of the line) to set baud rate to match the code.

Setting device to 9600 baud

I then changed the jumpers and ran the software again…

So, it looks like the RAK811 module was set to 115200 baud (web based setup instructions), but the later library versions were 9600 baud, but the instructions didn’t mention the need to change the speed with the RAK Serial port tool.

Image of code and setup from RAK instructions

Now that my device is trying to connect to a network I need to configure the LoRaWAN network settings. I’m going to use the RAK7246G LPWAN Developer Gateway and the nationwide LoRaWAN network operated by Spark in New Zealand.

Wilderness Labs nRF24L01 Wireless field gateway Meadow client

After a longish pause in development work on my nrf24L01 AdaFruit.IO and Azure IOT Hub field gateways I figured a client based on my port of the techfooninja nRF24 library to Wilderness Labs Meadow would be a good test.

This sample client is an Wilderness Labs Meadow with a Sensiron SHT31 Temperature & humidity sensor (supported by meadow foundation), and a generic nRF24L01 device connected with jumper cables.

Bill of materials (prices as at March 2020)

  • Wilderness Labs Meadow 7F Micro device USD50
  • Seeedstudio Temperature and Humidity Sensor(SHT31) USD11.90
  • Seeedstudio 4 pin Male Jumper to Grove 4 pin Conversion Cable USD2.90
  • 2.4G Wireless Module nRF24L01+PA USD9.90

The initial version of the code was pretty basic with limited error handling and no power conservation support.

namespace devMobile.IoT.FieldGateway.Client
{
   using System;
   using System.Text;
   using System.Threading;

   using Radios.RF24;

   using Meadow;
   using Meadow.Devices;
   using Meadow.Foundation.Leds;
   using Meadow.Foundation.Sensors.Atmospheric;
   using Meadow.Hardware;
   using Meadow.Peripherals.Leds;

   public class MeadowClient : App<F7Micro, MeadowClient>
   {
      private const string BaseStationAddress = "Base1";
      private const string DeviceAddress = "WLAB1";
      private const byte nRF24Channel = 15;
      private RF24 Radio = new RF24();
      private readonly TimeSpan periodTime = new TimeSpan(0, 0, 60);
      private readonly Sht31D sensor;
      private readonly ILed Led;

      public MeadowClient()
      {
         Led = new Led(Device, Device.Pins.OnboardLedGreen);

         try
         {
            sensor = new Sht31D(Device.CreateI2cBus());

            var config = new Meadow.Hardware.SpiClockConfiguration(
                           2000,
                           SpiClockConfiguration.Mode.Mode0);

            ISpiBus spiBus = Device.CreateSpiBus(
               Device.Pins.SCK,
               Device.Pins.MOSI,
               Device.Pins.MISO, config);

            Radio.OnDataReceived += Radio_OnDataReceived;
            Radio.OnTransmitFailed += Radio_OnTransmitFailed;
            Radio.OnTransmitSuccess += Radio_OnTransmitSuccess;

            Radio.Initialize(Device, spiBus, Device.Pins.D09, Device.Pins.D10, Device.Pins.D11);
            //Radio.Address = Encoding.UTF8.GetBytes(Environment.MachineName);
            Radio.Address = Encoding.UTF8.GetBytes(DeviceAddress);

            Radio.Channel = nRF24Channel;
            Radio.PowerLevel = PowerLevel.Low;
            Radio.DataRate = DataRate.DR250Kbps;
            Radio.IsEnabled = true;

            Radio.IsAutoAcknowledge = true;
            Radio.IsDyanmicAcknowledge = false;
            Radio.IsDynamicPayload = true;

            Console.WriteLine($"Address: {Encoding.UTF8.GetString(Radio.Address)}");
            Console.WriteLine($"PowerLevel: {Radio.PowerLevel}");
            Console.WriteLine($"IsAutoAcknowledge: {Radio.IsAutoAcknowledge}");
            Console.WriteLine($"Channel: {Radio.Channel}");
            Console.WriteLine($"DataRate: {Radio.DataRate}");
            Console.WriteLine($"IsDynamicAcknowledge: {Radio.IsDyanmicAcknowledge}");
            Console.WriteLine($"IsDynamicPayload: {Radio.IsDynamicPayload}");
            Console.WriteLine($"IsEnabled: {Radio.IsEnabled}");
            Console.WriteLine($"Frequency: {Radio.Frequency}");
            Console.WriteLine($"IsInitialized: {Radio.IsInitialized}");
            Console.WriteLine($"IsPowered: {Radio.IsPowered}");
         }
         catch (Exception ex)
         {
            Console.WriteLine(ex.Message);
         }

         while (true)
         {
            sensor.Update();

            Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-TX T:{sensor.Temperature:0.0}C H:{sensor.Humidity:0}%");

            Led.IsOn = true;

            string values = "T " + sensor.Temperature.ToString("F1") + ",H " + sensor.Humidity.ToString("F0");

            // Stuff the 2 byte header ( payload type & deviceIdentifierLength ) + deviceIdentifier into payload
            byte[] payload = new byte[1 + Radio.Address.Length + values.Length];
            payload[0] = (byte)((1 << 4) | Radio.Address.Length);
            Array.Copy(Radio.Address, 0, payload, 1, Radio.Address.Length);
            Encoding.UTF8.GetBytes(values, 0, values.Length, payload, Radio.Address.Length + 1);

            Radio.SendTo(Encoding.UTF8.GetBytes(BaseStationAddress), payload);

            Thread.Sleep(periodTime);
         }
      }

      private void Radio_OnDataReceived(byte[] data)
      {
         // Display as Unicode
         string unicodeText = Encoding.UTF8.GetString(data);
         Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-RX Unicode Length {0} Unicode Length {1} Unicode text {2}", data.Length, unicodeText.Length, unicodeText);

         // display as hex
         Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-RX Hex Length {data.Length} Payload {BitConverter.ToString(data)}");
      }

      private void Radio_OnTransmitSuccess()
      {
         Led.IsOn = false;

         Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-TX Succeeded!");
      }

      private void Radio_OnTransmitFailed()
      {
         Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss}-TX failed!");
      }
   }
}

After sorting out power to the SHT31 (I had to push the jumper cable further into the back of the jumper cable plug). I could see temperature and humidity values getting uploaded to Adafruit.IO.

Visual Studio 2019 debug output

Adafruit.IO “automagically” provisions new feeds which is helpful when building a proof of concept (PoC)

Adafruit.IO feed with default feed IDs

I then modified the feed configuration to give it a user friendly name.

Feed Configuration

All up configuration took about 10 minutes.

Meadow device temperature and humidity

AllThingsTalk with MQTTnet

As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.

This PoC was to confirm that I could connect to the AllThingsTalk MQTT API then format topics and payloads correctly.

MQTTNet Console Client

The AllThingsTalk MQTT broker, username, and device ID are required command line parameters.

namespace devmobile.Mqtt.TestClient.AllThingsTalk
{
	using System;
	using System.Diagnostics;
	using System.Threading;
	using System.Threading.Tasks;

	using MQTTnet;
	using MQTTnet.Client;
	using MQTTnet.Client.Disconnecting;
	using MQTTnet.Client.Options;
	using MQTTnet.Client.Receiving;

	using Newtonsoft.Json;
	using Newtonsoft.Json.Linq;

	class Program
	{
		private static IMqttClient mqttClient = null;
		private static IMqttClientOptions mqttOptions = null;
		private static string server;
		private static string username;
		private static string deviceID;

		static void Main(string[] args)
		{
			MqttFactory factory = new MqttFactory();
			mqttClient = factory.CreateMqttClient();

			if ((args.Length != 3))
			{
				Console.WriteLine("[MQTT Server] [UserName] [ClientID]");
				Console.WriteLine("Press <enter> to exit");
				Console.ReadLine();
				return;
			}

			server = args[0];
			username = args[1];
			deviceID = args[2];

			Console.WriteLine($"MQTT Server:{server} DeviceID:{deviceID}");

			// AllThingsTalk formatted device state update topic
			string topicD2C = $"device/{deviceID}/state";

			mqttOptions = new MqttClientOptionsBuilder()
				.WithTcpServer(server)
				.WithCredentials(username, "HighlySecurePassword")
				.WithClientId(deviceID)
				.WithTls()
				.Build();

			mqttClient.UseDisconnectedHandler(new MqttClientDisconnectedHandlerDelegate(e => MqttClient_Disconnected(e)));
			mqttClient.UseApplicationMessageReceivedHandler(new MqttApplicationMessageReceivedHandlerDelegate(e => MqttClient_ApplicationMessageReceived(e)));
			mqttClient.ConnectAsync(mqttOptions).Wait();

			// AllThingsTalk formatted device command with wildcard topic
			string topicC2D = $"device/{deviceID}/asset/+/command";

			mqttClient.SubscribeAsync(topicC2D, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce).GetAwaiter().GetResult();

			while (true)
			{
				JObject payloadJObject = new JObject();

				double temperature = 22.0 + (DateTime.UtcNow.Millisecond / 1000.0);
				temperature = Math.Round( temperature, 1 );
				double humidity = 50 + (DateTime.UtcNow.Millisecond / 100.0);
				humidity = Math.Round(humidity, 1);

				JObject temperatureJObject = new JObject
				{
					{ "value", temperature }
				};
				payloadJObject.Add("Temperature", temperatureJObject);

				JObject humidityJObject = new JObject
				{
					{ "value", humidity }
				};
				payloadJObject.Add("Humidity", humidityJObject);

				string payload = JsonConvert.SerializeObject(payloadJObject);
				Console.WriteLine($"Topic:{topicD2C} Payload:{payload}");

				var message = new MqttApplicationMessageBuilder()
					.WithTopic(topicD2C)
					.WithPayload(payload)
					.WithAtMostOnceQoS()
//					.WithAtLeastOnceQoS()
					.Build();

				Console.WriteLine("PublishAsync start");
				mqttClient.PublishAsync(message).Wait();
				Console.WriteLine("PublishAsync finish");

				Thread.Sleep(15100);
			}
		}

		private static void MqttClient_ApplicationMessageReceived(MqttApplicationMessageReceivedEventArgs e)
		{
			Console.WriteLine($"ClientId:{e.ClientId} Topic:{e.ApplicationMessage.Topic} Payload:{e.ApplicationMessage.ConvertPayloadToString()}");
		}

		private static async void MqttClient_Disconnected(MqttClientDisconnectedEventArgs e)
		{
			Debug.WriteLine("Disconnected");
			await Task.Delay(TimeSpan.FromSeconds(5));

			try
			{
				await mqttClient.ConnectAsync(mqttOptions);
			}
			catch (Exception ex)
			{
				Debug.WriteLine("Reconnect failed {0}", ex.Message);
			}
		}
	}

The AllThingsTalk device configuration was relatively easy but I need to investigate “Gateway” functionality and configuration further.

Configuring an Asset
Configuration a watchdog to check for sensor data
Sending a command to an actuator
Processing a command on the client

The ability to look at message payloads in the Debug tab would be very helpful when working out why a payload was not being processed as expected.

Asset debug information

Overall the AllThingsTalk configuration went fairly smoothly, though I need to investigate the “Gateway” configuration and functionality further. The way that assets are name by the system could make support in my MQTT Gateway more complex.

Azure IoT Hub MQTT+TLS Overheads

An Azure IoT Hub has a series of metrics and one I had been using was “Total Device Data Usage”. To better understand what it was displaying I modified my Azure IoT Hub MQTT Test Application to display the size of the JOSN payload.

MQTTNet based client displaying payload length

The size of the packets sent and the total device data appeared to map pretty well but I was also interested in the Transport Layer Security (TLS) and Messaging Queuing Telemetry Transport (MQTT) overheads.

Azure IoT Hub Metrics

To get an idea of the overheads I fired up LiveTcpUdpWatch by Nirsoft and noted down the traffic measure on port 8883.

Conenction LiveTcpUdpWatch main screen

Launching the MQTTNet client sending every 30 seconds resulted in traffic like this

4179b - Establishing connection
4284b - 105b
4317b - 33b
4386b - 69b
4455b - 69b
4524b - 69b
4593b - 69b
4662b - 69b
4731b - 69b
4800b - 69b
4869b - 69b
4938b - 69b
5007b - 69b
5076b - 69b
5145b - 69b
5214b - 69b
5288b - 69b

So it looks like my very rough numbers are close to the numbers discussed in the above article. I need to explore the impact of keep-alive messages and other background operations.