The Things Network HTTP Integration Part2

Basic JSON Deserialisation

This is the second in a series of posts about building an HTTP Integration for a The Things Network(TTN) application.

I used JSON2Csharp and a payload I downloaded in Part 1 to generate C# classes which would deserialise my minimalist messages.

// First version of classes for unpacking HTTP payload https://json2csharp.com/
public class GatewayV1
{
   public string gtw_id { get; set; }
   public int timestamp { get; set; }
   public DateTime time { get; set; }
   public int channel { get; set; }
   public int rssi { get; set; }
   public double snr { get; set; }
   public int rf_chain { get; set; }
   public double latitude { get; set; }
   public double longitude { get; set; }
   public int altitude { get; set; }
}

public class MetadataV1
{
   public string time { get; set; }
   public double frequency { get; set; }
   public string modulation { get; set; }
   public string data_rate { get; set; }
   public string coding_rate { get; set; }
   public List<GatewayV1> gateways { get; set; }
}

public class PayloadV1
{
   public string app_id { get; set; }
   public string dev_id { get; set; }
   public string hardware_serial { get; set; }
   public int port { get; set; }
   public int counter { get; set; }
   public bool confirmed { get; set; }
   public string payload_raw { get; set; }
   public MetadataV1 metadata { get; set; }
   public string downlink_url { get; set; }
}

I added a new to controller to my application which used the generated classes to deserialise the body of the POST from the TTN Application Integration.

[Route("[controller]")]
[ApiController]
public class ClassSerialisationV1 : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public IActionResult Post([FromBody] PayloadV1 payload)
   {
      // Check that the post data is good
      if (!this.ModelState.IsValid)
      {
         log.WarnFormat("ClassSerialisationV1 validation failed {0}", this.ModelState.Messages());

         return this.BadRequest(this.ModelState);
      }
      log.Info($"DevEUI:{payload.hardware_serial} Payload Base64:{payload.payload_raw}");

      return this.Ok();
   }
}

I then updated the TTN application integration to send messages to my new endpoint.

TTN Application configuration overview

In the body of the Application Insights events I could see the devEUI, port, and the raw payload had been extracted from the message.

DevEUI:1234567890123456 Port:1 Payload Base64:VGlueUNMUiBMb1JhV0FO

I then added another controller which decoded the Base64 encoded payload_raw.

[Route("[controller]")]
[ApiController]
public class ClassSerialisationV2Base64Decoded : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public IActionResult Post([FromBody] PayloadV2 payload)
   {
      // Check that the post data is good
      if (!this.ModelState.IsValid)
      {
         log.WarnFormat("ClassSerialisationV2BCDDecoded validation failed {0}", this.ModelState.Messages());

         return this.BadRequest(this.ModelState);
      }

      log.Info($"DevEUI:{payload.hardware_serial} Port:{payload.port} Payload:{ Encoding.UTF8.GetString(Convert.FromBase64String(payload.payload_raw))}");

      return this.Ok();
   }
}
DevEUI:1234567890123456 Port:1 Payload:TinyCLR LoRaWAN

Then after a while the deserialisation started to fail with an HTTP 400-Bad request. When I ran the same request with Telerik Fiddler on my desktop the raw response was

HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Content-Type: application/problem+json; charset=utf-8
Server: Microsoft-IIS/10.0
Request-Context: appId=cid-v1:f4f72f2e-1144-4578-923f-d3ebdcfb7766
X-Powered-By: ASP.NET
Date: Mon, 31 Aug 2020 09:07:30 GMT

17a
{"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1",
"title":"One or more validation errors occurred.",
"status":400,
"traceId":"00-45033ec030b63d4ebb82b95b67cb8142-9fc52a18be202848-00",
"errors":{
"$.metadata.gateways[0].timestamp":["The JSON value could not be converted to System.Int32. 
Path: $.metadata.gateways[0].timestamp | LineNumber: 21 | BytePositionInLine: 35."]}}
0

The line in the payload was the gateway timestamp. The value was 2,426,973,100 which larger than 2,147,483,647 the maximum number that can be stored in a signed 32 bit integer. The JSON2CSharp generator had made a reasonable choice of datatype but in this case the range was not sufficient.

public class GatewayV2 
{
   public string gtw_id { get; set; }
   public ulong timestamp { get; set; }
   public DateTime time { get; set; }
   public int channel { get; set; }
   public int rssi { get; set; }
   public double snr { get; set; }
   public int rf_chain { get; set; }
   public double latitude { get; set; }
   public double longitude { get; set; }
   public int altitude { get; set; }
}

I checked the TTN code where the variable was declared as an unsigned 64 bit integer.

This issue could occur for other variables so I need to manually check all the generated classes.

The Things Network HTTP Integration Part1

Infrastructure and payloads

This is the first in a series of posts about building an HTTP Integration for a The Things Network(TTN) application. I have assumed that readers are familiar with the configuration and operation of a TTN instance so I’m not going to cover that in detail.

I’m using a Seeeduino LoRaWAN device as a client so I configured the sample Over the Air Activation(OTAA) application to connect to my local RAK7246 Developer gateway.

#include <LoRaWan.h>

unsigned char data[] = {0x53, 0x65, 0x65, 0x65, 0x64, 0x75, 0x69, 0x6E, 0x6F, 0x20, 0x4C, 0x6F, 0x52, 0x61, 0x57, 0x41, 0x4E};
char buffer[256];

void setup(void)
{
  SerialUSB.begin(9600);
  while (!SerialUSB);

  lora.init();

  memset(buffer, 0, 256);
  lora.getVersion(buffer, 256, 1);
  SerialUSB.print("Ver:");
  SerialUSB.print(buffer);

  memset(buffer, 0, 256);
  lora.getId(buffer, 256, 1);
  SerialUSB.print(buffer);
  SerialUSB.print("ID:");

  lora.setKey(NULL, NULL, "12345678901234567890123456789012");
  lora.setId(NULL, "1234567890123456", "1234567890123456");

  lora.setPort(10);

  lora.setDeciveMode(LWOTAA);
  lora.setDataRate(DR0, AS923);

  lora.setDutyCycle(false);
  lora.setJoinDutyCycle(false);

  lora.setPower(14);


  while (!lora.setOTAAJoin(JOIN, 10))
  {
    SerialUSB.println("");
  }
    SerialUSB.println( "Joined");
}

void loop(void)
{
  bool result = false;

  //result = lora.transferPacket("Hello World!", 10);
  result = lora.transferPacket(data, sizeof(data));

  if (result)
  {
    short length;
    short rssi;

    memset(buffer, 0, 256);
    length = lora.receivePacket(buffer, 256, &rssi);

    if (length)
    {
      SerialUSB.print("Length is: ");
      SerialUSB.println(length);
      SerialUSB.print("RSSI is: ");
      SerialUSB.println(rssi);
      SerialUSB.print("Data is: ");
      for (unsigned char i = 0; i < length; i ++)
      {
        SerialUSB.print("0x");
        SerialUSB.print(buffer[i], HEX);
        SerialUSB.print(" ");
      }
      SerialUSB.println();
    }
  }
  delay( 10000);
}

The SetKey and SetId parameters are not obvious and it would be much easier if there were two methods one for OTTA and the other for Activation by-Personalization(ABP).
I then built an Net Core 3.1 Web API application which had a single controller to receive messages from TTN.

[Route("[controller]")]
[ApiController]
public class Raw : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   [HttpGet]
   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public void PostRaw([FromBody]JsonElement body)
   {
      string json = JsonSerializer.Serialize(body);

      log.Info(json);
   }
}

I then configured my TTN application integration to send messages to my shinny new endpoint

TTN Application configuration overview

My controller logged events to Azure application Insights so I could see if there were any errors and inspect message payloads. The TTN developers provide sample payloads to illustrate the message format but they were a bit chunky for my initial testing.

Application Insights event list

I could then inspect individual events and payloads

Application Insights event display

At this point I could download message payloads and save them locally.

{
   "app_id": "rak811wisnodetest",
   "dev_id": "rak811wisnode1",
   "hardware_serial": "1234567890123456",
   "port": 1,
   "counter": 2,
   "confirmed": true,
   "payload_raw": "VGlueUNMUiBMb1JhV0FO",
   "metadata": {
      "time": "2020-08-26T00:50:36.182774822Z",
      "frequency": 924.2,
      "modulation": "LORA",
      "data_rate": "SF7BW125",
      "coding_rate": "4/5",
      "gateways": [
         {
            "gtw_id": "eui-b827ebfffe6c279d",
            "timestamp": 1584148244,
            "time": "2020-08-26T00:50:35.012774Z",
            "channel": 5,
            "rssi": -63,
            "snr": 9.2,
            "rf_chain": 0,
            "latitude": -43.49889,
            "longitude": 172.60104,
            "altitude": 16
         }
      ]
   },
   "downlink_url": "https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/rak811wisnodetest/azure-webapi-endpoint?key=ttn-account-v2.12345678901234567_12345_1234567-dduo"

}

These were useful because I could then use a tool like Telerik Fiddler to submit messages to my application when it was running locally in the Visual Studio 2019 debugger.

Adafruit.IO LoRa Field Gateway BCD Addressing

After some testing with more client devices, especially the Easy Sensors Arduino Nano radio shield RFM69/95 or NRF24L01+ I have decided to move to non text addresses for devices and the LoRa field gateway.

THIS IS A BREAKING CHANGE

The unique identifier provided by the SHA204A crypto and authentication chip on the EasySensors shield highlighted this issue. The Binary Coded Decimal(BCD) version of the 72 bit identifier was too long to fit in the from address.

My later Arduino based sample clients have some helper functions to populate the message header, add values, and prepare the message payload for reuse.

On the server side I have added code to log the build version and Raspbery PI shield type

// Log the Application build, shield information etc.
LoggingFields appllicationBuildInformation = new LoggingFields();
#if DRAGINO
   appllicationBuildInformation.AddString("Shield", "DraginoLoRaGPSHat");
#endif
…
#if UPUTRONICS_RPIPLUS_CS1
   appllicationBuildInformation.AddString("Shield", "UputronicsPiPlusLoRaExpansionBoardCS1");
#endif
appllicationBuildInformation.AddString("Timezone", TimeZoneSettings.CurrentTimeZoneDisplayName);
appllicationBuildInformation.AddString("OSVersion", Environment.OSVersion.VersionString);
appllicationBuildInformation.AddString("MachineName", Environment.MachineName);

// This is from the application manifest 
Package package = Package.Current;
PackageId packageId = package.Id;
PackageVersion version = packageId.Version;

appllicationBuildInformation.AddString("ApplicationVersion", string.Format($"{version.Major}.{version.Minor}.{version.Build}.{version.Revision}"));
this.loggingChannel.LogEvent("Application starting", appllicationBuildInformation, LoggingLevel.Information);

Then when the message payload is populated the from address byte array is converted to BCD

private async void Rfm9XDevice_OnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e)
{
   string addressBcdText;
   string messageBcdText;
   string messageText = "";
   char[] sensorReadingSeparator = new char[] { ',' };
   char[] sensorIdAndValueSeparator = new char[] { ' ' };

   addressBcdText = BitConverter.ToString(e.Address);

   messageBcdText = BitConverter.ToString(e.Data);
   try
   {
      messageText = UTF8Encoding.UTF8.GetString(e.Data);
   }
   catch (Exception)
   {
      this.loggingChannel.LogMessage("Failure converting payload to text", LoggingLevel.Error);
   return;
   }

#if DEBUG
    Debug.WriteLine(@"{0:HH:mm:ss}-RX From {1} PacketSnr {2:0.0} Packet RSSI {3}dBm RSSI {4}dBm = {5} byte message ""{6}""", DateTime.Now, addressBcdText, e.PacketSnr, e.PacketRssi, e.Rssi, e.Data.Length, messageText);
#endif
   LoggingFields messagePayload = new LoggingFields();
   messagePayload.AddInt32("AddressLength", e.Address.Length);
   messagePayload.AddString("Address-BCD", addressBcdText);
   messagePayload.AddInt32("Message-Length", e.Data.Length);
   messagePayload.AddString("Message-BCD", messageBcdText);
   messagePayload.AddString("Nessage-Unicode", messageText);
   messagePayload.AddDouble("Packet SNR", e.PacketSnr);
   messagePayload.AddInt32("Packet RSSI", e.PacketRssi);
   messagePayload.AddInt32("RSSI", e.Rssi);
   this.loggingChannel.LogEvent("Message Data", messagePayload, LoggingLevel.Verbose);

			
   // Check the address is not to short/long 
   if (e.Address.Length < AddressLengthMinimum)
   {
      this.loggingChannel.LogMessage("From address too short", LoggingLevel.Warning);
      return;
   }

   if (e.Address.Length > MessageLengthMaximum)
   {
      this.loggingChannel.LogMessage("From address too long", LoggingLevel.Warning);
      return;
   }

   // Check the payload is not too short/long 
   if (e.Data.Length < MessageLengthMinimum)
   {
      this.loggingChannel.LogMessage("Message too short to contain any data", LoggingLevel.Warning);
      return;
   }

   if (e.Data.Length > MessageLengthMaximum)
   {
      this.loggingChannel.LogMessage("Message too long to contain valid data", LoggingLevel.Warning);
      return;
   }

   // Adafruit IO is case sensitive & only does lower case ?
   string deviceId = addressBcdText.ToLower();

   // Chop up the CSV text payload
   string[] sensorReadings = messageText.Split(sensorReadingSeparator, StringSplitOptions.RemoveEmptyEntries);
   if (sensorReadings.Length == 0)
   {
      this.loggingChannel.LogMessage("Payload contains no sensor readings", LoggingLevel.Warning);
      return;
   }

   Group_feed_data groupFeedData = new Group_feed_data();

   LoggingFields sensorData = new LoggingFields();
   sensorData.AddString("DeviceID", deviceId);

   // Chop up each sensor reading into an ID & value
   foreach (string sensorReading in sensorReadings)
   {
      string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparator, StringSplitOptions.RemoveEmptyEntries);

      // Check that there is an id & value
      if (sensorIdAndValue.Length != 2)
      {
         this.loggingChannel.LogMessage("Sensor reading invalid format", LoggingLevel.Warning);
         return;
      }

      string sensorId = sensorIdAndValue[0].ToLower();
      string value = sensorIdAndValue[1];

      // Construct the sensor ID from SensordeviceID & Value ID
      groupFeedData.Feeds.Add(new Anonymous2() { Key = string.Format("{0}{1}", deviceId, sensorId), Value = value });

      sensorData.AddString(sensorId, value);

      Debug.WriteLine(" Sensor {0}{1} Value {2}", deviceId, sensorId, value);
   }
   this.loggingChannel.LogEvent("Sensor readings", sensorData, LoggingLevel.Verbose);

   try
   {
      Debug.WriteLine(" CreateGroupDataAsync start");
      await this.adaFruitIOClient.CreateGroupDataAsync(this.applicationSettings.AdaFruitIOUserName,
this.applicationSettings.AdaFruitIOGroupName.ToLower(), groupFeedData);
      Debug.WriteLine(" CreateGroupDataAsync finish");
   }
   catch (Exception ex)
   {
      Debug.WriteLine(" CreateGroupDataAsync failed {0}", ex.Message);
				this.loggingChannel.LogMessage("CreateGroupDataAsync failed " + ex.Message, LoggingLevel.Error);
   }
}
AfaFruit.IO Data Display

This does mean longer field names but I usually copy n paste them from the Arduino serial monitor or the Event Tracing For Windows (ETW) logging.

AdaFruit.IO Field gateway ETW Logging

Azure IoT Hubs LoRa Field Gateway BCD Addressing

After some testing with more client devices, especially the Easy Sensors Arduino Nano radio shield RFM69/95 or NRF24L01+ I have decided to move to non text addresses for devices and the LoRa field gateway.

THIS IS A BREAKING CHANGE

The unique identifier provided by the SHA204A crypto and authentication chip on the EasySensors shield highlighted this issue. The Binary Coded Decimal(BCD) version of the 72 bit identifier was too long to fit in the from address.

My Arduino MKR1300 sample code has some helper functions to populate the message header, add values, and prepare the message payload for reuse.

On the server side I have added code to log the build version and Raspbery PI shield type

// Log the Application build, shield information etc.
LoggingFields appllicationBuildInformation = new LoggingFields();

#if DRAGINO
   appllicationBuildInformation.AddString("Shield", "DraginoLoRaGPSHat");
#endif

...

#if UPUTRONICS_RPIPLUS_CS1
   appllicationBuildInformation.AddString("Shield", "UputronicsPiPlusLoRaExpansionBoardCS1");
#endif
appllicationBuildInformation.AddString("Timezone", TimeZoneSettings.CurrentTimeZoneDisplayName);
appllicationBuildInformation.AddString("OSVersion", Environment.OSVersion.VersionString);
appllicationBuildInformation.AddString("MachineName", Environment.MachineName);

// This is from the application manifest 
Package package = Package.Current;
PackageId packageId = package.Id;
PackageVersion version = packageId.Version;

appllicationBuildInformation.AddString("ApplicationVersion", string.Format($"{version.Major}.{version.Minor}.{version.Build}.{version.Revision}"));
this.logging.LogEvent("Application starting", appllicationBuildInformation, LoggingLevel.Information);

Then when the message payload is populated the from address byte array is converted to BCD

private async void Rfm9XDevice_OnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e)
{
   string addressBcdText;
   string messageBcdText;
   string messageText = "";
   char[] sensorReadingSeparators = new char[] { ',' };
   char[] sensorIdAndValueSeparators = new char[] { ' ' };

   addressBcdText = BitConverter.ToString(e.Address);
   messageBcdText = BitConverter.ToString(e.Data);
   try
   {
      messageText = UTF8Encoding.UTF8.GetString(e.Data);
   }
   catch (Exception)
   {
      this.logging.LogMessage("Failure converting payload to text", 
   LoggingLevel.Error);
      return;
   }


#if DEBUG
   Debug.WriteLine(@"{0:HH:mm:ss}-RX From {1} PacketSnr {2:0.0} Packet 
   RSSI {3}dBm RSSI {4}dBm = {5} byte message ""{6}""", DateTime.Now, 
   messageBcdText, e.PacketSnr, e.PacketRssi, e.Rssi, e.Data.Length, 
   messageText);
#endif
   LoggingFields messagePayload = new LoggingFields();
   messagePayload.AddInt32("AddressLength", e.Address.Length);
   messagePayload.AddString("Address-BCD", addressBcdText);
   messagePayload.AddInt32("Message-Length", e.Data.Length);
   messagePayload.AddString("Message-BCD", messageBcdText);
   messagePayload.AddString("Message-Unicode", messageText);
   messagePayload.AddDouble("Packet SNR", e.PacketSnr);
   messagePayload.AddInt32("Packet RSSI", e.PacketRssi);
   messagePayload.AddInt32("RSSI", e.Rssi);
   this.logging.LogEvent("Message Data", messagePayload, LoggingLevel.Verbose);

//...

   JObject telemetryDataPoint = new JObject(); // This could be simplified but for field gateway will use this style
   LoggingFields sensorData = new LoggingFields();

   telemetryDataPoint.Add("DeviceID", addressBcdText);
   sensorData.AddString("DeviceID", addressBcdText);
   telemetryDataPoint.Add("PacketSNR", e.PacketSnr.ToString("F1"));
   sensorData.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
   telemetryDataPoint.Add("PacketRSSI", e.PacketRssi);
   sensorData.AddInt32("PacketRSSI", e.PacketRssi);
   telemetryDataPoint.Add("RSSI", e.Rssi);
   sensorData.AddInt32("RSSI", e.Rssi);

   //Chop up each sensor read into an ID & value
   foreach (string sensorReading in sensorReadings)
   {
      string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);

      // Check that there is an id & value
      if (sensorIdAndValue.Length != 2)
      {
         this.logging.LogMessage("Sensor reading invalid format", LoggingLevel.Warning);
         return;
      }

      string sensorId = sensorIdAndValue[0];
      string value = sensorIdAndValue[1];

      try
      {
         if (this.applicationSettings.SensorIDIsDeviceIDSensorID)
         {
            // Construct the sensor ID from SensordeviceID & Value ID
            telemetryDataPoint.Add(string.Format("{0}{1}", addressBcdText, sensorId), value);

            sensorData.AddString(string.Format("{0}{1}", addressBcdText, sensorId), value);
            Debug.WriteLine(" Sensor {0}{1} Value {2}", addressBcdText, sensorId, value);
         }
         else
         {
            telemetryDataPoint.Add(sensorId, value);
            sensorData.AddString(sensorId, value);
            Debug.WriteLine(" Device {0} Sensor {1} Value {2}", addressBcdText, sensorId, value);
         }
      }
      catch (Exception ex)
      {
         this.logging.LogMessage("Sensor reading invalid JSON format " + ex.Message, LoggingLevel.Warning);
         return;
      }
   }

  this.logging.LogEvent("Sensor readings", sensorData, LoggingLevel.Information);

   try
   {
      using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
      {
         Debug.WriteLine(" AzureIoTHubClient SendEventAsync start");
         await this.azureIoTHubClient.SendEventAsync(message);
         Debug.WriteLine(" AzureIoTHubClient SendEventAsync finish");
      }
   }
   catch (Exception ex)
   {
      this.logging.LogMessage("AzureIoTHubClient SendEventAsync failed " + ex.Message, LoggingLevel.Error);
   }
}

This does mean longer field names but I usually copy n paste them from the Arduino serial monitor of the Event Tracing For Windows (ETW) logging.

Azure IoT Hub LoRa Field Gateway ETW Logging