TTI V3 Connector Device EUI Representation

Posted on April 15, 2021 by devmobilenz

While debugging The Things Industries(TTI) V3 connector on my desktop I had noticed the Device EUI‘s were wrong.

TTI V3 Connector application running as a console application showing incorrect DeviceEUIs

The TTI V3 Connector code…

foreach (V3EndDevice device in endDevices.End_devices)
{
   if (DeviceAzureEnabled(device))
   {
      _logger.LogInformation("Config-ApplicationID:{0} DeviceID:{1} Device EUI:{2}", device.Ids.Application_ids.Application_id, device.Ids.Device_id, BitConverter.ToString(device.Ids.Dev_eui));

      tasks.Add(DeviceRegistration(device.Ids.Application_ids.Application_id, device.Ids.Device_id, _programSettings.ResolveDeviceModelId(device.Ids.Application_ids.Application_id, device.Attributes), stoppingToken));
   }
}

…uses some classes generated by nSwag based on the TheThingsNetwork/LoRaWAN-stack api.swagger.json

public partial class V3EndDeviceIdentifiers 
{
        [Newtonsoft.Json.JsonProperty("device_id", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string Device_id { get; set; }
    
        [Newtonsoft.Json.JsonProperty("application_ids", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public V3ApplicationIdentifiers Application_ids { get; set; }
    
        /// <summary>The LoRaWAN DevEUI.</summary>
        [Newtonsoft.Json.JsonProperty("dev_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Dev_eui { get; set; }
    
        /// <summary>The LoRaWAN JoinEUI (AppEUI until LoRaWAN 1.0.3 end devices).</summary>
        [Newtonsoft.Json.JsonProperty("join_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Join_eui { get; set; }
    
        /// <summary>The LoRaWAN DevAddr.</summary>
        [Newtonsoft.Json.JsonProperty("dev_addr", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Dev_addr { get; set; }
}

After some research I found references to the underlying problem in TTN and OpenAPI forums. The Dev_addr and Dev_eui fields are Base16(Hexidecimal) encoded binary but are being processed as if they were Base64(mime) encoded.

The TTI connector only displays the Device EUI so I changed the Dev_eui property to a string

public partial class V3EndDeviceIdentifiers 
{
        [Newtonsoft.Json.JsonProperty("device_id", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string Device_id { get; set; }
    
        [Newtonsoft.Json.JsonProperty("application_ids", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public V3ApplicationIdentifiers Application_ids { get; set; }
    
        /// <summary>The LoRaWAN DevEUI.</summary>
        [Newtonsoft.Json.JsonProperty("dev_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string Dev_eui { get; set; }

      /// <summary>The LoRaWAN JoinEUI (AppEUI until LoRaWAN 1.0.3 end devices).</summary>
      [Newtonsoft.Json.JsonProperty("join_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Join_eui { get; set; }
    
        /// <summary>The LoRaWAN DevAddr.</summary>
        [Newtonsoft.Json.JsonProperty("dev_addr", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Dev_addr { get; set; }
}

I also had to remove the BitConverter.ToString call

foreach (V3EndDevice device in endDevices.End_devices)
{
   if (DeviceAzureEnabled(device))
   {
      _logger.LogInformation("Config-ApplicationID:{0} DeviceID:{1} Device EUI:{2}", device.Ids.Application_ids.Application_id, device.Ids.Device_id, device.Ids.Dev_eui);

      tasks.Add(DeviceRegistration(device.Ids.Application_ids.Application_id, device.Ids.Device_id, _programSettings.ResolveDeviceModelId(device.Ids.Application_ids.Application_id, device.Attributes), stoppingToken));
   }
}

Now the DeviceEUI values are displayed correctly and searching for EndDevices in Azure Application Insights is easier

TTI V3 Connector application running as a console application showing correct DeviceEUIs

Modifying the nSwag generated classes is a really nasty way of fixing the problem but I think this approach is okay as it’s only one field and any other solution I could find was significantly more complex.

The Things Network HTTP Integration Part3

Posted on September 2, 2020 by devmobilenz

When Serialisation goes bad- payload_fields

This is the third in a series of posts about building an HTTP Integration for a The Things Network(TTN) application.

In part 1 & part 2 I had been ignoring the payload_fields property of the Payload class. The documentation indicates that payload_fields property is populated when an uplink message is Decoded.

There is a built in decoder for Cayenne Low Power Payload(LPP) which looked like the simplest option to start with.

TTN Application integration payload decoder selection

I modified the Seeeduino LoRaWAN Over The Air Activation(OTAA) sample application and added the CayenneLPP library from Electronic Cats.

#include <LoRaWan.h>
#include <CayenneLPP.h> 

CayenneLPP lpp(64);
char buffer[256];

void setup(void)
{
    SerialUSB.begin(9600);
    while(!SerialUSB);

    lora.init();

    memset(buffer, 0, 256);
    lora.getVersion(buffer, 256, 1);
    SerialUSB.print("Ver:");
    SerialUSB.print(buffer); 
 
    memset(buffer, 0, 256);
    lora.getId(buffer, 256, 1);
    SerialUSB.print("ID:");
    SerialUSB.println(buffer);

    lora.setKey(NULL, NULL, "12345678901234567890123456789012");
    lora.setId(NULL, "1234567890123456", "1234567890123456");

    lora.setPort(10);
        
    lora.setDeciveMode(LWOTAA);
    lora.setDataRate(DR0, AS923);

    lora.setDutyCycle(false);
    lora.setJoinDutyCycle(false);
 
    lora.setPower(14);

    while(!lora.setOTAAJoin(JOIN, 10));
}
 
void loop(void)
{   
    bool result = false;

    lpp.reset ();

    // Original LPPv1 data types only these work
    // https://www.thethingsnetwork.org/docs/devices/arduino/api/cayennelpp.html
    // https://loranow.com/cayennelpp/
    //
    lpp.addAnalogInput( 0, 0.01234) ;
    lpp.addAnalogOutput( 0, 0.56789);
    lpp.addDigitalInput(0, false);    
    lpp.addDigitalInput(1, true);    
    
    lpp.addGPS (1, -43.5309, 172.6371, 6.192);
    lpp.addAccelerometer(0, 0.0, 0.0, 1.0);
    lpp.addGyrometer(1, 0.0,0.0,0.0);
    
    lpp.addLuminosity(0, 0);    // Pitch black
    lpp.addLuminosity(1, 8000); // 40w fluro   
    lpp.addPresence(0, 0);
    lpp.addPresence(1, 1);

    lpp.addBarometricPressure(0,0.0);
    lpp.addBarometricPressure(0,1013.25);
    lpp.addRelativeHumidity (0, 50.0);
    lpp.addRelativeHumidity (1, 60.0);
    lpp.addTemperature (0, -273.00);
    lpp.addTemperature (1, 0.0);
    lpp.addTemperature (2, 100.0);

    // Additional data types don't think any of these worked
    //lpp.addUnixTime(1, millis()); 
    //lpp.addGenericSensor(1, 1.23456);
    //lpp.addVoltage(1, 4.5);
    //lpp.addCurrent(0, 1.0);
    //lpp.addFrequency (1, 50); 
    //lpp.addPercentage(1, 50);
    //lpp.addAltitude(1, 20.5);
    //lpp.addPower(1, 1500);
    //lpp.addDistance(1, 120.0);
    //lpp.addEnergy(1, 2.345);
    //lpp.addDirection(1, -98.76);
    //lpp.addSwitch(0, 1);
    //lpp.addConcentration(0, 10);
    //lpp.addColour(1, 255, 255, 255);

    uint8_t *lppBuffer = lpp.getBuffer();
    uint8_t lppLen = lpp.getSize();

    SerialUSB.print("Length is: ");
    SerialUSB.println(lppLen);

    // Dump buffer content for debugging
    PrintHexBuffer (lppBuffer, lppLen);    

    //result = lora.transferPacket("Hello World!", 10);
    result = lora.transferPacket(lppBuffer, lppLen);

    if(result)
    {
        short length;
        short rssi;
 
        memset(buffer, 0, sizeof(buffer));
        length = lora.receivePacket(buffer, 256, &rssi);
 
        if(length)
        {
            SerialUSB.print("Length is: ");
            SerialUSB.println(length);
            SerialUSB.print("RSSI is: ");
            SerialUSB.println(rssi);
            SerialUSB.print("Data is: ");
            for(unsigned char i = 0; i < length; i ++)
            {
                SerialUSB.print("0x");
                SerialUSB.print(buffer[i], HEX);
                SerialUSB.print(" ");
            }
            SerialUSB.println();
        }
    }
    delay( 30000);
}

void PrintHexBuffer( uint8_t *buffer, uint8_t size )
{

    for( uint8_t i = 0; i < size; i++ )
    {
        if(buffer[i] < 0x10)
        {
            Serial.print('0');
        }
        SerialUSB.print( buffer[i], HEX );
        Serial.print(" ");
    }
    SerialUSB.println( );
}

I then copied and saved to files the payloads from the Azure Application Insights events generated when an uplink messages were processed.

{
   "app_id": "rak811wisnodetest",
   "dev_id": "seeeduinolorawan4",
   "hardware_serial": "1234567890123456",
   "port": 10,
   "counter": 1,
   "is_retry": true,
   "payload_raw": "AWcBEAFlAGQBAAEBAgAyAYgAqYgGIxgBJuw=",
   "payload_fields": {
      "analog_in_1": 0.5,
      "digital_in_1": 1,
      "gps_1": {
         "altitude": 755,
         "latitude": 4.34,
         "longitude": 40.22
      },
      "luminosity_1": 100,
      "temperature_1": 27.2
   },
   "metadata": {
      "time": "2020-08-28T10:41:04.496594225Z",
      "frequency": 923.4,
      "modulation": "LORA",
      "data_rate": "SF12BW125",
      "coding_rate": "4/5",
      "gateways": [
         {
            "gtw_id": "eui-b827ebfffe6c279d",
            "timestamp": 3971612260,
            "time": "2020-08-28T10:41:03.313471Z",
            "channel": 1,
            "rssi": -53,
            "snr": 11.2,
            "rf_chain": 0,
            "latitude": -43.49885,
            "longitude": 172.60095,
            "altitude": 25
         }
      ]
   },
   "downlink_url": "https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/rak811wisnodetest/azure-webapi-endpoint?key=ttn-account-v2.12345678901234567_12345_1234567-dduo"
}

I used JSON2Csharp to generate C# classes which would deserialise the above uplink message.

// Third version of classes for unpacking HTTP payload 
public class Gps1V3
{
   public int altitude { get; set; }
   public double latitude { get; set; }
   public double longitude { get; set; }
}

public class PayloadFieldsV3
{
   public double analog_in_1 { get; set; }
   public int digital_in_1 { get; set; }
   public Gps1V3 gps_1 { get; set; }
   public int luminosity_1 { get; set; }
   public double temperature_1 { get; set; }
}

public class GatewayV3 
{
   public string gtw_id { get; set; }
   public ulong timestamp { get; set; }
   public DateTime time { get; set; }
   public int channel { get; set; }
   public int rssi { get; set; }
   public double snr { get; set; }
   public int rf_chain { get; set; }
   public double latitude { get; set; }
   public double longitude { get; set; }
   public int altitude { get; set; }
}

public class MetadataV3
{
   public string time { get; set; }
   public double frequency { get; set; }
   public string modulation { get; set; }
   public string data_rate { get; set; }
   public string coding_rate { get; set; }
   public List<GatewayV3> gateways { get; set; }
}

public class PayloadV3
{
   public string app_id { get; set; }
   public string dev_id { get; set; }
   public string hardware_serial { get; set; }
   public int port { get; set; }
   public int counter { get; set; }
   public bool is_retry { get; set; }
   public string payload_raw { get; set; }
   public PayloadFieldsV3 payload_fields { get; set; }
   public MetadataV3 metadata { get; set; }
   public string downlink_url { get; set; }
}

I added a new to controller to my application which used the generated classes to deserialise the body of the POST from the TTN Application Integration.

[Route("[controller]")]
[ApiController]
public class ClassSerialisationV3Fields : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public IActionResult Post([FromBody] PayloadV3 payload)
   {
      // Check that the post data is good
      if (!this.ModelState.IsValid)
      {
         log.WarnFormat("ClassSerialisationV3Fields validation failed {0}", this.ModelState.Messages());

         return this.BadRequest(this.ModelState);
      }

      log.Info($"DevEUI:{payload.hardware_serial} Payload Base64:{payload.payload_raw} analog_in_1:{payload.payload_fields.analog_in_1} digital_in_1:{payload.payload_fields.digital_in_1} gps_1:{payload.payload_fields.gps_1.latitude},{payload.payload_fields.gps_1.longitude},{payload.payload_fields.gps_1.altitude} luminosity_1:{payload.payload_fields.luminosity_1} temperature_1:{payload.payload_fields.temperature_1}");

      return this.Ok();
   }
}

I then updated the TTN application integration to send messages to my new endpoint. In the body of the Application Insights events I could see the devEUI, port, and the payload fields had been extracted from the message.

DevEUI:1234567890123456 Payload Base64:AWcBEAFlAGQBAAEBAgAyAYgAqYgGIxgBJuw= analog_in_1:0.5 digital_in_1:1 gps_1:4.34,40.22,755 luminosity_1:100 temperature_1:27.2

This arrangement was pretty nasty and sort of worked but in the “real world” would not have been viable. I would need to generate lots of custom classes for each application taking into account the channel numbers (e,g, analog_in_1,analog_in_2) and datatypes used.

I also explored which datatypes were supported by the TTN decoder, after some experimentation (Aug 2019) it looks like only the LPPV1 ones are.

AnalogInput
AnalogOutput
DigitalInput
DigitalOutput
GPS
Accelerometer
Gyrometer
Luminosity
Presence
BarometricPressure
RelativeHumidity
Temperature

What I need is a more flexible way to stored and decode payload_fields property..

The Things Network HTTP Integration Part2

Posted on September 1, 2020 by devmobilenz

Basic JSON Deserialisation

This is the second in a series of posts about building an HTTP Integration for a The Things Network(TTN) application.

I used JSON2Csharp and a payload I downloaded in Part 1 to generate C# classes which would deserialise my minimalist messages.

// First version of classes for unpacking HTTP payload https://json2csharp.com/
public class GatewayV1
{
   public string gtw_id { get; set; }
   public int timestamp { get; set; }
   public DateTime time { get; set; }
   public int channel { get; set; }
   public int rssi { get; set; }
   public double snr { get; set; }
   public int rf_chain { get; set; }
   public double latitude { get; set; }
   public double longitude { get; set; }
   public int altitude { get; set; }
}

public class MetadataV1
{
   public string time { get; set; }
   public double frequency { get; set; }
   public string modulation { get; set; }
   public string data_rate { get; set; }
   public string coding_rate { get; set; }
   public List<GatewayV1> gateways { get; set; }
}

public class PayloadV1
{
   public string app_id { get; set; }
   public string dev_id { get; set; }
   public string hardware_serial { get; set; }
   public int port { get; set; }
   public int counter { get; set; }
   public bool confirmed { get; set; }
   public string payload_raw { get; set; }
   public MetadataV1 metadata { get; set; }
   public string downlink_url { get; set; }
}

I added a new to controller to my application which used the generated classes to deserialise the body of the POST from the TTN Application Integration.

[Route("[controller]")]
[ApiController]
public class ClassSerialisationV1 : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public IActionResult Post([FromBody] PayloadV1 payload)
   {
      // Check that the post data is good
      if (!this.ModelState.IsValid)
      {
         log.WarnFormat("ClassSerialisationV1 validation failed {0}", this.ModelState.Messages());

         return this.BadRequest(this.ModelState);
      }
      log.Info($"DevEUI:{payload.hardware_serial} Payload Base64:{payload.payload_raw}");

      return this.Ok();
   }
}

I then updated the TTN application integration to send messages to my new endpoint.

In the body of the Application Insights events I could see the devEUI, port, and the raw payload had been extracted from the message.

DevEUI:1234567890123456 Port:1 Payload Base64:VGlueUNMUiBMb1JhV0FO

I then added another controller which decoded the Base64 encoded payload_raw.

[Route("[controller]")]
[ApiController]
public class ClassSerialisationV2Base64Decoded : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public IActionResult Post([FromBody] PayloadV2 payload)
   {
      // Check that the post data is good
      if (!this.ModelState.IsValid)
      {
         log.WarnFormat("ClassSerialisationV2BCDDecoded validation failed {0}", this.ModelState.Messages());

         return this.BadRequest(this.ModelState);
      }

      log.Info($"DevEUI:{payload.hardware_serial} Port:{payload.port} Payload:{ Encoding.UTF8.GetString(Convert.FromBase64String(payload.payload_raw))}");

      return this.Ok();
   }
}

DevEUI:1234567890123456 Port:1 Payload:TinyCLR LoRaWAN

Then after a while the deserialisation started to fail with an HTTP 400-Bad request. When I ran the same request with Telerik Fiddler on my desktop the raw response was

HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Content-Type: application/problem+json; charset=utf-8
Server: Microsoft-IIS/10.0
Request-Context: appId=cid-v1:f4f72f2e-1144-4578-923f-d3ebdcfb7766
X-Powered-By: ASP.NET
Date: Mon, 31 Aug 2020 09:07:30 GMT

17a
{"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1",
"title":"One or more validation errors occurred.",
"status":400,
"traceId":"00-45033ec030b63d4ebb82b95b67cb8142-9fc52a18be202848-00",
"errors":{
"$.metadata.gateways[0].timestamp":["The JSON value could not be converted to System.Int32. 
Path: $.metadata.gateways[0].timestamp | LineNumber: 21 | BytePositionInLine: 35."]}}
0

The line in the payload was the gateway timestamp. The value was 2,426,973,100 which larger than 2,147,483,647 the maximum number that can be stored in a signed 32 bit integer. The JSON2CSharp generator had made a reasonable choice of datatype but in this case the range was not sufficient.

public class GatewayV2 
{
   public string gtw_id { get; set; }
   public ulong timestamp { get; set; }
   public DateTime time { get; set; }
   public int channel { get; set; }
   public int rssi { get; set; }
   public double snr { get; set; }
   public int rf_chain { get; set; }
   public double latitude { get; set; }
   public double longitude { get; set; }
   public int altitude { get; set; }
}

I checked the TTN code where the variable was declared as an unsigned 64 bit integer.

This issue could occur for other variables so I need to manually check all the generated classes.

The Things Network HTTP Integration Part1

Posted on August 31, 2020 by devmobilenz

Infrastructure and payloads

This is the first in a series of posts about building an HTTP Integration for a The Things Network(TTN) application. I have assumed that readers are familiar with the configuration and operation of a TTN instance so I’m not going to cover that in detail.

I’m using a Seeeduino LoRaWAN device as a client so I configured the sample Over the Air Activation(OTAA) application to connect to my local RAK7246 Developer gateway.

#include <LoRaWan.h>

unsigned char data[] = {0x53, 0x65, 0x65, 0x65, 0x64, 0x75, 0x69, 0x6E, 0x6F, 0x20, 0x4C, 0x6F, 0x52, 0x61, 0x57, 0x41, 0x4E};
char buffer[256];

void setup(void)
{
  SerialUSB.begin(9600);
  while (!SerialUSB);

  lora.init();

  memset(buffer, 0, 256);
  lora.getVersion(buffer, 256, 1);
  SerialUSB.print("Ver:");
  SerialUSB.print(buffer);

  memset(buffer, 0, 256);
  lora.getId(buffer, 256, 1);
  SerialUSB.print(buffer);
  SerialUSB.print("ID:");

  lora.setKey(NULL, NULL, "12345678901234567890123456789012");
  lora.setId(NULL, "1234567890123456", "1234567890123456");

  lora.setPort(10);

  lora.setDeciveMode(LWOTAA);
  lora.setDataRate(DR0, AS923);

  lora.setDutyCycle(false);
  lora.setJoinDutyCycle(false);

  lora.setPower(14);


  while (!lora.setOTAAJoin(JOIN, 10))
  {
    SerialUSB.println("");
  }
    SerialUSB.println( "Joined");
}

void loop(void)
{
  bool result = false;

  //result = lora.transferPacket("Hello World!", 10);
  result = lora.transferPacket(data, sizeof(data));

  if (result)
  {
    short length;
    short rssi;

    memset(buffer, 0, 256);
    length = lora.receivePacket(buffer, 256, &rssi);

    if (length)
    {
      SerialUSB.print("Length is: ");
      SerialUSB.println(length);
      SerialUSB.print("RSSI is: ");
      SerialUSB.println(rssi);
      SerialUSB.print("Data is: ");
      for (unsigned char i = 0; i < length; i ++)
      {
        SerialUSB.print("0x");
        SerialUSB.print(buffer[i], HEX);
        SerialUSB.print(" ");
      }
      SerialUSB.println();
    }
  }
  delay( 10000);
}

The SetKey and SetId parameters are not obvious and it would be much easier if there were two methods one for OTTA and the other for Activation by-Personalization(ABP).
I then built an Net Core 3.1 Web API application which had a single controller to receive messages from TTN.

[Route("[controller]")]
[ApiController]
public class Raw : ControllerBase
{
   private static readonly ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   [HttpGet]
   public string Index()
   {
      return "move along nothing to see";
   }

   [HttpPost]
   public void PostRaw([FromBody]JsonElement body)
   {
      string json = JsonSerializer.Serialize(body);

      log.Info(json);
   }
}

I then configured my TTN application integration to send messages to my shinny new endpoint

My controller logged events to Azure application Insights so I could see if there were any errors and inspect message payloads. The TTN developers provide sample payloads to illustrate the message format but they were a bit chunky for my initial testing.

I could then inspect individual events and payloads

At this point I could download message payloads and save them locally.

{
   "app_id": "rak811wisnodetest",
   "dev_id": "rak811wisnode1",
   "hardware_serial": "1234567890123456",
   "port": 1,
   "counter": 2,
   "confirmed": true,
   "payload_raw": "VGlueUNMUiBMb1JhV0FO",
   "metadata": {
      "time": "2020-08-26T00:50:36.182774822Z",
      "frequency": 924.2,
      "modulation": "LORA",
      "data_rate": "SF7BW125",
      "coding_rate": "4/5",
      "gateways": [
         {
            "gtw_id": "eui-b827ebfffe6c279d",
            "timestamp": 1584148244,
            "time": "2020-08-26T00:50:35.012774Z",
            "channel": 5,
            "rssi": -63,
            "snr": 9.2,
            "rf_chain": 0,
            "latitude": -43.49889,
            "longitude": 172.60104,
            "altitude": 16
         }
      ]
   },
   "downlink_url": "https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/rak811wisnodetest/azure-webapi-endpoint?key=ttn-account-v2.12345678901234567_12345_1234567-dduo"

}

These were useful because I could then use a tool like Telerik Fiddler to submit messages to my application when it was running locally in the Visual Studio 2019 debugger.

Azure IoT Hub, Event Grid to Application Insights

Posted on February 2, 2020 by devmobilenz

For a second Proof of Concept (PoC) I wanted to upload sensor data from my MQTT LoRa Telemetry Field Gateway to an Azure IoT Hub, then using Azure EventGrid subscribe to the stream of telemetry data events, logging the payloads in Azure Application Insights (the aim was minimal code so no database etc.).

The first step was to create and deploy a simple Azure Function for unpacking the telemetry event payload.

Then wire the Azure function to the Microsoft.Devices.Device.Telemetry Event Type

On the Windows 10 IoT Core device in the Event Tracing Window(ETW) logging on the device I could see LoRa messages arriving and being unpacked.

Windows 10 Device ETW showing message payload

Then in Application Insights after some mucking around with code I could see in a series of Trace statements the event payload as it was unpacked.

{"id":"29108ebf-e5d5-7b95-e739-7d9048209d53","topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-123456789012/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/MQTTNetClient",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-02-01T04:30:51.427Z",
"data":
{
 "properties":{},
"systemProperties":{"iothub-connection-device-id":"MQTTNetClient","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149890997219611",
"iothub-enqueuedtime":"2020-02-01T04:30:51.427Z",
"iothub-message-source":"Telemetry"
},
"body":"eyJPZmZpY2VUZW1wZXJhdHVyZSI6IjIyLjUiLCJPZmZpY2VIdW1pZGl0eSI6IjkyIn0="
},
"dataVersion":"",
"metadataVersion":"1"
}

Application Insights logging with message unpacking

Application Insights logging message payload

Then in the last log entry the decoded message payload

/*
    Copyright ® 2020 Feb devMobile Software, All Rights Reserved
 
    MIT License

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE

    Default URL for triggering event grid function in the local environment.
    http://localhost:7071/runtime/webhooks/EventGrid?functionName=functionname
 */
namespace EventGridProcessorAzureIotHub
{
   using System;
   using System.IO;
   using System.Reflection;

   using Microsoft.Azure.WebJobs;
   using Microsoft.Azure.EventGrid.Models;
   using Microsoft.Azure.WebJobs.Extensions.EventGrid;

   using log4net;
   using log4net.Config;
   using Newtonsoft.Json;

   public static class Telemetry
    {
        [FunctionName("Telemetry")]
        public static void Run([EventGridTrigger]Microsoft.Azure.EventGrid.Models.EventGridEvent eventGridEvent, ExecutionContext executionContext )//, TelemetryClient telemetryClient)
        {
			ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

		   var logRepository = LogManager.GetRepository(Assembly.GetEntryAssembly());
			XmlConfigurator.Configure(logRepository, new FileInfo(Path.Combine(executionContext.FunctionAppDirectory, "log4net.config")));

         log.Info($"eventGridEvent.Data-{eventGridEvent}");

         log.Info($"eventGridEvent.Data.ToString()-{eventGridEvent.Data.ToString()}");

        IotHubDeviceTelemetryEventData iOThubDeviceTelemetryEventData = (IotHubDeviceTelemetryEventData)JsonConvert.DeserializeObject(eventGridEvent.Data.ToString(), typeof(IotHubDeviceTelemetryEventData));

         log.Info($"iOThubDeviceTelemetryEventData.Body.ToString()-{iOThubDeviceTelemetryEventData.Body.ToString()}");

         byte[] base64EncodedBytes = System.Convert.FromBase64String(iOThubDeviceTelemetryEventData.Body.ToString());

         log.Info($"System.Text.Encoding.UTF8.GetString(-{System.Text.Encoding.UTF8.GetString(base64EncodedBytes)}");
      }
	}
}

Overall it took roughly half a page of code (mainly generated by a tool) to unpack and log the contents of an Azure IoT Hub EventGrid payload to Application Insights.

Azure IOT Hub and Event Grid Part1

Posted on January 24, 2020 by devmobilenz

I have one an Azure IoT Hub LoRa Telemetry Field Gateway running in my office and I wanted to process the data collected by the sensors around my property without using a Software as a Service(SaaS) Internet of Things (IoT) package.

Rather than lots of screen grabs of my configuration steps I figured people reading this series of posts would be able to figure the details out themselves.

I created an Azure Resource Group for this project, and created an Azure IoT Hub.

I then provisioned an Azure IoT Hub device so I could get the connection string for my Windows 10 Azure IoT Hub LoRa Telemetry Field gateway.

LoRa Field Gateway Provisioned in Azure IoT Hub

I downloaded the JSON configuration file template from my Windows 10 device (which is created on first startup after installation) and configured the Azure IoT Hub connection string.

{
   "AzureIoTHubDeviceConnectionString": "HostName=FieldGatewayHub.azure-devices.net;DeviceId=LoRa915MHz;SharedAccessKey=123456789012345678901234567890123456789/arg=",
   "AzureIoTHubTransportType": "amqp",
   "SensorIDIsDeviceIDSensorID": false,
   "Address": "LoRaIoT1",
   "Frequency": 915000000.0,
   "PABoost": true
}

I then uploaded this to my Windows 10 IoT Core device and restarted the Azure IoT Hub Field gateway so it picked up the new settings.

I could then see on the device messages from sensor nodes being unpacked and uploaded to my Azure IoT Hub.

In the Azure IoT Hub metrics I graphed the number of devices connected and the number of telemetry messages sent and could see my device connect then start uploading telemetry.

One of my customers uses Azure Event Grid for application integration and I wanted to explore using it in an IoT solution. The first step was to create an Event Grid Domain.

I then used the Azure IoT Hub Events tab to wire up these events.

Microsoft.Devices.DeviceConnected
Microsoft.Devices.DeviceDisconnected
Microsoft.Devices.DeviceTelemetry

To confirm my event subscriptions were successful I previously found the “simplest” approach was to use an Azure storage queue endpoint. I had to create an Azure Storage Account with two Azure Storage Queues one for device connectivity (.DeviceConnected & .DeviceDisconnected) events and the other for device telemetry (.DeviceTelemetry) events.

I created a couple of other subscriptions so I could compare the different Event schemas (Event Grid Schema & Cloud Event Schema v1.0). At this stage I didn’t configure any Filters or Additional Features.

I use Cerebrate Cerculean for monitoring and managing a couple of other customer projects so I used it to inspect the messages in the storage queues.

Cerebrate Ceculean Storage queue Inspector

The message are quite verbose

{
"id":"b48b6376-b7f4-ee7d-82d9-12345678901a",
"source":"/SUBSCRIPTIONS/12345678-901234789-0123-456789012345/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-24T04:27:30.842Z","data":
{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-24T04:27:30.842Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTY5LCJSU1NJIjotMTA5LCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijc2LjYiLCJhaCI6IjU4Iiwid3NhIjoiMiIsIndzZyI6IjUiLCJ3ZCI6IjMyMi44OCIsInIiOiIwLjAwIn0="
}
}

The message payload is base64 encoded, so I used an online tool to decode it.

{
 PacketSNR":"10.0",
"PacketRSSI":-69,
"RSSI":-109,
"DeviceAddressBCD":"4D-61-64-75-69-6E-6F-32",
"at":"76.6",
"ah":"58",
"wsa":"2",
"wsg":"5",
"wd":"322.88",
"r":"0.00"
}

Without writing any code (I will script the configuration) I could upload sensor data to an Azure IoT Hub, subscribe to a selection of events the Azure IoT Hub publishes and then inspect them in an Azure Storage Queue.

I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the device I would get several DeviceTelemetry events before the DeviceConnected event arrived.

devMobile's blog

Random wanderings through Microsoft Azure, Windows 10 Iot Core, .Net Micro Framework, .Net Nano Framework and GHI Electronics TinyCLR

Tag Archives: Base64