Azure IoT Hub, Event Grid to Application Insights

For a second Proof of Concept (PoC) I wanted to upload sensor data from my MQTT LoRa Telemetry Field Gateway to an Azure IoT Hub, then using Azure EventGrid subscribe to the stream of telemetry data events, logging the payloads in Azure Application Insights (the aim was minimal code so no database etc.).

The first step was to create and deploy a simple Azure Function for unpacking the telemetry event payload.

Azure IoT Hub Azure Function Handler

Then wire the Azure function to the Microsoft.Devices.Device.Telemetry Event Type

Azure IoT Hub Event Metrics

On the Windows 10 IoT Core device in the Event Tracing Window(ETW) logging on the device I could see LoRa messages arriving and being unpacked.

Windows 10 Device ETW showing message payload

Then in Application Insights after some mucking around with code I could see in a series of Trace statements the event payload as it was unpacked.

{"id":"29108ebf-e5d5-7b95-e739-7d9048209d53","topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-123456789012/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/MQTTNetClient",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-02-01T04:30:51.427Z",
"data":
{
 "properties":{},
"systemProperties":{"iothub-connection-device-id":"MQTTNetClient","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149890997219611",
"iothub-enqueuedtime":"2020-02-01T04:30:51.427Z",
"iothub-message-source":"Telemetry"
},
"body":"eyJPZmZpY2VUZW1wZXJhdHVyZSI6IjIyLjUiLCJPZmZpY2VIdW1pZGl0eSI6IjkyIn0="
},
"dataVersion":"",
"metadataVersion":"1"
}
Application Insights logging with message unpacking
Application Insights logging message payload

Then in the last log entry the decoded message payload

/*
    Copyright ® 2020 Feb devMobile Software, All Rights Reserved
 
    MIT License

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE

    Default URL for triggering event grid function in the local environment.
    http://localhost:7071/runtime/webhooks/EventGrid?functionName=functionname
 */
namespace EventGridProcessorAzureIotHub
{
   using System;
   using System.IO;
   using System.Reflection;

   using Microsoft.Azure.WebJobs;
   using Microsoft.Azure.EventGrid.Models;
   using Microsoft.Azure.WebJobs.Extensions.EventGrid;

   using log4net;
   using log4net.Config;
   using Newtonsoft.Json;

   public static class Telemetry
    {
        [FunctionName("Telemetry")]
        public static void Run([EventGridTrigger]Microsoft.Azure.EventGrid.Models.EventGridEvent eventGridEvent, ExecutionContext executionContext )//, TelemetryClient telemetryClient)
        {
			ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

		   var logRepository = LogManager.GetRepository(Assembly.GetEntryAssembly());
			XmlConfigurator.Configure(logRepository, new FileInfo(Path.Combine(executionContext.FunctionAppDirectory, "log4net.config")));

         log.Info($"eventGridEvent.Data-{eventGridEvent}");

         log.Info($"eventGridEvent.Data.ToString()-{eventGridEvent.Data.ToString()}");

        IotHubDeviceTelemetryEventData iOThubDeviceTelemetryEventData = (IotHubDeviceTelemetryEventData)JsonConvert.DeserializeObject(eventGridEvent.Data.ToString(), typeof(IotHubDeviceTelemetryEventData));

         log.Info($"iOThubDeviceTelemetryEventData.Body.ToString()-{iOThubDeviceTelemetryEventData.Body.ToString()}");

         byte[] base64EncodedBytes = System.Convert.FromBase64String(iOThubDeviceTelemetryEventData.Body.ToString());

         log.Info($"System.Text.Encoding.UTF8.GetString(-{System.Text.Encoding.UTF8.GetString(base64EncodedBytes)}");
      }
	}
}

Overall it took roughly half a page of code (mainly generated by a tool) to unpack and log the contents of an Azure IoT Hub EventGrid payload to Application Insights.

Azure IoT Hub MQTT+TLS Overheads

An Azure IoT Hub has a series of metrics and one I had been using was “Total Device Data Usage”. To better understand what it was displaying I modified my Azure IoT Hub MQTT Test Application to display the size of the JOSN payload.

MQTTNet based client displaying payload length

The size of the packets sent and the total device data appeared to map pretty well but I was also interested in the Transport Layer Security (TLS) and Messaging Queuing Telemetry Transport (MQTT) overheads.

Azure IoT Hub Metrics

To get an idea of the overheads I fired up LiveTcpUdpWatch by Nirsoft and noted down the traffic measure on port 8883.

Conenction LiveTcpUdpWatch main screen

Launching the MQTTNet client sending every 30 seconds resulted in traffic like this

4179b - Establishing connection
4284b - 105b
4317b - 33b
4386b - 69b
4455b - 69b
4524b - 69b
4593b - 69b
4662b - 69b
4731b - 69b
4800b - 69b
4869b - 69b
4938b - 69b
5007b - 69b
5076b - 69b
5145b - 69b
5214b - 69b
5288b - 69b

So it looks like my very rough numbers are close to the numbers discussed in the above article. I need to explore the impact of keep-alive messages and other background operations.

Azure IOT Hub and Event Grid Part2

I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the Windows 10 IoT Core device I would get several DeviceTelemetry events before the DeviceConnected event arrived.

I was using Advanced Message Queueing Protocol (AMQP) so I modified the configuration file so I could try all the available options.

C# TransportType enumeration

namespace Microsoft.Azure.Devices.Client
{
	//
	// Summary:
	//     Transport types supported by DeviceClient - AMQP/TCP, HTTP 1.1, MQTT/TCP, AMQP/WS,
	//     MQTT/WS
	public enum TransportType
	{
		//
		// Summary:
		//     Advanced Message Queuing Protocol transport. Try Amqp over TCP first and fallback
		//     to Amqp over WebSocket if that fails
		Amqp = 0,
		//
		// Summary:
		//     HyperText Transfer Protocol version 1 transport.
		Http1 = 1,
		//
		// Summary:
		//     Advanced Message Queuing Protocol transport over WebSocket only.
		Amqp_WebSocket_Only = 2,
		//
		// Summary:
		//     Advanced Message Queuing Protocol transport over native TCP only
		Amqp_Tcp_Only = 3,
		//
		// Summary:
		//     Message Queuing Telemetry Transport. Try Mqtt over TCP first and fallback to
		//     Mqtt over WebSocket if that fails
		Mqtt = 4,
		//
		// Summary:
		//     Message Queuing Telemetry Transport over Websocket only.
		Mqtt_WebSocket_Only = 5,
		//
		// Summary:
		//     Message Queuing Telemetry Transport over native TCP only
		Mqtt_Tcp_Only = 6
	}
}

Windows 10 IoT Core LoRa Telemetry Field gateway config.json file

{
   "AzureIoTHubDeviceConnectionString": "HostName=FieldGatewayHub.azure-devices.net;DeviceId=LoRa915MHz;SharedAccessKey=y12345678901234567890123456789012345678/arg=",

   "AzureIoTHubTransportType-Amqp": "amqp",
   "AzureIoTHubTransportType-Http1": "Http1",
   "AzureIoTHubTransportType-Amqp_WebSocket_Only": "Amqp_WebSocket_Only",
   "AzureIoTHubTransportType-Amqp_Tcp_Only": "Amqp_Tcp_Only",
   "AzureIoTHubTransportType-Mqtt": "Mqtt",
   "AzureIoTHubTransportType-Mqtt_WebSocket_Only": "Mqtt_WebSocket_Only",
   "AzureIoTHubTransportType-Amqp": "Mqtt_Tcp_Only",

   "AzureIoTHubTransportType": "Mqtt_Tcp_Only",
   "SensorIDIsDeviceIDSensorID": false,
   "Address": "LoRaIoT1",
   "Frequency": 915000000.0,
   "PABoost": true
}

So in alphabetical order here are my not terribly scientific results

AMQP

Device ETW

Device Telemetry messages

{
"id":"07e9a772-d86a-963d-139a-9d2ea2a0866c",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry","eventTime":"2020-01-25T00:57:18.477Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T00:57:18.477Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTcxLCJSU1NJIjotMTEwLCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijg0LjAiLCJhaCI6IjUwIiwid3NhIjoiMSIsIndzZyI6IjMiLCJ3ZCI6IjE5My44OCIsInIiOiIwLjAwIn0="},
"dataVersion":"",
"metadataVersion":"1"
}
…
{
"id":"ca8e8531-10da-ec99-aad7-49e68f3f9500","topic":"/SUBSCRIPTIONS/712345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T01:02:23.377Z",
"data":{
"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T01:02:23.377Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjgiLCJQYWNrZXRSU1NJIjotNzAsIlJTU0kiOi0xMDgsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODUuMSIsImFoIjoiNTEiLCJ3c2EiOiIxIiwid3NnIjoiMyIsIndkIjoiMjM5LjI1IiwiciI6IjAuMDAifQ=="},
"dataVersion":"",
"metadataVersion":"1"
}

Device connectivity messages

{
"id":"d8a393ff-6549-69d2-d728-37eee2437693",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:01:28.4887191Z","data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000001E"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"b345d18a-bdf5-3397-35c1-fd5c35046d85",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:02:24.9605306Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000001F"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 00:57:18, the DeviceConnected arrived 01:01:28 so approximately a 4 minute delay, the DeviceDisconnected arrived within a minute of me shutting the device down.

Amqp-Tcp-Only

Device Telemetry messages

{
"id":"b8cdbc73-5cb8-134c-a328-beed47be5f27",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB","subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:16:48.732Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:16:48.732Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzIsIlJTU0kiOi0xMDcsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODYuNSIsImFoIjoiNDUiLCJ3c2EiOiIxIiwid3NnIjoiMiIsIndkIjoiOTkuNzUiLCJyIjoiMC4wMCJ9"},
"dataVersion":"",
"metadataVersion":"1"
}
...
{
"id":"56aaf46d-e1ee-9419-2621-aa97c0564778",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:21:53.402Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:21:53.402Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMDksIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODQuOSIsImFoIjoiNDYiLCJ3c2EiOiIzIiwid3NnIjoiNyIsIndkIjoiMjE3LjUwIiwiciI6IjAuMDAifQ=="},
"dataVersion":"",
"metadataVersion":"1"
}

Device connectivity messages

{
"id":"a157468c-7d65-00b1-73d6-12e43fd1356b",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:20:39.7309538Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000026"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"2e6d7e57-3db3-9e1d-c01b-b0f787b16e05",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:22:33.5276985Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000027"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:16:48, the DeviceConnected arrived 04:20:39 so approximately a 4 minute delay, the DeviceDisconnected arrived within a minute of me shutting the device down.

Amqp WebSocket Only

Device Telemetry messages

{
"id":"f82943da-425c-f49c-49b0-13ff2609544b",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:05:36.723Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:05:36.723Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjMiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMDksIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODUuMyIsImFoIjoiNDMiLCJ3c2EiOiIyIiwid3NnIjoiMyIsIndkIjoiMjcyLjYzIiwiciI6IjAuMDAifQ=="}
}
...
{
"id":"8419d4f7-3340-8e29-6370-29f95d40e68c",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:11:42.684Z",
"data":{"properties":{},"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:11:42.684Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjgiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMTIsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODcuMyIsImFoIjoiNDMiLCJ3c2EiOiIxIiwid3NnIjoiMyIsIndkIjoiMjcuMDAiLCJyIjoiMC4wMCJ9"}
}

Device connectivity messages

{
"id":"7f10a3e3-0c2c-2b18-e38c-d5ea498304ab",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:09:52.5786541Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000024"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{"id":"474a695b-8c4f-e3fe-0b1b-b6bc6b6d4dbe",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:11:53.076926Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000025"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:05:36, DeviceConnected arrived 04:09:52 so approximately a 4 minute delay, the DeviceDisconnected arrived within a minute of me shutting the device down.

HTTP

I waited for 20 minutes and there wasn’t a DeviceConnected message which I sort of expected as HTTP is a connectionless protocol.

MQTT

Device ETW

Device Telemetry messages

{
"id":"bc26c412-c694-3954-5888-baa118cc9f88",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz","time":"2020-01-25T01:11:33.493Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T01:11:33.493Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMTEsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODUuNiIsImFoIjoiNTAiLCJ3c2EiOiIyIiwid3NnIjoiNCIsIndkIjoiMjg1LjAwIiwiciI6IjAuMDAifQ=="}
}
...
{
"id":"de95b90f-ac96-5c76-7a46-00f5a0eef8cf",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:12:12.101Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T01:12:12.101Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTU4LCJSU1NJIjotMTA5LCJEZXZpY2VBZGRyZXNzQkNEIjoiMDEtMjMtMjktMDctMjMtMEMtODgtNjMtRUUiLCJoIjoiMzQiLCJ0IjoiNDEuOCIsInMiOiI0IiwidiI6IjQuMDcifQ=="}
}

Device connectivity messages

{
"id":"f8f5ee54-394d-05e3-784d-87bc648e8267",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:11:25.2530139Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000020"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"82d6aa6c-4c71-9623-a4ac-9e562345afad",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:12:26.6368519Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000021"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 01:11:33, the DeviceConnected arrived 01:11:25 so they arrived in order and within 10 seconds, the DeviceDisconnected arrived within a 15 seconds of me shutting the device down.

Mqtt-TCP-Only

Device ETW

Device Telemetry messages

{
"id":"bb86bfd9-6d12-4a27-2444-bd6953be7ffd",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:42:15.345Z",
"data":{"properties":{},"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}","iothub-connection-auth-generation-id":"637149227434620853","iothub-enqueuedtime":"2020-01-25T04:42:15.345Z","iothub-message-source":"Telemetry"},"body":"eyJQYWNrZXRTTlIiOiIxMC44IiwiUGFja2V0UlNTSSI6LTcxLCJSU1NJIjotMTA4LCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijg2LjkiLCJhaCI6IjQ1Iiwid3NhIjoiMSIsIndzZyI6IjMiLCJ3ZCI6IjI5LjYzIiwiciI6IjAuMDAifQ=="},"dataVersion":"","metadataVersion":"1"
}
...
{
"id":"c5991ed8-42f2-437a-9161-a526248c955f",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB","subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:44:16.986Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:44:16.986Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTcxLCJSU1NJIjotMTEyLCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijg4LjIiLCJhaCI6IjQ0Iiwid3NhIjoiMiIsIndzZyI6IjMiLCJ3ZCI6IjYxLjUwIiwiciI6IjAuMDAifQ=="},
"dataVersion":"",
"metadataVersion":"1"}
}

Device connectivity messages

{
"id":"7861f3c1-5f1d-e9a1-c214-19feea2bf1a3","source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:42:06.0123436Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000002A"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"c2090a7b-c827-73cc-d1e8-7d49fe4a03a1",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:44:48.816748Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000002B"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:42:15, the DeviceConnected arrived 04:42:06 so they arrived in order and within 10 seconds, the DeviceDisconnected arrived within a 20 seconds of me shutting device down.

Mqtt-Web-Socket-Only

Device ETW

Device Telemetry messages

{
"id":"3afa1a9c-f30d-d051-077e-cd25cc3b2245",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:36:08.871Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:36:08.871Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzAsIlJTU0kiOi0xMTIsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODMuOCIsImFoIjoiNDciLCJ3c2EiOiIwIiwid3NnIjoiMSIsIndkIjoiMzQ3LjI1IiwiciI6IjAuMDAifQ=="}
}
...
{
"id":"fa082b67-db32-312d-e716-1b1d37f57d94",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:37:09.516Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:37:09.516Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjgiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMTEsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODQuNCIsImFoIjoiNDciLCJ3c2EiOiIxIiwid3NnIjoiMiIsIndkIjoiMzU4LjUwIiwiciI6IjAuMDAifQ=="}
}

Device connectivity messages

{
"id":"245e4d68-06e6-4d76-1167-39a9a67b01ac",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz","time":"2020-01-25T04:36:03.8275263Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000028"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"9612e8af-84e0-a679-4c7d-28c5e968da3c",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:37:47.5794771Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000029"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:36:08, the DeviceConnected arrived 04:36:03 so they arrived in order and within 10 seconds, the DeviceDisconnected arrived within a 30 seconds of me shutting device down.

Summary

My LoRa sensors nodes are sending data roughly every minute which reduces the precision of the times.

It looks like for AMQP based messaging it can take 4-5 minutes for a Devices.DeviceConnected message to arrive, for based MQTT messaging it’s 5-10 seconds.

Azure IOT Hub and Event Grid Part1

I have one an Azure IoT Hub LoRa Telemetry Field Gateway running in my office and I wanted to process the data collected by the sensors around my property without using a Software as a Service(SaaS) Internet of Things (IoT) package.

Rather than lots of screen grabs of my configuration steps I figured people reading this series of posts would be able to figure the details out themselves.

Raspberry PI with M2M LoRa Hat

I created an Azure Resource Group for this project, and created an Azure IoT Hub.

Azure Resource Group with IoT Hub

I then provisioned an Azure IoT Hub device so I could get the connection string for my Windows 10 Azure IoT Hub LoRa Telemetry Field gateway.

LoRa Field Gateway Provisioned in Azure IoT Hub

I downloaded the JSON configuration file template from my Windows 10 device (which is created on first startup after installation) and configured the Azure IoT Hub connection string.

{
   "AzureIoTHubDeviceConnectionString": "HostName=FieldGatewayHub.azure-devices.net;DeviceId=LoRa915MHz;SharedAccessKey=123456789012345678901234567890123456789/arg=",
   "AzureIoTHubTransportType": "amqp",
   "SensorIDIsDeviceIDSensorID": false,
   "Address": "LoRaIoT1",
   "Frequency": 915000000.0,
   "PABoost": true
}

I then uploaded this to my Windows 10 IoT Core device and restarted the Azure IoT Hub Field gateway so it picked up the new settings.

I could then see on the device messages from sensor nodes being unpacked and uploaded to my Azure IoT Hub.

ETW logging on device

In the Azure IoT Hub metrics I graphed the number of devices connected and the number of telemetry messages sent and could see my device connect then start uploading telemetry.

Azure IoT Hub metrics

One of my customers uses Azure Event Grid for application integration and I wanted to explore using it in an IoT solution. The first step was to create an Event Grid Domain.

I then used the Azure IoT Hub Events tab to wire up these events.

  • Microsoft.Devices.DeviceConnected
  • Microsoft.Devices.DeviceDisconnected
  • Microsoft.Devices.DeviceTelemetry
Azure IoT Hub Event Metrics

To confirm my event subscriptions were successful I previously found the “simplest” approach was to use an Azure storage queue endpoint. I had to create an Azure Storage Account with two Azure Storage Queues one for device connectivity (.DeviceConnected & .DeviceDisconnected) events and the other for device telemetry (.DeviceTelemetry) events.

I created a couple of other subscriptions so I could compare the different Event schemas (Event Grid Schema & Cloud Event Schema v1.0). At this stage I didn’t configure any Filters or Additional Features.

Azure IoT Hub Telemetry Event Metrics

I use Cerebrate Cerculean for monitoring and managing a couple of other customer projects so I used it to inspect the messages in the storage queues.

Cerebrate Ceculean Storage queue Inspector

The message are quite verbose

{
"id":"b48b6376-b7f4-ee7d-82d9-12345678901a",
"source":"/SUBSCRIPTIONS/12345678-901234789-0123-456789012345/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-24T04:27:30.842Z","data":
{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-24T04:27:30.842Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTY5LCJSU1NJIjotMTA5LCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijc2LjYiLCJhaCI6IjU4Iiwid3NhIjoiMiIsIndzZyI6IjUiLCJ3ZCI6IjMyMi44OCIsInIiOiIwLjAwIn0="
}
}

The message payload is base64 encoded, so I used an online tool to decode it.

{
 PacketSNR":"10.0",
"PacketRSSI":-69,
"RSSI":-109,
"DeviceAddressBCD":"4D-61-64-75-69-6E-6F-32",
"at":"76.6",
"ah":"58",
"wsa":"2",
"wsg":"5",
"wd":"322.88",
"r":"0.00"
}

Without writing any code (I will script the configuration) I could upload sensor data to an Azure IoT Hub, subscribe to a selection of events the Azure IoT Hub publishes and then inspect them in an Azure Storage Queue.

I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the device I would get several DeviceTelemetry events before the DeviceConnected event arrived.

Armtronix IA005 SX1276 loRa node

A month ago I ordered a pair of IA005: SX1276 Lora node STM32F103 devices from the Armtronix shop on Tindie for evaluation. At USD18 each these devices were competitively priced and I was interested in trialling another maple like device.

Bill of materials (Prices as at December 2019)

  • IA005 SX1276 loRa node USD36 (USD18 each)
  • Grove – Temperature&Humidity Sensor USD11.5
  • Grove – 4 pin Female Jumper to Grove 4 pin Conversion Cable USD3.90
Armtronix device with Seeedstudio temperature & humidity sensor

I used a modified version of my Arduino client code which worked after I got the pins sorted and the female jumper sockets in the right order.

/*
  Copyright ® 2019 December devMobile Software, All Rights Reserved

  THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
  KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
  IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR
  PURPOSE.

  You can do what you want with this code, acknowledgment would be nice.

  http://www.devmobile.co.nz

*/
#include <stdlib.h>
#include <LoRa.h>
#include <TH02_dev.h>

//#define DEBUG
//#define DEBUG_TELEMETRY
//#define DEBUG_LORA

// LoRa field gateway configuration (these settings must match your field gateway)
const char FieldGatewayAddress[] = {"LoRaIoT1"};
const char DeviceAddress[] = {"ArmTronix01"};
const float FieldGatewayFrequency =  915000000.0;
const byte FieldGatewaySyncWord = 0x12 ;

// Payload configuration
const int ChipSelectPin = PA4;
const int InterruptPin = PA11;
const int ResetPin = PC13;

// LoRa radio payload configuration
const byte SensorIdValueSeperator = ' ' ;
const byte SensorReadingSeperator = ',' ;
const int LoopSleepDelaySeconds = 30 ;

const byte PayloadSizeMaximum = 64 ;
byte payload[PayloadSizeMaximum];
byte payloadLength = 0 ;


void setup()
{
  Serial.begin(9600);
#ifdef DEBUG
  while (!Serial);
#endif
  Serial.println("Setup called");

  Serial.println("LoRa setup start");
  
  // override the default chip select and reset pins
  LoRa.setPins(ChipSelectPin, ResetPin, InterruptPin); 
  if (!LoRa.begin(FieldGatewayFrequency))
  {
    Serial.println("LoRa begin failed");
    while (true); // Drop into endless loop requiring restart
  }

  // Need to do this so field gateways pays attention to messsages from this device
  LoRa.enableCrc();
  LoRa.setSyncWord(FieldGatewaySyncWord);

#ifdef DEBUG_LORA
  LoRa.dumpRegisters(Serial);
#endif
  Serial.println("LoRa Setup done.");

  // Configure the Seeedstudio TH02 temperature & humidity sensor
  Serial.println("TH02 setup start");
  TH02.begin();
  delay(100);
  Serial.println("TH02 setup done");

  PayloadHeader((byte*)FieldGatewayAddress,strlen(FieldGatewayAddress), (byte*)DeviceAddress, strlen(DeviceAddress));

  Serial.println("Setup done");
  Serial.println();
}


void loop()
{
  float temperature ;
  float humidity ;

  Serial.println("Loop called");

  PayloadReset();

  // Read the temperature & humidity & battery voltage values then display nicely
  temperature = TH02.ReadTemperature();
  Serial.print("T:");
  Serial.print( temperature, 1 ) ;
  Serial.println( "C " ) ;

  PayloadAdd( "T", temperature, 1);

  humidity = TH02.ReadHumidity();
  Serial.print("H:");
  Serial.print( humidity, 0 ) ;
  Serial.println( "% " ) ;

  PayloadAdd( "H", humidity, 0) ;

#ifdef DEBUG_TELEMETRY
  Serial.print( "RFM9X/SX127X Payload len:");
  Serial.print( payloadLength );
  Serial.println( " bytes" );
#endif

  LoRa.beginPacket();
  LoRa.write( payload, payloadLength );
  LoRa.endPacket();

  Serial.println("Loop done");
  Serial.println();
  delay(LoopSleepDelaySeconds * 1000l);
}


void PayloadHeader( byte *to, byte toAddressLength, byte *from, byte fromAddressLength)
{
  byte addressesLength = toAddressLength + fromAddressLength ;

#ifdef DEBUG_TELEMETRY
  Serial.print("PayloadHeader- ");
  Serial.print( "To len:");
  Serial.print( toAddressLength );
  Serial.print( " From len:");
  Serial.print( fromAddressLength );
  Serial.print( " Header len:");
  Serial.print( addressesLength );
  Serial.println( );
#endif

  payloadLength = 0 ;

  // prepare the payload header with "To" Address length (top nibble) and "From" address length (bottom nibble)
  payload[payloadLength] = (toAddressLength << 4) | fromAddressLength ;
  payloadLength += 1;

  // Copy the "To" address into payload
  memcpy(&payload[payloadLength], to, toAddressLength);
  payloadLength += toAddressLength ;

  // Copy the "From" into payload
  memcpy(&payload[payloadLength], from, fromAddressLength);
  payloadLength += fromAddressLength ;
}


void PayloadAdd( const char *sensorId, float value, byte decimalPlaces)
{
  byte sensorIdLength = strlen( sensorId ) ;

#ifdef DEBUG_TELEMETRY
  Serial.print("PayloadAdd-float ");
  Serial.print( "SensorId:");
  Serial.print( sensorId );
  Serial.print( " Len:");
  Serial.print( sensorIdLength );
  Serial.print( " Value:");
  Serial.print( value, decimalPlaces );
  Serial.print( " payloadLen:");
  Serial.print( payloadLength);
#endif

  memcpy( &payload[payloadLength], sensorId,  sensorIdLength) ;
  payloadLength += sensorIdLength ;
  payload[ payloadLength] = SensorIdValueSeperator;
  payloadLength += 1 ;
  payloadLength += strlen( dtostrf(value, -1, decimalPlaces, (char *)&payload[payloadLength]));
  payload[ payloadLength] = SensorReadingSeperator;
  payloadLength += 1 ;
  
#ifdef DEBUG_TELEMETRY
  Serial.print( " payloadLen:");
  Serial.println( payloadLength);
#endif
}


void PayloadAdd( const char *sensorId, int value )
{
  byte sensorIdLength = strlen( sensorId ) ;

#ifdef DEBUG_TELEMETRY
  Serial.print("PayloadAdd-int ");
  Serial.print( "SensorId:");
  Serial.print( sensorId );
  Serial.print( " Len:");
  Serial.print( sensorIdLength );
  Serial.print( " Value:");
  Serial.print( value );
  Serial.print( " payloadLen:");
  Serial.print( payloadLength);
#endif  

  memcpy( &payload[payloadLength], sensorId,  sensorIdLength) ;
  payloadLength += sensorIdLength ;
  payload[ payloadLength] = SensorIdValueSeperator;
  payloadLength += 1 ;
  payloadLength += strlen( itoa( value,(char *)&payload[payloadLength],10));
  payload[ payloadLength] = SensorReadingSeperator;
  payloadLength += 1 ;
  
#ifdef DEBUG_TELEMETRY
  Serial.print( " payloadLen:");
  Serial.println( payloadLength);
#endif
}


void PayloadAdd( const char *sensorId, unsigned int value )
{
  byte sensorIdLength = strlen( sensorId ) ;

#ifdef DEBUG_TELEMETRY
  Serial.print("PayloadAdd-unsigned int ");
  Serial.print( "SensorId:");
  Serial.print( sensorId );
  Serial.print( " Len:");
  Serial.print( sensorIdLength );
  Serial.print( " Value:");
  Serial.print( value );
  Serial.print( " payloadLen:");
  Serial.print( payloadLength);
#endif  

  memcpy( &payload[payloadLength], sensorId,  sensorIdLength) ;
  payloadLength += sensorIdLength ;
  payload[ payloadLength] = SensorIdValueSeperator;
  payloadLength += 1 ;
  payloadLength += strlen( utoa( value,(char *)&payload[payloadLength],10));
  payload[ payloadLength] = SensorReadingSeperator;
  payloadLength += 1 ;

#ifdef DEBUG_TELEMETRY
  Serial.print( " payloadLen:");
  Serial.println( payloadLength);
#endif
}


void PayloadReset()
{
  byte fromAddressLength = payload[0] & 0xf ;
  byte toAddressLength = payload[0] >> 4 ;
  byte addressesLength = toAddressLength + fromAddressLength ;

  payloadLength = addressesLength + 1;

#ifdef DEBUG_TELEMETRY
  Serial.print("PayloadReset- ");
  Serial.print( "To len:");
  Serial.print( toAddressLength );
  Serial.print( " From len:");
  Serial.print( fromAddressLength );
  Serial.print( " Header len:");
  Serial.println( addressesLength );
#endif
}

To get the application to download I had to configure the board in the Arduino IDE

Then change the jumpers

Initially I had some problems deploying my software because I hadn’t followed the instructions (the wiki everyone referred to appeared to be offline) and run the installation batch file (New dev machine since my previous maple based project).

15:40:56.207 -> LoRa Setup done.
15:40:56.207 -> TH02 setup start
15:40:56.307 -> TH02 setup done
15:40:56.307 -> PayloadHeader- To len:8 From len:11 Header len:19
15:40:56.354 -> Setup done
15:40:56.354 -> 
15:40:56.354 -> Loop called
15:40:56.354 -> PayloadReset- To len:8 From len:11 Header len:19
15:40:56.408 -> T:23.9C 
15:40:56.408 -> PayloadAdd-float SensorId:T Len:1 Value:23.9 payloadLen:20 payloadLen:27
15:40:56.508 -> H:70% 
15:40:56.508 -> PayloadAdd-float SensorId:H Len:1 Value:70 payloadLen:27 payloadLen:32
15:40:56.608 -> RFM9X/SX127X Payload len:32 bytes
15:40:56.655 -> Loop done
15:40:56.655 -> 
15:41:26.647 -> Loop called
15:41:26.647 -> PayloadReset- To len:8 From len:11 Header len:19
15:41:26.684 -> T:24.0C 
15:41:26.730 -> PayloadAdd-float SensorId:T Len:1 Value:24.0 payloadLen:20 payloadLen:27
15:41:26.784 -> H:69% 
15:41:26.784 -> PayloadAdd-float SensorId:H Len:1 Value:69 payloadLen:27 payloadLen:32
15:41:26.884 -> RFM9X/SX127X Payload len:32 bytes
15:41:26.931 -> Loop done
15:41:26.931 -> 
15:41:56.904 -> Loop called
15:41:56.904 -> PayloadReset- To len:8 From len:11 Header len:19
15:41:56.948 -> T:24.1C 
15:41:56.982 -> PayloadAdd-float SensorId:T Len:1 Value:24.1 payloadLen:20 payloadLen:27
15:41:57.054 -> H:69% 
15:41:57.054 -> PayloadAdd-float SensorId:H Len:1 Value:69 payloadLen:27 payloadLen:32
15:41:57.157 -> RFM9X/SX127X Payload len:32 bytes
15:41:57.191 -> Loop done
15:41:57.191 -> 
15:42:27.211 -> Loop called
15:42:27.211 -> PayloadReset- To len:8 From len:11 Header len:19
15:42:27.258 -> T:24.1C 
15:42:27.258 -> PayloadAdd-float SensorId:T Len:1 Value:24.1 payloadLen:20 payloadLen:27
15:42:27.343 -> H:69% 
15:42:27.343 -> PayloadAdd-float SensorId:H Len:1 Value:69 payloadLen:27 payloadLen:32
15:42:27.427 -> RFM9X/SX127X Payload len:32 bytes
15:42:27.481 -> Loop done
15:42:27.481 -> 
15:42:57.504 -> Loop called
15:42:57.504 -> PayloadReset- To len:8 From len:11 Header len:19
15:42:57.504 -> T:24.1C 
15:42:57.550 -> PayloadAdd-float SensorId:T Len:1 Value:24.1 payloadLen:20 payloadLen:27
15:42:57.604 -> H:68% 
15:42:57.604 -> PayloadAdd-float SensorId:H Len:1 Value:68 payloadLen:27 payloadLen:32
15:42:57.704 -> RFM9X/SX127X Payload len:32 bytes
15:42:57.755 -> Loop done
15:42:57.755 -> 

I configured the device to upload to my Azure IoT Hub/Azure IoT Central gateway and it has been running reliably for a couple of days.

Azure IoT Central temperature and humidity values

Initially I had some configuration problems but I contacted Armtronix support and they promptly provided a couple of updated links for product and device documentation.

Azure IoT Hub MQTT LoRa Field Gateway

Back in April I started working on an MQTT LoRa Field gateway which was going to support a selection of different Software as a service(SaaS) Internet of Things IoT) platforms.

I now have a working Azure IoT Hub plug-in (Azure IoT Central support as planned as well) with the first iteration focused on Device to Cloud (D2C) messaging. In a future iteration I will add Cloud to Device messaging(C2D).

My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)

I have a number of sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield etc. clients. These work with both my platform specific (Adafruit.IO, Azure IoT Hub/Central) gateways and protocol specific field gateways.

Azure IoT Hub Device Explorer Data Display

When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.

{
  "MQTTUserName": "YourIoTHubHub.azure-devices.net/MQTTLoRa915MHz/api-version=2018-06-30",
  "MQTTPassword": "SharedAccessSignature sr=YourIoTHubHub.azure-devices.net%2Fdevices%2FMQTTLoRa915MHz&sig=123456789012345678901234567890123456789012345%3D&se=1574673583",
  "MQTTClientID": "MQTTLoRa915MHz",
  "MQTTServer": "YourIoTHubHub.azure-devices.net",
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0,
  "MessageHandlerAssembly": "Mqtt.IoTCore.FieldGateway.LoRa.AzureIoTHub",
  "PlatformSpecificConfiguration": ""
}

The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel

MQTT LoRa Gateway with Azure IoT Hub plug-in

The message handler uploads all values in an inbound messages in one MQTT message.

namespace devMobile.Mqtt.IoTCore.FieldGateway
{
   using System;
   using System.Diagnostics;
   using System.Text;
   using Windows.Foundation.Diagnostics;

   using devMobile.IoT.Rfm9x;
   using MQTTnet;
   using MQTTnet.Client;
   using Newtonsoft.Json.Linq;
   using Newtonsoft.Json;

   public class MessageHandler : IMessageHandler
   {
      private LoggingChannel Logging { get; set; }
      private IMqttClient MqttClient { get; set; }
      private Rfm9XDevice Rfm9XDevice { get; set; }
      private string PlatformSpecificConfiguration { get; set; }

      void IMessageHandler.Initialise(LoggingChannel logging, IMqttClient mqttClient, Rfm9XDevice rfm9XDevice, string platformSpecificConfiguration)
      {
         LoggingFields processInitialiseLoggingFields = new LoggingFields();

         this.Logging = logging;
         this.MqttClient = mqttClient;
         this.Rfm9XDevice = rfm9XDevice;
         this.PlatformSpecificConfiguration = platformSpecificConfiguration;
      }

      async void IMessageHandler.Rfm9XOnReceive(Rfm9XDevice.OnDataReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();
         char[] sensorReadingSeparators = { ',' };
         char[] sensorIdAndValueSeparators = { ' ' };

         processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
         processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
         processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

         string addressBcdText = BitConverter.ToString(e.Address);
         processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
         processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

         string messageText;
         try
         {
            messageText = UTF8Encoding.UTF8.GetString(e.Data);
            processReceiveLoggingFields.AddString("MessageText", messageText);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         // Chop up the CSV text
         string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
         if (sensorReadings.Length < 1)
         {
            this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         JObject payloadJObject = new JObject();

         JObject feeds = new JObject();

         // Chop up each sensor read into an ID & value
         foreach (string sensorReading in sensorReadings)
         {
            string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);

            // Check that there is an id & value
            if (sensorIdAndValue.Length != 2)
            {
               this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
               return;
            }

            string sensorId = string.Concat(addressBcdText, sensorIdAndValue[0]);
            string value = sensorIdAndValue[1];

            feeds.Add(sensorId.ToLower(), value);
         }
         payloadJObject.Add("feeds", feeds);

         string topic = $"devices/{MqttClient.Options.ClientId}/messages/events/";

         try
         {
            var message = new MqttApplicationMessageBuilder()
               .WithTopic(topic)
               .WithPayload(JsonConvert.SerializeObject(payloadJObject))
               .WithAtLeastOnceQoS()
               .Build();
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
            await MqttClient.PublishAsync(message);
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);

            this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Information);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Error);
         }
      }

      void IMessageHandler.MqttApplicationMessageReceived(MqttApplicationMessageReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();

         processReceiveLoggingFields.AddString("ClientId", e.ClientId);
#if DEBUG
         processReceiveLoggingFields.AddString("Payload", e.ApplicationMessage.ConvertPayloadToString());
#endif
         processReceiveLoggingFields.AddString("QualityOfServiceLevel", e.ApplicationMessage.QualityOfServiceLevel.ToString());
         processReceiveLoggingFields.AddBoolean("Retain", e.ApplicationMessage.Retain);
         processReceiveLoggingFields.AddString("Topic", e.ApplicationMessage.Topic);

         this.Logging.LogEvent("MqttApplicationMessageReceived topic not processed", processReceiveLoggingFields, LoggingLevel.Error);
      }

      void IMessageHandler.Rfm9xOnTransmit(Rfm9XDevice.OnDataTransmitedEventArgs e)
      {
      }
   }
}

The formatting of the username and generation of password are password are a bit awkward and will be fixed in a future refactoring. Along with regenerating the SAS connection token just before it is due to expire.

Azure IoT Hub with MQTTnet

As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.

This PoC was to confirm that my device could connect to the Microsoft Azure IoT Hub MQTT API then format topics and payloads correctly.

Azure IoT Hub MQTT Console Client

I had tried with a couple of different MQTT libraries from micro controllers and embedded devices without success. With the benefit of hindsight (plus this article) I think I had the SAS key format wrong.

The Azure IoT Hub MQTT broker requires only a server name (fully resolved CName), device ID and SAS Key.

   class Program
   {
      private static IMqttClient mqttClient = null;
      private static IMqttClientOptions mqttOptions = null;
      private static string server;
      private static string username;
      private static string password;
      private static string clientId;
      private static string topicD2C;
      private static string topicC2D;

      static void Main(string[] args)
      {
         MqttFactory factory = new MqttFactory();
         mqttClient = factory.CreateMqttClient();

         if (args.Length != 3)
         {
            Console.WriteLine("[AzureIoTHubHostName] [deviceID] [SASKey]");
            Console.WriteLine("Press <enter> to exit");
            Console.ReadLine();
            return;
         }

         server = args[0];
         clientId = args[1];
         sasKey= args[2];

         username = $"{server}/{clientId}/api-version=2018-06-30";
         topicD2C = $"devices/{clientId}/messages/events/";
         topicC2D = $"devices/{clientId}/messages/devicebound/#";

         Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId}");

         mqttOptions = new MqttClientOptionsBuilder()
            .WithTcpServer(server, 8883)
            .WithCredentials(username, sasKey)
            .WithClientId(clientId)
            .WithTls()
            .Build();

         mqttClient.UseDisconnectedHandler(new MqttClientDisconnectedHandlerDelegate(e => MqttClient_Disconnected(e)));
         mqttClient.UseApplicationMessageReceivedHandler(new MqttApplicationMessageReceivedHandlerDelegate(e => MqttClient_ApplicationMessageReceived(e)));
         mqttClient.ConnectAsync(mqttOptions).Wait();

         mqttClient.SubscribeAsync(topicC2D, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce).GetAwaiter().GetResult();

         while (true)
         {
            JObject payloadJObject = new JObject();

            payloadJObject.Add("OfficeTemperature", "22." + DateTime.UtcNow.Millisecond.ToString());
            payloadJObject.Add("OfficeHumidity", (DateTime.UtcNow.Second + 40).ToString());

            string payload = JsonConvert.SerializeObject(payloadJObject);
            Console.WriteLine($"Topic:{topicD2C} Payload:{payload}");

            var message = new MqttApplicationMessageBuilder()
               .WithTopic(topicD2C)
               .WithPayload(payload)
               .WithAtLeastOnceQoS()
            .Build();

            Console.WriteLine("PublishAsync start");
            mqttClient.PublishAsync(message).Wait();
            Console.WriteLine("PublishAsync finish");

            Thread.Sleep(30100);
         }
      }

      private static void MqttClient_ApplicationMessageReceived(MqttApplicationMessageReceivedEventArgs e)
      {
         Console.WriteLine($"ClientId:{e.ClientId} Topic:{e.ApplicationMessage.Topic} Payload:{e.ApplicationMessage.ConvertPayloadToString()}");
      }

      private static async void MqttClient_Disconnected(MqttClientDisconnectedEventArgs e)
      {
         Debug.WriteLine("Disconnected");
         await Task.Delay(TimeSpan.FromSeconds(5));

         try
         {
            await mqttClient.ConnectAsync(mqttOptions);
         }
         catch (Exception ex)
         {
            Debug.WriteLine("Reconnect failed {0}", ex.Message);
         }
      }
   }

Overall the initial configuration went smoothly after I figured out the required Quality of Service (QoS) settings, and the SAS Key format.

Using the approach described in the Microsoft documentation I manually generated the SAS Key.(In my Netduino samples I have code for generating a SAS Key in my HTTPS Azure IoT Hub Client)

Azure Device Explorer Device Management
Azure Device Explorer SAS Key Generator

Once I had the configuration correct I could see telemetry from the device and send it messages.

Azure Device Explorer Data View

In a future post I will upload data to the Azure IoT Central for display. Then explore using a “module” attached to a device which maybe useful for my field gateway.