Azure IOT Hub and Event Grid Part2

I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the Windows 10 IoT Core device I would get several DeviceTelemetry events before the DeviceConnected event arrived.

I was using Advanced Message Queueing Protocol (AMQP) so I modified the configuration file so I could try all the available options.

C# TransportType enumeration

namespace Microsoft.Azure.Devices.Client
{
	//
	// Summary:
	//     Transport types supported by DeviceClient - AMQP/TCP, HTTP 1.1, MQTT/TCP, AMQP/WS,
	//     MQTT/WS
	public enum TransportType
	{
		//
		// Summary:
		//     Advanced Message Queuing Protocol transport. Try Amqp over TCP first and fallback
		//     to Amqp over WebSocket if that fails
		Amqp = 0,
		//
		// Summary:
		//     HyperText Transfer Protocol version 1 transport.
		Http1 = 1,
		//
		// Summary:
		//     Advanced Message Queuing Protocol transport over WebSocket only.
		Amqp_WebSocket_Only = 2,
		//
		// Summary:
		//     Advanced Message Queuing Protocol transport over native TCP only
		Amqp_Tcp_Only = 3,
		//
		// Summary:
		//     Message Queuing Telemetry Transport. Try Mqtt over TCP first and fallback to
		//     Mqtt over WebSocket if that fails
		Mqtt = 4,
		//
		// Summary:
		//     Message Queuing Telemetry Transport over Websocket only.
		Mqtt_WebSocket_Only = 5,
		//
		// Summary:
		//     Message Queuing Telemetry Transport over native TCP only
		Mqtt_Tcp_Only = 6
	}
}

Windows 10 IoT Core LoRa Telemetry Field gateway config.json file

{
   "AzureIoTHubDeviceConnectionString": "HostName=FieldGatewayHub.azure-devices.net;DeviceId=LoRa915MHz;SharedAccessKey=y12345678901234567890123456789012345678/arg=",

   "AzureIoTHubTransportType-Amqp": "amqp",
   "AzureIoTHubTransportType-Http1": "Http1",
   "AzureIoTHubTransportType-Amqp_WebSocket_Only": "Amqp_WebSocket_Only",
   "AzureIoTHubTransportType-Amqp_Tcp_Only": "Amqp_Tcp_Only",
   "AzureIoTHubTransportType-Mqtt": "Mqtt",
   "AzureIoTHubTransportType-Mqtt_WebSocket_Only": "Mqtt_WebSocket_Only",
   "AzureIoTHubTransportType-Amqp": "Mqtt_Tcp_Only",

   "AzureIoTHubTransportType": "Mqtt_Tcp_Only",
   "SensorIDIsDeviceIDSensorID": false,
   "Address": "LoRaIoT1",
   "Frequency": 915000000.0,
   "PABoost": true
}

So in alphabetical order here are my not terribly scientific results

AMQP

Device ETW

Device Telemetry messages

{
"id":"07e9a772-d86a-963d-139a-9d2ea2a0866c",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry","eventTime":"2020-01-25T00:57:18.477Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T00:57:18.477Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTcxLCJSU1NJIjotMTEwLCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijg0LjAiLCJhaCI6IjUwIiwid3NhIjoiMSIsIndzZyI6IjMiLCJ3ZCI6IjE5My44OCIsInIiOiIwLjAwIn0="},
"dataVersion":"",
"metadataVersion":"1"
}
…
{
"id":"ca8e8531-10da-ec99-aad7-49e68f3f9500","topic":"/SUBSCRIPTIONS/712345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T01:02:23.377Z",
"data":{
"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T01:02:23.377Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjgiLCJQYWNrZXRSU1NJIjotNzAsIlJTU0kiOi0xMDgsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODUuMSIsImFoIjoiNTEiLCJ3c2EiOiIxIiwid3NnIjoiMyIsIndkIjoiMjM5LjI1IiwiciI6IjAuMDAifQ=="},
"dataVersion":"",
"metadataVersion":"1"
}

Device connectivity messages

{
"id":"d8a393ff-6549-69d2-d728-37eee2437693",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:01:28.4887191Z","data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000001E"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"b345d18a-bdf5-3397-35c1-fd5c35046d85",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:02:24.9605306Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000001F"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 00:57:18, the DeviceConnected arrived 01:01:28 so approximately a 4 minute delay, the DeviceDisconnected arrived within a minute of me shutting the device down.

Amqp-Tcp-Only

Device Telemetry messages

{
"id":"b8cdbc73-5cb8-134c-a328-beed47be5f27",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB","subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:16:48.732Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:16:48.732Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzIsIlJTU0kiOi0xMDcsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODYuNSIsImFoIjoiNDUiLCJ3c2EiOiIxIiwid3NnIjoiMiIsIndkIjoiOTkuNzUiLCJyIjoiMC4wMCJ9"},
"dataVersion":"",
"metadataVersion":"1"
}
...
{
"id":"56aaf46d-e1ee-9419-2621-aa97c0564778",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:21:53.402Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:21:53.402Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMDksIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODQuOSIsImFoIjoiNDYiLCJ3c2EiOiIzIiwid3NnIjoiNyIsIndkIjoiMjE3LjUwIiwiciI6IjAuMDAifQ=="},
"dataVersion":"",
"metadataVersion":"1"
}

Device connectivity messages

{
"id":"a157468c-7d65-00b1-73d6-12e43fd1356b",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:20:39.7309538Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000026"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"2e6d7e57-3db3-9e1d-c01b-b0f787b16e05",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:22:33.5276985Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000027"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:16:48, the DeviceConnected arrived 04:20:39 so approximately a 4 minute delay, the DeviceDisconnected arrived within a minute of me shutting the device down.

Amqp WebSocket Only

Device Telemetry messages

{
"id":"f82943da-425c-f49c-49b0-13ff2609544b",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:05:36.723Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:05:36.723Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjMiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMDksIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODUuMyIsImFoIjoiNDMiLCJ3c2EiOiIyIiwid3NnIjoiMyIsIndkIjoiMjcyLjYzIiwiciI6IjAuMDAifQ=="}
}
...
{
"id":"8419d4f7-3340-8e29-6370-29f95d40e68c",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:11:42.684Z",
"data":{"properties":{},"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:11:42.684Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjgiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMTIsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODcuMyIsImFoIjoiNDMiLCJ3c2EiOiIxIiwid3NnIjoiMyIsIndkIjoiMjcuMDAiLCJyIjoiMC4wMCJ9"}
}

Device connectivity messages

{
"id":"7f10a3e3-0c2c-2b18-e38c-d5ea498304ab",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:09:52.5786541Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000024"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{"id":"474a695b-8c4f-e3fe-0b1b-b6bc6b6d4dbe",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:11:53.076926Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000025"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:05:36, DeviceConnected arrived 04:09:52 so approximately a 4 minute delay, the DeviceDisconnected arrived within a minute of me shutting the device down.

HTTP

I waited for 20 minutes and there wasn’t a DeviceConnected message which I sort of expected as HTTP is a connectionless protocol.

MQTT

Device ETW

Device Telemetry messages

{
"id":"bc26c412-c694-3954-5888-baa118cc9f88",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz","time":"2020-01-25T01:11:33.493Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T01:11:33.493Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMTEsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODUuNiIsImFoIjoiNTAiLCJ3c2EiOiIyIiwid3NnIjoiNCIsIndkIjoiMjg1LjAwIiwiciI6IjAuMDAifQ=="}
}
...
{
"id":"de95b90f-ac96-5c76-7a46-00f5a0eef8cf",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:12:12.101Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T01:12:12.101Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTU4LCJSU1NJIjotMTA5LCJEZXZpY2VBZGRyZXNzQkNEIjoiMDEtMjMtMjktMDctMjMtMEMtODgtNjMtRUUiLCJoIjoiMzQiLCJ0IjoiNDEuOCIsInMiOiI0IiwidiI6IjQuMDcifQ=="}
}

Device connectivity messages

{
"id":"f8f5ee54-394d-05e3-784d-87bc648e8267",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:11:25.2530139Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000020"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"82d6aa6c-4c71-9623-a4ac-9e562345afad",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T01:12:26.6368519Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000021"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 01:11:33, the DeviceConnected arrived 01:11:25 so they arrived in order and within 10 seconds, the DeviceDisconnected arrived within a 15 seconds of me shutting the device down.

Mqtt-TCP-Only

Device ETW

Device Telemetry messages

{
"id":"bb86bfd9-6d12-4a27-2444-bd6953be7ffd",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:42:15.345Z",
"data":{"properties":{},"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}","iothub-connection-auth-generation-id":"637149227434620853","iothub-enqueuedtime":"2020-01-25T04:42:15.345Z","iothub-message-source":"Telemetry"},"body":"eyJQYWNrZXRTTlIiOiIxMC44IiwiUGFja2V0UlNTSSI6LTcxLCJSU1NJIjotMTA4LCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijg2LjkiLCJhaCI6IjQ1Iiwid3NhIjoiMSIsIndzZyI6IjMiLCJ3ZCI6IjI5LjYzIiwiciI6IjAuMDAifQ=="},"dataVersion":"","metadataVersion":"1"
}
...
{
"id":"c5991ed8-42f2-437a-9161-a526248c955f",
"topic":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB","subject":"devices/LoRa915MHz",
"eventType":"Microsoft.Devices.DeviceTelemetry",
"eventTime":"2020-01-25T04:44:16.986Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:44:16.986Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTcxLCJSU1NJIjotMTEyLCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijg4LjIiLCJhaCI6IjQ0Iiwid3NhIjoiMiIsIndzZyI6IjMiLCJ3ZCI6IjYxLjUwIiwiciI6IjAuMDAifQ=="},
"dataVersion":"",
"metadataVersion":"1"}
}

Device connectivity messages

{
"id":"7861f3c1-5f1d-e9a1-c214-19feea2bf1a3","source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:42:06.0123436Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000002A"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"c2090a7b-c827-73cc-d1e8-7d49fe4a03a1",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:44:48.816748Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA90000000700000000000000000000002B"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:42:15, the DeviceConnected arrived 04:42:06 so they arrived in order and within 10 seconds, the DeviceDisconnected arrived within a 20 seconds of me shutting device down.

Mqtt-Web-Socket-Only

Device ETW

Device Telemetry messages

{
"id":"3afa1a9c-f30d-d051-077e-cd25cc3b2245",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:36:08.871Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:36:08.871Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjUiLCJQYWNrZXRSU1NJIjotNzAsIlJTU0kiOi0xMTIsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODMuOCIsImFoIjoiNDciLCJ3c2EiOiIwIiwid3NnIjoiMSIsIndkIjoiMzQ3LjI1IiwiciI6IjAuMDAifQ=="}
}
...
{
"id":"fa082b67-db32-312d-e716-1b1d37f57d94",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:37:09.516Z",
"data":{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz","iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-25T04:37:09.516Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiI5LjgiLCJQYWNrZXRSU1NJIjotNzEsIlJTU0kiOi0xMTEsIkRldmljZUFkZHJlc3NCQ0QiOiI0RC02MS02NC03NS02OS02RS02Ri0zMiIsImF0IjoiODQuNCIsImFoIjoiNDciLCJ3c2EiOiIxIiwid3NnIjoiMiIsIndkIjoiMzU4LjUwIiwiciI6IjAuMDAifQ=="}
}

Device connectivity messages

{
"id":"245e4d68-06e6-4d76-1167-39a9a67b01ac",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceConnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz","time":"2020-01-25T04:36:03.8275263Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000028"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

{
"id":"9612e8af-84e0-a679-4c7d-28c5e968da3c",
"source":"/SUBSCRIPTIONS/12345678-9012-3456-7890-D12345678901/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceDisconnected",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-25T04:37:47.5794771Z",
"data":{"deviceConnectionStateEventInfo":{"sequenceNumber":"000000000000000001D5CDC11D3AACA900000007000000000000000000000029"},
"hubName":"FieldGatewayHub",
"deviceId":"LoRa915MHz"}
}

The first telemetry data arrived 04:36:08, the DeviceConnected arrived 04:36:03 so they arrived in order and within 10 seconds, the DeviceDisconnected arrived within a 30 seconds of me shutting device down.

Summary

My LoRa sensors nodes are sending data roughly every minute which reduces the precision of the times.

It looks like for AMQP based messaging it can take 4-5 minutes for a Devices.DeviceConnected message to arrive, for based MQTT messaging it’s 5-10 seconds.

Azure IOT Hub and Event Grid Part1

I have one an Azure IoT Hub LoRa Telemetry Field Gateway running in my office and I wanted to process the data collected by the sensors around my property without using a Software as a Service(SaaS) Internet of Things (IoT) package.

Rather than lots of screen grabs of my configuration steps I figured people reading this series of posts would be able to figure the details out themselves.

Raspberry PI with M2M LoRa Hat

I created an Azure Resource Group for this project, and created an Azure IoT Hub.

Azure Resource Group with IoT Hub

I then provisioned an Azure IoT Hub device so I could get the connection string for my Windows 10 Azure IoT Hub LoRa Telemetry Field gateway.

LoRa Field Gateway Provisioned in Azure IoT Hub

I downloaded the JSON configuration file template from my Windows 10 device (which is created on first startup after installation) and configured the Azure IoT Hub connection string.

{
   "AzureIoTHubDeviceConnectionString": "HostName=FieldGatewayHub.azure-devices.net;DeviceId=LoRa915MHz;SharedAccessKey=123456789012345678901234567890123456789/arg=",
   "AzureIoTHubTransportType": "amqp",
   "SensorIDIsDeviceIDSensorID": false,
   "Address": "LoRaIoT1",
   "Frequency": 915000000.0,
   "PABoost": true
}

I then uploaded this to my Windows 10 IoT Core device and restarted the Azure IoT Hub Field gateway so it picked up the new settings.

I could then see on the device messages from sensor nodes being unpacked and uploaded to my Azure IoT Hub.

ETW logging on device

In the Azure IoT Hub metrics I graphed the number of devices connected and the number of telemetry messages sent and could see my device connect then start uploading telemetry.

Azure IoT Hub metrics

One of my customers uses Azure Event Grid for application integration and I wanted to explore using it in an IoT solution. The first step was to create an Event Grid Domain.

I then used the Azure IoT Hub Events tab to wire up these events.

  • Microsoft.Devices.DeviceConnected
  • Microsoft.Devices.DeviceDisconnected
  • Microsoft.Devices.DeviceTelemetry
Azure IoT Hub Event Metrics

To confirm my event subscriptions were successful I previously found the “simplest” approach was to use an Azure storage queue endpoint. I had to create an Azure Storage Account with two Azure Storage Queues one for device connectivity (.DeviceConnected & .DeviceDisconnected) events and the other for device telemetry (.DeviceTelemetry) events.

I created a couple of other subscriptions so I could compare the different Event schemas (Event Grid Schema & Cloud Event Schema v1.0). At this stage I didn’t configure any Filters or Additional Features.

Azure IoT Hub Telemetry Event Metrics

I use Cerebrate Cerculean for monitoring and managing a couple of other customer projects so I used it to inspect the messages in the storage queues.

Cerebrate Ceculean Storage queue Inspector

The message are quite verbose

{
"id":"b48b6376-b7f4-ee7d-82d9-12345678901a",
"source":"/SUBSCRIPTIONS/12345678-901234789-0123-456789012345/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-24T04:27:30.842Z","data":
{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-24T04:27:30.842Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTY5LCJSU1NJIjotMTA5LCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijc2LjYiLCJhaCI6IjU4Iiwid3NhIjoiMiIsIndzZyI6IjUiLCJ3ZCI6IjMyMi44OCIsInIiOiIwLjAwIn0="
}
}

The message payload is base64 encoded, so I used an online tool to decode it.

{
 PacketSNR":"10.0",
"PacketRSSI":-69,
"RSSI":-109,
"DeviceAddressBCD":"4D-61-64-75-69-6E-6F-32",
"at":"76.6",
"ah":"58",
"wsa":"2",
"wsg":"5",
"wd":"322.88",
"r":"0.00"
}

Without writing any code (I will script the configuration) I could upload sensor data to an Azure IoT Hub, subscribe to a selection of events the Azure IoT Hub publishes and then inspect them in an Azure Storage Queue.

I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the device I would get several DeviceTelemetry events before the DeviceConnected event arrived.

Azure IoT Hub SAS Tokens revisited yet again

Based my previous post on SAS Token Expiry I wrote a test harness to better understand DateTimeOffset

using System;

namespace UnixEpochTester
{
   class Program
   {
      static void Main(string[] args)
      {
         Console.WriteLine($"DIY                {new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)}");
         Console.WriteLine($"DateTime.UnixEpoch {DateTime.UnixEpoch} {DateTime.UnixEpoch.Kind}");
         Console.WriteLine();

         TimeSpan fromUnixEpochNow = DateTime.UtcNow - DateTime.UnixEpoch;
         Console.WriteLine($"Epoc now {fromUnixEpochNow} {fromUnixEpochNow.TotalSeconds.ToString("f0")} sec");
         Console.WriteLine();

         TimeSpan fromUnixEpochFixed = new DateTime(2019, 11, 30, 2, 0, 0, DateTimeKind.Utc) - DateTime.UnixEpoch;
         Console.WriteLine($"Epoc  {fromUnixEpochFixed} {fromUnixEpochFixed.TotalSeconds.ToString("f0")} sec");
         Console.WriteLine();

         DateTimeOffset dateTimeOffset = new DateTimeOffset( new DateTime( 2019,11,30,2,0,0, DateTimeKind.Utc));
         Console.WriteLine($"Epoc DateTimeOffset {fromUnixEpochFixed} {dateTimeOffset.ToUnixTimeSeconds()}");
         Console.WriteLine();

         TimeSpan fromEpochStart = new DateTime(2019, 11, 30, 2, 0, 0, DateTimeKind.Utc) - DateTime.UnixEpoch;
         Console.WriteLine($"Epoc DateTimeOffset {fromEpochStart} {fromEpochStart.TotalSeconds.ToString("F0")}");
         Console.WriteLine();


         // https://www.epochconverter.com/ matches
         // https://www.unixtimestamp.com/index.php matches

         Console.WriteLine("Press ENTER to exit");
         Console.ReadLine();
      }
   }
}

I validated my numbers against a couple of online calculators and they matched which was a good start.

DateTimeOffset test harness

As I was testing my Azure MQTT Test Client I had noticed some oddness with MQTT connection timeouts.

string token = generateSasToken($"{server}/devices/{clientId}", password, "", new TimeSpan(0,5,0));
1/12/2019 1:29:52 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.391","OfficeHumidity":"93"}]
1/12/2019 1:30:22 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.29","OfficeHumidity":"64"}]
...
1/12/2019 1:43:56 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.591","OfficeHumidity":"98"}]
1/12/2019 1:44:26 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.754","OfficeHumidity":"68"}]


string token = generateSasToken($"{server}/devices/{clientId}", password, "", new TimeSpan(0,5,0));
1/12/2019 1:29:52 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.391","OfficeHumidity":"93"}]
1/12/2019 1:30:22 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.29","OfficeHumidity":"64"}]
...
1/12/2019 2:01:37 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.334","OfficeHumidity":"79"}]
1/12/2019 2:02:07 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.503","OfficeHumidity":"49"}]


string token = generateSasToken($"{server}/devices/{clientId}", password, "", new TimeSpan(0,5,0));
2/12/2019 9:27:21 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.196","OfficeHumidity":"61"}]
2/12/2019 9:27:51 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.788","OfficeHumidity":"91"}]
...
2/12/2019 9:36:24 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.670","OfficeHumidity":"64"}]
2/12/2019 9:36:54 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.836","OfficeHumidity":"94"}]


string token = generateSasToken($"{server}/devices/{clientId}", password, "", new TimeSpan(0,5,0));
2/12/2019 9:40:52 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.46","OfficeHumidity":"92"}]
2/12/2019 9:41:22 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.443","OfficeHumidity":"62"}]
...
2/12/2019 9:50:55 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.742","OfficeHumidity":"95"}]


string token = generateSasToken($"{server}/devices/{clientId}", password, "", new TimeSpan(0,10,0));
approx 15min as only 30 sec resolution
1/12/2019 12:50:23 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.630","OfficeHumidity":"65"}]
1/12/2019 12:50:53 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.798","OfficeHumidity":"95"}]
...
1/12/2019 1:03:59 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.677","OfficeHumidity":"41"}]
1/12/2019 1:04:30 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.26","OfficeHumidity":"72"}]


string token = generateSasToken($"{server}/devices/{clientId}", password, "", new TimeSpan(0,10,0));
approx 15min as only 30 sec resolution
1/12/2019 1:09:30 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.106","OfficeHumidity":"72"}]
1/12/2019 1:10:00 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.463","OfficeHumidity":"42"}]
...
1/12/2019 1:23:35 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.366","OfficeHumidity":"77"}]
1/12/2019 1:24:05 PM> Device: [MQTTLoRa915MHz], Data:[{"OfficeTemperature":"22.537","OfficeHumidity":"47"}]

The dataset with the 5 minute expiry which remained connected for approximately 30 mins was hopefully a configuration issue.

The updated SAS Token code not uses ToUnixTimeSeconds to eliminate the scope for local vs. UTC issues.

      public static string generateSasToken(string resourceUri, string key, string policyName, TimeSpan timeToLive)
      {
         DateTimeOffset expiryDateTimeOffset = new DateTimeOffset(DateTime.UtcNow.Add(timeToLive));

         string expiryEpoch = expiryDateTimeOffset.ToUnixTimeSeconds().ToString();
         string stringToSign = WebUtility.UrlEncode(resourceUri) + "\n" + expiryEpoch;

         HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(key));
         string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

         string token = $"SharedAccessSignature sr={WebUtility.UrlEncode(resourceUri)}&sig={WebUtility.UrlEncode(signature)}&se={expiryEpoch}";

         if (!String.IsNullOrEmpty(policyName))
         {
            token += "&skn=" + policyName;
         }

         return token;
      }

I need to test the expiry of my SAS Tokens some more especially with the client running on my development machine (NZT which is currently UTC+13) and in Azure (UTC timezone)

Azure IoT Hub MQTT LoRa Field Gateway

Back in April I started working on an MQTT LoRa Field gateway which was going to support a selection of different Software as a service(SaaS) Internet of Things IoT) platforms.

I now have a working Azure IoT Hub plug-in (Azure IoT Central support as planned as well) with the first iteration focused on Device to Cloud (D2C) messaging. In a future iteration I will add Cloud to Device messaging(C2D).

My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)

I have a number of sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield etc. clients. These work with both my platform specific (Adafruit.IO, Azure IoT Hub/Central) gateways and protocol specific field gateways.

Azure IoT Hub Device Explorer Data Display

When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.

{
  "MQTTUserName": "YourIoTHubHub.azure-devices.net/MQTTLoRa915MHz/api-version=2018-06-30",
  "MQTTPassword": "SharedAccessSignature sr=YourIoTHubHub.azure-devices.net%2Fdevices%2FMQTTLoRa915MHz&sig=123456789012345678901234567890123456789012345%3D&se=1574673583",
  "MQTTClientID": "MQTTLoRa915MHz",
  "MQTTServer": "YourIoTHubHub.azure-devices.net",
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0,
  "MessageHandlerAssembly": "Mqtt.IoTCore.FieldGateway.LoRa.AzureIoTHub",
  "PlatformSpecificConfiguration": ""
}

The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel

MQTT LoRa Gateway with Azure IoT Hub plug-in

The message handler uploads all values in an inbound messages in one MQTT message.

namespace devMobile.Mqtt.IoTCore.FieldGateway
{
   using System;
   using System.Diagnostics;
   using System.Text;
   using Windows.Foundation.Diagnostics;

   using devMobile.IoT.Rfm9x;
   using MQTTnet;
   using MQTTnet.Client;
   using Newtonsoft.Json.Linq;
   using Newtonsoft.Json;

   public class MessageHandler : IMessageHandler
   {
      private LoggingChannel Logging { get; set; }
      private IMqttClient MqttClient { get; set; }
      private Rfm9XDevice Rfm9XDevice { get; set; }
      private string PlatformSpecificConfiguration { get; set; }

      void IMessageHandler.Initialise(LoggingChannel logging, IMqttClient mqttClient, Rfm9XDevice rfm9XDevice, string platformSpecificConfiguration)
      {
         LoggingFields processInitialiseLoggingFields = new LoggingFields();

         this.Logging = logging;
         this.MqttClient = mqttClient;
         this.Rfm9XDevice = rfm9XDevice;
         this.PlatformSpecificConfiguration = platformSpecificConfiguration;
      }

      async void IMessageHandler.Rfm9XOnReceive(Rfm9XDevice.OnDataReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();
         char[] sensorReadingSeparators = { ',' };
         char[] sensorIdAndValueSeparators = { ' ' };

         processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
         processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
         processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

         string addressBcdText = BitConverter.ToString(e.Address);
         processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
         processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

         string messageText;
         try
         {
            messageText = UTF8Encoding.UTF8.GetString(e.Data);
            processReceiveLoggingFields.AddString("MessageText", messageText);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         // Chop up the CSV text
         string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
         if (sensorReadings.Length < 1)
         {
            this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
            return;
         }

         JObject payloadJObject = new JObject();

         JObject feeds = new JObject();

         // Chop up each sensor read into an ID & value
         foreach (string sensorReading in sensorReadings)
         {
            string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);

            // Check that there is an id & value
            if (sensorIdAndValue.Length != 2)
            {
               this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
               return;
            }

            string sensorId = string.Concat(addressBcdText, sensorIdAndValue[0]);
            string value = sensorIdAndValue[1];

            feeds.Add(sensorId.ToLower(), value);
         }
         payloadJObject.Add("feeds", feeds);

         string topic = $"devices/{MqttClient.Options.ClientId}/messages/events/";

         try
         {
            var message = new MqttApplicationMessageBuilder()
               .WithTopic(topic)
               .WithPayload(JsonConvert.SerializeObject(payloadJObject))
               .WithAtLeastOnceQoS()
               .Build();
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
            await MqttClient.PublishAsync(message);
            Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);

            this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Information);
         }
         catch (Exception ex)
         {
            processReceiveLoggingFields.AddString("Exception", ex.ToString());
            this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Error);
         }
      }

      void IMessageHandler.MqttApplicationMessageReceived(MqttApplicationMessageReceivedEventArgs e)
      {
         LoggingFields processReceiveLoggingFields = new LoggingFields();

         processReceiveLoggingFields.AddString("ClientId", e.ClientId);
#if DEBUG
         processReceiveLoggingFields.AddString("Payload", e.ApplicationMessage.ConvertPayloadToString());
#endif
         processReceiveLoggingFields.AddString("QualityOfServiceLevel", e.ApplicationMessage.QualityOfServiceLevel.ToString());
         processReceiveLoggingFields.AddBoolean("Retain", e.ApplicationMessage.Retain);
         processReceiveLoggingFields.AddString("Topic", e.ApplicationMessage.Topic);

         this.Logging.LogEvent("MqttApplicationMessageReceived topic not processed", processReceiveLoggingFields, LoggingLevel.Error);
      }

      void IMessageHandler.Rfm9xOnTransmit(Rfm9XDevice.OnDataTransmitedEventArgs e)
      {
      }
   }
}

The formatting of the username and generation of password are password are a bit awkward and will be fixed in a future refactoring. Along with regenerating the SAS connection token just before it is due to expire.

Azure IoT Hub SAS Tokens revisited again

This post has been edited (2019-11-24) my original assumption about how DateTime.Kind unspecified was handled were incorrect.

As I was testing my Azure MQTT Test Client I noticed some oddness with MQTT connection timeouts and this got me wondering about token expiry times. So, I went searching again and found this Azure IoT Hub specific sample code

public static string generateSasToken(string resourceUri, string key, string policyName, int expiryInSeconds = 3600)
{
    TimeSpan fromEpochStart = DateTime.UtcNow - new DateTime(1970, 1, 1);
    string expiry = Convert.ToString((int)fromEpochStart.TotalSeconds + expiryInSeconds);

    string stringToSign = WebUtility.UrlEncode(resourceUri) + "\n" + expiry;

    HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(key));
    string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

    string token = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}", WebUtility.UrlEncode(resourceUri), WebUtility.UrlEncode(signature), expiry);

    if (!String.IsNullOrEmpty(policyName))
    {
        token += "&skn=" + policyName;
    }

    return token;
}

This code worked first time and was more flexible than mine which was a bonus. Though while running my MQTTNet based client I noticed the connection would drop after approximately 10mins (EDIT this was probably an unrelated networking issue).

A long time ago (25 years) I had issues sharing a Unix time value between an applications written with Borland C and Microsoft Visual C which made me wonder about Unix epoch base offsets.

So to test my theory I built a Unix epoch test harness console application

using System;

namespace UnixEpocTest
{
   class Program
   {
      static void Main(string[] args)
      {
         TimeSpan ttl = new TimeSpan(0, 0, 0);

         Console.WriteLine("Current time");
         Console.WriteLine($"Local     {DateTime.Now} {DateTime.Now.Kind}");
         Console.WriteLine($"UTC       {DateTime.UtcNow} {DateTime.UtcNow.Kind}");
         Console.WriteLine($"Unix DIY  {new DateTime(1970, 1, 1)} {new DateTime(1970, 1, 1).Kind}");
         Console.WriteLine($"Unix DIY+ {new DateTime(1970, 1, 1).ToUniversalTime()} {new DateTime(1970, 1, 1).ToUniversalTime().Kind}");
         Console.WriteLine($"Unix DIY  {new DateTime(1970, 1, 1, 0,0,0, DateTimeKind.Utc)}");
         Console.WriteLine($"Unix      {DateTime.UnixEpoch} {DateTime.UnixEpoch.Kind}");
         Console.WriteLine();

         TimeSpan fromEpochStart = DateTime.UtcNow - new DateTime(1970, 1, 1);
         TimeSpan fromEpochStartUtc = DateTime.UtcNow - new DateTime(1970, 1, 1,0,0,0, DateTimeKind.Utc);
         TimeSpan fromEpochStartUnixEpoch = DateTime.UtcNow - DateTime.UnixEpoch;

         Console.WriteLine("Epoch comparison");
         Console.WriteLine($"Local {fromEpochStart} {fromEpochStart.TotalSeconds.ToString("f0")} sec");
         Console.WriteLine($"UTC   {fromEpochStartUtc} {fromEpochStartUtc.TotalSeconds.ToString("f0")} sec");
         Console.WriteLine($"Epoc  {fromEpochStartUnixEpoch} {fromEpochStartUnixEpoch.TotalSeconds.ToString("f0")} sec");
         Console.WriteLine();

         TimeSpan afterEpoch = DateTime.UtcNow.Add(ttl) - new DateTime(1970, 1, 1);
         TimeSpan afterEpochUtC = DateTime.UtcNow.Add(ttl) - new DateTime(1970, 1, 1).ToUniversalTime();
         TimeSpan afterEpochEpoch = DateTime.UtcNow.Add(ttl) - DateTime.UnixEpoch;

         Console.WriteLine("Epoch calculation");
         Console.WriteLine($"Local {afterEpoch}");
         Console.WriteLine($"UTC   {afterEpochUtC}");
         Console.WriteLine($"Epoch {afterEpochEpoch}");
         Console.WriteLine();

         Console.WriteLine("Epoch DateTime");
         Console.WriteLine($"Local :{new DateTime(1970, 1, 1)}");
         Console.WriteLine($"UTC   :{ new DateTime(1970, 1, 1).ToUniversalTime()}");

         Console.WriteLine("Press ENTER to exit");
         Console.ReadLine();

         Console.WriteLine("Hello World!");
      }
   }
}

EDIT: I now think the UtcNow to “unspecified” kind mathematics was being handled correctly. I have updated the code to use the DateTime.UnixEpoch constant so the code is more readable.

public static string generateSasToken(string resourceUri, string key, string policyName, int expiryInSeconds = 900)
      {
         TimeSpan fromEpochStart = DateTime.UtcNow - DateTime.UnixEpoch;
         string expiry = Convert.ToString((int)fromEpochStart.TotalSeconds + expiryInSeconds);

         string stringToSign = WebUtility.UrlEncode(resourceUri) + "\n" + expiry;

         HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(key));
         string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

         string token = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}", WebUtility.UrlEncode(resourceUri), WebUtility.UrlEncode(signature), expiry);

         if (!String.IsNullOrEmpty(policyName))
         {
            token += "&skn=" + policyName;
         }

         return token;
      }

I need to test the expiry of my SAS Tokens some more especially with the client running on my development machine (NZT which is currently UTC+13) and in Azure (UTC timezone)

Azure IoT Hub SAS Tokens revisited

A long time ago I wrote a post about uploading telemetry data to an Azure Event Hub from a Netduino 3 Wifi using HTTPS. To send messages to the EventHub I had to create a valid SAS Token which took a surprising amount of effort because of the reduced text encoding/decoding and cryptographic functionality available in .NET Micro Framework v4.3 (NetMF)

// Create a SAS token for a specified scope. SAS tokens are described in http://msdn.microsoft.com/en-us/library/windowsazure/dn170477.aspx.
private static string CreateSasToken(string uri, string keyName, string key)
{
   // Set token lifetime to 20 minutes. When supplying a device with a token, you might want to use a longer expiration time.
   uint tokenExpirationTime = GetExpiry(20 * 60);
 
   string stringToSign = HttpUtility.UrlEncode(uri) + "\n" + tokenExpirationTime;
 
   var hmac = SHA.computeHMAC_SHA256(Encoding.UTF8.GetBytes(key), Encoding.UTF8.GetBytes(stringToSign));
   string signature = Convert.ToBase64String(hmac);
 
   signature = Base64NetMf42ToRfc4648(signature);
 
   string token = "SharedAccessSignature sr=" + HttpUtility.UrlEncode(uri) + "&sig=" + HttpUtility.UrlEncode(signature) + "&se=" + tokenExpirationTime.ToString() + "&skn=" + keyName;
 
   return token;
}
 
private static string Base64NetMf42ToRfc4648(string base64netMf)
{
   var base64Rfc = string.Empty;
 
   for (var i = 0; i < base64netMf.Length; i++)
   {
      if (base64netMf[i] == '!')
      {
         base64Rfc += '+';
      }
      else if (base64netMf[i] == '*')
      {
         base64Rfc += '/';
      }
      else
      {
         base64Rfc += base64netMf[i];
      }
   }
   return base64Rfc;
}
 
static uint GetExpiry(uint tokenLifetimeInSeconds)
{
   const long ticksPerSecond = 1000000000 / 100; // 1 tick = 100 nano seconds
 
   DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0);
   TimeSpan diff = DateTime.Now.ToUniversalTime() - origin;
 
   return ((uint)(diff.Ticks / ticksPerSecond)) + tokenLifetimeInSeconds;
}

Initially for testing my Azure MQTT Test Client I manually generated the SAS tokens using Azure Device Explorer but figured it would be better if the application generated them.

An initial search lead to this article about how to generate a SAS token for an Azure Event Hub in multiple languages. For my first attempt I “copied and paste” the code sample for C# (I also wasn’t certain what to put in the KeyName parameter) and it didn’t work.

private static string createToken(string resourceUri, string keyName, string key)
{
    TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);
    var week = 60 * 60 * 24 * 7;
    var expiry = Convert.ToString((int)sinceEpoch.TotalSeconds + week);
    string stringToSign = HttpUtility.UrlEncode(resourceUri) + "\n" + expiry;
    HMACSHA256 hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));
    var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
    var sasToken = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}", HttpUtility.UrlEncode(resourceUri), HttpUtility.UrlEncode(signature), expiry, keyName);
    return sasToken;
}

By comparing the Device Explorer and C# generated SAS keys I worked out the keyName parameter was unnecessary so I removed.

private static string createToken(string resourceUri, string key)
{
    TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);
    var week = 60 * 60 * 24 * 7;
    var expiry = Convert.ToString((int)sinceEpoch.TotalSeconds + week);
    string stringToSign = HttpUtility.UrlEncode(resourceUri) + "\n" + expiry;
    HMACSHA256 hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));
    var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
    var sasToken = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}", HttpUtility.UrlEncode(resourceUri), HttpUtility.UrlEncode(signature), expiry);
    return sasToken;
}

The shared SAS token now looked closer to what I was expecting but the MQTTNet ConnectAsync was failing with an authentication exception. After looking at the Device Explorer SAS Key code, my .NetMF implementation and the code for the IoT Hub SDK I noticed the encoding for the HMAC Key was different. Encoding.UTF8.GetBytes vs. Convert.FromBase64String.

 private static string createToken(string resourceUri,string key, TimeSpan ttl)
      {
         TimeSpan afterEpoch = DateTime.UtcNow.Add( ttl ) - new DateTime(1970, 1, 1);

         string expiry = afterEpoch.TotalSeconds.ToString("F0");
         string stringToSign = HttpUtility.UrlEncode(resourceUri) + "\n" + expiry;
         HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(key));
         string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
         return  String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}", HttpUtility.UrlEncode(resourceUri), HttpUtility.UrlEncode(signature), expiry);
      }

This approach appears to work reliably in my test harness.

MQTTnet client with new SAS Key Generator

User beware DIY Crypto often ends badly