The QueueTrigger function retry “rabbit hole”

My first couple of attempts at an Azure Queue Trigger Function which could do retries when an uplink message couldn’t be processed immediately(I didn’t want to throw an exception as this was just a transient issue) didn’t work. I wanted to return the uplink message to the Azure Storage Queue with the initial visibility set to a couple of seconds without throwing an exception.

I tried decorating the method with an Azure Storage Queue output binding but finally settled on the approach below. I can insert a single message into the storage queue and the application would start looping every minute.

public static class UplinkMessageProcessor
{
   const string RunTag = "Processor001";
   static int ConcurrentThreadCount = 0;
   static int MessagesProcessed = 0;

   [FunctionName("UplinkMessageProcessor")]
   public static void Run(
      [QueueTrigger("%UplinkQueueName%", Connection = "AzureStorageConnectionString")] 
      CloudQueueMessage cloudQueueMessage, 
      IBinder binder, ILogger log)
   {
      try
      {
         Interlocked.Increment(ref ConcurrentThreadCount);
         Interlocked.Increment(ref MessagesProcessed);

         log.LogInformation($"{MessagesProcessed} {RunTag} Threads:{ConcurrentThreadCount}");

         CloudQueue outputQueue = binder.Bind<CloudQueue>(new QueueAttribute("%UplinkQueueName%"));

         CloudQueueMessage message = new CloudQueueMessage(cloudQueueMessage.AsString);

         outputQueue.AddMessage(message, initialVisibilityDelay: new TimeSpan(0, 1, 0));
    
         Thread.Sleep(2000);

         Interlocked.Decrement(ref ConcurrentThreadCount);
      }
      catch (Exception ex)
      {
         log.LogError(ex, "Processing of Uplink message failed");

         throw;
      }
   }
}

I used the binder.bind method to get the CloudQueue and CloudQueueMessage details so I could insert a hidden messages back into the queue.

The version of Azure Storage queue libraries used by the function bindings (Sep 2020) may cause some compile time warnings if you select the wrong NuGet package.

Hopefully this has enough keywords that someone trying todo the same thing finds it.

Azure IOT Hub and Event Grid Part1

I have one an Azure IoT Hub LoRa Telemetry Field Gateway running in my office and I wanted to process the data collected by the sensors around my property without using a Software as a Service(SaaS) Internet of Things (IoT) package.

Rather than lots of screen grabs of my configuration steps I figured people reading this series of posts would be able to figure the details out themselves.

Raspberry PI with M2M LoRa Hat

I created an Azure Resource Group for this project, and created an Azure IoT Hub.

Azure Resource Group with IoT Hub

I then provisioned an Azure IoT Hub device so I could get the connection string for my Windows 10 Azure IoT Hub LoRa Telemetry Field gateway.

LoRa Field Gateway Provisioned in Azure IoT Hub

I downloaded the JSON configuration file template from my Windows 10 device (which is created on first startup after installation) and configured the Azure IoT Hub connection string.

{
   "AzureIoTHubDeviceConnectionString": "HostName=FieldGatewayHub.azure-devices.net;DeviceId=LoRa915MHz;SharedAccessKey=123456789012345678901234567890123456789/arg=",
   "AzureIoTHubTransportType": "amqp",
   "SensorIDIsDeviceIDSensorID": false,
   "Address": "LoRaIoT1",
   "Frequency": 915000000.0,
   "PABoost": true
}

I then uploaded this to my Windows 10 IoT Core device and restarted the Azure IoT Hub Field gateway so it picked up the new settings.

I could then see on the device messages from sensor nodes being unpacked and uploaded to my Azure IoT Hub.

ETW logging on device

In the Azure IoT Hub metrics I graphed the number of devices connected and the number of telemetry messages sent and could see my device connect then start uploading telemetry.

Azure IoT Hub metrics

One of my customers uses Azure Event Grid for application integration and I wanted to explore using it in an IoT solution. The first step was to create an Event Grid Domain.

I then used the Azure IoT Hub Events tab to wire up these events.

  • Microsoft.Devices.DeviceConnected
  • Microsoft.Devices.DeviceDisconnected
  • Microsoft.Devices.DeviceTelemetry
Azure IoT Hub Event Metrics

To confirm my event subscriptions were successful I previously found the “simplest” approach was to use an Azure storage queue endpoint. I had to create an Azure Storage Account with two Azure Storage Queues one for device connectivity (.DeviceConnected & .DeviceDisconnected) events and the other for device telemetry (.DeviceTelemetry) events.

I created a couple of other subscriptions so I could compare the different Event schemas (Event Grid Schema & Cloud Event Schema v1.0). At this stage I didn’t configure any Filters or Additional Features.

Azure IoT Hub Telemetry Event Metrics

I use Cerebrate Cerculean for monitoring and managing a couple of other customer projects so I used it to inspect the messages in the storage queues.

Cerebrate Ceculean Storage queue Inspector

The message are quite verbose

{
"id":"b48b6376-b7f4-ee7d-82d9-12345678901a",
"source":"/SUBSCRIPTIONS/12345678-901234789-0123-456789012345/RESOURCEGROUPS/AZUREIOTHUBEVENTGRIDAZUREFUNCTION/PROVIDERS/MICROSOFT.DEVICES/IOTHUBS/FIELDGATEWAYHUB",
"specversion":"1.0",
"type":"Microsoft.Devices.DeviceTelemetry",
"dataschema":"#",
"subject":"devices/LoRa915MHz",
"time":"2020-01-24T04:27:30.842Z","data":
{"properties":{},
"systemProperties":{"iothub-connection-device-id":"LoRa915MHz",
"iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id":"637149227434620853",
"iothub-enqueuedtime":"2020-01-24T04:27:30.842Z",
"iothub-message-source":"Telemetry"},
"body":"eyJQYWNrZXRTTlIiOiIxMC4wIiwiUGFja2V0UlNTSSI6LTY5LCJSU1NJIjotMTA5LCJEZXZpY2VBZGRyZXNzQkNEIjoiNEQtNjEtNjQtNzUtNjktNkUtNkYtMzIiLCJhdCI6Ijc2LjYiLCJhaCI6IjU4Iiwid3NhIjoiMiIsIndzZyI6IjUiLCJ3ZCI6IjMyMi44OCIsInIiOiIwLjAwIn0="
}
}

The message payload is base64 encoded, so I used an online tool to decode it.

{
 PacketSNR":"10.0",
"PacketRSSI":-69,
"RSSI":-109,
"DeviceAddressBCD":"4D-61-64-75-69-6E-6F-32",
"at":"76.6",
"ah":"58",
"wsa":"2",
"wsg":"5",
"wd":"322.88",
"r":"0.00"
}

Without writing any code (I will script the configuration) I could upload sensor data to an Azure IoT Hub, subscribe to a selection of events the Azure IoT Hub publishes and then inspect them in an Azure Storage Queue.

I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the device I would get several DeviceTelemetry events before the DeviceConnected event arrived.