Azure Functions with VB.Net 4.X

As part of my “day job” I spend a lot of time working with C# and VB.Net 4.X “legacy” projects doing upgrades, bugs fixes and moving applications to Azure. For the last couple of months I have been working on a project replacing Microsoft message queue(MSMQ) queues with Azure Storage Queues so the solution is easier to deploy in Azure.

The next phase of the project is to replace a number of Windows Services with Azure Queue Trigger and Timer Trigger functions. The aim is a series of small steps which we can test before deployment rather than major changes, hence the use of V1 Azure functions for the first release.

Silver Fox systems sells a Visual Studio extension which generates an HTTP Trigger VB.Net project. I needed Timer and Queue Trigger functions so I created C# examples and then used them to figure out how to build VB.Net equivalents

Visual Studio Solution Explorer

After quite a few failed attempts I found this sequence worked for me

Add a new VB.Net class library
Provide a name for new class library
Select target framework

Even though the target platform is not .NET 5.0 ignore this and continue.

Microsoft.NET.Sdk.Functions

Added Microsoft.NET.Sdk.Functions (make sure version 1.0.38)

Visual Studio project with Azure Function Icon.

Then unload the project and open the file.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <RootNamespace>TimerClass</RootNamespace>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.38" />
  </ItemGroup>

</Project>

Add the TargetFramework and AzureFunctionsVersion lines

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <RootNamespace>TimerClass</RootNamespace>
    <TargetFramework>net48</TargetFramework>
    <AzureFunctionsVersion>v1</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.38" />
  </ItemGroup>

</Project>

At this point the project should compile but won’t do much, so update the class to look like the code below.

Imports System.Threading

Imports Microsoft.Azure.WebJobs
Imports Microsoft.Extensions.Logging


Public Class TimerTrigger
   Shared executionCount As Int32

   <FunctionName("Timer")>
   Public Shared Sub Run(<TimerTrigger("0 */1 * * * *")> myTimer As TimerInfo, log As ILogger)
      Interlocked.Increment(executionCount)

      log.LogInformation("VB.Net TimerTrigger next trigger:{0} Execution count:{1}", myTimer.ScheduleStatus.Next, executionCount)

   End Sub
End Class

Then add an empty hosts.json file (make sure “copy if newer” is configured in properties) to the project directory, then depending on deployment model configure the AzureWebJobsStorage and AzureWebJobsDashboard connection strings via environment variables or a local.settings.json file.

Visual Studio Environment variables for AzureWebJobsStorage and AzureWebJobsDashboard connection strings

Blob Trigger Sample code

Imports System.IO
Imports System.Threading

Imports Microsoft.Azure.WebJobs
Imports Microsoft.Extensions.Logging


Public Class BlobTrigger
   Shared executionCount As Int32

   ' This function will get triggered/executed when a new message is written on an Azure Queue called events.
   <FunctionName("Notifications")>
   Public Shared Async Sub Run(<BlobTrigger("notifications/{name}", Connection:="BlobEndPoint")> payload As Stream, name As String, log As ILogger)
      Interlocked.Increment(executionCount)

      log.LogInformation("VB.Net BlobTrigger processed blob name:{0} Size:{1} bytes Execution count:{2}", name, payload.Length, executionCount)
   End Sub
End Class

HTTP Trigger Sample code

Imports System.Net
Imports System.Net.Http
Imports System.Threading

Imports Microsoft.Azure.WebJobs
Imports Microsoft.Azure.WebJobs.Extensions.Http
Imports Microsoft.Extensions.Logging


Public Class HttpTrigger
   Shared executionCount As Int32

   <FunctionName("Notifications")>
   Public Shared Async Function Run(<HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route:=Nothing)> req As HttpRequestMessage, log As ILogger) As Task(Of HttpResponseMessage)
      Interlocked.Increment(executionCount)

      log.LogInformation($"VB.Net HTTP trigger Execution count:{0} Method:{1}", executionCount, req.Method)

      Return New HttpResponseMessage(HttpStatusCode.OK)
   End Function
End Class

Queue Trigger Sample Code

Imports System.Threading

Imports Microsoft.Azure.WebJobs
Imports Microsoft.Extensions.Logging


Public Class QueueTrigger
   Shared ConcurrencyCount As Long
   Shared ExecutionCount As Long

   <FunctionName("Alerts")>
   Public Shared Sub ProcessQueueMessage(<QueueTrigger("notifications", Connection:="QueueEndpoint")> message As String, log As ILogger)
      Interlocked.Increment(ConcurrencyCount)
      Interlocked.Increment(ExecutionCount)

      log.LogInformation("VB.Net Concurrency:{0} Message:{1} Execution count:{2}", ConcurrencyCount, message, ExecutionCount)

      ' Wait for a bit to force some consurrency
      Thread.Sleep(5000)

      Interlocked.Decrement(ConcurrencyCount)
   End Sub
End Class

As well as counting the number of executions I also wanted to check that >1 instances were started to process messages when the queues had many messages. I added a “queues” section to the hosts.json file so I could tinker with the options.

{
  "queues": {
    "maxPollingInterval": 100,
    "visibilityTimeout": "00:00:05",
    "batchSize": 16,
    "maxDequeueCount": 5,
    "newBatchThreshold": 8
  }
}

The QueueMessageGenerator application inserts many messages into a queue for processing.

When I started the QueueTrigger function I could see the concurrency count was > 0

Timer Trigger Sample Code

Imports System.Threading

Imports Microsoft.Azure.WebJobs
Imports Microsoft.Extensions.Logging


Public Class TimerTrigger
   Shared executionCount As Int32

   <FunctionName("Timer")>
   Public Shared Sub Run(<TimerTrigger("0 */1 * * * *")> myTimer As TimerInfo, log As ILogger)
      Interlocked.Increment(executionCount)

      log.LogInformation("VB.Net TimerTrigger next trigger:{0} Execution count:{1}", myTimer.ScheduleStatus.Next, executionCount)

   End Sub
End Class

The source code for the C# and VB.Net functions is available on GitHub

TTI V3 Connector Device EUI Representation

While debugging The Things Industries(TTI) V3 connector on my desktop I had noticed the Device EUI‘s were wrong.

TTI V3 Connector application running as a console application showing incorrect DeviceEUIs

The TTI V3 Connector code…

foreach (V3EndDevice device in endDevices.End_devices)
{
   if (DeviceAzureEnabled(device))
   {
      _logger.LogInformation("Config-ApplicationID:{0} DeviceID:{1} Device EUI:{2}", device.Ids.Application_ids.Application_id, device.Ids.Device_id, BitConverter.ToString(device.Ids.Dev_eui));

      tasks.Add(DeviceRegistration(device.Ids.Application_ids.Application_id, device.Ids.Device_id, _programSettings.ResolveDeviceModelId(device.Ids.Application_ids.Application_id, device.Attributes), stoppingToken));
   }
}

…uses some classes generated by nSwag based on the TheThingsNetwork/LoRaWAN-stack api.swagger.json

public partial class V3EndDeviceIdentifiers 
{
        [Newtonsoft.Json.JsonProperty("device_id", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string Device_id { get; set; }
    
        [Newtonsoft.Json.JsonProperty("application_ids", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public V3ApplicationIdentifiers Application_ids { get; set; }
    
        /// <summary>The LoRaWAN DevEUI.</summary>
        [Newtonsoft.Json.JsonProperty("dev_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Dev_eui { get; set; }
    
        /// <summary>The LoRaWAN JoinEUI (AppEUI until LoRaWAN 1.0.3 end devices).</summary>
        [Newtonsoft.Json.JsonProperty("join_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Join_eui { get; set; }
    
        /// <summary>The LoRaWAN DevAddr.</summary>
        [Newtonsoft.Json.JsonProperty("dev_addr", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Dev_addr { get; set; }
}

After some research I found references to the underlying problem in TTN and OpenAPI forums. The Dev_addr and Dev_eui fields are Base16(Hexidecimal) encoded binary but are being processed as if they were Base64(mime) encoded.

The TTI connector only displays the Device EUI so I changed the Dev_eui property to a string

public partial class V3EndDeviceIdentifiers 
{
        [Newtonsoft.Json.JsonProperty("device_id", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string Device_id { get; set; }
    
        [Newtonsoft.Json.JsonProperty("application_ids", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public V3ApplicationIdentifiers Application_ids { get; set; }
    
        /// <summary>The LoRaWAN DevEUI.</summary>
        [Newtonsoft.Json.JsonProperty("dev_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public string Dev_eui { get; set; }

      /// <summary>The LoRaWAN JoinEUI (AppEUI until LoRaWAN 1.0.3 end devices).</summary>
      [Newtonsoft.Json.JsonProperty("join_eui", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Join_eui { get; set; }
    
        /// <summary>The LoRaWAN DevAddr.</summary>
        [Newtonsoft.Json.JsonProperty("dev_addr", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
        public byte[] Dev_addr { get; set; }
}

I also had to remove the BitConverter.ToString call

foreach (V3EndDevice device in endDevices.End_devices)
{
   if (DeviceAzureEnabled(device))
   {
      _logger.LogInformation("Config-ApplicationID:{0} DeviceID:{1} Device EUI:{2}", device.Ids.Application_ids.Application_id, device.Ids.Device_id, device.Ids.Dev_eui);

      tasks.Add(DeviceRegistration(device.Ids.Application_ids.Application_id, device.Ids.Device_id, _programSettings.ResolveDeviceModelId(device.Ids.Application_ids.Application_id, device.Attributes), stoppingToken));
   }
}

Now the DeviceEUI values are displayed correctly and searching for EndDevices in Azure Application Insights is easier

TTI V3 Connector application running as a console application showing correct DeviceEUIs

Modifying the nSwag generated classes is a really nasty way of fixing the problem but I think this approach is okay as it’s only one field and any other solution I could find was significantly more complex.

TTI V3 Gateway provisioning Dragino LHT65 Uplink

This very long post is about how to connect a Dragino LHT65 Temperature and Humidity sensor to Azure IoT Central using my TTI/TTN V3Azure IoT Connector and the Digital Twin Definition Language (DTDL).

Dragino LHT65 temperature and Humidity sensor

The first step was to add an application(dragino-lht65) in my The Things Industries(TTI) tenant

TTI/TTN application for my Dragino LHT65 devices
Adding devMobile as a collaborator on the new application
TTI Application API Key configuration

The new Application API Key used by the MQTTnet managed client only needs to have write downlink and read uplink traffic enabled.

FTDI Adapter and modified LHT64 cable

So I could reliably connect to my LHT65 devices to configure them I modified a programming cable so I could use it with a spare FTDI adaptor without jumper wires. Todo this I used a small jewelers screwdriver to “pop” out the VCC cable and move the transmit data line.

After entering the device password and checking the firmware version I used the AT+CFG command to display the device settings

AT+CFG: Print all configurations

[334428]***** UpLinkCounter= 0 *****
[334430]TX on freq 923200000 Hz at DR 2
[334804]txDone
[339807]RX on freq 923200000 Hz at DR 2
[339868]rxTimeOut
[340807]RX on freq 923200000 Hz at DR 2
[340868]rxTimeOut

Correct Password

Stop Tx events,Please wait for all configurations to print
Printf all config...
AT+DEUI=a8 .. .. .. .. .. .. d6
AT+DADDR=01......D6

AT+APPKEY=9d .. .. .. .. .. .. .. .. .. .. .. .. .. .. 2e
AT+NWKSKEY=f6 .. .. .. .. .. .. .. .. .. .. .. .. .. .. 69
AT+APPSKEY=4c 35 .. .. .. .. .. .. .. .. .. .. .. .. .. 3d
AT+APPEUI=a0 .. .. .. .. .. .. 00
AT+ADR=1
AT+TXP=0
AT+DR=0
AT+DCS=0
AT+PNM=1
AT+RX2FQ=923200000
AT+RX2DR=2
AT+RX1DL=1000
AT+RX2DL=2000
AT+JN1DL=5000
AT+JN2DL=6000
AT+NJM=1
AT+NWKID=00 00 00 00
AT+FCU=0
AT+FCD=0
AT+CLASS=A
AT+NJS=0
AT+RECVB=0:
AT+RECV=0:
AT+VER=v1.7 AS923

AT+CFM=0
AT+CFS=0
AT+SNR=0
AT+RSSI=0
AT+TDC=1200000
AT+PORT=2
AT+PWORD=123456
AT+CHS=0
AT+DATE=21/3/26 07:49:15
AT+SLEEP=0
AT+EXT=4,2
AT+RTP=20
AT+BAT=3120
AT+WMOD=0
AT+ARTEMP=-40,125
AT+CITEMP=1
Start Tx events OK


[399287]***** UpLinkCounter= 0 *****

[399289]TX on freq 923400000 Hz at DR 2

[399663]txDone

[404666]RX on freq 923400000 Hz at DR 2

[404726]rxTimeOut

[405666]RX on freq 923200000 Hz at DR 2

[405726]rxTimeOut

I copied the AppEUI and DevEUI for use on the TI Dragino LHT65 Register end device form provided by the TTI/TTN.

TTYI/TTN Dragino LHT65 Register end device

The Dragino LHT65 uses the DeviceEUI as the DeviceID which meant I had todo more redaction in my TTI/TTN and Azure Application Insights screen captures. The rules around the re-use of EndDevice ID were a pain in the arse(PITA) in my development focused tenant.

Dragino LHT 65 Device uplink payload formatter

The connector supports both uplink and downlink messages with JSON encoded payloads. The Dragino LHT65 has a vendor supplied formatter which is automatically configured when an EndDevice is created. The EndDevice formatter configuration can also be overridden at the Application level in the app.settings.json file.

Device Live Data Uplink Data Payload

Once an EndDevice is configured in TTI/TTN I usually use the “Live data Uplink Payload” to work out the decoded payload JSON property names and data types.

LHT65 Uplink only Azure IoT Central Device Template
LHT65 Device Template View Identity

For Azure IoT Central “automagic” provisioning the DTDLModelId has to be copied from the Azure IoT Central Template into the TTI/TTN EndDevice or app.settings.json file application configuration.

LHT65 Device Template copy DTDL @ID
TTI EndDevice configuring the DTDLV2 @ID at the device level

Configuring the DTDLV2 @ID at the TTI application level in the app.settings.json file

{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "Microsoft": "Debug",
      "Microsoft.Hosting.Lifetime": "Debug"
    },
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Debug"
      }
    }
  },

  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0ne...DD9",
            "GroupEnrollmentKey": "eFR...w=="
          }
        },
        "DTDLModelId": "dtmi:ttnv3connectorclient:FezduinoWisnodeV14x8;4",
        "MQTTAccessKey": "NNSXS.HCY...RYQ",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_0": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_1": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "TemperatureOOBAlertMinimumAndMaximum": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          }
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0ne...DD9",
            "GroupEnrollmentKey": "AtN...g=="
          },
        },
        "DTDLModelId": "dtmi:ttnv3connectorclient:SeeeduinoLoRaWAN4cz;1",
        "MQTTAccessKey": "NNSXS.V44...42A",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      },
      "dragino-lht65": {
        "AzureSettings": {
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0ne...DD9",
            "GroupEnrollmentKey": "SLB...w=="
          }
        },
        "DTDLModelId": "dtmi:ttnv3connectorclient:DraginoLHT656w6;1",
        "MQTTAccessKey": "NNSXS.RIJ...NZQ",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
      "MqttServerName": "eu1.cloud.thethings.industries",
      "MqttClientId": "MQTTClient",
      "MqttAutoReconnectDelay": "00:00:05",
      "Tenant": "...-test",
      "ApiBaseUrl": "https://...-test.eu1.cloud.thethings.industries/api/v3",
      "ApiKey": "NNSXS.NR7...ZSA",
      "Collaborator": "devmobile",
      "DevicePageSize": 10,
      "DeviceIntegrationDefault": true
    }
  }
}

The Azure Device Provisioning Service(DPS) is configured at the TTI application level in the app.settings.json file. The IDScope and one of the Primary or Secondary Shared Access Signature(SAS) keys should be copied into DeviceProvisioningServiceSettings of an Application in the app.settings.json file. I usually set the “Automatically connect devices in this group” flag as part of the “automagic” provisioning process.

Azure IoT Central Group Enrollment Key
Then device templates need to be mapped to an Enrollment Group then Device Group.

For testing the connector application can be run locally with diagnostic information displayed in the application console window as it “automagically’ provisions devices and uploads telemetry data.

Connector application Diagnostics
Azure IoT Central Device list before my LHT65 device is “automagically” provisioned
Azure IoT Central Device list after my LHT65 device is “automagically” provisioned

One a device has been provisioned I check on the raw data display that all the fields I configured have been mapped correctly.

Azure IoT Central raw data display

I then created a dashboard to display the telemetry data from the LHT65 sensors.

Azure IoT Central dashboard displaying LHT65 temperature, humidity and battery voltage graphs.

The dashboard also has a few KPI displays which highlighted an issue which occurs a couple of times a month with the LHT65 onboard temperature sensor values (327.7°). I have connected Dragino technical support and have also been unable to find a way to remove the current an/or filter out future aberrant values.

Azure Application Insights logging

I also noticed that the formatting of the DeviceEUI values in the Application Insights logging was incorrect after trying to search for one of my Seeedstudio LoRaWAN device with its DeviceEUI.

TTN V3 Gateway Downlink Broken

While adding Azure Device Provisioning Service (DPS) support to my The Things Industries(TTI)/The Things Network(TTN) Azure IoT Hub/Azure IoT Central Connector I broke Cloud to Device(C2D)/Downlink messaging. I had copied the Advanced Message Queuing Protocol(AMQP) connection pooling configuration code from my The Things Network Integration assuming it worked.

return DeviceClient.CreateFromConnectionString(connectionString, deviceId,
	new ITransportSettings[]
	{
		new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
		{
			PrefetchCount = 0,
			AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
			{
				Pooling = true,
			}
		}
	});

I hadn’t noticed this issue in my Azure IoT The Things Network Integration because I hadn’t built support for C2D messaging. After some trial and error I figured out the issue was the PrefetchCount initialisation.

return DeviceClient.CreateFromConnectionString(connectionString, deviceId,
	new ITransportSettings[]
	{
		new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
		{
			AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
			{
				Pooling = true,
			}
		}
	});

From the Azure Service Bus (I couldn’t find any specifically Azure IoT Hub ) documentation

Even though the Service Bus APIs do not directly expose such an option today, a lower-level AMQP protocol client can use the link-credit model to turn the “pull-style” interaction of issuing one unit of credit for each receive request into a “push-style” model by issuing a large number of link credits and then receive messages as they become available without any further interaction. Push is supported through the MessagingFactory.PrefetchCount or MessageReceiver.PrefetchCount property settings. When they are non-zero, the AMQP client uses it as the link credit.

n this context, it’s important to understand that the clock for the expiration of the lock on the message inside the entity starts when the message is taken from the entity, not when the message is put on the wire. Whenever the client indicates readiness to receive messages by issuing link credit, it is therefore expected to be actively pulling messages across the network and be ready to handle them. Otherwise the message lock may have expired before the message is even delivered. The use of link-credit flow control should directly reflect the immediate readiness to deal with available messages dispatched to the receiver.

In the Azure IoT Hub SDK the prefetch count is set to 50 (around line 57) and throws an exception if less that zero (around line 90) and there is some information about tuning the prefetch value for Azure Service Bus.

The best explanation I count find was Github issue which was a query “What exactly does the PrefetchCount property control?”

“You are correct, the pre-fetch count is used to set the link credit over AMQP. What this signifies is the max. no. of messages that can be “in-flight” from the service to the client, at any given time. (This value defaults to 50 for the IoT Hub .NET client).
The client specifies its link-credit, that the service must respect. In simplest terms, any time the service sends a message to the client, it decrements the link credit, and will continue sending messages until linkCredit > 0. Once the client acknowledges the message, it will increment the link credit.”

In summary if Prefetch count is set to zero on startup in my application no messages will be sent to the client….

TTN V3 Gateway Azure Configuration Simplication

To reduce complexity the initial version of the V3 TTI gateway didn’t support the Azure Device Provisioning Service(DPS). In preparation for this I had included DeviceProvisioningServiceSettings object in both the Application and AzureSettingsDefault sections.

After trialing a couple of different approaches I have removed the AzureSettingsDefault. If an application has a connectionstring configured that is used, if there is not one then the DPS configuration is used, if there are neither currently the application logs an error. In the future I will look at adding a configuration option to make the application optionally shutdown

{
  ...
  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          "IoTHubConnectionString": "HostName=TT...n1.azure-devices.net;SharedAccessKeyName=device;SharedAccessKey=Am...M=",
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0n...3B",
            "GroupEnrollmentKey": "Kl...Y="
          }
        },
        "MQTTAccessKey": "NNSXS.HC...YQ",
        "DeviceIntegrationDefault": false,
        "DevicePageSize": 10
      },
      "seeeduinolorawan": {
        "AzureSettings": {
          "IoTHubConnectionString": "HostName=TT...n2.azure-devices.net;SharedAccessKeyName=device;SharedAccessKey=D2q...L8=",
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0n...3B",
            "GroupEnrollmentKey": "Kl...Y="
          }
        },
        "MQTTAccessKey": "NNSXS.V44...42A",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },

    "TheThingsIndustries": {
      "MqttServerName": "eu1.cloud.thethings.industries",
      "MqttClientId": "MQTTClient",
      "MqttAutoReconnectDelay": "00:00:05",
      "Tenant": "br...st",
      "ApiBaseUrl": "https://br..st.eu1.cloud.thethings.industries/api/v3",
      "ApiKey": "NNSXS.NR...SA",
      "Collaborator": "de...le",
      "DevicePageSize": 10,
      "DeviceIntegrationDefault": true
    }
  }
}

The implementation of failing back from application to default settings wasn’t easy to implement, explain or document.

Downlink messages NahYeah

While running my The Things IndustriesTTI) gateway I noticed an exception in the logs every so often

Exception of type 'Microsoft.Azure.Devices.Client.Exceptions.DeviceMessageLockLostException' was thrown.

My client subscribes to Message Queue Telemetry Transport Topics(MQTT) (using MQTTNet) for each TTI Application and establishes a connection (using an Azure DeviceClient) for each TTI Device to an Azure IoT Hub(s).

  • v3/{application id}@{tenant id}/devices/{device id}/up
  • v3/{application id}@{tenant id}/devices/{device id}/down/queued
  • v3/{application id}@{tenant id}/devices/{device id}/down/sent
  • v3/{application id}@{tenant id}/devices/{device id}/down/ack
  • v3/{application id}@{tenant id}/devices/{device id}/down/nack
  • v3/{application id}@{tenant id}/devices/{device id}/down/failed

The application subscribes to the queued, ack, nack, and failed topics so the progress of a downlink message can be monitored. For downlink messages the correlation_id “az:LockToken:” contains the message.LockToken so that they can be Abandoned, Completed or Rejected in the MQTT receive messageHandler.

Below is the logging from my application for an odd sequence of messages

*****Nothing much happening for a couple of hours the .'s represent approx 1 second. Wisnode 4 sends roughly every 5 minues

.....................................................................................................................................................................................................................................................................................................................
03:36:08 TTN Uplink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 Port: 5
.....................................................................................................................................................................................................................................................................................................................
03:41:18 TTN Uplink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 Port: 5
...........................................................................
03:42:34 Azure IoT Hub downlink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 LockToken: 57ea0fad-b6b3-492e-b194-10c4ff3e53cb
 Body: vu8=

*****I then started sending 5 messages to Wisnode 5 same payload vu8=, port 71 thru 75 

***** 71 Queued
03:42:34 Queued: v3/application1@tenant1/devices/wisnodetest04/down/queued
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"}},
	"correlation_ids":[
"az:LockToken:57ea0fad-b6b3-492e-b194-10c4ff3e53cb",
"as:downlink:01EXX9B1CA4DB68PKCDAK4SS4H"],
	"downlink_queued":{"f_port":71,"frm_payload":"vu8=","confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:57ea0fad-b6b3-492e-b194-10c4ff3e53cb",
"as:downlink:01EXX9B1CA4DB68PKCDAK4SS4H"]}}
...
03:42:37 Azure IoT Hub downlink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 LockToken: e2fef28c-fb1f-42cd-bb40-3ad8e6051da9
 Body: vu8=
.

***** 72 Queued
03:42:38 Queued: v3/application1@tenant1/devices/wisnodetest04/down/queued
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"}},
	"correlation_ids":[
"az:LockToken:e2fef28c-fb1f-42cd-bb40-3ad8e6051da9",
"as:downlink:01EXX9B4RGSCJ4BN21GHPM85W5"],
	"downlink_queued":{"f_port":72,"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:e2fef28c-fb1f-42cd-bb40-3ad8e6051da9",
"as:downlink:01EXX9B4RGSCJ4BN21GHPM85W5"]}}
...
03:42:41 Azure IoT Hub downlink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 LockToken: 70d61d71-9b24-44d2-b54b-7cc08da4d072
 Body: vu8=

***** 73 Queued
03:42:41 Queued: v3/application1@tenant1/devices/wisnodetest04/down/queued
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"}},
	"correlation_ids":[
"az:LockToken:70d61d71-9b24-44d2-b54b-7cc08da4d072","as:downlink:01EXX9B800WF7FEP56J3EZ3M8A"],
	"downlink_queued":{"f_port":73,"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:70d61d71-9b24-44d2-b54b-7cc08da4d072",
"as:downlink:01EXX9B800WF7FEP56J3EZ3M8A"]}}
...

***** 74 Queued
03:42:45 Azure IoT Hub downlink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 LockToken: 12537728-de4a-4489-ace5-92923e49b8e4
 Body: vu8=
.
03:42:45 Queued: v3/application1@tenant1/devices/wisnodetest04/down/queued
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"}},
	"correlation_ids":[
"az:LockToken:12537728-de4a-4489-ace5-92923e49b8e4",
"as:downlink:01EXX9BBWA2YNCN2DFE5FC3BP3"],
	"downlink_queued":{
"f_port":74,"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:12537728-de4a-4489-ace5-92923e49b8e4",
"as:downlink:01EXX9BBWA2YNCN2DFE5FC3BP3"]}}
...

***** 75 Queued
03:42:48 Azure IoT Hub downlink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 LockToken: 388efc11-4514-406e-8147-9109289095f4
 Body: vu8=

03:42:49 Queued: v3/application1@tenant1/devices/wisnodetest04/down/queued
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"}},
	"correlation_ids":[
"az:LockToken:388efc11-4514-406e-8147-9109289095f4",
"as:downlink:01EXX9BFCM2G51EPYNWGDWPS0N"],
	"downlink_queued":{"f_port":75,"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:388efc11-4514-406e-8147-9109289095f4",
"as:downlink:01EXX9BFCM2G51EPYNWGDWPS0N"]}}

***** Waiting for Wisniode
..........................................................................................................................................................................
03:47:18 TTN Uplink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 Port: 5

***** Waiting for Wisniode again, I think might have been such a long delay becuase TTI didn't get
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
***** 71 Nack'd
03:56:52 Nack: v3/application1@tenant1/devices/wisnodetest04/down/nack
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"},
	"dev_eui":"60C5A8FFFE781691","join_eui":"70B3D57ED0000000","dev_addr":"26083BE1"},
	"correlation_ids":[
"as:downlink:01EXX9B1CA4DB68PKCDAK4SS4H",
"as:up:01EXXA572VHN7X7G5KFTHBQPNG",
"az:LockToken:57ea0fad-b6b3-492e-b194-10c4ff3e53cb",
"gs:conn:01EXRPTTFGFNTRGH7V8FTC3R0S",
"gs:up:host:01EXRPTTFTEXBNV87KZFYFWP5V",
"gs:uplink:01EXXA56VPK14XG5S8JB9Q0V0X",
"ns:uplink:01EXXA56VYCHGGPPN1K77REMNM",
"rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01EXXA56VRG6811HRCF803VJ34"],
	"received_at":"2021-02-07T03:56:53.211893610Z",
	"downlink_nack":{
"session_key_id":"AXd6GPmneD3dKVoArcS36g==",
"f_port":71,"f_cnt":35,
"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:57ea0fad-b6b3-492e-b194-10c4ff3e53cb",
"as:downlink:01EXX9B1CA4DB68PKCDAK4SS4H"]}}

 Found az:LockToken:

03:56:52 TTN Uplink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 Port: 5

03:56:52 Azure IoT Hub downlink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 LockToken: 856f5a9b-bc37-435c-8de9-19d2213999f8
 Body: vu8=

03:56:53 Queued: v3/application1@tenant1/devices/wisnodetest04/down/queued
 payload: {
"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"},
	"correlation_ids":[
"az:LockToken:856f5a9b-bc37-435c-8de9-19d2213999f8",
"as:downlink:01EXXA57JJWWYEDX3Z55TNSTP5"],
	"downlink_queued":{"f_port":71,
"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":
["az:LockToken:856f5a9b-bc37-435c-8de9-19d2213999f8",
"as:downlink:01EXXA57JJWWYEDX3Z55TNSTP5"]}}

......
***** 71 Ack'd
03:56:58 Ack: v3/application1@tenant1/devices/wisnodetest04/down/ack
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"},
	"dev_eui":"60C5A8FFFE781691","join_eui":"70B3D57ED0000000","dev_addr":"26083BE1"},
	"correlation_ids":[
"as:downlink:01EXX9B1CA4DB68PKCDAK4SS4H",
"as:up:01EXXA5D45E77S19TXEV1E4GAJ",
"az:LockToken:57ea0fad-b6b3-492e-b194-10c4ff3e53cb",
"gs:conn:01EXRPTTFGFNTRGH7V8FTC3R0S",
"gs:up:host:01EXRPTTFTEXBNV87KZFYFWP5V",
"gs:uplink:01EXXA5CV73THH2RKEAC2T9MDP",
"ns:uplink:01EXXA5CVDCWPFBTXGGGB3T02W",
"rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01EXXA5CVDEXDFBPYXC0J01Q3E"],
	"received_at":"2021-02-07T03:56:59.397330003Z",
	"downlink_ack":{
"session_key_id":"AXd6GPmneD3dKVoArcS36g==",
"f_port":71,"f_cnt":36,"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL",
	"correlation_ids":[
"az:LockToken:57ea0fad-b6b3-492e-b194-10c4ff3e53cb",
"as:downlink:01EXX9B1CA4DB68PKCDAK4SS4H"]}}

 Found az:LockToken:
Exception of type 'Microsoft.Azure.Devices.Client.Exceptions.DeviceMessageLockLostException' was thrown.

03:56:59 TTN Uplink message
 ApplicationID: application1
 DeviceID: wisnodetest04
 Port: 0
......
03:57:04 Ack: v3/application1@tenant1/devices/wisnodetest04/down/ack
 payload: {"end_device_ids":{"device_id":"wisnodetest04","application_ids":{"application_id":"application1"},
"dev_eui":"60C5A8FFFE781691","join_eui":"70B3D57ED0000000","dev_addr":"26083BE1"},
"correlation_ids":[
"as:downlink:01EXX9B4RGSCJ4BN21GHPM85W5",
"as:up:01EXXA5K2FWGP9DGD7THWZ8HNR",
"az:LockToken:e2fef28c-fb1f-42cd-bb40-3ad8e6051da9",
"gs:conn:01EXRPTTFGFNTRGH7V8FTC3R0S",
"gs:up:host:01EXRPTTFTEXBNV87KZFYFWP5V",
"gs:uplink:01EXXA5JVDR102TKCWQ77P4YYF",
"ns:uplink:01EXXA5JVGNGMZN33FNT47G6PF",
"rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01EXXA5JVGJFFQVEWX2M1XSFKK"],
"received_at":"2021-02-07T03:57:05.487910418Z","downlink_ack":{"session_key_id":"AXd6GPmneD3dKVoArcS36g==",
"f_port":72,"f_cnt":37,
"frm_payload":"vu8=",
"confirmed":true,"priority":"NORMAL","correlation_ids":
["az:LockToken:e2fef28c-fb1f-42cd-bb40-3ad8e6051da9","as:downlink:01EXX9B4RGSCJ4BN21GHPM85W5"]}}

The sequence of messages is a bit odd, in the Azure DeviceClient ReceiveMessageHandler a downlink message is published, then a queued message is received, then a nak and finally an ack, The exception was because my client was trying to Complete the delivery of a message that had already been Abandoned.

Application Insights & Configuration

As part of my The Things IndustriesTTI) Integration my current approach is to use an Azure web job and configure the Azure App Service host so it doesn’t get shutdown after a period of inactivity. This so my application won’t have to repeatedly use the TTI API to request the Application and Device configuration information to reload the cache (still not certain if this is going to be implemented with a ConcurrentDictionary or ObjectCache).

namespace devMobile.TheThingsNetwork.WorkerService
{
   using System.Collections.Generic;

   public class AzureDeviceProvisiongServiceSettings
   {
      public string IdScope { get; set; }
      public string GroupEnrollmentKey { get; set; }
   }

   public class AzureSettings
   {
      public string IoTHubConnectionString { get; set; }
      public AzureDeviceProvisiongServiceSettings DeviceProvisioningServiceSettings { get; set; }
   }

   public class ApplicationSetting
   {
      public AzureSettings AzureSettings { get; set; }

      public string MQTTAccessKey { get; set; }

      public byte? ApplicationPageSize { get; set; }

      public bool? DeviceIntegrationDefault { get; set; }
      public byte? DevicePageSize { get; set; }
   }

   public class TheThingsIndustries
   {
      public string MqttServerName { get; set; }
      public string MqttClientName { get; set; }

      public string Tennant { get; set; }
      public string ApiBaseUrl { get; set; }
      public string ApiKey { get; set; }

      public bool ApplicationIntegrationDefault { get; set; }
      public byte ApplicationPageSize { get; set; }

      public bool DeviceIntegrationDefault { get; set; }
      public byte DevicePageSize { get; set; }
   }

   public class ProgramSettings
   {
      public TheThingsIndustries TheThingsIndustries { get; set; }

      public AzureSettings AzureSettingsDefault { get; set; }

      public Dictionary<string, ApplicationSetting> Applications { get; set; }
   }
}

The amount of configuration required to support multiple TTI Applications containing many Devices is also starting to get out of hand.

I need to subscribe to a Message Queue Telemetry Transport Topics(MQTT using MQTTNet) for each Application and establish a connection (using an Azure DeviceClient) for each TTI Device to the configured Azure IoT Hub(s).

  • v3/{application id}@{tenant id}/devices/{device id}/up
  • v3/{application id}@{tenant id}/devices/{device id}/down/queued
  • v3/{application id}@{tenant id}/devices/{device id}/down/sent
  • v3/{application id}@{tenant id}/devices/{device id}/down/ack
  • v3/{application id}@{tenant id}/devices/{device id}/down/nack
  • v3/{application id}@{tenant id}/devices/{device id}/down/failed

The Azure DeviceClient has to be configured and OpenAsync called just before/after subscribing to the TTI Application /up topic so the SendEventAsync method can be called to send messages to the configured Azure IoT Hub(s). For downlink messages the SetReceiveMessageHandler method will need to be called just before/after subscribing to ../down/queued, ../down/sent,../down/ack,…/down/nack and ,…/down/failed downlink topics.

The ordering of downloading the Application and Device configuration so downlink messages can be sent and uplink message received as soon as possible (so no messages are lost) is important. I have considered making the downlink process multi-threaded so API calls are made concurrently but I’m not certain the additional complexity would be worth it, especially in initial versions.

I’m also currently not certain about how to register my program for Application and Device registry changes so it doesn’t have to be restarted when configuration changes. I have also considered reverting to an HTTP Integration so that I could use Azure Storage queues to buffer uplink and downlink messages. This may also introduce ordering issues when multiple threads are created for Azure Queue Trigger functions to process a message backlog.

For debugging the application and monitoring in production I was planning on using the Apache Log4Net library but now I’m not certain the additional configuration complexity and dependencies are worth it. The built in Microsoft.Extensions.Logging library with Azure Application Insights integration looks like a “light weight” alternative with sufficient functionality .

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
   while (!stoppingToken.IsCancellationRequested)
   {
      _logger.LogDebug("Debug worker running at: {time}", DateTimeOffset.Now);
      _logger.LogInformation("Info worker running at: {time}", DateTimeOffset.Now);
      _logger.LogWarning("Warning worker running at: {time}", DateTimeOffset.Now);
      _logger.LogError("Error running at: {time}", DateTimeOffset.Now);

      using (_logger.BeginScope("TheThingsIndustries configuration"))
      {
         _logger.LogInformation("Tennant: {0}", _programSettings.TheThingsIndustries.Tennant);
         _logger.LogInformation("ApiBaseUrl: {0}", _programSettings.TheThingsIndustries.ApiBaseUrl);
         _logger.LogInformation("ApiKey: {0}", _programSettings.TheThingsIndustries.ApiKey);

         _logger.LogInformation("ApplicationPageSize: {0}", _programSettings.TheThingsIndustries.ApplicationPageSize);
         _logger.LogInformation("DevicePageSize: {0}", _programSettings.TheThingsIndustries.DevicePageSize);

         _logger.LogInformation("ApplicationIntegrationDefault: {0}", _programSettings.TheThingsIndustries.ApplicationIntegrationDefault);
         _logger.LogInformation("DeviceIntegrationDefault: {0}", _programSettings.TheThingsIndustries.DeviceIntegrationDefault);

         _logger.LogInformation("MQTTServerName: {0}", _programSettings.TheThingsIndustries.MqttServerName);
         _logger.LogInformation("MQTTClientName: {0}", _programSettings.TheThingsIndustries.MqttClientName);
      }

      using (_logger.BeginScope("Azure default configuration"))
      {
         if (_programSettings.AzureSettingsDefault.IoTHubConnectionString != null)
         {
            _logger.LogInformation("AzureSettingsDefault.IoTHubConnectionString: {0}", _programSettings.AzureSettingsDefault.IoTHubConnectionString);
         }

         if (_programSettings.AzureSettingsDefault.DeviceProvisioningServiceSettings != null)
         {
            _logger.LogInformation("AzureSettings.DeviceProvisioningServiceSettings.IdScope: {0}", _programSettings.AzureSettingsDefault.DeviceProvisioningServiceSettings.IdScope);
            _logger.LogInformation("AzureSettings.DeviceProvisioningServiceSettings.GroupEnrollmentKey: {0}", _programSettings.AzureSettingsDefault.DeviceProvisioningServiceSettings.GroupEnrollmentKey);
         }
      }
    
      foreach (var application in _programSettings.Applications)
      {
         using (_logger.BeginScope(new[] { new KeyValuePair<string, object>("Application", application.Key)}))
         {
            _logger.LogInformation("MQTTAccessKey: {0} ", application.Value.MQTTAccessKey);

            if (application.Value.ApplicationPageSize.HasValue)
            {
               _logger.LogInformation("ApplicationPageSize: {0} ", application.Value.ApplicationPageSize.Value);
            }

            if (application.Value.DeviceIntegrationDefault.HasValue)
            {
               _logger.LogInformation("DeviceIntegation: {0} ", application.Value.DeviceIntegrationDefault.Value);
            }

            if (application.Value.DevicePageSize.HasValue)
            {
               _logger.LogInformation("DevicePageSize: {0} ", application.Value.DevicePageSize.Value);
            }

            if (application.Value.AzureSettings.IoTHubConnectionString != null)
            {
               _logger.LogInformation("AzureSettings.IoTHubConnectionString: {0} ", application.Value.AzureSettings.IoTHubConnectionString);
            }

            if (application.Value.AzureSettings.DeviceProvisioningServiceSettings != null)
            {
               _logger.LogInformation("AzureSettings.DeviceProvisioningServiceSettings.IdScope: {0} ", application.Value.AzureSettings.DeviceProvisioningServiceSettings.IdScope);
               _logger.LogInformation("AzureSettings.DeviceProvisioningServiceSettings.GroupEnrollmentKey: {0} ", application.Value.AzureSettings.DeviceProvisioningServiceSettings.GroupEnrollmentKey);
            }
         }
      }

      await Task.Delay(300000, stoppingToken);
   }
}

The logging information formatting is sufficiently readable when running locally

Extensive use of the BeginScope method to include additional meta-data on logged records should make debugging easier.

This long post is to explain some of my design decisions and which ones are still to be decided

Azure IoT Central Connectivity Part4

The Things Network(TTN) Friendly Commands

I have built a several Proof of Concept(PoC) applications (Azure IoT Central Basic Telemetry, Basic Commands, and Request Commands) to explore to how an Azure IoT Central integration with TTN could work. This blog post is about how to configure queued and non queued Cloud to Device(C2D) Commands with request parameters so they should work with my TTN Message Queue Telemetry Transport(MQTT) Data API connector.

I have focused on commands with Analog values but the same approach should be valid for other parameter types like Boolean, Date, DateTime, Double, Duration, Enumeration, Float, Geopoint, Vector, Integer, Long, String, and Time.

Multiple versions of my Azure IoT Central templates

There was a lot of “trial and error” (26 template versions) required to figure out how to configure commands and queued commands so they can and used in TTN downlink payloads.

{
  "end_device_ids": {
    "device_id": "dev1",
    "application_ids": {
      "application_id": "app1"
    },
    "dev_eui": "4200000000000000",
    "join_eui": "4200000000000000",
    "dev_addr": "00E6F42A"
  },
  "correlation_ids": [
    "my-correlation-id",
    "..."
  ],
  "downlink_ack": {
    "session_key_id": "AWnj0318qrtJ7kbudd8Vmw==",
    "f_port": 15,
    "f_cnt": 11,
    "frm_payload": "....",
    "decoded_payload": 
    {
      "Value_0":"1.23"
      ...
    }
    "confirmed": true,
    "priority": "NORMAL",
    "correlation_ids": [
      "my-correlation-id",
      "..."
    ]
  }
}

My Azure IoT Central client application displays the generated message message including the decoded_payload field which is used by the built in Cayenned Low Power Protocol(LPP) decoder/encoder and other custom encoders/decoders.

Azure IoT Central commands for TTN/TTI integration

From the “Device Commands” form I can send commands and a queued commands which have float parameters or object parameters which contain one or more float values in a JSON payload.

For commands which call the methodHander which was been registered by calling SetMethodDefaultHandlerAsync the request payload can be JSON or plain text. If the payload is valid JSON it is “grafted”(couldn’t think of a better word) into the decoded_payload field. If the payload is not valid a JSON object with the method name as the “name” and the text payload as the value is added the decoded_payload.

private static async Task<MethodResponse> MethodCallbackDefaultHandler(MethodRequest methodRequest, object userContext)
{
   AzureIoTMethodHandlerContext receiveMessageHandlerConext = (AzureIoTMethodHandlerContext)userContext;

   Console.WriteLine($"Default handler method {methodRequest.Name} was called.");

   Console.WriteLine($"Payload:{methodRequest.DataAsJson}");
   Console.WriteLine();

   if (string.IsNullOrWhiteSpace(methodRequest.Name))
   {
      Console.WriteLine($"   Method Request Name null or white space");
      return new MethodResponse(400);
   }

   string payloadText = Encoding.UTF8.GetString(methodRequest.Data);
   if (string.IsNullOrWhiteSpace(payloadText))
   {
       Console.WriteLine($"   Payload null or white space");
       return new MethodResponse(400);
   }

   // At this point would check to see if Azure DeviceClient is in cache, this is so nasty
   if ( String.Compare( methodRequest.Name, "Analog_Output_1", true) ==0 )
   {
      Console.WriteLine($"   Device not found");
      return new MethodResponse(UTF8Encoding.UTF8.GetBytes("Device not found"), 404);
   }

   JObject payload;

   if (IsValidJSON(payloadText))
   {
      payload = JObject.Parse(payloadText);
   }
   else
   {
      payload = new JObject
      {
         { methodRequest.Name, payloadText }
      };
   }

   string downlinktopic = $"v3/{receiveMessageHandlerConext.ApplicationId}@{receiveMessageHandlerConext.TenantId}/devices/{receiveMessageHandlerConext.DeviceId}/down/push";

   DownlinkPayload downlinkPayload = new DownlinkPayload()
   {
      Downlinks = new List<Downlink>()
      {
         new Downlink()
         {
            Confirmed = false,
            //PayloadRaw = messageBody,
            PayloadDecoded = payload,
            Priority = DownlinkPriority.Normal,
            Port = 10,
            /*
            CorrelationIds = new List<string>()
            {
               methodRequest.LockToken
            }
            */
         }
      }
   };

   Console.WriteLine($"TTN Topic :{downlinktopic}");
   Console.WriteLine($"TTN downlink JSON :{JsonConvert.SerializeObject(downlinkPayload, Formatting.Indented)}");

   return new MethodResponse(200);
}
Configuration of unqueued Commands with a typed payload
The output of my test harness for a Command for a typed payload
Configuring fields of object payload(JSON)

A JSON request payload also supports downlink messages with more that one value.

The output of my test harness for a Command with an object payload(JSON)

For queued commands which call the ReceiveMessageHandler which has was registered by calling SetReceiveMessageHandler the request payload is JSON or plain text.

private async static Task ReceiveMessageHandler(Message message, object userContext)
{
   AzureIoTMessageHandlerContext receiveMessageHandlerConext = (AzureIoTMessageHandlerContext)userContext;

   Console.WriteLine($"ReceiveMessageHandler handler method was called.");

   Console.WriteLine($" Message ID:{message.MessageId}");
   Console.WriteLine($" Message Schema:{message.MessageSchema}");
   Console.WriteLine($" Correlation ID:{message.CorrelationId}");
   Console.WriteLine($" Lock Token:{message.LockToken}");
   Console.WriteLine($" Component name:{message.ComponentName}");
   Console.WriteLine($" To:{message.To}");
   Console.WriteLine($" Module ID:{message.ConnectionModuleId}");
   Console.WriteLine($" Device ID:{message.ConnectionDeviceId}");
   Console.WriteLine($" User ID:{message.UserId}");
   Console.WriteLine($" CreatedAt:{message.CreationTimeUtc}");
   Console.WriteLine($" EnqueuedAt:{message.EnqueuedTimeUtc}");
   Console.WriteLine($" ExpiresAt:{message.ExpiryTimeUtc}");
   Console.WriteLine($" Delivery count:{message.DeliveryCount}");
   Console.WriteLine($" InputName:{message.InputName}");
   Console.WriteLine($" SequenceNumber:{message.SequenceNumber}");

   foreach (var property in message.Properties)
   {
      Console.WriteLine($"   Key:{property.Key} Value:{property.Value}");
   }

   Console.WriteLine($" Content encoding:{message.ContentEncoding}");
   Console.WriteLine($" Content type:{message.ContentType}");
   string payloadText = Encoding.UTF8.GetString(message.GetBytes());
   Console.WriteLine($" Content:{payloadText}");
   Console.WriteLine();

   if (!message.Properties.ContainsKey("method-name"))
   {
      await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message);
      Console.WriteLine($"   Property method-name not found");
      return;
   }

   string methodName = message.Properties["method-name"];
   if (string.IsNullOrWhiteSpace( methodName))
   {
      await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message);
      Console.WriteLine($"   Property null or white space");
      return;
   }

   if (string.IsNullOrWhiteSpace(payloadText))
   {
      await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message);
      Console.WriteLine($"   Payload null or white space");
      return;
   }

   JObject payload;

   if (IsValidJSON(payloadText))
   {
      payload = JObject.Parse(payloadText);
   }
   else
   {
      payload = new JObject
      {
         { methodName, payloadText }
      };
   }

   string downlinktopic = $"v3/{receiveMessageHandlerConext.ApplicationId}@{receiveMessageHandlerConext.TenantId}/devices/{receiveMessageHandlerConext.DeviceId}/down/push";

   DownlinkPayload downlinkPayload = new DownlinkPayload()
   {
      Downlinks = new List<Downlink>()
      {
         new Downlink()
         {
            Confirmed = false,
            //PayloadRaw = messageBody,
            PayloadDecoded = payload,
            Priority = DownlinkPriority.Normal,
            Port = 10,
            CorrelationIds = new List<string>()
            {
               message.LockToken
            }
         }
      }
   };

   Console.WriteLine($"TTN Topic :{downlinktopic}");
   Console.WriteLine($"TTN downlink JSON :{JsonConvert.SerializeObject(downlinkPayload, Formatting.Indented)}");

   //await receiveMessageHandlerConext.AzureIoTHubClient.AbandonAsync(message); // message retries
   //await receiveMessageHandlerConext.AzureIoTHubClient.CompleteAsync(message);
   await receiveMessageHandlerConext.AzureIoTHubClient.CompleteAsync(message.LockToken);
   //await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message); // message gone no retry
}

When I initiated an Analog queued command the message handler was invoked with the name of the command capability (Analog_Output_2) in a message property called “method-name”. For a typed parameter the message content was a string representation of the value. For an object parameter the payload contains a JSON representation of the request field(s)

The output of my test harness for a Queued Command with a typed payload

A JSON request payload supports downlink message with more that one value.

The output of my test harness for a Queued Command with an object payload(JSON)

The choice of Value_0, Value_1 (I think they are float64 type) etc. for the decoded_payload is specified in the Cayenne LPP downlink decode/encoder source code.

The context information for both comments and queued commands provides additional information required to construct the MQTT topic for publishing the downlink messages.

For queued commands the correlation_id will contain the message.LockToken so that messages can be Abandoned, Completed or Rejected. The MQTT broker publishes a series of topics so the progress of the transmission of downlink message can be monitored.

If the device is not known the Abandon method will be called immediately. For command messages Completed will be called as soon as the message is “sent”

  • v3/{application id}@{tenant id}/devices/{device id}/down/queued
  • v3/{application id}@{tenant id}/devices/{device id}/down/sent
  • v3/{application id}@{tenant id}/devices/{device id}/down/ack
  • v3/{application id}@{tenant id}/devices/{device id}/down/nack
  • v3/{application id}@{tenant id}/devices/{device id}/down/failed

For queued messages the point in the delivery process where the Abandoned, Completed and Rejected methods will be called will be configurable.

Azure IoT Central Connectivity Part3

Request Commands

I have built a couple of proof of Concept(PoC) applications to explore the Basic Telemetry and Basic Command functionality of Azure IoT Central. This blog post is about queued and non queued Cloud to Device(C2D) Commands with request parameters.

I initially created an Azure IoT Central Device Template with command and telemetry device capabilities.

“Collapsed” Command Request template
Command Request Template digital commands

I tried typed request and object based parameters to explorer how an integration with The Things Network(TTN)/The Things Industries(TTI) using the Message Queue Telemetry Transport(MQTT) interface could work.

Object parameter schema designer

With object based parameters the request JSON could contain more than one value though the validation of user provided information didn’t appear to be as robust.

Object parameter schema definition

I “migrated” my third preconfigured device to the CommandRequest template to see how the commands with Request parameters interacted with my PoC application.

After “migrating” my device I went back and created a Template view so I could visualise the simulated telemetry from my PoC application and provide a way to initiate commands (Didn’t really need four command tiles as they all open the Device commands form).

CommandRequest device template default view

From the Device Commands form I could send commands and a queued commands which had analog or digital parameters.

Device Three Command Tab

When I initiated an Analog non-queued command the default method handler was invoked with the name of the command capability (Analog_Output_1) as the method name and the payload contained a JSON representation of the request values(s). With a typed parameter a string representation of the value was in the message payload. With a typed parameter a string representation of the value was in the message payload rather than JSON.

Console application displaying Analog request and Analog Request queued commands

When I initiated an Analog queued command the message handler was invoked with the name of the command capability (Analog_Output_2) in a message property called “method-name” and the payload contained a JSON representation of the request value(s). With a typed parameter a string representation of the value was in the message payload rather than JSON.

When I initiated a Digital non-queued command the default method handler was invoked with the name of the command capability (Digital_Output_1) as the method name and the payload contained a JSON representation of the request values(s). With a typed parameter a string representation of the value was in the message payload rather than JSON.

Console application displaying Digital request and Digital Request queued commands

When I initiated a Digital queued command the message handler was invoked with the name of the command capability(Digital_Output_2) in a message property called “method-name” and the payload contained a JSON representation of the request value(s). With a typed parameter a string representation of the value was in the message payload rather than JSON.

The validation of user input wasn’t as robust as I expected, with problems selecting checkboxes with a mouse when there were several Boolean fields. I often had to click on a nearby input field and use the TAB button to navigate to the desired checkbox. I also had problems with ISO 8601 format date validation as the built in Date Picker returned a month, day, year date which was not editable and wouldn’t pass validation.

The next logical step would be to look at commands with a Response parameter but as the MQTT interface is The Things Network(TTN) and The Things Industries(TTI) is asynchronous and devices reporting every 5 minutes to a couple of times a day there could be a significant delay between sending a message and receiving an optional delivery confirmation or response.

Azure IoT Central Connectivity Part2

Basic Commands

I have been struggling with making The Things Network(TTN) and The Things Industries(TTI) uplink/downlink messages work well Azure IoT Central. To explore different messaging approaches I have built a proof of Concept(PoC) application which simulates TTN/TTI connectivity to an Azure IoT Hub, or Azure IoT Central.

This blog post is about queued and non queued Cloud to Device(C2D) commands without request or response parameters. I have mostly used non queued commands in other projects (my Azure IoT Hub LoRa and RF24L01 gateways) to “Restart” devices etc..

The first step was to create an Azure IoT Central Device Template with command and telemetry device capabilities.

CommandBasic device template device with command & telemetry capabilities

I then “migrated” my second preconfigured device to the CommandBasic template.

Migrating a device to TelemetryBasic template

I then went back and created a Template view to visualise the telemetry from my console application and initiate commands.

CommandBasic device template default view

I modified the PoC application adding handlers for Methods (SetMethodDefaultHandlerAsync) and Messages (SetReceiveMessageHandlerAsync).

private static async Task ApplicationCore(CommandLineOptions options)
{
   DeviceClient azureIoTHubClient;
   Timer MessageSender;

   try
   {
      // Open up the connection
      azureIoTHubClient = DeviceClient.CreateFromConnectionString(options.AzureIoTHubconnectionString, TransportType.Amqp_Tcp_Only);

      await azureIoTHubClient.OpenAsync();
      await azureIoTHubClient.SetReceiveMessageHandlerAsync(ReceiveMessageHandler, azureIoTHubClient);
      await azureIoTHubClient.SetMethodHandlerAsync("Named", MethodCallbackNamedHandler, null);
      await azureIoTHubClient.SetMethodDefaultHandlerAsync(MethodCallbackDefaultHandler, null);

      MessageSender = new Timer(TimerCallbackAsync, azureIoTHubClient, new TimeSpan(0, 0, 10), new TimeSpan(0, 2, 0));

      Console.WriteLine("Press any key to exit");
      while (!Console.KeyAvailable)
      {
         await Task.Delay(100);
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine($"Main {ex.Message}");
      Console.WriteLine("Press <enter> to exit");
      Console.ReadLine();
   }
}

The method handler displays the method name and the message payload.

private static async Task<MethodResponse> MethodCallbackDefaultHandler(MethodRequest methodRequest, object userContext)
{
   Console.WriteLine($"Default handler method {methodRequest.Name} was called.");

   Console.WriteLine($"Payload:{methodRequest.DataAsJson}");
   Console.WriteLine();

   //return new MethodResponse(400);
   //return new MethodResponse(404);
   return new MethodResponse(200);
}

The message handler displays a selection the message properties, any attributes and the message payload.

 private async static Task ReceiveMessageHandler(Message message, object userContext)
{
   DeviceClient azureIoTHubClient = (DeviceClient)userContext;

   Console.WriteLine($"ReceiveMessageHandler handler method was called.");

   Console.WriteLine($" Message ID:{message.MessageId}");
   Console.WriteLine($" Message Schema:{message.MessageSchema}");
   Console.WriteLine($" Correlation ID:{message.CorrelationId}");
   Console.WriteLine($" Component name:{message.ComponentName}");
   Console.WriteLine($" To:{message.To}");
   Console.WriteLine($" Module ID:{message.ConnectionModuleId}");
   Console.WriteLine($" Device ID:{message.ConnectionDeviceId}");
   Console.WriteLine($" CreatedAt:{message.CreationTimeUtc}");
   Console.WriteLine($" EnqueuedAt:{message.EnqueuedTimeUtc}");
   Console.WriteLine($" ExpiresAt:{message.ExpiryTimeUtc}");
   Console.WriteLine($" Delivery count:{message.DeliveryCount}");
   Console.WriteLine($" InputName:{message.InputName}");
   Console.WriteLine($" SequenceNumber:{message.SequenceNumber}");

   foreach (var property in message.Properties)
   {
     Console.WriteLine($"   Key:{property.Key} Value:{property.Value}");
   }

   Console.WriteLine($" Content encoding:{message.ContentEncoding}");
   Console.WriteLine($" Content type:{message.ContentType}");
   Console.WriteLine($" Content:{Encoding.UTF8.GetString(message.GetBytes())}");
   Console.WriteLine();

   //await azureIoTHubClient.AbandonAsync(message); // message retries
   await azureIoTHubClient.CompleteAsync(message);
   //await azureIoTHubClient.RejectAsync(message); // message gone no retry
}

From the Device Commands tab I can could non queued and a queued commands

Device Two Commands tab

When I sent a non-queued command the default method handler was invoked with the name of the command capability (Digital_Output_0) as the method name and an empty payload. In the Azure IoT Central interface I couldn’t see any difference for successful (HTTP 200 OK) or failure (HTTP 400 Bad Request or HTTP 404 Not Found) responses. If the application was not running the command failed immediately.

Console application displaying non-queued call

With Azure IoT Explorer failure responses were visible.

Azure IoT Explorer show message with 404 response

When I sent a queued command the message handler was invoked with the name of the command capability(Digital_Output_1) in a message property called “method-name” and the payload contained only an “@” character.

Console application displaying queued call

If the application was not running the command was queued until the Console application was started. When the console application was running and AbandonAsync was called rather than CompleteAsync the message was retried 10 times. If RejectAsync was called rather than CompleteAsync the message was deleted from the queue and not retried. There didn’t appear to be any difference for the displayed Azure IoT Central or Azure IoT Hub explorer results when AbandonAsync or RejectAsync were called.

I also created a personal dashboard to visualise the telemetry data and initiate commands. The way the two commands were presented on the dashboard was quite limited so I will go back to the documentation and see what I missed