TTN V3 EndDevice API Basic Client

The next step was to enumerate all the EndDevices of a The Things Network(TTN) Application and display their attributes. I have to establish an Azure DeviceClient connection to an Azure IoT Hub for each TTN EndDevice to get downlink messages. To do this I will have to enumerate the TTN Applications in the instance then enumerate the LoRaWAN EndDevices.

using (HttpClient httpClient = new HttpClient())
{
	EndDeviceRegistryClient endDeviceRegistryClient = new EndDeviceRegistryClient(baseUrl, httpClient)
	{
		ApiKey = apiKey
	};

	try
	{
#if FIELDS_MINIMUM
		string[] fieldMaskPathsDevice = { "attributes" }; // think this is the bare minimum required for integration
#else
		string[] fieldMaskPathsDevice = { "name", "description", "attributes" };
#endif
		V3EndDevices endDevices = await endDeviceRegistryClient.ListAsync(applicationID, field_mask_paths:fieldMaskPathsDevice);
		if ((endDevices != null) && (endDevices.End_devices != null)) // If there are no devices returns null rather than empty list
		{
			foreach (V3EndDevice endDevice in endDevices.End_devices)
			{
#if FIELDS_MINIMUM
				Console.WriteLine($"EndDevice ID:{endDevice.Ids.Device_id}");
#else
				Console.WriteLine($"Device ID:{endDevice.Ids.Device_id} Name:{endDevice.Name} Description:{endDevice.Description}");
				Console.WriteLine($"  CreatedAt: {endDevice.Created_at:dd-MM-yy HH:mm:ss} UpdatedAt: {endDevice.Updated_at:dd-MM-yy HH:mm:ss}");
#endif
				if (endDevice.Attributes != null)
				{
					Console.WriteLine("  EndDevice attributes");

					foreach (KeyValuePair<string, string> attribute in endDevice.Attributes)
					{
						Console.WriteLine($"    Key: {attribute.Key} Value: {attribute.Value}");
					}
				}
				Console.WriteLine();
			}
		}
	}
	catch (Exception ex)
	{
		Console.WriteLine(ex.Message);
	}

	Console.WriteLine("Press <enter> to exit");
	Console.ReadLine();
}

Like the applicationRegistryClient.ListAsync call the endDeviceRegistryClient.ListAsync also returns null rather than an empty list.

I also wanted to explore whether I could use EndDevice attributes to populate the ClientOptions ModelId of my CreateFromConnectionString call. The modelId would contain the Digital Twins Definition Language(DTDL) ID of the LoRaWAN device so it could be automatically provisioned.

TTN V3 Application API Basic Paging and Filtering Client

The next step was to enumerate The Things Network(TTN) Applications so I could connect only to the required Azure IoT hub(s). There would also be a single configuration setting for the client (establish a connection for every TTN application, or don’t establish a connection for any) and this could be overridden with a TTN application attribute

long pageSize = long.Parse(args[3]);
Console.WriteLine($"Page size: {pageSize}");

Console.WriteLine();

using (HttpClient httpClient = new HttpClient())
{
	ApplicationRegistryClient applicationRegistryClient = new ApplicationRegistryClient(baseUrl, httpClient)
	{
		ApiKey = apiKey
	};

	try
	{
		int page = 1;

		string[] fieldMaskPathsApplication = { "attributes" }; // think this is the bare minimum required for integration

		V3Applications applications = await applicationRegistryClient.ListAsync(collaborator, field_mask_paths: fieldMaskPathsApplication, limit: pageSize, page: page);
		while ((applications != null) && (applications.Applications != null)) 
		{
			Console.WriteLine($"Applications:{applications.Applications.Count} Page:{page} Page size:{pageSize}");
			foreach (V3Application application in applications.Applications)
			{
				bool applicationIntegration = ApplicationAzureintegrationDefault;

				Console.WriteLine($"Application ID:{application.Ids.Application_id}");
				if (application.Attributes != null)
				{
					string ApplicationAzureIntegrationValue = string.Empty;
					if (application.Attributes.TryGetValue(ApplicationAzureIntegrationField, out ApplicationAzureIntegrationValue))
					{
						bool.TryParse(ApplicationAzureIntegrationValue, out applicationIntegration);
					}

					if (applicationIntegration)
					{
						Console.WriteLine("  Application attributes");

						foreach (KeyValuePair<string, string> attribute in application.Attributes)
						{
							Console.WriteLine($"   Key: {attribute.Key} Value: {attribute.Value}");
						}
					}
				}
				Console.WriteLine();
			}
			page += 1;
			applications = await applicationRegistryClient.ListAsync(collaborator, field_mask_paths: fieldMaskPathsApplication, limit: pageSize, page: page);
		};
	}
	catch (Exception ex)
	{
		Console.WriteLine(ex.Message);
	}

	Console.WriteLine("Press <enter> to exit");
	Console.ReadLine();
}

I Used the field_mask_paths parameter (don’t need created_at, updated_at, name etc.) to minimise the data returned to my client.

public async System.Threading.Tasks.Task<V3Applications> ListAsync(string collaborator_organization_ids_organization_id = null, string collaborator_user_ids_user_id = null, string collaborator_user_ids_email = null, System.Collections.Generic.IEnumerable<string> field_mask_paths = null, string order = null, long? limit = null, long? page = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken))
{
   var urlBuilder_ = new System.Text.StringBuilder();
   urlBuilder_.Append(BaseUrl != null ? BaseUrl.TrimEnd('/') : "").Append("/applications?");
   if (collaborator_organization_ids_organization_id != null) 
   {
         urlBuilder_.Append(System.Uri.EscapeDataString("collaborator.organization_ids.organization_id") + "=").Append(System.Uri.EscapeDataString(ConvertToString(collaborator_organization_ids_organization_id, System.Globalization.CultureInfo.InvariantCulture))).Append("&");
   }
   if (collaborator_user_ids_user_id != null) 
   {
         urlBuilder_.Append(System.Uri.EscapeDataString("collaborator.user_ids.user_id") + "=").Append(System.Uri.EscapeDataString(ConvertToString(collaborator_user_ids_user_id, System.Globalization.CultureInfo.InvariantCulture))).Append("&");
   }
   if (collaborator_user_ids_email != null) 
   {
         urlBuilder_.Append(System.Uri.EscapeDataString("collaborator.user_ids.email") + "=").Append(System.Uri.EscapeDataString(ConvertToString(collaborator_user_ids_email, System.Globalization.CultureInfo.InvariantCulture))).Append("&");
   }
   if (field_mask_paths != null) 
   {
         foreach (var item_ in field_mask_paths) { urlBuilder_.Append(System.Uri.EscapeDataString("field_mask.paths") + "=").Append(System.Uri.EscapeDataString(ConvertToString(item_, System.Globalization.CultureInfo.InvariantCulture))).Append("&"); }
   }
   if (order != null) 
   {
         urlBuilder_.Append(System.Uri.EscapeDataString("order") + "=").Append(System.Uri.EscapeDataString(ConvertToString(order, System.Globalization.CultureInfo.InvariantCulture))).Append("&");
   }
   if (limit != null) 
   {
         urlBuilder_.Append(System.Uri.EscapeDataString("limit") + "=").Append(System.Uri.EscapeDataString(ConvertToString(limit, System.Globalization.CultureInfo.InvariantCulture))).Append("&");
   }
   if (page != null) 
   {
         urlBuilder_.Append(System.Uri.EscapeDataString("page") + "=").Append(System.Uri.EscapeDataString(ConvertToString(page, System.Globalization.CultureInfo.InvariantCulture))).Append("&");
   }
}

I was hoping that there would be a away to further “shape” the returned data, but in the NSwag generated code the construction of the URL with field_mask_paths, order, limit, and page parameters meant this appears not to be possible.

TTN V3 Application API Basic Paging Client

The next step was to enumerate The Things Network(TTN) Applications and their attributes. I’m planning on using attributes to manage which applications (and in future EndDevices) are enabled in my Advanced Message Queuing Protocol(AMQP) client.

In the code I have left the different paging implementations which I trialled but abandoned.

using (HttpClient httpClient = new HttpClient())
{
	ApplicationRegistryClient applicationRegistryClient = new ApplicationRegistryClient(baseUrl, httpClient)
	{
		ApiKey = apiKey
	};

	try
	{
		int page = 1;
		string[] fieldMaskPathsApplication = { "attributes" };

		V3Applications applications = await applicationRegistryClient.ListAsync(collaborator, field_mask_paths: fieldMaskPathsApplication, limit:pageSize, page: page);
		while ((applications != null) && (applications.Applications != null))
		{ 
			Console.WriteLine($"Applications:{applications.Applications.Count} Page:{page} Page size:{pageSize}");
			foreach (V3Application application in applications.Applications)
			{
				Console.WriteLine($"Application ID:{application.Ids.Application_id}"); 
				if (application.Attributes != null)
				{
					Console.WriteLine("  Application attributes");

					foreach (KeyValuePair<string, string> attribute in application.Attributes)
					{
						Console.WriteLine($"   Key: {attribute.Key} Value: {attribute.Value}");
					}
				}
				Console.WriteLine();
			}
			page += 1;
			applications = await applicationRegistryClient.ListAsync(collaborator, field_mask_paths: fieldMaskPathsApplication, limit: pageSize, page: page);
		}
	}   
}

For each LoraWAN client I have to have an open connection to the Azure IoT hub to get Cloud to Device (C2D) messages so I’m looking at using connection pooling to reduce the overall number of connections.

I think the Azure ClientDevice library supports up to 995 devices per connection and has quiet a lot of additional functionality.

/// <summary>
/// contains Amqp Connection Pool settings for DeviceClient
/// </summary>
public sealed class AmqpConnectionPoolSettings
{
   private static readonly TimeSpan s_defaultConnectionIdleTimeout = TimeSpan.FromMinutes(2);
    private uint _maxPoolSize;
    internal const uint MaxDevicesPerConnection = 995; // IotHub allows upto 999 tokens per connection. Setting the threshold just below that.

    /// <summary>
    /// The default size of the pool
    /// </summary>
    /// <remarks>
    /// Allows up to 100,000 devices
    /// </remarks>
    private const uint DefaultPoolSize = 100;

    /// <summary>
    /// The maximum value that can be used for the MaxPoolSize property
    /// </summary>
     public const uint AbsoluteMaxPoolSize = ushort.MaxValue;

    /// <summary>
    /// Creates an instance of AmqpConnecitonPoolSettings with default properties
    /// </summary>
    public AmqpConnectionPoolSettings()
    {
       _maxPoolSize = DefaultPoolSize;
       Pooling = false;
    }

Whereas I think AMQPNetLite may support more, but will require me to implement more of the Azure IoT client interface

/// <summary>
/// The default maximum frame size used by the library.
/// </summary>
public const uint DefaultMaxFrameSize = 64 * 1024;
internal const ushort DefaultMaxConcurrentChannels = 8 * 1024;
internal const uint DefaultMaxLinkHandles = 256 * 1024;
internal const uint DefaultHeartBeatInterval = 90000;
internal const uint MinimumHeartBeatIntervalMs = 5 * 1000;

I have got todo some more research to see which library is easier/requires more code/complex/scales better.

TTN V3 Application API Basic Client

After reviewing the initial implementation I found I had to have one connection per The Things Network(TTN) device. Todo this I first have to enumerate the LoRaWAN Devices for each Application in my instance. First I had to add the TTN APIKey to the application and device registry requests.

namespace devMobile.TheThingsNetwork.API
{
	public partial class EndDeviceRegistryClient
	{
		public string ApiKey { set; get; }

		partial void PrepareRequest(System.Net.Http.HttpClient client, System.Net.Http.HttpRequestMessage request, string url)
		{
			if (!client.DefaultRequestHeaders.Contains("Authorization"))
			{
				client.DefaultRequestHeaders.Add("Authorization", $"Bearer {ApiKey}");
			}
		}
	}

	public partial class ApplicationRegistryClient
	{
		public string ApiKey { set; get; }

		partial void PrepareRequest(System.Net.Http.HttpClient client, System.Net.Http.HttpRequestMessage request, string url)
		{
			if (!client.DefaultRequestHeaders.Contains("Authorization"))
			{
				client.DefaultRequestHeaders.Add("Authorization", $"Bearer {ApiKey}");
			}
		}
	}
}

The first step was to enumerate Applications and their attributes

#if FIELDS_MINIMUM
	string[] fieldMaskPathsApplication = { "attributes" }; // think this is the bare minimum required for integration
#else
	string[] fieldMaskPathsApplication = { "name", "description", "attributes" };
#endif

	V3Applications applications = await applicationRegistryClient.ListAsync(collaborator, field_mask_paths: fieldMaskPathsApplication);
	if ((applications != null) && (applications.Applications != null)) // If there are no applications returns null rather than empty list
	{
		foreach (V3Application application in applications.Applications)
		{
#if FIELDS_MINIMUM
			Console.WriteLine($"Application ID:{application.Ids.Application_id}");
#else
			Console.WriteLine($"Application ID:{application.Ids.Application_id} Name:{application.Name} Description:{application.Description}");
			Console.WriteLine($"  CreatedAt: {application.Created_at:dd-MM-yy HH:mm:ss} UpdatedAt: {application.Updated_at:dd-MM-yy HH:mm:ss}");
#endif
			if (application.Attributes != null)
			{
				Console.WriteLine("  Application attributes");

				foreach (KeyValuePair<string, string> attribute in application.Attributes)
				{
					Console.WriteLine($"    Key: {attribute.Key} Value: {attribute.Value}");
				}
			}
			Console.WriteLine();
		}
	}
}

The applicationRegistryClient.ListAsync call returns null rather than an empty list which tripped me up. I only found this when I deleted all the applications in my instance and started from scratch.

The Things Network HTTP Integration Part13

Connection multiplexing

For the Proof of Concept(PoC) I had used a cache to store Azure IoT Hub connections to reduce the number of calls to the Device Provisioning Service(DPS).

Number of connections with no pooling

When stress testing with 1000’s of devices my program hit the host connection limit so I enabled Advanced Message Queuing Protocol(AMQP) connection pooling.

return DeviceClient.Create(result.AssignedHub,
                  authentication,
                  new ITransportSettings[]
                  {
                     new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
                     {
                        PrefetchCount = 0,
                        AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
                        {
                           Pooling = true,
                        }
                     }
                  }
               );

My first attempt failed as I hadn’t configured “TransportType.Amqp_Tcp_Only” which would have allowed the AMQP implementation to fallback to other protocols which don’t support pooling.

Exception caused by not using TransportType.Amqp_Tcp_Only

I then deployed the updated code and ran my 1000 device stress test (note the different x axis scales)

Number of connections with pooling

This confirmed what I found in the Azure.AMQP source code

/// <summary>
/// The default size of the pool
/// </summary>
/// <remarks>
/// Allows up to 100,000 devices
/// </remarks>
/// private const uint DefaultPoolSize = 100;

The Things Network V2 MQTT Client

Another option for I had been looking at for connecting an Azure IoT Hub and The Things Network(TTN) was a Message Queue Telemetry Transport(MQTT) integration.

To trial this approach I build a .Net Core console application which sent message to and received messages from an application running on a GHI Electronics TinyCLRV2 Fezduino with RakWireless Wisduino Evaluation Board(EVB).

The console application uses MQTTNet to connect to TTN. It subscribes to to the TTN application device uplink topic (did try subscribing to the uplink messages for all the devices in the application but this was to noisy), and the downlink message scheduled, sent and acknowledged topics. To send messages to the device I published them on the device downlink topic.

//string uplinktopic = $"{applicationId}/devices/+/up";
string uplinktopic = $"{applicationId}/devices/{deviceId}/up";
await mqttClient.SubscribeAsync(uplinktopic, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce);

string downlinkAcktopic = $"{applicationId}/devices/{deviceId}/events/down/acks";
await mqttClient.SubscribeAsync(downlinkAcktopic, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce);

string downlinkScheduledtopic = $"{applicationId}/devices/{deviceId}/events/down/scheduled";
await mqttClient.SubscribeAsync(downlinkScheduledtopic, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce);

string downlinkSenttopic = $"{applicationId}/devices/{deviceId}/events/down/sent";
await mqttClient.SubscribeAsync(downlinkSenttopic, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce);

string downlinktopic = $"{applicationId}/devices/{deviceId}/down";

I used the classes from one of my earlier blog posts to deserialise the uplink message payload so I could display a subset of the fields.

MQTTNet based .Net Core console client
Things Network Device Data view

In the TTN Device data tab I could see messages being sent, to and received from from the device.

Visual Studio 2019 Tiny CLR debugger Output

In the Visual Studio 2019 debugger output window I could see messages being sent and received by the Fezduino.

Malformed TTN downlink payload

I had some problems with the downlink messages silently failing as the TTN sample payload JSON was malformed and I had copied it without noticing.

I have a working TTN HTTP Integration (uplink messages only) but have been exploring alternatives using TTN MQTT and Azure IoT Hub AMQP clients.

The next step is to build an Azure IoT Hub client (using native AMQP) then join them together.

The Things Network HTTP Integration Part12

Removing the DIY cache

For the Proof of Concept(PoC) I had written a simple cache using a ConcurrentDictionary to store Azure IoT Hub connections to reduce the number of calls to the Device Provisioning Service(DPS).

Device Provisioning Service calls in stress test

For a PoC the DIY cache was ok but I wanted to replace it with something more robust like the .Net ObjectCache which is in the System.Runtime.Caching namespace.

I started by replacing the ConcurrentDictionary declaration

static readonly ConcurrentDictionary<string, DeviceClient> DeviceClients = new ConcurrentDictionary<string, DeviceClient>();
     

With an ObjectCache declaration.

static readonly ObjectCache DeviceClients = MemoryCache.Default;
  

Then, where there were compiler errors I updated the method call.

// See if the device has already been provisioned or is being provisioned on another thread.
if (DeviceClients.Add(registrationId, deviceContext, cacheItemPolicy))
{
   log.LogInformation("RegID:{registrationId} Device provisioning start", registrationId);
...

One difference I found was that ObjectCache throws an exception if the value is null. I was using a null value to indicate that the Device Provisioning Service(DPS) process had been initiated on another thread and was underway.

I have been planning to add support for downlink messages so I added a new class to store the uplink (Azure IoT Hub DeviceClient) and downlink ( downlink_url in the uplink message) details.

 public class DeviceContext
   {
      public DeviceClient Uplink { get; set; }
      public Uri Downlink { get; set; }
   }

For the first version the only functionality I’m using is sliding expiration which is set to one day

CacheItemPolicy cacheItemPolicy = new CacheItemPolicy()
{
   SlidingExpiration = new TimeSpan(1, 0, 0, 0),
   //RemovedCallback
};

DeviceContext deviceContext = new DeviceContext()
{
   Uplink = null,
   Downlink = new Uri(payload.DownlinkUrl)
};

I didn’t have to make many changes and I’ll double check my implementation in the next round of stress and soak testing.

The Things Network HTTP Integration Part11

Moving Secrets to KeyVault

The application configuration file contained sensitive information like Device Provision Service(DPS) Group Enrollment Symmetric Keys and Azure IoT Hub connection strings which is OK for a proof of concept (PoC) but sub-optimal for production deployments.

"DeviceProvisioningService": {
      "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
      "ScopeID": "",
      "EnrollmentGroupSymmetricKeyDefault": "TopSecretKey",
      "DeviceProvisioningPollingDelay": 500,
      "ApplicationEnrollmentGroupMapping": {
         "Application1": "TopSecretKey1",
         "Application2": "TopSecretKey2"
      }
   }

The Azure Key Vault is intended for securing sensitive information like connection strings so I added one to my resource group.

Azure Key Vault overview and basic metrics

I wrote a wrapper which resolves configuration settings based on the The Things Network(TTN) application identifier and port information in the uplink message payload. The resolve methods start by looking for configuration for the applicationId and port (separated by a – ), then the applicationId and then finally falling back to a default value. This functionality is used for AzureIoTHub connection strings, DPS IDScopes, DPS Enrollment Group Symmetric Keys, and is also used to format the cache keys.

public class ApplicationConfiguration
{
const string DpsGlobaDeviceEndpointDefault = "global.azure-devices-provisioning.net";

private IConfiguration Configuration;

public void Initialise( )
{
   // Check that KeyVault URI is configured in environment variables. Not a lot we can do if it isn't....
   if (Configuration == null)
   {
      string keyVaultUri = Environment.GetEnvironmentVariable("KeyVaultURI");
      if (string.IsNullOrWhiteSpace(keyVaultUri))
      {
         throw new ApplicationException("KeyVaultURI environment variable not set");
      }

      // Load configuration from KeyVault 
      Configuration = new ConfigurationBuilder()
         .AddEnvironmentVariables()
         .AddAzureKeyVault(keyVaultUri)
         .Build();
   }
}

public string DpsGlobaDeviceEndpointResolve()
{
   string globaDeviceEndpoint = Configuration.GetSection("DPSGlobaDeviceEndpoint").Value;
   if (string.IsNullOrWhiteSpace(globaDeviceEndpoint))
   {
      globaDeviceEndpoint = DpsGlobaDeviceEndpointDefault;
   }

   return globaDeviceEndpoint;
}

public string ConnectionStringResolve(string applicationId, int port)
{
   // Check to see if there is application + port specific configuration
   string connectionString = Configuration.GetSection($"AzureIotHubConnectionString-{applicationId}-{port}").Value;
   if (!string.IsNullOrWhiteSpace(connectionString))
   {
      return connectionString;
   }

   // Check to see if there is application specific configuration, otherwise run with default
   connectionString = Configuration.GetSection($"AzureIotHubConnectionString-{applicationId}").Value;
   if (!string.IsNullOrWhiteSpace(connectionString))
   {
      return connectionString;
   }

   // get the default as not a specialised configuration
   connectionString = Configuration.GetSection("AzureIotHubConnectionStringDefault").Value;

   return connectionString;
}

public string DpsIdScopeResolve(string applicationId, int port)
{
   // Check to see if there is application + port specific configuration
   string idScope = Configuration.GetSection($"DPSIDScope-{applicationId}-{port}").Value;
   if (!string.IsNullOrWhiteSpace(idScope))
   {
      return idScope;
   }

   // Check to see if there is application specific configuration, otherwise run with default
   idScope = Configuration.GetSection($"DPSIDScope-{applicationId}").Value;
   if (!string.IsNullOrWhiteSpace(idScope))
   {
      return idScope;
   }

   // get the default as not a specialised configuration
   idScope = Configuration.GetSection("DPSIDScopeDefault").Value;

   if (string.IsNullOrWhiteSpace(idScope))
   {
      throw new ApplicationException($"DPSIDScope configuration invalid");
   }

   return idScope;
}

The values of Azure function configuration settings are replaced by a reference to the secret in the Azure Key Vault.

Azure Function configuration value replacement

In the Azure Key Vault “Access Policies” I configured an “Application Access Policy” so my Azure TTNAzureIoTHubMessageV2Processor function identity could retrieve secrets.

Azure Key Vault Secrets

I kept on making typos in the secret names and types which was frustrating.

Azure Key Vault secret

While debugging in Visual Studio you may need to configure the Azure Identity so the application can access the Azure Key Vault.

The Things Network HTTP Azure IoT Integration Soak Testing

I wanted to do some testing to make sure the application would reliably process messages from 1000’s of devices…

The first thing I learnt was “don’t forget to restart your Azure Function after deleting all the devices from the Azure IoT Hub” as the DeviceClients are cached. Also make sure you delete the devices from both your Azure Device Provisioning service(DPS) and Azure IoT Hub instances.

Applications Insights provisioning event tracking

The next “learning” was that if you forget to enable “always on” the caching won’t work and your application will call the DPS way more often than expected.

Azure Application “always on configuration

The next “learning” was if your soak test sends 24000 messages it will start to fail just after you go out to get a coffee because of the 8000 msgs/day limit on the free version of IoT Hub.

Azure IoT Hub Free tier 8000 messages/day limit

After these “learnings” the application appeared to be working and every so often a message would briefly appear in Azure Storage Explorer queue view.

Azure storage explorer view of uplink messages queue

The console test application simulated 1000 devices sending 24 messages every so often and took roughly 8 hours to complete.

Message generator finished

In the Azure IoT Hub telemetry 24000 messages had been received after roughly 8 hours confirming the test rig was working as expected.

The notch was another “learning”, if you go and do some gardening then after roughly 40 minutes of inactivity your desktop PC will go into power save mode and the test client will stop sending messages.

The caching of settings appeared to be work as there were only a couple of requests to my Azure Key Vault where sensitive information like connection strings, symmetric keys etc. are stored.

Memory consumption did look to bad and topped out at roughly 120M.

In the application logging you can see the 1000 calls to DPS at the beginning (the yellow dependency events) then the regular processing of messages.

Application Insights logging

Even with the “learnings” the testing went pretty well overall. I do need to run the test rig for longer and with even more simulated devices.

I think this should do

48K Telemetry messages

If you get lots of errors in the logs “Host thresholds exceeded: [Connections]…. might need to bump your plan to something a bit larger

The Things Network HTTP Azure IoT Central Integration

This post is an overview of the Azure IoT Central configuration required to process The Things Network(TTN) HTTP integration uplink messages. I have assumed that the reader is already reasonably familiar with these products. There is an overview of configuring TTN HTTP integration in my “Simplicating and securing the HTTP handler” post.

The first step is to copy the IDScope from the Device connection blade.

Device connection blade

Then copy one of the primary or secondary keys

For more complex deployment the ApplicationEnrollmentGroupMapping configuration enables The Things Network(TTN) devices to be provisioned using different GroupEnrollment keys based on the applicationid in the Uplink message which initiates their provisoning.

"DeviceProvisioningService": {
      "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
      "IDScope": "",
      "EnrollmentGroupSymmetricKeyDefault": "TopSecretKey",
      "DeviceProvisioningPollingDelay": 500,
      "ApplicationEnrollmentGroupMapping": {
         "Application1": "TopSecretKey1",
         "Application2": "TopSecretKey2"
      }
   }

Shortly after the first uplink message from a TTN device is processed, it will listed in the “Unassociated devices” blade with the DevEUI as the Device ID.

Unassociated devices blade

The device can then be associated with an Azure IoT Central Device Template.

Unassociated devices blade showing recently associated device

The device template provides for the mapping of uplink message payload_fields to measurements. In this example the payload field has been generated by the TTN HTTP integration Cayenne Low Power Protocol(LPP) decoder. Many LoRaWAN devices use LPP to minimise the size of the network payload.

Azure IoT Central Device template blade

Once the device has been associated with a template a user friendly device name etc. can be configured.

Azure IoT Central Device properties blade

In the telemetry event payload sent to Azure IoT Central there are some extra fields to help with debugging and tracing.

// Assemble the JSON payload to send to Azure IoT Hub/Central.
log.LogInformation($"{messagePrefix} Payload assembly start");
JObject telemetryEvent = new JObject();
try
{
   JObject payloadFields = (JObject)payloadObect.payload_fields;
   telemetryEvent.Add("HardwareSerial", payloadObect.hardware_serial);
   telemetryEvent.Add("Retry", payloadObect.is_retry);
   telemetryEvent.Add("Counter", payloadObect.counter);
   telemetryEvent.Add("DeviceID", payloadObect.dev_id);
   telemetryEvent.Add("ApplicationID", payloadObect.app_id);
   telemetryEvent.Add("Port", payloadObect.port);
   telemetryEvent.Add("PayloadRaw", payloadObect.payload_raw);
   telemetryEvent.Add("ReceivedAtUTC", payloadObect.metadata.time);

   // If the payload has been unpacked in TTN backend add fields to telemetry event payload
   if (payloadFields != null)
   {
      foreach (JProperty child in payloadFields.Children())
      {
         EnumerateChildren(telemetryEvent, child);
      }
   }
}
catch (Exception ex)
{
   log.LogError(ex, $"{messagePrefix} Payload processing or Telemetry event assembly failed");
   throw;
}

Azure IoT Central has mapping functionality which can be used to display the location of a device.

Azure Device

The format of the location payload generated by the TTN LPP decoder is different to the one required by Azure IoT Central. I have added temporary code (“a cost effective modification to expedite deployment” aka. a hack) to format the TelemetryEvent payload so it can be processed.

if (token.First is JValue)
{
   // Temporary dirty hack for Azure IoT Central compatibility
   if (token.Parent is JObject possibleGpsProperty)
   {
      if (possibleGpsProperty.Path.StartsWith("GPS", StringComparison.OrdinalIgnoreCase))
      {
         if (string.Compare(property.Name, "Latitude", true) == 0)
         {
            jobject.Add("lat", property.Value);
         }
         if (string.Compare(property.Name, "Longitude", true) == 0)
         {
            jobject.Add("lon", property.Value);
         }
         if (string.Compare(property.Name, "Altitude", true) == 0)
         {
            jobject.Add("alt", property.Value);
         }
      }
   }
   jobject.Add(property.Name, property.Value);
}

I need review the IoT Plug and Play specification documentation to see what other payload transformations maybe required.

I did observe that if a device had not reported its position the default location was zero degrees latitude and zero degrees longitude which is about 610 KM south of Ghana and 1080 KM west of Gabon in the Atlantic Ocean.

Azure IoT Central mapping default position

After configuring a device template, associating my devices with the template, and modifying each device’s properties I could create a dashboard to view the temperature and humidity information returned by my Seeeduino LoRaWAN devices.

Azure IoT Central dashboard