TTI V3 Connector Azure IoT Central Cloud to Device(C2D)

Handling Cloud to Device(D2C) Azure IoT Central messages (The Things Industries(TTI) downlink) is a bit more complex than Device To Cloud(D2C) messaging. The format of the command messages is reasonably well documented and I have already explored in detail with basic telemetry, basic commands, request commands, and The Things Industries Friendly commands and Digital Twin Definition Language(DTDL) support.

public class IoTHubApplicationSetting
{
	public string DtdlModelId { get; set; }
}

public class IoTHubSettings
{
	public string IoTHubConnectionString { get; set; } = string.Empty;

	public Dictionary<string, IoTHubApplicationSetting> Applications { get; set; }
}


public class DeviceProvisiongServiceApplicationSetting
{
	public string DtdlModelId { get; set; } = string.Empty;

	public string GroupEnrollmentKey { get; set; } = string.Empty;
}

public class DeviceProvisiongServiceSettings
{
	public string IdScope { get; set; } = string.Empty;

	public Dictionary<string, DeviceProvisiongServiceApplicationSetting> Applications { get; set; }
}


public class IoTCentralMethodSetting
{
	public byte Port { get; set; } = 0;

	public bool Confirmed { get; set; } = false;

	public Models.DownlinkPriority Priority { get; set; } = Models.DownlinkPriority.Normal;

	public Models.DownlinkQueue Queue { get; set; } = Models.DownlinkQueue.Replace;
}

public class IoTCentralSetting
{
	public Dictionary<string, IoTCentralMethodSetting> Methods { get; set; }
}

public class AzureIoTSettings
{
	public IoTHubSettings IoTHub { get; set; }

	public DeviceProvisiongServiceSettings DeviceProvisioningService { get; set; }

	public IoTCentralSetting IoTCentral { get; set; }
}

Azure IoT Central appears to have no support for setting message properties so the LoRaWAN port, confirmed flag, priority, and queuing so these a retrieved from configuration.

Azure Function Configuration
Models.Downlink downlink;
Models.DownlinkQueue queue;

string payloadText = Encoding.UTF8.GetString(message.GetBytes()).Trim();

if (message.Properties.ContainsKey("method-name"))
{
	#region Azure IoT Central C2D message processing
	string methodName = message.Properties["method-name"];

	if (string.IsNullOrWhiteSpace(methodName))
	{
		_logger.LogWarning("Downlink-DeviceID:{0} MessagedID:{1} LockToken:{2} method-name property empty", receiveMessageHandlerContext.DeviceId, message.MessageId, message.LockToken);

		await deviceClient.RejectAsync(message);
		return;
	}

	// Look up the method settings to get confirmed, port, priority, and queue
	if ((_azureIoTSettings == null) || (_azureIoTSettings.IoTCentral == null) || !_azureIoTSettings.IoTCentral.Methods.TryGetValue(methodName, out IoTCentralMethodSetting methodSetting))
	{
		_logger.LogWarning("Downlink-DeviceID:{0} MessagedID:{1} LockToken:{2} method-name:{3} has no settings", receiveMessageHandlerContext.DeviceId, message.MessageId, message.LockToken, methodName);
							
		await deviceClient.RejectAsync(message);
		return;
	}

	downlink = new Models.Downlink()
	{
		Confirmed = methodSetting.Confirmed,
		Priority = methodSetting.Priority,
		Port = methodSetting.Port,
		CorrelationIds = AzureLockToken.Add(message.LockToken),
	};

	queue = methodSetting.Queue;

	// Check to see if special case for Azure IoT central command with no request payload
	if (payloadText.IsPayloadEmpty())
	{
		downlink.PayloadRaw = "";
	}

	if (!payloadText.IsPayloadEmpty())
	{
		if (payloadText.IsPayloadValidJson())
		{
			downlink.PayloadDecoded = JToken.Parse(payloadText);
			}
		else
		{
			downlink.PayloadDecoded = new JObject(new JProperty(methodName, payloadText));
		}
	}

	logger.LogInformation("Downlink-IoT Central DeviceID:{0} Method:{1} MessageID:{2} LockToken:{3} Port:{4} Confirmed:{5} Priority:{6} Queue:{7}",
		receiveMessageHandlerContext.DeviceId,
		methodName,
		message.MessageId,
		message.LockToken,
		downlink.Port,
		downlink.Confirmed,
		downlink.Priority,
		queue);
	#endregion
}

The reboot command payload only contains an “@” so the TTTI payload will be empty, the minimum and maximum command payloads will contain only a numeric value which is added to the decoded payload with the method name, the combined minimum and maximum command has a JSON payload which is “grafted” into the decoded payload.

Azure IoT Central Device Template

Azure Device Provisioning Service(DPS) when transient isn’t

After some updates to my Device Provisioning Service(DPS) code the RegisterAsync method was exploding with an odd exception.

TTI Webhook Integration running in desktop emulator

In the Visual Studio 2019 Debugger the exception text was “IsTransient = true” so I went and made a coffee and tried again.

Visual Studio 2019 Quickwatch displaying short from error message

The call was still failing so I dumped out the exception text so I had some key words to search for

Microsoft.Azure.Devices.Provisioning.Client.ProvisioningTransportException: AMQP transport exception
 ---> System.UnauthorizedAccessException: Sys
   at Microsoft.Azure.Amqp.ExceptionDispatcher.Throw(Exception exception)
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.HandleTransportOpened(IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.OnTransportOpenCompete(IAsyncResult result)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.AmqpClientConnection.OpenAsync(TimeSpan timeout, Boolean useWebSocket, X509Certificate2 clientCert, IWebProxy proxy, RemoteCertificateValidationCallback remoteCerificateValidationCallback)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, TimeSpan timeout, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, CancellationToken cancellationToken)
   at devMobile.IoT.TheThingsIndustries.AzureIoTHub.Integration.Uplink(HttpRequestData req, FunctionContext executionContext) in C:\Users\BrynLewis\source\repos\TTIV3AzureIoTConnector\TTIV3WebHookAzureIoTHubIntegration\TTIUplinkHandler.cs:line 245

I tried a lot of keywords and went and looked at the source code on github

One of the many keyword searches

Another of the many keyword searches

I then tried another program which did used the Device provisioning Service and it worked first time so it was something wrong with the code.

using (var securityProvider = new SecurityProviderSymmetricKey(deviceId, deviceKey, null))
{
	using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
	{
		DeviceRegistrationResult result;

		ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
			Constants.AzureDpsGlobalDeviceEndpoint,
			 dpsApplicationSetting.GroupEnrollmentKey, <<= Should be _azureIoTSettings.DeviceProvisioningService.IdScope,
			securityProvider,
			transport);

		try
		{
				result = await provClient.RegisterAsync();
		}
		catch (ProvisioningTransportException ex)
		{
			logger.LogInformation(ex, "Uplink-DeviceID:{0} RegisterAsync failed IDScope and/or GroupEnrollmentKey invalid", deviceId);

			return req.CreateResponse(HttpStatusCode.Unauthorized);
		}

		if (result.Status != ProvisioningRegistrationStatusType.Assigned)
		{
			_logger.LogError("Uplink-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);

			return req.CreateResponse(HttpStatusCode.FailedDependency);
		}

		IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

		deviceClient = DeviceClient.Create(result.AssignedHub, authentication, TransportSettings);

		await deviceClient.OpenAsync();

		logger.LogInformation("Uplink-DeviceID:{0} Azure IoT Hub connected (Device Provisioning Service)", deviceId);
	}
}

I then carefully inspected my source code and worked back through the file history and realised I had accidentally replaced the IDScope with the GroupEnrollment setting so it was never going to work i.e. IsTransient != true. So, for the one or two other people who get this error message check your IDScope and GroupEnrollment key make sure they are the right variables and that values they contain are correct.

TTI V3 Connector Azure IoT Central Device to Cloud(D2C)

This post is largely about adapting the output of The Things Industries(TTI) MyDevices Cayenne Low Power Protocol(LPP) payload formatter so that it can be injested by Azure IoT Central. The Azure function for processing TTI Uplink messages first deserialises the JSON payload discarding any LoRaWAN control messages and messages with empty payloads.

[Function("Uplink")]
public async Task<HttpResponseData> Uplink([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
{
	Models.PayloadUplink payload;
	var logger = executionContext.GetLogger("Queued");

	// Wrap all the processing in a try\catch so if anything blows up we have logged it.
	try
	{
		string payloadText = await req.ReadAsStringAsync();

		try
		{
			payload = JsonConvert.DeserializeObject<Models.PayloadUplink>(payloadText);
		}
		catch(JsonException ex)
		{
			logger.LogInformation(ex, "Uplink-Payload Invalid JSON:{0}", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		if (payload == null)
		{
			logger.LogInformation("Uplink-Payload invalid:{0}", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
		string deviceId = payload.EndDeviceIds.DeviceId;

		if ((payload.UplinkMessage.Port == null) || (!payload.UplinkMessage.Port.HasValue) || (payload.UplinkMessage.Port.Value == 0))
		{
			logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Payload Raw:{2} Control message", applicationId, deviceId, payload.UplinkMessage.PayloadRaw);

			return req.CreateResponse(HttpStatusCode.UnprocessableEntity);
		}

		int port = payload.UplinkMessage.Port.Value;

		logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Port:{2} Payload Raw:{3}", applicationId, deviceId, port, payload.UplinkMessage.PayloadRaw);

		if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
		{
...		
		}

		JObject telemetryEvent = new JObject
		{
			{ "ApplicationID", applicationId },
			{ "DeviceID", deviceId },
			{ "Port", port },
			{ "Simulated", payload.Simulated },
			{ "ReceivedAtUtc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture) },
			{ "PayloadRaw", payload.UplinkMessage.PayloadRaw }
		};

		// If the payload has been decoded by payload formatter, put it in the message body.
		if (payload.UplinkMessage.PayloadDecoded != null)
		{
			EnumerateChildren(telemetryEvent, payload.UplinkMessage.PayloadDecoded);
		}

		// Send the message to Azure IoT Hub
		using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
		{
			// Ensure the displayed time is the acquired time rather than the uploaded time. 
			ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture));
			ioTHubmessage.Properties.Add("ApplicationId", applicationId);
			ioTHubmessage.Properties.Add("DeviceEUI", payload.EndDeviceIds.DeviceEui);
			ioTHubmessage.Properties.Add("DeviceId", deviceId);
			ioTHubmessage.Properties.Add("port", port.ToString());
			ioTHubmessage.Properties.Add("Simulated", payload.Simulated.ToString());

			await deviceClient.SendEventAsync(ioTHubmessage);

			logger.LogInformation("Uplink-DeviceID:{0} SendEventAsync success", payload.EndDeviceIds.DeviceId);
		}
	}
	catch (Exception ex)
	{
		logger.LogError(ex, "Uplink-Message processing failed");

		return req.CreateResponse(HttpStatusCode.InternalServerError);
	}

	return req.CreateResponse(HttpStatusCode.OK);
}

If the message has been successfully decoded by a payload formatter the PayloadDecoded contents will be “grafted” into the Azure IoT Central Telemetry message.

TTI JSON GPS position format

The Azure IoT Central Location Telemetry messages have a slightly different format to the output of the TTI LPP Payload formatter so the payload has to be “post processed”.

private void EnumerateChildren(JObject jobject, JToken token)
{
	if (token is JProperty property)
	{
		if (token.First is JValue)
		{
			// Temporary dirty hack for Azure IoT Central compatibility
			if (token.Parent is JObject possibleGpsProperty)
			{
				// TODO Need to check if similar approach necessary accelerometer and gyro LPP payloads
				if (possibleGpsProperty.Path.StartsWith("GPS_", StringComparison.OrdinalIgnoreCase))
				{
					if (string.Compare(property.Name, "Latitude", true) == 0)
					{
						jobject.Add("lat", property.Value);
					}
					if (string.Compare(property.Name, "Longitude", true) == 0)
					{
						jobject.Add("lon", property.Value);
					}
					if (string.Compare(property.Name, "Altitude", true) == 0)
					{
						jobject.Add("alt", property.Value);
					}
				}
			}
			jobject.Add(property.Name, property.Value);
		}
		else
		{
			JObject parentObject = new JObject();
			foreach (JToken token2 in token.Children())
			{
				EnumerateChildren(parentObject, token2);
				jobject.Add(property.Name, parentObject);
			}
		}
	}
	else
	{
		foreach (JToken token2 in token.Children())
		{
			EnumerateChildren(jobject, token2);
		}
	}
}

I may have to extend this method for other LPP datatypes

“Post processed” TTI JSON GPS Position data suitable for Azure IoT Central

To test the telemetry message JSON I created an Azure IoT Central Device Template which had a “capability type” of Location.

Azure IoT Central Device Template with Location Capability

For initial development and testing I ran the function application in the desktop emulator and simulated TTI webhook calls with Telerik Fiddler and modified sample payloads. After some issues with iothub-creation-time-utc decoded telemetry messages were displayed in the Device Raw Data tab

Azure IoT Central Device Raw Data tab with successfully decoded GPS location payloads
Azure IoT Central map displaying with device location highlighted

This post uses a lot of the work done for my The Things Network V2 integration. I also found the first time a device connected to the Azure IoT Central Azure IoT hub (using the Azure IoT Central Device Provisioning Service(DPS) to get the connection string) there was always an exception.

Microsoft.Azure.Devices.Client.Exceptions.IotHubException: error(condition:com.microsoft:connection-closed-on-new-connection,description:Backend initiated disconnection.

TTI V3 Gateway Azure IoT Central first call exception

This exception occurs when the SetMethodDefaultHandlerAsync method is called which is a bit odd. This exception does not occur when I use Device Provisioning Service(DPS) and Azure IoT Hub instances I have provisioned.

TTI V3 Connector Cloud to Device(C2D)

The TTI V3 Connector Minimalist Cloud to Device only required a port number, and there was no way to specify whether delivery of message had to be confirmed, the way the message was queued, or the priority of message delivery. Like the port number these optional settings can be specified in message properties.

  • Confirmation – True/False
  • Queue – Push/Replace
  • Priority – Lowest/Low/BelowNormal/Normal/AboveNormal/High/Highest

If any of these properties are incorrect DeviceClient.RejectAsync is called which deletes the message from the device queue and indicates to the server that the message could not be processed.

private async Task AzureIoTHubClientReceiveMessageHandler(Message message, object userContext)
{
	try
	{
		Models.AzureIoTHubReceiveMessageHandlerContext receiveMessageHandlerContext = (Models.AzureIoTHubReceiveMessageHandlerContext)userContext;

		if (!_DeviceClients.TryGetValue(receiveMessageHandlerContext.DeviceId, out DeviceClient deviceClient))
		{
			_logger.LogWarning("Downlink-DeviceID:{0} unknown", receiveMessageHandlerContext.DeviceId);
			return;
		}

		using (message)
		{
			string payloadText = Encoding.UTF8.GetString(message.GetBytes()).Trim();

			if (!AzureDownlinkMessage.PortTryGet(message.Properties, out byte port))
			{
				_logger.LogWarning("Downlink-Port property is invalid");

				await deviceClient.RejectAsync(message);
				return;
			}

			if (!AzureDownlinkMessage.ConfirmedTryGet(message.Properties, out bool confirmed))
			{
				_logger.LogWarning("Downlink-Confirmed flag is invalid");

				await deviceClient.RejectAsync(message);
				return;
			}

			if (!AzureDownlinkMessage.PriorityTryGet(message.Properties, out Models.DownlinkPriority priority))
			{
				_logger.LogWarning("Downlink-Priority value is invalid");

				await deviceClient.RejectAsync(message);
				return;
			}

			if (!AzureDownlinkMessage.QueueTryGet(message.Properties, out Models.DownlinkQueue queue))
			{
				_logger.LogWarning("Downlink-Queue value is invalid");

				await deviceClient.RejectAsync(message.LockToken);
				return;
			}

			Models.Downlink downlink = new Models.Downlink()
			{
				Confirmed = confirmed,
				Priority = priority,
				Port = port,
				CorrelationIds = AzureLockToken.Add(message.LockToken),
			};

			// Split over multiple lines in an attempt to improve readability. In this scenario a valid JSON string should start/end with {/} for an object or [/] for an array
			if ((payloadText.StartsWith("{") && payloadText.EndsWith("}"))
													||
				((payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
			{
				try
				{
					downlink.PayloadDecoded = JToken.Parse(payloadText);
				}
				catch (JsonReaderException)
				{
					downlink.PayloadRaw = payloadText;
				}
			}
			else
			{
				downlink.PayloadRaw = payloadText;
			}

			_logger.LogInformation("Downlink-IoT Hub DeviceID:{0} MessageID:{2} LockToken:{3} Port:{4} Confirmed:{5} Priority:{6} Queue:{7}",
				receiveMessageHandlerContext.DeviceId,
				message.MessageId,
				message.LockToken,
				downlink.Port,
				downlink.Confirmed,
				downlink.Priority,
				queue);

			Models.DownlinkPayload Payload = new Models.DownlinkPayload()
			{
				Downlinks = new List<Models.Downlink>()
				{
					downlink
				}
			};

			string url = $"{receiveMessageHandlerContext.WebhookBaseURL}/{receiveMessageHandlerContext.ApplicationId}/webhooks/{receiveMessageHandlerContext.WebhookId}/devices/{receiveMessageHandlerContext.DeviceId}/down/{queue}".ToLower();

			using (var client = new WebClient())
			{
				client.Headers.Add("Authorization", $"Bearer {receiveMessageHandlerContext.ApiKey}");

				client.UploadString(new Uri(url), JsonConvert.SerializeObject(Payload));
			}

			_logger.LogInformation("Downlink-DeviceID:{0} LockToken:{1} success", receiveMessageHandlerContext.DeviceId, message.LockToken);
		}
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "Downlink-ReceiveMessge processing failed");
	}
}

A correlation identifier containing the Message LockToken is added to the downlink payload.

Azure IoT Explorer Cloud to Device sending an unconfirmed downlink message

For unconfirmed messages The TTI Connector calls the DeviceClient.CompletedAsync method (with the LockToken from the CorrelationIDs list) which deletes the message from the device queue.

[Function("Queued")]
public async Task<HttpResponseData> Queued([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
{
	var logger = executionContext.GetLogger("Queued");

	// Wrap all the processing in a try\catch so if anything blows up we have logged it.
	try
	{
		string payloadText = await req.ReadAsStringAsync();

		Models.DownlinkQueuedPayload payload = JsonConvert.DeserializeObject<Models.DownlinkQueuedPayload>(payloadText);
		if (payload == null)
		{
			logger.LogInformation("Queued-Payload {0} invalid", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
		string deviceId = payload.EndDeviceIds.DeviceId;

		logger.LogInformation("Queued-ApplicationID:{0} DeviceID:{1} ", applicationId, deviceId);

		if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
		{
			logger.LogInformation("Queued-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		// If the message is not confirmed "complete" it as soon as with network
		if (!payload.DownlinkQueued.Confirmed)
		{
			if (!AzureLockToken.TryGet(payload.DownlinkQueued.CorrelationIds, out string lockToken))
			{
				logger.LogWarning("Queued-DeviceID:{0} LockToken missing from payload:{1}", payload.EndDeviceIds.DeviceId, payloadText);

				return req.CreateResponse(HttpStatusCode.BadRequest);
			}

			try
			{
				await deviceClient.CompleteAsync(lockToken);
			}
			catch (DeviceMessageLockLostException)
			{
				logger.LogWarning("Queued-CompleteAsync DeviceID:{0} LockToken:{1} timeout", payload.EndDeviceIds.DeviceId, lockToken);

				return req.CreateResponse(HttpStatusCode.Conflict);
			}

			logger.LogInformation("Queued-DeviceID:{0} LockToken:{1} success", payload.EndDeviceIds.DeviceId, lockToken);
		}
	}
	catch (Exception ex)
	{
		logger.LogError(ex, "Queued message processing failed");

		return req.CreateResponse(HttpStatusCode.InternalServerError);
	}

	return req.CreateResponse(HttpStatusCode.OK);
}

The Things Industries Live Data tab for an unconfirmed message-Queued
Azure Application Insights for an unconfirmed message
The Things Industries Live Data tab for an unconfirmed message-Sent
Azure IoT Explorer Cloud to Device sending a confirmed downlink message
Azure Application Insights for a confirmed message
The Things Industries Live Data tab for a confirmed message-Sent
The Things Industries Live Data tab for a confirmed message-Ack

If message delivery succeeds the deviceClient.CompleteAsync method (with the LockToken from the CorrelationIDs list) is called which removes the message from the device queue.

[Function("Ack")]
public async Task<HttpResponseData> Ack([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
{
	var logger = executionContext.GetLogger("Queued");

	// Wrap all the processing in a try\catch so if anything blows up we have logged it.
	try
	{
		string payloadText = await req.ReadAsStringAsync();

		Models.DownlinkAckPayload payload = JsonConvert.DeserializeObject<Models.DownlinkAckPayload>(payloadText);
		if (payload == null)
		{
			logger.LogInformation("Ack-Payload {0} invalid", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
		string deviceId = payload.EndDeviceIds.DeviceId;

		logger.LogInformation("Ack-ApplicationID:{0} DeviceID:{1} ", applicationId, deviceId);

		if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
		{
			logger.LogInformation("Ack-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		if (!AzureLockToken.TryGet(payload.DownlinkAck.CorrelationIds, out string lockToken))
		{
			logger.LogWarning("Ack-DeviceID:{0} LockToken missing from payload:{1}", payload.EndDeviceIds.DeviceId, payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		try
		{
			await deviceClient.CompleteAsync(lockToken);
		}
		catch (DeviceMessageLockLostException)
		{
			logger.LogWarning("Ack-CompleteAsync DeviceID:{0} LockToken:{1} timeout", payload.EndDeviceIds.DeviceId, lockToken);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		logger.LogInformation("Ack-DeviceID:{0} LockToken:{1} success", payload.EndDeviceIds.DeviceId, lockToken);
	}
	catch (Exception ex)
	{
		logger.LogError(ex, "Ack message processing failed");

		return req.CreateResponse(HttpStatusCode.InternalServerError);
	}

	return req.CreateResponse(HttpStatusCode.OK);
}

Azure Application Insights for an confirmed message Ack

If message delivery fails the deviceClient.AbandonAsync method (with the LockToken from the CorrelationIDs list) is called which puts the downlink message back onto the device queue.

[Function("Failed")]
public async Task<HttpResponseData> Failed([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
{
	var logger = executionContext.GetLogger("Queued");

	// Wrap all the processing in a try\catch so if anything blows up we have logged it.
	try
	{
		string payloadText = await req.ReadAsStringAsync();

		Models.DownlinkFailedPayload payload = JsonConvert.DeserializeObject<Models.DownlinkFailedPayload>(payloadText);
		if (payload == null)
		{
			logger.LogInformation("Failed-Payload {0} invalid", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
		string deviceId = payload.EndDeviceIds.DeviceId;

		logger.LogInformation("Failed-ApplicationID:{0} DeviceID:{1} ", applicationId, deviceId);

		if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
		{
			logger.LogInformation("Failed-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		if (!AzureLockToken.TryGet(payload.DownlinkFailed.CorrelationIds, out string lockToken))
		{
			logger.LogWarning("Failed-DeviceID:{0} LockToken missing from payload:{1}", payload.EndDeviceIds.DeviceId, payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		try
		{
			await deviceClient.RejectAsync(lockToken);
		}
		catch (DeviceMessageLockLostException)
		{
			logger.LogWarning("Failed-RejectAsync DeviceID:{0} LockToken:{1} timeout", payload.EndDeviceIds.DeviceId, lockToken);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		logger.LogInformation("Failed-DeviceID:{0} LockToken:{1} success", payload.EndDeviceIds.DeviceId, lockToken);
	}
	catch (Exception ex)
	{
		logger.LogError(ex, "Failed message processing failed");

		return req.CreateResponse(HttpStatusCode.InternalServerError);
	}

	return req.CreateResponse(HttpStatusCode.OK);
}

If message delivery is unsuccessful the deviceClient.RejectAsync method (with the LockToken from the CorrelationIDs list) is called which deletes the message from the device queue and indicates to the server that the message could not be processed.

[Function("Nack")]
public async Task<HttpResponseData> Nack([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
{
	var logger = executionContext.GetLogger("Queued");

	// Wrap all the processing in a try\catch so if anything blows up we have logged it.
	try
	{
		string payloadText = await req.ReadAsStringAsync();

		Models.DownlinkNackPayload payload = JsonConvert.DeserializeObject<Models.DownlinkNackPayload>(payloadText);
		if (payload == null)
		{
			logger.LogInformation("Nack-Payload {0} invalid", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
		string deviceId = payload.EndDeviceIds.DeviceId;

		logger.LogInformation("Nack-ApplicationID:{0} DeviceID:{1} ", applicationId, deviceId);

		if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
		{
			logger.LogInformation("Nack-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		if (!AzureLockToken.TryGet(payload.DownlinkNack.CorrelationIds, out string lockToken))
		{
			logger.LogWarning("Nack-DeviceID:{0} LockToken missing from payload:{1}", payload.EndDeviceIds.DeviceId, payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		try
		{
			await deviceClient.RejectAsync(lockToken);
		}
		catch (DeviceMessageLockLostException)
		{
			logger.LogWarning("Nack-RejectAsync DeviceID:{0} LockToken:{1} timeout", payload.EndDeviceIds.DeviceId, lockToken);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		logger.LogInformation("Nack-DeviceID:{0} LockToken:{1} success", payload.EndDeviceIds.DeviceId, lockToken);
	}
	catch (Exception ex)
	{
		logger.LogError(ex, "Nack message processing failed");

		return req.CreateResponse(HttpStatusCode.InternalServerError);
	}

	return req.CreateResponse(HttpStatusCode.OK);
}

The way message Failed(Abandon), Ack(CompleteAsync) and Nack(RejectAsync) are handled needs some more testing to confirm my understanding of the sequencing of TTI confirmed message delivery.

BEWARE

The use of Confirmed messaging with devices that send uplink messages irregularly can cause weird problems if the Azure IoT hub downlink message times out.

TTI V3 Connector Minimalist Device to Cloud(D2C)

After pausing my Azure Storage Queued based approach I built a quick Proof of Concept(PoC) with an HTTPTrigger Azure Function. The application has a single endpoint for processing uplink messages which is called by a The Things Industries(TTI) Webhooks integration.

The Things Industries Application Webhook configuration
namespace devMobile.IoT.TheThingsIndustries.AzureIoTHub
{
	using System.Collections.Concurrent;
	using Microsoft.Azure.Devices.Client;
...

	public partial class Integration
	{
...
		private static readonly ConcurrentDictionary<string, DeviceClient> _DeviceClients = new ConcurrentDictionary<string, DeviceClient>();
...
	}
}

The connector uses a ConcurrentDictionary(indexed by TTI deviceID) to cache Azure IoT Hub DeviceClient instances.

public partial class Webhooks
{
	[Function("Uplink")]
	public async Task<HttpResponseData> Uplink([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
	{
		var logger = executionContext.GetLogger("Uplink");

		// Wrap all the processing in a try\catch so if anything blows up we have logged it. Will need to specialise for connectivity failues etc.
		try
		{
			Models.PayloadUplink payload = JsonConvert.DeserializeObject<Models.PayloadUplink>(await req.ReadAsStringAsync());
			if (payload == null)
			{
				logger.LogInformation("Uplink: Payload {0} invalid", await req.ReadAsStringAsync());

				return req.CreateResponse(HttpStatusCode.BadRequest);
			}

			string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
			string deviceId = payload.EndDeviceIds.DeviceId;

			if ((payload.UplinkMessage.Port == null ) || (!payload.UplinkMessage.Port.HasValue) || (payload.UplinkMessage.Port.Value == 0))
			{
				logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Payload Raw:{2} Control nessage", applicationId, deviceId, payload.UplinkMessage.PayloadRaw);

				return req.CreateResponse(HttpStatusCode.BadRequest);
			}

			int port = payload.UplinkMessage.Port.Value;

			logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Port:{2} Payload Raw:{3}", applicationId, deviceId, port, payload.UplinkMessage.PayloadRaw);

			if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
			{
				logger.LogInformation("Uplink-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

				deviceClient = DeviceClient.CreateFromConnectionString(_configuration.GetConnectionString("AzureIoTHub"), deviceId);

				try
				{
					await deviceClient.OpenAsync();
				}
				catch (DeviceNotFoundException)
				{
					logger.LogWarning("Uplink-Unknown DeviceID:{0}", deviceId);

					return req.CreateResponse(HttpStatusCode.NotFound);
				}

				if (!_DeviceClients.TryAdd(deviceId, deviceClient))
				{
					logger.LogWarning("Uplink-TryAdd failed for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

					return req.CreateResponse(HttpStatusCode.Conflict);
				}
			}

			JObject telemetryEvent = new JObject
			{
				{ "ApplicationID", applicationId },
				{ "DeviceID", deviceId },
				{ "Port", port },
				{ "PayloadRaw", payload.UplinkMessage.PayloadRaw }
			};

			// If the payload has been decoded by payload formatter, put it in the message body.
			if (payload.UplinkMessage.PayloadDecoded != null)
			{
				telemetryEvent.Add("PayloadDecoded", payload.UplinkMessage.PayloadDecoded);
			}

			// Send the message to Azure IoT Hub
			using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
			{
				// Ensure the displayed time is the acquired time rather than the uploaded time. 
				ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture));
				ioTHubmessage.Properties.Add("ApplicationId", applicationId);
				ioTHubmessage.Properties.Add("DeviceEUI", payload.EndDeviceIds.DeviceEui);
				ioTHubmessage.Properties.Add("DeviceId", deviceId);
				ioTHubmessage.Properties.Add("port", port.ToString());

				await deviceClient.SendEventAsync(ioTHubmessage);
			}
		}
		catch (Exception ex)
		{
			logger.LogError(ex, "Uplink message processing failed");

			return req.CreateResponse(HttpStatusCode.InternalServerError);
		}

		return req.CreateResponse(HttpStatusCode.OK);
	}
}

For initial development and testing I ran the function application in the desktop emulator and simulated TTI webhook calls with Telerik Fiddler and modified TTI sample payloads.

Azure Functions Desktop development environment

I then deployed my function to Azure and configured the Azure IoT Hub connection string, Azure Application Insights key etc.

Azure Function configuration

I then used Azure IoT Explorer to configure devices, view uplink traffic etc. When I connected to my Azure IoT Hub shortly after starting the application all the devices were disconnected.

Azure IoT Explorer – no connected devices

The SeeeduinoLoRaWAN devices report roughly every 15 minutes so it took a while for them all to connect. (the SeeeduinoLoRaWAN4 & SeeeduinoLoRaWAN6 need to be repaired) .

Azure IoT Explorer – some connected devices

After a device had connected I could use Azure IoT Explorer to inspect the Seeeduino LoRaWAN device uplink message payloads.

Azure IoT Explorer displaying device telemetry

I also used Azure Application Insights to monitor the performance of the function and device activity.

Azure Application Insights displaying device telemetry

The Azure functions uplink message processor was then “soak tested” for a week without an issues.

Azure HTTP Trigger Functions with .NET Core 5

My updated The Things Industries(TTI) connector will use a number of Azure Functions to process Application Integration webhooks (with HTTP Triggers) and Azure Storage Queue messages(with Output Bindings & QueueTriggers).

On a couple of customer projects we had been updating Azure Functions from .NET 4.X to .NET Core 3.1, and most recently .NET Core 5. This process has been surprisingly painful so I decided to build a series of small proof of concept (PoC) projects to explore the problem.

Visual Studio Azure Function Trigger type selector

I started with the Visual Studio 2019 Azure Function template and created a plain HTTPTrigger.

public static class Function1
{
   [Function("Function1")]
   public static HttpResponseData Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req,
      FunctionContext executionContext)
   {
      var logger = executionContext.GetLogger("Function1");
      logger.LogInformation("C# HTTP trigger function processed a request.");

      var response = req.CreateResponse(HttpStatusCode.OK);
      response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

      response.WriteString("Welcome to Azure Functions!");

      return response;
   }
}

I changed the AuthorizationLevel to Anonymous to make testing in Azure with Telerik Fiddler easier

public static class Function1
{
	[Function("PlainAsync")]
	public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequestData request, FunctionContext executionContext)
	{
		var logger = executionContext.GetLogger("UplinkMessage");

		logger.LogInformation("C# HTTP trigger function processed a request.");

		var response = request.CreateResponse(HttpStatusCode.OK);

		response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

		response.WriteString("Welcome to Azure Functions!");

		return new OkResult();
	}
}

With not a lot of work I had an Azure Function I could run in the Visual Studio debugger

Azure Functions Debug Diagnostic Output

I could invoke the function using the endpoint displayed as debugging environment started.

Telerik Fiddler Composer invoking Azure Function running locally

I then added more projects to explore asynchronicity, and output bindings

Azure Functions Solution PoC Projects

After a bit of “trial and error” I had an HTTPTrigger Function that inserted a message containing the payload of an HTTP POST into an Azure Storage Queue.

[StorageAccount("AzureWebJobsStorage")]
public static class Function1
{
	[Function("Uplink")]
	public static async Task<HttpTriggerUplinkOutputBindingType> Uplink([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext context)
	{
		var logger = context.GetLogger("UplinkMessage");

		logger.LogInformation("Uplink processed");
			
		var response = req.CreateResponse(HttpStatusCode.OK);

		return new HttpTriggerUplinkOutputBindingType()
		{
			Name = await req.ReadAsStringAsync(),
			HttpReponse = response
		};
	}

	public class HttpTriggerUplinkOutputBindingType
	{
		[QueueOutput("uplink")]
		public string Name { get; set; }

		public HttpResponseData HttpReponse { get; set; }
	}
}

The key was Multiple Output Bindings so the function could return a result for both the HttpResponseData and Azure Storage Queue operations

Azure Functions Debug Diagnostic Output

After getting the function running locally I deployed it to a Function App running in an App Service plan

Azure HTTP Trigger function Host Key configuration

Using the Azure Portal I configured an x-functions-key which I could use in Telerik Fiddler

After fixing an accidental truncation of the x-functions-key a message with the body of the POST was created in the Azure Storage Queue.

Azure Storage Queue Message containing HTTP Post Payload

The aim of this series of PoCs was to have an Azure function that securely (x-functions-key) processed an Hyper Text Transfer Protocol(HTTP) POST with an HTTPTrigger and inserted a message containing the payload into an Azure Storage Queue using an OutputBinding.

Use the contents of this blog post with care as it may not age well.

.NET Core RAK811 LoRaWAN library Part3

The massive refactor

After refactoring my RAK3172 device library I have applied a similar approach to code on my RAK811 device library. My test-rig is a RaspberryPI 3B with a PI Supply RAK811 pHat and external antenna.

PI Supply RAK811 LoRaWAN pHat

In the new code a Thread reads lines of text from the SerialPort and processes them, checking for command responses, failures and downlink messages.

Unlike most of the devices I have worked with the RAK811 Join and Send commands are synchronous so return once the process has completed. The RAK811 responses also have quite a few empty, null prefixed or null suffixed lines which is a bit odd.

public void SerialPortProcessor()
{
	string line;

	while (CommandProcessResponses)
	{
		try
		{
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine before");
#endif
			line = SerialDevice.ReadLine().Trim('\0').Trim();
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine after:{line}");
#endif
			// consume empty lines
			if (String.IsNullOrWhiteSpace(line))
			{
				continue;
			}

			// Consume the response from set work mode
			if (line.StartsWith("?LoRa (R)") || line.StartsWith("RAK811 ") || line.StartsWith("UART1 ") || line.StartsWith("UART3 ") || line.StartsWith("LoRa work mode"))
			{
				continue;
			}

			// See if device successfully joined network
			if (line.StartsWith("OK Join Success"))
			{
				OnJoinCompletion?.Invoke(true);

				CommandResponseExpectedEvent.Set();

				continue;
			}

			if (line.StartsWith("at+recv="))
			{
				string[] payloadFields = line.Split("=,:".ToCharArray());

				byte port = byte.Parse(payloadFields[1]);
				int rssi = int.Parse(payloadFields[2]);
				int snr = int.Parse(payloadFields[3]);
				int length = int.Parse(payloadFields[4]);

				if (this.OnMessageConfirmation != null)
				{
					OnMessageConfirmation?.Invoke(rssi, snr);
				}
				if (length > 0)
				{
					string payload = payloadFields[5];

					if (this.OnReceiveMessage != null)
					{
						OnReceiveMessage.Invoke(port, rssi, snr, payload);
					}
				}
				continue;
			}

			switch (line)
			{
				case "OK":
				case "Initialization OK":
				case "OK Wake Up":
				case "OK Sleep":
					CommandResult = Result.Success;
					break;

				case "ERROR: 1":
					CommandResult = Result.ATCommandUnsuported;
					break;
				case "ERROR: 2":
					CommandResult = Result.ATCommandInvalidParameter;
					break;
				case "ERROR: 3": //There is an error when reading or writing flash.
				case "ERROR: 4": //There is an error when reading or writing through IIC.
					CommandResult = Result.ErrorReadingOrWritingFlash;
					break;
				case "ERROR: 5": //There is an error when sending through UART
					CommandResult = Result.ATCommandInvalidParameter;
					break;
				case "ERROR: 41": //The BLE works in an invalid state, so that it can’t be operated.
					CommandResult = Result.ResponseInvalid;
					break;
				case "ERROR: 80":
					CommandResult = Result.LoRaBusy;
					break;
				case "ERROR: 81":
					CommandResult = Result.LoRaServiceIsUnknown;
					break;
				case "ERROR: 82":
					CommandResult = Result.LoRaParameterInvalid;
					break;
				case "ERROR: 83":
					CommandResult = Result.LoRaFrequencyInvalid;
					break;
				case "ERROR: 84":
					CommandResult = Result.LoRaDataRateInvalid;
					break;
				case "ERROR: 85":
					CommandResult = Result.LoRaFrequencyAndDataRateInvalid;
					break;
				case "ERROR: 86":
					CommandResult = Result.LoRaDeviceNotJoinedNetwork;
					break;
				case "ERROR: 87":
					CommandResult = Result.LoRaPacketToLong;
					break;
				case "ERROR: 88":
					CommandResult = Result.LoRaServiceIsClosedByServer;
					break;
				case "ERROR: 89":
					CommandResult = Result.LoRaRegionUnsupported;
					break;
				case "ERROR: 90":
					CommandResult = Result.LoRaDutyCycleRestricted;
					break;
				case "ERROR: 91":
					CommandResult = Result.LoRaNoValidChannelFound;
					break;
				case "ERROR: 92":
					CommandResult = Result.LoRaNoFreeChannelFound;
					break;
				case "ERROR: 93":
					CommandResult = Result.StatusIsError;
					break;
				case "ERROR: 94":
					CommandResult = Result.LoRaTransmitTimeout;
					break;
				case "ERROR: 95":
					CommandResult = Result.LoRaRX1Timeout;
					break;
				case "ERROR: 96":
					CommandResult = Result.LoRaRX2Timeout;
					break;
				case "ERROR: 97":
					CommandResult = Result.LoRaRX1ReceiveError;
					break;
				case "ERROR: 98":
					CommandResult = Result.LoRaRX2ReceiveError;
					break;
				case "ERROR: 99":
					CommandResult = Result.LoRaJoinFailed;
					break;
				case "ERROR: 100":
					CommandResult = Result.LoRaDownlinkRepeated;
					break;
				case "ERROR: 101":
					CommandResult = Result.LoRaPayloadSizeNotValidForDataRate;
					break;
				case "ERROR: 102":
					CommandResult = Result.LoRaTooManyDownlinkFramesLost;
					break;
				case "ERROR: 103":
					CommandResult = Result.LoRaAddressFail;
					break;
				case "ERROR: 104":
					CommandResult = Result.LoRaMicVerifyError;
					break;
				default:
					CommandResult = Result.ResponseInvalid;
					break;
			}
		}
		catch (TimeoutException)
		{
			// Intentionally ignored, not certain this is a good idea
		}

		CommandResponseExpectedEvent.Set();
	}
}

After a lot of testing I think my thread based approach works reliably. Initially, I was having some signal strength issues because I had forgotten to configure the external antenna. I need to add some validation to the metrics and payload field unpacking (though I’m not certain what todo if they are the wrong format).

RAK3172LoRaWAN-NetCore on Github

The source code for a late beta version of my .Net Core C# library for RAK3172 modules is now available on GitHub.

Using SerialDataReceivedEventHandler was causing memory management problems so the core command processor now runs in its own Thread.(Though I worry about the continuous System.TimeoutExceptions)

A sample application which shows how to connect using Over the Air Activation(OTAA) or Activation By Personalisation(ABP) then send and receive byte array/hexadecimal messages.

//---------------------------------------------------------------------------------
// Copyright (c) September 2021, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// Must have one of following options defined in the project\build definitions
//    PAYLOAD_BCD or PAYLOAD_BYTES
//    OTAA or ABP
//
// Optional definitions
//    CONFIRMED For confirmed messages
//		DEVEUI_SET
//
//---------------------------------------------------------------------------------
namespace devMobile.IoT.LoRaWAN.NetCore.RAK3172
{
	using System;
	using System.IO.Ports;
	using System.Threading;


	public class Program
	{
		private const string SerialPortId = "/dev/ttyS0";
		private const LoRaClass Class = LoRaClass.A;
		private const string Band = "8-1";
		private const byte MessagePort = 10;
		private static readonly TimeSpan MessageSendTimerDue = new TimeSpan(0, 0, 15);
		private static readonly TimeSpan MessageSendTimerPeriod = new TimeSpan(0, 5, 0);
		private static Timer MessageSendTimer ;
		private const int JoinRetryAttempts = 2;
		private const int JoinRetryIntervalSeconds = 10;
#if PAYLOAD_BCD
		private const string PayloadBcd = "48656c6c6f204c6f526157414e"; // Hello LoRaWAN in BCD
#endif
#if PAYLOAD_BYTES
		private static readonly byte[] PayloadBytes = { 0x48, 0x65 , 0x6c, 0x6c, 0x6f, 0x20, 0x4c, 0x6f, 0x52, 0x61, 0x57, 0x41, 0x4e}; // Hello LoRaWAN in bytes
#endif

		public static void Main()
		{
			Result result;

			Console.WriteLine("devMobile.IoT.LoRaWAN.NetCore.RAK3172 RAK3712LoRaWANDeviceClient starting");

			Console.WriteLine($"Serial ports:{String.Join(",", SerialPort.GetPortNames())}");

			try
			{
				using (Rak3172LoRaWanDevice device = new Rak3172LoRaWanDevice())
				{
					result = device.Initialise(SerialPortId, 9600, Parity.None, 8, StopBits.One);
					if (result != Result.Success)
					{
						Console.WriteLine($"Initialise failed {result}");
						return;
					}

					MessageSendTimer = new Timer(SendMessageTimerCallback, device,Timeout.Infinite, Timeout.Infinite);

					device.OnJoinCompletion += OnJoinCompletionHandler;
					device.OnReceiveMessage += OnReceiveMessageHandler;
#if CONFIRMED
					device.OnMessageConfirmation += OnMessageConfirmationHandler;
#endif

#if DEVEUI_SET
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} DevEUI {Config.devEui}");
					result = device.DeviceEui(Config.devEui);
					if (result != Result.Success)
					{
						Console.WriteLine($"DevEUI failed {result}");
						return;
					}
#endif

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Class {Class}");
					result = device.Class(Class);
					if (result != Result.Success)
					{
						Console.WriteLine($"Class failed {result}");
						return;
					}

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Band {Band}");
					result = device.Band(Band);
					if (result != Result.Success)
					{
						Console.WriteLine($"Region failed {result}");
						return;
					}

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} ADR On");
					result = device.AdrOn();
					if (result != Result.Success)
					{
						Console.WriteLine($"ADR on failed {result}");
						return;
					}

#if CONFIRMED
               Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Confirmed");
               result = device.UplinkMessageConfirmationOn();
               if (result != Result.Success)
               {
                  Console.WriteLine($"Confirm on failed {result}");
                  return;
               }
#else
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Unconfirmed");
					result = device.UplinkMessageConfirmationOff();
					if (result != Result.Success)
					{
						Console.WriteLine($"Confirm off failed {result}");
						return;
					}
#endif

#if OTAA
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} OTAA");
					result = device.OtaaInitialise(Config.JoinEui, Config.AppKey);
					if (result != Result.Success)
					{
						Console.WriteLine($"OTAA Initialise failed {result}");
						return;
					}
#endif

#if ABP
               Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} ABP");
               result = device.AbpInitialise(Config.DevAddress, Config.NwksKey, Config.AppsKey);
               if (result != Result.Success)
               {
                  Console.WriteLine($"ABP Initialise failed {result}");
                  return;
               }
#endif

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Join start");
					result = device.Join(JoinRetryAttempts, JoinRetryIntervalSeconds);
					if (result != Result.Success)
					{
						Console.WriteLine($"Join failed {result}");
						return;
					}
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Join started");

					Thread.Sleep(Timeout.Infinite);
				}
			}
			catch (Exception ex)
			{
				Console.WriteLine(ex.Message);
			}
		}

		private static void OnJoinCompletionHandler(bool result)
		{
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Join finished:{result}");

			if (result)
			{ 
				MessageSendTimer.Change(MessageSendTimerDue, MessageSendTimerPeriod);
			}
		}

		private static void SendMessageTimerCallback(object state)
		{
			Rak3172LoRaWanDevice device = (Rak3172LoRaWanDevice)state;

#if PAYLOAD_BCD
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} port:{MessagePort} payload BCD:{PayloadBcd}");
			Result result = device.Send(MessagePort, PayloadBcd );
#endif
#if PAYLOAD_BYTES
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} port:{MessagePort} payload bytes:{Rak3172LoRaWanDevice.BytesToBcd(PayloadBytes)}");
         Result result = device.Send(MessagePort, PayloadBytes);
#endif
			if (result != Result.Success)
			{
				Console.WriteLine($"Send failed {result}");
			}
		}

#if CONFIRMED
		private static void OnMessageConfirmationHandler()
      {
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Send successful");
		}
#endif

		private static void OnReceiveMessageHandler(byte port, int rssi, int snr, string payload)
		{
			byte[] payloadBytes = Rak3172LoRaWanDevice.HexToByes(payload); // Done this way so both conversion methods tested

			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Receive Message RSSI:{rssi} SNR:{snr} Port:{port} Payload:{payload} PayLoadBytes:{BitConverter.ToString(payloadBytes)}");
		}
	}
}

I have added XML Documentation comments which will need some rework, after I have “soak tested” the code for at least a week.

I have also added a method so the DevEUI can be set (intended for use after device firmware has been updated), fixed up my mistake with Binary Coded Decimal(BCD) vs. Hexadecimal strings.

I will also go back and apply the “learnings” from this refactoring to my other LoRaWAN module and platform libraries

.NET Core RAK3172 LoRaWAN library Part5

The massive refactor

After getting Activation By Personalisation(ABP) and Over The Air Activation(OTAA) working on my RAK3172 test rig I was looking at the code and SerialDataReceivedEventHandler was really ugly.

Raspberry Pi3 with Grove Base Hat and RAK3172 Breakout (using UART2)

After some experimentation in the BreakOutSerial project I decided to reimplement the RAK3172 command processing. In the new code a Thread reads lines of text from the SerialPort and processes them. I have replaced the Join and Send(Confirmed) methods with ones that block only while the command are sent to the RAK3172. Then, when completed the OnJoinCompletion or OnMessagesConfirmation event handlers are called.

private Result SendCommand(string command)
{
	if (command == null)
	{
		throw new ArgumentNullException(nameof(command));
	}

	if (command == string.Empty)
	{
		throw new ArgumentException($"command cannot be empty", nameof(command));
	}

	serialDevice.WriteLine(command);

	this.CommandResponseExpectedEvent.Reset();

	if (!this.CommandResponseExpectedEvent.WaitOne(CommandTimeoutDefaultmSec, false))
	{
		return Result.Timeout;
	}

	return CommandResult;
}

private void SerialPortProcessor()
{
	string line;

	while (CommandProcessResponses)
	{
		try
		{
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine before");
#endif
			line = serialDevice.ReadLine();
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine after:{line}");
#endif

			// See if device successfully joined network
			if (line.StartsWith("+EVT:JOINED"))
			{
				OnJoinCompletion?.Invoke(true);

				continue;
			}

			// See if device failed ot join network
			if (line.StartsWith("+EVT:JOIN FAILED"))
			{
				OnJoinCompletion?.Invoke(false);

				continue;
			}

			// Applicable only if confirmed messages enabled 
			if (line.StartsWith("+EVT:SEND CONFIRMED OK"))
			{
				OnMessageConfirmation?.Invoke();

				continue;
			}

			// Check for A/B/C downlink message
			if (line.StartsWith("+EVT:RX_1") || line.StartsWith("+EVT:RX_2") || line.StartsWith("+EVT:RX_3") || line.StartsWith("+EVT:RX_C"))
			{
				// TODO beef up validation, nto certain what todo if borked
				string[] metricsFields= line.Split(' ', ',');

				int rssi = int.Parse(metricsFields[3]);
				int snr = int.Parse(metricsFields[6]);

				line = serialDevice.ReadLine();

#if DIAGNOSTICS
				Debug.WriteLine($" {DateTime.UtcNow:HH:mm:ss} UNICAST :{line}");
#endif
				line = serialDevice.ReadLine();
#if DIAGNOSTICS
				Debug.WriteLine($" {DateTime.UtcNow:HH:mm:ss} Payload:{line}");
#endif
				// TODO beef up validation, nto certain what todo if borked
				string[] payloadFields = line.Split(':');

				byte port = byte.Parse(payloadFields[1]);
				string payload = payloadFields[2];

				OnReceiveMessage?.Invoke(port, rssi, snr, payload);

				continue;
			}

#if DIAGNOSTICS
           Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine Result");
#endif
			line = serialDevice.ReadLine();
#if DIAGNOSTICS
             Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine Result:{line}");
#endif
			switch (line)
			{
				case "OK":
					CommandResult = Result.Success;
					break;
				case "AT_ERROR":
					CommandResult = Result.AtError;
					break;
				case "AT_PARAM_ERROR":
					CommandResult = Result.ParameterError;
					break;
				case "AT_BUSY_ERROR":
					CommandResult = Result.BusyError;
					break;
				case "AT_TEST_PARAM_OVERFLOW":
					CommandResult = Result.ParameterOverflow;
					break;
				case "AT_NO_NETWORK_JOINED":
					CommandResult = Result.NotJoined;
					break;
				case "AT_RX_ERROR":
					CommandResult = Result.ReceiveError;
					break;
				case "AT_DUTYCYLE_RESTRICTED":
					CommandResult = Result.DutyCycleRestricted;
					break;
				default:
					CommandResult = Result.Undefined;
					break;
			}

			CommandResponseExpectedEvent.Set();
		}
		catch (TimeoutException)
		{
			// Intentionally ignored, not certain this is a good idea
		}
	}
}

After a lot of testing I think my thread based approach works reliably. I also had to modify the code to shutdown the command processor thread and free any non managed resources.

/// <summary>
/// Ensures unmanaged serial port and thread resources are released in a "responsible" manner.
/// </summary>
public void Dispose()
{
	CommandProcessResponses = false;

	if (CommandResponsesProcessorThread != null)
	{
		CommandResponsesProcessorThread.Join();
		CommandResponsesProcessorThread = null;
	}

	if (serialDevice != null)
	{
		serialDevice.Dispose();
		serialDevice = null;
	}
}

I need to add some validation to the metrics and payload field unpacking (though I’m not certain what todo if they are the wrong format) and review the handling of multi-line event messages.

.NET Core RAK3172 LoRaWAN library Part4

Starting again with Threads

After getting Activation By Personalisation(ABP) and Over The Air Activation(OTAA) working on my RAK3172 test rig I was looking at the code and SerialDataReceivedEventHandler was really ugly.

Raspberry Pi3 with Grove Base Hat and RAK3172 Breakout (using UART2)

After some experimentation in the BreakOutSerial project I decided to reimplement the RAK3172 command processing. In the new code a Thread reads lines of text from the SerialPort and processes them. I have replaced the Join and Send(Confirmed) methods with ones that block only while the command are sent to the RAK3172. Then, when completed the OnJoinCompletion or OnMessagesConfirmation event handlers are called.

private Result SendCommand(string command)
{
   if (command == null)
   {
      throw new ArgumentNullException(nameof(command));
   }

   if (command == string.Empty)
   {
      throw new ArgumentException($"command invalid length cannot be empty", nameof(command));
    }

   serialDevice.ReadTimeout = (int)CommandTimeoutDefault.TotalMilliseconds;
   serialDevice.WriteLine(command);

   this.atExpectedEvent.Reset();

   if (!this.atExpectedEvent.WaitOne((int)CommandTimeoutDefault.TotalMilliseconds, false))
      return Result.Timeout;

   return result;
}

public void SerialPortProcessor()
{
   string line;

   while (true)
   {
      this.serialDevice.ReadTimeout = -1;

      Debug.WriteLine("ReadLine before");
      line = serialDevice.ReadLine();
      Debug.WriteLine($"ReadLine after:{line}");

            // check for +EVT:JOINED
      if (line.StartsWith("+EVT:JOINED"))
      {
            OnJoinCompletion?.Invoke(true);

            continue;
      }

      if (line.StartsWith("+EVT:JOIN FAILED"))
      {
	     OnJoinCompletion?.Invoke(false);

         continue;
      }

      if (line.StartsWith("+EVT:SEND CONFIRMED OK"))
      {
         OnMessageConfirmation?.Invoke();

         continue;
      }

      // Check for A/B/C downlink message
      if (line.StartsWith("+EVT:RX_1") || line.StartsWith("+EVT:RX_2") || line.StartsWith("+EVT:RX_3") || line.StartsWith("+EVT:RX_C"))
      {
         string[] fields1 = line.Split(' ', ',');

         int rssi = int.Parse(fields1[3]);
         int snr = int.Parse(fields1[6]);
 
         line = serialDevice.ReadLine();
         Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} UNICAST :{line}");

         line = serialDevice.ReadLine();
         Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} Payload:{line}");

         string[] fields2 = line.Split(':');

         int port = int.Parse(fields2[1]);
         string payload = fields2[2];

         OnReceiveMessage?.Invoke(port, rssi, snr, payload);

         continue;
      }

      try
      {
         this.serialDevice.ReadTimeout = 3000;

         Debug.WriteLine("ReadLine Result");
         line = serialDevice.ReadLine();
         Debug.WriteLine($"ReadLine Result after:{line}");

         switch (line)
         {
            case "OK":
               result = Result.Success;
               break;
         case "AT_ERROR":
               result = Result.Error;
               break;
         case "AT_PARAM_ERROR":
               result = Result.ParameterError;
               break;
         case "AT_BUSY_ERROR":
               result = Result.BusyError;
               break;
         case "AT_TEST_PARAM_OVERFLOW":
               result = Result.ParameterOverflow;
               break;
         case "AT_NO_NETWORK_JOINED":
               result = Result.NotJoined;
               break;
         case "AT_RX_ERROR":
               result = Result.ReceiveError;
               break;
         case "AT_DUTYCYLE_RESTRICTED":
               result = Result.DutyCycleRestricted;
               break;
         default:
               result = Result.Undefined;
               break;
         }
      }
      catch (TimeoutException) 
      {
         result = Result.Timeout;
      }
   atExpectedEvent.Set();
}

The code is not suitable for production but it confirmed my thread based approach works. I need to add code to shutdown the message processing thread in a controlled way, support for Class B & C devices, replace the OnJoinCompletionHandler timer magic numbers and soak test for 5-7 days.

Visual Studio Displaying RAK3172 device joining network then sending messages

In the Visual Studio 2019 debug output I could see messages getting sent and then after a short delay they were visible in the TTN console.

TTN Displaying RAK3172 device joining network then sending messages