libcamera on Raspberry Pi OS Bullseye

This is a “note to self” post about using libcamera(replacement for raspistill) on my Raspberry PI 4 Model B to capture an image from my Raspberry Pi Camera Module 2 with an application built with .NET Core.

I wanted one of my ML.Net demos to use the Raspberry PI Camera rather than a security camera (so it was more portable) but it took a bit more work than I expected.

Version 1 used Process.Start to launch the libcamera-jpeg application with a command line to store an image to the local file system.

libcamera-jpeg -o latest.jpg
libcamera-jpeg with diagnostic information displayed

There was a lot of diagnostic information which I didn’t want displayed so after reading many stackoverflow posts (lots of different approaches none of which worked in my scenario), then some trial and error I found that I only had to enable RedirectStandardError.

libcamera-jpeg without diagnostic information displayed

At this point there was a lot less noise but the image was upside down.

Inverted picture of my 30th anniversary Mini Cooper in the backyard

I then added a vertical flip to the command line parameters

libcamera-jpeg -o latest.jpg --vflip
My 30th anniversary Mini Cooper in the backyard

The image was backwards so I added a horizontal flip to the commandline parameters

libcamera-jpeg -o latest.jpg --vflip --hflip
My 30th anniversary Mini Cooper in the backyard with the correct orientation

The libcamera code is in a Timer callback so I added the _cameraBusy boolean flag to stop reentrancy problems.

private static void ImageUpdateTimerCallback(object state)
{
	try
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image update start");

		// Just incase - stop code being called while photo already in progress
		if (_cameraBusy)
		{
			return;
		}

		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image capture start");

		using (Process process = new Process())
		{
			process.StartInfo.FileName = @"libcamera-jpeg";
			// V1 it works
			//process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal}";
			// V3 Image right way up
			//process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal} --vflip";
			// V3 Image right way round
			process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal} --vflip --hflip";
			//process.StartInfo.RedirectStandardOutput = true;
			// V2 No diagnostics
			process.StartInfo.RedirectStandardError = true;
			//process.StartInfo.UseShellExecute = false;
			//process.StartInfo.CreateNoWindow = true; 

			process.Start();

			if (!process.WaitForExit(10000) || (process.ExitCode != 0))
			{
				Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image update failure {process.ExitCode}");
			}
		}

		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image capture done");
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image update error {ex.Message}");
	}
	finally
	{
		_cameraBusy = false;
	}
}

This was the simplest way I could get an image onto the local file system without lots of dependencies on third party libraries. The image capture process takes about 5 seconds which a bit longer than I was expecting.

TTI V3 Connector Device Provisioning Service(DPS) support

The previous versions of my Things Network Industries(TTI) and The Things Network(TTN) connectors supported the Azure IoT Hub Device Provisioning Service(DPS) with Symmetric Key Attestation(SAS) to “automagically” setup the LoRaWAN devices in a TTI Application.(See my V2 Gateway DPS setup post for more detail).

Azure Device Provisioning Service configuring Azure IoT Hubs

I used an “evenly weighted distribution” to spread the devices across five Azure IoT Hubs.

Azure IoT Hub no registered devices

In the Azure Portal I configured the DPS ID Scope (AzureSettings:DeviceProvisioningServiceSettings:IdScope) and the Group Enrollment Key(AzureSettings:DeviceProvisioningServiceSettings:GroupEnrollmentKey) then saved the configuration which restarted the AppService.

Azure Portal AppService configration

The first time a device sent an uplink message the cache query fails and the RegisterAsync method of the ProvisioningDeviceClient is called to get a device connection string.

	logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Port:{2} Payload Raw:{3}", applicationId, deviceId, port, payload.UplinkMessage.PayloadRaw);

	if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
	{
		logger.LogInformation("Uplink-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

		// Check that only one of Azure Connection string or DPS is configured
		if (string.IsNullOrEmpty(_azureSettings.IoTHubConnectionString) && (_azureSettings.DeviceProvisioningServiceSettings == null))
		{
			logger.LogError("Uplink-Neither Azure IoT Hub connection string or Device Provisioning Service configured");

			return req.CreateResponse(HttpStatusCode.UnprocessableEntity);
		}

		// Check that only one of Azure Connection string or DPS is configured
		if (!string.IsNullOrEmpty(_azureSettings.IoTHubConnectionString) && (_azureSettings.DeviceProvisioningServiceSettings != null))
		{
			logger.LogError("Uplink-Both Azure IoT Hub connection string and Device Provisioning Service configured");

			return req.CreateResponse(HttpStatusCode.UnprocessableEntity);
		}

		// User Azure IoT Connection string if configured and Device Provisioning Service isn't
		if (!string.IsNullOrEmpty(_azureSettings.IoTHubConnectionString))
		{
			deviceClient = DeviceClient.CreateFromConnectionString(_azureSettings.IoTHubConnectionString, deviceId, transportSettings);

			try
			{
				await deviceClient.OpenAsync();
			}
			catch (DeviceNotFoundException)
			{
				logger.LogWarning("Uplink-Unknown DeviceID:{0}", deviceId);

				return req.CreateResponse(HttpStatusCode.NotFound);
			}
		}

		// Azure IoT Hub Device provisioning service if configured
		if (_azureSettings.DeviceProvisioningServiceSettings != null) 
		{
			string deviceKey;

			if ( string.IsNullOrEmpty(_azureSettings.DeviceProvisioningServiceSettings.IdScope) || string.IsNullOrEmpty(_azureSettings.DeviceProvisioningServiceSettings.GroupEnrollmentKey))
			{
				logger.LogError("Uplink-Device Provisioning Service requires ID Scope and Group Enrollment Key configured");

				return req.CreateResponse(HttpStatusCode.UnprocessableEntity);
			}

			using (var hmac = new HMACSHA256(Convert.FromBase64String(_azureSettings.DeviceProvisioningServiceSettings.GroupEnrollmentKey)))
			{
				deviceKey = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(deviceId)));
			}

			using (var securityProvider = new SecurityProviderSymmetricKey(deviceId, deviceKey, null))
			{
				using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
				{
					ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
						Constants.AzureDpsGlobalDeviceEndpoint,
						_azureSettings.DeviceProvisioningServiceSettings.IdScope,
						securityProvider,
						transport);

					DeviceRegistrationResult result = await provClient.RegisterAsync();

					if (result.Status != ProvisioningRegistrationStatusType.Assigned)
					{
						_logger.LogError("Config-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);

						return req.CreateResponse(HttpStatusCode.FailedDependency);
					}

					IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

					deviceClient = DeviceClient.Create(result.AssignedHub, authentication, transportSettings);

					await deviceClient.OpenAsync();
				}
			}
		}

		if (!_DeviceClients.TryAdd(deviceId, deviceClient))
		{
			logger.LogWarning("Uplink-TryAdd failed for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

			return req.CreateResponse(HttpStatusCode.Conflict);
		}

		Models.AzureIoTHubReceiveMessageHandlerContext context = new Models.AzureIoTHubReceiveMessageHandlerContext()
		{
			DeviceId = deviceId,
			ApplicationId = applicationId,
			WebhookId = _theThingsIndustriesSettings.WebhookId,
			WebhookBaseURL = _theThingsIndustriesSettings.WebhookBaseURL,
			ApiKey = _theThingsIndustriesSettings.ApiKey
		};

		await deviceClient.SetReceiveMessageHandlerAsync(AzureIoTHubClientReceiveMessageHandler, context);

		await deviceClient.SetMethodDefaultHandlerAsync(AzureIoTHubClientDefaultMethodHandler, context);
	}

	JObject telemetryEvent = new JObject
	{
		{ "ApplicationID", applicationId },
		{ "DeviceID", deviceId },
		{ "Port", port },
		{ "Simulated", payload.Simulated },
		{ "ReceivedAtUtc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture) },
		{ "PayloadRaw", payload.UplinkMessage.PayloadRaw }
	};

	// If the payload has been decoded by payload formatter, put it in the message body.
	if (payload.UplinkMessage.PayloadDecoded != null)
	{
		telemetryEvent.Add("PayloadDecoded", payload.UplinkMessage.PayloadDecoded);
	}

	// Send the message to Azure IoT Hub
	using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
	{
		// Ensure the displayed time is the acquired time rather than the uploaded time. 
		ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture));
		ioTHubmessage.Properties.Add("ApplicationId", applicationId);
		ioTHubmessage.Properties.Add("DeviceEUI", payload.EndDeviceIds.DeviceEui);
		ioTHubmessage.Properties.Add("DeviceId", deviceId);
		ioTHubmessage.Properties.Add("port", port.ToString());
		ioTHubmessage.Properties.Add("Simulated", payload.Simulated.ToString());

		await deviceClient.SendEventAsync(ioTHubmessage);

		logger.LogInformation("Uplink-DeviceID:{0} SendEventAsync success", payload.EndDeviceIds.DeviceId);
	}
}
catch (Exception ex)
{
	logger.LogError(ex, "Uplink-Message processing failed");

	return req.CreateResponse(HttpStatusCode.InternalServerError);
}

I used Telerik Fiddler and some sample payloads copied from my Azure Storage Queue sample to simulate many devices and the registrations were spread across my five Azure IoT Hubs.

DPS Device Registrations tab showing distribution of LoRaWAN Devices

I need to review the HTTP Error codes returned for different errors and ensure failures are handled robustly.

TTI V3 Connector Minimalist Cloud to Device(C2D)

In a previous version of my Things Network Industries(TTI) The Things Network(TTN) connector I queried the The Things Stack(TTS) Application Programing Interface(API) to get a list of Applications and their Devices. For a large number of Applications and/or Devices this process could take many 10’s of seconds. Application and Device creation and deletion then had to be tracked to keep the AzureDeviceClient connection list current, which added significant complexity.

In this version a downlink message can be sent to a device only after an uplink message. I’m looking at adding an Azure Function which initiates a connection to the configured Azure IoT Hub for the specified device to mitigate with this issue.

To send a TTN downlink message to a device the minimum required info is the LoRaWAN port number (specified in a Custom Property on the Azure IoT Hub cloud to device message), the device Id (from uplink message payload, which has been validated by a successful Azure IoT Hub connection) web hook id, web hook base URL, and an API Key (The Web Hook parameters are stored in the Connector configuration).

Azure IoT Explorer Clod to Device message with LoRaWAN Port number custom parameter

When a LoRaWAN device sends an Uplink message a session is established using Advanced Message Queuing Protocol(AMQP) so connections can be multiplexed)

I used Azure IoT Explorer to send Cloud to Device messages to the Azure IoT Hub (to initiate the sending of a downlink message to the Device by the Connector) after simulating a TTN uplink message with Telerik Fiddler and a modified TTN sample payload.

BEWARE – TTN URLs and Azure IoT Hub device identifiers are case sensitive

...
if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
{
   logger.LogInformation("Uplink-Unknown device for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

   deviceClient = DeviceClient.CreateFromConnectionString(_configuration.GetConnectionString("AzureIoTHub"), deviceId, 
                    new ITransportSettings[]
                    {
                        new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
                        {
                            AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
                            {
                                Pooling = true,
                            }
                        }
                    });

   try
   {
      await deviceClient.OpenAsync();
   }
   catch (DeviceNotFoundException)
   {
      logger.LogWarning("Uplink-Unknown DeviceID:{0}", deviceId);

      return req.CreateResponse(HttpStatusCode.NotFound);
   }

   if (!_DeviceClients.TryAdd(deviceId, deviceClient))
   {
      logger.LogWarning("Uplink-TryAdd failed for ApplicationID:{0} DeviceID:{1}", applicationId, deviceId);

      return req.CreateResponse(HttpStatusCode.Conflict);
   }

   Models.AzureIoTHubReceiveMessageHandlerContext context = new Models.AzureIoTHubReceiveMessageHandlerContext()
   { 
      DeviceId = deviceId,
      ApplicationId = applicationId,
      WebhookId = _configuration.GetSection("TheThingsIndustries").GetSection("WebhookId").Value,
      WebhookBaseURL = _configuration.GetSection("TheThingsIndustries").GetSection("WebhookBaseURL").Value,
      ApiKey = _configuration.GetSection("TheThingsIndustries").GetSection("APiKey").Value,
   };      

   await deviceClient.SetReceiveMessageHandlerAsync(AzureIoTHubClientReceiveMessageHandler, context);

   await deviceClient.SetMethodDefaultHandlerAsync(AzureIoTHubClientDefaultMethodHandler, context);
 }

...

An Azure IoT Hub can invoke methods(synchronous) or send messages(asynchronous) to a device for processing. The Azure IoT Hub DeviceClient has two methods SetMethodDefaultHandlerAsync and SetReceiveMessageHandlerAsync which enable the processing of direct methods and messages.

private async Task<MethodResponse> AzureIoTHubClientDefaultMethodHandler(MethodRequest methodRequest, object userContext)
{
	if (methodRequest.DataAsJson != null)
	{
		_logger.LogWarning("AzureIoTHubClientDefaultMethodHandler name:{0} payload:{1}", methodRequest.Name, methodRequest.DataAsJson);
	}
	else
	{
		_logger.LogWarning("AzureIoTHubClientDefaultMethodHandler name:{0} payload:NULL", methodRequest.Name);
	}

	return new MethodResponse(404);
}

After some experimentation in previous TTN Connectors I found the synchronous nature of DirectMethods didn’t work well with LoRAWAN “irregular” connectivity so currently they are ignored.

public partial class Integration
{
	private async Task AzureIoTHubClientReceiveMessageHandler(Message message, object userContext)
	{
		try
		{
			Models.AzureIoTHubReceiveMessageHandlerContext receiveMessageHandlerContext = (Models.AzureIoTHubReceiveMessageHandlerContext)userContext;

			if (!_DeviceClients.TryGetValue(receiveMessageHandlerContext.DeviceId, out DeviceClient deviceClient))
			{
				_logger.LogWarning("Downlink-DeviceID:{0} unknown", receiveMessageHandlerContext.DeviceId);
				return;
			}

			using (message)
			{
				string payloadText = Encoding.UTF8.GetString(message.GetBytes()).Trim();

				if (!AzureDownlinkMessage.PortTryGet(message.Properties, out byte port))
				{
					_logger.LogWarning("Downlink-Port property is invalid");

					await deviceClient.RejectAsync(message);
					return;
				}

				// Split over multiple lines in an attempt to improve readability. In this scenario a valid JSON string should start/end with {/} for an object or [/] for an array
				if ((payloadText.StartsWith("{") && payloadText.EndsWith("}"))
														||
					((payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
				{
					try
					{
						downlink.PayloadDecoded = JToken.Parse(payloadText);
					}
					catch (JsonReaderException)
					{
						downlink.PayloadRaw = payloadText;
					}
				}
				else
				{
					downlink.PayloadRaw = payloadText;
				}

				_logger.LogInformation("Downlink-IoT Hub DeviceID:{0} MessageID:{1} LockToken :{2} Port{3}",
					receiveMessageHandlerContext.DeviceId,
					message.MessageId,
		            message.LockToken,
					downlink.Port);

				Models.DownlinkPayload Payload = new Models.DownlinkPayload()
				{
					Downlinks = new List<Models.Downlink>()
					{
						downlink
					}
				};

				string url = $"{receiveMessageHandlerContext.WebhookBaseURL}/{receiveMessageHandlerContext.ApplicationId}/webhooks/{receiveMessageHandlerContext.WebhookId}/devices/{receiveMessageHandlerContext.DeviceId}/down/replace");

				using (var client = new WebClient())
				{
					client.Headers.Add("Authorization", $"Bearer {receiveMessageHandlerContext.ApiKey}");

					client.UploadString(new Uri(url), JsonConvert.SerializeObject(Payload));
				}

				_logger.LogInformation("Downlink-DeviceID:{0} LockToken:{1} success", receiveMessageHandlerContext.DeviceId, message.LockToken);
			}
		}
		catch (Exception ex)
		{
			_logger.LogError(ex, "Downlink-ReceiveMessge processing failed");
		}
	}
}

If the message body contains a valid JavaScript Object Notation(JSON) payload it is “spliced” into the DownLink message decoded_payload field otherwise the Base64 encoded frm_payload is populated.

The Things “Industries Live” data tab downlink message

The SetReceiveMessageHandlerAsync context information is used to construct a TTN downlink message payload and request URL(with default queuing, message priority and confirmation options)

Arduino Serial Monitor displaying Uplink and Downlink messages

TTI V3 Connector Azure Storage Queues Paused

After running my The Things Industries(TTI) V3 HTTPStorageQueueOutput application for a week I think there are some problems with my approach so I have paused development while I build another HTTPTrigger Azure Functions based Proof of Concept(PoC).

The HTTPTrigger and Azure Storage Queue OutputBinding based code which inserts messages into an Azure Storage Queue was minimal

[StorageAccount("AzureWebJobsStorage")]
public static class Webhooks
{
	[Function("Uplink")]
	public static async Task<HttpTriggerUplinkOutputBindingType> Uplink([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext context)
	{
		var logger = context.GetLogger("UplinkMessage");

		logger.LogInformation("Uplink processed");
			
		var response = req.CreateResponse(HttpStatusCode.OK);

		return new HttpTriggerUplinkOutputBindingType()
		{
			Name = await req.ReadAsStringAsync(),
			HttpReponse = response
		};
	}
}

With Azure Storage Explorer I could inspect uplink, queued, sent, and acknowledgment(ACK) messages. It was difficult to generate failed and Negative Acknowledgement (Nack) and failed messages

Azure Storage Explorer displaying Uplink messages
Azure Storage Explorer displaying queued messages
Azure Storage Explorer displaying sent messages
Azure Storage Explorer Displaying Ack messages

After some experimentation I realised that I had forgotten that the order of message processing was important e.g. a TTI Queued message should be processed before the associated Ack. This could (and did happen) because I had a queue for each message type and in addition the Azure Queue Storage trigger binding would use parallel execution to process backlogs of messages. My approach caused issues with both intra and inter queue message ordering

Sensirion SHT 20 library for .NET Core 5.0

As part of project I needed to connect a Sensirion SHT20 driver to a.NET Core 5 application running on a Raspberry Pi so I wrote this library. For initial testing I used a DF Robot Waterproof SHT20 temperature and humidity sensor, Seeedstudio Gove Base Hat, Grove Screw Terminal, and a Grove – Universal 4 Pin Buckled 5cm Cable.

Sensirion SHT20 connected to Raspberry PI3

I have included sample application in the Github repository to show how to use the library

namespace devMobile.IoT.NetCore.Sensirion
{
	using System;
	using System.Device.I2c;
	using System.Threading;

	class Program
	{
		static void Main(string[] args)
		{
			// bus id on the raspberry pi 3
			const int busId = 1;

			I2cConnectionSettings i2cConnectionSettings = new(busId, Sht20.DefaultI2cAddress);

			using I2cDevice i2cDevice = I2cDevice.Create(i2cConnectionSettings);

			using (Sht20 sht20 = new Sht20(i2cDevice))
			{
				sht20.Reset();

				while (true)
				{
					double temperature = sht20.Temperature();
					double humidity = sht20.Humidity();

#if HEATER_ON_OFF
					sht20.HeaterOn();
					Console.WriteLine($"{DateTime.Now:HH:mm:ss} HeaterOn:{sht20.IsHeaterOn()}");
#endif
					Console.WriteLine($"{DateTime.Now:HH:mm:ss} Temperature:{temperature:F1}°C Humidity:{humidity:F0}% HeaterOn:{sht20.IsHeaterOn()}");
#if HEATER_ON_OFF
					sht20.HeaterOff();
					Console.WriteLine($"{DateTime.Now:HH:mm:ss} HeaterOn:{sht20.IsHeaterOn()}");
#endif

					Thread.Sleep(1000);
				}
			}
		}
	}
}

The Sensiron SHT20 has a heater which is intended to be used for functionality diagnosis – relative humidity drops upon rising temperature. The heater consumes about 5.5mW and provides a temperature increase of about 0.5 – 1.5°C.

Beware when the device is soft reset the heater bit is not cleared.

Grove Base Hat for Raspberry PI with .NET Core 5.0

Over the weekend I ported my Windows 10 IoT Core library for Seeedstudio Grove Base Hat for RPI Zero and Grove Base Hat for Raspberry Pi to .NET Core 5.

RaspberryP and RaspberryPI Zero testrig

I have included sample application to show how to use the library

namespace devMobile.IoT.NetCore.GroveBaseHat
{
	using System;
	using System.Device.I2c;
	using System.Threading;

	class Program
	{
		static void Main(string[] args)
		{
			// bus id on the raspberry pi 3
			const int busId = 1;

			I2cConnectionSettings i2cConnectionSettings = new(busId, AnalogPorts.DefaultI2cAddress);

			using (I2cDevice i2cDevice = I2cDevice.Create(i2cConnectionSettings))
			using (AnalogPorts AnalogPorts = new AnalogPorts(i2cDevice))
			{
				Console.WriteLine($"{DateTime.Now:HH:mm:SS} Version:{AnalogPorts.Version()}");
				Console.WriteLine();

				double powerSupplyVoltage = AnalogPorts.PowerSupplyVoltage();
				Console.WriteLine($"{DateTime.Now:HH:mm:SS} Power Supply Voltage:{powerSupplyVoltage:F2}v");

				while (true)
				{
					double value = AnalogPorts.Read(AnalogPorts.AnalogPort.A0);
					double rawValue = AnalogPorts.ReadRaw(AnalogPorts.AnalogPort.A0);
					double voltageValue = AnalogPorts.ReadVoltage(AnalogPorts.AnalogPort.A0);

					Console.WriteLine($"{DateTime.Now:HH:mm:SS} Value:{value:F2} Raw:{rawValue:F2} Voltage:{voltageValue:F2}v");
					Console.WriteLine();

					Thread.Sleep(1000);
				}
			}
		}
	}
}

The GROVE_BASE_HAT_RPI and GROVE_BASE_HAT_RPI_ZERO are used to specify the number of available analog ports.

Azure HTTP Trigger Functions with .NET Core 5

My updated The Things Industries(TTI) connector will use a number of Azure Functions to process Application Integration webhooks (with HTTP Triggers) and Azure Storage Queue messages(with Output Bindings & QueueTriggers).

On a couple of customer projects we had been updating Azure Functions from .NET 4.X to .NET Core 3.1, and most recently .NET Core 5. This process has been surprisingly painful so I decided to build a series of small proof of concept (PoC) projects to explore the problem.

Visual Studio Azure Function Trigger type selector

I started with the Visual Studio 2019 Azure Function template and created a plain HTTPTrigger.

public static class Function1
{
   [Function("Function1")]
   public static HttpResponseData Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req,
      FunctionContext executionContext)
   {
      var logger = executionContext.GetLogger("Function1");
      logger.LogInformation("C# HTTP trigger function processed a request.");

      var response = req.CreateResponse(HttpStatusCode.OK);
      response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

      response.WriteString("Welcome to Azure Functions!");

      return response;
   }
}

I changed the AuthorizationLevel to Anonymous to make testing in Azure with Telerik Fiddler easier

public static class Function1
{
	[Function("PlainAsync")]
	public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequestData request, FunctionContext executionContext)
	{
		var logger = executionContext.GetLogger("UplinkMessage");

		logger.LogInformation("C# HTTP trigger function processed a request.");

		var response = request.CreateResponse(HttpStatusCode.OK);

		response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

		response.WriteString("Welcome to Azure Functions!");

		return new OkResult();
	}
}

With not a lot of work I had an Azure Function I could run in the Visual Studio debugger

Azure Functions Debug Diagnostic Output

I could invoke the function using the endpoint displayed as debugging environment started.

Telerik Fiddler Composer invoking Azure Function running locally

I then added more projects to explore asynchronicity, and output bindings

Azure Functions Solution PoC Projects

After a bit of “trial and error” I had an HTTPTrigger Function that inserted a message containing the payload of an HTTP POST into an Azure Storage Queue.

[StorageAccount("AzureWebJobsStorage")]
public static class Function1
{
	[Function("Uplink")]
	public static async Task<HttpTriggerUplinkOutputBindingType> Uplink([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext context)
	{
		var logger = context.GetLogger("UplinkMessage");

		logger.LogInformation("Uplink processed");
			
		var response = req.CreateResponse(HttpStatusCode.OK);

		return new HttpTriggerUplinkOutputBindingType()
		{
			Name = await req.ReadAsStringAsync(),
			HttpReponse = response
		};
	}

	public class HttpTriggerUplinkOutputBindingType
	{
		[QueueOutput("uplink")]
		public string Name { get; set; }

		public HttpResponseData HttpReponse { get; set; }
	}
}

The key was Multiple Output Bindings so the function could return a result for both the HttpResponseData and Azure Storage Queue operations

Azure Functions Debug Diagnostic Output

After getting the function running locally I deployed it to a Function App running in an App Service plan

Azure HTTP Trigger function Host Key configuration

Using the Azure Portal I configured an x-functions-key which I could use in Telerik Fiddler

After fixing an accidental truncation of the x-functions-key a message with the body of the POST was created in the Azure Storage Queue.

Azure Storage Queue Message containing HTTP Post Payload

The aim of this series of PoCs was to have an Azure function that securely (x-functions-key) processed an Hyper Text Transfer Protocol(HTTP) POST with an HTTPTrigger and inserted a message containing the payload into an Azure Storage Queue using an OutputBinding.

Use the contents of this blog post with care as it may not age well.

.NET Core RAK811 LoRaWAN library Part3

The massive refactor

After refactoring my RAK3172 device library I have applied a similar approach to code on my RAK811 device library. My test-rig is a RaspberryPI 3B with a PI Supply RAK811 pHat and external antenna.

PI Supply RAK811 LoRaWAN pHat

In the new code a Thread reads lines of text from the SerialPort and processes them, checking for command responses, failures and downlink messages.

Unlike most of the devices I have worked with the RAK811 Join and Send commands are synchronous so return once the process has completed. The RAK811 responses also have quite a few empty, null prefixed or null suffixed lines which is a bit odd.

public void SerialPortProcessor()
{
	string line;

	while (CommandProcessResponses)
	{
		try
		{
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine before");
#endif
			line = SerialDevice.ReadLine().Trim('\0').Trim();
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine after:{line}");
#endif
			// consume empty lines
			if (String.IsNullOrWhiteSpace(line))
			{
				continue;
			}

			// Consume the response from set work mode
			if (line.StartsWith("?LoRa (R)") || line.StartsWith("RAK811 ") || line.StartsWith("UART1 ") || line.StartsWith("UART3 ") || line.StartsWith("LoRa work mode"))
			{
				continue;
			}

			// See if device successfully joined network
			if (line.StartsWith("OK Join Success"))
			{
				OnJoinCompletion?.Invoke(true);

				CommandResponseExpectedEvent.Set();

				continue;
			}

			if (line.StartsWith("at+recv="))
			{
				string[] payloadFields = line.Split("=,:".ToCharArray());

				byte port = byte.Parse(payloadFields[1]);
				int rssi = int.Parse(payloadFields[2]);
				int snr = int.Parse(payloadFields[3]);
				int length = int.Parse(payloadFields[4]);

				if (this.OnMessageConfirmation != null)
				{
					OnMessageConfirmation?.Invoke(rssi, snr);
				}
				if (length > 0)
				{
					string payload = payloadFields[5];

					if (this.OnReceiveMessage != null)
					{
						OnReceiveMessage.Invoke(port, rssi, snr, payload);
					}
				}
				continue;
			}

			switch (line)
			{
				case "OK":
				case "Initialization OK":
				case "OK Wake Up":
				case "OK Sleep":
					CommandResult = Result.Success;
					break;

				case "ERROR: 1":
					CommandResult = Result.ATCommandUnsuported;
					break;
				case "ERROR: 2":
					CommandResult = Result.ATCommandInvalidParameter;
					break;
				case "ERROR: 3": //There is an error when reading or writing flash.
				case "ERROR: 4": //There is an error when reading or writing through IIC.
					CommandResult = Result.ErrorReadingOrWritingFlash;
					break;
				case "ERROR: 5": //There is an error when sending through UART
					CommandResult = Result.ATCommandInvalidParameter;
					break;
				case "ERROR: 41": //The BLE works in an invalid state, so that it can’t be operated.
					CommandResult = Result.ResponseInvalid;
					break;
				case "ERROR: 80":
					CommandResult = Result.LoRaBusy;
					break;
				case "ERROR: 81":
					CommandResult = Result.LoRaServiceIsUnknown;
					break;
				case "ERROR: 82":
					CommandResult = Result.LoRaParameterInvalid;
					break;
				case "ERROR: 83":
					CommandResult = Result.LoRaFrequencyInvalid;
					break;
				case "ERROR: 84":
					CommandResult = Result.LoRaDataRateInvalid;
					break;
				case "ERROR: 85":
					CommandResult = Result.LoRaFrequencyAndDataRateInvalid;
					break;
				case "ERROR: 86":
					CommandResult = Result.LoRaDeviceNotJoinedNetwork;
					break;
				case "ERROR: 87":
					CommandResult = Result.LoRaPacketToLong;
					break;
				case "ERROR: 88":
					CommandResult = Result.LoRaServiceIsClosedByServer;
					break;
				case "ERROR: 89":
					CommandResult = Result.LoRaRegionUnsupported;
					break;
				case "ERROR: 90":
					CommandResult = Result.LoRaDutyCycleRestricted;
					break;
				case "ERROR: 91":
					CommandResult = Result.LoRaNoValidChannelFound;
					break;
				case "ERROR: 92":
					CommandResult = Result.LoRaNoFreeChannelFound;
					break;
				case "ERROR: 93":
					CommandResult = Result.StatusIsError;
					break;
				case "ERROR: 94":
					CommandResult = Result.LoRaTransmitTimeout;
					break;
				case "ERROR: 95":
					CommandResult = Result.LoRaRX1Timeout;
					break;
				case "ERROR: 96":
					CommandResult = Result.LoRaRX2Timeout;
					break;
				case "ERROR: 97":
					CommandResult = Result.LoRaRX1ReceiveError;
					break;
				case "ERROR: 98":
					CommandResult = Result.LoRaRX2ReceiveError;
					break;
				case "ERROR: 99":
					CommandResult = Result.LoRaJoinFailed;
					break;
				case "ERROR: 100":
					CommandResult = Result.LoRaDownlinkRepeated;
					break;
				case "ERROR: 101":
					CommandResult = Result.LoRaPayloadSizeNotValidForDataRate;
					break;
				case "ERROR: 102":
					CommandResult = Result.LoRaTooManyDownlinkFramesLost;
					break;
				case "ERROR: 103":
					CommandResult = Result.LoRaAddressFail;
					break;
				case "ERROR: 104":
					CommandResult = Result.LoRaMicVerifyError;
					break;
				default:
					CommandResult = Result.ResponseInvalid;
					break;
			}
		}
		catch (TimeoutException)
		{
			// Intentionally ignored, not certain this is a good idea
		}

		CommandResponseExpectedEvent.Set();
	}
}

After a lot of testing I think my thread based approach works reliably. Initially, I was having some signal strength issues because I had forgotten to configure the external antenna. I need to add some validation to the metrics and payload field unpacking (though I’m not certain what todo if they are the wrong format).

RAK3172LoRaWAN-NetCore on Github

The source code for a late beta version of my .Net Core C# library for RAK3172 modules is now available on GitHub.

Using SerialDataReceivedEventHandler was causing memory management problems so the core command processor now runs in its own Thread.(Though I worry about the continuous System.TimeoutExceptions)

A sample application which shows how to connect using Over the Air Activation(OTAA) or Activation By Personalisation(ABP) then send and receive byte array/hexadecimal messages.

//---------------------------------------------------------------------------------
// Copyright (c) September 2021, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// Must have one of following options defined in the project\build definitions
//    PAYLOAD_BCD or PAYLOAD_BYTES
//    OTAA or ABP
//
// Optional definitions
//    CONFIRMED For confirmed messages
//		DEVEUI_SET
//
//---------------------------------------------------------------------------------
namespace devMobile.IoT.LoRaWAN.NetCore.RAK3172
{
	using System;
	using System.IO.Ports;
	using System.Threading;


	public class Program
	{
		private const string SerialPortId = "/dev/ttyS0";
		private const LoRaClass Class = LoRaClass.A;
		private const string Band = "8-1";
		private const byte MessagePort = 10;
		private static readonly TimeSpan MessageSendTimerDue = new TimeSpan(0, 0, 15);
		private static readonly TimeSpan MessageSendTimerPeriod = new TimeSpan(0, 5, 0);
		private static Timer MessageSendTimer ;
		private const int JoinRetryAttempts = 2;
		private const int JoinRetryIntervalSeconds = 10;
#if PAYLOAD_BCD
		private const string PayloadBcd = "48656c6c6f204c6f526157414e"; // Hello LoRaWAN in BCD
#endif
#if PAYLOAD_BYTES
		private static readonly byte[] PayloadBytes = { 0x48, 0x65 , 0x6c, 0x6c, 0x6f, 0x20, 0x4c, 0x6f, 0x52, 0x61, 0x57, 0x41, 0x4e}; // Hello LoRaWAN in bytes
#endif

		public static void Main()
		{
			Result result;

			Console.WriteLine("devMobile.IoT.LoRaWAN.NetCore.RAK3172 RAK3712LoRaWANDeviceClient starting");

			Console.WriteLine($"Serial ports:{String.Join(",", SerialPort.GetPortNames())}");

			try
			{
				using (Rak3172LoRaWanDevice device = new Rak3172LoRaWanDevice())
				{
					result = device.Initialise(SerialPortId, 9600, Parity.None, 8, StopBits.One);
					if (result != Result.Success)
					{
						Console.WriteLine($"Initialise failed {result}");
						return;
					}

					MessageSendTimer = new Timer(SendMessageTimerCallback, device,Timeout.Infinite, Timeout.Infinite);

					device.OnJoinCompletion += OnJoinCompletionHandler;
					device.OnReceiveMessage += OnReceiveMessageHandler;
#if CONFIRMED
					device.OnMessageConfirmation += OnMessageConfirmationHandler;
#endif

#if DEVEUI_SET
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} DevEUI {Config.devEui}");
					result = device.DeviceEui(Config.devEui);
					if (result != Result.Success)
					{
						Console.WriteLine($"DevEUI failed {result}");
						return;
					}
#endif

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Class {Class}");
					result = device.Class(Class);
					if (result != Result.Success)
					{
						Console.WriteLine($"Class failed {result}");
						return;
					}

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Band {Band}");
					result = device.Band(Band);
					if (result != Result.Success)
					{
						Console.WriteLine($"Region failed {result}");
						return;
					}

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} ADR On");
					result = device.AdrOn();
					if (result != Result.Success)
					{
						Console.WriteLine($"ADR on failed {result}");
						return;
					}

#if CONFIRMED
               Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Confirmed");
               result = device.UplinkMessageConfirmationOn();
               if (result != Result.Success)
               {
                  Console.WriteLine($"Confirm on failed {result}");
                  return;
               }
#else
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Unconfirmed");
					result = device.UplinkMessageConfirmationOff();
					if (result != Result.Success)
					{
						Console.WriteLine($"Confirm off failed {result}");
						return;
					}
#endif

#if OTAA
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} OTAA");
					result = device.OtaaInitialise(Config.JoinEui, Config.AppKey);
					if (result != Result.Success)
					{
						Console.WriteLine($"OTAA Initialise failed {result}");
						return;
					}
#endif

#if ABP
               Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} ABP");
               result = device.AbpInitialise(Config.DevAddress, Config.NwksKey, Config.AppsKey);
               if (result != Result.Success)
               {
                  Console.WriteLine($"ABP Initialise failed {result}");
                  return;
               }
#endif

					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Join start");
					result = device.Join(JoinRetryAttempts, JoinRetryIntervalSeconds);
					if (result != Result.Success)
					{
						Console.WriteLine($"Join failed {result}");
						return;
					}
					Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Join started");

					Thread.Sleep(Timeout.Infinite);
				}
			}
			catch (Exception ex)
			{
				Console.WriteLine(ex.Message);
			}
		}

		private static void OnJoinCompletionHandler(bool result)
		{
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Join finished:{result}");

			if (result)
			{ 
				MessageSendTimer.Change(MessageSendTimerDue, MessageSendTimerPeriod);
			}
		}

		private static void SendMessageTimerCallback(object state)
		{
			Rak3172LoRaWanDevice device = (Rak3172LoRaWanDevice)state;

#if PAYLOAD_BCD
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} port:{MessagePort} payload BCD:{PayloadBcd}");
			Result result = device.Send(MessagePort, PayloadBcd );
#endif
#if PAYLOAD_BYTES
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} port:{MessagePort} payload bytes:{Rak3172LoRaWanDevice.BytesToBcd(PayloadBytes)}");
         Result result = device.Send(MessagePort, PayloadBytes);
#endif
			if (result != Result.Success)
			{
				Console.WriteLine($"Send failed {result}");
			}
		}

#if CONFIRMED
		private static void OnMessageConfirmationHandler()
      {
			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Send successful");
		}
#endif

		private static void OnReceiveMessageHandler(byte port, int rssi, int snr, string payload)
		{
			byte[] payloadBytes = Rak3172LoRaWanDevice.HexToByes(payload); // Done this way so both conversion methods tested

			Console.WriteLine($"{DateTime.UtcNow:hh:mm:ss} Receive Message RSSI:{rssi} SNR:{snr} Port:{port} Payload:{payload} PayLoadBytes:{BitConverter.ToString(payloadBytes)}");
		}
	}
}

I have added XML Documentation comments which will need some rework, after I have “soak tested” the code for at least a week.

I have also added a method so the DevEUI can be set (intended for use after device firmware has been updated), fixed up my mistake with Binary Coded Decimal(BCD) vs. Hexadecimal strings.

I will also go back and apply the “learnings” from this refactoring to my other LoRaWAN module and platform libraries

.NET Core RAK3172 LoRaWAN library Part5

The massive refactor

After getting Activation By Personalisation(ABP) and Over The Air Activation(OTAA) working on my RAK3172 test rig I was looking at the code and SerialDataReceivedEventHandler was really ugly.

Raspberry Pi3 with Grove Base Hat and RAK3172 Breakout (using UART2)

After some experimentation in the BreakOutSerial project I decided to reimplement the RAK3172 command processing. In the new code a Thread reads lines of text from the SerialPort and processes them. I have replaced the Join and Send(Confirmed) methods with ones that block only while the command are sent to the RAK3172. Then, when completed the OnJoinCompletion or OnMessagesConfirmation event handlers are called.

private Result SendCommand(string command)
{
	if (command == null)
	{
		throw new ArgumentNullException(nameof(command));
	}

	if (command == string.Empty)
	{
		throw new ArgumentException($"command cannot be empty", nameof(command));
	}

	serialDevice.WriteLine(command);

	this.CommandResponseExpectedEvent.Reset();

	if (!this.CommandResponseExpectedEvent.WaitOne(CommandTimeoutDefaultmSec, false))
	{
		return Result.Timeout;
	}

	return CommandResult;
}

private void SerialPortProcessor()
{
	string line;

	while (CommandProcessResponses)
	{
		try
		{
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine before");
#endif
			line = serialDevice.ReadLine();
#if DIAGNOSTICS
			Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine after:{line}");
#endif

			// See if device successfully joined network
			if (line.StartsWith("+EVT:JOINED"))
			{
				OnJoinCompletion?.Invoke(true);

				continue;
			}

			// See if device failed ot join network
			if (line.StartsWith("+EVT:JOIN FAILED"))
			{
				OnJoinCompletion?.Invoke(false);

				continue;
			}

			// Applicable only if confirmed messages enabled 
			if (line.StartsWith("+EVT:SEND CONFIRMED OK"))
			{
				OnMessageConfirmation?.Invoke();

				continue;
			}

			// Check for A/B/C downlink message
			if (line.StartsWith("+EVT:RX_1") || line.StartsWith("+EVT:RX_2") || line.StartsWith("+EVT:RX_3") || line.StartsWith("+EVT:RX_C"))
			{
				// TODO beef up validation, nto certain what todo if borked
				string[] metricsFields= line.Split(' ', ',');

				int rssi = int.Parse(metricsFields[3]);
				int snr = int.Parse(metricsFields[6]);

				line = serialDevice.ReadLine();

#if DIAGNOSTICS
				Debug.WriteLine($" {DateTime.UtcNow:HH:mm:ss} UNICAST :{line}");
#endif
				line = serialDevice.ReadLine();
#if DIAGNOSTICS
				Debug.WriteLine($" {DateTime.UtcNow:HH:mm:ss} Payload:{line}");
#endif
				// TODO beef up validation, nto certain what todo if borked
				string[] payloadFields = line.Split(':');

				byte port = byte.Parse(payloadFields[1]);
				string payload = payloadFields[2];

				OnReceiveMessage?.Invoke(port, rssi, snr, payload);

				continue;
			}

#if DIAGNOSTICS
           Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine Result");
#endif
			line = serialDevice.ReadLine();
#if DIAGNOSTICS
             Debug.WriteLine($" {DateTime.UtcNow:hh:mm:ss} ReadLine Result:{line}");
#endif
			switch (line)
			{
				case "OK":
					CommandResult = Result.Success;
					break;
				case "AT_ERROR":
					CommandResult = Result.AtError;
					break;
				case "AT_PARAM_ERROR":
					CommandResult = Result.ParameterError;
					break;
				case "AT_BUSY_ERROR":
					CommandResult = Result.BusyError;
					break;
				case "AT_TEST_PARAM_OVERFLOW":
					CommandResult = Result.ParameterOverflow;
					break;
				case "AT_NO_NETWORK_JOINED":
					CommandResult = Result.NotJoined;
					break;
				case "AT_RX_ERROR":
					CommandResult = Result.ReceiveError;
					break;
				case "AT_DUTYCYLE_RESTRICTED":
					CommandResult = Result.DutyCycleRestricted;
					break;
				default:
					CommandResult = Result.Undefined;
					break;
			}

			CommandResponseExpectedEvent.Set();
		}
		catch (TimeoutException)
		{
			// Intentionally ignored, not certain this is a good idea
		}
	}
}

After a lot of testing I think my thread based approach works reliably. I also had to modify the code to shutdown the command processor thread and free any non managed resources.

/// <summary>
/// Ensures unmanaged serial port and thread resources are released in a "responsible" manner.
/// </summary>
public void Dispose()
{
	CommandProcessResponses = false;

	if (CommandResponsesProcessorThread != null)
	{
		CommandResponsesProcessorThread.Join();
		CommandResponsesProcessorThread = null;
	}

	if (serialDevice != null)
	{
		serialDevice.Dispose();
		serialDevice = null;
	}
}

I need to add some validation to the metrics and payload field unpacking (though I’m not certain what todo if they are the wrong format) and review the handling of multi-line event messages.