ubidots MQTT LoRa Field Gateway

Back in April I started working on an MQTT LoRa Field gateway which was going to support a selection of different Software as a service(SaaS) Internet of Things(IoT) platforms.

After a long pause in development I have a working ubidots client and have 3 proof of concept (PoC) integrations for Adafruit.IO, AskSensors, and Losant. I am also working on Azure IoT Hub, Azure IoT Central and MyDevice Cayenne clients. The first iteration is focused on Device to Cloud (D2C) messaging in the next iteration I will add Cloud to Device where viable(C2D).

My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)

I have a number of sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield etc. clients. These work with both my platform specific (Adafruit.IO, Azure IoT Central) gateways and protocol specific field gateways.

Ubidots dashboard

When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.

{
  "MQTTUserName": "Ubidots generated usname here",
  "MQTTPassword": "NotVerySecure",
  "MQTTClientID": "MQTTLoRaGateway",
  "MQTTServer": "industrial.api.ubidots.com",
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0,
  "MessageHandlerAssembly": "Mqtt.IoTCore.FieldGateway.LoRa.Ubidots",
  "PlatformSpecificConfiguration": ""
}

The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel

MQTT LoRa Field Gateway with ubidots plugin generated telemetry
ubidots device management
ubidot managment

The message handler uploads all values in an inbound messages in one MQTT message using the ubidots MQTT message format

async void IMessageHandler.Rfm9XOnReceive(object sender, Rfm9XDevice.OnDataReceivedEventArgs e)
{
	LoggingFields processReceiveLoggingFields = new LoggingFields();
	JObject telemetryDataPoint = new JObject();
	char[] sensorReadingSeparators = { ',' };
	char[] sensorIdAndValueSeparators = { ' ' };

	processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
	processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
	processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);

	string addressBcdText = BitConverter.ToString(e.Address);
	processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
	processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);

	string messageText;
	try
	{
		messageText = UTF8Encoding.UTF8.GetString(e.Data);
		processReceiveLoggingFields.AddString("MessageText", messageText);
	}
	catch (Exception ex)
	{
		processReceiveLoggingFields.AddString("Exception", ex.ToString());
		this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
		return;
	}

	// Chop up the CSV text
	string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
	if (sensorReadings.Length < 1)
	{
		this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
		return;
	}

	// Chop up each sensor read into an ID & value
	foreach (string sensorReading in sensorReadings)
	{
		string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);
		// Check that there is an id & value
		if (sensorIdAndValue.Length != 2)
		{
			this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
			return;
		}

		string sensorId = sensorIdAndValue[0];
		string value = sensorIdAndValue[1];

		telemetryDataPoint.Add(addressBcdText + sensorId, Convert.ToDouble(value));
	}
	processReceiveLoggingFields.AddString("MQTTClientId", MqttClient.Options.ClientId);

	string stateTopic = string.Format(stateTopicFormat, MqttClient.Options.ClientId);

	try
	{
		var message = new MqttApplicationMessageBuilder()
			.WithTopic(stateTopic)
			.WithPayload(JsonConvert.SerializeObject(telemetryDataPoint))
			.WithAtLeastOnceQoS()
			.Build();
		Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
		await MqttClient.PublishAsync(message);
		Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);

		this.Logging.LogEvent("PublishAsync Ubidots payload", processReceiveLoggingFields, LoggingLevel.Information);
	}
	catch (Exception ex)
	{
		processReceiveLoggingFields.AddString("Exception", ex.ToString());
		this.Logging.LogEvent("PublishAsync Ubidots payload", processReceiveLoggingFields, LoggingLevel.Error);
	}
}

The “automagic” provisioning of feeds does make setting up small scale systems easier, though I’m not certain how well it would scale.

Some of the fields weren’t obviously editable e.g.”ÄPI Label” in device configuration which I only discovered by clicking on them..

The limitations of the free account meant I couldn’t evaluate ubidots in much depth but what was available appeared to be robust and reliable (Nov 2019).

Adafruit MQTT Cloud to Device Messaging

After getting MQ Telemetry Transport (MQTT) Device to Cloud (D2C) messaging working for AdaFruit.IO I have also got Cloud to Device (C2D) messaging working as well.

The MQTT broker, username, API key, client ID, optional group name (to keep MQTT aligned with REST API terminology), command topic and feed name are command line options.

The Adafruit IO MQTT documentation suggests an approach for naming topics which allows a bit more structure for feed (D2C and C2D) names than the REST API (which only does D2C).

class Program
{
	private static IMqttClient mqttClient = null;
	private static IMqttClientOptions mqttOptions = null;
	private static string server;
	private static string username;
	private static string password;
	private static string clientId;
	private static string commandTopic;
	private static string groupname;
	private static string feedname;

	static void Main(string[] args)
	{
		MqttFactory factory = new MqttFactory();
		mqttClient = factory.CreateMqttClient();

		if ((args.Length != 6) && (args.Length != 7))
		{
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID] [CommandTopic] [GroupName] [FeedName]");
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID] [CommandTopic] [FeedName]");
			Console.WriteLine("Press <enter> to exit");
			Console.ReadLine();
			return;
		}

		server = args[0];
		username = args[1];
		password = args[2];
		clientId = args[3];
		commandTopic = args[4];
		if (args.Length == 6)
		{
			feedname = args[5].ToLower();
			Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId} CommandTopic:{commandTopic} Feedname:{feedname}");
		}

		if (args.Length == 7)
		{
			groupname = args[5].ToLower();
			feedname = args[6].ToLower();
			Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId} CommandTopic:{commandTopic} Groupname:{groupname} Feedname:{feedname}");
		}

		mqttOptions = new MqttClientOptionsBuilder()
			.WithTcpServer(server)
			.WithCredentials(username, password)
			.WithClientId(clientId)
			.WithTls()
			.Build();

		mqttClient.Disconnected += MqttClient_Disconnected;
		mqttClient.ConnectAsync(mqttOptions).Wait();
		mqttClient.ApplicationMessageReceived += MqttClient_ApplicationMessageReceived;

		// Adafruit.IO format for topics which are called feeds
		string topic = string.Empty;

		if (args.Length == 6)
		{
			topic = $"{username}/feeds/{feedname}";				
		}

		if (args.Length == 7)
		{
			topic = $"{username}/feeds/{groupname}.{feedname}";
		}

		mqttClient.SubscribeAsync(commandTopic, MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce).GetAwaiter().GetResult();

		while (true)
		{
			string value = "22." + DateTime.UtcNow.Millisecond.ToString();
			Console.WriteLine($"Topic:{topic} Value:{value}");

			var message = new MqttApplicationMessageBuilder()
				.WithTopic(topic)
				.WithPayload(value)
				.WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
			.WithExactlyOnceQoS()
			.WithRetainFlag()
			.Build();

			Console.WriteLine("PublishAsync start");
			mqttClient.PublishAsync(message).Wait();
			Console.WriteLine("PublishAsync finish");

			Thread.Sleep(30100);
		}
	}

	private static void MqttClient_ApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e)
	{
		Console.WriteLine($"ClientId:{e.ClientId} Topic:{e.ApplicationMessage.Topic} Payload:{e.ApplicationMessage.ConvertPayloadToString()}");
	}

	private static async void MqttClient_Disconnected(object sender, MqttClientDisconnectedEventArgs e)
	{
		Debug.WriteLine("Disconnected");
		await Task.Delay(TimeSpan.FromSeconds(5));

		try
		{
			await mqttClient.ConnectAsync(mqttOptions);
		}
		catch (Exception ex)
		{
			Debug.WriteLine("Reconnect failed {0}", ex.Message);
		}
	}
}

I configured a slider on the dashboard for my home called “setpoint” (yet again I was tripped up “automatically” camel casing the name because I’m a C# developer) which my MQTT client subscribed to.

AdaFruit.IO Home monitoring dashboard
setpoint feed configuration

After figuring out the format of the command topic I found that when the slider was moved the MQTT client subscription event fired reliably.

AdaFruit Client showing the setpoint value change notifications

Overall the process went pretty well, though the manual configuration of the subscriptions to AdaFruit.IO feeds could become a bit of a problem at scale.

Ask Sensors with MQTTnet

After a 6 month pause I’m back working on my Message Queue Telemetry Transport(MQTT) LoRa gateway.

As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.

This PoC was to confirm that I could connect to the Ask Sensors MQTT API then format topics and payloads correctly.

Console test application

The MQTT broker, username, password, and client ID are command line options.

class Program
{
	private static IMqttClient mqttClient = null;
	private static IMqttClientOptions mqttOptions = null;
	private static string server;
	private static string username;
	private static string apiKey;
	private static string clientID;

	static void Main(string[] args)
	{
		MqttFactory factory = new MqttFactory();
		mqttClient = factory.CreateMqttClient();
		bool heatPumpOn = false;

		if (args.Length != 4)
		{
			Console.WriteLine("[MQTT Server] [UserName] [APIKey] [ClientID]");
			Console.WriteLine("Press <enter> to exit");
			Console.ReadLine();
			return;
		}

		server = args[0];
		username = args[1];
		apiKey = args[2];
		clientID = args[3];

		Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientID}");

		mqttOptions = new MqttClientOptionsBuilder()
			.WithTcpServer(server)
			.WithCredentials(username, "")
			.WithClientId(clientID)
			//.WithTls() // This is a bit of a worry
			.Build();

		mqttClient.ApplicationMessageReceived += MqttClient_ApplicationMessageReceived;
		mqttClient.Disconnected += MqttClient_Disconnected;
		mqttClient.ConnectAsync(mqttOptions).Wait();

		// AskSensors formatted client state update topic
		string stateTopic = $"{username}/{apiKey}";

		while (true)
		{
			string payloadText;
			double temperature = 22.0 + (DateTime.UtcNow.Millisecond / 1000.0);
			double humidity = 50 + (DateTime.UtcNow.Millisecond / 100.0);
			double speed = 10 + (DateTime.UtcNow.Millisecond / 100.0);
			Console.WriteLine($"Topic:{stateTopic} Temperature:{temperature:0.00} Humidity:{humidity:0} HeatPumpOn:{heatPumpOn}");

			// First JSON attempt didn't work
			payloadText = @"{""Humidity"":55}";

            // Second attempt worked
            payloadText = $"module1=22";

            // Third attempt with "real" values injected
            payloadText = $"module1={temperature}&m2={humidity}";

            var message = new MqttApplicationMessageBuilder()
					.WithTopic(stateTopic)
					.WithPayload(payloadText)
					.WithQualityOfServiceLevel(global::MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
				   .WithExactlyOnceQoS()
				   //.WithAtLeastOnceQoS()
				   //.WithRetainFlag()
				   .Build();

				Console.WriteLine("PublishAsync start");
				mqttClient.PublishAsync(message).Wait();
				Console.WriteLine("PublishAsync finish");

				Thread.Sleep(30100);
			}
		}

	private static async void MqttClient_Disconnected(object sender, MqttClientDisconnectedEventArgs e)
	{
		Debug.WriteLine("Disconnected");
		await Task.Delay(TimeSpan.FromSeconds(5));

		try
		{
			await mqttClient.ConnectAsync(mqttOptions);
		}
		catch (Exception ex)
		{
			Debug.WriteLine("Reconnect failed {0}", ex.Message);
		}
	}

The Ask Sensors screen designer has 8 different types of Graph (line, bar, scatter, gauge, table, binary. digital, map)

Ask sensors dashboard configuration was fairly simple, though sequential numbering of inputs (modules) might require some mapping code.

Overall the initial configuration went smoothly after I figured out the payload format (not JSON), though the functionality (of a free subscription) did appear to be quite limited.

Since I first started building my MQTT gateway there have been several breaking updates to the MQTTNet API which so I will have to refresh all the applications in my solution.

Windows 10 IoT Core Cognitive Services Azure IoT Hub Client

This application builds on Windows 10 IoT Core Cognitive Services Vision API client. It uses my Lego brick classifier model and a new m&m object detection model.

m&m counter test rig

I created a new Visual Studio 2017 Windows IoT Core project and copied across the Windows 10 IoT Core Cognitive Services Custom Vision API code, (changing the namespace and manifest details) and added the Azure Devices Client NuGet package.

Azure Devices Client NuGet

In the start up code I added code to initialise the Azure IoT Hub client, retrieve the device twin settings, and update the device twin properties.

try
{
	this.azureIoTHubClient = DeviceClient.CreateFromConnectionString(this.azureIoTHubConnectionString, this.transportType);
}
catch (Exception ex)
{
	this.logging.LogMessage("AzureIOT Hub DeviceClient.CreateFromConnectionString failed " + ex.Message, LoggingLevel.Error);
	return;
}

try
{
	TwinCollection reportedProperties = new TwinCollection();

	// This is from the OS
	reportedProperties["Timezone"] = TimeZoneSettings.CurrentTimeZoneDisplayName;
	reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
	reportedProperties["MachineName"] = Environment.MachineName;

	reportedProperties["ApplicationDisplayName"] = package.DisplayName;
	reportedProperties["ApplicationName"] = packageId.Name;
	reportedProperties["ApplicationVersion"] = string.Format($"{version.Major}.{version.Minor}.{version.Build}.{version.Revision}");

	// Unique identifier from the hardware
	SystemIdentificationInfo systemIdentificationInfo = SystemIdentification.GetSystemIdForPublisher();
	using (DataReader reader = DataReader.FromBuffer(systemIdentificationInfo.Id))
	{
		byte[] bytes = new byte[systemIdentificationInfo.Id.Length];
		reader.ReadBytes(bytes);
		reportedProperties["SystemId"] = BitConverter.ToString(bytes);
	}
	this.azureIoTHubClient.UpdateReportedPropertiesAsync(reportedProperties).Wait();
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure IoT Hub client UpdateReportedPropertiesAsync failed " + ex.Message, LoggingLevel.Error);
	return;
}

try
{
	LoggingFields configurationInformation = new LoggingFields();

	Twin deviceTwin = this.azureIoTHubClient.GetTwinAsync().GetAwaiter().GetResult();

	if (!deviceTwin.Properties.Desired.Contains("ImageUpdateDue") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdateDue"].value.ToString(), out imageUpdateDue))
	{
		this.logging.LogMessage("DeviceTwin.Properties ImageUpdateDue setting missing or invalid format", LoggingLevel.Warning);
		return;
	}
	configurationInformation.AddTimeSpan("ImageUpdateDue", imageUpdateDue);

	if (!deviceTwin.Properties.Desired.Contains("ImageUpdatePeriod") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdatePeriod"].value.ToString(), out imageUpdatePeriod))
	{
		this.logging.LogMessage("DeviceTwin.Properties ImageUpdatePeriod setting missing or invalid format", LoggingLevel.Warning);
		return;
	}
…
	if (!deviceTwin.Properties.Desired.Contains("DebounceTimeout") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["DebounceTimeout"].value.ToString(), out debounceTimeout))
	{
		this.logging.LogMessage("DeviceTwin.Properties DebounceTimeout setting missing or invalid format", LoggingLevel.Warning);
		return;
	}
				configurationInformation.AddTimeSpan("DebounceTimeout", debounceTimeout);

	this.logging.LogEvent("Configuration settings", configurationInformation);
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure IoT Hub client GetTwinAsync failed or property missing/invalid" + ex.Message, LoggingLevel.Error);
	return;
}

When the digital input (configured in the app.settings file) is strobed or the timer fires (configured in the device properties) an image is captured, uploaded to Azure Cognitive Services Custom Vision for processing.

The returned results are then post processed to make them Azure IoT Central friendly, and finally uploaded to an Azure IoT Hub.

For testing I have used a simple object detection model.

I trained the model with images of 6 different colours of m&m’s.

For my first dataset I tagged the location of a single m&m of each of the colour in 15 images.

Testing the training of the model

I then trained the model multiple times adding additional images where the model was having trouble distiguishing colours.

The published name comes from the training performance tab

Project settings

The projectID, AzureCognitiveServicesSubscriptionKey (PredictionKey) and PublishedName (From the Performance tab in project) are from the custom vision project properties.

All of the Custom Vision model settings are configured in the Azure IoT Hub device properties.

The app.settings file contains only the hardware configuration settings and the Azure IoT Hub connection string.

{
  "InterruptPinNumber": 24,
  "interruptTriggerOn": "RisingEdge",
  "DisplayPinNumber": 35,
  "AzureIoTHubConnectionString": "",
  "TransportType": "Mqtt"
} 

The LED connected to the display pin is illuminated while an image is being processed or briefly flashed if the insufficient time between image captures has passed.

The image data is post processed differently based on the model.

// Post process the predictions based on the type of model
switch (modelType)
{
	case ModelType.Classification:
		// Use only the tags above the specified minimum probability
		foreach (var prediction in imagePrediction.Predictions)
		{
			if (prediction.Probability >= probabilityThreshold)
			{
				// Display and log the individual tag probabilities
				Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Probability:0.00}");
				imageInformation.AddDouble($"Tag valid:{prediction.TagName}", prediction.Probability);
					telemetryDataPoint.Add(prediction.TagName, prediction.Probability);
			}
		}
		break;

	case ModelType.Detection:
		// Group the tags to get the count, include only the predictions above the specified minimum probability
		var groupedPredictions = from prediction in imagePrediction.Predictions
										 where prediction.Probability >= probabilityThreshold
										 group prediction by new { prediction.TagName }
				into newGroup
										 select new
										 {
											 TagName = newGroup.Key.TagName,
											 Count = newGroup.Count(),
										 };

		// Display and log the agregated predictions
		foreach (var prediction in groupedPredictions)
		{
			Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Count}");
			imageInformation.AddInt32($"Tag valid:{prediction.TagName}", prediction.Count);
			telemetryDataPoint.Add(prediction.TagName, prediction.Count);
		}
		break;
	default:
		throw new ArgumentException("ModelType Invalid");
}

For a classifier only the tags with a probability greater than or equal the specified threshold are uploaded.

For a detection model the instances of each tag are counted. Only the tags with a prediction value greater than the specified threshold are included in the count.

19-08-14 05:26:14 Timer triggered
Prediction count 33
 Tag:Blue 0.0146500813
 Tag:Blue 0.61186564
 Tag:Blue 0.0923164859
 Tag:Blue 0.7813785
 Tag:Brown 0.0100603029
 Tag:Brown 0.128318727
 Tag:Brown 0.0135991769
 Tag:Brown 0.687322736
 Tag:Brown 0.846672833
 Tag:Brown 0.1826635
 Tag:Brown 0.0183384717
 Tag:Green 0.0200069249
 Tag:Green 0.367765248
 Tag:Green 0.011428359
 Tag:Orange 0.678825438
 Tag:Orange 0.03718319
 Tag:Orange 0.8643157
 Tag:Orange 0.0296728313
 Tag:Red 0.02141669
 Tag:Red 0.7183208
 Tag:Red 0.0183610674
 Tag:Red 0.0130951973
 Tag:Red 0.82097
 Tag:Red 0.0618815944
 Tag:Red 0.0130757084
 Tag:Yellow 0.04150853
 Tag:Yellow 0.0106579047
 Tag:Yellow 0.0210028365
 Tag:Yellow 0.03392527
 Tag:Yellow 0.129197285
 Tag:Yellow 0.8089519
 Tag:Yellow 0.03723789
 Tag:Yellow 0.74729687
 Tag valid:Blue 2
 Tag valid:Brown 2
 Tag valid:Orange 2
 Tag valid:Red 2
 Tag valid:Yellow 2
 05:26:17 AzureIoTHubClient SendEventAsync start
 05:26:18 AzureIoTHubClient SendEventAsync finish

The debugging output of the application includes the different categories identified in the captured image.

I found my small model was pretty good at detection of individual m&m as long as the ambient lighting was consistent, and the background fairly plain.

Sample image from test rig

Every so often the camera contrast setting went bad and could only be restored by restarting the device which needs further investigation.

Image with contrast problem

This application could be the basis for projects which need to run an Azure Cognitive Services model to count or classify then upload the results to an Azure IoT Hub or Azure IoT Central for presentation.

With a suitable model this application could be used to count the number of people in a room, which could be displayed along with the ambient temperature, humidity, CO2, and noise levels in Azure IoT Central.

The code for this application is available In on GitHub.

Windows 10 IoT Core Cognitive Services Custom Vision API

This application was inspired by one of teachers I work with wanting to count ducks in the stream on the school grounds. The school was having problems with water quality and the they wanted to see if the number of ducks was a factor. (Manually counting the ducks several times a day would be impractical).

I didn’t have a source of training images so built an image classifier using my son’s Lego for testing. In a future post I will build an object detection model once I have some sample images of the stream captured by my Windows 10 IoT Core time lapse camera application.

To start with I added the Azure Cognitive Services Custom Vision API NuGet packages to a new Visual Studio 2017 Windows IoT Core project.

Azure Custom Vision Service NuGet packacges

Then I initialised the Computer Vision API client

try
{
	this.customVisionClient = new CustomVisionPredictionClient(new System.Net.Http.DelegatingHandler[] { })
	{
		ApiKey = this.azureCognitiveServicesSubscriptionKey,
		Endpoint = this.azureCognitiveServicesEndpoint,
	};
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure Cognitive Services Custom Vision Client configuration failed " + ex.Message, LoggingLevel.Error);
	return;
}

Every time the digital input is strobed by the infra red proximity sensor or touch button an image is captured, uploaded for processing, and results displayed in the debug output.

For testing I have used a simple multiclass classifier that I trained with a selection of my son’s Lego. I tagged the brick size height x width x length (1x2x3, smallest of width/height first) and colour (red, green, blue etc.)

Azure Cognitive Services Classifier project creation
Custom vision projects
Lego classifier project properties

The projectID, AzureCognitiveServicesSubscriptionKey (PredictionKey) and PublishedName (From the Performance tab in project) in the app.settings file come from the custom vision project properties.

{
  "InterruptPinNumber": 24,
  "interruptTriggerOn": "RisingEdge",
  "DisplayPinNumber": 35,
  "AzureCognitiveServicesEndpoint": "https://australiaeast.api.cognitive.microsoft.com",
  "AzureCognitiveServicesSubscriptionKey": "41234567890123456789012345678901s,
  "DebounceTimeout": "00:00:30",
  "PublishedName": "LegoBrickClassifierV3",
  "TriggerTag": "1x2x4",
  "TriggerThreshold": "0.4",
  "ProjectID": "c1234567-abcdefghijklmn-1234567890ab"
} 

The sample application only supports one trigger tag + probability and if this condition satisfied the Light Emitting Diode (LED) is turned on for 5 seconds. If an image is being processed or the minimum period between images has not passed the LED is illuminated for 5 milliseconds .

private async void InterruptGpioPin_ValueChanged(GpioPin sender, GpioPinValueChangedEventArgs args)
{
	DateTime currentTime = DateTime.UtcNow;
	Debug.WriteLine($"Digital Input Interrupt {sender.PinNumber} triggered {args.Edge}");

	if (args.Edge != this.interruptTriggerOn)
	{
		return;
	}

	// Check that enough time has passed for picture to be taken
	if ((currentTime - this.imageLastCapturedAtUtc) < this.debounceTimeout)
	{
		this.displayGpioPin.Write(GpioPinValue.High);
		this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
		return;
	}

	this.imageLastCapturedAtUtc = currentTime;

	// Just incase - stop code being called while photo already in progress
	if (this.cameraBusy)
	{
		this.displayGpioPin.Write(GpioPinValue.High);
		this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
		return;
	}

	this.cameraBusy = true;

	try
	{
		using (Windows.Storage.Streams.InMemoryRandomAccessStream captureStream = new Windows.Storage.Streams.InMemoryRandomAccessStream())
		{
			this.mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), captureStream).AsTask().Wait();
			captureStream.FlushAsync().AsTask().Wait();
			captureStream.Seek(0);

			IStorageFile photoFile = await KnownFolders.PicturesLibrary.CreateFileAsync(ImageFilename, CreationCollisionOption.ReplaceExisting);
			ImageEncodingProperties imageProperties = ImageEncodingProperties.CreateJpeg();
			await this.mediaCapture.CapturePhotoToStorageFileAsync(imageProperties, photoFile);

			ImageAnalysis imageAnalysis = await this.computerVisionClient.AnalyzeImageInStreamAsync(captureStream.AsStreamForRead());

			Debug.WriteLine($"Tag count {imageAnalysis.Categories.Count}");

			if (imageAnalysis.Categories.Intersect(this.categoryList, new CategoryComparer()).Any())
			{
				this.displayGpioPin.Write(GpioPinValue.High);

				// Start the timer to turn the LED off
				this.displayOffTimer.Change(this.timerPeriodFaceIlluminated, this.timerPeriodInfinite);
					}

					LoggingFields imageInformation = new LoggingFields();

					imageInformation.AddDateTime("TakenAtUTC", currentTime);
					imageInformation.AddInt32("Pin", sender.PinNumber);
					Debug.WriteLine($"Categories:{imageAnalysis.Categories.Count}");
					imageInformation.AddInt32("Categories", imageAnalysis.Categories.Count);
					foreach (Category category in imageAnalysis.Categories)
					{
						Debug.WriteLine($" Category:{category.Name} {category.Score}");
						imageInformation.AddDouble($"Category:{category.Name}", category.Score);
					}

					this.logging.LogEvent("Captured image processed by Cognitive Services", imageInformation);
				}
			}
			catch (Exception ex)
			{
				this.logging.LogMessage("Camera photo or save failed " + ex.Message, LoggingLevel.Error);
			}
			finally
			{
				this.cameraBusy = false;
			}
		}

		private void TimerCallback(object state)
		{
			this.displayGpioPin.Write(GpioPinValue.Low);
		}

		internal class CategoryComparer : IEqualityComparer<Category>
		{
			public bool Equals(Category x, Category y)
			{
				if (string.Equals(x.Name, y.Name, StringComparison.OrdinalIgnoreCase))
				{
					return true;
				}

				return false;
			}

			public int GetHashCode(Category obj)
			{
				return obj.Name.GetHashCode();
			}
		}

I found my small model was pretty good at tagging images of Lego bricks as long as the ambient lighting was consistent and the background fairly plain.

When tagging many bricks my ability to distinguish pearl light grey, light grey, sand blue and grey bricks was a problem. I should have started with a limited palette (red, green, blue) of colours and shapes for my models while evaluating different tagging approaches.

The debugging output of the application includes the different categories identified in the captured image.

Digital Input Interrupt 24 triggered RisingEdge
Digital Input Interrupt 24 triggered FallingEdge
Prediction count 54
 Tag:Lime 0.529844046
 Tag:1x1x2 0.4441353
 Tag:Green 0.252290249
 Tag:1x1x3 0.1790101
 Tag:1x2x3 0.132092983
 Tag:Turquoise 0.128928885
 Tag:DarkGreen 0.09383947
 Tag:DarkTurquoise 0.08993266
 Tag:1x2x2 0.08145093
 Tag:1x2x4 0.060960535
 Tag:LightBlue 0.0525473
 Tag:MediumAzure 0.04958712
 Tag:Violet 0.04894981
 Tag:SandGreen 0.048463434
 Tag:LightOrange 0.044860106
 Tag:1X1X1 0.0426577441
 Tag:Azure 0.0416654423
 Tag:Aqua 0.0400410332
 Tag:OliveGreen 0.0387720577
 Tag:Blue 0.035169173
 Tag:White 0.03497391
 Tag:Pink 0.0321456343
 Tag:Transparent 0.0246597622
 Tag:MediumBlue 0.0245670844
 Tag:BrightPink 0.0223842952
 Tag:Flesh 0.0221406389
 Tag:Magenta 0.0208457354
 Tag:Purple 0.0188888311
 Tag:DarkPurple 0.0187285
 Tag:MaerskBlue 0.017609369
 Tag:DarkPink 0.0173041821
 Tag:Lavender 0.0162359159
 Tag:PearlLightGrey 0.0152829709
 Tag:1x1x4 0.0133710662
 Tag:Red 0.0122602312
 Tag:Yellow 0.0118704
 Tag:Clear 0.0114340987
 Tag:LightYellow 0.009903331
 Tag:Black 0.00877647
 Tag:BrightLightYellow 0.00871937349
 Tag:Mediumorange 0.0078356415
 Tag:Tan 0.00738664949
 Tag:Sand 0.00713921571
 Tag:Grey 0.00710422
 Tag:Orange 0.00624707434
 Tag:SandBlue 0.006215865
 Tag:DarkGrey 0.00613187673
 Tag:DarkBlue 0.00578308525
 Tag:DarkOrange 0.003790971
 Tag:DarkTan 0.00348462746
 Tag:LightGrey 0.00321317
 Tag:ReddishBrown 0.00304117263
 Tag:LightBluishGrey 0.00273489812
 Tag:Brown 0.00199119

I’m going to run this application repeatedly, adding more images and retraining the model to see how it performs. Once the model is working wll I’ll try downloading it and running it on a device

Custom Vision Test Harness running on my desk

This sample could be used as a basis for projects like this cat door which stops your pet bringing in dead or wounded animals. The model could be trained with tags to indicate whether the cat is carrying a “present” for their human and locking the door if it is.

Windows 10 IoT Core Cognitive Services Computer Vision API

This application was inspired by one of teachers I work with wanting to check occupancy of different areas in the school library. I had been using the Computer Vision service to try and identify objects around my home and office which had been moderately successful but not terribly useful or accurate.

I added the Azure Cognitive Services Computer Vision API NuGet packages to my Visual Studio 2017 Windows IoT Core project.

Azure Cognitive Services Computer Vision API library

Then I initialised the Computer Vision API client

try
{
	this.computerVisionClient = new ComputerVisionClient(
			 new Microsoft.Azure.CognitiveServices.Vision.ComputerVision.ApiKeyServiceClientCredentials(this.azureCognitiveServicesSubscriptionKey),
			 new System.Net.Http.DelegatingHandler[] { })
	{
		Endpoint = this.azureCognitiveServicesEndpoint,
	};
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure Cognitive Services Computer Vision client configuration failed " + ex.Message, LoggingLevel.Error);
	return;
}

Every time the digital input is strobed by the passive infra red motion detector an image is captured, then uploaded for processing, and finally results displayed. For this sample I’m looking for categories which indicate the image is of a group of people (The categories are configured in the appsettings file)

{
  "InterruptPinNumber": 24,
  "interruptTriggerOn": "RisingEdge",
  "DisplayPinNumber": 35,
  "AzureCognitiveServicesEndpoint": "https://australiaeast.api.cognitive.microsoft.com/",
  "AzureCognitiveServicesSubscriptionKey": "1234567890abcdefghijklmnopqrstuv",
  "ComputerVisionCategoryNames":"people_group,people_many",
  "LocalImageFilenameFormatLatest": "{0}.jpg",
  "LocalImageFilenameFormatHistoric": "{1:yyMMddHHmmss}.jpg",
  "DebounceTimeout": "00:00:30"
} 

If any of the specified categories are identified in the image I illuminate a Light Emitting Diode (LED) for 5 seconds, if an image is being processed or the minimum period between images has not passed the LED is illuminated for 5 milliseconds .

		private async void InterruptGpioPin_ValueChanged(GpioPin sender, GpioPinValueChangedEventArgs args)
		{
			DateTime currentTime = DateTime.UtcNow;
			Debug.WriteLine($"Digital Input Interrupt {sender.PinNumber} triggered {args.Edge}");

			if (args.Edge != this.interruptTriggerOn)
			{
				return;
			}

			// Check that enough time has passed for picture to be taken
			if ((currentTime - this.imageLastCapturedAtUtc) < this.debounceTimeout)
			{
				this.displayGpioPin.Write(GpioPinValue.High);
				this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
				return;
			}

			this.imageLastCapturedAtUtc = currentTime;

			// Just incase - stop code being called while photo already in progress
			if (this.cameraBusy)
			{
				this.displayGpioPin.Write(GpioPinValue.High);
				this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
				return;
			}

			this.cameraBusy = true;

			try
			{
				using (Windows.Storage.Streams.InMemoryRandomAccessStream captureStream = new Windows.Storage.Streams.InMemoryRandomAccessStream())
				{
					this.mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), captureStream).AsTask().Wait();
					captureStream.FlushAsync().AsTask().Wait();
					captureStream.Seek(0);

					IStorageFile photoFile = await KnownFolders.PicturesLibrary.CreateFileAsync(ImageFilename, CreationCollisionOption.ReplaceExisting);
					ImageEncodingProperties imageProperties = ImageEncodingProperties.CreateJpeg();
					await this.mediaCapture.CapturePhotoToStorageFileAsync(imageProperties, photoFile);

					ImageAnalysis imageAnalysis = await this.computerVisionClient.AnalyzeImageInStreamAsync(captureStream.AsStreamForRead());

					Debug.WriteLine($"Tag count {imageAnalysis.Categories.Count}");

					if (imageAnalysis.Categories.Intersect(this.categoryList, new CategoryComparer()).Any())
					{
						this.displayGpioPin.Write(GpioPinValue.High);

						// Start the timer to turn the LED off
						this.displayOffTimer.Change(this.timerPeriodFaceIlluminated, this.timerPeriodInfinite);
					}

					LoggingFields imageInformation = new LoggingFields();

					imageInformation.AddDateTime("TakenAtUTC", currentTime);
					imageInformation.AddInt32("Pin", sender.PinNumber);
					Debug.WriteLine($"Categories:{imageAnalysis.Categories.Count}");
					imageInformation.AddInt32("Categories", imageAnalysis.Categories.Count);
					foreach (Category category in imageAnalysis.Categories)
					{
						Debug.WriteLine($" Category:{category.Name} {category.Score}");
						imageInformation.AddDouble($"Category:{category.Name}", category.Score);
					}

					this.logging.LogEvent("Captured image processed by Cognitive Services", imageInformation);
				}
			}
			catch (Exception ex)
			{
				this.logging.LogMessage("Camera photo or save failed " + ex.Message, LoggingLevel.Error);
			}
			finally
			{
				this.cameraBusy = false;
			}
		}

		private void TimerCallback(object state)
		{
			this.displayGpioPin.Write(GpioPinValue.Low);
		}

		internal class CategoryComparer : IEqualityComparer<Category>
		{
			public bool Equals(Category x, Category y)
			{
				if (string.Equals(x.Name, y.Name, StringComparison.OrdinalIgnoreCase))
				{
					return true;
				}

				return false;
			}

			public int GetHashCode(Category obj)
			{
				return obj.Name.GetHashCode();
			}
		}

I found that the Computer vision service was pretty good at categorising photos of images like this displayed on my second monitor as containing a group of people.

The debugging output of the application includes the different categories identified in the captured image.

Digital Input Interrupt 24 triggered RisingEdge
Digital Input Interrupt 24 triggered FallingEdge
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Diagnostics.DiagnosticSource.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Collections.NonGeneric.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Runtime.Serialization.Formatters.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Diagnostics.TraceSource.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Collections.Specialized.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Drawing.Primitives.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Runtime.Serialization.Primitives.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Data.Common.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Xml.ReaderWriter.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Private.Xml.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'Anonymously Hosted DynamicMethods Assembly'. 
Tag count 1
Categories:1
 Category:people_group 0.8671875
The thread 0x634 has exited with code 0 (0x0).

I used an infrared motion sensor to trigger capture and processing of an image to simulate a application for detecting if there is a group of people in an area of the school library.

I’m going to run this application alongside one of my time-lapse applications to record a days worth of images and manually check the accuracy of the image categorisation. I think that camera location maybe important as well so I’ll try a selection of different USB cameras and locations.

Trial PIR triggered computer vision client

I also found the small PIR motion detector didn’t work very well in a larger space so I’m going to trial a configurable sensor and a repurposed burglar alarm sensor.

Windows 10 IoT Core Cognitive Services Face API

After building a series of Windows 10 IoT Core applications to capture images and store them

I figured some sample applications which used Azure Cognitive Services Vision Services to process captured images would be interesting.

This application was inspired by one of my students who has been looking at an Arduino based LoRa wireless connected sensor for monitoring Ultraviolet(UV) light levels and wanted to check that juniors at the school were wearing their hats on sunny days before going outside.

First I needed create a Cognitive Services instance and get the subscription key and endpoint.

Azure Cognitive Services Instance Creation

Then I added the Azure Cognitive Services Face API NuGet packages into my Visual Studio Windows IoT Core project

Azure Cognitive Services Vision Face API library

Then initialise the Face API client

try
{
	this.faceClient = new FaceClient(
			 new Microsoft.Azure.CognitiveServices.Vision.Face.ApiKeyServiceClientCredentials(this.azureCognitiveServicesSubscriptionKey),
											 new System.Net.Http.DelegatingHandler[] { })
	{
		Endpoint = this.azureCognitiveServicesEndpoint,
	};
}
catch (Exception ex)
{
	this.logging.LogMessage("Azure Cognitive Services Face Client configuration failed " + ex.Message, LoggingLevel.Error);
	return;
}

Then every time a digital input is strobed and image is captured, then uploaded for processing, and finally results displayed. The interrupt handler has code to stop re-entrancy and contactor bounce causing issues. I also requested that the Face service include age and gender attributes with associated confidence values.

If a face is found in the image I illuminate a Light Emitting Diode (LED) for 5 seconds, if an image is being processed or the minimum period between images has not passed the LED is illuminated for 5 milliseconds .

private async void InterruptGpioPin_ValueChanged(GpioPin sender, GpioPinValueChangedEventArgs args)
{
	DateTime currentTime = DateTime.UtcNow;
	Debug.WriteLine($"Digital Input Interrupt {sender.PinNumber} triggered {args.Edge}");

	if (args.Edge != this.interruptTriggerOn)
	{
		return;
	}

	// Check that enough time has passed for picture to be taken
	if ((currentTime - this.imageLastCapturedAtUtc) < this.debounceTimeout)
	{
		this.displayGpioPin.Write(GpioPinValue.High);
		this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
		return;
	}

	this.imageLastCapturedAtUtc = currentTime;

	// Just incase - stop code being called while photo already in progress
	if (this.cameraBusy)
	{
		this.displayGpioPin.Write(GpioPinValue.High);
		this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
		return;
	}

	this.cameraBusy = true;

	try
	{
		using (Windows.Storage.Streams.InMemoryRandomAccessStream captureStream = new Windows.Storage.Streams.InMemoryRandomAccessStream())
		{
			this.mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), captureStream).AsTask().Wait();
			captureStream.FlushAsync().AsTask().Wait();
			captureStream.Seek(0);
			IStorageFile photoFile = await KnownFolders.PicturesLibrary.CreateFileAsync(ImageFilename, CreationCollisionOption.ReplaceExisting);
			ImageEncodingProperties imageProperties = ImageEncodingProperties.CreateJpeg();
			await this.mediaCapture.CapturePhotoToStorageFileAsync(imageProperties, photoFile);

			IList<FaceAttributeType> returnfaceAttributes = new List<FaceAttributeType>();
			returnfaceAttributes.Add(FaceAttributeType.Gender);
			returnfaceAttributes.Add(FaceAttributeType.Age);

			IList<DetectedFace> detectedFaces = await this.faceClient.Face.DetectWithStreamAsync(captureStream.AsStreamForRead(), returnFaceAttributes: returnfaceAttributes);

			Debug.WriteLine($"Count {detectedFaces.Count}");

			if (detectedFaces.Count > 0)
			{
				this.displayGpioPin.Write(GpioPinValue.High);

						// Start the timer to turn the LED off
				this.displayOffTimer.Change(this.timerPeriodFaceIlluminated, this.timerPeriodInfinite);
			}

			LoggingFields imageInformation = new LoggingFields();
			imageInformation.AddDateTime("TakenAtUTC", currentTime);
			imageInformation.AddInt32("Pin", sender.PinNumber);
			imageInformation.AddInt32("Faces", detectedFaces.Count);
			foreach (DetectedFace detectedFace in detectedFaces)
			{
				Debug.WriteLine("Face");
				if (detectedFace.FaceId.HasValue)
				{
					imageInformation.AddGuid("FaceId", detectedFace.FaceId.Value);
					Debug.WriteLine($" Id:{detectedFace.FaceId.Value}");
				}
				imageInformation.AddInt32("Left", detectedFace.FaceRectangle.Left);
				imageInformation.AddInt32("Width", detectedFace.FaceRectangle.Width);
				imageInformation.AddInt32("Top", detectedFace.FaceRectangle.Top);
				imageInformation.AddInt32("Height", detectedFace.FaceRectangle.Height);
				Debug.WriteLine($" L:{detectedFace.FaceRectangle.Left} W:{detectedFace.FaceRectangle.Width} T:{detectedFace.FaceRectangle.Top} H:{detectedFace.FaceRectangle.Height}");
				if (detectedFace.FaceAttributes != null)
				{
					if (detectedFace.FaceAttributes.Gender.HasValue)
					{
						imageInformation.AddString("Gender", detectedFace.FaceAttributes.Gender.Value.ToString());
						Debug.WriteLine($" Gender:{detectedFace.FaceAttributes.Gender.ToString()}");
					}

					if (detectedFace.FaceAttributes.Age.HasValue)
					{
						imageInformation.AddDouble("Age", detectedFace.FaceAttributes.Age.Value);
						Debug.WriteLine($" Age:{detectedFace.FaceAttributes.Age.Value.ToString("F1")}");
					}
				}
			}

			this.logging.LogEvent("Captured image processed by Cognitive Services", imageInformation);
		}
	}
	catch (Exception ex)
	{
		this.logging.LogMessage("Camera photo or save failed " + ex.Message, LoggingLevel.Error);
	}
	finally
	{
		this.cameraBusy = false;
	}
}

private void TimerCallback(object state)
{
	this.displayGpioPin.Write(GpioPinValue.Low);
}

This is the image uploaded to the Cognitive Services Vision Face API from my DragonBoard 410C

Which was a photo of this sample image displayed on my second monitor

The debugging output of the application includes the bounding box, gender, age and unique identifier of each detected face.

Digital Input Interrupt 24 triggered RisingEdge
Digital Input Interrupt 24 triggered FallingEdge
Count 13
Face
 Id:41ab8a38-180e-4b63-ab47-d502b8534467
 L:12 W:51 T:129 H:51
 Gender:Female
 Age:24.0
Face
 Id:554f7557-2b78-4392-9c73-5e51fedf0300
 L:115 W:48 T:146 H:48
 Gender:Female
 Age:19.0
Face
 Id:f67ae4cc-1129-46a8-8c5b-0e79f350cbaa
 L:547 W:46 T:162 H:46
 Gender:Female
 Age:56.0
Face
 Id:fad453fb-0923-4ae2-8c9d-73c9d89eaaf4
 L:585 W:45 T:116 H:45
 Gender:Female
 Age:25.0
Face
 Id:c2d2ca4e-faa6-49e8-8cd9-8d21abfc374c
 L:410 W:44 T:154 H:44
 Gender:Female
 Age:23.0
Face
 Id:6fb75edb-654c-47ff-baf0-847a31d2fd85
 L:70 W:44 T:57 H:44
 Gender:Male
 Age:37.0
Face
 Id:d6c97a9a-c49f-4d9c-8eac-eb2fbc03abc1
 L:469 W:44 T:122 H:44
 Gender:Female
 Age:38.0
Face
 Id:e193bf15-6d8c-4c30-adb5-4ca5fb0f0271
 L:206 W:44 T:117 H:44
 Gender:Male
 Age:33.0
Face
 Id:d1ba5a42-0475-4b65-afc8-0651439e1f1e
 L:293 W:44 T:74 H:44
 Gender:Male
 Age:59.0
Face
 Id:b6a7c551-bdad-4e38-8976-923b568d2721
 L:282 W:43 T:144 H:43
 Gender:Female
 Age:28.0
Face
 Id:8be87f6d-7350-4bc3-87f5-3415894b8fac
 L:513 W:42 T:78 H:42
 Gender:Male
 Age:36.0
Face
 Id:e73bd4d7-81a4-403c-aa73-1408ae1068c0
 L:163 W:36 T:94 H:36
 Gender:Female
 Age:44.0
Face
 Id:462a6948-a05e-4fea-918d-23d8289e0401
 L:407 W:36 T:73 H:36
 Gender:Male
 Age:27.0
The thread 0x8e0 has exited with code 0 (0x0).

I used a simple infrared proximity sensor trigger the image capture to simulate an application for monitoring the number of people in or people entering a room.

Infrared Proximity Sensor triggered Face API test client

Overall I found that with not a lot of code I could capture an image, upload it to Azure Cognitive Services Face API for processing and the algorithm would reasonably reliably detect faces and features.