Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
This post revisits a previous post “Don’t forget to flush” Application insights and shows how to configure the instrumentation key in code or via the ApplicationInsights.config file.
class Program
{
static void Main(string[] args)
{
#if INSTRUMENTATION_KEY_TELEMETRY_CONFIGURATION
if (args.Length != 1)
{
Console.WriteLine("Usage AzureApplicationInsightsClientConsole <instrumentationKey>");
return;
}
TelemetryConfiguration telemetryConfiguration = new TelemetryConfiguration(args[0]);
TelemetryClient telemetryClient = new TelemetryClient(telemetryConfiguration);
telemetryClient.TrackTrace("INSTRUMENTATION_KEY_TELEMETRY_CONFIGURATION", SeverityLevel.Information);
#endif
#if INSTRUMENTATION_KEY_APPLICATION_INSIGHTS_CONFIG
TelemetryClient telemetryClient = new TelemetryClient();
telemetryClient.TrackTrace("INSTRUMENTATION_KEY_APPLICATION_INSIGHTS_CONFIG", SeverityLevel.Information);
#endif
telemetryClient.TrackTrace("This is an AI API Verbose message", SeverityLevel.Verbose);
telemetryClient.TrackTrace("This is an AI API Information message", SeverityLevel.Information);
telemetryClient.TrackTrace("This is an AI API Warning message", SeverityLevel.Warning);
telemetryClient.TrackTrace("This is an AI API Error message", SeverityLevel.Error);
telemetryClient.TrackTrace("This is an AI API Critical message", SeverityLevel.Critical);
telemetryClient.Flush();
Console.WriteLine("Press <enter> to exit");
Console.ReadLine();
}
Application Insights logging with message unpackingApplication Insights logging message payload
Then in the last log entry the decoded message payload
/*
Copyright ® 2020 Feb devMobile Software, All Rights Reserved
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
Default URL for triggering event grid function in the local environment.
http://localhost:7071/runtime/webhooks/EventGrid?functionName=functionname
*/
namespace EventGridProcessorAzureIotHub
{
using System;
using System.IO;
using System.Reflection;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.EventGrid.Models;
using Microsoft.Azure.WebJobs.Extensions.EventGrid;
using log4net;
using log4net.Config;
using Newtonsoft.Json;
public static class Telemetry
{
[FunctionName("Telemetry")]
public static void Run([EventGridTrigger]Microsoft.Azure.EventGrid.Models.EventGridEvent eventGridEvent, ExecutionContext executionContext )//, TelemetryClient telemetryClient)
{
ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
var logRepository = LogManager.GetRepository(Assembly.GetEntryAssembly());
XmlConfigurator.Configure(logRepository, new FileInfo(Path.Combine(executionContext.FunctionAppDirectory, "log4net.config")));
log.Info($"eventGridEvent.Data-{eventGridEvent}");
log.Info($"eventGridEvent.Data.ToString()-{eventGridEvent.Data.ToString()}");
IotHubDeviceTelemetryEventData iOThubDeviceTelemetryEventData = (IotHubDeviceTelemetryEventData)JsonConvert.DeserializeObject(eventGridEvent.Data.ToString(), typeof(IotHubDeviceTelemetryEventData));
log.Info($"iOThubDeviceTelemetryEventData.Body.ToString()-{iOThubDeviceTelemetryEventData.Body.ToString()}");
byte[] base64EncodedBytes = System.Convert.FromBase64String(iOThubDeviceTelemetryEventData.Body.ToString());
log.Info($"System.Text.Encoding.UTF8.GetString(-{System.Text.Encoding.UTF8.GetString(base64EncodedBytes)}");
}
}
}
Overall it took roughly half a page of code (mainly generated by a tool) to unpack and log the contents of an Azure IoT Hub EventGrid payload to Application Insights.
I have one an Azure IoT HubLoRa Telemetry Field Gateway running in my office and I wanted to process the data collected by the sensors around my property without using a Software as a Service(SaaS) Internet of Things (IoT) package.
Rather than lots of screen grabs of my configuration steps I figured people reading this series of posts would be able to figure the details out themselves.
I downloaded the JSON configuration file template from my Windows 10 device (which is created on first startup after installation) and configured the Azure IoT Hub connection string.
I then uploaded this to my Windows 10 IoT Core device and restarted the Azure IoT Hub Field gateway so it picked up the new settings.
I could then see on the device messages from sensor nodes being unpacked and uploaded to my Azure IoT Hub.
ETW logging on device
In the Azure IoT Hub metrics I graphed the number of devices connected and the number of telemetry messages sent and could see my device connect then start uploading telemetry.
Azure IoT Hub metrics
One of my customers uses Azure Event Grid for application integration and I wanted to explore using it in an IoT solution. The first step was to create an Event Grid Domain.
To confirm my event subscriptions were successful I previously found the “simplest” approach was to use an Azure storage queue endpoint. I had to create an Azure Storage Account with two Azure Storage Queues one for device connectivity (.DeviceConnected & .DeviceDisconnected) events and the other for device telemetry (.DeviceTelemetry) events.
I created a couple of other subscriptions so I could compare the different Event schemas (Event Grid Schema & Cloud Event Schema v1.0). At this stage I didn’t configure any Filters or Additional Features.
Azure IoT Hub Telemetry Event Metrics
I use Cerebrate Cerculean for monitoring and managing a couple of other customer projects so I used it to inspect the messages in the storage queues.
Without writing any code (I will script the configuration) I could upload sensor data to an Azure IoT Hub, subscribe to a selection of events the Azure IoT Hub publishes and then inspect them in an Azure Storage Queue.
I did notice that the .DeviceConnected and .DeviceDisconnected events did take a while to arrive. When I started the field gateway application on the device I would get several DeviceTelemetry events before the DeviceConnected event arrived.
I need to test the expiry of my SAS Tokens some more especially with the client running on my development machine (NZT which is currently UTC+13) and in Azure (UTC timezone)
I now have a working Azure IoT Hub plug-in (Azure IoT Central support as planned as well) with the first iteration focused on Device to Cloud (D2C) messaging. In a future iteration I will add Cloud to Device messaging(C2D).
My applications use a lightweight, easy to implemented protocol which is intended for hobbyist and educational use rather than commercial applications (I have been working on a more secure version as yet another side project)
When the application is first started it creates a minimal configuration file which should be downloaded, the missing information filled out, then uploaded using the File explorer in the Windows device portal.
The application logs debugging information to the Windows 10 IoT Core ETW logging Microsoft-Windows-Diagnostics-LoggingChannel
MQTT LoRa Gateway with Azure IoT Hub plug-in
The message handler uploads all values in an inbound messages in one MQTT message.
namespace devMobile.Mqtt.IoTCore.FieldGateway
{
using System;
using System.Diagnostics;
using System.Text;
using Windows.Foundation.Diagnostics;
using devMobile.IoT.Rfm9x;
using MQTTnet;
using MQTTnet.Client;
using Newtonsoft.Json.Linq;
using Newtonsoft.Json;
public class MessageHandler : IMessageHandler
{
private LoggingChannel Logging { get; set; }
private IMqttClient MqttClient { get; set; }
private Rfm9XDevice Rfm9XDevice { get; set; }
private string PlatformSpecificConfiguration { get; set; }
void IMessageHandler.Initialise(LoggingChannel logging, IMqttClient mqttClient, Rfm9XDevice rfm9XDevice, string platformSpecificConfiguration)
{
LoggingFields processInitialiseLoggingFields = new LoggingFields();
this.Logging = logging;
this.MqttClient = mqttClient;
this.Rfm9XDevice = rfm9XDevice;
this.PlatformSpecificConfiguration = platformSpecificConfiguration;
}
async void IMessageHandler.Rfm9XOnReceive(Rfm9XDevice.OnDataReceivedEventArgs e)
{
LoggingFields processReceiveLoggingFields = new LoggingFields();
char[] sensorReadingSeparators = { ',' };
char[] sensorIdAndValueSeparators = { ' ' };
processReceiveLoggingFields.AddString("PacketSNR", e.PacketSnr.ToString("F1"));
processReceiveLoggingFields.AddInt32("PacketRSSI", e.PacketRssi);
processReceiveLoggingFields.AddInt32("RSSI", e.Rssi);
string addressBcdText = BitConverter.ToString(e.Address);
processReceiveLoggingFields.AddInt32("DeviceAddressLength", e.Address.Length);
processReceiveLoggingFields.AddString("DeviceAddressBCD", addressBcdText);
string messageText;
try
{
messageText = UTF8Encoding.UTF8.GetString(e.Data);
processReceiveLoggingFields.AddString("MessageText", messageText);
}
catch (Exception ex)
{
processReceiveLoggingFields.AddString("Exception", ex.ToString());
this.Logging.LogEvent("PayloadProcess failure converting payload to text", processReceiveLoggingFields, LoggingLevel.Warning);
return;
}
// Chop up the CSV text
string[] sensorReadings = messageText.Split(sensorReadingSeparators, StringSplitOptions.RemoveEmptyEntries);
if (sensorReadings.Length < 1)
{
this.Logging.LogEvent("PayloadProcess payload contains no sensor readings", processReceiveLoggingFields, LoggingLevel.Warning);
return;
}
JObject payloadJObject = new JObject();
JObject feeds = new JObject();
// Chop up each sensor read into an ID & value
foreach (string sensorReading in sensorReadings)
{
string[] sensorIdAndValue = sensorReading.Split(sensorIdAndValueSeparators, StringSplitOptions.RemoveEmptyEntries);
// Check that there is an id & value
if (sensorIdAndValue.Length != 2)
{
this.Logging.LogEvent("PayloadProcess payload invalid format", processReceiveLoggingFields, LoggingLevel.Warning);
return;
}
string sensorId = string.Concat(addressBcdText, sensorIdAndValue[0]);
string value = sensorIdAndValue[1];
feeds.Add(sensorId.ToLower(), value);
}
payloadJObject.Add("feeds", feeds);
string topic = $"devices/{MqttClient.Options.ClientId}/messages/events/";
try
{
var message = new MqttApplicationMessageBuilder()
.WithTopic(topic)
.WithPayload(JsonConvert.SerializeObject(payloadJObject))
.WithAtLeastOnceQoS()
.Build();
Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync start", DateTime.UtcNow);
await MqttClient.PublishAsync(message);
Debug.WriteLine(" {0:HH:mm:ss} MQTT Client PublishAsync finish", DateTime.UtcNow);
this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Information);
}
catch (Exception ex)
{
processReceiveLoggingFields.AddString("Exception", ex.ToString());
this.Logging.LogEvent("PublishAsync Azure IoTHub payload", processReceiveLoggingFields, LoggingLevel.Error);
}
}
void IMessageHandler.MqttApplicationMessageReceived(MqttApplicationMessageReceivedEventArgs e)
{
LoggingFields processReceiveLoggingFields = new LoggingFields();
processReceiveLoggingFields.AddString("ClientId", e.ClientId);
#if DEBUG
processReceiveLoggingFields.AddString("Payload", e.ApplicationMessage.ConvertPayloadToString());
#endif
processReceiveLoggingFields.AddString("QualityOfServiceLevel", e.ApplicationMessage.QualityOfServiceLevel.ToString());
processReceiveLoggingFields.AddBoolean("Retain", e.ApplicationMessage.Retain);
processReceiveLoggingFields.AddString("Topic", e.ApplicationMessage.Topic);
this.Logging.LogEvent("MqttApplicationMessageReceived topic not processed", processReceiveLoggingFields, LoggingLevel.Error);
}
void IMessageHandler.Rfm9xOnTransmit(Rfm9XDevice.OnDataTransmitedEventArgs e)
{
}
}
}
The formatting of the username and generation of password are password are a bit awkward and will be fixed in a future refactoring. Along with regenerating the SAS connection token just before it is due to expire.
This post has been edited (2019-11-24) my original assumption about how DateTime.Kind unspecified was handled were incorrect.
As I was testing my Azure MQTT Test Client I noticed some oddness with MQTT connection timeouts and this got me wondering about token expiry times. So, I went searching again and found thisAzure IoT Hub specific sample code
This code worked first time and was more flexible than mine which was a bonus. Though while running my MQTTNet based client I noticed the connection would drop after approximately 10mins (EDIT this was probably an unrelated networking issue).
A long time ago (25 years) I had issues sharing a Unix time value between an applications written with Borland C and Microsoft Visual C which made me wonder about Unix epoch base offsets.
So to test my theory I built a Unix epoch test harness console application
EDIT: I now think the UtcNow to “unspecified” kind mathematics was being handled correctly. I have updated the code to use the DateTime.UnixEpoch constant so the code is more readable.
I need to test the expiry of my SAS Tokens some more especially with the client running on my development machine (NZT which is currently UTC+13) and in Azure (UTC timezone)
A long time ago I wrote a post about uploading telemetry data to an Azure Event Hub from a Netduino 3 Wifi using HTTPS. To send messages to the EventHub I had to create a valid SAS Token which took a surprising amount of effort because of the reduced text encoding/decoding and cryptographic functionality available in .NET Micro Framework v4.3 (NetMF)
// Create a SAS token for a specified scope. SAS tokens are described in http://msdn.microsoft.com/en-us/library/windowsazure/dn170477.aspx.
private static string CreateSasToken(string uri, string keyName, string key)
{
// Set token lifetime to 20 minutes. When supplying a device with a token, you might want to use a longer expiration time.
uint tokenExpirationTime = GetExpiry(20 * 60);
string stringToSign = HttpUtility.UrlEncode(uri) + "\n" + tokenExpirationTime;
var hmac = SHA.computeHMAC_SHA256(Encoding.UTF8.GetBytes(key), Encoding.UTF8.GetBytes(stringToSign));
string signature = Convert.ToBase64String(hmac);
signature = Base64NetMf42ToRfc4648(signature);
string token = "SharedAccessSignature sr=" + HttpUtility.UrlEncode(uri) + "&sig=" + HttpUtility.UrlEncode(signature) + "&se=" + tokenExpirationTime.ToString() + "&skn=" + keyName;
return token;
}
private static string Base64NetMf42ToRfc4648(string base64netMf)
{
var base64Rfc = string.Empty;
for (var i = 0; i < base64netMf.Length; i++)
{
if (base64netMf[i] == '!')
{
base64Rfc += '+';
}
else if (base64netMf[i] == '*')
{
base64Rfc += '/';
}
else
{
base64Rfc += base64netMf[i];
}
}
return base64Rfc;
}
static uint GetExpiry(uint tokenLifetimeInSeconds)
{
const long ticksPerSecond = 1000000000 / 100; // 1 tick = 100 nano seconds
DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0);
TimeSpan diff = DateTime.Now.ToUniversalTime() - origin;
return ((uint)(diff.Ticks / ticksPerSecond)) + tokenLifetimeInSeconds;
}
An initial search lead to this article about how to generate a SAS token for an Azure Event Hub in multiple languages. For my first attempt I “copied and paste” the code sample for C# (I also wasn’t certain what to put in the KeyName parameter) and it didn’t work.
The shared SAS token now looked closer to what I was expecting but the MQTTNet ConnectAsync was failing with an authentication exception. After looking at the Device Explorer SAS Key code, my .NetMF implementation and the code for the IoT Hub SDK I noticed the encoding for the HMAC Key was different. Encoding.UTF8.GetBytes vs. Convert.FromBase64String.
As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.
This PoC was to confirm that my device could connect to the Microsoft Azure IoT HubMQTT API then format topics and payloads correctly.
Azure IoT Hub MQTT Console Client
I had tried with a couple of different MQTT libraries from micro controllers and embedded devices without success. With the benefit of hindsight (plus this article) I think I had the SAS key format wrong.
The Azure IoT Hub MQTT broker requires only a server name (fully resolved CName), device ID and SAS Key.
Overall the initial configuration went smoothly after I figured out the required Quality of Service (QoS) settings, and the SAS Key format.
Using the approach described in the Microsoft documentation I manually generated the SAS Key.(In my Netduino samples I have code for generating a SAS Key in my HTTPSAzure IoT Hub Client)
Azure Device Explorer Device ManagementAzure Device Explorer SAS Key Generator
Once I had the configuration correct I could see telemetry from the device and send it messages.
Azure Device Explorer Data View
In a future post I will upload data to the Azure IoT Central for display. Then explore using a “module” attached to a device which maybe useful for my field gateway.
try
{
this.azureIoTHubClient = DeviceClient.CreateFromConnectionString(this.azureIoTHubConnectionString, this.transportType);
}
catch (Exception ex)
{
this.logging.LogMessage("AzureIOT Hub DeviceClient.CreateFromConnectionString failed " + ex.Message, LoggingLevel.Error);
return;
}
try
{
TwinCollection reportedProperties = new TwinCollection();
// This is from the OS
reportedProperties["Timezone"] = TimeZoneSettings.CurrentTimeZoneDisplayName;
reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
reportedProperties["MachineName"] = Environment.MachineName;
reportedProperties["ApplicationDisplayName"] = package.DisplayName;
reportedProperties["ApplicationName"] = packageId.Name;
reportedProperties["ApplicationVersion"] = string.Format($"{version.Major}.{version.Minor}.{version.Build}.{version.Revision}");
// Unique identifier from the hardware
SystemIdentificationInfo systemIdentificationInfo = SystemIdentification.GetSystemIdForPublisher();
using (DataReader reader = DataReader.FromBuffer(systemIdentificationInfo.Id))
{
byte[] bytes = new byte[systemIdentificationInfo.Id.Length];
reader.ReadBytes(bytes);
reportedProperties["SystemId"] = BitConverter.ToString(bytes);
}
this.azureIoTHubClient.UpdateReportedPropertiesAsync(reportedProperties).Wait();
}
catch (Exception ex)
{
this.logging.LogMessage("Azure IoT Hub client UpdateReportedPropertiesAsync failed " + ex.Message, LoggingLevel.Error);
return;
}
try
{
LoggingFields configurationInformation = new LoggingFields();
Twin deviceTwin = this.azureIoTHubClient.GetTwinAsync().GetAwaiter().GetResult();
if (!deviceTwin.Properties.Desired.Contains("ImageUpdateDue") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdateDue"].value.ToString(), out imageUpdateDue))
{
this.logging.LogMessage("DeviceTwin.Properties ImageUpdateDue setting missing or invalid format", LoggingLevel.Warning);
return;
}
configurationInformation.AddTimeSpan("ImageUpdateDue", imageUpdateDue);
if (!deviceTwin.Properties.Desired.Contains("ImageUpdatePeriod") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdatePeriod"].value.ToString(), out imageUpdatePeriod))
{
this.logging.LogMessage("DeviceTwin.Properties ImageUpdatePeriod setting missing or invalid format", LoggingLevel.Warning);
return;
}
…
if (!deviceTwin.Properties.Desired.Contains("DebounceTimeout") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["DebounceTimeout"].value.ToString(), out debounceTimeout))
{
this.logging.LogMessage("DeviceTwin.Properties DebounceTimeout setting missing or invalid format", LoggingLevel.Warning);
return;
}
configurationInformation.AddTimeSpan("DebounceTimeout", debounceTimeout);
this.logging.LogEvent("Configuration settings", configurationInformation);
}
catch (Exception ex)
{
this.logging.LogMessage("Azure IoT Hub client GetTwinAsync failed or property missing/invalid" + ex.Message, LoggingLevel.Error);
return;
}
When the digital input (configured in the app.settings file) is strobed or the timer fires (configured in the device properties) an image is captured, uploaded to Azure Cognitive Services Custom Vision for processing.
The returned results are then post processed to make them Azure IoT Central friendly, and finally uploaded to an Azure IoT Hub.
For testing I have used a simple object detection model.
I trained the model with images of 6 different colours of m&m’s.
For my first dataset I tagged the location of a single m&m of each of the colour in 15 images.
Testing the training of the model
I then trained the model multiple times adding additional images where the model was having trouble distiguishing colours.
The published name comes from the training performance tab
Project settings
The projectID, AzureCognitiveServicesSubscriptionKey (PredictionKey) and PublishedName (From the Performance tab in project) are from the custom vision project properties.
All of the Custom Vision model settings are configured in the Azure IoT Hub device properties.
The app.settings file contains only the hardware configuration settings and the Azure IoT Hub connection string.
The LED connected to the display pin is illuminated while an image is being processed or briefly flashed if the insufficient time between image captures has passed.
The image data is post processed differently based on the model.
// Post process the predictions based on the type of model
switch (modelType)
{
case ModelType.Classification:
// Use only the tags above the specified minimum probability
foreach (var prediction in imagePrediction.Predictions)
{
if (prediction.Probability >= probabilityThreshold)
{
// Display and log the individual tag probabilities
Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Probability:0.00}");
imageInformation.AddDouble($"Tag valid:{prediction.TagName}", prediction.Probability);
telemetryDataPoint.Add(prediction.TagName, prediction.Probability);
}
}
break;
case ModelType.Detection:
// Group the tags to get the count, include only the predictions above the specified minimum probability
var groupedPredictions = from prediction in imagePrediction.Predictions
where prediction.Probability >= probabilityThreshold
group prediction by new { prediction.TagName }
into newGroup
select new
{
TagName = newGroup.Key.TagName,
Count = newGroup.Count(),
};
// Display and log the agregated predictions
foreach (var prediction in groupedPredictions)
{
Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Count}");
imageInformation.AddInt32($"Tag valid:{prediction.TagName}", prediction.Count);
telemetryDataPoint.Add(prediction.TagName, prediction.Count);
}
break;
default:
throw new ArgumentException("ModelType Invalid");
}
For a classifier only the tags with a probability greater than or equal the specified threshold are uploaded.
For a detection model the instances of each tag are counted. Only the tags with a prediction value greater than the specified threshold are included in the count.
The debugging output of the application includes the different categories identified in the captured image.
I found my small model was pretty good at detection of individual m&m as long as the ambient lighting was consistent, and the background fairly plain.
Sample image from test rig
Every so often the camera contrast setting went bad and could only be restored by restarting the device which needs further investigation.
Image with contrast problem
This application could be the basis for projects which need to run an Azure Cognitive Services model to count or classify then upload the results to an Azure IoT Hub or Azure IoT Central for presentation.
With a suitable model this application could be used to count the number of people in a room, which could be displayed along with the ambient temperature, humidity, CO2, and noise levels in Azure IoT Central.
The code for this application is available In on GitHub.
This application was inspired by one of teachers I work with wanting to count ducks in the stream on the school grounds. The school was having problems with water quality and the they wanted to see if the number of ducks was a factor. (Manually counting the ducks several times a day would be impractical).
I didn’t have a source of training images so built an image classifier using my son’s Lego for testing. In a future post I will build an object detection model once I have some sample images of the stream captured by my Windows 10 IoT Core time lapse camera application.
Every time the digital input is strobed by the infra red proximity sensor or touch button an image is captured, uploaded for processing, and results displayed in the debug output.
For testing I have used a simple multiclass classifier that I trained with a selection of my son’s Lego. I tagged the brick size height x width x length (1x2x3, smallest of width/height first) and colour (red, green, blue etc.)
The projectID, AzureCognitiveServicesSubscriptionKey (PredictionKey) and PublishedName (From the Performance tab in project) in the app.settings file come from the custom vision project properties.
The sample application only supports one trigger tag + probability and if this condition satisfied the Light Emitting Diode (LED) is turned on for 5 seconds. If an image is being processed or the minimum period between images has not passed the LED is illuminated for 5 milliseconds .
private async void InterruptGpioPin_ValueChanged(GpioPin sender, GpioPinValueChangedEventArgs args)
{
DateTime currentTime = DateTime.UtcNow;
Debug.WriteLine($"Digital Input Interrupt {sender.PinNumber} triggered {args.Edge}");
if (args.Edge != this.interruptTriggerOn)
{
return;
}
// Check that enough time has passed for picture to be taken
if ((currentTime - this.imageLastCapturedAtUtc) < this.debounceTimeout)
{
this.displayGpioPin.Write(GpioPinValue.High);
this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
return;
}
this.imageLastCapturedAtUtc = currentTime;
// Just incase - stop code being called while photo already in progress
if (this.cameraBusy)
{
this.displayGpioPin.Write(GpioPinValue.High);
this.displayOffTimer.Change(this.timerPeriodDetectIlluminated, this.timerPeriodInfinite);
return;
}
this.cameraBusy = true;
try
{
using (Windows.Storage.Streams.InMemoryRandomAccessStream captureStream = new Windows.Storage.Streams.InMemoryRandomAccessStream())
{
this.mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), captureStream).AsTask().Wait();
captureStream.FlushAsync().AsTask().Wait();
captureStream.Seek(0);
IStorageFile photoFile = await KnownFolders.PicturesLibrary.CreateFileAsync(ImageFilename, CreationCollisionOption.ReplaceExisting);
ImageEncodingProperties imageProperties = ImageEncodingProperties.CreateJpeg();
await this.mediaCapture.CapturePhotoToStorageFileAsync(imageProperties, photoFile);
ImageAnalysis imageAnalysis = await this.computerVisionClient.AnalyzeImageInStreamAsync(captureStream.AsStreamForRead());
Debug.WriteLine($"Tag count {imageAnalysis.Categories.Count}");
if (imageAnalysis.Categories.Intersect(this.categoryList, new CategoryComparer()).Any())
{
this.displayGpioPin.Write(GpioPinValue.High);
// Start the timer to turn the LED off
this.displayOffTimer.Change(this.timerPeriodFaceIlluminated, this.timerPeriodInfinite);
}
LoggingFields imageInformation = new LoggingFields();
imageInformation.AddDateTime("TakenAtUTC", currentTime);
imageInformation.AddInt32("Pin", sender.PinNumber);
Debug.WriteLine($"Categories:{imageAnalysis.Categories.Count}");
imageInformation.AddInt32("Categories", imageAnalysis.Categories.Count);
foreach (Category category in imageAnalysis.Categories)
{
Debug.WriteLine($" Category:{category.Name} {category.Score}");
imageInformation.AddDouble($"Category:{category.Name}", category.Score);
}
this.logging.LogEvent("Captured image processed by Cognitive Services", imageInformation);
}
}
catch (Exception ex)
{
this.logging.LogMessage("Camera photo or save failed " + ex.Message, LoggingLevel.Error);
}
finally
{
this.cameraBusy = false;
}
}
private void TimerCallback(object state)
{
this.displayGpioPin.Write(GpioPinValue.Low);
}
internal class CategoryComparer : IEqualityComparer<Category>
{
public bool Equals(Category x, Category y)
{
if (string.Equals(x.Name, y.Name, StringComparison.OrdinalIgnoreCase))
{
return true;
}
return false;
}
public int GetHashCode(Category obj)
{
return obj.Name.GetHashCode();
}
}
I found my small model was pretty good at tagging images of Lego bricks as long as the ambient lighting was consistent and the background fairly plain.
When tagging many bricks my ability to distinguish pearl light grey, light grey, sand blue and grey bricks was a problem. I should have started with a limited palette (red, green, blue) of colours and shapes for my models while evaluating different tagging approaches.
The debugging output of the application includes the different categories identified in the captured image.
I’m going to run this application repeatedly, adding more images and retraining the model to see how it performs. Once the model is working wll I’ll try downloading it and running it on a device
Custom Vision Test Harness running on my desk
This sample could be used as a basis for projects like this cat door which stops your pet bringing in dead or wounded animals. The model could be trained with tags to indicate whether the cat is carrying a “present” for their human and locking the door if it is.