This application builds on Windows 10 IoT Core Cognitive Services Vision API client. It uses my Lego brick classifier model and a new m&m object detection model.
I created a new Visual Studio 2017 Windows IoT Core project and copied across the Windows 10 IoT Core Cognitive Services Custom Vision API code, (changing the namespace and manifest details) and added the Azure Devices Client NuGet package.
In the start up code I added code to initialise the Azure IoT Hub client, retrieve the device twin settings, and update the device twin properties.
try
{
this.azureIoTHubClient = DeviceClient.CreateFromConnectionString(this.azureIoTHubConnectionString, this.transportType);
}
catch (Exception ex)
{
this.logging.LogMessage("AzureIOT Hub DeviceClient.CreateFromConnectionString failed " + ex.Message, LoggingLevel.Error);
return;
}
try
{
TwinCollection reportedProperties = new TwinCollection();
// This is from the OS
reportedProperties["Timezone"] = TimeZoneSettings.CurrentTimeZoneDisplayName;
reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
reportedProperties["MachineName"] = Environment.MachineName;
reportedProperties["ApplicationDisplayName"] = package.DisplayName;
reportedProperties["ApplicationName"] = packageId.Name;
reportedProperties["ApplicationVersion"] = string.Format($"{version.Major}.{version.Minor}.{version.Build}.{version.Revision}");
// Unique identifier from the hardware
SystemIdentificationInfo systemIdentificationInfo = SystemIdentification.GetSystemIdForPublisher();
using (DataReader reader = DataReader.FromBuffer(systemIdentificationInfo.Id))
{
byte[] bytes = new byte[systemIdentificationInfo.Id.Length];
reader.ReadBytes(bytes);
reportedProperties["SystemId"] = BitConverter.ToString(bytes);
}
this.azureIoTHubClient.UpdateReportedPropertiesAsync(reportedProperties).Wait();
}
catch (Exception ex)
{
this.logging.LogMessage("Azure IoT Hub client UpdateReportedPropertiesAsync failed " + ex.Message, LoggingLevel.Error);
return;
}
try
{
LoggingFields configurationInformation = new LoggingFields();
Twin deviceTwin = this.azureIoTHubClient.GetTwinAsync().GetAwaiter().GetResult();
if (!deviceTwin.Properties.Desired.Contains("ImageUpdateDue") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdateDue"].value.ToString(), out imageUpdateDue))
{
this.logging.LogMessage("DeviceTwin.Properties ImageUpdateDue setting missing or invalid format", LoggingLevel.Warning);
return;
}
configurationInformation.AddTimeSpan("ImageUpdateDue", imageUpdateDue);
if (!deviceTwin.Properties.Desired.Contains("ImageUpdatePeriod") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["ImageUpdatePeriod"].value.ToString(), out imageUpdatePeriod))
{
this.logging.LogMessage("DeviceTwin.Properties ImageUpdatePeriod setting missing or invalid format", LoggingLevel.Warning);
return;
}
…
if (!deviceTwin.Properties.Desired.Contains("DebounceTimeout") || !TimeSpan.TryParse(deviceTwin.Properties.Desired["DebounceTimeout"].value.ToString(), out debounceTimeout))
{
this.logging.LogMessage("DeviceTwin.Properties DebounceTimeout setting missing or invalid format", LoggingLevel.Warning);
return;
}
configurationInformation.AddTimeSpan("DebounceTimeout", debounceTimeout);
this.logging.LogEvent("Configuration settings", configurationInformation);
}
catch (Exception ex)
{
this.logging.LogMessage("Azure IoT Hub client GetTwinAsync failed or property missing/invalid" + ex.Message, LoggingLevel.Error);
return;
}
When the digital input (configured in the app.settings file) is strobed or the timer fires (configured in the device properties) an image is captured, uploaded to Azure Cognitive Services Custom Vision for processing.
The returned results are then post processed to make them Azure IoT Central friendly, and finally uploaded to an Azure IoT Hub.
For testing I have used a simple object detection model.
I trained the model with images of 6 different colours of m&m’s.
For my first dataset I tagged the location of a single m&m of each of the colour in 15 images.
I then trained the model multiple times adding additional images where the model was having trouble distiguishing colours.
The published name comes from the training performance tab
The projectID, AzureCognitiveServicesSubscriptionKey (PredictionKey) and PublishedName (From the Performance tab in project) are from the custom vision project properties.
All of the Custom Vision model settings are configured in the Azure IoT Hub device properties.
The app.settings file contains only the hardware configuration settings and the Azure IoT Hub connection string.
{
"InterruptPinNumber": 24,
"interruptTriggerOn": "RisingEdge",
"DisplayPinNumber": 35,
"AzureIoTHubConnectionString": "",
"TransportType": "Mqtt"
}
The LED connected to the display pin is illuminated while an image is being processed or briefly flashed if the insufficient time between image captures has passed.
The image data is post processed differently based on the model.
// Post process the predictions based on the type of model
switch (modelType)
{
case ModelType.Classification:
// Use only the tags above the specified minimum probability
foreach (var prediction in imagePrediction.Predictions)
{
if (prediction.Probability >= probabilityThreshold)
{
// Display and log the individual tag probabilities
Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Probability:0.00}");
imageInformation.AddDouble($"Tag valid:{prediction.TagName}", prediction.Probability);
telemetryDataPoint.Add(prediction.TagName, prediction.Probability);
}
}
break;
case ModelType.Detection:
// Group the tags to get the count, include only the predictions above the specified minimum probability
var groupedPredictions = from prediction in imagePrediction.Predictions
where prediction.Probability >= probabilityThreshold
group prediction by new { prediction.TagName }
into newGroup
select new
{
TagName = newGroup.Key.TagName,
Count = newGroup.Count(),
};
// Display and log the agregated predictions
foreach (var prediction in groupedPredictions)
{
Debug.WriteLine($" Tag valid:{prediction.TagName} {prediction.Count}");
imageInformation.AddInt32($"Tag valid:{prediction.TagName}", prediction.Count);
telemetryDataPoint.Add(prediction.TagName, prediction.Count);
}
break;
default:
throw new ArgumentException("ModelType Invalid");
}
For a classifier only the tags with a probability greater than or equal the specified threshold are uploaded.
For a detection model the instances of each tag are counted. Only the tags with a prediction value greater than the specified threshold are included in the count.
19-08-14 05:26:14 Timer triggered Prediction count 33 Tag:Blue 0.0146500813 Tag:Blue 0.61186564 Tag:Blue 0.0923164859 Tag:Blue 0.7813785 Tag:Brown 0.0100603029 Tag:Brown 0.128318727 Tag:Brown 0.0135991769 Tag:Brown 0.687322736 Tag:Brown 0.846672833 Tag:Brown 0.1826635 Tag:Brown 0.0183384717 Tag:Green 0.0200069249 Tag:Green 0.367765248 Tag:Green 0.011428359 Tag:Orange 0.678825438 Tag:Orange 0.03718319 Tag:Orange 0.8643157 Tag:Orange 0.0296728313 Tag:Red 0.02141669 Tag:Red 0.7183208 Tag:Red 0.0183610674 Tag:Red 0.0130951973 Tag:Red 0.82097 Tag:Red 0.0618815944 Tag:Red 0.0130757084 Tag:Yellow 0.04150853 Tag:Yellow 0.0106579047 Tag:Yellow 0.0210028365 Tag:Yellow 0.03392527 Tag:Yellow 0.129197285 Tag:Yellow 0.8089519 Tag:Yellow 0.03723789 Tag:Yellow 0.74729687 Tag valid:Blue 2 Tag valid:Brown 2 Tag valid:Orange 2 Tag valid:Red 2 Tag valid:Yellow 2 05:26:17 AzureIoTHubClient SendEventAsync start 05:26:18 AzureIoTHubClient SendEventAsync finish
The debugging output of the application includes the different categories identified in the captured image.
I found my small model was pretty good at detection of individual m&m as long as the ambient lighting was consistent, and the background fairly plain.
Every so often the camera contrast setting went bad and could only be restored by restarting the device which needs further investigation.
This application could be the basis for projects which need to run an Azure Cognitive Services model to count or classify then upload the results to an Azure IoT Hub or Azure IoT Central for presentation.
With a suitable model this application could be used to count the number of people in a room, which could be displayed along with the ambient temperature, humidity, CO2, and noise levels in Azure IoT Central.
The code for this application is available In on GitHub.
