This project builds on my ML.Net YoloV5 + Camera + GPIO on ARM64 Raspberry PI with the addition of basic support for Azure IoT Hubs, the Azure IoT Hub Device Provisioning Service(DPS), and Azure IoT Central.
My backyard test-rig has consists of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) module, and an ASUS PE100A.
The application can be compiled with support for Azure IoT Connection strings or the Device Provisioning Service(DPS). The appsetings.json file has configuration options for Azure IoT Hub connection string or DPS Global Device Endpoint+ScopeID+Group Enrollment key.
{
"ApplicationSettings": {
"DeviceId": "NotTheEdgeCamera",
"ImageTimerDue": "0.00:00:15",
"ImageTimerPeriod": "0.00:00:30",
"CameraUrl": "http://10.0.0.55:85/images/snapshot.jpg",
"CameraUserName": ",,,",
"CameraUserPassword": "...",
"ButtonPinNumer": 6,
"LedPinNumer": 5,
"InputImageFilenameLocal": "InputLatest.jpg",
"OutputImageFilenameLocal": "OutputLatest.jpg",
"ProcessWaitForExit": 10000,
"YoloV5ModelPath": "Assets/YoloV5/yolov5s.onnx",
"PredicitionScoreThreshold": 0.5,
"AzureIoTHubConnectionString": "...",
"GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
"AzureIoTHubDpsIDScope": "...",
"AzureIoTHubDpsGroupEnrollmentKey": "..."
}
}
After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period is started.
private static async void ImageUpdateTimerCallback(object state)
{
DateTime requestAtUtc = DateTime.UtcNow;
// Just incase - stop code being called while photo already in progress
if (_cameraBusy)
{
return;
}
_cameraBusy = true;
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing start");
try
{
#if SECURITY_CAMERA
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download start");
SecurityCameraImageCapture();
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download done");
#endif
#if RASPBERRY_PI_CAMERA
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture start");
RaspberryPICameraImageCapture();
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture done");
#endif
List<YoloPrediction> predictions;
// Process the image on local file system
using (Image image = Image.FromFile(_applicationSettings.InputImageFilenameLocal))
{
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing start");
predictions = _scorer.Predict(image);
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing done");
#if OUTPUT_IMAGE_MARKUP
using (Graphics graphics = Graphics.FromImage(image))
{
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup start");
foreach (var prediction in predictions) // iterate predictions to draw results
{
double score = Math.Round(prediction.Score, 2);
graphics.DrawRectangles(new Pen(prediction.Label.Color, 1), new[] { prediction.Rectangle });
var (x, y) = (prediction.Rectangle.X - 3, prediction.Rectangle.Y - 23);
graphics.DrawString($"{prediction.Label.Name} ({score})", new Font("Consolas", 16, GraphicsUnit.Pixel), new SolidBrush(prediction.Label.Color), new PointF(x, y));
}
image.Save(_applicationSettings.OutputImageFilenameLocal);
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup done");
}
#endif
}
#if AZURE_IOT_HUB_CONNECTION || AZURE_IOT_HUB_DPS_CONNECTION
await AzureIoTHubTelemetry(requestAtUtc, predictions);
#endif
}
catch (Exception ex)
{
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Camera image download, post procesing, image upload, or telemetry failed {ex.Message}");
}
finally
{
_cameraBusy = false;
}
TimeSpan duration = DateTime.UtcNow - requestAtUtc;
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing done {duration.TotalSeconds:f2} sec");
Console.WriteLine();
}
In the ImageUpdateTimerCallback method a camera image is captured (Raspberry Pi Camera Module 2 or Unv ADZK-10 Security Camera) and written to the local file system.
The YoloV5 model ML.Net support library then loads the image and processes the prediction output (can be inspected with Netron) generating list of objects that have been detected, their Minimum Bounding Rectangle(MBR) and class.
public static async Task AzureIoTHubTelemetry(DateTime requestAtUtc, List<YoloPrediction> predictions)
{
JObject telemetryDataPoint = new JObject();
foreach (var predictionTally in predictions.Where(p => p.Score >= _applicationSettings.PredicitionScoreThreshold).GroupBy(p => p.Label.Name)
.Select(p => new
{
Label = p.Key,
Count = p.Count()
}))
{
Console.WriteLine(" {0} {1}", predictionTally.Label, predictionTally.Count);
telemetryDataPoint.Add(predictionTally.Label, predictionTally.Count);
}
try
{
using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
{
message.Properties.Add("iothub-creation-time-utc", requestAtUtc.ToString("s", CultureInfo.InvariantCulture));
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information start");
await _deviceClient.SendEventAsync(message);
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information finish");
}
}
catch (Exception ex)
{
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync cow counting failed {ex.Message}");
}
}
The list of predictions is post processed with a Language Integrated Query(LINQ) which filters out predictions with a score below a configurable threshold and returns a count of each class.
The aggregated YoloV5 prediction results are then uploaded to an Azure IoT Hub or Azure IoT Central
devMobile's blog 