The SmartEdgeCameraAzureIoTService application uses the same You Only Look Once(YOLOV5) + ML.Net + Open Neural Network Exchange(ONNX) plumbing as the SmartEdgeCameraAzureStorageService.
If there is an object with a label in the PredictionLabelsOfInterest list, a tally of each of the different object classes is sent to an Azure IoT Hub.
"Application": {
"DeviceID": "",
"ImageTimerDue": "0.00:00:15",
"ImageTimerPeriod": "0.00:00:30",
"ImageCameraFilepath": "ImageCamera.jpg",
"YoloV5ModelPath": "YoloV5/yolov5s.onnx",
"PredicitionScoreThreshold": 0.7,
"PredictionLabelsOfInterest": [
"person"
],
}
The Azure IoT hub can configured via a Shared Access Signature(SAS) device policy connection string or the Azure IoT Hub Device Provisioning Service(DPS)
After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period is started. Using some Language Integrated Query (LINQ) code any predictions with a score < PredictionScoreThreshold are discarded, then the list of predictions is checked to see if there are any in the PredictionLabelsOfInterest. If there are any matching predictions a count of the instances of each class is generated with more LINQ code.
private async void ImageUpdateTimerCallback(object state)
{
DateTime requestAtUtc = DateTime.UtcNow;
// Just incase - stop code being called while photo already in progress
if (_cameraBusy)
{
return;
}
_cameraBusy = true;
_logger.LogInformation("Image processing start");
try
{
#if CAMERA_RASPBERRY_PI
RaspberryPIImageCapture();
#endif
#if CAMERA_SECURITY
SecurityCameraImageCapture();
#endif
List<YoloPrediction> predictions;
using (Image image = Image.FromFile(_applicationSettings.ImageCameraFilepath))
{
_logger.LogTrace("Prediction start");
predictions = _scorer.Predict(image);
_logger.LogTrace("Prediction done");
}
if (_logger.IsEnabled(LogLevel.Trace))
{
_logger.LogTrace("Predictions {0}", predictions.Select(p => new { p.Label.Name, p.Score }));
}
var predictionsOfInterest = predictions.Where(p => p.Score > _applicationSettings.PredicitionScoreThreshold)
.Select(c => c.Label.Name)
.Intersect(_applicationSettings.PredictionLabelsOfInterest, StringComparer.OrdinalIgnoreCase);
if (predictionsOfInterest.Any())
{
if (_logger.IsEnabled(LogLevel.Trace))
{
_logger.LogTrace("Predictions of interest {0}", predictionsOfInterest.ToList());
}
var predictionsTally = predictions.GroupBy(p => p.Label.Name)
.Select(p => new
{
Label = p.Key,
Count = p.Count()
});
if (_logger.IsEnabled(LogLevel.Information))
{
_logger.LogInformation("Predictions tally {0}", predictionsTally.ToList());
}
JObject telemetryDataPoint = new JObject();
foreach (var predictionTally in predictionsTally)
{
telemetryDataPoint.Add(predictionTally.Label, predictionTally.Count);
}
using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
{
message.Properties.Add("iothub-creation-time-utc", requestAtUtc.ToString("s", CultureInfo.InvariantCulture));
await _deviceClient.SendEventAsync(message);
}
}
}
catch (Exception ex)
{
_logger.LogError(ex, "Camera image download, post processing, telemetry failed");
}
finally
{
_cameraBusy = false;
}
TimeSpan duration = DateTime.UtcNow - requestAtUtc;
_logger.LogInformation("Image processing done {0:f2} sec", duration.TotalSeconds);
}
The list of prediction class counts is used to populate a Newtonsoft JObject which serialised to generate a Java Script Object Notation(JSON) payload for an Azure IoT Hub message.
The test-rig consisted of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) and my HP Prodesk 400G4 DM (i7-8700T)
Pingback: Smartish Edge Camera – Azure IoT Central Part 2 | devMobile's blog
Pingback: Smartish Edge Camera – Azure IoT Image Upload | devMobile's blog