The AzureIoTSmartEdgeCameraService was a useful proof of concept(PoC) but the codebase was starting to get unwieldy so it has been split into the SmartEdgeCameraAzureStorageService and SmartEdgeCameraAzureIoTService.
The initial ML.Net +You only look once V5(YoloV5) project uploaded raw (effectively a time lapse camera) and marked-up (with searchable tags) images to Azure Storage. But, after using it in a “real” project I found…
- The time-lapse functionality which continually uploaded images wasn’t that useful. I have another standalone application which has that functionality.
- If an object with a label in the “PredictionLabelsOfInterest” and a score greater than PredicitionScoreThreshold was detected it was useful to have the option to upload the camera and/or marked-up (including objects below the threshold) image(s).
- Having both camera and marked-up images tagged so they were searchable with an application like Azure Storage Explorer was very useful.
After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period was started.
private async void ImageUpdateTimerCallback(object state)
{
DateTime requestAtUtc = DateTime.UtcNow;
// Just incase - stop code being called while photo already in progress
if (_cameraBusy)
{
return;
}
_cameraBusy = true;
_logger.LogInformation("Image processing start");
try
{
#if CAMERA_RASPBERRY_PI
RaspberryPIImageCapture();
#endif
#if CAMERA_SECURITY
SecurityCameraImageCapture();
#endif
List<YoloPrediction> predictions;
using (Image image = Image.FromFile(_applicationSettings.ImageCameraFilepath))
{
_logger.LogTrace("Prediction start");
predictions = _scorer.Predict(image);
_logger.LogTrace("Prediction done");
OutputImageMarkup(image, predictions, _applicationSettings.ImageMarkedUpFilepath);
}
if (_logger.IsEnabled(LogLevel.Trace))
{
_logger.LogTrace("Predictions {0}", predictions.Select(p => new { p.Label.Name, p.Score }));
}
var predictionsOfInterest = predictions.Where(p => p.Score > _applicationSettings.PredicitionScoreThreshold).Select(c => c.Label.Name).Intersect(_applicationSettings.PredictionLabelsOfInterest, StringComparer.OrdinalIgnoreCase);
if (_logger.IsEnabled(LogLevel.Trace))
{
_logger.LogTrace("Predictions of interest {0}", predictionsOfInterest.ToList());
}
var predictionsTally = predictions.Where(p => p.Score >= _applicationSettings.PredicitionScoreThreshold)
.GroupBy(p => p.Label.Name)
.Select(p => new
{
Label = p.Key,
Count = p.Count()
});
if (predictionsOfInterest.Any())
{
BlobUploadOptions blobUploadOptions = new BlobUploadOptions()
{
Tags = new Dictionary<string, string>()
};
foreach (var predicition in predictionsTally)
{
blobUploadOptions.Tags.Add(predicition.Label, predicition.Count.ToString());
}
if (_applicationSettings.ImageCameraUpload)
{
_logger.LogTrace("Image camera upload start");
string imageFilenameCloud = string.Format(_azureStorageSettings.ImageCameraFilenameFormat, requestAtUtc);
await _imagecontainerClient.GetBlobClient(imageFilenameCloud).UploadAsync(_applicationSettings.ImageCameraFilepath, blobUploadOptions);
_logger.LogTrace("Image camera upload done");
}
if (_applicationSettings.ImageMarkedupUpload)
{
_logger.LogTrace("Image marked-up upload start");
string imageFilenameCloud = string.Format(_azureStorageSettings.ImageMarkedUpFilenameFormat, requestAtUtc);
await _imagecontainerClient.GetBlobClient(imageFilenameCloud).UploadAsync(_applicationSettings.ImageMarkedUpFilepath, blobUploadOptions);
_logger.LogTrace("Image marked-up upload done");
}
}
if (_logger.IsEnabled(LogLevel.Information))
{
_logger.LogInformation("Predictions tally {0}", predictionsTally.ToList());
}
}
catch (Exception ex)
{
_logger.LogError(ex, "Camera image download, post procesing, image upload, or telemetry failed");
}
finally
{
_cameraBusy = false;
}
TimeSpan duration = DateTime.UtcNow - requestAtUtc;
_logger.LogInformation("Image processing done {0:f2} sec", duration.TotalSeconds);
}
The test-rig consisted of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) module, D-Link Switch and a Raspberry Pi 4B 8G, or ASUS PE100A, or my HP Prodesk 400G4 DM (i7-8700T)
Excluding the first download it takes on average 0.16 secs to download a security camera image with my network setup.
The HP Prodesk 400G4 DM (i7-8700T) took on average 1.16 seconds to download an image from the camera, run the model, and upload the two images to Azure Storage
The Raspberry Pi 4B 8G took on average 2.18 seconds to download an image from the camera, run the model, then upload the two images to Azure Storage
The ASUS PE100A took on average 3.79 seconds to download an image from the camera, run the model, then upload the two images to Azure Storage.