Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
Most of my applications have focused on telemetry but I had been thinking about local control for solutions that have to run disconnected. In “real-world” deployments connectivity to Azure EventGrid MQTT Broker isn’t 100% reliable (also delay and jitter issues) which are an issue for control at the edge.
When I started with the Security Camera HTTP code and added code to process the images with Ultralytics Object Detection model I found the order of the prompts could make a difference. My first attempt at adding YoloSharp to the SecurityCameraHttpClient application with Github Copilot didn’t go well and needed some “human intervention”. When I thought more about the order of the prompts the adding the same functionality went a lot better.
// Use a stream rather than loading image from a file // Use YoloSharp to run an onnx Object Detection model on the image // Make the YoloPredictor a class variable // Save image if object with specified image class name detected // Modify so objectDetected supports multiple image class names // Modify code to make use of GPU configurable // Make display of detections configurable in app settings // Make saving of image configurable in app settings
internal class Program
{
private static HttpClient _client;
private static bool _isRetrievingImage = false;
private static ApplicationSettings _applicationSettings;
private static YoloPredictor _yoloPredictor;
static void Main(string[] args)
{
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} SecurityCameraClient starting");
#if RELEASE
Console.WriteLine("RELEASE");
#else
Console.WriteLine("DEBUG");
#endif
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.AddUserSecrets<Program>()
.Build();
_applicationSettings = configuration.GetSection("ApplicationSettings").Get<ApplicationSettings>();
// Initialize YoloPredictor with GPU configuration
_yoloPredictor = new YoloPredictor(_applicationSettings.OnnxModelPath, new YoloPredictorOptions()
{
UseCuda = _applicationSettings.UseCuda, // Configurable GPU usage
});
using (HttpClientHandler handler = new HttpClientHandler { Credentials = new NetworkCredential(_applicationSettings.Username, _applicationSettings.Password) })
using (_client = new HttpClient(handler))
using (var timer = new Timer(async _ => await RetrieveImageAsync(), null, _applicationSettings.TimerDue, _applicationSettings.TimerPeriod))
{
Console.WriteLine("Press any key to exit...");
Console.ReadKey();
}
}
private static async Task RetrieveImageAsync()
{
if (_isRetrievingImage) return;
_isRetrievingImage = true;
try
{
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} SecurityCameraClient download starting");
HttpResponseMessage response = await _client.GetAsync(_applicationSettings.CameraUrl);
response.EnsureSuccessStatusCode();
using (Stream imageStream = await response.Content.ReadAsStreamAsync())
{
var detections = _yoloPredictor.Detect(imageStream);
bool objectDetected = false;
foreach (var detection in detections)
{
if (_applicationSettings.LogDetections) // Check if logging detections is enabled
{
Console.WriteLine($"Detected {detection.Name.Name} with confidence {detection.Confidence}");
}
if (_applicationSettings.ClassNames.Contains(detection.Name.Name))
{
objectDetected = true;
}
}
if (objectDetected && _applicationSettings.SaveImage) // Check if saving images is enabled
{
string savePath = string.Format(_applicationSettings.SavePath, DateTime.UtcNow);
using (FileStream fileStream = new FileStream(savePath, FileMode.Create, FileAccess.Write, FileShare.None))
{
imageStream.Position = 0;
await imageStream.CopyToAsync(fileStream);
}
}
}
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} SecurityCameraClient download done");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
finally
{
_isRetrievingImage = false;
}
}
}
public class ApplicationSettings
{
public string CameraUrl { get; set; } = "";
public string SavePath { get; set; } = "";
public string Username { get; set; } = "";
public string Password { get; set; } = "";
public TimeSpan TimerDue { get; set; } = TimeSpan.Zero;
public TimeSpan TimerPeriod { get; set; } = TimeSpan.Zero;
public string OnnxModelPath { get; set; } = "";
public bool UseCuda { get; set; } = false; // Configurable GPU usage
public List<string> ClassNames { get; set; } //= new List<string>();
public bool LogDetections { get; set; } = false; // Configurable logging of detections
public bool SaveImage { get; set; } = false; // Configurable saving of images
}
}
One odd thing that I noticed when scrolling “back and forth” through the images around when there was exception was that the date and time on the top left of the image was broken.
I wonder if the image was “broken” in some subtle way and FFMpegCore is handling this differently to the other libraries I’m trialing.
There was another sample application RabbitOM.Streaming.Tests.Mjpeg which displayed JPEG images. After looking at the code I figured out I need to use the RtpFrameBuilder class to assemble the RTSP packets into frames.
private static readonly RtpFrameBuilder _frameBuilder = new JpegFrameBuilder();
...
_frameBuilder.FrameReceived += OnFrameReceived;
...
client.PacketReceived += (sender, e) =>
{
var interleavedPacket = e.Packet as RtspInterleavedPacket;
if (interleavedPacket != null && interleavedPacket.Channel > 0)
{
// In most of case, avoid this packet
Console.ForegroundColor = ConsoleColor.DarkCyan;
Console.WriteLine("Skipping some data : size {0}", e.Packet.Data.Length);
return;
}
_frameBuilder.Write(interleavedPacket.Data);
};
I use GitHub Copilot with Visual Studio 2020 and I wanted to see if I could write a simple application only using “Ask Copilot”. Retrieving a static image from a security camera (like my HiLook IPC-T250H) felt like a reasonable challenge.
please write a console application that uses an HttpClient to retrieve an image from a security camera
The Github Copilot generated code compiled first time and looked reasonable for a Proof of Concept (PoC). After changing the “cameraUrl” the program could retrieve an image from one of my cameras when I disabled authentication.
please write a console application that uses an httpclient to retrieve an image from a security camera. The security camera needs a username and password
static async Task Main(string[] args)
{
string cameraUrl = "http://your-security-camera-url/image";
string username = "your-username";
string password = "your-password";
using (HttpClient client = new HttpClient())
{
var byteArray = new System.Text.ASCIIEncoding().GetBytes($"{username}:{password}");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
HttpResponseMessage response = await client.GetAsync(cameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
File.WriteAllBytes("camera_image.jpg", imageBytes);
Console.WriteLine("Image retrieved and saved as camera_image.jpg");
}
The Github Copilot generated code compiled and after changing the “cameraUrl”, “username” and “password” the program could retrieve an image from each of my cameras.
By this stage I was an elite “prompt engineer” and after a couple of revisions of my Github Copilot prompt to released unmanaged resources (the initial version used .Dispose and was a bit ugly).
modify the code to dispose of the HttpClientHandler, HttpClient and Timer with “using” statements
The application had a reentrancy issue when retrieving an image from a camera took too long
modify the code to stop RetrieveImageAsync getting called while an image is already being retrieved
I then decided to try a “cosmetic” change
modify the code _timer does not have to be class level variable
The savePath was a constant and I wanted to store a series of images
modify the code to use String.Format to generate the savepath
After some discussion with my co-workers I decided to explore how much Github Copilot helped building a “non-trivial” computer vision Proof of Concept (PoC). In previous blog posts I have written about capturing still images from a Security Camera so figured that was a good place to start.
Even though I have been using Copilot for a while I still say please.
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
class Program
{
static async Task Main(string[] args)
{
string cameraUrl = "http://your-security-camera-url/image.jpg"; // Replace with your security camera URL
string savePath = "C:\\path\\to\\save\\image.jpg"; // Replace with the path where you want to save the image
using (HttpClient client = new HttpClient())
{
try
{
HttpResponseMessage response = await client.GetAsync(cameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
await File.WriteAllBytesAsync(savePath, imageBytes);
Console.WriteLine("Image retrieved and saved successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
}
}
The generated code compiled first time, looked “ok” and worked with some of my security cameras when the cameraurl and save path were modified
I then modified the code so “sensitive” information was not stored in code or configuration files.
//using System;
//using System.IO;
//using System.Net.Http;
//using System.Reflection;
//using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
class Program
{
static async Task Main(string[] args)
{
//string cameraUrl = "http://your-security-camera-url/image.jpg"; // Replace with your security camera URL
//string savePath = "C:\\path\\to\\save\\image.jpg"; // Replace with the path where you want to save the image
Model.ApplicationSettings applicationSettings;
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.2.CameraHTTPClient starting");
#if RELEASE
Console.WriteLine("RELEASE");
#else
Console.WriteLine("DEBUG");
#endif
// load the app settings into configuration
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.AddUserSecrets<Program>()
.Build();
applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
using (HttpClient client = new HttpClient())
{
try
{
HttpResponseMessage response = await client.GetAsync(applicationSettings.CameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
await File.WriteAllBytesAsync(applicationSettings.SavePath, imageBytes);
Console.WriteLine("Image retrieved and saved successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
}
}
namespace Model
{
public class ApplicationSettings
{
public string CameraUrl { get; set; } = "";
public string SavePath { get; set; } = "Images";
}
}
// please write c# console application that uses httpclient and an httpget to retrieve image from security camera with NetworkCredential security
//
// Sure! Here's a C# console application that uses HttpClient and HttpGet to retrieve an image from a security camera with NetworkCredential security:
// In this example:
//
// Replace http://your-security-camera-url/image.jpg with the actual URL of your security camera's image.
//
// Replace C:\\path\\to\\save\\image.jpg with the path where you want to save the retrieved image.
//
// Replace your-username and your-password with your actual credentials.
//
// This application uses HttpClient with NetworkCredential to authenticate the request, retrieves the image as a byte array, and saves it to the specified path. If there's an error during the process, it will be caught and displayed in the console.
//
// Happy coding! 🚀
//using System;
//using System.IO;
using System.Net;
//using System.Net.Http;
//using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
class Program
{
static async Task Main(string[] args)
{
//string cameraUrl = "http://your-security-camera-url/image.jpg"; // Replace with your security camera URL
//string savePath = "C:\\path\\to\\save\\image.jpg"; // Replace with the path where you want to save the image
//string username = "your-username"; // Replace with your username
//string password = "your-password"; // Replace with your password
Model.ApplicationSettings applicationSettings;
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.3.CameraHTTPClient starting");
#if RELEASE
Console.WriteLine("RELEASE");
#else
Console.WriteLine("DEBUG");
#endif
// load the app settings into configuration
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.AddUserSecrets<Program>()
.Build();
applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
using (HttpClientHandler handler = new HttpClientHandler { Credentials = new NetworkCredential(applicationSettings.Username, applicationSettings.Password) })
using (HttpClient client = new HttpClient(handler))
{
try
{
HttpResponseMessage response = await client.GetAsync(applicationSettings.CameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
await File.WriteAllBytesAsync(applicationSettings.SavePath, imageBytes);
Console.WriteLine("Image retrieved and saved successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
}
}
namespace Model
{
public class ApplicationSettings
{
public string CameraUrl { get; set; } = "";
public string SavePath { get; set; } = "Images";
public string Username { get; set; } = "";
public string Password { get; set; } = "";
}
}
My test harness code was “inspired” by the Nager.VideoStream.TestConsole application with a slightly different file format for the start-stop marker text and camera images files.
I used Path.Combine so no code or configuration changes were required when the application was run on different operating systems (still need to ensure ImageFilepathLocal in the appsettings.json is the correct format).
Developer Desktop
I used my desktop computer a 13th Gen Intel(R) Core(TM) i7-13700 2.10 GHz with 32.0 GB running Windows 11 Pro 24H2.
In the test results below (representative of multiple runs while testing) the delay between starting streaming and the first image file was on average 3.7 seconds with the gap between the images roughly 100mSec.
In the test results below (representative of multiple runs while testing) the delay between starting streaming and the first image file was on average roughly 3.7 seconds but the time between images varied a lot from 30mSec to >300mSec.
At 10FPS the results for my developer desktop were more consistent, and the reComputer J3011 had significantly more “jitter”. Both could cope with 1oFPS so the next step is to integrate YoloDotNet library to process the video frames.
Enabling CUDA reduced the total image scaling, pre-processing, inferencing, and post processing time from 115mSec to 36mSec which is a significant improvement.
Several of my projects use the NickSwardh/YoloDotNetNuGet which supports NVIDIA CUDA but not TensorRT. The first step before “putting the NuGet on a diet” was to fix up my test application because some of the method signatures had changed in the latest release.
// load the app settings into configuration
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.Build();
_applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model load start : {_applicationSettings.ModelPath}");
//using (var predictor = new Yolo(_applicationSettings.ModelPath, false))
using var yolo = new Yolo(new YoloOptions()
{
OnnxModel = _applicationSettings.ModelPath,
Cuda = false,
PrimeGpu = false,
ModelType = ModelType.ObjectDetection,
});
{
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model load done");
Console.WriteLine();
//using (var image = await SixLabors.ImageSharp.Image.LoadAsync<Rgba32>(_applicationSettings.ImageInputPath))
using (var image = SKImage.FromEncodedData(_applicationSettings.ImageInputPath))
{
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model detect start");
var predictions = yolo.RunObjectDetection(image);
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model detect done");
Console.WriteLine();
foreach (var predicition in predictions)
{
Console.WriteLine($" Class {predicition.Label.Name} {(predicition.Confidence * 100.0):f1}% X:{predicition.BoundingBox.Location.X} Y:{predicition.BoundingBox.Location.Y} Width:{predicition.BoundingBox.Width} Height:{predicition.BoundingBox.Height}");
}
Console.WriteLine();
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} Plot and save : {_applicationSettings.ImageOutputPath}");
using (SKImage skImage = image.Draw(predictions))
{
//await image.SaveAsJpegAsync(_applicationSettings.ImageOutputPath);
skImage.Save(_applicationSettings.ImageOutputPath, SKEncodedImageFormat.Jpeg);
}
}
}
The YoloDotNet code was looking for specific text in the model description which wasn’t present in the description of my Ultralytics Hub trained models.
I downloaded the YoloDotNet source and included the core project in my solution so I could temporarily modify the GetModelVersion method in OnnxPropertiesExtension.cs.
/// <summary>
/// Get ONNX model version
/// </summary>
private static ModelVersion GetModelVersion(string modelDescription) => modelDescription.ToLower() switch
{
var version when version.Contains("yolo") is false => ModelVersion.V8,
var version when version.Contains("yoloV8") is false => ModelVersion.V8, // <========
var version when version.StartsWith("ultralytics yolov8") => ModelVersion.V8,
var version when version.StartsWith("ultralytics yolov9") => ModelVersion.V9,
var version when version.StartsWith("ultralytics yolov10") => ModelVersion.V10,
var version when version.StartsWith("ultralytics yolo11") => ModelVersion.V11, // Note the missing v in Yolo11
var version when version.Contains("worldv2") => ModelVersion.V11,
_ => throw new NotSupportedException("Onnx model not supported!")
};
After getting the test application running in the Visual Studio 2022 debugger it looked like the CustomMetadata Version info would be a better choice.
To check my assumption, I inspected some of the sample ONNX Model properties with Netron.
YoloV8s model 8.1.1
YoloV10s Model – 8.2.5.1
It looks like the CustomMetadata Version info increments but doesn’t nicely map to the Ultralytics Yolo version.
devMobile's blog 