Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
There was another sample application RabbitOM.Streaming.Tests.Mjpeg which displayed JPEG images. After looking at the code I figured out I need to use the RtpFrameBuilder class to assemble the RTSP packets into frames.
private static readonly RtpFrameBuilder _frameBuilder = new JpegFrameBuilder();
...
_frameBuilder.FrameReceived += OnFrameReceived;
...
client.PacketReceived += (sender, e) =>
{
var interleavedPacket = e.Packet as RtspInterleavedPacket;
if (interleavedPacket != null && interleavedPacket.Channel > 0)
{
// In most of case, avoid this packet
Console.ForegroundColor = ConsoleColor.DarkCyan;
Console.WriteLine("Skipping some data : size {0}", e.Packet.Data.Length);
return;
}
_frameBuilder.Write(interleavedPacket.Data);
};
I use GitHub Copilot with Visual Studio 2020 and I wanted to see if I could write a simple application only using “Ask Copilot”. Retrieving a static image from a security camera (like my HiLook IPC-T250H) felt like a reasonable challenge.
please write a console application that uses an HttpClient to retrieve an image from a security camera
The Github Copilot generated code compiled first time and looked reasonable for a Proof of Concept (PoC). After changing the “cameraUrl” the program could retrieve an image from one of my cameras when I disabled authentication.
please write a console application that uses an httpclient to retrieve an image from a security camera. The security camera needs a username and password
static async Task Main(string[] args)
{
string cameraUrl = "http://your-security-camera-url/image";
string username = "your-username";
string password = "your-password";
using (HttpClient client = new HttpClient())
{
var byteArray = new System.Text.ASCIIEncoding().GetBytes($"{username}:{password}");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
HttpResponseMessage response = await client.GetAsync(cameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
File.WriteAllBytes("camera_image.jpg", imageBytes);
Console.WriteLine("Image retrieved and saved as camera_image.jpg");
}
The Github Copilot generated code compiled and after changing the “cameraUrl”, “username” and “password” the program could retrieve an image from each of my cameras.
By this stage I was an elite “prompt engineer” and after a couple of revisions of my Github Copilot prompt to released unmanaged resources (the initial version used .Dispose and was a bit ugly).
modify the code to dispose of the HttpClientHandler, HttpClient and Timer with “using” statements
The application had a reentrancy issue when retrieving an image from a camera took too long
modify the code to stop RetrieveImageAsync getting called while an image is already being retrieved
I then decided to try a “cosmetic” change
modify the code _timer does not have to be class level variable
The savePath was a constant and I wanted to store a series of images
modify the code to use String.Format to generate the savepath
After some discussion with my co-workers I decided to explore how much Github Copilot helped building a “non-trivial” computer vision Proof of Concept (PoC). In previous blog posts I have written about capturing still images from a Security Camera so figured that was a good place to start.
Even though I have been using Copilot for a while I still say please.
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
class Program
{
static async Task Main(string[] args)
{
string cameraUrl = "http://your-security-camera-url/image.jpg"; // Replace with your security camera URL
string savePath = "C:\\path\\to\\save\\image.jpg"; // Replace with the path where you want to save the image
using (HttpClient client = new HttpClient())
{
try
{
HttpResponseMessage response = await client.GetAsync(cameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
await File.WriteAllBytesAsync(savePath, imageBytes);
Console.WriteLine("Image retrieved and saved successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
}
}
The generated code compiled first time, looked “ok” and worked with some of my security cameras when the cameraurl and save path were modified
I then modified the code so “sensitive” information was not stored in code or configuration files.
//using System;
//using System.IO;
//using System.Net.Http;
//using System.Reflection;
//using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
class Program
{
static async Task Main(string[] args)
{
//string cameraUrl = "http://your-security-camera-url/image.jpg"; // Replace with your security camera URL
//string savePath = "C:\\path\\to\\save\\image.jpg"; // Replace with the path where you want to save the image
Model.ApplicationSettings applicationSettings;
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.2.CameraHTTPClient starting");
#if RELEASE
Console.WriteLine("RELEASE");
#else
Console.WriteLine("DEBUG");
#endif
// load the app settings into configuration
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.AddUserSecrets<Program>()
.Build();
applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
using (HttpClient client = new HttpClient())
{
try
{
HttpResponseMessage response = await client.GetAsync(applicationSettings.CameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
await File.WriteAllBytesAsync(applicationSettings.SavePath, imageBytes);
Console.WriteLine("Image retrieved and saved successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
}
}
namespace Model
{
public class ApplicationSettings
{
public string CameraUrl { get; set; } = "";
public string SavePath { get; set; } = "Images";
}
}
// please write c# console application that uses httpclient and an httpget to retrieve image from security camera with NetworkCredential security
//
// Sure! Here's a C# console application that uses HttpClient and HttpGet to retrieve an image from a security camera with NetworkCredential security:
// In this example:
//
// Replace http://your-security-camera-url/image.jpg with the actual URL of your security camera's image.
//
// Replace C:\\path\\to\\save\\image.jpg with the path where you want to save the retrieved image.
//
// Replace your-username and your-password with your actual credentials.
//
// This application uses HttpClient with NetworkCredential to authenticate the request, retrieves the image as a byte array, and saves it to the specified path. If there's an error during the process, it will be caught and displayed in the console.
//
// Happy coding! 🚀
//using System;
//using System.IO;
using System.Net;
//using System.Net.Http;
//using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
class Program
{
static async Task Main(string[] args)
{
//string cameraUrl = "http://your-security-camera-url/image.jpg"; // Replace with your security camera URL
//string savePath = "C:\\path\\to\\save\\image.jpg"; // Replace with the path where you want to save the image
//string username = "your-username"; // Replace with your username
//string password = "your-password"; // Replace with your password
Model.ApplicationSettings applicationSettings;
Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.3.CameraHTTPClient starting");
#if RELEASE
Console.WriteLine("RELEASE");
#else
Console.WriteLine("DEBUG");
#endif
// load the app settings into configuration
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.AddUserSecrets<Program>()
.Build();
applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
using (HttpClientHandler handler = new HttpClientHandler { Credentials = new NetworkCredential(applicationSettings.Username, applicationSettings.Password) })
using (HttpClient client = new HttpClient(handler))
{
try
{
HttpResponseMessage response = await client.GetAsync(applicationSettings.CameraUrl);
response.EnsureSuccessStatusCode();
byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
await File.WriteAllBytesAsync(applicationSettings.SavePath, imageBytes);
Console.WriteLine("Image retrieved and saved successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
}
}
namespace Model
{
public class ApplicationSettings
{
public string CameraUrl { get; set; } = "";
public string SavePath { get; set; } = "Images";
public string Username { get; set; } = "";
public string Password { get; set; } = "";
}
}
My test harness code was “inspired” by the Nager.VideoStream.TestConsole application with a slightly different file format for the start-stop marker text and camera images files.
I used Path.Combine so no code or configuration changes were required when the application was run on different operating systems (still need to ensure ImageFilepathLocal in the appsettings.json is the correct format).
Developer Desktop
I used my desktop computer a 13th Gen Intel(R) Core(TM) i7-13700 2.10 GHz with 32.0 GB running Windows 11 Pro 24H2.
In the test results below (representative of multiple runs while testing) the delay between starting streaming and the first image file was on average 3.7 seconds with the gap between the images roughly 100mSec.
In the test results below (representative of multiple runs while testing) the delay between starting streaming and the first image file was on average roughly 3.7 seconds but the time between images varied a lot from 30mSec to >300mSec.
At 10FPS the results for my developer desktop were more consistent, and the reComputer J3011 had significantly more “jitter”. Both could cope with 1oFPS so the next step is to integrate YoloDotNet library to process the video frames.
Enabling CUDA reduced the total image scaling, pre-processing, inferencing, and post processing time from 115mSec to 36mSec which is a significant improvement.
Several of my projects use the NickSwardh/YoloDotNetNuGet which supports NVIDIA CUDA but not TensorRT. The first step before “putting the NuGet on a diet” was to fix up my test application because some of the method signatures had changed in the latest release.
// load the app settings into configuration
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", false, true)
.Build();
_applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model load start : {_applicationSettings.ModelPath}");
//using (var predictor = new Yolo(_applicationSettings.ModelPath, false))
using var yolo = new Yolo(new YoloOptions()
{
OnnxModel = _applicationSettings.ModelPath,
Cuda = false,
PrimeGpu = false,
ModelType = ModelType.ObjectDetection,
});
{
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model load done");
Console.WriteLine();
//using (var image = await SixLabors.ImageSharp.Image.LoadAsync<Rgba32>(_applicationSettings.ImageInputPath))
using (var image = SKImage.FromEncodedData(_applicationSettings.ImageInputPath))
{
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model detect start");
var predictions = yolo.RunObjectDetection(image);
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} YoloV8 Model detect done");
Console.WriteLine();
foreach (var predicition in predictions)
{
Console.WriteLine($" Class {predicition.Label.Name} {(predicition.Confidence * 100.0):f1}% X:{predicition.BoundingBox.Location.X} Y:{predicition.BoundingBox.Location.Y} Width:{predicition.BoundingBox.Width} Height:{predicition.BoundingBox.Height}");
}
Console.WriteLine();
Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss.fff} Plot and save : {_applicationSettings.ImageOutputPath}");
using (SKImage skImage = image.Draw(predictions))
{
//await image.SaveAsJpegAsync(_applicationSettings.ImageOutputPath);
skImage.Save(_applicationSettings.ImageOutputPath, SKEncodedImageFormat.Jpeg);
}
}
}
The YoloDotNet code was looking for specific text in the model description which wasn’t present in the description of my Ultralytics Hub trained models.
I downloaded the YoloDotNet source and included the core project in my solution so I could temporarily modify the GetModelVersion method in OnnxPropertiesExtension.cs.
/// <summary>
/// Get ONNX model version
/// </summary>
private static ModelVersion GetModelVersion(string modelDescription) => modelDescription.ToLower() switch
{
var version when version.Contains("yolo") is false => ModelVersion.V8,
var version when version.Contains("yoloV8") is false => ModelVersion.V8, // <========
var version when version.StartsWith("ultralytics yolov8") => ModelVersion.V8,
var version when version.StartsWith("ultralytics yolov9") => ModelVersion.V9,
var version when version.StartsWith("ultralytics yolov10") => ModelVersion.V10,
var version when version.StartsWith("ultralytics yolo11") => ModelVersion.V11, // Note the missing v in Yolo11
var version when version.Contains("worldv2") => ModelVersion.V11,
_ => throw new NotSupportedException("Onnx model not supported!")
};
After getting the test application running in the Visual Studio 2022 debugger it looked like the CustomMetadata Version info would be a better choice.
To check my assumption, I inspected some of the sample ONNX Model properties with Netron.
YoloV8s model 8.1.1
YoloV10s Model – 8.2.5.1
It looks like the CustomMetadata Version info increments but doesn’t nicely map to the Ultralytics Yolo version.
The initial version of the YoloV8.dll in the version 4.2 of the NuGet was 96.5KB. Most of my applications deployed to edge devices and Azure do not require plotting functionality so I started by commenting out (not terribly subtle).
This post is for Azure Function developers having issues with the SendGrid binding throwing exceptions like the one below.
System.Private.CoreLib: Exception while executing function: Functions.AzureBlobFileUploadEmailer. Microsoft.Azure.WebJobs.Extensions.SendGrid: A 'To' address must be specified for the message
I wasn’t paying close enough attention to the sample code and used the System.Text.Json rather than Newtonsoft.Json to serialize the SendGridMessage object. They use different attributes for property names etc. so the JSON generated was wrong.
namespace SendGrid.Helpers.Mail
{
/// <summary>
/// Class SendGridMessage builds an object that sends an email through Twilio SendGrid.
/// </summary>
[JsonObject(IsReference = false)]
public class SendGridMessage
{
/// <summary>
/// Gets or sets an email object containing the email address and name of the sender. Unicode encoding is not supported for the from field.
/// </summary>
//[JsonProperty(PropertyName = "from")]
[JsonPropertyName("from")]
public EmailAddress From { get; set; }
/// <summary>
/// Gets or sets the subject of your email. This may be overridden by personalizations[x].subject.
/// </summary>
//[JsonProperty(PropertyName = "subject")]
[JsonPropertyName("subject")]
public string Subject { get; set; }
/// <summary>
/// Gets or sets a list of messages and their metadata. Each object within personalizations can be thought of as an envelope - it defines who should receive an individual message and how that message should be handled. For more information, please see our documentation on Personalizations. Parameters in personalizations will override the parameters of the same name from the message level.
/// https://sendgrid.com/docs/Classroom/Send/v3_Mail_Send/personalizations.html.
/// </summary>
//[JsonProperty(PropertyName = "personalizations", IsReference = false)]
[JsonPropertyName("personalizations")]
public List<Personalization> Personalizations { get; set; }
...
}
namespace SendGrid.Helpers.Mail
{
/// <summary>
/// An email object containing the email address and name of the sender or recipient.
/// </summary>
[JsonObject(IsReference = false)]
public class EmailAddress : IEquatable<EmailAddress>
{
/// <summary>
/// Initializes a new instance of the <see cref="EmailAddress"/> class.
/// </summary>
public EmailAddress()
{
}
/// <summary>
/// Initializes a new instance of the <see cref="EmailAddress"/> class.
/// </summary>
/// <param name="email">The email address of the sender or recipient.</param>
/// <param name="name">The name of the sender or recipient.</param>
public EmailAddress(string email, string name = null)
{
this.Email = email;
this.Name = name;
}
/// <summary>
/// Gets or sets the name of the sender or recipient.
/// </summary>
[JsonProperty(PropertyName = "name")]
public string Name { get; set; }
/// <summary>
/// Gets or sets the email address of the sender or recipient.
/// </summary>
[JsonProperty(PropertyName = "email")]
public string Email { get; set; }
...
}
Many of the property name “mismatch” issues were in the Personalization class with the Toos, Ccs, bccs etc. properties
namespace SendGrid.Helpers.Mail
{
/// <summary>
/// An array of messages and their metadata. Each object within personalizations can be thought of as an envelope - it defines who should receive an individual message and how that message should be handled. For more information, please see our documentation on Personalizations. Parameters in personalizations will override the parameters of the same name from the message level.
/// https://sendgrid.com/docs/Classroom/Send/v3_Mail_Send/personalizations.html.
/// </summary>
[JsonObject(IsReference = false)]
public class Personalization
{
/// <summary>
/// Gets or sets an array of recipients. Each email object within this array may contain the recipient’s name, but must always contain the recipient’s email.
/// </summary>
[JsonProperty(PropertyName = "to", IsReference = false)]
[JsonConverter(typeof(RemoveDuplicatesConverter<EmailAddress>))]
public List<EmailAddress> Tos { get; set; }
/// <summary>
/// Gets or sets an array of recipients who will receive a copy of your email. Each email object within this array may contain the recipient’s name, but must always contain the recipient’s email.
/// </summary>
[JsonProperty(PropertyName = "cc", IsReference = false)]
[JsonConverter(typeof(RemoveDuplicatesConverter<EmailAddress>))]
public List<EmailAddress> Ccs { get; set; }
/// <summary>
/// Gets or sets an array of recipients who will receive a blind carbon copy of your email. Each email object within this array may contain the recipient’s name, but must always contain the recipient’s email.
/// </summary>
[JsonProperty(PropertyName = "bcc", IsReference = false)]
[JsonConverter(typeof(RemoveDuplicatesConverter<EmailAddress>))]
public List<EmailAddress> Bccs { get; set; }
/// <summary>
/// Gets or sets the from email address. The domain must match the domain of the from email property specified at root level of the request body.
/// </summary>
[JsonProperty(PropertyName = "from")]
public EmailAddress From { get; set; }
/// <summary>
/// Gets or sets the subject line of your email.
/// </summary>
[JsonProperty(PropertyName = "subject")]
public string Subject { get; set; }
...
}
The Myriota Connector only supports Direct Methods which provide immediate confirmation of the result being queued by the Myriota Cloud API. The Myriota (API) control message send method responds with 400 Bad Request if there is already a message being sent to a device.
Myriota Azure Function Environment Variable configuration
Sense and Locate Azure IoT Central Template Fan Speed Enumeration
The fan speed value in the message payload is configured in the fan speed enumeration.
Sense and Locate Azure IoT Central Command Fan Speed Selection
The FanSpeed.cs payload formatter extracts the FanSpeed value from the Javascript Object Notation(JSON) payload and returns a two-byte array containing the message type and speed of the fan.
using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public class FormatterDownlink : PayloadFormatter.IFormatterDownlink
{
public byte[] Evaluate(string terminalId, string methodName, JObject payloadJson, byte[] payloadBytes)
{
byte? status = payloadJson.Value<byte?>("FanSpeed");
if (!status.HasValue)
{
return new byte[] { };
}
return new byte[] { 1, status.Value };
}
}
Sense and Locate Azure IoT Central Command Fan Speed History
Each Azure Application Insights log entry starts with the TerminalID (to simplify searching for all the messages related to device) and the requestId a Globally Unique Identifier (GUID) to simplify searching for all the “steps” associated with sending/receiving a message) with the rest of the logging message containing “step” specific diagnostic information.
Sense and Locate Azure IoT Central Command Fan Speed Application Insights
In the Myriota Device Manager the status of Control Messages can be tracked and they can be cancelled if in the “pending” state.
Myriota Control Message status Pending
A Control Message can take up to 24hrs to be delivered and confirmation of delivery has to be implemented by the application developer.
