Tag Archives: Object Classification

Building Cloud AI with Github Copilot- YoloSharp Azure HTTP Functions

Posted on by

Introduction

For this post I have used Github Copilot prompts to generate Azure HTTP Trigger functions which use Ultralytics YoloV8 and Compunet YoloSharp for object classification, object detection, and pose estimation.

I started with the Visual Studio 2022 Azure functions quick start code which ran first time.

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;

namespace YoloSharpxxxxxHttpTriggerFunction
{
    public class Function1
    {
        private readonly ILogger<Function1> _logger;

        public Function1(ILogger<Function1> logger)
        {
            _logger = logger;
        }

        [Function("Function1")]
        public IActionResult Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req)
        {
            _logger.LogInformation("C# HTTP trigger function processed a request.");
            return new OkObjectResult("Welcome to Azure Functions!");
        }
    }
}

The code generated by Github Copilot for the three functions changed the Function attribute to FunctionName and didn’t initialise the ILogger correctly.

[FunctionName("DetectObjects")]
public static async Task<IActionResult> Run(
     [HttpTrigger(AuthorizationLevel.Function, "post", Route = "detect")] HttpRequest req,
     ILogger log)
{
   log.LogInformation("DetectObjects function received a request for object detection.");
   ...
}

Every so often, when uploading more than one image at a time there was a “System.IO.InvalidData Exception: The stream exceeded the data limit 16384” error. In previous examples I had tried the batch processing of multiple images but had hit memory issues, so putting the image to be in the message body rather than Multipart Form Data for future development seems like a reasonable approach.

Image in the body of the request

Object Classification

I started with an object classification function because in previous examples Github Copilot was better at unpacking the results.

please write an httpTrigger azure function that uses YoloSharp and a Yolo image classification onnx model
Image classification not object detection
The image is in the form data
The multipart/form-data check can be removed
The YoloPredictor should be released after use
Many image files could be uploaded in one request
Only one image file can be uploaded in one request

[Function("YoloImageClassificationFunction")]
public async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req)
{
   _logger.LogInformation("C# HTTP trigger function processed a request.");

   var form = await req.ReadFormAsync();
   var files = form.Files;

   if (files.Count == 0)
   {
      return new BadRequestObjectResult("No images uploaded.");
   }

   if (files.Count > 1)
   {
      return new BadRequestObjectResult("Only one image file can be uploaded in one request.");
   }

   var file = files[0];
   if (file.Length > 0)
   {
      using (var memoryStream = new MemoryStream())
      {
         await file.CopyToAsync(memoryStream);
         var imageBytes = memoryStream.ToArray();

         using (var yoloModel = new YoloPredictor("yolov8s-cls.onnx"))
         {
            var classifications = yoloModel.Classify(imageBytes);

            return new OkObjectResult(new { file.FileName, classifications });
         }
      }
   }

   return new BadRequestObjectResult("Invalid image file.");
}

Image in the form data of the request

When I compared the results with previous applications, they were the same. I should have added code which discarded any result with less that a specified confidence to reduce the size of the results.

Object Detection

please write an azure function with an http trigger that uses yolosharp and an onnx file to detect objects in an image
The image is not in the body it is in the form
Image.Load is not used
yolo.Detect can process an image file stream
The YoloPredictor should be released after use
Many image files could be uploaded in one request
Only one image per request
Add multipart/form-data content type check
Add check that there are headers — Didn’t get this right
Add check that req.Headers is not null
Add check that request has ContentType and it is multipart/form-data

[Function("ObjectDetectionFunction")]
public async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req)
{
   _logger.LogInformation("C# HTTP trigger function processed a request.");

   // Intellisense fix
   if (req.Headers == null || !req.Headers.TryGetValue("Content-Type", out Microsoft.Extensions.Primitives.StringValues value) || !value.ToString().Contains("multipart/form-data"))
   {
      return new BadRequestObjectResult("The request must have a Content-Type header with multipart/form-data.");
   }

   // Read the images from the form data
   var form = await req.ReadFormAsync();
   var files = form.Files;
   if (files.Count == 0)
   {
      return new BadRequestObjectResult("No image files uploaded.");
   }

   if (files.Count > 1)
   {
      return new BadRequestObjectResult("Only one image file is allowed per request.");
   }

   var file = files[0];
   if (file.Length == 0)
   {
      return new BadRequestObjectResult("The uploaded image file is empty.");
   }

   // Load the YOLOv8 model
   using (var yolo = new YoloPredictor("yolov8s.onnx"))
   {
      // Perform object detection
      using (var stream = file.OpenReadStream())
      {
         var items = yolo.Detect(stream);
         var result = new { FileName = file.FileName, Detections = items };

         // Return the detection results
         return new OkObjectResult(result);
      }
   }
}

After a couple of manual fixes the function ran in the desktop emulator

When I compared the results with previous applications, they were the same.

Pose Estimation

I forgot the ILogger initialisation so had to do it at the end

please write an azure function with an http trigger that uses yolosharp to estimate the pose of humans in an uploaded image.
Yolo v8 pose estimation model and yolosharp library
Make into azure function
The image files are in the form of the request
Modify the code so more than one image per request can be processed
Initialise ILogger in the constructor

//[FunctionName("PoseEstimation")]
[Function("PoseEstimation")]
public async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req)
{
   _log.LogInformation("Pose estimation function processed a request.");

   if (!req.HasFormContentType || !req.Form.Files.Any())
   {
      return new BadRequestObjectResult("Please upload image files.");
   }

   var results = new List<object>();

   foreach (var file in req.Form.Files)
   {
      using var memoryStream = new MemoryStream();
      await file.CopyToAsync(memoryStream);
      memoryStream.Position = 0;

      using var image = Image.Load<Rgba32>(memoryStream);

      // Initialize the YOLO model
      //using var predictor = new YoloPredictor("path/to/model.onnx");
      using var predictor = new YoloPredictor("yolov8s-pose.onnx");

      // Perform pose estimation
      var result = await predictor.PoseAsync(image);

      // Format the results
      //var poses = result.Poses.Select(pose => new
      var poses = result.Select(pose => new
      {
         //Keypoints = pose.Keypoints.Select(k => new { k.X, k.Y }),
         Keypoints = pose.Select(k => new { k.Point.X, k.Point.Y }),
         Confidence = pose.Confidence
      });

      results.Add(new
      {
         Image = file.FileName,
         Poses = poses
      });
   }

   return new OkObjectResult(new { results });
}

After a couple of manual fixes including changing the way the results were generated the function ran in the desktop emulator.

Summary

The generated code worked but required manual fixes and was pretty ugly

The Github Copilot generated code in this post is not suitable for production

Building Edge AI with AI- YoloDotNet Client

Posted on by

Introduction

For this post I have used Copilot prompts to generate code which uses Ultralytics YoloV8 and YoloDoNet by NickSwardh for object detection, object classification, and pose estimation.

Object Detection

static void Main(string[] args)
{
   string modelPath = "path/to/yolo_model.onnx";  // Update with actual model path
   string imagePath = "path/to/image.jpg";        // Update with actual image path

   // Load YOLO model
   var yolo = new YoloModel(modelPath);

   // Load image
   using Bitmap image = new Bitmap(imagePath);

   // Run classification
   var results = yolo.Predict(image);

   // Display results
   foreach (var result in results)
   {
      Console.WriteLine($"Detected: {result.Label} with confidence {result.Confidence:P}");
   }
}

The code didn’t compile so I tried a different prompt

This didn’t compile and looked like it was based on https://github.com/techwingslab/yolov5-net. The generated code also added code to check that the model and image files existed.

static void Main(string[] args)
{
   string modelPath = "yolov8.onnx"; // Replace with your actual model path
   string imagePath = "image.jpg"; // Replace with your actual image path

   if (!File.Exists(modelPath))
   {
      Console.WriteLine("Error: Model file not found!");
      return;
   }

   if (!File.Exists(imagePath))
   {
      Console.WriteLine("Error: Image file not found!");
      return;
   }

   try
   {
      // Load the YOLO model
      using var yolo = new Yolo(modelPath);

      // Load image from disk
      using var image = new Bitmap(imagePath);

      // Run object detection
      var results = yolo.Predict(image);

      // Display detected objects
      foreach (var result in results)
      {
         Console.WriteLine($"Detected: {result.Label} - Confidence: {result.Confidence}");
         Console.WriteLine($"Bounding Box: {result.BoundingBox}");
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine($"Error: {ex.Message}");
   }
}

This didn’t compile so tried again 

static void Main(string[] args)
{
   string modelPath = "yolov8.onnx"; // Replace with your actual model path
   string imagePath = "image.jpg"; // Replace with your actual image path
...
   try
   {
      var yoloOptions = new YoloOptions
      {
         ConfidenceThreshold = 0.5, // Confidence threshold (adjust as needed)
         IoUThreshold = 0.4        // Intersection over Union threshold
      };

      // Load the YOLO model
      //using var yolo = new Yolo(modelPath);
      using var yolo = new Yolo(yoloOptions);

      // Load image from disk
      using var image = new Bitmap(imagePath);

      // Run object detection
      //var results = yolo.Predict(image);
      var results = yolo.RunObjectDetection(image);

      // Display detected objects
      foreach (var result in results)
      {
         Console.WriteLine($"Detected: {result.Label} - Confidence: {result.Confidence}");
         Console.WriteLine($"Bounding Box: {result.BoundingBox}");
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine($"Error: {ex.Message}");
   }
}

This didn’t compile so tried a different approach

I manually modified the code removing ConfidenceThreshold and IoUThreshold, then used intellisense to “discover” then add ModelType & modelPath

static void Main(string[] args)
{
   string modelPath = "yolov8.onnx"; // Replace with your actual model path
   string imagePath = "image.jpg"; // Replace with your actual image path
...
   try
   {
      var yoloOptions = new YoloOptions
      {
         ModelType = ModelType.ObjectDetection,
         OnnxModel = modelPath
      };

      // Load the YOLO model
      //using var yolo = new Yolo(modelPath);
      //using var yolo = new Yolo(yoloOptions);
      //using var yolo = new Yolo(modelPath, yoloOptions);
      using var yolo = new Yolo(yoloOptions);

      // Load image using SkiaSharp
      using var skBitmap = SKBitmap.Decode(imagePath);

      // Convert SKBitmap to a format YOLO can process
      using var skImage = SKImage.FromBitmap(skBitmap);
      using var skData = skImage.Encode(SKEncodedImageFormat.Jpeg, 100);
      using var memoryStream = new MemoryStream(skData.ToArray());
      //var results = yolo.Predict(memoryStream);
      var results = yolo.RunObbDetection(skImage);

      // Display detected objects
      foreach (var result in results)
      {
         Console.WriteLine($"Detected: {result.Label} - Confidence: {result.Confidence}");
         Console.WriteLine($"Bounding Box: {result.BoundingBox}");
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine($"Error: {ex.Message}");
   }
}

The code compiled and ran but didn’t work because YoloDoNet assumed that my computer had CUDA support

static void Main(string[] args)
{
   string modelPath = "yolov8.onnx"; // Replace with your actual model path
   string imagePath = "image.jpg"; // Replace with your actual image path
...
         try
         {
            var yoloOptions = new YoloOptions
            {
               ModelType = ModelType.ObjectDetection,
               OnnxModel = modelPath,
               Cuda = false
            };

            // Load the YOLO model
            //using var yolo = new Yolo(modelPath);
            //using var yolo = new Yolo(yoloOptions);
            //using var yolo = new Yolo(modelPath, yoloOptions);
            using var yolo = new Yolo(yoloOptions);

            // Load image using SkiaSharp
            using var skBitmap = SKBitmap.Decode(imagePath);

            // Convert SKBitmap to a format YOLO can process
            using var skImage = SKImage.FromBitmap(skBitmap);
            using var skData = skImage.Encode(SKEncodedImageFormat.Jpeg, 100);
            using var memoryStream = new MemoryStream(skData.ToArray());
            //var results = yolo.Predict(memoryStream);
            var results = yolo.RunObbDetection(skImage);

            // Display detected objects
            foreach (var result in results)
            {
               Console.WriteLine($"Detected: {result.Label} - Confidence: {result.Confidence}");
               Console.WriteLine($"Bounding Box: {result.BoundingBox}");
            }
         }
         catch (Exception ex)
         {
            Console.WriteLine($"Error: {ex.Message}");
         }
      }
   }
}

The code compiled and ran but didn’t work because I had accidentally used oriented bounding box rather than the object detection.

I was getting sick of prompts so manually fixed SKImage loading, RunObjectDetection and then removed the “cruft”.

static void Main(string[] args)
{
   string modelPath = "yolov8s.onnx"; // Replace with your actual model path
   string imagePath = "sports.jpg"; // Replace with your actual image path
...
   try
   {
      var yoloOptions = new YoloOptions
      {
         ModelType = ModelType.ObjectDetection,
         OnnxModel = modelPath,
         Cuda = false
      };

      using var yolo = new Yolo(yoloOptions);

     using var skImage = SKImage.FromEncodedData(imagePath);

      var results = yolo.RunObjectDetection(skImage);

      foreach (var result in results)
      {
         Console.WriteLine($"Detected: {result.Label} - Confidence: {result.Confidence:F2}");
         Console.WriteLine($"Bounding Box: {result.BoundingBox}");
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine($"Error: {ex.Message}");
   }

   Console.WriteLine("Press Enter to exit the application");
   Console.ReadLine();
}

I tested the implementation with sample “sports” image from the YoloSharp Github repository

The console application output looked reasonable

Classification

My initial Copilot prompt

Don’t understand why reference to OpenCV was included

static void Main(string[] args)
{
   string modelPath = "path/to/yolo_model.onnx"; // Update with actual model path
   string imagePath = "path/to/image.jpg"; // Update with actual image path

   // Load YOLO model
   var yolo = new YoloModel(modelPath);

   // Load image
   using Bitmap image = new Bitmap(imagePath);

   // Run classification
   var results = yolo.Predict(image);

   // Display results
  foreach (var result in results)
  {
      Console.WriteLine($"Detected: {result.Label} with confidence {result.Confidence:P}");
   }
}

The code didn’t compile so I prompted the code be modified to use SkiaSharp which is used by YoloDoNet

This was a bit strange, so I tried again

I was getting sick of prompts so manually fixed SKImage loading, RunClassification and then removed the “cruft”.

static void Main(string[] args)
{
   string modelPath = "yolov8s-cls.onnx";  // Update with actual model path
   string imagePath = "pizza.jpg";        // Update with actual image path

   var yolo = new Yolo(new YoloOptions()
   {
      ModelType = ModelType.Classification,
      OnnxModel = modelPath,
      Cuda = false
   });

   // Load image
   using SKImage image = SKImage.FromEncodedData(imagePath);

   // Run classification
   var results = yolo.RunClassification(image);

   // Display results
   foreach (var result in results)
   {
      Console.WriteLine($"Detected: {result.Label} with confidence {result.Confidence:P}");
   }

   Console.WriteLine("Press Enter to exit the application");
   Console.ReadLine();
}

At this point the code compiled and ran

Pretty confident this i a picture of a pizza

Pose

My Copilot prompt

Replace, path/to/yolo_model.onnx, and path/to/image.jpg with the actual paths to your model files and input image

This example assumes that YoloDotNet V2 supports the loaded YOLO model. Verify compatibility with the YOLO ObjectDetection variant.

Copilot had “assumed” I meant Ultralytics Yolo V2 and the code didn’t compile. So, I tried again without V2.

At this point I gave up

Summary

Using Copilot prompts to generate code which uses Ultralytics YoloV8 and YoloDoNet didn’t go well. In addition Visual Studio intellisense often struggled to offer useful options.

Overall, the experience wasn’t great.

Building Cloud AI with AI-YoloSharp Object detection Azure HTTP Trigger function

Posted on by

Introduction

Using Copilot to generate the code to retrieve an image from a security camera and small applications for object detection, object classification, with YoloSharp also went surprisingly well. An Azure HTTP Trigger function is a bit more complex so I decided to do one next. I started with the Visual Studio 2022 generated code 

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;

namespace YoloObjectDetectionFunction
{
    public class Function1
    {
        private readonly ILogger<Function1> _logger;

        public Function1(ILogger<Function1> logger)
        {
            _logger = logger;
        }

        [Function("Function1")]
        public IActionResult Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequest req)
        {
            _logger.LogInformation("C# HTTP trigger function processed a request.");
            return new OkObjectResult("Welcome to Azure Functions!");
        }
    }
}

I then generated the first version of the code with Copilot desktop

The code looked “okay” but had lots of extra “usings”, used FunctionName vs Function attributes etc. so didn’t compile.

I think the “var yolo = new Yolo(modelPath)” comes from YoloDotNet by Niklas Swärd and “var detections = yolo.Predict(image)” is from YoloV8.Net by Samuel Stainback

using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using YoloSharp;
using YoloSharp.DataStructures;

public static class YoloObjectDetectionFunction
{
    private static readonly string modelPath = "path/to/your/yolo-model.onnx";

    [FunctionName("YoloObjectDetection")]
    public static async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequestMessage req,
        ILogger log)
    {
        log.LogInformation("Yolo Object Detection function processed a request.");

        // Read the image from the request
        byte[] imageBytes = await req.Content.ReadAsByteArrayAsync();
        using (Image<Rgba32> image = Image.Load<Rgba32>(imageBytes))
        {
            // Initialize YoloSharp
            using (var yolo = new Yolo(modelPath))
            {
                // Detect objects in the image
                var detections = yolo.Predict(image);

                // Process the detections
                foreach (var detection in detections)
                {
                    // Draw bounding boxes on the image
                    image.Mutate(ctx => ctx.DrawPolygon(Rgba32.Red, 2, detection.BoundingBox.ToPolygon()));
                }

                // Save the modified image to a memory stream
                using (var ms = new MemoryStream())
                {
                    image.SaveAsPng(ms);
                    ms.Seek(0, SeekOrigin.Begin);

                    // Return the modified image as a response
                    var result = new HttpResponseMessage(HttpStatusCode.OK)
                    {
                        Content = new StreamContent(ms)
                    };
                    result.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
                    return new FileStreamResult(ms, "image/png");
                }
            }
        }
    }
}

I manually fixed all the issues in the code with a lot of assistance from Visual Studio 2022 Intellisense

using System.Net;

using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using SixLabors.ImageSharp.Drawing.Processing;

//using YoloSharp;
//using YoloSharp.DataStructures;
using Compunet.YoloSharp;


public static class YoloObjectDetectionFunction
{
   //private static readonly string modelPath = "path/to/your/yolo-model.onnx";
   private static readonly string modelPath = "yolov8s.onnx";

   //[FunctionName("YoloObjectDetection")]
   [Function("YoloObjectDetection")]
   public static async Task<IActionResult> Run(
       [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequestMessage req,
       ILogger log)
   {
      log.LogInformation("Yolo Object Detection function processed a request.");

      // Read the image from the request
      byte[] imageBytes = await req.Content.ReadAsByteArrayAsync();
      using (Image<Rgba32> image = Image.Load<Rgba32>(imageBytes))
      {
         // Initialize YoloSharp
         //using (var yolo = new Yolo(modelPath))
         using (var yolo = new YoloPredictor(modelPath))
         {
            // Detect objects in the image
            //var detections = yolo.Predict(image);
            var detections = yolo.Detect(image);

            // Process the detections
            foreach (var detection in detections)
            {
               // Draw bounding boxes on the image
               //image.Mutate(ctx => ctx.DrawPolygon(Rgba32.Red, 2, detection.BoundingBox.ToPolygon()));

               var rectangle = new PointF[] {new Point(detection.Bounds.Bottom, detection.Bounds.Left), new Point(detection.Bounds.Bottom, detection.Bounds.Right), new Point(detection.Bounds.Right, detection.Bounds.Top), new Point(detection.Bounds.Left, detection.Bounds.Top)};

               image.Mutate(ctx => ctx.DrawPolygon(Rgba32.ParseHex("FF0000"), 2, rectangle));
            }

            // Save the modified image to a memory stream
            using (var ms = new MemoryStream())
            {
               image.SaveAsPng(ms);
               ms.Seek(0, SeekOrigin.Begin);

               // Return the modified image as a response
               var result = new HttpResponseMessage(HttpStatusCode.OK)
               {
                  Content = new StreamContent(ms)
               };
               result.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
               return new FileStreamResult(ms, "image/png");
            }
         }
      }
   }
}

When I posted an image to the function with Telerik Fiddler Classic it threw an exception, because the ILogger injected into the Run method was null.

I then asked Github Copilot to “Modify code so the ILogger is parameter of constructor”

using System.Net;

using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using SixLabors.ImageSharp.Drawing.Processing;

//using YoloSharp;
//using YoloSharp.DataStructures;
using Compunet.YoloSharp;


//public static class YoloObjectDetectionFunction
public class YoloObjectDetectionFunction
{
   private static readonly ILogger _log;
   //private static readonly string modelPath = "path/to/your/yolo-model.onnx";
   private static readonly string modelPath = "yolov8s.onnx";

   public YoloObjectDetectionFunction(ILogger<YoloObjectDetectionFunction> log)
   {
      _log = log;
   }

   //[FunctionName("YoloObjectDetection")]
   [Function("YoloObjectDetection")]
   //public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequestMessage req, ILogger log)
   public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequestMessage req)
   {
      _log.LogInformation("Yolo Object Detection function processed a request.");

      // Read the image from the request
      byte[] imageBytes = await req.Content.ReadAsByteArrayAsync();
      using (Image<Rgba32> image = Image.Load<Rgba32>(imageBytes))
      {
         // Initialize YoloSharp
         //using (var yolo = new Yolo(modelPath))
         using (var yolo = new YoloPredictor(modelPath))
         {
            // Detect objects in the image
            //var detections = yolo.Predict(image);
            var detections = yolo.Detect(image);

            // Process the detections
            foreach (var detection in detections)
            {
               // Draw bounding boxes on the image
               //image.Mutate(ctx => ctx.DrawPolygon(Rgba32.Red, 2, detection.BoundingBox.ToPolygon()));

               var rectangle = new PointF[] {new Point(detection.Bounds.Bottom, detection.Bounds.Left), new Point(detection.Bounds.Bottom, detection.Bounds.Right), new Point(detection.Bounds.Right, detection.Bounds.Top), new Point(detection.Bounds.Left, detection.Bounds.Top)};

               image.Mutate(ctx => ctx.DrawPolygon(Rgba32.ParseHex("FF0000"), 2, rectangle));
            }

            // Save the modified image to a memory stream
            using (var ms = new MemoryStream())
            {
               image.SaveAsPng(ms);
               ms.Seek(0, SeekOrigin.Begin);

               // Return the modified image as a response
               var result = new HttpResponseMessage(HttpStatusCode.OK)
               {
                  Content = new StreamContent(ms)
               };
               result.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
               return new FileStreamResult(ms, "image/png");
            }
         }
      }
   }
}

When I posted an image to the function it threw an exception, because content of the HttpRequestMessage was null.

I then asked Github Copilot to “Modify the code so that the image is read from the form”

// Read the image from the form
var form = await req.ReadFormAsync();
var file = form.Files["image"];
if (file == null || file.Length == 0)
{
   return new BadRequestObjectResult("Image file is missing or empty.");
}

When I posted an image to the function it returned a 400 Bad Request Error.

After inspecting the request I realized that the name field was wrong, as the generated code was looking for “image”

Content-Disposition: form-data; name=”image”; filename=”sports.jpg”

Then, when I posted an image to the function it returned a 500 error.

But, the FileStreamResult was failing so I modified the code to return a FileContentResult

using (var ms = new MemoryStream())
{
   image.SaveAsJpeg(ms);

   return new FileContentResult(ms.ToArray(), "image/jpg");
}

Then, when I posted an image to the function it succeeded

But, the bounding boxes around the detected objects were wrong.

I then manually fixed up the polygon code so the lines for each bounding box were drawn in the correct order.

// Process the detections
foreach (var detection in detections)
{
   var rectangle = new PointF[] {
      new Point(detection.Bounds.Left, detection.Bounds.Bottom),
      new Point(detection.Bounds.Right, detection.Bounds.Bottom),
      new Point(detection.Bounds.Right, detection.Bounds.Top),
      new Point(detection.Bounds.Left, detection.Bounds.Top)
 };

Then, when I posted an image to the function it succeeded

The bounding boxes around the detected objects were correct.

I then “refactored” the code, removing all the unused “using”s, removed any commented out code, changed ILogger to be initialised using a Primary Constructor etc.

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using SixLabors.ImageSharp.Drawing.Processing;

using Compunet.YoloSharp;

public class YoloObjectDetectionFunction(ILogger<YoloObjectDetectionFunction> log)
{
   private readonly ILogger<YoloObjectDetectionFunction> _log = log;
   private readonly string modelPath = "yolov8s.onnx";

   [Function("YoloObjectDetection")]
   public async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req)
   {
      _log.LogInformation("Yolo Object Detection function processed a request.");

      // Read the image from the form
      var form = await req.ReadFormAsync();
      var file = form.Files["image"];
      if (file == null || file.Length == 0)
      {
         return new BadRequestObjectResult("Image file is missing or empty.");
      }

      using (var stream = file.OpenReadStream())
      using (Image<Rgba32> image = Image.Load<Rgba32>(stream))
      {
         // Initialize YoloSharp
         using (var yolo = new YoloPredictor(modelPath))
         {
            // Detect objects in the image
            var detections = yolo.Detect(image);

            // Process the detections
            foreach (var detection in detections)
            {
               var rectangle = new PointF[] {
                  new Point(detection.Bounds.Left, detection.Bounds.Bottom),
                  new Point(detection.Bounds.Right, detection.Bounds.Bottom),
                  new Point(detection.Bounds.Right, detection.Bounds.Top),
                  new Point(detection.Bounds.Left, detection.Bounds.Top)
               };

               image.Mutate(ctx => ctx.DrawPolygon(Rgba32.ParseHex("FF0000"), 2, rectangle));
            }

            // Save the modified image to a memory stream
            using (var ms = new MemoryStream())
            {
               image.SaveAsJpeg(ms);

              return new FileContentResult(ms.ToArray(), "image/jpg");
            }
         }
      }
   }
}

Summary

The initial code generated by Copilot was badly broken but with the assistance of Visual Studio 2022 Intellisense was fixed fairly quickly. The ILogger not being initialised, me using the “wrong” upload file name were easy to debug, but the FileContentResult exception was a bit more difficult.

It took me a quite a bit longer to write the function with Copilot desktop/Github Copilot than what it would have taken me normally. But, I think a lot of this was due to having to take screen shots, writing this blog post as I went, and having already written several Azure HTTP Trigger function for processing uploaded images.

The Copilot generated code in this post is not suitable for production

IoT and AI Building Edge AI with AI- YoloSharp Client

Posted on by

Introduction

Using Copilot to generate the code to retrieve an image from a security camera went surprisingly well so next step was something a bit more complex. For this post I have used Copilot prompts to generate code which uses Ultralytics YoloV8 and Compunet YoloSharp for object detection, object classification, pose estimation, and instance segmentation 

using System;
using System.Drawing;
using YOLOSharp;

class Program
{
    static void Main(string[] args)
    {
        string modelPath = "path/to/yolov8.onnx"; // Replace with the path to your YOLOv8 model
        string imagePath = "path/to/image.jpg"; // Replace with the path to your image

        // Load the YOLOv8 model
        var yolo = new YOLO(modelPath);

        // Load the image from disk
        using (var image = Image.FromFile(imagePath))
        {
            // Run the YOLOv8 model on the image
            var results = yolo.Predict(image);

            // Display the results
            foreach (var result in results)
            {
                Console.WriteLine($"Label: {result.Label}, Confidence: {result.Confidence}, BoundingBox: {result.BoundingBox}");
            }
        }
    }
}

In this example:

Replace path/to/yolov8.onnx with the actual path to your YOLOv8 model file.

Replace path/to/image.jpg with the actual path to your image file.

This application loads the YOLOv8 model using YOLOSharp, loads an image from disk, runs the model on the image, and displays the results, including the label, confidence, and bounding box for each detected object.

Happy coding! 🚀

Copilot had “assumed” I meant object detection(reasonable) with YoloSharp and the code didn’t compile.

Object Detection

After some minor fixes and shifting the configuration to an appsettings file the application compiled. 

using Compunet.YoloSharp;
using Microsoft.Extensions.Configuration;
using SixLabors.ImageSharp;

class Program
{
   static void Main(string[] args)
   {
      Model.ApplicationSettings applicationSettings;

      Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.3.CameraHTTPClient starting");
#if RELEASE
       Console.WriteLine("RELEASE");
#else
      Console.WriteLine("DEBUG");
#endif
      // load the app settings into configuration
      var configuration = new ConfigurationBuilder()
           .AddJsonFile("appsettings.json", false, true)
      .Build();

      applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();

      // Load the YOLOv8 model
      var yolo = new YoloPredictor(applicationSettings.ModelPath);

      // Load the image from disk
      using (var image = Image.Load(applicationSettings.ImagePath))
      {
         // Run the YOLOv8 model on the image
         var results = yolo.Detect(image);

         // Display the results
         foreach (var result in results)
         {
            Console.WriteLine($"Label: {result.Name}, Confidence: {result.Confidence}, BoundingBox: {result.Bounds}");
         }
      }
   }
}

I tested the implementation with sample “sports” image from the YoloSharp Github repository

The console application output looked reasonable

I manually compared the output of the console application with the Ultralytics Hub model preview tab and the results looked close enough.

Pose

My Copilot prompt

please write a C# console application that uses Compunet YoloSharp to run a yolov8 pose model on an image loaded from disk

using Compunet.YoloSharp;
using Microsoft.Extensions.Configuration;
using SixLabors.ImageSharp;

class Program
{
   static void Main(string[] args)
   {
      Model.ApplicationSettings applicationSettings;

      Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.3.CameraHTTPClient starting");
#if RELEASE
       Console.WriteLine("RELEASE");
#else
      Console.WriteLine("DEBUG");
#endif
      // load the app settings into configuration
      var configuration = new ConfigurationBuilder()
           .AddJsonFile("appsettings.json", false, true)
      .Build();

      applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();

      // Load the YOLOv8 pose model
      var yolo = new YoloPredictor(applicationSettings.ModelPath);

      // Load the image from disk
      using (var image = Image.Load(applicationSettings.ImagePath))
      {
         // Run the YOLOv8 pose model on the image
         var results = yolo.Pose(image);

         // Display the results
         foreach (var result in results)
         {
            Console.WriteLine($"Label: {result.Name.Name}, Confidence: {result.Confidence}, BoundingBox: {result.Bounds}");
            Console.WriteLine("Keypoints:");
            foreach (var keypoint in result)
            {
               Console.WriteLine($"  - {keypoint.Point}");
            }
         }
      }
   }
}

After some minor fixes and shifting the configuration to an appsettings file the application compiled. I tested the implementation with sample “sports” image from the YoloSharp Github repository

The console application output looked reasonable

I manually compared the output of the console application with the Ultralytics Hub model preview tab and the results were reasonable

Classification

My Copilot prompt

please write a C# console application that uses Compunet YoloSharp to run a yolov8 pose model on an image loaded from disk

using Compunet.YoloSharp;
using Microsoft.Extensions.Configuration;
using SixLabors.ImageSharp;

class Program
{
   static void Main(string[] args)
   {
      Model.ApplicationSettings applicationSettings;

      Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} 0.3.CameraHTTPClient starting");
#if RELEASE
       Console.WriteLine("RELEASE");
#else
      Console.WriteLine("DEBUG");
#endif

      // load the app settings into configuration
      var configuration = new ConfigurationBuilder()
           .AddJsonFile("appsettings.json", false, true)
      .Build();

      applicationSettings = configuration.GetSection("ApplicationSettings").Get<Model.ApplicationSettings>();

      // Load the YOLOv8 classification model
      var yolo = new YoloPredictor(applicationSettings.ModelPath);

      // Load the image from disk
      using (var image = Image.Load(applicationSettings.ImagePath))
      {
         // Run the YOLOv8 classification model on the image
         var results = yolo.Classify(image);

         // Display the results
         foreach (var result in results)
         {
             Console.WriteLine($"Label: {result.Name.Name}, Confidence: {result.Confidence}");
         }
      }
   }
}

After some minor fixes and shifting the configuration to an appsettings file the application compiled. I tested the implementation with sample “toaster” image from the YoloSharp Github repository

The console application output looked reasonable

I’m pretty confident the input image was a toaster.

Summary

The Copilot prompts to generate code which uses Ultralytics YoloV8 and Compunet YoloSharp and may have produced better code with some “prompt engineering”. Using Visual Studio intellisense the generated code was easy to fix.

The Copilot generated code in this post is not suitable for production