For the last month I have been using preview releases of ML.Net with a focus on Open Neural Network Exchange(ONNX) support. A company I work with has a YoloV5 based solution for tracking the cattle in stockyards so I figured I would try getting YoloV5 working with .Net Core and ML.Net on ARM64.
After some searching I found a repository created by Github user Mentalstack for an ONNX based YoloV5 implementation which I cloned and started hacking. I stared by updating the NuGet packages for the scorer and sample application and fixing what broke.
I didn’t update the System.Drawing.Common Nuget as my Raspberry PI V4 has got .Net Core V5 installed.
The sample application only had one dependency Microsoft.ML.OnnxRuntime which I was able to drop as it was referenced by the YoloV5Net.Scorer.
I then modified the sample application to process all the images in an “input” folder and save the processed images with Minimum Bounding Boxes(MBRs) to the output folder.
using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using Yolov5Net.Scorer;
using Yolov5Net.Scorer.Models;
namespace Yolov5Net.App
{
class Program
{
static void Main(string[] args)
{
var scorer = new YoloScorer<YoloCocoP5Model>("Assets/Weights/yolov5s.onnx");
DateTime startedAtUtc = DateTime.UtcNow;
Console.WriteLine($"{startedAtUtc:yyyy:MM:dd HH:mm:ss} Start");
string[] imageFilesPaths = Directory.GetFiles("Assets/inputs");
foreach (string imageFilePath in imageFilesPaths)
{
using (Image image = Image.FromFile(imageFilePath))
using (Graphics graphics = Graphics.FromImage(image))
{
List<YoloPrediction> predictions = scorer.Predict(image);
foreach (var prediction in predictions) // iterate predictions to draw results
{
double score = Math.Round(prediction.Score, 2);
graphics.DrawRectangles(new Pen(prediction.Label.Color, 1), new[] { prediction.Rectangle });
var (x, y) = (prediction.Rectangle.X - 3, prediction.Rectangle.Y - 23);
graphics.DrawString($"{prediction.Label.Name} ({score})", new Font("Consolas", 16, GraphicsUnit.Pixel), new SolidBrush(prediction.Label.Color), new PointF(x, y));
}
image.Save($"Assets/outputs/{Path.GetFileName(imageFilePath)}");
}
}
DateTime finishedAtUtc = DateTime.UtcNow;
TimeSpan duration = finishedAtUtc - startedAtUtc;
Console.WriteLine($"{finishedAtUtc:yyyy:MM:dd HH:mm:ss} Finish Duration:{duration.TotalMilliseconds}mSec");
}
}
}
The sample images are from wikimedia commons site. Go to Wikimediacommon.md to refer to the image urls and their licenses.
The next step of my Proof of Concept(PoC) was to get the YoloV5 Object Detection sample application working on my Intel(R) Core(TM) i7-8700T CPU @ 2.40GHz 2.40 GHz desktop development system. After debugging the software with Visual Studio I “published” the application to a folder.
The application took roughly 0.9 seconds to process each of my 5 sample images. The next task was to get the YoloV5 sample application working on a Raspberry Pi 4 running Bullseye.
To deploy applications I often copy the contents of the “publish” directory to the device with WinSCP. Getting the Object sample application running on my Raspberry Pi4 took a couple of attempts…
I had forgotten then Unix paths are case sensitive inputs vs. Inputs
pi@raspberrypi4a:~/vsdbg/Yolov5Net.App $ dotnet Yolov5Net.App.dll
Unhandled exception. System.TypeInitializationException: The type initializer for 'Microsoft.ML.OnnxRuntime.NativeMethods' threw an exception.
---> System.DllNotFoundException: Unable to load shared library 'onnxruntime' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libonnxruntime: cannot open shared object file: No such file or directory
at Microsoft.ML.OnnxRuntime.NativeMethods.OrtGetApiBase()
at Microsoft.ML.OnnxRuntime.NativeMethods..cctor()
--- End of inner exception stack trace ---
at Microsoft.ML.OnnxRuntime.SessionOptions..ctor()
at Yolov5Net.Scorer.YoloScorer`1..ctor(String weights, SessionOptions opts) in C:\Users\BrynLewis\source\repos\yolov5-net\src\Yolov5Net.Scorer\YoloScorer.cs:line 326
at Yolov5Net.App.Program.Main(String[] args) in C:\Users\BrynLewis\source\repos\yolov5-net\src\Yolov5Net.App\Program.cs:line 14
Aborted
pi@raspberrypi4a:~/vsdbg/Yolov5Net.App $
\
The ONNX runtime was missing so I confirmed the processor architecture with uname then copied the platform specific file to the application folder with Winscp.
I then checked Yolo V4 Sample application was generating output images with WinSCP.
On the Raspberry PI4B the application took roughly 8.3 seconds to process each of my 5 sample images.
I was “standing on the shoulders of giants” the Mentalstack code just worked, my changes were minimal and largely so I could collect some basic performance metrics. I need to spend some more time figuring out how the implementation works.
Need some wisdom on the missing ONNX runtime.
I see libonnxruntime.so in the linux-x64/native folder on the machine I am compiling (Ubuntu 64 using dotnet code)
I still get the “unable to load shared library onnxruntime” when running the app.
I must be doing something wrong here….
Hi
On my dev box I use VS2K19 so I publish my application to a folder
C:\Users\BrynLewis\source\repos.…\ObjectDetectionCamera\bin\Debug\net5.0\publish
The contents of the “publish’ folder with the application dlls, library dependences etc. end up in a folder on the device.
On my desktop the platform native runtimes are in sub folders of the publish folder
C:\Users\BrynLewis\source\repos...\ObjectDetectionCamera\bin\Debug\net5.0\publish\runtimes\android\native\onnxruntime.aar
…
C:\Users\BrynLewis\source\repos...\ObjectDetectionCamera\bin\Debug\net5.0\publish\runtimes\linux-aarch64\native\libonnxruntime.so
etc.
Another another part of the build process (initially just Winscp until things have stabilised) copies the applicable native runtime to the application folder.
In my case
C:\Users\BrynLewis\source\repos...\ObjectDetectionCamera\bin\Debug\net5.0\publish\runtimes\linux-aarch64\native\libonnxruntime.so
Ends up with rest of binaries in
/home/pi/ObjectDetectionCamera
Hope that helps
@KiwiBryn
Pingback: ML.Net YoloV5 + Camera on ARM64 Raspberry PI | devMobile's blog