Azure Smartish Edge Camera – The basics

This project builds on my ML.Net YoloV5 + Camera + GPIO on ARM64 Raspberry PI with the addition of basic support for Azure IoT Hubs, the Azure IoT Hub Device Provisioning Service(DPS), and Azure IoT Central.

My backyard test-rig has consists of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) module, and an ASUS PE100A.

Backyard test-rig

The application can be compiled with support for Azure IoT Connection strings or the Device Provisioning Service(DPS). The appsetings.json file has configuration options for Azure IoT Hub connection string or DPS Global Device Endpoint+ScopeID+Group Enrollment key.

{
  "ApplicationSettings": {
    "DeviceId": "NotTheEdgeCamera",

    "ImageTimerDue": "0.00:00:15",
    "ImageTimerPeriod": "0.00:00:30",

    "CameraUrl": "http://10.0.0.55:85/images/snapshot.jpg",
    "CameraUserName": ",,,",
    "CameraUserPassword": "...",

    "ButtonPinNumer": 6,
    "LedPinNumer": 5,

    "InputImageFilenameLocal": "InputLatest.jpg",
    "OutputImageFilenameLocal": "OutputLatest.jpg",

    "ProcessWaitForExit": 10000,

    "YoloV5ModelPath": "Assets/YoloV5/yolov5s.onnx",

    "PredicitionScoreThreshold": 0.5,

    "AzureIoTHubConnectionString": "...",

    "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
    "AzureIoTHubDpsIDScope": "...",
    "AzureIoTHubDpsGroupEnrollmentKey": "..."
  }
}

After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period is started.

private static async void ImageUpdateTimerCallback(object state)
{
	DateTime requestAtUtc = DateTime.UtcNow;

	// Just incase - stop code being called while photo already in progress
	if (_cameraBusy)
	{
		return;
	}
	_cameraBusy = true;

	Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing start");

	try
	{
#if SECURITY_CAMERA
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download start");
		SecurityCameraImageCapture();
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download done");
#endif

#if RASPBERRY_PI_CAMERA
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture start");
		RaspberryPICameraImageCapture();
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture done");
#endif

		List<YoloPrediction> predictions;

		// Process the image on local file system
		using (Image image = Image.FromFile(_applicationSettings.InputImageFilenameLocal))
		{
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing start");
			predictions = _scorer.Predict(image);
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing done");

#if OUTPUT_IMAGE_MARKUP
			using (Graphics graphics = Graphics.FromImage(image))
			{
				Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup start");

				foreach (var prediction in predictions) // iterate predictions to draw results
				{
					double score = Math.Round(prediction.Score, 2);

					graphics.DrawRectangles(new Pen(prediction.Label.Color, 1), new[] { prediction.Rectangle });

					var (x, y) = (prediction.Rectangle.X - 3, prediction.Rectangle.Y - 23);

					graphics.DrawString($"{prediction.Label.Name} ({score})", new Font("Consolas", 16, GraphicsUnit.Pixel), new SolidBrush(prediction.Label.Color), new PointF(x, y));
				}

				image.Save(_applicationSettings.OutputImageFilenameLocal);

				Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup done");
			}
#endif
		}

#if AZURE_IOT_HUB_CONNECTION || AZURE_IOT_HUB_DPS_CONNECTION
		await AzureIoTHubTelemetry(requestAtUtc, predictions);
#endif
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Camera image download, post procesing, image upload, or telemetry failed {ex.Message}");
	}
	finally
	{
		_cameraBusy = false;
	}

	TimeSpan duration = DateTime.UtcNow - requestAtUtc;

	Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing done {duration.TotalSeconds:f2} sec");
	Console.WriteLine();
}

In the ImageUpdateTimerCallback method a camera image is captured (Raspberry Pi Camera Module 2 or Unv ADZK-10 Security Camera) and written to the local file system.

SSH Connection to Azure PE100 running Smartish Camera application

The YoloV5 model ML.Net support library then loads the image and processes the prediction output (can be inspected with Netron) generating list of objects that have been detected, their Minimum Bounding Rectangle(MBR) and class.

public static async Task AzureIoTHubTelemetry(DateTime requestAtUtc, List<YoloPrediction> predictions)
{
	JObject telemetryDataPoint = new JObject();

	foreach (var predictionTally in predictions.Where(p => p.Score >= _applicationSettings.PredicitionScoreThreshold).GroupBy(p => p.Label.Name)
					.Select(p => new
					{
						Label = p.Key,
						Count = p.Count()
					}))
	{
		Console.WriteLine("  {0} {1}", predictionTally.Label, predictionTally.Count);

		telemetryDataPoint.Add(predictionTally.Label, predictionTally.Count);
	}

	try
	{
		using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
		{
			message.Properties.Add("iothub-creation-time-utc", requestAtUtc.ToString("s", CultureInfo.InvariantCulture));

			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information start");
			await _deviceClient.SendEventAsync(message);
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information finish");
		}
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync cow counting failed {ex.Message}");
	}
}

The list of predictions is post processed with a Language Integrated Query(LINQ) which filters out predictions with a score below a configurable threshold and returns a count of each class.

My backyard from the deck

The aggregated YoloV5 prediction results are then uploaded to an Azure IoT Hub or Azure IoT Central

Azure IoT Explorer Displaying message payloads from the Smartish Edge Camera
Azure IoT Central displaying message payloads from the Smartish Edge Camera

ML.Net ONNX Object Detection on ARM64 ASUS PE100A

I work on applications which need a device that will survive in a farm shed that is open to all weathers. The ASUS PE100A an ARM64 device which, with the right parts has an operational temperature range of -20~60°C should be fine for New Zealand conditions. The devices are usually shipped with Windows 10 IoT Core or Yocto but an Ubuntu image (which this post uses) is also available.

The Ubuntu install is distributed as a zip file which contains the NXP IMX flashing utility (uuu.exe), installation scripts and the device image. I won’t cover the process in detail as the very helpful local ASUS support person and the readme file were more than sufficient.

Contents of PE100 Ubuntu update
PE100 device dip switches which control boot process(see readme for details)
PE100 flashing process complete

After remembering to reset the DIP switches before powering up the device it booted to a simple console.

PE100 Ubuntu home screen

I then created a new user, set the password, updated the users permissions and manually installed the .Net Core runtime (using a hybrid of the Microsoft and these instructions). I then had a device that I could SSH into, copy files to with WinSCP and run simple console applications on.

I then deployed my ONNX Object Detection console application to the device and it wouldn’t start. I had forgotten to install support for System.Drawing.Common with

sudo apt-get install -y libgdiplus

Object Detection console application with code to draw MBRs on images
Object Detection console application without code to draw MBRs on images

The version of the application which draws Minimum Bounding Boxes(MBRs) on the output images was only slightly slower that the version which didn’t. (the PE100 has a 16G on board eMMC so disk access is going to be fairly quick)

The required operational temperature range and price point make the PE100A good platform for our product.