Azure Smartish Edge Camera – The basics

This project builds on my ML.Net YoloV5 + Camera + GPIO on ARM64 Raspberry PI with the addition of basic support for Azure IoT Hubs, the Azure IoT Hub Device Provisioning Service(DPS), and Azure IoT Central.

My backyard test-rig has consists of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) module, and an ASUS PE100A.

Backyard test-rig

The application can be compiled with support for Azure IoT Connection strings or the Device Provisioning Service(DPS). The appsetings.json file has configuration options for Azure IoT Hub connection string or DPS Global Device Endpoint+ScopeID+Group Enrollment key.

{
  "ApplicationSettings": {
    "DeviceId": "NotTheEdgeCamera",

    "ImageTimerDue": "0.00:00:15",
    "ImageTimerPeriod": "0.00:00:30",

    "CameraUrl": "http://10.0.0.55:85/images/snapshot.jpg",
    "CameraUserName": ",,,",
    "CameraUserPassword": "...",

    "ButtonPinNumer": 6,
    "LedPinNumer": 5,

    "InputImageFilenameLocal": "InputLatest.jpg",
    "OutputImageFilenameLocal": "OutputLatest.jpg",

    "ProcessWaitForExit": 10000,

    "YoloV5ModelPath": "Assets/YoloV5/yolov5s.onnx",

    "PredicitionScoreThreshold": 0.5,

    "AzureIoTHubConnectionString": "...",

    "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
    "AzureIoTHubDpsIDScope": "...",
    "AzureIoTHubDpsGroupEnrollmentKey": "..."
  }
}

After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period is started.

private static async void ImageUpdateTimerCallback(object state)
{
	DateTime requestAtUtc = DateTime.UtcNow;

	// Just incase - stop code being called while photo already in progress
	if (_cameraBusy)
	{
		return;
	}
	_cameraBusy = true;

	Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing start");

	try
	{
#if SECURITY_CAMERA
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download start");
		SecurityCameraImageCapture();
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download done");
#endif

#if RASPBERRY_PI_CAMERA
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture start");
		RaspberryPICameraImageCapture();
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture done");
#endif

		List<YoloPrediction> predictions;

		// Process the image on local file system
		using (Image image = Image.FromFile(_applicationSettings.InputImageFilenameLocal))
		{
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing start");
			predictions = _scorer.Predict(image);
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing done");

#if OUTPUT_IMAGE_MARKUP
			using (Graphics graphics = Graphics.FromImage(image))
			{
				Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup start");

				foreach (var prediction in predictions) // iterate predictions to draw results
				{
					double score = Math.Round(prediction.Score, 2);

					graphics.DrawRectangles(new Pen(prediction.Label.Color, 1), new[] { prediction.Rectangle });

					var (x, y) = (prediction.Rectangle.X - 3, prediction.Rectangle.Y - 23);

					graphics.DrawString($"{prediction.Label.Name} ({score})", new Font("Consolas", 16, GraphicsUnit.Pixel), new SolidBrush(prediction.Label.Color), new PointF(x, y));
				}

				image.Save(_applicationSettings.OutputImageFilenameLocal);

				Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup done");
			}
#endif
		}

#if AZURE_IOT_HUB_CONNECTION || AZURE_IOT_HUB_DPS_CONNECTION
		await AzureIoTHubTelemetry(requestAtUtc, predictions);
#endif
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Camera image download, post procesing, image upload, or telemetry failed {ex.Message}");
	}
	finally
	{
		_cameraBusy = false;
	}

	TimeSpan duration = DateTime.UtcNow - requestAtUtc;

	Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing done {duration.TotalSeconds:f2} sec");
	Console.WriteLine();
}

In the ImageUpdateTimerCallback method a camera image is captured (Raspberry Pi Camera Module 2 or Unv ADZK-10 Security Camera) and written to the local file system.

SSH Connection to Azure PE100 running Smartish Camera application

The YoloV5 model ML.Net support library then loads the image and processes the prediction output (can be inspected with Netron) generating list of objects that have been detected, their Minimum Bounding Rectangle(MBR) and class.

public static async Task AzureIoTHubTelemetry(DateTime requestAtUtc, List<YoloPrediction> predictions)
{
	JObject telemetryDataPoint = new JObject();

	foreach (var predictionTally in predictions.Where(p => p.Score >= _applicationSettings.PredicitionScoreThreshold).GroupBy(p => p.Label.Name)
					.Select(p => new
					{
						Label = p.Key,
						Count = p.Count()
					}))
	{
		Console.WriteLine("  {0} {1}", predictionTally.Label, predictionTally.Count);

		telemetryDataPoint.Add(predictionTally.Label, predictionTally.Count);
	}

	try
	{
		using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
		{
			message.Properties.Add("iothub-creation-time-utc", requestAtUtc.ToString("s", CultureInfo.InvariantCulture));

			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information start");
			await _deviceClient.SendEventAsync(message);
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information finish");
		}
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync cow counting failed {ex.Message}");
	}
}

The list of predictions is post processed with a Language Integrated Query(LINQ) which filters out predictions with a score below a configurable threshold and returns a count of each class.

My backyard from the deck

The aggregated YoloV5 prediction results are then uploaded to an Azure IoT Hub or Azure IoT Central

Azure IoT Explorer Displaying message payloads from the Smartish Edge Camera
Azure IoT Central displaying message payloads from the Smartish Edge Camera

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.