Smartish Edge Camera – Azure IoT Updateable Properties (not persisted)

This post builds on my Smartish Edge Camera -Azure IoT Direct Methods post adding two updateable properties for the image capture and processing timer the due and period values. The two properties can be updated together or independently but the values are not persisted.

When I was searching for answers I found this code in many posts and articles but it didn’t really cover my scenario.

private static async Task OnDesiredPropertyChanged(TwinCollection desiredProperties, 
  object userContext)
{
   Console.WriteLine("desired property chPleange:");
   Console.WriteLine(JsonConvert.SerializeObject(desiredProperties));
   Console.WriteLine("Sending current time as reported property");
   TwinCollection reportedProperties = new TwinCollection
   {
       ["DateTimeLastDesiredPropertyChangeReceived"] = DateTime.Now
   };

    await Client.UpdateReportedPropertiesAsync(reportedProperties).ConfigureAwait(false);
}

When AZURE_DEVICE_PROPERTIES is defined in the SmartEdgeCameraAzureIoTService project properties the device reports a number of properties on startup and SetDesiredPropertyUpdateCallbackAsync is used to configure the method called whenever the client receives a state update(desired or reported) from the Azure IoT Hub.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
	_logger.LogInformation("Azure IoT Smart Edge Camera Service starting");

	try
	{
#if AZURE_IOT_HUB_CONNECTION
		_deviceClient = await AzureIoTHubConnection();
#endif
#if AZURE_IOT_HUB_DPS_CONNECTION
		_deviceClient = await AzureIoTHubDpsConnection();
#endif

#if AZURE_DEVICE_PROPERTIES
		_logger.LogTrace("ReportedPropeties upload start");

		TwinCollection reportedProperties = new TwinCollection();

		reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
		reportedProperties["MachineName"] = Environment.MachineName;
		reportedProperties["ApplicationVersion"] = Assembly.GetAssembly(typeof(Program)).GetName().Version;
		reportedProperties["ImageTimerDue"] = _applicationSettings.ImageTimerDue;
		reportedProperties["ImageTimerPeriod"] = _applicationSettings.ImageTimerPeriod;
		reportedProperties["YoloV5ModelPath"] = _applicationSettings.YoloV5ModelPath;

		reportedProperties["PredictionScoreThreshold"] = _applicationSettings.PredictionScoreThreshold;
		reportedProperties["PredictionLabelsOfInterest"] = _applicationSettings.PredictionLabelsOfInterest;
		reportedProperties["PredictionLabelsMinimum"] = _applicationSettings.PredictionLabelsMinimum;

		await _deviceClient.UpdateReportedPropertiesAsync(reportedProperties, stoppingToken);

		_logger.LogTrace("ReportedPropeties upload done");
#endif

		_logger.LogTrace("YoloV5 model setup start");
		_scorer = new YoloScorer<YoloCocoP5Model>(_applicationSettings.YoloV5ModelPath);
		_logger.LogTrace("YoloV5 model setup done");

		_ImageUpdatetimer = new Timer(ImageUpdateTimerCallback, null, _applicationSettings.ImageTimerDue, _applicationSettings.ImageTimerPeriod);

		await _deviceClient.SetMethodHandlerAsync("ImageTimerStart", ImageTimerStartHandler, null);
		await _deviceClient.SetMethodHandlerAsync("ImageTimerStop", ImageTimerStopHandler, null);
		await _deviceClient.SetMethodDefaultHandlerAsync(DefaultHandler, null);

		await _deviceClient.SetDesiredPropertyUpdateCallbackAsync(OnDesiredPropertyChangedAsync, null);

		try
		{
			await Task.Delay(Timeout.Infinite, stoppingToken);
		}
		catch (TaskCanceledException)
		{
			_logger.LogInformation("Application shutown requested");
		}
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "Application startup failure");
	}
	finally
	{
		_deviceClient?.Dispose();
	}

	_logger.LogInformation("Azure IoT Smart Edge Camera Service shutdown");
}

// Lots of other code here

private async Task OnDesiredPropertyChangedAsync(TwinCollection desiredProperties, object userContext)
{
	TwinCollection reportedProperties = new TwinCollection();

	_logger.LogInformation("OnDesiredPropertyChanged handler");

	// NB- This approach does not save the ImageTimerDue or ImageTimerPeriod, a stop/start with return to appsettings.json configuration values. If only
	// one parameter specified other is default from appsettings.json. If timer settings changed I think they won't take
	// effect until next time Timer fires.

	try
	{
		// Check to see if either of ImageTimerDue or ImageTimerPeriod has changed
		if (!desiredProperties.Contains("ImageTimerDue") && !desiredProperties.Contains("ImageTimerPeriod"))
		{
			_logger.LogInformation("OnDesiredPropertyChanged neither ImageTimerDue or ImageTimerPeriod present");
			return;
		}

		TimeSpan imageTimerDue = _applicationSettings.ImageTimerDue;

		// Check that format of ImageTimerDue valid if present
		if (desiredProperties.Contains("ImageTimerDue"))
		{
			if (TimeSpan.TryParse(desiredProperties["ImageTimerDue"].Value, out imageTimerDue))
			{
				reportedProperties["ImageTimerDue"] = imageTimerDue;
			}
			else
			{
				_logger.LogInformation("OnDesiredPropertyChanged ImageTimerDue invalid");
				return;
			}
		}

		TimeSpan imageTimerPeriod = _applicationSettings.ImageTimerPeriod;

		// Check that format of ImageTimerPeriod valid if present
		if (desiredProperties.Contains("ImageTimerPeriod"))
		{
			if (TimeSpan.TryParse(desiredProperties["ImageTimerPeriod"].Value, out imageTimerPeriod))
			{
				reportedProperties["ImageTimerPeriod"] = imageTimerPeriod;
			}
			else
			{
				_logger.LogInformation("OnDesiredPropertyChanged ImageTimerPeriod invalid");
				return;
			}
		}

		_logger.LogInformation("Desired Due:{0} Period:{1}", imageTimerDue, imageTimerPeriod);

		if (!_ImageUpdatetimer.Change(imageTimerDue, imageTimerPeriod))
		{
			_logger.LogInformation("Desired Due:{0} Period:{1} failed", imageTimerDue, imageTimerPeriod);
		}

		await _deviceClient.UpdateReportedPropertiesAsync(reportedProperties);
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "OnDesiredPropertyChangedAsync handler failed");
	}
}

The TwinCollection desiredProperties is checked for ImageTimerDue and ImageTimerPeriod properties and if either of these are present and valid the Timer.Change method is called.

The AzureMLMetSmartEdgeCamera supports both Azure IoT Hub and Azure IoT Central so I have included images from Azure IoT Explorer and my Azure IoT Central Templates.

SmartEdge Camera Device Twin properties in Azure IoT Explorer

When I modified, then saved the Azure IoT Hub Device Twin desired properties JavaScript Object Notation(JSON) in Azure IoT Hub Explorer the method configured with SetDesiredPropertyUpdateCallbackAsync was invoked on the device.

In Azure IoT Central I added two Capabilities to the device template, the time properties ImageTimerDue, and ImageTimerPeriod.

Azure IoT Central SmartEdgeCamera Device template capabilities

I added a View to the template so the two properties could be changed (I didn’t configure either as required)

Azure IoT Central SmartEdgeCamera Device Default view designer

In the “Device Properties”, “Operation Tab” when I changed the ImageTimerDue and/or ImageTimerPeriod there was visual feedback that there was an update in progress.

Azure IoT Central SmartEdgeCamera Device Properties update start

Then on the device the SmartEdgeCameraAzureIoTService the method configured with SetDesiredPropertyUpdateCallbackAsync was invoked on the device.

SmartEdge Camera Console application displaying updated properties

Once the properties have been updated on the device the UpdateReportedPropertiesAsync method is called

Then a message with the updated property values from the device was visible in the telemetry

Azure IoT Central SmartEdgeCamera Device Properties update done

Then finally the “Operation Tab” displayed a visual confirmation that the value(s) had been updated.

Smartish Edge Camera – Azure IoT Readonly Properties

This post builds on my Smartish Edge Camera – Azure IoT Direct Methods post adding a number of read only properties. In this version the application reports the OSVersion, MachineName, ApplicationVersion, ImageTimerDue, ImageTimerPeriod, YoloV5ModelPath, PredictionScoreThreshold, PredictionLabelsOfInterest, and PredictionLabelsMinimum.

Azure IoT Explorer displaying the reported “readonly” property values

The AzureMLMetSmartEdgeCamera application supports both Azure IoT Hub and Azure IoT Central connectivity so I have have covered inspecting the properties with Azure IoT Explorer and adding them to an Azure IoT Central Template.

Azure IoT Central Template Readonly properties

The code populates a TwinCollection then calls UpdateReportedPropertiesAsync to push the properties upto my Azure IoT Hub. (This functionality is not available on all Azure IoT hub Tiers)

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
	_logger.LogInformation("Azure IoT Smart Edge Camera Service starting");

	try
	{
#if AZURE_IOT_HUB_CONNECTION
		_deviceClient = await AzureIoTHubConnection();
#endif
#if AZURE_IOT_HUB_DPS_CONNECTION
		_deviceClient = await AzureIoTHubDpsConnection();
#endif

#if AZURE_DEVICE_PROPERTIES
		_logger.LogTrace("ReportedPropeties upload start");

		TwinCollection reportedProperties = new TwinCollection();

		reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
		reportedProperties["MachineName"] = Environment.MachineName;
		reportedProperties["ApplicationVersion"] = Assembly.GetAssembly(typeof(Program)).GetName().Version;
		reportedProperties["ImageTimerDue"] = _applicationSettings.ImageTimerDue;
		reportedProperties["ImageTimerPeriod"] = _applicationSettings.ImageTimerPeriod;
		reportedProperties["YoloV5ModelPath"] = _applicationSettings.YoloV5ModelPath;

		reportedProperties["PredictionScoreThreshold"] = _applicationSettings.PredictionScoreThreshold;
		reportedProperties["PredictionLabelsOfInterest"] = _applicationSettings.PredictionLabelsOfInterest;
		reportedProperties["PredictionLabelsMinimum"] = _applicationSettings.PredictionLabelsMinimum;

		await _deviceClient.UpdateReportedPropertiesAsync(reportedProperties, stoppingToken);

		_logger.LogTrace("ReportedPropeties upload done");
#endif

		_logger.LogTrace("YoloV5 model setup start");
		_scorer = new YoloScorer<YoloCocoP5Model>(_applicationSettings.YoloV5ModelPath);
		_logger.LogTrace("YoloV5 model setup done");
...

Azure IoT Central Dashboard with readonly properties before UpdateReportedPropertiesAsync called
Azure IoT Central Telemetry displaying property update payloads
Azure IoT Central Dashboard displaying readonly properties

While testing the application I noticed the reported property version was increasing every time I deployed the application. I was retrieving the version information as the application started with AssemblyName.Version

reportedProperties["ApplicationVersion"] = Assembly.GetAssembly(typeof(Program)).GetName().Version;
Visual Studio 2019 Application Package information

I had also configured the Assembly Version in the SmartEdgeCameraAzureIoTService project Package tab to update the assembly build number each time the application was compiled. This was forcing an update of the reported properties version every time the application started

Smartish Edge Camera – Azure IoT Direct Methods

This post builds on my Smartish Edge Camera – Azure IoT Image-Upload post adding two Direct Methods for Starting and Stopping the image capture and processing timer. The AzureMLMetSmartEdgeCamera supports both Azure IoT Hub and Azure IoT Central connectivity.

Azure IoT Explorer invoking a Direct Method

BEWARE – The Direct Method names are case sensitive which regularly trips me up when I use Azure IoT Explorer. If the Direct Method name is unknown a default handler is called, the issue logged and a Hyper Text Transfer Protocol(HTTP) Not Implemented(501) error returned

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
	_logger.LogInformation("Azure IoT Smart Edge Camera Service starting");

	try
	{
#if AZURE_IOT_HUB_CONNECTION
		_deviceClient = await AzureIoTHubConnection();
#endif
#if AZURE_IOT_HUB_DPS_CONNECTION
		_deviceClient = await AzureIoTHubDpsConnection();
#endif

...
		_logger.LogTrace("YoloV5 model setup start");
		_scorer = new YoloScorer<YoloCocoP5Model>(_applicationSettings.YoloV5ModelPath);
		_logger.LogTrace("YoloV5 model setup done");

		_ImageUpdatetimer = new Timer(ImageUpdateTimerCallback, null, _applicationSettings.ImageTimerDue, _applicationSettings.ImageTimerPeriod);

		await _deviceClient.SetMethodHandlerAsync("ImageTimerStart", ImageTimerStartHandler, null);
		await _deviceClient.SetMethodHandlerAsync("ImageTimerStop", ImageTimerStopHandler, null);
		await _deviceClient.SetMethodDefaultHandlerAsync(DefaultHandler, null);
...
		try
		{
			await Task.Delay(Timeout.Infinite, stoppingToken);
		}
		catch (TaskCanceledException)
		{
			_logger.LogInformation("Application shutown requested");
		}
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "Application startup failure");
	}
	finally
	{
		_deviceClient?.Dispose();
	}

	_logger.LogInformation("Azure IoT Smart Edge Camera Service shutdown");
}

private async Task<MethodResponse> ImageTimerStartHandler(MethodRequest methodRequest, object userContext)
{
	_logger.LogInformation("ImageUpdatetimer Start Due:{0} Period:{1}", _applicationSettings.ImageTimerDue, _applicationSettings.ImageTimerPeriod);

	_ImageUpdatetimer.Change(_applicationSettings.ImageTimerDue, _applicationSettings.ImageTimerPeriod);

	return new MethodResponse((short)HttpStatusCode.OK);
}

private async Task<MethodResponse> ImageTimerStopHandler(MethodRequest methodRequest, object userContext)
{
	_logger.LogInformation("ImageUpdatetimer Stop");

	_ImageUpdatetimer.Change(Timeout.Infinite, Timeout.Infinite);

	return new MethodResponse((short)HttpStatusCode.OK);
}

private async Task<MethodResponse> DefaultHandler(MethodRequest methodRequest, object userContext)
{
	_logger.LogInformation("Direct Method default handler Name:{0}", methodRequest.Name);

	return new MethodResponse((short)HttpStatusCode.NotFound);
}

I created an Azure IoT Central Template with two command capabilities. (For more detail see my post TTI V3 Connector Azure IoT Central Cloud to Device(C2D)).

Azure IoT Central Template Direct Method configuration
Azure IoT Central Template Direct Method invocation
Azure Smart Edge Camera console application Start Direct Method call

Initially, I had one long post which covered Direct Methods, Readonly Properties and Updateable Properties but it got too long so I split it into three.

Smartish Edge Camera – Azure IoT Image Upload

This post builds on my Smartish Edge Camera – Azure Storage Service, Azure IoT Hub, and Azure IoT Central projects adding optional camera and marked-up image upload to Azure Blob Storage for Azure IoT Hubs and Azure IoT Central.

Azure IoT Hub – File upload storage account configuration
Azure IoT Central – File upload storage account configuration

The “new improved” process of uploading files to an Azure IoT Hub and Azure IoT Central is surprisingly complex to use and make robust(I think the initial approach with DeviceClient.UploadToBlobAsync which is now “deprecated” was easier to use).

public async Task UploadImage(List<YoloPrediction> predictions, string filepath, string blobpath)
{
	var fileUploadSasUriRequest = new FileUploadSasUriRequest()
	{
		BlobName = blobpath 
	};

	FileUploadSasUriResponse sasUri = await _deviceClient.GetFileUploadSasUriAsync(fileUploadSasUriRequest);

	var blockBlobClient = new BlockBlobClient(sasUri.GetBlobUri());

	var fileUploadCompletionNotification = new FileUploadCompletionNotification()
	{
		// Mandatory. Must be the same value as the correlation id returned in the sas uri response
		CorrelationId = sasUri.CorrelationId,

		IsSuccess = true
	};

	try
	{
		using (FileStream fileStream = File.OpenRead(filepath))
		{
			Response<BlobContentInfo> response = await blockBlobClient.UploadAsync(fileStream); //, blobUploadOptions);

			fileUploadCompletionNotification.StatusCode = response.GetRawResponse().Status;

			if (fileUploadCompletionNotification.StatusCode != ((int)HttpStatusCode.Created))
			{
				fileUploadCompletionNotification.IsSuccess = false;

				fileUploadCompletionNotification.StatusDescription = response.GetRawResponse().ReasonPhrase;
			}
		}
	}
	catch (RequestFailedException ex)
	{
		fileUploadCompletionNotification.StatusCode = ex.Status;

		fileUploadCompletionNotification.IsSuccess = false;

		fileUploadCompletionNotification.StatusDescription = ex.Message;
	}
	finally
	{
		await _deviceClient.CompleteFileUploadAsync(fileUploadCompletionNotification);
	}
}

If there is an object with a label in the PredictionLabelsOfInterest list, the camera and marked-up images can (configured with ImageCameraUpload & ImageMarkedupUpload) be uploaded to an Azure Storage Blob container associated with an Azure IoT Hub/ Azure IoT Central instance.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },

  "Application": {
    "DeviceID": "",
    "ImageTimerDue": "0.00:00:15",
    "ImageTimerPeriod": "0.00:00:30",

    "ImageCameraFilepath": "ImageCamera.jpg",
    "ImageMarkedUpFilepath": "ImageMarkedup.jpg",

    "ImageCameraUpload": false,
    "ImageMarkedupUpload": true,

    "ImageUploadFilepath": "ImageMarkedup.jpg",

    "YoloV5ModelPath": "YoloV5/yolov5s.onnx",

    "PredictionScoreThreshold": 0.7,
    "PredictionLabelsOfInterest": [
      "bicycle",
      "person"
    ],

    "PredictionLabelsMinimum": [
      "bicycle",
      "car",
      "person"
    ],

    "ImageCameraFilenameFormat": "{0:yyyyMMdd}/{0:HHmmss}.jpg"
  },

  "SecurityCamera": {
    "CameraUrl": "",
    "CameraUserName": "",
    "CameraUserPassword": ""
  },

  "RaspberryPICamera": {
    "ProcessWaitForExit": 1000,
    "Rotation": 180
  },

  "AzureIoTHub": {
    "ConnectionString": ""
  },

  "AzureIoTHubDPS": {
    "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
    "IDScope": "",
    "GroupEnrollmentKey": ""
  },

  "AzureStorage": {
    "ImageCameraFilenameFormat": "{0:yyyyMMdd}/camera/{0:HHmmss}.jpg",
    "ImageMarkedUpFilenameFormat": "{0:yyyyMMdd}/markedup/{0:HHmmss}.jpg"
  }
}

The Blob’s path is prefixed with the device id (My Azure Storage Service created an Azure Blob Storage container for each device).

Azure IoT Central SmartEdge Camera devices

The format of the Azure Storage Blob path is configurable(ImageCameraFilenameFormat & ImageMarkedUpFilenameFormat + Universal Coordinated Time(UTC)) so images can be grouped.

Configurable Blob paths in Azure Storage Explorer

After creating a new Azure IoT Hub uploads started failing with an exception and there weren’t a lot of useful search results (April 2022). I found error this was caused by missing or incorrect Azure Storage Account configuration.

Azure IoT Hub Upload application failure logging
{"Message":"{\"errorCode\":400022,\"trackingId\":\"1175af36ec884cc4a54978f77b877a01-G:0-TimeStamp:04/12/2022 10:19:04\",\"message\":\"BadRequest\",\"timestampUtc\":\"2022-04-12T10:19:04.5925999Z\"}","ExceptionMessage":""}

   at Microsoft.Azure.Devices.Client.Transport.HttpClientHelper.<ExecuteAsync>d__23.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Devices.Client.Transport.HttpClientHelper.<PostAsync>d__19`2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.Azure.Devices.Client.Transport.HttpTransportHandler.<GetFileUploadSasUriAsync>d__15.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at devMobile.IoT.MachineLearning.SmartEdgeCameraAzureIoTService.Worker.<UploadImage>d__14.MoveNext() in C:\Users\BrynLewis\source\repos\AzureMLNetSmartEdgeCamera\SmartEdgeCameraAzureIoTService\Worker.cs:line 430
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at devMobile.IoT.MachineLearning.SmartEdgeCameraAzureIoTService.Worker.<ImageUpdateTimerCallback>d__10.MoveNext() in C:\Users\BrynLewis\source\repos\AzureMLNetSmartEdgeCamera\SmartEdgeCameraAzureIoTService\Worker.cs:line 268

While testing the application I noticed an “unexpected” object detected in my backyard…

Unexpected object detection diagnostic logging
Unexpected object detection results marked-up image

The mentalstack/yolov5-net and NuGet have been incredibly useful and MentalStack team have done a marvelous job building and supporting this project. For this project my test-rig consisted of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) and my HP Prodesk 400G4 DM (i7-8700T).

Smartish Edge Camera – Azure IoT Central

This post builds on Smartish Edge Camera – Azure Hub Part 1 using the Azure IoT Hub Device Provisioning Service(DPS) to connect to Azure IoT Central.

The list of object classes is in the YoloCocoP5Model.cs file in the mentalstack/yolov5-net repository.

public override List<YoloLabel> Labels { get; set; } = new List<YoloLabel>()
{
    new YoloLabel { Id = 1, Name = "person" },
    new YoloLabel { Id = 2, Name = "bicycle" },
    new YoloLabel { Id = 3, Name = "car" },
    new YoloLabel { Id = 4, Name = "motorcycle" },
    new YoloLabel { Id = 5, Name = "airplane" },
    new YoloLabel { Id = 6, Name = "bus" },
    new YoloLabel { Id = 7, Name = "train" },
    new YoloLabel { Id = 8, Name = "truck" },
    new YoloLabel { Id = 9, Name = "boat" },
    new YoloLabel { Id = 10, Name = "traffic light" },
    new YoloLabel { Id = 11, Name = "fire hydrant" },
    new YoloLabel { Id = 12, Name = "stop sign" },
    new YoloLabel { Id = 13, Name = "parking meter" },
    new YoloLabel { Id = 14, Name = "bench" },
    new YoloLabel { Id = 15, Name = "bird" },
    new YoloLabel { Id = 16, Name = "cat" },
    new YoloLabel { Id = 17, Name = "dog" },
    new YoloLabel { Id = 18, Name = "horse" },
    new YoloLabel { Id = 19, Name = "sheep" },
    new YoloLabel { Id = 20, Name = "cow" },
    new YoloLabel { Id = 21, Name = "elephant" },
    new YoloLabel { Id = 22, Name = "bear" },
    new YoloLabel { Id = 23, Name = "zebra" },
    new YoloLabel { Id = 24, Name = "giraffe" },
    new YoloLabel { Id = 25, Name = "backpack" },
    new YoloLabel { Id = 26, Name = "umbrella" },
    new YoloLabel { Id = 27, Name = "handbag" },
    new YoloLabel { Id = 28, Name = "tie" },
    new YoloLabel { Id = 29, Name = "suitcase" },
    new YoloLabel { Id = 30, Name = "frisbee" },
    new YoloLabel { Id = 31, Name = "skis" },
    new YoloLabel { Id = 32, Name = "snowboard" },
    new YoloLabel { Id = 33, Name = "sports ball" },
    new YoloLabel { Id = 34, Name = "kite" },
    new YoloLabel { Id = 35, Name = "baseball bat" },
    new YoloLabel { Id = 36, Name = "baseball glove" },
    new YoloLabel { Id = 37, Name = "skateboard" },
    new YoloLabel { Id = 38, Name = "surfboard" },
    new YoloLabel { Id = 39, Name = "tennis racket" },
    new YoloLabel { Id = 40, Name = "bottle" },
    new YoloLabel { Id = 41, Name = "wine glass" },
    new YoloLabel { Id = 42, Name = "cup" },
    new YoloLabel { Id = 43, Name = "fork" },
    new YoloLabel { Id = 44, Name = "knife" },
    new YoloLabel { Id = 45, Name = "spoon" },
    new YoloLabel { Id = 46, Name = "bowl" },
    new YoloLabel { Id = 47, Name = "banana" },
    new YoloLabel { Id = 48, Name = "apple" },
    new YoloLabel { Id = 49, Name = "sandwich" },
    new YoloLabel { Id = 50, Name = "orange" },
    new YoloLabel { Id = 51, Name = "broccoli" },
    new YoloLabel { Id = 52, Name = "carrot" },
    new YoloLabel { Id = 53, Name = "hot dog" },
    new YoloLabel { Id = 54, Name = "pizza" },
    new YoloLabel { Id = 55, Name = "donut" },
    new YoloLabel { Id = 56, Name = "cake" },
    new YoloLabel { Id = 57, Name = "chair" },
    new YoloLabel { Id = 58, Name = "couch" },
    new YoloLabel { Id = 59, Name = "potted plant" },
    new YoloLabel { Id = 60, Name = "bed" },
    new YoloLabel { Id = 61, Name = "dining table" },
    new YoloLabel { Id = 62, Name = "toilet" },
    new YoloLabel { Id = 63, Name = "tv" },
    new YoloLabel { Id = 64, Name = "laptop" },
    new YoloLabel { Id = 65, Name = "mouse" },
    new YoloLabel { Id = 66, Name = "remote" },
    new YoloLabel { Id = 67, Name = "keyboard" },
    new YoloLabel { Id = 68, Name = "cell phone" },
    new YoloLabel { Id = 69, Name = "microwave" },
    new YoloLabel { Id = 70, Name = "oven" },
    new YoloLabel { Id = 71, Name = "toaster" },
    new YoloLabel { Id = 72, Name = "sink" },
    new YoloLabel { Id = 73, Name = "refrigerator" },
    new YoloLabel { Id = 74, Name = "book" },
    new YoloLabel { Id = 75, Name = "clock" },
    new YoloLabel { Id = 76, Name = "vase" },
    new YoloLabel { Id = 77, Name = "scissors" },
    new YoloLabel { Id = 78, Name = "teddy bear" },
    new YoloLabel { Id = 79, Name = "hair drier" },
    new YoloLabel { Id = 80, Name = "toothbrush" }
};

Some of the label choices seem a bit arbitrary(frisbee, surfboard) and American(fire hydrant, baseball bat, baseball glove) It was quite tedious configuring the 80 labels in my Azure IoT Central template.

Azure IoT Central Template with all the YoloV5 labels configured

If there is an object with a label in the PredictionLabelsOfInterest list, a tally of each of the different object classes in the image is sent to an Azure IoT Hub/ Azure IoT Central.

"Application": {
  "DeviceID": "",
  "ImageTimerDue": "0.00:00:15",
  "ImageTimerPeriod": "0.00:00:30",

  "ImageCameraFilepath": "ImageCamera.jpg",
  "ImageMarkedUpFilepath": "ImageMarkedup.jpg",

  "YoloV5ModelPath": "YoloV5/yolov5s.onnx",

  "PredictionScoreThreshold": 0.7,
  "PredictionLabelsOfInterest": [
    "bicycle",
    "person"
  ],
  "PredictionLabelsMinimum": [
    "bicycle",
    "car",
    "person"
  ]
}
My backyard just after the car left (the dry patch in shingle on the right)
Smartish Edge Camera Service console just after car left
Smartish Edge Camera Azure IoT Central graphs showing missing data points

After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period is started.

private async void ImageUpdateTimerCallback(object state)
{
	DateTime requestAtUtc = DateTime.UtcNow;

	// Just incase - stop code being called while photo already in progress
	if (_cameraBusy)
	{
		return;
	}
	_cameraBusy = true;

	_logger.LogInformation("Image processing start");

	try
	{
#if CAMERA_RASPBERRY_PI
		RaspberryPIImageCapture();
#endif
#if CAMERA_SECURITY
		SecurityCameraImageCapture();
#endif
		List<YoloPrediction> predictions;

		using (Image image = Image.FromFile(_applicationSettings.ImageCameraFilepath))
		{
			_logger.LogTrace("Prediction start");
			predictions = _scorer.Predict(image);
			_logger.LogTrace("Prediction done");

			OutputImageMarkup(image, predictions, _applicationSettings.ImageMarkedUpFilepath);
		}

		if (_logger.IsEnabled(LogLevel.Trace))
		{
			_logger.LogTrace("Predictions {0}", predictions.Select(p => new { p.Label.Name, p.Score }));
		}

		var predictionsValid = predictions.Where(p => p.Score >= _applicationSettings.PredictionScoreThreshold).Select(p => p.Label.Name);

		// Count up the number of each class detected in the image
		var predictionsTally = predictionsValid.GroupBy(p => p)
				.Select(p => new
				{
					Label = p.Key,
					Count = p.Count()
				});

		if (_logger.IsEnabled(LogLevel.Information))
		{
			_logger.LogInformation("Predictions tally before {0}", predictionsTally.ToList());
		}

		// Add in any missing counts the cloudy side is expecting
		if (_applicationSettings.PredictionLabelsMinimum != null)
		{
			foreach( String label in _applicationSettings.PredictionLabelsMinimum)
			{
				if (!predictionsTally.Any(c=>c.Label == label ))
				{
					predictionsTally = predictionsTally.Append(new {Label = label, Count = 0 });
				}
			}
		}

		if (_logger.IsEnabled(LogLevel.Information))
		{
			_logger.LogInformation("Predictions tally after {0}", predictionsTally.ToList());
		}

		if ((_applicationSettings.PredictionLabelsOfInterest == null) || (predictionsValid.Select(c => c).Intersect(_applicationSettings.PredictionLabelsOfInterest, StringComparer.OrdinalIgnoreCase).Any()))
		{
			JObject telemetryDataPoint = new JObject();

			foreach (var predictionTally in predictionsTally)
			{
				telemetryDataPoint.Add(predictionTally.Label, predictionTally.Count);
			}

			using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
			{
				message.Properties.Add("iothub-creation-time-utc", requestAtUtc.ToString("s", CultureInfo.InvariantCulture));

				await _deviceClient.SendEventAsync(message);
			}
		}
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "Camera image download, post processing, or telemetry failed");
	}
	finally
	{
		_cameraBusy = false;
	}

	TimeSpan duration = DateTime.UtcNow - requestAtUtc;

	_logger.LogInformation("Image processing done {0:f2} sec", duration.TotalSeconds);
}

Using some Language Integrated Query (LINQ) code any predictions with a score < PredictionScoreThreshold are discarded. A count of the instances of each class is generated with some more LINQ code.

The PredictionLabelsMinimum(optional) is then used to add additional labels with a count of 0 to PredictionsTally so there are no missing datapoints. This is specifically for Azure IoT Central Dashboard so the graph lines are continuous.

Smartish Edge Camera Service console just after put bike in-front of the garage

If any of the list of valid predictions labels is in the PredictionLabelsOfInterest list (if the PredictionLabelsOfInterest is empty any label is a label of interest) the list of prediction class counts is used to populate a Newtonsoft JObject which is serialised to generate a Java Script Object Notation(JSON) Azure IoT Hub message payload.

The “automagic” graph scaling can be sub-optimal

The mentalstack/yolov5-net and NuGet have been incredibly useful and MentalStack team have done a marvelous job building and supporting this project.

The test-rig consisted of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) and my HP Prodesk 400G4 DM (i7-8700T).

TTI V3 Connector Azure IoT Central Device Provisioning Service(DPS) support

The TTI Connector supports the Azure IoT Hub Device Provisioning Service(DPS) which is required (it is possible to provision individual devices but this intended for small deployments or testing) for Azure IoT Central applications. The TTI Connector implementation also supports Azure IoT Central Digital Twin Definition Language (DTDL V2) for “automagic” device provisioning.

The first step was to configure and Azure IoT Central enrollment group (ensure “Automatically connect devices in this group” is on for “zero touch” provisioning) and copy the IDScope and Group Enrollment key to the TTI Connector configuration

RAK3172 Enrollment Group creation
Azure IoT Hub Device Provisioning Service configuration

I then created an Azure IoT Central template for my RAK3172 breakout board based.Net Core powered test device.

{
    "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7;1",
    "@type": "Interface",
    "contents": [
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:temperature_0;1",
            "@type": [
                "Telemetry",
                "Temperature"
            ],
            "displayName": {
                "en": "Temperature"
            },
            "name": "temperature_0",
            "schema": "double",
            "unit": "degreeCelsius"
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:relative_humidity_0;1",
            "@type": [
                "Telemetry",
                "RelativeHumidity"
            ],
            "displayName": {
                "en": "Humidity"
            },
            "name": "relative_humidity_0",
            "schema": "double",
            "unit": "percent"
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:value_0;1",
            "@type": "Command",
            "displayName": {
                "en": "Temperature OOB alert minimum"
            },
            "name": "value_0",
            "request": {
                "@type": "CommandPayload",
                "displayName": {
                    "en": "Minimum"
                },
                "name": "value_0",
                "schema": "double"
            },
            "durable": true
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:value_1;1",
            "@type": "Command",
            "displayName": {
                "en": "Temperature OOB alert maximum"
            },
            "name": "value_1",
            "request": {
                "@type": "CommandPayload",
                "displayName": {
                    "en": "Maximum"
                },
                "name": "value_1",
                "schema": "double"
            },
            "durable": true
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:TemperatureOOBAlertMinimumAndMaximum;1",
            "@type": "Command",
            "displayName": {
                "en": "Temperature OOB alert minimum and maximum"
            },
            "name": "TemperatureOOBAlertMinimumAndMaximum",
            "request": {
                "@type": "CommandPayload",
                "displayName": {
                    "en": "Alert Temperature"
                },
                "name": "AlertTemperature",
                "schema": {
                    "@type": "Object",
                    "displayName": {
                        "en": "Object"
                    },
                    "fields": [
                        {
                            "displayName": {
                                "en": "minimum"
                            },
                            "name": "value_0",
                            "schema": "double"
                        },
                        {
                            "displayName": {
                                "en": "maximum"
                            },
                            "name": "value_1",
                            "schema": "double"
                        }
                    ]
                }
            },
            "durable": true
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:value_2;1",
            "@type": "Command",
            "displayName": {
                "en": "Fan"
            },
            "name": "value_2",
            "request": {
                "@type": "CommandPayload",
                "displayName": {
                    "en": "On"
                },
                "name": "value_3",
                "schema": {
                    "@type": "Enum",
                    "displayName": {
                        "en": "Enum"
                    },
                    "enumValues": [
                        {
                            "displayName": {
                                "en": "On"
                            },
                            "enumValue": 1,
                            "name": "On"
                        },
                        {
                            "displayName": {
                                "en": "Off"
                            },
                            "enumValue": 0,
                            "name": "Off"
                        }
                    ],
                    "valueSchema": "integer"
                }
            },
            "durable": true
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:LightsGoOn;1",
            "@type": "Command",
            "displayName": {
                "en": "LightsGoOn"
            },
            "name": "LightsGoOn",
            "durable": true
        },
        {
            "@id": "dtmi:ttnv3connectorclient:RASK3172Breakout1c7:LightsGoOff;1",
            "@type": "Command",
            "displayName": {
                "en": "LightsGoOff"
            },
            "name": "LightsGoOff",
            "durable": true
        }
    ],
    "displayName": {
        "en": "RASK3172 Breakout"
    },
    "@context": [
        "dtmi:iotcentral:context;2",
        "dtmi:dtdl:context;2"
    ]
}

The Device Template @Id can also be set for a TTI application using an optional dtdlmodelid which is specified the the TTI Connector configuration.

Azure Smartish Edge Camera – The basics

This project builds on my ML.Net YoloV5 + Camera + GPIO on ARM64 Raspberry PI with the addition of basic support for Azure IoT Hubs, the Azure IoT Hub Device Provisioning Service(DPS), and Azure IoT Central.

My backyard test-rig has consists of a Unv ADZK-10 Security Camera, Power over Ethernet(PoE) module, and an ASUS PE100A.

Backyard test-rig

The application can be compiled with support for Azure IoT Connection strings or the Device Provisioning Service(DPS). The appsetings.json file has configuration options for Azure IoT Hub connection string or DPS Global Device Endpoint+ScopeID+Group Enrollment key.

{
  "ApplicationSettings": {
    "DeviceId": "NotTheEdgeCamera",

    "ImageTimerDue": "0.00:00:15",
    "ImageTimerPeriod": "0.00:00:30",

    "CameraUrl": "http://10.0.0.55:85/images/snapshot.jpg",
    "CameraUserName": ",,,",
    "CameraUserPassword": "...",

    "ButtonPinNumer": 6,
    "LedPinNumer": 5,

    "InputImageFilenameLocal": "InputLatest.jpg",
    "OutputImageFilenameLocal": "OutputLatest.jpg",

    "ProcessWaitForExit": 10000,

    "YoloV5ModelPath": "Assets/YoloV5/yolov5s.onnx",

    "PredicitionScoreThreshold": 0.5,

    "AzureIoTHubConnectionString": "...",

    "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
    "AzureIoTHubDpsIDScope": "...",
    "AzureIoTHubDpsGroupEnrollmentKey": "..."
  }
}

After the You Only Look Once(YOLOV5)+ML.Net+Open Neural Network Exchange(ONNX) plumbing has loaded a timer with a configurable due time and period is started.

private static async void ImageUpdateTimerCallback(object state)
{
	DateTime requestAtUtc = DateTime.UtcNow;

	// Just incase - stop code being called while photo already in progress
	if (_cameraBusy)
	{
		return;
	}
	_cameraBusy = true;

	Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing start");

	try
	{
#if SECURITY_CAMERA
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download start");
		SecurityCameraImageCapture();
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Security Camera Image download done");
#endif

#if RASPBERRY_PI_CAMERA
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture start");
		RaspberryPICameraImageCapture();
		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Raspberry PI Image capture done");
#endif

		List<YoloPrediction> predictions;

		// Process the image on local file system
		using (Image image = Image.FromFile(_applicationSettings.InputImageFilenameLocal))
		{
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing start");
			predictions = _scorer.Predict(image);
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} YoloV5 inferencing done");

#if OUTPUT_IMAGE_MARKUP
			using (Graphics graphics = Graphics.FromImage(image))
			{
				Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup start");

				foreach (var prediction in predictions) // iterate predictions to draw results
				{
					double score = Math.Round(prediction.Score, 2);

					graphics.DrawRectangles(new Pen(prediction.Label.Color, 1), new[] { prediction.Rectangle });

					var (x, y) = (prediction.Rectangle.X - 3, prediction.Rectangle.Y - 23);

					graphics.DrawString($"{prediction.Label.Name} ({score})", new Font("Consolas", 16, GraphicsUnit.Pixel), new SolidBrush(prediction.Label.Color), new PointF(x, y));
				}

				image.Save(_applicationSettings.OutputImageFilenameLocal);

				Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss:fff} Image markup done");
			}
#endif
		}

#if AZURE_IOT_HUB_CONNECTION || AZURE_IOT_HUB_DPS_CONNECTION
		await AzureIoTHubTelemetry(requestAtUtc, predictions);
#endif
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Camera image download, post procesing, image upload, or telemetry failed {ex.Message}");
	}
	finally
	{
		_cameraBusy = false;
	}

	TimeSpan duration = DateTime.UtcNow - requestAtUtc;

	Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image processing done {duration.TotalSeconds:f2} sec");
	Console.WriteLine();
}

In the ImageUpdateTimerCallback method a camera image is captured (Raspberry Pi Camera Module 2 or Unv ADZK-10 Security Camera) and written to the local file system.

SSH Connection to Azure PE100 running Smartish Camera application

The YoloV5 model ML.Net support library then loads the image and processes the prediction output (can be inspected with Netron) generating list of objects that have been detected, their Minimum Bounding Rectangle(MBR) and class.

public static async Task AzureIoTHubTelemetry(DateTime requestAtUtc, List<YoloPrediction> predictions)
{
	JObject telemetryDataPoint = new JObject();

	foreach (var predictionTally in predictions.Where(p => p.Score >= _applicationSettings.PredicitionScoreThreshold).GroupBy(p => p.Label.Name)
					.Select(p => new
					{
						Label = p.Key,
						Count = p.Count()
					}))
	{
		Console.WriteLine("  {0} {1}", predictionTally.Label, predictionTally.Count);

		telemetryDataPoint.Add(predictionTally.Label, predictionTally.Count);
	}

	try
	{
		using (Message message = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryDataPoint))))
		{
			message.Properties.Add("iothub-creation-time-utc", requestAtUtc.ToString("s", CultureInfo.InvariantCulture));

			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information start");
			await _deviceClient.SendEventAsync(message);
			Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync prediction information finish");
		}
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} AzureIoTHubClient SendEventAsync cow counting failed {ex.Message}");
	}
}

The list of predictions is post processed with a Language Integrated Query(LINQ) which filters out predictions with a score below a configurable threshold and returns a count of each class.

My backyard from the deck

The aggregated YoloV5 prediction results are then uploaded to an Azure IoT Hub or Azure IoT Central

Azure IoT Explorer Displaying message payloads from the Smartish Edge Camera
Azure IoT Central displaying message payloads from the Smartish Edge Camera

TTI V3 Connector Azure IoT Central Cloud to Device(C2D)

Handling Cloud to Device(D2C) Azure IoT Central messages (The Things Industries(TTI) downlink) is a bit more complex than Device To Cloud(D2C) messaging. The format of the command messages is reasonably well documented and I have already explored in detail with basic telemetry, basic commands, request commands, and The Things Industries Friendly commands and Digital Twin Definition Language(DTDL) support.

public class IoTHubApplicationSetting
{
	public string DtdlModelId { get; set; }
}

public class IoTHubSettings
{
	public string IoTHubConnectionString { get; set; } = string.Empty;

	public Dictionary<string, IoTHubApplicationSetting> Applications { get; set; }
}


public class DeviceProvisiongServiceApplicationSetting
{
	public string DtdlModelId { get; set; } = string.Empty;

	public string GroupEnrollmentKey { get; set; } = string.Empty;
}

public class DeviceProvisiongServiceSettings
{
	public string IdScope { get; set; } = string.Empty;

	public Dictionary<string, DeviceProvisiongServiceApplicationSetting> Applications { get; set; }
}


public class IoTCentralMethodSetting
{
	public byte Port { get; set; } = 0;

	public bool Confirmed { get; set; } = false;

	public Models.DownlinkPriority Priority { get; set; } = Models.DownlinkPriority.Normal;

	public Models.DownlinkQueue Queue { get; set; } = Models.DownlinkQueue.Replace;
}

public class IoTCentralSetting
{
	public Dictionary<string, IoTCentralMethodSetting> Methods { get; set; }
}

public class AzureIoTSettings
{
	public IoTHubSettings IoTHub { get; set; }

	public DeviceProvisiongServiceSettings DeviceProvisioningService { get; set; }

	public IoTCentralSetting IoTCentral { get; set; }
}

Azure IoT Central appears to have no support for setting message properties so the LoRaWAN port, confirmed flag, priority, and queuing so these a retrieved from configuration.

Azure Function Configuration
Models.Downlink downlink;
Models.DownlinkQueue queue;

string payloadText = Encoding.UTF8.GetString(message.GetBytes()).Trim();

if (message.Properties.ContainsKey("method-name"))
{
	#region Azure IoT Central C2D message processing
	string methodName = message.Properties["method-name"];

	if (string.IsNullOrWhiteSpace(methodName))
	{
		_logger.LogWarning("Downlink-DeviceID:{0} MessagedID:{1} LockToken:{2} method-name property empty", receiveMessageHandlerContext.DeviceId, message.MessageId, message.LockToken);

		await deviceClient.RejectAsync(message);
		return;
	}

	// Look up the method settings to get confirmed, port, priority, and queue
	if ((_azureIoTSettings == null) || (_azureIoTSettings.IoTCentral == null) || !_azureIoTSettings.IoTCentral.Methods.TryGetValue(methodName, out IoTCentralMethodSetting methodSetting))
	{
		_logger.LogWarning("Downlink-DeviceID:{0} MessagedID:{1} LockToken:{2} method-name:{3} has no settings", receiveMessageHandlerContext.DeviceId, message.MessageId, message.LockToken, methodName);
							
		await deviceClient.RejectAsync(message);
		return;
	}

	downlink = new Models.Downlink()
	{
		Confirmed = methodSetting.Confirmed,
		Priority = methodSetting.Priority,
		Port = methodSetting.Port,
		CorrelationIds = AzureLockToken.Add(message.LockToken),
	};

	queue = methodSetting.Queue;

	// Check to see if special case for Azure IoT central command with no request payload
	if (payloadText.IsPayloadEmpty())
	{
		downlink.PayloadRaw = "";
	}

	if (!payloadText.IsPayloadEmpty())
	{
		if (payloadText.IsPayloadValidJson())
		{
			downlink.PayloadDecoded = JToken.Parse(payloadText);
			}
		else
		{
			downlink.PayloadDecoded = new JObject(new JProperty(methodName, payloadText));
		}
	}

	logger.LogInformation("Downlink-IoT Central DeviceID:{0} Method:{1} MessageID:{2} LockToken:{3} Port:{4} Confirmed:{5} Priority:{6} Queue:{7}",
		receiveMessageHandlerContext.DeviceId,
		methodName,
		message.MessageId,
		message.LockToken,
		downlink.Port,
		downlink.Confirmed,
		downlink.Priority,
		queue);
	#endregion
}

The reboot command payload only contains an “@” so the TTTI payload will be empty, the minimum and maximum command payloads will contain only a numeric value which is added to the decoded payload with the method name, the combined minimum and maximum command has a JSON payload which is “grafted” into the decoded payload.

Azure IoT Central Device Template

Azure Device Provisioning Service(DPS) when transient isn’t

After some updates to my Device Provisioning Service(DPS) code the RegisterAsync method was exploding with an odd exception.

TTI Webhook Integration running in desktop emulator

In the Visual Studio 2019 Debugger the exception text was “IsTransient = true” so I went and made a coffee and tried again.

Visual Studio 2019 Quickwatch displaying short from error message

The call was still failing so I dumped out the exception text so I had some key words to search for

Microsoft.Azure.Devices.Provisioning.Client.ProvisioningTransportException: AMQP transport exception
 ---> System.UnauthorizedAccessException: Sys
   at Microsoft.Azure.Amqp.ExceptionDispatcher.Throw(Exception exception)
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.HandleTransportOpened(IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.OnTransportOpenCompete(IAsyncResult result)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.AmqpClientConnection.OpenAsync(TimeSpan timeout, Boolean useWebSocket, X509Certificate2 clientCert, IWebProxy proxy, RemoteCertificateValidationCallback remoteCerificateValidationCallback)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, TimeSpan timeout, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, CancellationToken cancellationToken)
   at devMobile.IoT.TheThingsIndustries.AzureIoTHub.Integration.Uplink(HttpRequestData req, FunctionContext executionContext) in C:\Users\BrynLewis\source\repos\TTIV3AzureIoTConnector\TTIV3WebHookAzureIoTHubIntegration\TTIUplinkHandler.cs:line 245

I tried a lot of keywords and went and looked at the source code on github

One of the many keyword searches

Another of the many keyword searches

I then tried another program which did used the Device provisioning Service and it worked first time so it was something wrong with the code.

using (var securityProvider = new SecurityProviderSymmetricKey(deviceId, deviceKey, null))
{
	using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
	{
		DeviceRegistrationResult result;

		ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
			Constants.AzureDpsGlobalDeviceEndpoint,
			 dpsApplicationSetting.GroupEnrollmentKey, <<= Should be _azureIoTSettings.DeviceProvisioningService.IdScope,
			securityProvider,
			transport);

		try
		{
				result = await provClient.RegisterAsync();
		}
		catch (ProvisioningTransportException ex)
		{
			logger.LogInformation(ex, "Uplink-DeviceID:{0} RegisterAsync failed IDScope and/or GroupEnrollmentKey invalid", deviceId);

			return req.CreateResponse(HttpStatusCode.Unauthorized);
		}

		if (result.Status != ProvisioningRegistrationStatusType.Assigned)
		{
			_logger.LogError("Uplink-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);

			return req.CreateResponse(HttpStatusCode.FailedDependency);
		}

		IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

		deviceClient = DeviceClient.Create(result.AssignedHub, authentication, TransportSettings);

		await deviceClient.OpenAsync();

		logger.LogInformation("Uplink-DeviceID:{0} Azure IoT Hub connected (Device Provisioning Service)", deviceId);
	}
}

I then carefully inspected my source code and worked back through the file history and realised I had accidentally replaced the IDScope with the GroupEnrollment setting so it was never going to work i.e. IsTransient != true. So, for the one or two other people who get this error message check your IDScope and GroupEnrollment key make sure they are the right variables and that values they contain are correct.

TTI V3 Connector Azure IoT Central Device to Cloud(D2C)

This post is largely about adapting the output of The Things Industries(TTI) MyDevices Cayenne Low Power Protocol(LPP) payload formatter so that it can be injested by Azure IoT Central. The Azure function for processing TTI Uplink messages first deserialises the JSON payload discarding any LoRaWAN control messages and messages with empty payloads.

[Function("Uplink")]
public async Task<HttpResponseData> Uplink([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, FunctionContext executionContext)
{
	Models.PayloadUplink payload;
	var logger = executionContext.GetLogger("Queued");

	// Wrap all the processing in a try\catch so if anything blows up we have logged it.
	try
	{
		string payloadText = await req.ReadAsStringAsync();

		try
		{
			payload = JsonConvert.DeserializeObject<Models.PayloadUplink>(payloadText);
		}
		catch(JsonException ex)
		{
			logger.LogInformation(ex, "Uplink-Payload Invalid JSON:{0}", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		if (payload == null)
		{
			logger.LogInformation("Uplink-Payload invalid:{0}", payloadText);

			return req.CreateResponse(HttpStatusCode.BadRequest);
		}

		string applicationId = payload.EndDeviceIds.ApplicationIds.ApplicationId;
		string deviceId = payload.EndDeviceIds.DeviceId;

		if ((payload.UplinkMessage.Port == null) || (!payload.UplinkMessage.Port.HasValue) || (payload.UplinkMessage.Port.Value == 0))
		{
			logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Payload Raw:{2} Control message", applicationId, deviceId, payload.UplinkMessage.PayloadRaw);

			return req.CreateResponse(HttpStatusCode.UnprocessableEntity);
		}

		int port = payload.UplinkMessage.Port.Value;

		logger.LogInformation("Uplink-ApplicationID:{0} DeviceID:{1} Port:{2} Payload Raw:{3}", applicationId, deviceId, port, payload.UplinkMessage.PayloadRaw);

		if (!_DeviceClients.TryGetValue(deviceId, out DeviceClient deviceClient))
		{
...		
		}

		JObject telemetryEvent = new JObject
		{
			{ "ApplicationID", applicationId },
			{ "DeviceID", deviceId },
			{ "Port", port },
			{ "Simulated", payload.Simulated },
			{ "ReceivedAtUtc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture) },
			{ "PayloadRaw", payload.UplinkMessage.PayloadRaw }
		};

		// If the payload has been decoded by payload formatter, put it in the message body.
		if (payload.UplinkMessage.PayloadDecoded != null)
		{
			EnumerateChildren(telemetryEvent, payload.UplinkMessage.PayloadDecoded);
		}

		// Send the message to Azure IoT Hub
		using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
		{
			// Ensure the displayed time is the acquired time rather than the uploaded time. 
			ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture));
			ioTHubmessage.Properties.Add("ApplicationId", applicationId);
			ioTHubmessage.Properties.Add("DeviceEUI", payload.EndDeviceIds.DeviceEui);
			ioTHubmessage.Properties.Add("DeviceId", deviceId);
			ioTHubmessage.Properties.Add("port", port.ToString());
			ioTHubmessage.Properties.Add("Simulated", payload.Simulated.ToString());

			await deviceClient.SendEventAsync(ioTHubmessage);

			logger.LogInformation("Uplink-DeviceID:{0} SendEventAsync success", payload.EndDeviceIds.DeviceId);
		}
	}
	catch (Exception ex)
	{
		logger.LogError(ex, "Uplink-Message processing failed");

		return req.CreateResponse(HttpStatusCode.InternalServerError);
	}

	return req.CreateResponse(HttpStatusCode.OK);
}

If the message has been successfully decoded by a payload formatter the PayloadDecoded contents will be “grafted” into the Azure IoT Central Telemetry message.

TTI JSON GPS position format

The Azure IoT Central Location Telemetry messages have a slightly different format to the output of the TTI LPP Payload formatter so the payload has to be “post processed”.

private void EnumerateChildren(JObject jobject, JToken token)
{
	if (token is JProperty property)
	{
		if (token.First is JValue)
		{
			// Temporary dirty hack for Azure IoT Central compatibility
			if (token.Parent is JObject possibleGpsProperty)
			{
				// TODO Need to check if similar approach necessary accelerometer and gyro LPP payloads
				if (possibleGpsProperty.Path.StartsWith("GPS_", StringComparison.OrdinalIgnoreCase))
				{
					if (string.Compare(property.Name, "Latitude", true) == 0)
					{
						jobject.Add("lat", property.Value);
					}
					if (string.Compare(property.Name, "Longitude", true) == 0)
					{
						jobject.Add("lon", property.Value);
					}
					if (string.Compare(property.Name, "Altitude", true) == 0)
					{
						jobject.Add("alt", property.Value);
					}
				}
			}
			jobject.Add(property.Name, property.Value);
		}
		else
		{
			JObject parentObject = new JObject();
			foreach (JToken token2 in token.Children())
			{
				EnumerateChildren(parentObject, token2);
				jobject.Add(property.Name, parentObject);
			}
		}
	}
	else
	{
		foreach (JToken token2 in token.Children())
		{
			EnumerateChildren(jobject, token2);
		}
	}
}

I may have to extend this method for other LPP datatypes

“Post processed” TTI JSON GPS Position data suitable for Azure IoT Central

To test the telemetry message JSON I created an Azure IoT Central Device Template which had a “capability type” of Location.

Azure IoT Central Device Template with Location Capability

For initial development and testing I ran the function application in the desktop emulator and simulated TTI webhook calls with Telerik Fiddler and modified sample payloads. After some issues with iothub-creation-time-utc decoded telemetry messages were displayed in the Device Raw Data tab

Azure IoT Central Device Raw Data tab with successfully decoded GPS location payloads
Azure IoT Central map displaying with device location highlighted

This post uses a lot of the work done for my The Things Network V2 integration. I also found the first time a device connected to the Azure IoT Central Azure IoT hub (using the Azure IoT Central Device Provisioning Service(DPS) to get the connection string) there was always an exception.

Microsoft.Azure.Devices.Client.Exceptions.IotHubException: error(condition:com.microsoft:connection-closed-on-new-connection,description:Backend initiated disconnection.

TTI V3 Gateway Azure IoT Central first call exception

This exception occurs when the SetMethodDefaultHandlerAsync method is called which is a bit odd. This exception does not occur when I use Device Provisioning Service(DPS) and Azure IoT Hub instances I have provisioned.