Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
There are now separate Data Transfer Objects(DTO) for the uplink and queue message payloads mainly, because the UplinkPayloadQueueDto has additional fields for the client (based on the x-api-key) and when the webhook was called.
public class UplinkPayloadQueueDto
{
public ulong PacketId { get; set; }
public byte DeviceType { get; set; }
public uint DeviceId { get; set; }
public ushort UserApplicationId { get; set; }
public uint OrganizationId { get; set; }
public string Data { get; set; } = string.Empty;
public byte Length { get; set; }
public int Status { get; set; }
public DateTime SwarmHiveReceivedAtUtc { get; set; }
public DateTime UplinkWebHookReceivedAtUtc { get; set; }
public string Client { get; set; } = string.Empty;
}
public class UplinkPayloadWebDto
{
public ulong PacketId { get; set; }
public byte DeviceType { get; set; }
public uint DeviceId { get; set; }
public ushort UserApplicationId { get; set; }
public uint OrganizationId { get; set; }
public string Data { get; set; } = string.Empty;
[Range(Constants.PayloadLengthMinimum, Constants.PayloadLengthMaximum)]
public byte Len { get; set; }
public int Status { get; set; }
public DateTime HiveRxTime { get; set; }
}
I did consider using AutoMapper to copy the values from the UplinkPayloadWebDto to the UplinkPayloadQueueDto but the additional complexity/configuration required for one mapping wasn’t worth it.
The UplinkController has a single POST method, which has a JSON payload(FromBody) and a single header (FromHeader) “x-api-key” which is to secure the method and identify the caller.
[HttpPost]
public async Task<IActionResult> Post([FromHeader(Name = "x-api-key")] string xApiKeyValue, [FromBody] Models.UplinkPayloadWebDto payloadWeb)
{
if (!_applicationSettings.XApiKeys.TryGetValue(xApiKeyValue, out string apiKeyName))
{
_logger.LogWarning("Authentication unsuccessful X-API-KEY value:{xApiKeyValue}", xApiKeyValue);
return this.Unauthorized("Unauthorized client");
}
_logger.LogInformation("Authentication successful X-API-KEY value:{apiKeyName}", apiKeyName);
// Could of used AutoMapper but didn't seem worth it for one place
Models.UplinkPayloadQueueDto payloadQueue = new()
{
PacketId = payloadWeb.PacketId,
DeviceType = payloadWeb.DeviceType,
DeviceId = payloadWeb.DeviceId,
UserApplicationId = payloadWeb.UserApplicationId,
OrganizationId = payloadWeb.OrganizationId,
Data = payloadWeb.Data,
Length = payloadWeb.Len,
Status = payloadWeb.Status,
SwarmHiveReceivedAtUtc = payloadWeb.HiveRxTime,
UplinkWebHookReceivedAtUtc = DateTime.UtcNow,
Client = apiKeyName,
};
_logger.LogInformation("SendAsync queue name:{QueueName}", _applicationSettings.QueueName);
QueueClient queueClient = _queueServiceClient.GetQueueClient(_applicationSettings.QueueName);
await queueClient.SendMessageAsync(Convert.ToBase64String(JsonSerializer.SerializeToUtf8Bytes(payloadQueue)));
return this.Ok();
}
This post could have been much longer with more screen grabs and code snippets, so this is the “highlights package”. This post took a lot longer than I expected as building, testing locally, then deploying the different implementations was time consuming.
Swarm Space Connector Functions Projects
I built the projects to investigate the different options taking into account reliability, robustness, amount of code, performance (I think slow startup could be a problem). The code is very “plain” I used the default options, no copyright notices, default formatting, context sensitive error messages were used to add any required “using” statements, libraries etc.
The desktop emulator hosting the six functions
I also deployed the Azure Functions and ASP .NET CoreWebAPI application to check there were no difference (beyond performance) in the way they worked. I included a “default” function (generated by the new project wizard) for reference while I was building the others.
The function application with six functions in deployed to Azure
namespace WebhookHttpTrigger
{
public static class TypedAutomagic
{
[FunctionName("TypedAutomagic")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] UplinkPayload payload,
ILogger log)
{
log.LogInformation($"C# HTTP trigger function typed TypedAutomagic UplinkPayload processed a request PacketId:{payload.PacketId}");
return new OkObjectResult("Hello, This HTTP triggered automagic function executed successfully.");
}
}
}
Successful execution of TypedAutomagic function
The “TypedAutomagic” implementation also detected when the JSON property values in the payload couldn’t be deserialised successfully, but if the hiveRxTime was invalid the value was set to 1/1/0001 12:00:00 am.
namespace WebhookHttpTrigger
{
public static class TypedDeserializeObjectAnnotations
{
[FunctionName("TypedDeserializeObjectAnnotations")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] string httpPayload,
ILogger log)
{
UplinkPayload uplinkPayload;
try
{
uplinkPayload = JsonConvert.DeserializeObject<UplinkPayload>(httpPayload);
}
catch (Exception ex)
{
log.LogWarning(ex, "JsonConvert.DeserializeObject failed");
return new BadRequestObjectResult(ex.Message);
}
var context = new ValidationContext(uplinkPayload, serviceProvider: null, items: null);
var results = new List<ValidationResult>();
var isValid = Validator.TryValidateObject(uplinkPayload, context, results,true);
if (!isValid)
{
log.LogWarning("Validator.TryValidateObject failed results:{results}", results);
return new BadRequestObjectResult(results);
}
log.LogInformation($"C# HTTP trigger function typed DeserializeObject UplinkPayload processed a request PacketId:{uplinkPayload.PacketId}");
return new OkObjectResult("Hello, This HTTP triggered DeserializeObject function executed successfully.");
}
}
}
I built an ASP .NET CoreWebAPI version with two uplink method implementations, one which used dependency injection (DI) and the other that didn’t. I also added code to validate the deserialisation of HiveRxTimeUtc.
...
[HttpPost]
public async Task<IActionResult> Post([FromBody] UplinkPayload payload)
{
// Check that the post data is good
if (!this.ModelState.IsValid)
{
_logger.LogWarning("QueuedController validation failed {0}", this.ModelState.ToString());
return this.BadRequest(this.ModelState);
}
if ( payload.HiveRxTimeUtc == DateTime.MinValue)
{
_logger.LogWarning("HiveRxTimeUtc validation failed");
return this.BadRequest();
}
try
{
QueueClient queueClient = new QueueClient(_configuration.GetConnectionString("AzureWebApi"), "uplink");
//await queueClient.CreateIfNotExistsAsync();
await queueClient.SendMessageAsync(Convert.ToBase64String(JsonSerializer.SerializeToUtf8Bytes(payload)));
}
catch (Exception ex)
{
_logger.LogError(ex,"Unable to open/create queue or send message", ex);
return this.Problem("Unable to open queue (creating if it doesn't exist) or send message", statusCode: 500, title: "Uplink payload not sent");
}
return this.Ok();
}
Swarm Space Azure IoT Connector Identity Translation Gateway Architecture
The new approach uses most of the existing building blocks but adds an Azure HTTP Trigger which receives the Swarm Space Bumble bee hive Webhook Delivery Method calls and writes them to an Azure Storage Queue.
Swarm Space Bumble bee hive Web Hook Delivery method
The uplink and downlink formatters are now called asynchronously so they have limited impact on the overall performance of the application.
Azure IoT Central with consecutive duplicate PacketIds
Then I started to pay more attention and noticed that duplicate PacketIds could be interleaved
Azure IoT Central with interleaved duplicate PacketIds
Shortly after noticing the interleaved PacketIds I checked the Delivery Method and found there were message delivery timeouts.
Swarm Space Delivery with method timeouts
In Azure Application Insights I could see that the UplinkController was taking up to 15 seconds to execute which was longer than the bumblebee hive delivery timeout.
In Telerik Fiddler I could see calls to the UplinkController taking 16 seconds to execute. (I did see 30+ seconds)
Telerik Fiddler showing duration of Uplink controller calls
To see if the problem was loading CS-Script I added code to load a simple function as the application started. After averaging the duration over many executions there was little difference in the duration.
public interface IApplication
{
public DateTime Startup(DateTime utcNow);
}
...
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
await Task.Yield();
_logger.LogInformation("StartUpService.ExecuteAsync start");
// Force the loading and startup of CS Script evaluator
dynamic application = CSScript.Evaluator
.LoadCode(
@"using System;
public class Application : IApplication
{
public DateTime Startup(DateTime utcNow)
{
return utcNow;
}
}");
DateTime result = application.Startup(DateTime.UtcNow);
try
{
await _swarmSpaceBumblebeeHive.Login(cancellationToken);
await _azureIoTDeviceClientCache.Load(cancellationToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "StartUpService.ExecuteAsync error");
throw;
}
_logger.LogInformation("StartUpService.ExecuteAsync finish");
}
using System;
using System.Globalization;
using System.Text;
using Microsoft.Azure.Devices.Client;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public class FormatterUplink : PayloadFormatter.IFormatterUplink
{
public Message Evaluate(int organisationId, int deviceId, int deviceType, int userApplicationId, JObject telemetryEvent, JObject payloadJson, string payloadText, byte[] payloadBytes)
{
if ((payloadText != "") && (payloadJson != null))
{
JObject location = new JObject();
location.Add("lat", payloadJson.GetValue("lt"));
location.Add("lon", payloadJson.GetValue("ln"));
location.Add("alt", payloadJson.GetValue("a"));
telemetryEvent.Add("DeviceLocation", location);
}
Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent)));
ioTHubmessage.Properties.Add("iothub-creation-time-utc", DateTimeOffset.FromUnixTimeSeconds((long)payloadJson.GetValue("d")).ToString("s", CultureInfo.InvariantCulture));
return ioTHubmessage;
}
}
I then added code to load the most complex uplink and downlink formatters as the application started. There was a significant reduction in the UplinkController execution durations, but it could still take more than 30 seconds.
I then added detailed telemetry to the code and found that the duration (also variability) was a combination of Azure IoT Device Provisoning Service(DPS) registration, Azure IoT Hub connection establishment, CS-Script payload formatter loading/compilation/execution, application startup tasks and message uploading durations.
After much experimentation It looks like that “synchronously” calling the payload processing code from the Uplink controller is not a viable approach as the Swarm Space Bumblebee hive calls regularly timeout resulting in duplicate messages.
public interface IFormatterUplink
{
public Message Evaluate(int organisationId, int deviceId, int deviceType, int userApplicationId, JObject telemetryEvent, JObject payloadJson, string payloadText, byte[] payloadBytes);
}
public class FormatterUplink : PayloadFormatter.IFormatterUplink
{
public Message Evaluate(int organisationId, int deviceId, int deviceType, int userApplicationId, JObject telemetryEvent, JObject payloadJson, string payloadText, byte[] payloadBytes)
{
Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent)));
return ioTHubmessage;
}
}
Satellite Passes with gap in coverage from 16:18 to 18:42 highlighted
In the Swarm Hive Delivery Method messages from the Swarm Eval Kit and Swarm Tracker in my backyard arriving in “clusters”.
Swarm Hive Delivery Methods webhook calls.
The messages in each “cluster” were processed by a payload formatter then forwarded to Azure IoT Central for processing. All the messages in a cluster had similar event creation times which was “breaking” graphs and device tracking maps. After running the application locally using Telerik Fiddler to try different payloads I realised that the Microsoft.Azure.Azure.Devices.Client message iothub-creation-time-utc property was set to the when the message was received by Swarm Space infrastructure.
_logger.LogDebug("Uplink-DeviceId:{0} PacketId:{1} TelemetryEvent before:{0}", payload.DeviceId, payload.PacketId, JsonConvert.SerializeObject(telemetryEvent, Formatting.Indented));
telemetryEvent = swarmSpaceFormatterUplink.Evaluate(telemetryEvent, payload.Data, payloadBytes, payloadText, payloadJson);
_logger.LogDebug("Uplink-DeviceId:{0} PacketId:{1} TelemetryEvent after:{0}", payload.DeviceId, payload.PacketId, JsonConvert.SerializeObject(telemetryEvent, Formatting.Indented));
// Send the message to Azure IoT Hub
using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
{
// Ensure the displayed time is the acquired time rather than the uploaded time.
ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.HiveRxTime.ToString("s", CultureInfo.InvariantCulture));
ioTHubmessage.Properties.Add("PacketId", payload.PacketId.ToString());
ioTHubmessage.Properties.Add("OrganizationId", payload.OrganizationId.ToString());
ioTHubmessage.Properties.Add("ApplicationId", payload.UserApplicationId.ToString());
ioTHubmessage.Properties.Add("DeviceId", payload.DeviceId.ToString());
ioTHubmessage.Properties.Add("deviceType", payload.DeviceType.ToString());
await deviceClient.SendEventAsync(ioTHubmessage);
_logger.LogInformation("Uplink-DeviceID:{deviceId} SendEventAsync success", payload.DeviceId);
}
The Swarm Eval Kit uplink (JSON) message generated by the sample firmware “d” field is the number of seconds since the Unix Epoch that the message payload was constructed.
Swarm Hive Messages with “d” field in the JSON payload highlighted
Online Unix Epoch Convertor displaying Unix Epoch 1672561286 in NZDT and UTC time
The revised 65355.cs payload formatter adds an “iothub-creation-time-utc” field to the TelemetryEvent
using System;
using System.Globalization;
using Newtonsoft.Json.Linq;
public class FormatterUplink : PayloadFormatter.IFormatterUplink
{
public JObject Evaluate(JObject telemetryEvent, string payloadBase64, byte[] payloadBytes, string payloadText, JObject payloadJson)
{
if ((payloadText != "" ) && ( payloadJson != null))
{
JObject location = new JObject();
location.Add("lat", payloadJson.GetValue("lt"));
location.Add("lon", payloadJson.GetValue("ln"));
location.Add("alt", payloadJson.GetValue("a"));
telemetryEvent.Add("DeviceLocation", location);
};
telemetryEvent.Add("iothub-creation-time-utc", DateTimeOffset.FromUnixTimeSeconds((long)payloadJson.GetValue("d")).ToString("s", CultureInfo.InvariantCulture));
return telemetryEvent;
}
}
The payload formatters of my Azure IoT Hub Cloud Identity Translation Gateway use CS-Script and even a simple one was taking more than half a second to compile each time it was called.
using System;
using System.Globalization;
using Newtonsoft.Json.Linq;
public class FormatterUplink : PayloadFormatter.IFormatterUplink
{
public JObject Evaluate(JObject telemetryEvent, string payloadBase64, byte[] payloadBytes, string payloadText, JObject payloadJson)
{
if ((payloadText != "" ) && ( payloadJson != null))
{
JObject location = new JObject();
location.Add("lat", payloadJson.GetValue("lt"));
location.Add("lon", payloadJson.GetValue("ln"));
location.Add("alt", payloadJson.GetValue("a"));
telemetryEvent.Add("Location", location);
};
return telemetryEvent;
}
}
Azure IoT Central uplink telemetry message payload
The formatter files are currently part of the SwarmSpaceAzureIoTConnector project (moving to Azure Blob Storage) so are configured as “content” (bonus syntax highlighting works) and “copy if newer” so they are included in the deployment package.
public class FormatterCache : IFormatterCache
{
private readonly ILogger<FormatterCache> _logger;
private readonly Models.ApplicationSettings _applicationSettings;
private readonly static IAppCache _payloadFormatters = new CachingService();
public FormatterCache(ILogger<FormatterCache>logger, IOptions<Models.ApplicationSettings> applicationSettings)
{
_logger = logger;
_applicationSettings = applicationSettings.Value;
}
public async Task<IFormatterUplink> UplinkGetAsync(int userApplicationId)
{
IFormatterUplink payloadFormatterUplink = await _payloadFormatters.GetOrAddAsync<PayloadFormatter.IFormatterUplink>($"U{userApplicationId}", (ICacheEntry x) => UplinkLoadAsync(userApplicationId), memoryCacheEntryOptions);
return payloadFormatterUplink;
}
private async Task<IFormatterUplink> UplinkLoadAsync(int userApplicationId)
{
string payloadformatterFilePath = $"{_applicationSettings.PayloadFormattersUplinkFilePath}\\{userApplicationId}.cs";
if (!File.Exists(payloadformatterFilePath))
{
_logger.LogInformation("PayloadFormatterUplink- UserApplicationId:{0} PayloadFormatterPath:{1} not found using default:{2}", userApplicationId, payloadformatterFilePath, _applicationSettings.PayloadFormatterUplinkDefault);
return CSScript.Evaluator.LoadFile<PayloadFormatter.IFormatterUplink>(_applicationSettings.PayloadFormatterUplinkDefault);
}
_logger.LogInformation("PayloadFormatterUplink- UserApplicationId:{0} loading PayloadFormatterPath:{1}", userApplicationId, payloadformatterFilePath);
return CSScript.Evaluator.LoadFile<PayloadFormatter.IFormatterUplink>(payloadformatterFilePath);
}
...
}
The default uplink and downlink formatters are configured in application settings and are used when a UserApplicationId specific formatter is not configured.
Fiddler Composer illustrating compiled formatter timings before and after caching
const string codeSwarmSpaceUplinkFormatterCode = @"
using Newtonsoft.Json.Linq;
public class UplinkFormatter : PayloadFormatter.ISwarmSpaceFormatterUplink
{
public JObject Evaluate(JObject telemetryEvent, string payloadBase64, byte[] payloadBytes, string payloadText, JObject payloadJson)
{
if ((payloadText != """" ) && ( payloadJson != null))
{
JObject location = new JObject() ;
location.Add(""Lat"", payloadJson.GetValue(""lt""));
location.Add(""Lon"", payloadJson.GetValue(""ln""));
location.Add(""Alt"", payloadJson.GetValue(""a""));
telemetryEvent.Add( ""location"", location);
};
return telemetryEvent;
}
}";
}
The PayloadFormatter namespace was added to reduce the length of the payload formatter C# interface declarations.
namespace PayloadFormatter
{
using Newtonsoft.Json.Linq;
public interface ISwarmSpaceFormatterUplink
{
public JObject Evaluate(JObject telemetry, string payloadBase64, byte[] payloadBytes, string payloadText, JObject payloadJson);
}
public interface ISwarmSpaceFormatterDownlink
{
public string Evaluate(JObject payloadJson, string payloadText, byte[] payloadBytes, string payloadBase64);
}
}
namespace devMobile.IoT.SwarmSpace.AzureIoT.Connector
{
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using CSScriptLib;
using PayloadFormatter;
public interface ISwarmSpaceFormatterCache
{
public Task<ISwarmSpaceFormatterUplink> PayloadFormatterGetOrAddAsync(int userApplicationId);
}
public class SwarmSpaceFormatterCache : ISwarmSpaceFormatterCache
{
private readonly ILogger<SwarmSpaceFormatterCache> _logger;
public SwarmSpaceFormatterCache(ILogger<SwarmSpaceFormatterCache>logger)
{
_logger = logger;
}
public async Task<ISwarmSpaceFormatterUplink> PayloadFormatterGetOrAddAsync(int deviceId)
{
return CSScript.Evaluator.LoadCode<PayloadFormatter.ISwarmSpaceFormatterUplink>(codeSwarmSpaceUplinkFormatterCode);
}
...
}
The parameters of the formatter are Base64 encoded, textual and a Newtonsoft JObject representations of the uplink payload and a telemetry event populated with some uplink message metadata.
Azure IoT Central uplink telemetry message payload
The initial “compile” of an uplink formatter was taking approximately 2.1 seconds so they will be “compiled” on demand and cached in a Dictionary with the UserApplicationId as the key. A default uplink formatter will be used when a UserApplicationId specific uplink formatter is not configured.
https://json2csharp.com/
// Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse);
public class Root
{
public int packetId { get; set; }
public int deviceType { get; set; }
public int deviceId { get; set; }
public int userApplicationId { get; set; }
public int organizationId { get; set; }
public string data { get; set; }
public int len { get; set; }
public int status { get; set; }
public DateTime hiveRxTime { get; set; }
}
*/
public class UplinkPayload
{
[JsonProperty("packetId")]
public int PacketId { get; set; }
[JsonProperty("deviceType")]
public int DeviceType { get; set; }
[JsonProperty("deviceId")]
public int DeviceId { get; set; }
[JsonProperty("userApplicationId")]
public int UserApplicationId { get; set; }
[JsonProperty("organizationId")]
public int OrganizationId { get; set; }
[JsonProperty("data")]
[JsonRequired]
public string Data { get; set; }
[JsonProperty("len")]
public int Len { get; set; }
[JsonProperty("status")]
public int Status { get; set; }
[JsonProperty("hiveRxTime")]
public DateTime HiveRxTime { get; set; }
}
This class is used to “automagically” deserialise Delivery Webhook payloads. There is also some additional payload validation which discards test messages (not certain this is a good idea) etc.
//---------------------------------------------------------------------------------
// Copyright (c) December 2022, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
//---------------------------------------------------------------------------------
namespace devMobile.IoT.SwarmSpace.AzureIoT.Connector.Controllers
{
using System.Globalization;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Devices.Client;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
[ApiController]
[Route("api/[controller]")]
public class UplinkController : ControllerBase
{
private readonly ILogger<UplinkController> _logger;
private readonly IAzureIoTDeviceClientCache _azureIoTDeviceClientCache;
public UplinkController(ILogger<UplinkController> logger, IAzureIoTDeviceClientCache azureIoTDeviceClientCache)
{
_logger = logger;
_azureIoTDeviceClientCache = azureIoTDeviceClientCache;
}
[HttpPost]
public async Task<IActionResult> Uplink([FromBody] Models.UplinkPayload payload)
{
DeviceClient deviceClient;
_logger.LogDebug("Payload {0}", JsonConvert.SerializeObject(payload, Formatting.Indented));
if (payload.PacketId == 0)
{
_logger.LogWarning("Uplink-payload simulated DeviceId:{DeviceId}", payload.DeviceId);
return this.Ok();
}
if ((payload.UserApplicationId < Constants.UserApplicationIdMinimum) || (payload.UserApplicationId > Constants.UserApplicationIdMaximum))
{
_logger.LogWarning("Uplink-payload invalid User Application Id:{UserApplicationId}", payload.UserApplicationId);
return this.BadRequest($"Invalid User Application Id {payload.UserApplicationId}");
}
if ((payload.Len < Constants.PayloadLengthMinimum) || string.IsNullOrEmpty(payload.Data))
{
_logger.LogWarning("Uplink-payload.Data is empty PacketId:{PacketId}", payload.PacketId);
return this.Ok("payload.Data is empty");
}
Models.AzureIoTDeviceClientContext context = new Models.AzureIoTDeviceClientContext()
{
OrganisationId = payload.OrganizationId,
UserApplicationId = payload.UserApplicationId,
DeviceType = payload.DeviceType,
DeviceId = payload.DeviceId,
};
deviceClient = await _azureIoTDeviceClientCache.GetOrAddAsync(payload.DeviceId.ToString(), context);
JObject telemetryEvent = new JObject
{
{ "packetId", payload.PacketId},
{ "deviceType" , payload.DeviceType},
{ "DeviceID", payload.DeviceId },
{ "organizationId", payload.OrganizationId },
{ "ApplicationId", payload.UserApplicationId},
{ "ReceivedAtUtc", payload.HiveRxTime.ToString("s", CultureInfo.InvariantCulture) },
{ "DataLength", payload.Len },
{ "Data", payload.Data },
{ "Status", payload.Status },
};
// Send the message to Azure IoT Hub
using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
{
// Ensure the displayed time is the acquired time rather than the uploaded time.
ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.HiveRxTime.ToString("s", CultureInfo.InvariantCulture));
ioTHubmessage.Properties.Add("OrganizationId", payload.OrganizationId.ToString());
ioTHubmessage.Properties.Add("ApplicationId", payload.UserApplicationId.ToString());
ioTHubmessage.Properties.Add("DeviceId", payload.DeviceId.ToString());
ioTHubmessage.Properties.Add("deviceType", payload.DeviceType.ToString());
await deviceClient.SendEventAsync(ioTHubmessage);
_logger.LogInformation("Uplink-DeviceID:{deviceId} SendEventAsync success", payload.DeviceId);
}
return this.Ok();
}
}
}
The webhook was configured to “acknowledge messages on successful delivery”. I then checked my Delivery Method configuration with a couple of “Test” messages.
My Swarm Space Eval Kit arrived md-week and after some issues with jumper settings it started reporting position and status information.
I started with a modified version of the first sample on Github.
public class Samples
{
const string codeMethod = @"
int Multiply(int a, int b)
{
return a * b;
}";
public void Execute1()
{
dynamic script = CSScript.Evaluator.LoadMethod(codeMethod);
int result = script.Multiply(3, 2);
Console.WriteLine($"Product 1:{result}");
}
...
internal class Program
{
static void Main(string[] args)
{
new Samples().Execute1();
...
Console.WriteLine($"Press Enter to exit");
Console.ReadLine();
}
}
I then modified it to use a C# interface and the application failed with an exception
CSScriptLib.CompilerException
HResult=0x80131600
Message=(2,39): error CS0246: The type or namespace name 'IMultiplier' could not be found (are you missing a using directive or an assembly reference?)
Source=CSScriptLib
StackTrace:
at CSScriptLib.RoslynEvaluator.Compile(String scriptText, String scriptFile, CompileInfo info)
at CSScriptLib.EvaluatorBase`1.LoadCode[T](String scriptText, Object[] args)
at devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.Samples.Execute2A() in C:\Users\BrynLewis\source\repos\SwarmSpaceAzureIoT\PayloadFormatterCSScipt\Program.cs:line 90
at devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.Program.Main(String[] args) in C:\Users\BrynLewis\source\repos\SwarmSpaceAzureIoT\PayloadFormatterCSScipt\Program.cs:line 375
After some trial and error, I figured out I had the namespace wrong
const string codeClassA = @"
public class Calculator : devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.IMultiplier
{
public int Multiply(int a, int b)
{
return a * b;
}
}";
public void Execute2A()
{
IMultiplier multiplierA = CSScript.Evaluator.LoadCode<IMultiplier>(codeClassA);
Console.WriteLine($"Product 2A:{multiplierA.Multiply(3, 2)} - Press Enter to exit");
}
The long namespace would have been a pain in the arse (PITA) for users creating payload formatters and after some experimentation I added another interface with a short namespace. (Not certain this is a good idea).
namespace PayloadFormatter // Additional namespace for shortening interface for formatters
{
public interface IMultiplier
{
int Multiply(int a, int b);
}
}
...
public void Execute2B()
{
PayloadFormatter.IMultiplier multiplierB = CSScript.Evaluator.LoadCode<PayloadFormatter.IMultiplier>(codeClassB);
Console.WriteLine($"Product 2B:{multiplierB.Multiply(3, 2)} - Press Enter to exit");
}
I then wanted to figure out how to limit the namepaces the script has access to
const string codeClassDebug = @"
using System.Diagnostics;
public class Calculator : devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.IMultiplier
{
public int Multiply(int a, int b)
{
Debug.WriteLine(""Oops""); // Comment out the using System.Diagnostics;
return a * b;
}
}";
public void Execute3()
{
CSScript.Evaluator.Reset(true);
IMultiplier multiplier = CSScript.Evaluator
.LoadCode<IMultiplier>(codeClassDebug);
int result = multiplier.Multiply(6, 2);
Console.WriteLine($"Product 3:{result}");
}
The CSScript.Evaluator.Reset(true); removes all of the “default” references but a using directive could make namespaces available, so this needs some more investigation
The next step was to build the simplest possible payload formatter a “pipe” which displayed the text encoded in Base64 string.
const string codeSwarmSpaceFormatterPipe = @"
public class SwarmSpaceFormatter:devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.ISwarmSpaceFormatterPipe
{
public string Pipe(string payloadBase64)
{
var payloadBase64Bytes = System.Convert.FromBase64String(payloadBase64);
return System.Text.Encoding.UTF8.GetString(payloadBase64Bytes);
}
}";
...
public void Execute4()
{
ISwarmSpaceFormatterPipe SwarmSpaceFormatter = CSScript.Evaluator
...
.LoadCode<ISwarmSpaceFormatterPipe>(codeSwarmSpaceFormatterPipe);
string payload = SwarmSpaceFormatter.Pipe(PayloadBase64);
Console.WriteLine($"Pipe:{payload}");
}
The Base64 encoded uplink payloads will have to be converted to JSON and the downlink JSON payloads will have to be converted to Base64 encoded binary, so I created an uplink and downlink formatters.
I found that having both the byte array and Base64 encoded representation of the uplink payloads was useful. The first formatter converts the temperature field of the downlink payload into a four byte array then reverses the array to illustrate how packed byte payloads could be constructed.
const string codeSwarmSpaceFormatter1 = @"
public class SwarmSpaceFormatter : devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.ISwarmSpaceFormatter
{
public string Pipe(string payloadBase64)
{
var payloadBase64Bytes = System.Convert.FromBase64String(payloadBase64);
return System.Text.Encoding.UTF8.GetString(payloadBase64Bytes);
}
public JObject Uplink(JObject telemetryEvent, string payloadBase64, byte[] payloadBytes)
{
var payloadBase64Bytes = System.Convert.FromBase64String(payloadBase64);
telemetryEvent.Add(""PayloadBase64"", payloadBase64Bytes);
telemetryEvent.Add(""PayloadBytes"",System.Text.Encoding.UTF8.GetString(payloadBytes));
return telemetryEvent;
}
public string Downlink(JObject command)
{
int temperature = command.Value<int>(""Temperature"");
return System.Convert.ToBase64String(BitConverter.GetBytes(temperature));
}
}";
const string codeSwarmSpaceFormatter2 = @"
public class SwarmSpaceFormatter:devMobile.IoT.SwarmSpace.AzureIoT.PayloadFormatterCSScript.ISwarmSpaceFormatter
{
public string Pipe(string payloadBase64)
{
var payloadBase64Bytes = System.Convert.FromBase64String(payloadBase64);
return System.Text.Encoding.UTF8.GetString(payloadBase64Bytes);
}
public JObject Uplink(JObject telemetryEvent, string payloadBase64, byte[] payloadBytes)
{
var payloadBase64Bytes = System.Convert.FromBase64String(payloadBase64);
telemetryEvent.Add(""PayloadBase64"", payloadBase64Bytes);
telemetryEvent.Add(""PayloadBytes"",System.Text.Encoding.UTF8.GetString(payloadBytes));
return telemetryEvent;
}
public string Downlink(JObject command)
{
int temperature = command.Value<int>(""Temperature"");
byte[] temperatureBytes = BitConverter.GetBytes(temperature);
Array.Reverse(temperatureBytes);
return System.Convert.ToBase64String(temperatureBytes);
}
}";
...
public void Execute6()
{
string namespaces = $"using Newtonsoft.Json.Linq;using System;\n";
string code1 = namespaces + codeSwarmSpaceFormatter1;
string code2 = namespaces + codeSwarmSpaceFormatter2;
JObject telemetry = new JObject
{
{ "ApplicationID", 12345 },
{ "DeviceID", 54321 },
{ "DeviceType", 2 },
{ "ReceivedAtUtc", DateTime.UtcNow.ToString("s", CultureInfo.InvariantCulture) },
};
var formatters = new Dictionary<string, ISwarmSpaceFormatter>();
Console.WriteLine($"Evaluator start");
DateTime evaluatorStartAtUtc = DateTime.UtcNow;
ISwarmSpaceFormatter SwarmSpaceFormatter1 = CSScript.Evaluator
.LoadCode<ISwarmSpaceFormatter>(code1);
ISwarmSpaceFormatter SwarmSpaceFormatter2 = CSScript.Evaluator
.LoadCode<ISwarmSpaceFormatter>(code2);
Console.WriteLine($"Evaluator:{DateTime.UtcNow - evaluatorStartAtUtc}");
Console.WriteLine("");
Console.WriteLine($"Evaluation start");
DateTime evaluationStartUtc = DateTime.UtcNow;
formatters.Add("F1", SwarmSpaceFormatter1);
formatters.Add("F2", SwarmSpaceFormatter2);
JObject command = new JObject
{
{"Temperature", 1},
};
ISwarmSpaceFormatter downlinkPayload;
downlinkPayload = formatters["F1"];
Console.WriteLine($"Downlink F1:{downlinkPayload.Downlink(command)}");
downlinkPayload = formatters["F2"];
Console.WriteLine($"Downlink F2:{downlinkPayload.Downlink(command)}");
Console.WriteLine($"Evaluation:{DateTime.UtcNow - evaluationStartUtc}");
Console.WriteLine("");
const int iterations = 100;
Console.WriteLine($"Evaluations start {iterations}");
DateTime evaluationsStartUtc = DateTime.UtcNow;
for (int i = 1; i <= iterations; i++)
{
JObject command1 = new JObject
{
{"Temperature", 1},
};
downlinkPayload = formatters["F1"];
Console.WriteLine($" Downlink F1:{downlinkPayload.Downlink(command1)}");
downlinkPayload = formatters["F2"];
Console.WriteLine($" Downlink F2:{downlinkPayload.Downlink(command1)}");
}
Console.WriteLine($"Evaluations:{iterations} Took:{DateTime.UtcNow - evaluationsStartUtc}");
}
On my development box the initial “compile” of each function was taking approximately 2.1 seconds so I cached the “compiled” formatters in a dictionary so they could be reused. Cached in the dictionary executing the two formatters 100 times took approximately 15 milliseconds (which is close to native .NET performance).
Compatibility
To check that the CS-Script tooling could run on a machine without the .NET 6 Software Development Kit (SDK) I tested the application on a laptop which had a “fresh” install of Windows 10.
CS-Script application failing due to missing .NET 6 runtime
Installing the .NET 6 Runtime
CS-Script application running after .NET runtime installation
The CS-Script library is pretty amazing and has made the development of uplink and downlink payload formatters significantly less complex than I was expecting.