I initially noticed a couple of duplicate Swarm Space message PacketIds in Azure IoT Central.
Then I started to pay more attention and noticed that duplicate PacketIds could be interleaved
Shortly after noticing the interleaved PacketIds I checked the Delivery Method and found there were message delivery timeouts.
In Azure Application Insights I could see that the UplinkController was taking up to 15 seconds to execute which was longer than the bumblebee hive delivery timeout.
In Telerik Fiddler I could see calls to the UplinkController taking 16 seconds to execute. (I did see 30+ seconds)
To see if the problem was loading CS-Script I added code to load a simple function as the application started. After averaging the duration over many executions there was little difference in the duration.
public interface IApplication
{
public DateTime Startup(DateTime utcNow);
}
...
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
await Task.Yield();
_logger.LogInformation("StartUpService.ExecuteAsync start");
// Force the loading and startup of CS Script evaluator
dynamic application = CSScript.Evaluator
.LoadCode(
@"using System;
public class Application : IApplication
{
public DateTime Startup(DateTime utcNow)
{
return utcNow;
}
}");
DateTime result = application.Startup(DateTime.UtcNow);
try
{
await _swarmSpaceBumblebeeHive.Login(cancellationToken);
await _azureIoTDeviceClientCache.Load(cancellationToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "StartUpService.ExecuteAsync error");
throw;
}
_logger.LogInformation("StartUpService.ExecuteAsync finish");
}
The Swarm Eval Kit uplink formatter (UserApplicationId 65535.cs) “unpacks” the uplink Javascript ObjectNotation(JSON) message, adds an Azure IoT Central compatible location which requires a number of libraries to be loaded.
using System;
using System.Globalization;
using System.Text;
using Microsoft.Azure.Devices.Client;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public class FormatterUplink : PayloadFormatter.IFormatterUplink
{
public Message Evaluate(int organisationId, int deviceId, int deviceType, int userApplicationId, JObject telemetryEvent, JObject payloadJson, string payloadText, byte[] payloadBytes)
{
if ((payloadText != "") && (payloadJson != null))
{
JObject location = new JObject();
location.Add("lat", payloadJson.GetValue("lt"));
location.Add("lon", payloadJson.GetValue("ln"));
location.Add("alt", payloadJson.GetValue("a"));
telemetryEvent.Add("DeviceLocation", location);
}
Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent)));
ioTHubmessage.Properties.Add("iothub-creation-time-utc", DateTimeOffset.FromUnixTimeSeconds((long)payloadJson.GetValue("d")).ToString("s", CultureInfo.InvariantCulture));
return ioTHubmessage;
}
}
I then added code to load the most complex uplink and downlink formatters as the application started. There was a significant reduction in the UplinkController execution durations, but it could still take more than 30 seconds.
try
{
await _swarmSpaceBumblebeeHive.Login(cancellationToken);
await _azureIoTDeviceClientCache.Load(cancellationToken);
await _formatterCache.UplinkGetAsync(65535);
await _formatterCache.DownlinkGetAsync(20);
}
catch (Exception ex)
{
_logger.LogError(ex, "StartUpService.ExecuteAsync error");
throw;
}
I then added detailed telemetry to the code and found that the duration (also variability) was a combination of Azure IoT Device Provisoning Service(DPS) registration, Azure IoT Hub connection establishment, CS-Script payload formatter loading/compilation/execution, application startup tasks and message uploading durations.
After much experimentation It looks like that “synchronously” calling the payload processing code from the Uplink controller is not a viable approach as the Swarm Space Bumblebee hive calls regularly timeout resulting in duplicate messages.
devMobile's blog 
Pingback: Swarm Space – Underlying Architecture Revised | devMobile's blog