Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
Initially, I was running the WebAPI Dapper DistributedCache application on my development box and it ran third/forth time after sorting out the Azure Cache for Redis connection string and firewall configuration.
StockItems list served from the cache running on my Desktop
There were many questions and tentative answers on stackoverflow some of them I tried, and which didn’t work in my scenario.
No connection is active/available to service this operation: HMGET Dapper WebAPI InstanceStockItems;
UnableToConnect on xxxxxxxxxxxxxxxxxxxxxxxxxxx.redis.cache.windows.net:6380/Interactive,
Initializing/NotStarted,
last: NONE,
origin: BeginConnectAsync,
outstanding: 0,
last-read: 10s ago, last-write: 10s ago, keep-alive: 60s,
state: Connecting,
mgr: 10 of 10 available,
last-heartbeat: never, global: 10s ago,
v: 2.2.4.27433,
mc: 1/1/0, mgr: 10 of 10 available,
clientName: XXXXXXXXXXXX, <----------------------This was important
IOCP: (Busy=0,Free=1000,Min=1,Max=1000),
WORKER: (Busy=5,Free=1018,Min=1,Max=1023),
v: 2.2.4.27433
UnableToConnect on xxxxxxxxxxxxxxxxxxxxxxxxxxx.cache.windows.net:6380/Interactive,
Initializing/NotStarted,
last: NONE,
origin: BeginConnectAsync,
outstanding: 0,
last-read: 10s ago, last-write: 10s ago, keep-alive: 60s,
state: Connecting,
mgr: 10 of 10 available,
last-heartbeat: never, global: 10s ago,
v: 2.2.4.27433
I then recreated my configuration from scratch so none of my random changes based on stackoverflow posts would mess my next round of debugging.
Azure AppService Virtual, Outbound and Additional Outbound IP Addresses
The error appeared to be a networking issue, most probably the firewall blocking connections so I added the Azure AppService Virtual IP Address which didn’t work…
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
builder.Services.AddApplicationInsightsTelemetry();
// Add services to the container.
builder.Services.AddTransient<IDapperContext>(s => new DapperContext(builder.Configuration));
builder.Services.AddControllers();
// Register IAppCache as a singleton CachingService
builder.Services.AddLazyCache();
var app = builder.Build();
// Configure the HTTP request pipeline.
app.UseHttpsRedirection();
app.MapControllers();
app.Run();
}
One of the applications I work uses a lot of “reference” data which is effectively static e.g. national/regional holidays in countries where they have customers, and data which is updated on a schedule e.g. exchange rates downloaded at a known time every day.
For the holiday information updates, the dataset could be deleted(“burped”) then refreshed the next time it is referenced or deleted then re-loaded. For exchange rate updates the cache could automatically expire the dataset shortly after the scheduled download.
The time to open the home page of one of the “legacy” applications I was working on was slowly increasing over time. After a quick investigation it looked like there a couple of Azure SQL stored procedures which were called many times as the home page was opening were the problem.
[HttpGet("Dapper")]
public async Task<ActionResult<IEnumerable<Model.StockItemListDtoV1>>> GetDapper()
{
IEnumerable<Model.StockItemListDtoV1> response;
using (IDbConnection db = dapperContext.ConnectionCreate())
{
response = await db.QueryAsync<Model.StockItemListDtoV1>(sql: sqlCommandText, commandType: CommandType.Text);
}
return this.Ok(response);
}
World Wide Importers database list of stockitems
The World Wide Importers database has approximately 250 StockItems which was representative of one of the problematic queries.
[HttpGet("DapperProfiled")]
public async Task<ActionResult<IEnumerable<Model.StockItemListDtoV1>>> GetDapperProfiled()
{
IEnumerable<Model.StockItemListDtoV1> response;
using (IDbConnection db = new ProfiledDbConnection((DbConnection)dapperContext.ConnectionCreate(), MiniProfiler.Current))
{
response = await db.QueryAsync<Model.StockItemListDtoV1>(sql: sqlCommandText, commandType: CommandType.Text);
}
return this.Ok(response);
}
When I executed the GetDapperProfiler() method of the StockItems controller on my development box it took roughly 2.4 seconds.
MiniProfiler results for StockItems query running on my desktop
I sometimes ran the website on my development box so when I used “toggle trivial gaps” it was easier to see what where the delays were.
When I executed the GetDapperProfiler() method of the StockItems controller running in an Azure AppService it took roughly 20 mSec.
MiniProfiler results for StockItems query running in an Azure AppService
In one application a QueryMultipleAsync is used to retrieve information about a product and a list of its attributes. The World Wide Importers database has Invoices which have invoice lines and Transactions which was representative of another problematic query.
[HttpGet("DapperProfiledQueryMultiple")]
public async Task<ActionResult<IEnumerable<Model.StockItemListDtoV1>>> GetDapperProfiledQueryMultiple([Required][Range(1, int.MaxValue, ErrorMessage = "Invoice id must greater than 0")] int id)
{
Model.InvoiceSummaryGetDtoV1 response = null;
using (ProfiledDbConnection db = new ProfiledDbConnection((DbConnection)dapperContext.ConnectionCreate(), MiniProfiler.Current))
{
var invoiceSummary = await db.QueryMultipleAsync("[Sales].[InvoiceSummaryGetV1]", param: new { InvoiceId = id }, commandType: CommandType.StoredProcedure);
response = await invoiceSummary.ReadSingleOrDefaultAsync<Model.InvoiceSummaryGetDtoV1>();
if (response == default)
{
return this.NotFound($"Invoice:{id} not found");
}
response.InvoiceLines = await invoiceSummary.ReadAsync<Model.InvoiceLineSummaryListDtoV1>();
response.StockItemTransactions = await invoiceSummary.ReadAsync<Model.StockItemTransactionSummaryListDtoV1>();
}
return this.Ok(response);
}
World Wide Importers database list of invoice lines and transactions for a StockItem
MiniProfiler results for Invoice Item Query Multiple running in an Azure AppService
I couldn’t see any results for reading the StockItem Invoice lines and Transactions so I wrapped each ReadAsync with a MiniProfiler.Current.Step.
[HttpGet("DapperProfiledQueryMultipleStep")]
public async Task<ActionResult<IEnumerable<Model.StockItemListDtoV1>>> GetDapperProfiledQueryMultipleStep([Required][Range(1, int.MaxValue, ErrorMessage = "Invoice id must greater than 0")] int id)
{
Model.InvoiceSummaryGetDtoV1 response = null;
using (IDbConnection db = new ProfiledDbConnection((DbConnection)dapperContext.ConnectionCreate(), MiniProfiler.Current))
{
SqlMapper.GridReader invoiceSummary;
using (MiniProfiler.Current.Step("db.QueryMultipleAsync"))
{
invoiceSummary = await db.QueryMultipleAsync("[Sales].[InvoiceSummaryGetV1]", param: new { InvoiceId = id }, commandType: CommandType.StoredProcedure);
}
using (MiniProfiler.Current.Step("invoiceSummary.ReadSingleOrDefaultAsync"))
{
response = await invoiceSummary.ReadSingleOrDefaultAsync<Model.InvoiceSummaryGetDtoV1>();
}
if (response == default)
{
return this.NotFound($"Invoice:{id} not found");
}
using (MiniProfiler.Current.Step("invoiceSummaryLine.ReadAsync"))
{
response.InvoiceLines = await invoiceSummary.ReadAsync<Model.InvoiceLineSummaryListDtoV1>();
}
using (MiniProfiler.Current.Step("TransactionSummary.ReadAsync"))
{
response.StockItemTransactions = await invoiceSummary.ReadAsync<Model.StockItemTransactionSummaryListDtoV1>();
}
}
return this.Ok(response);
}
With larger lists every so often there were ReadAsync calls that took a more than a “trivial” amount of time. I surmise was some sort of batching done by the underlying ReadAsync + NextResultAsync methods of a SqlDataReader.
Need to investigate the use of
using (IDbConnection db = new ProfiledDbConnection(new SqlConnection(_configuration.GetConnectionString("default"), MiniProfiler.Current))
{
//...
}
The application I’m currently working on has some tables with many columns and these were proving painful to update with HTTP PUT methods. Over the last couple of releases, I have been extending the customer facing API with PATCH methods so the client can specify only the values to changed.
The JSON Patch is a format for specifying updates to be applied to a resource.
Stock Items list before HTTP Patch operation
A JSON Patch document has an array of operations which identify a particular type of change.
Using Telerik Fiddler Composer functionality to apply an HTTP PATCH
Stock Items list after HTTP Patch operation
The StockItemPatchDtoV1 class is decorated with DataAnnotations to ensure the contents are valid.
public class StockItemPatchDtoV1
{
[Required]
[StringLength(100, MinimumLength = 1, ErrorMessage = "The name text must be at least {2} and no more than {1} characters long")] // These would be constants in a real application
public string Name { get; set; }
[Required]
[Range(0.0, 100.0)] // These would be constants in a real application
public decimal UnitPrice { get; set; }
[Required]
[Range(0.0, 1000000.0)] // These would be constants in a real application
public decimal RecommendedRetailPrice { get; set; }
}
The StockItemsController [HttpPatch(“{id}”)] method retrieves the stock item to be updated, then uses ApplyTo method and TryValidateModel to update only the specified fields.
[HttpPatch("{id}")]
public async Task<ActionResult<Model.StockItemGetDtoV1>> Patch([FromBody] JsonPatchDocument<Model.StockItemPatchDtoV1> stockItemPatch, int id)
{
Model.StockItemGetDtoV1 stockItem;
using (IDbConnection db = dapperContext.ConnectionCreate())
{
stockItem = await db.QuerySingleOrDefaultWithRetryAsync<Model.StockItemGetDtoV1>(sql: "[Warehouse].[StockItemsStockItemLookupV1]", param: new { stockItemId = id }, commandType: CommandType.StoredProcedure);
if (stockItem == default)
{
logger.LogInformation("StockItem:{id} not found", id);
return this.NotFound($"StockItem:{id} not found");
}
Model.StockItemPatchDtoV1 stockItemPatchDto = mapper.Map<Model.StockItemPatchDtoV1>(stockItem);
stockItemPatch.ApplyTo(stockItemPatchDto, ModelState);
if (!ModelState.IsValid || !TryValidateModel(stockItemPatchDto))
{
logger.LogInformation("stockItemPatchDto invalid {0}", string.Join(Environment.NewLine, ModelState.Values.SelectMany(v => v.Errors).Select(v => v.ErrorMessage + " " + v.Exception))); // would extract this out into shared module
return BadRequest(ModelState);
}
mapper.Map(stockItemPatchDto, stockItem);
await db.ExecuteWithRetryAsync(sql: "UPDATE Warehouse.StockItems SET StockItemName=@Name, UnitPrice=@UnitPrice, RecommendedRetailPrice=@RecommendedRetailPrice WHERE StockItemId=@Id", param: stockItem, commandType: CommandType.Text);
}
return this.Ok();
}
Initially the HTTP Patch method returned this error message.
HTTP/1.1 400 Bad Request
Content-Type: application/problem+json; charset=utf-8
Date: Tue, 27 Jun 2023 09:20:30 GMT
Server: Kestrel
Transfer-Encoding: chunked
1d7
{"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"00-665a6ee9ed1105a105237c421793af5d-1719bda40c0b7d5d-00","errors":{"$":["The JSON value could not be converted to Microsoft.AspNetCore.JsonPatch.JsonPatchDocument`1[devMobile.WebAPIDapper.HttpPatch.Model.StockItemPatchDtoV1]. Path: $ | LineNumber: 0 | BytePositionInLine: 1."],"stockItemPatch":["The stockItemPatch field is required."]}}
0
So far, to keep the code really obvious (tends to be more verbose) I have limited the use Dependency Injection(DI). I have been “injecting” an instance of an IConfiguration interface then retrieving the database connection string and other configuration. This isn’t a great approach as the database connection string name is in multiple files etc.
namespace devMobile.WebAPIDapper.ListsDIBasic
{
using System.Data;
using System.Data.SqlClient;
using Microsoft.Extensions.Configuration;
public interface IDapperContext
{
public IDbConnection ConnectionCreate();
public IDbConnection ConnectionCreate(string connectionStringName);
public IDbConnection ConnectionReadCreate();
public IDbConnection ConnectionWriteCreate();
}
public class DapperContext : IDapperContext
{
private readonly IConfiguration _configuration;
public DapperContext(IConfiguration configuration)
{
_configuration = configuration;
}
public IDbConnection ConnectionCreate()
{
return new SqlConnection(_configuration.GetConnectionString("default"));
}
public IDbConnection ConnectionCreate(string connectionStringName)
{
return new SqlConnection(_configuration.GetConnectionString(connectionStringName));
}
public IDbConnection ConnectionReadCreate()
{
return new SqlConnection(_configuration.GetConnectionString("default-read"));
}
public IDbConnection ConnectionWriteCreate()
{
return new SqlConnection(_configuration.GetConnectionString("default-write"));
}
}
}
public class Program
{
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
//builder.Services.AddSingleton<IDapperContext>(s => new DapperContext(builder.Configuration));
//builder.Services.AddTransient<IDapperContext>(s => new DapperContext(builder.Configuration));
//builder.Services.AddScoped<IDapperContext>(s => new DapperContext(builder.Configuration));
builder.Services.AddControllers();
var app = builder.Build();
// Configure the HTTP request pipeline.
app.UseHttpsRedirection();
app.MapControllers();
app.Run();
}
}
Then in the StockItems controller the IDapperContext interface implementation is used to create an IDbConnection for Dapper operations to use. I also added “WAITFOR DELAY ’00:00:02″ to the query to extend the duration of the requests.
[ApiController]
[Route("api/[controller]")]
public class StockItemsController : ControllerBase
{
private readonly ILogger<StockItemsController> logger;
private readonly IDapperContext dapperContext;
public StockItemsController(ILogger<StockItemsController> logger, IDapperContext dapperContext)
{
this.logger = logger;
this.dapperContext = dapperContext;
}
[HttpGet]
public async Task<ActionResult<IEnumerable<Model.StockItemListDtoV1>>> Get()
{
IEnumerable<Model.StockItemListDtoV1> response;
using (IDbConnection db = dapperContext.ConnectionCreate())
{
//response = await db.QueryWithRetryAsync<Model.StockItemListDtoV1>(sql: @"SELECT [StockItemID] as ""ID"", [StockItemName] as ""Name"", [RecommendedRetailPrice], [TaxRate] FROM [Warehouse].[StockItems]", commandType: CommandType.Text);
response = await db.QueryWithRetryAsync<Model.StockItemListDtoV1>(sql: @"SELECT [StockItemID] as ""ID"", [StockItemName] as ""Name"", [RecommendedRetailPrice], [TaxRate] FROM [Warehouse].[StockItems]; WAITFOR DELAY '00:00:02'", commandType: CommandType.Text);
}
return this.Ok(response);
}
...
}
I ran a stress testing application which simulated 50 concurrent users. When the stress test rig was stopped all the connections in the pool were closed after roughly 5 minutes.
SQL Server Management Studio(SSMS) sp_who query – stress test
public class Program
{
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
//builder.Services.AddSingleton<IDbConnection>(s => new SqlConnection(builder.Configuration.GetConnectionString("default")));
//builder.Services.AddScoped<IDbConnection>(s => new SqlConnection(builder.Configuration.GetConnectionString("default")));
//builder.Services.AddTransient<IDbConnection>(s => new SqlConnection(builder.Configuration.GetConnectionString("default")));
builder.Services.AddControllers();
var app = builder.Build();
app.UseHttpsRedirection();
app.MapControllers();
app.Run();
}
}
The code in the get method was reduced. I also added “WAITFOR DELAY ’00:00:02″ to the query to extend the duration of the requests.
public class StockItemsController : ControllerBase
{
private readonly ILogger<StockItemsController> logger;
private readonly IDbConnection dbConnection;
public StockItemsController(ILogger<StockItemsController> logger, IDbConnection dbConnection)
{
this.logger = logger;
this.dbConnection = dbConnection;
}
[HttpGet]
public async Task<ActionResult<IEnumerable<Model.StockItemListDtoV1>>> Get()
{
// return this.Ok(await dbConnection.QueryWithRetryAsync<Model.StockItemListDtoV1>(sql: @"SELECT [StockItemID] as ""ID"", [StockItemName] as ""Name"", [RecommendedRetailPrice], [TaxRate] FROM [Warehouse].[StockItems]; WAITFOR DELAY '00:00:02';", commandType: CommandType.Text));
return this.Ok(await dbConnection.QueryWithRetryAsync<Model.StockItemListDtoV1>(sql: @"SELECT [StockItemID] as ""ID"", [StockItemName] as ""Name"", [RecommendedRetailPrice], [TaxRate] FROM [Warehouse].[StockItems]", commandType: CommandType.Text));
}
...
}
With the stress test rig running the number of active connections was roughly the same as the DapperContext based implementation.
I don’t like this approach so will stick with DapperContext
The first message sent shortly after I powered up the device had the latitude and longitude of Null Island
The Asset Tracker UserApplicationId is 65002 and the payload is similar to the Swarm Eval Kit. I created some message payloads (location of Christchurch Cathedral) for testing.
namespace PayloadFormatter // Additional namespace for shortening interface when usage in formatter code
{
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
public interface IFormatterUplink
{
public JObject Evaluate(IDictionary<string, string> properties, uint organisationId, uint deviceId, byte deviceType, ushort userApplicationId, JObject payloadJson, string payloadText, byte[] payloadBytes);
}
public interface IFormatterDownlink
{
public byte[] Evaluate(IDictionary<string, string> properties, uint organisationId, uint deviceId, byte deviceType, ushort userApplicationId, JObject payloadJson, string payloadText, byte[] payloadBytes);
}
}
The definitions of the uplink & downlink payload formatter evaluator interfaces have been updated and shifted to a new project.
Visual Studio 2022 Solution with payloadformatter maintenance application
I built a console application to help with developing and debugging uplink or downlink formatters. The application has a number of command line parameters which specify the formatter to be used, UserApplicationId, OrganizationId, DeviceType etc.
public class CommandLineOptions
{
[Option('d', "Direction", Required = true, HelpText = "Test Uplink or DownLink formatter")]
public string Direction { get; set; }
[Option('p', "filename", HelpText = "Uplink or Downlink Payload file name")]
public string PayloadFilename { get; set; } = string.Empty;
[Option('o', "OrganisationId", Required = true, HelpText = "Organisation unique identifier")]
public uint OrganizationId { get; set; }
[Option('i', "DeviceId", Required = true, HelpText = "Device unique identitifer")]
public uint DeviceId { get; set; }
[Option('t', "DeviceType", Required = true, HelpText = "Device type number")]
public byte DeviceType { get; set; }
[Option('u', "UserApplicationId", Required = true, HelpText = "User Application Id")]
public ushort UserApplicationId { get; set; }
[Option('h', "SwarmHiveReceivedAtUtc", HelpText = "Swarm Hive received at time UTC")]
public DateTime? SwarmHiveReceivedAtUtc { get; set; }
[Option('w', "UplinkWebHookReceivedAtUtc", HelpText = "Webhook received at time UTC")]
public DateTime? UplinkWebHookReceivedAtUtc { get; set; }
[Option('s', "Status", HelpText = "Uplink local file system file name")]
public byte? Status { get; set; }
[Option('c', "Client", HelpText = "Uplink local file system file name")]
public string Client { get; set; }
}
The downlink formatter (similar approach for uplink) loads the sample file as an array of bytes, then tries to convert it to text, and finally to JSON. Then the formatter code is “compiled” and the executed with the file payload and command line parameters.
private static async Task DownlinkFormatterCore(CommandLineOptions options)
{
Dictionary<string, string> properties = new Dictionary<string, string>();
string formatterFolder = Path.Combine(Environment.CurrentDirectory, "downlink");
Console.WriteLine($"Downlink- uplinkFormatterFolder: {formatterFolder}");
string formatterFile = Path.Combine(formatterFolder, $"{options.UserApplicationId}.cs");
Console.WriteLine($"Downlink- UserApplicationId: {options.UserApplicationId}");
Console.WriteLine($"Downlink- Payload formatter file: {formatterFile}");
PayloadFormatter.IFormatterDownlink evalulator;
try
{
evalulator = CSScript.Evaluator.LoadFile<PayloadFormatter.IFormatterDownlink>(formatterFile);
}
catch (CSScriptLib.CompilerException cex)
{
Console.Write($"Loading or compiling file:{formatterFile} failed Exception:{cex}");
return;
}
string payloadFilename = Path.Combine(formatterFolder, options.PayloadFilename);
Console.WriteLine($"Downlink- payloadFilename:{payloadFilename}");
byte[] uplinkBytes;
try
{
uplinkBytes = File.ReadAllBytes(payloadFilename);
}
catch (DirectoryNotFoundException dex)
{
Console.WriteLine($"Uplink payload filename directory {formatterFolder} not found:{dex}");
return;
}
catch (FileNotFoundException fnfex)
{
Console.WriteLine($"Uplink payload filename {payloadFilename} not found:{fnfex}");
return;
}
catch (FormatException fex)
{
Console.WriteLine($"Uplink payload file invalid format {payloadFilename} not found:{fex}");
return;
}
// See if payload can be converted to a string
string uplinkText = string.Empty;
try
{
uplinkText = Encoding.UTF8.GetString(uplinkBytes);
}
catch (FormatException fex)
{
Console.WriteLine("Encoding.UTF8.GetString failed:{0}", fex.Message);
}
// See if payload can be converted to JSON
JObject uplinkJson;
try
{
uplinkJson = JObject.Parse(uplinkText);
}
catch (JsonReaderException jrex)
{
Console.WriteLine("JObject.Parse failed Exception:{1}", jrex);
uplinkJson = new JObject();
}
Console.WriteLine("Properties");
foreach (var property in properties)
{
Console.WriteLine($"{property.Key}:{property.Value}");
}
// Transform the byte and optional text and JSON payload
Byte[] payload;
try
{
payload = evalulator.Evaluate(properties, options.OrganizationId, options.DeviceId, options.DeviceType, options.UserApplicationId, uplinkJson, uplinkText, uplinkBytes);
}
catch (Exception ex)
{
Console.WriteLine($"evalulatorUplink.Evaluate failed Exception:{ex}");
return;
}
Console.WriteLine("Payload");
Console.WriteLine(Convert.ToBase64String(payload));
}
The sample JSON payload is what would be sent by Azure IoT Central to a device to configure the fan speed
Azure IoT Central M138 Breakout device template with the Fan Status command selected
{
"FanStatus": 2
}
If the downlink payload formatter is compiled and executes successfully the Base64 representation output is displayed
using System;
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
public class FormatterDownlink : PayloadFormatter.IFormatterDownlink
{
public byte[] Evaluate(IDictionary<string, string> properties, uint organisationId, uint deviceId, byte deviceType, ushort userApplicationId, JObject payloadJson, string payloadText, byte[] payloadBytes)
{
byte? status = payloadJson.Value<byte?>("FanStatus");
if ( status.HasValue )
{
return new byte[] { status.Value };
}
return new byte[]{};
}
}
If the downlink payload formatter syntax is incorrect e.g. { status.Value ; }; an error message with the line and column is displayed.
using System;
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
public class FormatterDownlink : PayloadFormatter.IFormatterDownlink
{
public byte[] Evaluate(IDictionary<string, string> properties, uint organisationId, uint deviceId, byte deviceType, ushort userApplicationId, JObject payloadJson, string payloadText, byte[] payloadBytes)
{
byte? status = payloadJson.Value<byte?>("FanStatus");
if ( status.HasValue )
{
return new byte[] { status.Value ; };
}
return new byte[]{};
}
}
If the downlink payload formatter syntax is correct but execution fails (in the example code division by zero) an error message is displayed.
using System;
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
public class FormatterDownlink : PayloadFormatter.IFormatterDownlink
{
public byte[] Evaluate(IDictionary<string, string> properties, uint organisationId, uint deviceId, byte deviceType, ushort userApplicationId, JObject payloadJson, string payloadText, byte[] payloadBytes)
{
byte? status = payloadJson.Value<byte?>("FanStatus");
if ( status.HasValue )
{
int divideByZero = 10;
divideByZero = divideByZero / 0;
return new byte[] { status.Value };
}
return new byte[]{};
}
}
Swarm Space Bumble hive classes in Visual Studio 2022
My SwarmSpaceAzureIoTConnector project only needed to login, get a list of devices and send messages so all the additional functionality was never going to be used. The method to send a message didn’t work, the class used for the payload (UserMessage) appears to be wrong.
"post": {
"tags": [ "messages" ],
"summary": "POST user messages",
"description": "<p>This endpoint submits a JSON formatted UserMessage object for delivery to a Swarm device. A JSON object is returned with a newly assigned <code>packetId</code> and <code>status</code> of<code>OK</code> on success, or <code>ERROR</code> (with a description of the error) on failure.</p><p>The current user must have access to the <code>userApplicationId</code> and <code>device</code> given inside the UserMessage JSON. The device must also have the ability to receive messages from the Hive (\"two-way communication\") enabled. If these conditions are not met, a response with status code 403 (Forbidden) will be returned.</p><p>Note that the <code>data</code> field is the <b>Base64-encoded</b> version of the data to be sent. This allows the sending of binary, as well as text, data.</p>",
"operationId": "addApplicationMessage",
"requestBody": {
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/UserMessage" } } },
"required": true
},
"responses": {
"401": {
"description": "Unauthorized",
"content": { "*/*": { "schema": { "$ref": "#/components/schemas/ApiError" } } }
},
"403": {
"description": "Forbidden",
"content": { "*/*": { "schema": { "$ref": "#/components/schemas/ApiError" } } }
},
"400": {
"description": "Bad Request",
"content": { "*/*": { "schema": { "$ref": "#/components/schemas/ApiError" } } }
},
"200": {
"description": "OK",
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/PacketPostReturn" } } }
}
}
}
},
[System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "13.17.0.0 (NJsonSchema v10.8.0.0 (Newtonsoft.Json v13.0.0.0))")]
public partial class UserMessage
{
/// <summary>
/// Swarm packet ID
/// </summary>
[Newtonsoft.Json.JsonProperty("packetId", Required = Newtonsoft.Json.Required.Always)]
public long PacketId { get; set; }
/// <summary>
/// Swarm message ID. There may be multiple messages for a single message ID. A message ID represents an intent to send a message, but there may be multiple Swarm packets that are required to fulfill that intent. For example, if a Hive -> device message fails to reach its destination, automatic retry attempts to send that message will have the same message ID.
/// </summary>
[Newtonsoft.Json.JsonProperty("messageId", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public long MessageId { get; set; }
/// <summary>
/// Swarm device type
/// </summary>
[Newtonsoft.Json.JsonProperty("deviceType", Required = Newtonsoft.Json.Required.Always)]
public int DeviceType { get; set; }
/// <summary>
/// Swarm device ID
/// </summary>
[Newtonsoft.Json.JsonProperty("deviceId", Required = Newtonsoft.Json.Required.Always)]
public int DeviceId { get; set; }
/// <summary>
/// Swarm device name
/// </summary>
[Newtonsoft.Json.JsonProperty("deviceName", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public string DeviceName { get; set; }
/// <summary>
/// Direction of message
/// </summary>
[Newtonsoft.Json.JsonProperty("direction", Required = Newtonsoft.Json.Required.Always)]
public int Direction { get; set; }
/// <summary>
/// Message data type, always = 6
/// </summary>
[Newtonsoft.Json.JsonProperty("dataType", Required = Newtonsoft.Json.Required.Always)]
public int DataType { get; set; }
/// <summary>
/// Application ID
/// </summary>
[Newtonsoft.Json.JsonProperty("userApplicationId", Required = Newtonsoft.Json.Required.Always)]
public int UserApplicationId { get; set; }
/// <summary>
/// Organization ID
/// </summary>
[Newtonsoft.Json.JsonProperty("organizationId", Required = Newtonsoft.Json.Required.Always)]
public int OrganizationId { get; set; }
/// <summary>
/// Length of data (in bytes) before base64 encoding
/// </summary>
[Newtonsoft.Json.JsonProperty("len", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public int Len { get; set; }
/// <summary>
/// Base64 encoded data string
/// </summary>
[Newtonsoft.Json.JsonProperty("data", Required = Newtonsoft.Json.Required.Always)]
[System.ComponentModel.DataAnnotations.Required(AllowEmptyStrings = true)]
public byte[] Data { get; set; }
/// <summary>
/// Swarm packet ID of acknowledging packet from device
/// </summary>
[Newtonsoft.Json.JsonProperty("ackPacketId", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public long AckPacketId { get; set; }
/// <summary>
/// Message status. Possible values:
/// <br/>0 = incoming message (from a device)
/// <br/>1 = outgoing message (to a device)
/// <br/>2 = incoming message, acknowledged as seen by customer. OR a outgoing message packet is on groundstation
/// <br/>3 = outgoing message, packet is on satellite
/// <br/>-1 = error
/// <br/>-3 = failed to deliver, retrying
/// <br/>-4 = failed to deliver, will not re-attempt
/// <br/>
/// </summary>
[Newtonsoft.Json.JsonProperty("status", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public int Status { get; set; }
/// <summary>
/// Time that the message was received by the Hive
/// </summary>
[Newtonsoft.Json.JsonProperty("hiveRxTime", Required = Newtonsoft.Json.Required.Always)]
[System.ComponentModel.DataAnnotations.Required(AllowEmptyStrings = true)]
public System.DateTimeOffset HiveRxTime { get; set; }
private System.Collections.Generic.IDictionary<string, object> _additionalProperties;
[Newtonsoft.Json.JsonExtensionData]
public System.Collections.Generic.IDictionary<string, object> AdditionalProperties
{
get { return _additionalProperties ?? (_additionalProperties = new System.Collections.Generic.Dictionary<string, object>()); }
set { _additionalProperties = value; }
}
}
After several attempts I gave up and have rebuilt the required Bumble bee hive integration with RestSharp
public async Task SendAsync(uint organisationId, uint deviceId, byte deviceType, ushort userApplicationId, byte[] data, CancellationToken cancellationToken)
{
await TokenRefresh(cancellationToken);
_logger.LogInformation("SendAsync: OrganizationId:{0} DeviceType:{1} DeviceId:{2} UserApplicationId:{3} Data:{4} Enabled:{5}", organisationId, deviceType, deviceId, userApplicationId, Convert.ToBase64String(data), _bumblebeeHiveSettings.DownlinkEnabled);
Models.MessageSendRequest message = new Models.MessageSendRequest()
{
OrganizationId = (int)organisationId,
DeviceType = deviceType,
DeviceId = (int)deviceId,
UserApplicationId = userApplicationId,
Data = data,
};
RestClientOptions restClientOptions = new RestClientOptions()
{
BaseUrl = new Uri(_bumblebeeHiveSettings.BaseUrl),
ThrowOnAnyError = true,
};
using (RestClient client = new RestClient(restClientOptions))
{
RestRequest request = new RestRequest("api/v1/messages", Method.Post);
request.AddBody(message);
request.AddHeader("Authorization", $"bearer {_token}");
// To save the limited monthly allocation of mesages downlinks can be disabled
if (_bumblebeeHiveSettings.DownlinkEnabled)
{
var response = await client.PostAsync<Models.MessageSendResponse>(request, cancellationToken);
_logger.LogInformation("SendAsync-Result:{Status} PacketId:{PacketId}", response.Status, response.PacketId);
}
}
}
The application now has a StartUpService which loads the Azure DeviceClient cache (Lazy Cache) in the background as the application starts up. If an uplink message is received from a SwarmDevice before, it has been loaded by the FunctionsStartup the DeviceClient information is cached and another connection to the Azure IoT Hub is not established.
I’m looking at building a webby user interface where users an interactivity list, create, edit, delete formatters with syntax highlighter support, and the executing the formatter with sample payloads.
Swarm Space Azure IoT Connector Identity Translation Gateway Architecture
This approach uses most of the existing building blocks, and that’s it no more changes.
I initially tried building a cache loader with BackgroundService so that the DeviceClient cache would start loading as the application started but interdependencies became problem.
public partial class Connector
{
[Function("BumblebeeHiveCacheRefresh")]
public async Task<IActionResult> BumblebeeHiveCacheRefreshRun([HttpTrigger(AuthorizationLevel.Function, "get")] CancellationToken cancellationToken)
{
_logger.LogInformation("BumblebeeHiveCacheRefresh start");
await _swarmSpaceBumblebeeHive.Login(cancellationToken);
foreach (SwarmSpace.BumblebeeHiveClient.Device device in await _swarmSpaceBumblebeeHive.DeviceListAsync(cancellationToken))
{
_logger.LogInformation("BumblebeeHiveCacheRefresh DeviceId:{DeviceId} DeviceName:{DeviceName}", device.DeviceId, device.DeviceName);
Models.AzureIoTDeviceClientContext context = new Models.AzureIoTDeviceClientContext()
{
// TODO seems a bit odd getting this from application settings
OrganisationId = _applicationSettings.OrganisationId,
//UserApplicationId = device.UserApplicationId, deprecated
DeviceType = (byte)device.DeviceType,
DeviceId = (uint)device.DeviceId,
};
switch (_azureIoTSettings.ApplicationType)
{
case Models.ApplicationType.AzureIotHub:
switch (_azureIoTSettings.AzureIotHub.ConnectionType)
{
case Models.AzureIotHubConnectionType.DeviceConnectionString:
await _azureDeviceClientCache.GetOrAddAsync<DeviceClient>(device.DeviceId.ToString(), (ICacheEntry x) => AzureIoTHubDeviceConnectionStringConnectAsync(device.DeviceId.ToString(), context));
break;
case Models.AzureIotHubConnectionType.DeviceProvisioningService:
await _azureDeviceClientCache.GetOrAddAsync<DeviceClient>(device.DeviceId.ToString(), (ICacheEntry x) => AzureIoTHubDeviceProvisioningServiceConnectAsync(device.DeviceId.ToString(), context, _azureIoTSettings.AzureIotHub.DeviceProvisioningService));
break;
default:
_logger.LogError("Azure IoT Hub ConnectionType unknown {0}", _azureIoTSettings.AzureIotHub.ConnectionType);
throw new NotImplementedException("AzureIoT Hub unsupported ConnectionType");
}
break;
case Models.ApplicationType.AzureIoTCentral:
await _azureDeviceClientCache.GetOrAddAsync<DeviceClient>(device.DeviceId.ToString(), (ICacheEntry x) => AzureIoTHubDeviceProvisioningServiceConnectAsync(device.DeviceId.ToString(), context, _azureIoTSettings.AzureIoTCentral.DeviceProvisioningService));
break;
default:
_logger.LogError("AzureIoT application type unknown {0}", _azureIoTSettings.ApplicationType);
throw new NotImplementedException("AzureIoT unsupported ApplicationType");
}
}
_logger.LogInformation("BumblebeeHiveCacheRefresh finish");
return new OkResult();
}
}
In the short-term loading the cache with a call to an Azure HTTPTrigger Function works but may timeout issues. When I ran the connector with my 100’s of devices simulator the function timed out every so often.
devMobile's blog 