Random wanderings through Microsoft Azure esp. PaaS plumbing, the IoT bits, AI on Micro controllers, AI on Edge Devices, .NET nanoFramework, .NET Core on *nix and ML.NET+ONNX
After going back and looking at the module documentation and the diagnostic output I realised that the downlink message and confirmation were sent in two responses.
The first (optional) part of the response had the port number and message payload
+MSGHEX: PORT: 20; RX: "0504030201"
The second had the signal strength information
+MSGHEX: RXWIN1, RSSI -31, SNR 14.0
I had to add some code to the SerialDevice_DataReceived method for assembling the two responses. It would be good if the Seeedstudio LoRa-E5 only used one response. (Sample below based on RAK811)
at+send=lora:1:5A00
OK
at+recv=1,-105,-12,0
at+send=lora:1:5A00
OK
at+recv=0,-105,-12,8,00010203
The other LoRa-E5 implementation detail which frustrated me was the inclusion of labels for values e.g. PORT, RSSI, SNR etc.
The code currently isn’t very robust so when I accidentally used an invalid region, then AppEUI the responses weren’t consistent. When the region configuration failed the response was +DR: ERROR(-1) which maps to “Parameters is invalid” and when the Join failed the response was “+JOIN: Join failed”.
// Set the Region to AS923
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes("AT+DR=AS924\r\n"));
Debug.WriteLine($"TX: DR {txByteCount} bytes");
Thread.Sleep(500);
//Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
I had to add some code to the SerialDevice_DataReceived method for handling the “+JOIN: Join failed” case. It would be good if the Seeedstudio LoRa-E5 reported errors in a consistent way for all commands, without the ERROR(..) marker.
My Over the Air Activation (OTAA) implementation is very “nasty” as it is assumed that there are no timeouts or failures and it only sends one BCD message “48656c6c6f204c6f526157414e” which is “hello LoRaWAN”. The code just sequentially steps through the necessary commands (with a suitable delay after each is sent) to join the TTI network.
public class Program
{
#if TINYCLR_V2_FEZDUINO
private static string SerialPortId = SC20100.UartPort.Uart5;
#endif
private const string AppKey = "................................";
//txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+ID=AppEui,{AppEui}\r\n"));
//private const string AppEui = "................";
//txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+ID=AppEui,\"{AppEui}\"\r\n"));
private const string AppEui = ".. .. .. .. .. .. .. ..";
private const byte messagePort = 1;
//private const string payload = "48656c6c6f204c6f526157414e"; // Hello LoRaWAN
private const string payload = "01020304"; // AQIDBA==
//private const string payload = "04030201"; // BAMCAQ==
public static void Main()
{
UartController serialDevice;
int txByteCount;
int rxByteCount;
Debug.WriteLine("devMobile.IoT.SeeedE5.NetworkJoinOTAA starting");
try
{
serialDevice = UartController.FromName(SerialPortId);
serialDevice.SetActiveSettings(new UartSetting()
{
BaudRate = 9600,
Parity = UartParity.None,
StopBits = UartStopBitCount.One,
Handshaking = UartHandshake.None,
DataBits = 8
});
serialDevice.Enable();
// Set the Region to AS923
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes("AT+DR=AS923\r\n"));
Debug.WriteLine($"TX: DR {txByteCount} bytes");
Thread.Sleep(500);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
// Set the Join mode
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes("AT+MODE=LWOTAA\r\n"));
Debug.WriteLine($"TX: MODE {txByteCount} bytes");
Thread.Sleep(500);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
// Set the appEUI
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+ID=AppEui,\"{AppEui}\"\r\n"));
Debug.WriteLine($"TX: ID=AppEui {txByteCount} bytes");
Thread.Sleep(500);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
// Set the appKey
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+KEY=APPKEY,{AppKey}\r\n"));
Debug.WriteLine($"TX: KEY=APPKEY {txByteCount} bytes");
Thread.Sleep(500);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
// Set the PORT
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+PORT={messagePort}\r\n"));
Debug.WriteLine($"TX: PORT {txByteCount} bytes");
Thread.Sleep(500);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
// Join the network
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes("AT+JOIN\r\n"));
Debug.WriteLine($"TX: JOIN {txByteCount} bytes");
Thread.Sleep(10000);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
while (true)
{
// Unconfirmed message
txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+MSGHEX=\"{payload}\"\r\n"));
Debug.WriteLine($"TX: MSGHEX {txByteCount} bytes");
// Confirmed message
//txByteCount = serialDevice.Write(UTF8Encoding.UTF8.GetBytes($"AT+CMSGHEX=\"{payload}\"\r\n"));
//Debug.WriteLine($"TX: CMSGHEX {txByteCount} bytes");
Thread.Sleep(10000);
// Read the response
rxByteCount = serialDevice.BytesToRead;
if (rxByteCount > 0)
{
byte[] rxBuffer = new byte[rxByteCount];
serialDevice.Read(rxBuffer);
Debug.WriteLine($"RX :{UTF8Encoding.UTF8.GetString(rxBuffer)}");
}
Thread.Sleep(30000);
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
The code is not suitable for production but it confirmed my software and hardware configuration worked.
After some research I found references to the underlying problem in TTN and OpenAPI forums. The Dev_addr and Dev_eui fields are Base16(Hexidecimal) encoded binary but are being processed as if they were Base64(mime) encoded.
The TTI connector only displays the Device EUI so I changed the Dev_eui property to a string
Now the DeviceEUI values are displayed correctly and searching for EndDevices in Azure Application Insights is easier
TTI V3 Connector application running as a console application showing correct DeviceEUIs
Modifying the nSwag generated classes is a really nasty way of fixing the problem but I think this approach is okay as it’s only one field and any other solution I could find was significantly more complex.
To reduce the impact of the RegisterAsync call duration this Proof of Concept(PoC) code uses the System.Tasks.Threading library to execute each request in its own thread and then wait for all the requests to finish.
The connector application paginates the retrieval of device configuration from TTI API and a Task is created for each device returned in a page. In the Application Insights Trace logging the duration of a single page of device registrations was approximately the duration of the longest call.
There will be a tradeoff between device page size (resource utilisation by many threads) and startup duration (to many sequential page operations) which will need to be explored.
While debugging the connector on my desktop I had noticed that using a connection string was quite a bit faster than using DPS and I had assumed this was just happenstance. While doing some testing in the Azure North Europe data-center (Closer to TTI European servers) I grabbed some screen shots of the trace messages in Azure Application Insights as the TTI Connector Application was starting.
I only have six LoRaWAN devices configured in my TTI dev instance, but I repeated each test several times and the results were consistent so the request durations are reasonable. My TTI Connector application, IoT Hub, DPS and Application insights instances are all in the same Azure Region and Azure Resource Group so networking overheads shouldn’t be significant.
Using my own DPS instance to provide the connection string and then establishing a connection took between 3 and 7 seconds.
Azure IoT Central DPS
For my Azure IoT Central instance getting a connection string and establishing a connection took between 4 and 7 seconds.
The Azure DPS client code was copied from one of the sample applications so I have assumed it is “correct”.
using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
{
ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
Constants.AzureDpsGlobalDeviceEndpoint,
deviceProvisiongServiceSettings.IdScope,
securityProvider,
transport);
DeviceRegistrationResult result;
if (!string.IsNullOrEmpty(modelId))
{
ProvisioningRegistrationAdditionalData provisioningRegistrationAdditionalData = new ProvisioningRegistrationAdditionalData()
{
JsonData = $"{{"modelId": "{modelId}"}}"
};
result = await provClient.RegisterAsync(provisioningRegistrationAdditionalData, stoppingToken);
}
else
{
result = await provClient.RegisterAsync(stoppingToken);
}
if (result.Status != ProvisioningRegistrationStatusType.Assigned)
{
_logger.LogError("Config-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);
return false;
}
IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());
deviceClient = DeviceClient.Create(result.AssignedHub, authentication, transportSettings);
}
I need to investigate why getting a connection string from the DPS then connecting take significantly longer (I appreciate that “behind the scenes” service calls maybe required). This wouldn’t be an issue for individual devices connecting from different locations but for my Identity Translation Cloud gateway which currently open connections sequentially this could be a problem when there are a large number of devices.
If the individual requests duration can’t be reduced (using connection pooling etc.) I may have to spin up multiple threads so multiple devices can be connecting concurrently.
The first step was to configure and Azure IoT Central enrollment group (ensure “Automatically connect devices in this group” is on) and copy the IDScope and Group Enrollment key to the appsettings.json file (see sample file below for more detail)
At startup the TTI Gateway enumerates through the devices in each application configured in the app.settings.json. The Azure Device Provisioning Service(DPS) is used to retrieve each device’s connection string and configure it in Azure IoT Central if required.
Azure IoT Central Device Group with no provisioned DevicesTTI Connector application connecting and provisioning EndDevices
Azure IoT Central devices mapped to an Azure IoT Central Template via the modelID
using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
{
ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
Constants.AzureDpsGlobalDeviceEndpoint,
deviceProvisiongServiceSettings.IdScope,
securityProvider,
transport);
DeviceRegistrationResult result;
if (!string.IsNullOrEmpty(modelId))
{
ProvisioningRegistrationAdditionalData provisioningRegistrationAdditionalData = new ProvisioningRegistrationAdditionalData()
{
JsonData = $"{{\"modelId\": \"{modelId}\"}}"
};
result = await provClient.RegisterAsync(provisioningRegistrationAdditionalData, stoppingToken);
}
else
{
result = await provClient.RegisterAsync(stoppingToken);
}
if (result.Status != ProvisioningRegistrationStatusType.Assigned)
{
_logger.LogError("Config-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);
return false;
}
IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());
deviceClient = DeviceClient.Create(result.AssignedHub, authentication, transportSettings);
}
My implementation was “inspired” by TemperatureController project in the PnP Device Samples.
Azure IoT Central Dashboard with Seeeduino LoRaWAN devices around my house that were “automagically” provisioned
I need to do some testing to confirm my code works reliably with both DPS and user provided connection strings. The RegisterAsync call is currently taking about four seconds which could be an issue for TTI applications with many devices.
Device Twin Explorer displaying telemetry from one of the Seeeduino devices
My integration uses only queued messages as often they won’t be delivered to the sensor node immediately, especially if the sensor node only sends an uplink message every 30 minutes/hour/day.
The confirmed flag should be used with care as the Azure IoT Hub messages may expire before a delivery Ack/Nack/Failed is received from the TTI.
PowerBI graph of temperature and humidity in my garage over 24 hours
Device explorer displaying a raw payload message which has been confirmed delivered
TTI device live data tab displaying raw payload in downlink message information tabAzure IoT Connector console application sending raw payload to sensor node with confirmation ack
Arduino monitor displaying received raw payload from TTI
If the Azure IoT Hub message payload is valid JSON it is copied into the payload decoded downlink message property. and if it is not valid JSON it assumed to be a Base64 encoded value and copied into the payload raw downlink message property.
try
{
// Split over multiple lines in an attempt to improve readability. A valid JSON string should start/end with {/} for an object or [/] for an array
if (!(payloadText.StartsWith("{") && payloadText.EndsWith("}"))
&&
(!(payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
{
throw new JsonReaderException();
}
downlink.PayloadDecoded = JToken.Parse(payloadText);
}
catch (JsonReaderException)
{
downlink.PayloadRaw = payloadText;
}
Like the Azure IoT Central JSON validation I had to add a check that the string started with a “{” and finished with a “}” (a JSON object) or started with a “[” and finished with a “]” (a JSON array) as part of the validation process.
Device explorer displaying a JSON payload message which has been confirmed delivered
I normally wouldn’t use exceptions for flow control but I can’t see a better way of doing this.
TTI device live data tab displaying JSON payload in downlink message information tabAzure IoT Connector console application sending JSON payload to sensor node with confirmation ackArduino monitor displaying received JSON payload from TTI
The build in TTI decoder only supports downlink decoded payloads with property names “value_0” through “value_x” custom encoders may support other property names.
The first step was to display the temperature and barometric pressure values from the Seeedstudio Grove BMP180 attached to my sensor node.
Sensor node displaying temperature and barometric pressure values
Azure IoT Central temperature and barometric pressure telemetry configuration
Azure IoT Central Telemetry Dashboard displaying temperature and barometric pressure values
The next step was to configure a simple Azure IoT Central command to send to the sensor node. This was a queued request with no payload. An example of this sort of command would be a request for a sensor node to reboot or turn on an actuator.
My integration uses only offline queued commands as often messages won’t be delivered to the sensor node immediately, especially if the sensor node only sends a message every half hour/hour/day. The confirmed flag should be used with care as the Azure IoT Hub messages may expire before a delivery Ack/Nack/Failed is received from the TTI and it consumes downlink bandwidth.
if (message.Properties.ContainsKey("method-name"))
{
}
Azure IoT Central command with out a request payload value command configuration
To send a downlink message, TTI needs a LoRaWAN port number (plus optional queue, confirmed and priority values) which can’t be provided via the Azure IoT Central command setup so these values are configured in the app.settings file.
Each TTI application has zero or more Azure IoT Central command configurations which supply the port, confirmed, priority and queue settings.
Azure IoT Central simple command dashboardAzure IoT Central simple command initiationAzure IoT TTI connector application sending a simple command to my sensor nodeSensor node display simple command information. The note message payload is empty
The next step was to configure a more complex Azure IoT Central command to send to the sensor node. This was a queued request with a single value payload. An example of this sort of command could be setting the speed of a fan or the maximum temperature of a freezer for an out of band (OOB) notification to be sent.
Azure IoT Central single value command configuration
The value_0 settings are for the minimum temperature the value_1 settings are for the maximum temperature value.
Azure IoT Central single value command initiationAzure IoT TTI connector application sending a single value command to my sensor nodeSensor node displaying single value command information. There are two downlink messages and each payload contains a single value
The single value command payload contains the textual representation of the value e.g. “true”/”false” or “1.23” which are also valid JSON. This initially caused issues as I was trying to splice a single value into the decoded payload.
I had to add a check that the string started with a “{” and finished with a “}” (a JSON object) or started with a “[” and finished with a “]” (a JSON array) as part of the validation process.
For a single value command the payload decoded has a single property with the method-name value as the name and the payload as the value. For a command with a JSON payload the message payload is copied into the PayloadDecoded.
I normally wouldn’t use exceptions for flow control but I can’t see a better way of doing this.
try
{
// Split over multiple lines to improve readability
if (!(payloadText.StartsWith("{") && payloadText.EndsWith("}"))
&&
(!(payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
{
throw new JsonReaderException();
}
downlink.PayloadDecoded = JToken.Parse(payloadText);
}
catch (JsonReaderException)
{
try
{
JToken value = JToken.Parse(payloadText);
downlink.PayloadDecoded = new JObject(new JProperty(methodName, value));
}
catch (JsonReaderException)
{
downlink.PayloadDecoded = new JObject(new JProperty(methodName, payloadText));
}
}
The final step was to configure an another Azure IoT Central command with a JSON payload to send to the sensor node. A “real-world” example of this sort of command would be setting the minimum and maximum temperatures of a freezer in a single downlink message.
Azure IoT Central JSON payload command setup
Azure IoT Central JSON payload command payload configuration
Azure IoT TTI connector application sending a JSON payload command to my sensor node
Sensor node displaying JSON command information. There is a single payload which contains a two values
The build in TTI decoder only supports downlink decoded payloads with property names “value_0” through “value_x” which results in some odd command names and JSON payload property names. (Custom encoders may support other property names). Case sensitivity of some configuration values also tripped me up.
devMobile's blog 