Azure IoT Hub nRF24L01 Windows 10 IoT Core Field Gateway

This project is now live on Hackster.IO and github.com with sample *duino, Devduino and Netduino clients. While building the AdaFruit.IO field gateway, Azure IOT Hub field gateways and sample clients I changed the structure of the message payload and spent a bit of time removing non-core functionality and code.

The diagnostics logging code was refactored several times and after reading this reference on docs.Microsoft.com I settled on the published approach.

I considered using the built in Universal Windows Platform (UWP) application data class but this would have made configuration in the field hard for most of the targeted users school students & IT departments.

I have the application running at my house and it has proved pretty robust, last week I though it had crashed because the telemetry data stopped for about 20 minutes. I had a look at the Device portal and it was because Windows 10 IoT core had downloaded some updates, applied them and then rebooted automatically (as configured).

I put a socket on the Raspberry PI nRF24L01 Shield rather than soldering the module to the board so that I could compare the performance of the Low and High power modules. The antenna end of the high power module tends to droop so I put a small piece of plastic foam underneath to prop them up.

I had code to generate an empty JSON configuration but I removed that as it added complexity compared to putting a sample in the github repository.

I considered using a binary format (the nRF24L01 max message length is 32 bytes) but the code required to make it sufficiently flexible rapidly got out of hand and as most of my devices didn’t have a lot of sensors (battery/solar powered *duinos) and it wasn’t a major hassle to send another message so I removed it.

I need to tidy up the project and remove the unused Visual Assets and have a look at the automated update support.

Wireless field gateway protocol V2

I have now built a couple of nRF2L01P field gateways (for AdaFriut.IO & Azure IoT Hubs) which run as a background tasks on Windows 10 IoT Core on RaspberyPI). I have also written several clients which run on Arduino, devDuino, Netduino, and Seeeduino devices.

I have tried to keep the protocol simple (telemetry only) to deploy and it will be used in high school student projects in the next couple of weeks.

To make the payload smaller the first byte of the message now specifies the message type in the top nibble and the length of the device unique identifier in the bottom nibble.

0 = Echo

The message is displayed by the field gateway as text & hexadecimal.

1 = Device identifier + Comma separated values (CSV) payload

[0] – Set to 0001, XXXX   Device identifier length

[1]..[1+Device identifier length] – Unique device identifier bytes e.g. Mac address

[1+Device identifier length+1 ]..[31] – CSV payload e.g.  SensorID value, SensorID value

 

Microsoft IoT Central dynamic payload desktop client

Unlike most of the Azure IoT Hub client examples the names and number of sensor values will only be known when messages received over the nRF24L01 wireless link are processed so the JSON message payload has to be constructed on the fly.

Using the Newtonsoft.Json NuGet package and Linq + JObject made this much easier than expected so I have added some code improve robustness.

/*

Copyright ® 2018 Jan devMobile Software, All Rights Reserved

THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR
PURPOSE.

You can do what you want with this code, acknowledgment would be nice.

http://www.devmobile.co.nz

*/
using System;
using System.Text;
using System.Threading;
using Microsoft.Azure.Devices.Client;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace devMobile.IoT.MicrosoftIoTCentral.Desktop.DynamicPayload
{
   class Program
   {
      const string DeviceConnectionString = "YourDeviceConnectionStringFromIoTCentralGoesHere";
      const string TelemetryDataPointPropertyNameFormat = @"{0}-{1}";
      const double temperatureBase = 20.0;
      const double temperatureRange = 10.0;
      const double humidityBase = 70.0;
      const double humidityRange = 20.0;
      const double batteryVoltageBase = 3.00;
      const double batteryVoltageRange = -1.00;
      static readonly TimeSpan feedUpdateDelay = new TimeSpan(0, 0, 15);

      static void Main(string[] args)
      {
         DeviceClient Client = null;

         try
         {
            Console.WriteLine("Connecting to IoI hub");
            Client = DeviceClient.CreateFromConnectionString(DeviceConnectionString, TransportType.Amqp);
            Console.WriteLine(" Connected");
         }
         catch (Exception ex)
         {
            Console.WriteLine("Error connecting or sending data to IoT Central: {0}", ex.Message);
            return;
         }

         while (true)
         {
            // Then send simulated temperature, humidity & battery voltage data
            Random random = new Random();
            double temperature = temperatureBase + random.NextDouble() * temperatureRange;
            double humidity = humidityBase + random.NextDouble() * humidityRange;
            double batteryVoltage = batteryVoltageBase + random.NextDouble() * batteryVoltageRange;

            Console.WriteLine("Temperature {0}°C  Humidity {1}% Battery Voltage {2}V", temperature.ToString("F1"), humidity.ToString("F0"), batteryVoltage.ToString("F2"));

            // Populate the data point -
            JObject telemetryDataPoint = new JObject(); // This could be simplified but for field gateway will use this style

            string sensorDeviceSerialNumber = "0123456789ABCDEF"; // intentionally created and initialised at this level as sensor device will send over NRF24 link

            telemetryDataPoint.Add(string.Format(TelemetryDataPointPropertyNameFormat, sensorDeviceSerialNumber, "T"), temperature.ToString("F1"));
            telemetryDataPoint.Add(string.Format(TelemetryDataPointPropertyNameFormat, sensorDeviceSerialNumber, "H"), humidity.ToString("F0"));
            telemetryDataPoint.Add(string.Format(TelemetryDataPointPropertyNameFormat, sensorDeviceSerialNumber, "V"), batteryVoltage.ToString("F2"));

            string messageString = JsonConvert.SerializeObject(telemetryDataPoint);

            Console.WriteLine("{0:hh:mm:ss} > Sending telemetry: {1}", DateTime.Now, messageString);

            try
            {
               using (Message message = new Message(Encoding.ASCII.GetBytes(messageString)))
               {
                  Client.SendEventAsync(message).Wait();
                  Console.WriteLine(" Sent");
               }
            }
            catch (Exception ex)
            {
               Console.WriteLine("Error sending data to IoT Central: {0}", ex.Message);
            }

            Thread.Sleep(feedUpdateDelay);
         }
      }
   }
}

The application produces very similar output to the basic desktop client

IoTCentralDashboardDynamicPayloadClient

Microsoft IoT Central basic desktop client

One of the replacement Internet of Things services which looked worth evaluating was Microsoft’s IoT Central. My first project was to build the simplest possible desktop client (.Net Core) which simulates a limited number of sensors (sensor names, value formats etc. configured in code) and only sends data to the cloud (no device management, control or provisioning capabilities).

The only required dependencies are the Newtonsoft.Json &  Microsoft.Azure.DevicesClient NuGet packages

/*

Copyright ® 2018 Jan devMobile Software, All Rights Reserved

THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR
PURPOSE
You can do what you want with this code, acknowledgement would be nice.
http://www.devmobile.co.nz

*/
using System;
using System.Text;
using System.Threading;
using Microsoft.Azure.Devices.Client;
using Newtonsoft.Json;

namespace devMobile.IoT.MicrosoftIoTCentral.Desktop.Basic
{
 class Program
 {
 private const string DeviceConnectionString = "YourDeviceConnectionStringFromIoTCentralGoesHere";
 const double temperatureBase = 20.0;
 const double temperatureRange = 10.0;
 const double humidityBase = 70.0;
 const double humidityRange = 20.0;
 const double batteryVoltageBase = 3.00;
 const double batteryVoltageRange = -1.00;
 static readonly TimeSpan feedUpdateDelay = new TimeSpan(0, 0, 15);

private class TelemetryDataPoint
 {
 [JsonProperty(PropertyName = "H")]
 public double Humidity { get; set; }
 [JsonProperty(PropertyName = "T")]
 public double Temperature { get; set; }
 [JsonProperty(PropertyName = "B")]
 public double BatteryVoltage { get; set; }
 }

static void Main(string[] args)
 {
 DeviceClient Client ;
 Random random = new Random();

try
 {
 Console.WriteLine("Connecting to IoI hub");
 Client = DeviceClient.CreateFromConnectionString(DeviceConnectionString, TransportType.Mqtt);

while (true)
 {
 double temperature = temperatureBase + random.NextDouble() * temperatureRange;
 double humidity = humidityBase + random.NextDouble() * humidityRange;
 double batteryVoltage = batteryVoltageBase + random.NextDouble() * batteryVoltageRange;

Console.WriteLine("Temperature {0}°C Humidity {1}% Battery Voltage {2}V", temperature.ToString("F1"), humidity.ToString("F0"), batteryVoltage.ToString("F2"));

// Populate the data point - this has a static structure and name which could be a problem for field gateway
 TelemetryDataPoint telemetryDataPoint = new TelemetryDataPoint()
 {
 BatteryVoltage = Math.Round(batteryVoltage, 2),
 Humidity = Math.Round(humidity, 0),
 Temperature = Math.Round( temperature, 1 )
 };

string messageString = JsonConvert.SerializeObject(telemetryDataPoint);
 Message message = new Message(Encoding.ASCII.GetBytes(messageString));

Console.WriteLine("{0:hh:mm:ss} > Sending telemetry: {1}", DateTime.Now, messageString);
 Client.SendEventAsync(message).Wait();
 Console.WriteLine(" Done");

Thread.Sleep(feedUpdateDelay);
 }
 }
 catch (Exception ex)
 {
 Console.WriteLine("Error connecting or sending data to IoT Central: {0}", ex.Message);
 Console.WriteLine("Press <ENTER> to exit");
 Console.ReadLine();
 }
 }
 }
}

I manually provisioned the device by copying the device connection string in the IoT Central dashboard

IoTCentralDashboardBasicClient

DesktopClientSimple

Simple dotNet Core 2 IoTCentral Client

A functional client in less than 100 lines of code with support for individual device configuration. For my FieldGateway I’m going to need more flexibility in the construction of telemetry payloads, device provisioning and configuration support.

.Net Core & WCF TransportWithMessageCredential

In one of my day jobs I look after a system which has been around since 2010 (Early adopter of Microsoft Azure, developement started on .Net 3.5). The product has a number of Windows Communication Foundation(WCF) services hosted in an Azure CloudService.

A client built with .Net Core wanted to be able to call one of the services which was implemented using wsHttpBinding and TransportWithMessageCredential and this proved a bit more painful than expected…

I first tried the Visual Studio 2017 Microsoft WCF Web Service Reference Provider fromt the WCF Core Team.

The “add connected service” extension dialog allowed me to select an endpoint

ConfigureWCFWebSeriveReference

But the code generation process failed

WCFWebServiceReferenceError.png

The error message wasn’t particularly helpful so I used the command line utility svcutil to generate client classes. Which I used to built a .net core client with and the associated .Net Core WCF NuGet packages.

The console application failed when I called the service with a “PlatformNotSupportedException”. After some searching I found that the .Net Core WCF libraries don’t support TransportWithMessageCredential (September 2017).

Some more searching lead to a StackOverflow article where an answer suggested using the SimpleSOAPClient NuGet package. I then created a new client using the generated classes as the basis for the ones used in my SimpleSOAPClient proof of concept(PoC)

[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.ServiceModel", "4.0.0.0")]
[System.ServiceModel.MessageContractAttribute(WrapperName="Redeem", WrapperNamespace="http://qwertyuiop.com/services2011/08", IsWrapped=true)]
public partial class RedeemRequest
{
    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=1)]
    public string voucherCode;

    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=2)]
    public string merchantId;

    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=3)]
    public string merchantReference;

    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=4)]
    public string terminalId;

    public RedeemRequest()
    {
    }

    public RedeemRequest(string voucherCode, string merchantId, string merchantReference, string terminalId)
    {
        this.voucherCode = voucherCode;
        this.merchantId = merchantId;
        this.merchantReference = merchantReference;
        this.terminalId = terminalId;
    }
}

became

[XmlRoot("Redeem", Namespace = "http://qwertyuiop.com/services2011/08")]
public partial class RedeemRequest
{
   [XmlElement("voucherCode")]
   public string voucherCode;
   [XmlElement("transactionAmount")]
   public decimal transactionAmount;
   [XmlElement("merchantId")]
   public string merchantId;
   [XmlElement("merchantReference")]
   public string merchantReference;
   [XmlElement("terminalId")]
   public string terminalId;
}

This client failed with a SOAPAction related exception so I fired up Telerik Fiddler and found that the header was missing. When I manually added the header in the request composer (after dragging one of my failed requests onto the composer tab) it worked.

I had a look at the code in the SimpleSOAPClient repository to see how to add a custom HTTP Header to a request.

RedeemRequest redeemRequest = new RedeemRequest()
{
   merchantId = "......",
   merchantReference = "......",
   terminalId = "......",
   voucherCode = "......",
};

using (var client = SoapClient.Prepare())
{
   client.HttpClient.DefaultRequestHeaders.Add("SOAPAction", "http://qwertyuiop.com/services2011/08/IRedemptionProxyServiceV1/Redeem");
   var responseEnvelope = await client.SendAsync(
      "https://qwertyuiop.com/RedemptionProxy.svc",
      "https://qwertyuiop.com/services2011/08/IRedemptionProxyServiceV1/Redeem",
      SoapEnvelope.Prepare()
      .WithHeaders(KnownHeader.Oasis.Security.UsernameTokenAndPasswordText(".....", "......"))
      .Body(redeemRequest), ct);

      var response = responseEnvelope.Body<RedeemResponse>();

      Console.WriteLine("Redeem Result:{0}  Message:{1}", response.Result, response.messageText);
   }
}

After sorting out a few typos my request worked as expected. Only a couple of hours lost from my life, hopefully this post will help someone else.

nRF24L01 Raspberry PI Gateway Hardware

For those who came to my MS Ignite AU Intelligent Cloud booth session

Building Wireless Field Gateways

Connecting wireless sensor nodes to the cloud is not the mission it used to be, because the Azure team (and many OS projects) have developed tooling which can help hobbyist and professional developers build solutions. How could you build a home scale robust, reliable and secure solution with off the shelf kit without blowing the budget?

Sparkfun nRF24L01 module &Adafruit perma proto hat

NRF24L01 Raspberry PI DIY Gateway Hardware

BoM (all prices as at Feb 2016)

You will also need some short lengths of wire and a soldering iron.

For those who want an “off the shelf” solution (still requires a minor modification for interrupt support) I have used the Raspberry Pi to NRF24l01+ Shield USD9.90

2015-09-25t072754-447z-20150925_091942-855x570_q85_pad_rcrop

Instructions for modifications and software to follow.

Microsoft Sync Framework timezones

Over the last few months I have been working with the Microsoft Sync Framework and the time zone issues have been a problem.

New Zealand has a 12hr standard time or 13 hr daylight savings time offset from Coordinated Universal Time (UTC) and at a glance our customer data could look ok if treated as either local or UTC.

After some experimentation I found that it was due to Windows Communication Foundation(WCF) serialisation issues (The proposed solutions looks like it might have some limitations, especially across daylight savings time transitions).

For the initial synchronisation DateTime values in the database were unchanged, but for any later incremental synchronisations the DateTime values were adjusted to the timezone of the server (Our Azure Cloud Services are UTC timezone, though I don’t understand why Microsoft by default has them set to US locale with MM/DD/YY date formats)

In our scenario having all of the DateTime values in the cloud local looked like a reasonable option and this article provided some useful insights.

In the end I found that setting the DateSetDateTime  for every DateTime column in each DataTable in the synchronisation DataSet to unspecified in the ProcessChangeBatch (our code was based on the samples) method meant that no adjustment was applied to the incremental updates

public override void ProcessChangeBatch(ConflictResolutionPolicy resolutionPolicy, ChangeBatch sourceChanges, object changeDataRetriever, SyncCallbacks syncCallbacks, SyncSessionStatistics sessionStatistics)
{
try
{
DbSyncContext context = changeDataRetriever as DbSyncContext;

if (context != null)
{
foreach (DataTable table in context.DataSet.Tables)
{
foreach (DataColumn column in table.Columns)
{
// Switching from UnspecifiedLocal to Unspecified is allowed even after the DataSet has rows.
if ((column.DataType == typeof(DateTime)) && (column.DateTimeMode == DataSetDateTime.UnspecifiedLocal))
{
column.DateTimeMode = DataSetDateTime.Unspecified;
}
}
}
...

Hope this helps someone else