Azure IoT Hubs LoRa Windows 10 IoT Core Field Gateway

This project is now live on github.com, sample Arduino with Dragino LoRa Shield for Arduino, MakerFabs Maduino, Dragino LoRa Mini Dev, M2M Low power Node and Netduino with Elecrow LoRa RFM95 Shield clients uploaded in the next couple of days.

AzureIOTHubExplorerScreenGrab20180912

The bare minimum configuration is

{
  "AzureIoTHubDeviceConnectionString": "HostName=qwertyuiop.azure-devices.net;DeviceId=LoRaGateway;SharedAccessKey=1234567890qwertyuiop987654321qwertyuiop1234g=",
  "AzureIoTHubTransportType": "Amqp",
  "SensorIDIsDeviceIDSensorID": true,
  "Address": "LoRaIoT1",
  "Frequency": 915000000.0
}

So far battery life and wireless communications range for the Arduino clients is looking pretty good. CRC presence checking and validation is turned so have a look at one of the sample clients.

ArduinoUnoR3DraginoLoRa
It took a bit longer than expected as upgrading to the latest version (v1.18.0 as at 12 Sep 2018) of Microsoft.Azure.Devices.Client (from 1.6.3) broke my field gateway with timeouts and exceptions.

I’ll be doing some more testing over the next couple of weeks so it is a work in progress.

ASP MVC Core V2.1 and Cross-Origin Resource Sharing

I’m working on an project for a customer which implements a number of application programming Interfaces(API) for a Single Page Application(SPA) and other clients. We are using entity tags (ETags) for versioning and the front end developers found the couldn’t access them from javascipt running in mainstream browser clients (June 2018).

The problems was understanding how Cross-Origin Resource Sharing (CORS) worked and how it interacted with our security model (API key and OAuth2.0 depending on the client)

In our scenario we first found the pre-flight check wasn’t working because in the HyperText Transfer Protocol (HTTP) OPTIONS method our X-API-KEY check was failing

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key
Access-Control-Request-Method: GET
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Server: Kestrel
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:48:30 GMT

13
API Key is invalid.
0

So I disabled X-API-KEY validation in startup.cs

public async Task Invoke(HttpContext context)
{
   if (context.Request.Method == "OPTIONS")
   {
      await this.next.Invoke(context);
      return;
   }

   var claims = new List();
…

OPTIONS then worked

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key
Access-Control-Request-Method: GET
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 404 Not Found
Server: Kestrel
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:52:20 GMT
Content-Length: 0

I then turned on CORS allowing pretty much anything

public void ConfigureServices(IServiceCollection services)
{
   services.AddCors(options =>
   {
      options.AddPolicy("CorsPolicy",
      builder => builder.AllowAnyOrigin()
         .AllowAnyMethod()
         .AllowAnyHeader()
         .AllowCredentials());
   });
   services.AddMvc();
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
   if (env.IsDevelopment())
   {
      app.UseDeveloperExceptionPage();
   }

   TelemetryConfiguration.Active.InstrumentationKey = this.configuration.GetSection("ApplicationInsights").GetSection("InstrumentationKey").Value;

   loggerFactory.AddLog4Net();
   this.log.Info("Startup.Configure called");

   app.ApplyUserKeyValidation();
   app.UseCors("CorsPolicy");
   app.UseMvc();
   }
}

OPTIONS then worked

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key
Access-Control-Request-Method: GET
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 204 No Content
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: x-api-key
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:57:33 GMT

GET then worked

GET http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
If-None-Match: 00-00-00-00-00-00-00-76
Host: xyz.azurewebsites.net
Connection: Keep-Alive

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
ETag: 00-00-00-00-00-00-00-76
Vary: Origin,Accept-Encoding
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:57:34 GMT
Content-Length: 2216

[{"...."}}

But HEAD didn’t work

HEAD http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
If-None-Match: 00-00-00-00-00-00-00-76
Host: xyz.azurewebsites.net
Connection: Keep-Alive

HTTP/1.1 400 Bad Request
Content-Length: 0
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:59:55 GMT

From the Application Insights logging and RestTest client (which I ran locally and remotely) I could see that the client side code couldn’t access the value of our eTag.  It had to be “exposed”

public void ConfigureServices(IServiceCollection services)
{
   services.AddCors(options =>
   {
      options.AddPolicy("CorsPolicy",
            builder => builder.AllowAnyOrigin()
            .AllowAnyMethod()
            .AllowAnyHeader()
            .WithExposedHeaders("etag")
            .AllowCredentials()
         );
      });
      services.AddMvc();
   }
...

GET then worked

GET http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
ETag: 00-00-00-00-00-00-00-76
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
If-None-Match: 00-00-00-00-00-00-00-76
Host: xyz.azurewebsites.net
Connection: Keep-Alive

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
ETag: 00-00-00-00-00-00-00-76
Vary: Origin,Accept-Encoding
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 07:53:41 GMT

[{"...."}}

HEAD then worked

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key,etag
Access-Control-Request-Method: HEAD
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 204 No Content
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: x-api-key,etag
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 07:57:31 GMT

HEAD http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
ETag: 00-00-00-00-00-00-00-76
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 304 Not Modified
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 07:57:31 GMT

I had some oddness with releasing code updates which I think was down to caching of pre-flight request responses.
Next steps tidy up the headers etc. and lock the CORS configuration down to expose the minimum necessary required for the application to work.

Azure Meetup-Budget tank of 91 IoT

The premise of my Azure Meetup presentation was could you build an interesting project on a rainy weekend afternoon with a constrained budget (tank of 91 octane petrol) and minimal soldering .

Budget

Our family car is a VW Passat V6 4Motion which has a 62 Litre tank. The driver usually doesn’t usually stop to fill up until the fuel light has been on for a bit which helped.

PetrolReceipt

Based on the most recent receipt the budget was NZD132.

Where possible I purchased parts locally (the tech equivalent of food miles) or on special.

My bill of materials (prices as at 2018-06) was on budget.

The devDuino V2.2 and nRF24L01 module were USD26.20 approx. NZD37.50 (including freight) from elecrow.

Tradeoffs

I powered my Raspberry PI with a spare cellphone charger (make sure it can supply enough current to reliably power the device).

The devDuino V2.has an ATSHA204A which provides a guaranteed unique 72-bit serial number (makes it harder to screw up provisioning devices in the field).

I use a 32G MicroSD rather than a 16G MicroSD card as I have had issued with 16G cards getting corrupted by more recent upgrades (possibly running out of space?)

The Raspberry PI shield requires a simple modification to enable interrupt driven operation.

My sample devDuino V2.2 client uses an external temperature and humidity sensor, modifying this code to use the onboard temperature sensor an MCP9700 will be covered in another post.

The devDuino V2 is a little bit cheaper USD15.99 NZD37.31, has the same onboard temperature sensor as the V2.2 but no unique serial number chip.

The devDuino V4.0 has an onboard HTU21D temperature + humidity sensor but no unique serial number and the batteries are expensive.

The code and deployment instructions for the nRF24L01 field gateway applications for AdaFruit.IO and Azure IoT Hub/Azure IoT Central are available on hackster.IO.

RPiWithnRF24Plate

AdaFruit.IO has free and USD10.00/month options which work well for many hobbyist projects.

AdaFruitIO

Azure Meetup Christchurch notes

For the people who came to my Azure meetup session this evening

Sources of sensors and development boards

http://www.adafruit.com
http://www.elecrow.com (watering kits)
http://www.ingenuitymicro.com (NZ based dev boards)
http://www.netduino.com (.NetMF development boards)
http://www.makerfabs.com
http://www.seeedstudio.com
http://www.tindie.com

nRF24Shields for RPI devices
http://www.tindie.com/products/ceech/new-raspberry-pi-to-nrf24l01-shield/

nRF24Shields for *duino devices in AU
embeddedcoolness.com

Raspberry PI Source in CHC
http://www.wavetech.co.nz

RFM69 & LoRa Modules
http://www.wisen.com.au

local sensor and device resellers quick turnaround
http://www.mindkits.co.nz
http://www.nicegear.co.nz

http://www.diyelectricskateboard.com

The watch development platform
http://www.hexiwear.com

http://www.gowifi.co.nz (Antennas & other wireless kit based in Rangiora)

my projects
http://www.hackster.io/KiwiBryn
io.adafruit.com/BrynHLewis/dashboards/home-environment

NLog and Application Insights

Another of my clients has an application which uses NLog and sooner or later they are going to want to move their logging to Azure Application Insights.

The application consists of a number of Azure websites and some embedded clients. The Azure applications log information to the local file system on each box but the number of boxes is growing so finding and tracing issues is becoming painful.

//---------------------------------------------------------------------------------
// Copyright (c) 2018, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//---------------------------------------------------------------------------------
using System;
using Microsoft.ApplicationInsights.Extensibility;
using NLog;

namespace devMobile.Azure.ApplicationInsightsNLogClient
{
   class Program
   {
      private static Logger log = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType.ToString());

      static void Main(string[] args)
      {
         if (args.Length != 1)
         {
            Console.WriteLine("Command line argument InstrumentationKey missing");
            return;
         }
         TelemetryConfiguration.Active.InstrumentationKey = args[0];

         log.Trace("This is nLog");
         log.Debug("This is a Debug message");
         log.Info("This is a Info message");
         log.Warn("This is a Warning message");
         log.Error("This is an Error message");
         log.Fatal("This is a Fatal message");

         new Microsoft.ApplicationInsights.TelemetryClient().Flush();
      }
   }
}

Sample code ApplicationInsightsNLogClient

Two clients to go, one which uses serilog the other has a DIY system which I’m ignoring as long as possible.

Apache Log4net and Application Insights

One of my clients had built a Fintech application which uses Apache log4net and I have just finished moving the logging to Azure Application Insights.

The application used to run on a couple of dedicated servers but its core processing engine is now run on an Azure Virtual Machine Scale Set(VMSS).

The application used to log information to the local file system on each server but with more machines (upto a dozen) and them being started up and shutdown in response to customer demand this wasn’t a viable approach.

//---------------------------------------------------------------------------------
// Copyright (c) 2018, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//---------------------------------------------------------------------------------
using System;
using log4net;
using Microsoft.ApplicationInsights.Extensibility;

namespace devMobile.Azure.ApplicationInsightsLog4NetClient
{
   class Program
   {
      public static ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

      static void Main(string[] args)
      {
         if (args.Length != 1)
         {
            Console.WriteLine("Command line argument InstrumentationKey missing");
            return;
         }
         TelemetryConfiguration.Active.InstrumentationKey = args[0];

         log.Info("This is Log4net");
         log.Debug("This is a Debug message");
         log.Info("This is a Info message");
         log.Warn("This is a Warning message");
         log.Error("This is an Error message");
         log.Fatal("This is a Fatal message");

         new Microsoft.ApplicationInsights.TelemetryClient().Flush();
      }
   }
}

Sample code ApplicationInsightsLog4NetClient

Microsoft Enterprise Library and Application Insights

One of my clients has a largish application (120+ projects) which uses the Microsoft Patterns and Practices Enterprise Library V6 data access, exception handling, logging and transient fault handling blocks.

To get consistent logging across Classic Cloud services, Azure websites and Azure functions etc. we are in the process of moving all our diagnostics to Azure application insights.

My proof of concept uses a community developed Enterprise Library listener and it appears to be working well.

Beware the Visual Studio configuration tool plug-in rewrites and application config file removing the application insights enterprise library trace listener setup.

The code for a smallest example application is below (I pass the instrumentation key as a command line parameter).

//---------------------------------------------------------------------------------
// Copyright (c) 2018, devMobile Software
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//---------------------------------------------------------------------------------
using System;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Practices.EnterpriseLibrary.Logging;
using Microsoft.Practices.EnterpriseLibrary.ExceptionHandling;
namespace ApplicationInsightsEnterpriseLibraryClient
{
   class Program
   {
      static void Main(string[] args)
      {
         if (args.Length != 1)
         {
            Console.WriteLine("Command line argument InstrumentationKey missing");
            return;
         }
         TelemetryConfiguration.Active.InstrumentationKey = args[0];

         LogWriterFactory logWriterFactory = new LogWriterFactory();
         LogWriter logWriter = logWriterFactory.Create();
         Logger.SetLogWriter(logWriter);

         ExceptionManager exceptionManager = new ExceptionPolicyFactory().CreateManager();
         ExceptionPolicy.SetExceptionManager(exceptionManager);

         logWriter.Write("This is Entlib", "General");

         logWriter.Write("Application startup", "Startup");

         logWriter.Write("General category", "General");
         logWriter.Write(new LogEntry() { Severity = System.Diagnostics.TraceEventType.Error, Categories = { "General" }, Message = "General category more complex overload", Title = "Dumpster fire" });

         try
         {
            throw new ApplicationException("Something bad has happened");
         }
         catch (Exception ex)
         {
            bool rethrow = ExceptionPolicy.HandleException(ex, "ProgramMain");
            if (rethrow)
               throw;
         }

         logWriter.Write("Application shutdown", "Shutdown");

         new Microsoft.ApplicationInsights.TelemetryClient().Flush();
      }
   }
}

Sample project ApplicationInsightsEnterpriseLibraryClient

Thanks to bveerendrakumar for sharing your code