Netduino 3 Wifi Queued Azure Event Hub Field Gateway V1.0

My ADSL connection had been a bit flaky which had meant I had lost some sensor data with my initial Azure Event Hub gateway. In attempt make the solution more robust this version of the gateway queues unsent messages using the on-board MicroSD card support.

The code assumes that a file move is an “atomic operation”, so it streams the events received from the devices into a temporary directory (configurable) then moves them to the upload directory (configurable).

This code is proof of concept and needs to be soak tested, improved error handling and some additional multi threading locking added plus the magic constants refactored.

This code is called in the nRF24 receive messages handler

private void OnReceive(byte[] data)
{
   activityLed.Write(!activityLed.Read());

   // Ensure that we have a payload
   if (data.Length < 1 )
   {
      Debug.Print( "ERROR - Message has no payload" ) ;
      return ;
   }

   string message = new String(Encoding.UTF8.GetChars(data));
   Debug.Print("+" + DateTime.UtcNow.ToString("HH:mm:ss") + " L=" + data.Length + " M=" + message);

   string filename = DateTime.UtcNow.ToString("yyyyMMddhhmmssff") + ".txt";

   string tempDirectory = Path.Combine("\\sd", "temp");
   string tempFilePath = Path.Combine(tempDirectory, filename);

   string queueDirectory = Path.Combine("\\sd", "data");
   string queueFilePath = Path.Combine(queueDirectory, filename);

   File.WriteAllBytes(tempFilePath, data);

   File.Move(tempFilePath, queueFilePath);

   new Microsoft.SPOT.IO.VolumeInfo("\\sd").FlushAll();
}

A timer initiates the upload process which uses the AMQPNetlite library

bool UploadInProgress = false;

      
void uploaderCallback(object state)
{
   Debug.Print("uploaderCallback - start");

   if (UploadInProgress)
   {
      return;
   }
   UploadInProgress = true;

   string[] eventFilesToSend = Directory.GetFiles(Path.Combine("\\sd", "data")) ;

   if ( eventFilesToSend.Length == 0 )
   {
      Debug.Print("uploaderCallback - no files");
      UploadInProgress = false;
      return ;
   }

   try
   {
      Debug.Print("uploaderCallback - Connect");
      Connection connection = new Connection(new Address(serviceBusHost, serviceBusPort, serviceBusSasKeyName, serviceBusSasKey));

      Session session = new Session(connection);

      SenderLink sender = new SenderLink(session, "send-link", eventHubName);

      for (int index = 0; index < System.Math.Min(eventUploadBatchSizeMaximum, eventFilesToSend.Length); index++)
      {
         string eventFile = eventFilesToSend[ index ] ;

         Debug.Print("-" + DateTime.UtcNow.ToString("HH:mm:ss") + " " + eventFile ); ;

         Message message = new Message()
         {
            BodySection = new Data()
            {
               Binary = File.ReadAllBytes(eventFile),
            },
         ApplicationProperties = new Amqp.Framing.ApplicationProperties(),
         };

         FileInfo fileInfo = new FileInfo(eventFile);

         message.ApplicationProperties["AcquiredAtUtc"] = fileInfo.CreationTimeUtc;
         message.ApplicationProperties["UploadedAtUtc"] = DateTime.UtcNow;
         message.ApplicationProperties["GatewayId"] = gatewayId;
         message.ApplicationProperties["DeviceId"] = deviceId;
         message.ApplicationProperties["EventId"] = Guid.NewGuid();

         sender.Send(message);

         File.Delete(eventFile);

         new Microsoft.SPOT.IO.VolumeInfo("\\sd").FlushAll();
      }

      sender.Close();
      session.Close();
      connection.Close();
   }
   catch (Exception ex)
   {
      Debug.Print("ERROR: Upload failed with error: " + ex.Message);
   }
   finally
   {
      Debug.Print("uploaderCallback - finally");
      UploadInProgress = false;
   }
}

The timer period and number of files uploaded in each batch is configurable. I need to test the application to see how it handles power outages and MicroSD card corruption. The source is Netduino NRF24L01 AMQPNetLite Queued Azure EventHub Gatewaywith all the usual caveats.

This project wouldn’t have been possible without

Netduino 3 AnalogInput read rates

At CodeClub some of the students build a power consumption meter and as part of that project we measure the AnalogInput sample rates to check they are sufficient for our application.

Earlier this term when we measured the sampling rates in a CodeClub session we had a mix of Netduino 2 and Netduino 3 devices and some of the results differed from my previous observations. I used the same code on all the devices

int value;
AnalogInput x1 = new AnalogInput(Cpu.AnalogChannel.ANALOG_0);
stopwatch.Start();
for (int i = 0; i < SampleCount; i++)
{
value = x1.ReadRaw();
}
stopwatch.Stop();

Netduino Plus 2

Duration = 2081 mSec 48053/sec
Duration = 2082 mSec 48030/sec
Duration = 2081 mSec 48053/sec
Duration = 2081 mSec 48053/sec
Duration = 2082 mSec 48030/sec
Duration = 2081 mSec 48053/sec
Duration = 2081 mSec 48053/sec
Duration = 2081 mSec 48053/sec
Duration = 2081 mSec 48053/sec
Duration = 2081 mSec 48053/sec

Netduino 3

Duration = 2071 mSec 48285/sec
Duration = 2069 mSec 48332/sec
Duration = 2070 mSec 48309/sec
Duration = 2071 mSec 48285/sec
Duration = 2071 mSec 48285/sec
Duration = 2070 mSec 48309/sec
Duration = 2070 mSec 48309/sec
Duration = 2071 mSec 48285/sec
Duration = 2071 mSec 48285/sec
Duration = 2071 mSec 48285/sec

Netduino 3 Ethernet
Duration = 2136 mSec 46816/sec
Duration = 2137 mSec 46794/sec
Duration = 2136 mSec 46816/sec
Duration = 2135 mSec 46838/sec
Duration = 2135 mSec 46838/sec
Duration = 2137 mSec 46794/sec
Duration = 2137 mSec 46794/sec
Duration = 2135 mSec 46838/sec
Duration = 2136 mSec 46816/sec
Duration = 2135 mSec 46838/sec

Netduino 3 Wifii
Duration = 3902 mSec 25627/sec
Duration = 3901 mSec 25634/sec
Duration = 3902 mSec 25627/sec
Duration = 3902 mSec 25627/sec
Duration = 3901 mSec 25634/sec
Duration = 3903 mSec 25621/sec
Duration = 3903 mSec 25621/sec
Duration = 3902 mSec 25627/sec
Duration = 3902 mSec 25627/sec
Duration = 3903 mSec 25621/sec

The results for the Netduino 3 & Netduino 3 Ethernet were comparable with the Netduino Plus 2 in my earlier post. The reduction in the sampling rate of the Netduino 3 Wifi warrants some further investigation.

Netduino 3 Wifi pollution Sensor Part 2

In a previous post I had started building a driver for the Seeedstudio Grove Dust Sensor. It was a proof of concept and it didn’t handle some edge cases well.

While building the pollution monitor with a student we started by simulating the negative occupancy of the Shinyei PPD42NJ Particle sensor with the Netduino’s on-board button. This worked and reduced initial complexity. But it also made it harder to simulate the button being pressed as the program launches (the on-board button is also the reset button), or simulate if the button was pressed at the start or end of the period.

Dust sensor simulation with button

Netduino 3 Wifi Test Harness

The first sample code processes button press interrupts and displays the values of the data1 & data2 parameters

public class Program
{
   public static void Main()
   {
      InterruptPort button = new InterruptPort(Pins.GPIO_PIN_D5, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeBoth);
      button.OnInterrupt += button_OnInterrupt;

      Thread.Sleep(Timeout.Infinite);
   }

   static void button_OnInterrupt(uint data1, uint data2, DateTime time)
   {
      Debug.Print(time.ToString("hh:mm:ss.fff") + " data1 =" + data1.ToString() + " data2 = " + data2.ToString());
   }
}

Using the debugging output from this application we worked out that data1 was the Microcontroller Pin number and data2 was the button state.

12:00:14.389 data1 =24 data2 = 0
12:00:14.389 data1 =24 data2 = 1
12:00:14.389 data1 =24 data2 = 0
12:00:15.851 data1 =24 data2 = 1
12:00:16.078 data1 =24 data2 = 0

We then extended the code to record the duration of each button press.

public class Program
{
   static DateTime buttonLastPressedAtUtc = DateTime.UtcNow;

   public static void Main()
   {
      InterruptPort button = new InterruptPort(Pins.ONBOARD_BTN, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeBoth);
      button.OnInterrupt += button_OnInterrupt;

      Thread.Sleep(Timeout.Infinite);
   }

   static void button_OnInterrupt(uint data1, uint data2, DateTime time)
   {
      if (data2 == 0)
      {
         TimeSpan duration = time - buttonLastPressedAtUtc;

         Debug.Print(duration.ToString());
      }
      else
      {
         buttonLastPressedAtUtc = time;
      }
   }
}

The thread ” (0x4) has exited with code 0 (0x0).
00:00:00.2031790
00:00:00.1954150
00:00:00.1962350

The next step was to keep track of the total duration of the button presses since the program started executing.

public class Program
{
   static DateTime buttonLastPressedAtUtc = DateTime.UtcNow;
   static TimeSpan buttonPressedDurationTotal;

   public static void Main()
   {
      InterruptPort button = new InterruptPort(Pins.ONBOARD_BTN, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeBoth);
      button.OnInterrupt += button_OnInterrupt;

      Thread.Sleep(Timeout.Infinite);
   }

   static void button_OnInterrupt(uint data1, uint data2, DateTime time)
   {
      if (data2 == 0)
      {
         TimeSpan duration = time - buttonLastPressedAtUtc;

         buttonPressedDurationTotal += duration;
          Debug.Print(duration.ToString() + " " + buttonPressedDurationTotal.ToString());
      }
      else
      {
         buttonLastPressedAtUtc = time;
      }
   }
}

The thread ” (0x4) has exited with code 0 (0x0).
00:00:00.2476460 00:00:00.2476460
00:00:00.2193600 00:00:00.4670060
00:00:00.2631400 00:00:00.7301460
00:00:00.0001870 00:00:00.7303330

We then added a timer to display the amount of time the button was pressed in the configured period.

public class Program
{
   static TimeSpan measurementDueTime = new TimeSpan(0, 0, 30);
   static TimeSpan measurementperiodTime = new TimeSpan(0, 0, 30);
   static DateTime buttonLastPressedAtUtc = DateTime.UtcNow;
   static TimeSpan buttonPressedDurationTotal;


   public static void Main()
   {
      InterruptPort button = new InterruptPort(Pins.GPIO_PIN_D5, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeBoth);
      button.OnInterrupt += button_OnInterrupt;

      Timer periodTimer = new Timer(periodTimerProc, button, measurementDueTime, measurementperiodTime);

      Thread.Sleep(Timeout.Infinite);
   }

   static void periodTimerProc(object status)
   {
      InterruptPort button = (InterruptPort)status;

      if (button.Read())
      {
         TimeSpan duration = DateTime.UtcNow - buttonLastPressedAtUtc;

         buttonPressedDurationTotal += duration; 
      }

      Debug.Print(buttonPressedDurationTotal.ToString());

      buttonPressedDurationTotal = new TimeSpan(0, 0, 0);
      buttonLastPressedAtUtc = DateTime.UtcNow;
   }

   static void button_OnInterrupt(uint data1, uint data2, DateTime time)
   {
      if (data2 == 0)
      {
         TimeSpan duration = time - buttonLastPressedAtUtc;

         buttonPressedDurationTotal += duration;

         Debug.Print(duration.ToString() + " " + buttonPressedDurationTotal.ToString());
      }
      else
      {
         buttonLastPressedAtUtc = time;
      }
   }
}

The thread ” (0x4) has exited with code 0 (0x0).
00:00:00
00:00:00
00:00:00.2299050 00:00:00.2299050
00:00:00.1956980 00:00:00.4256030
00:00:00.1693190 00:00:00.5949220
00:00:00.5949220

After some testing we identified that the handling of button presses at the period boundaries was problematic and revised the code some more. We added a timer for the startup period to simplify the interrupt handling code.

public class Program
{
   static TimeSpan measurementDueTime = new TimeSpan(0, 0, 60);
   static TimeSpan measurementperiodTime = new TimeSpan(0, 0, 30);
   static DateTime buttonLastPressedAtUtc = DateTime.UtcNow;
   static TimeSpan buttonPressedDurationTotal;

   public static void Main()
   {
      InterruptPort button = new InterruptPort(Pins.GPIO_PIN_D5, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeBoth);
      button.OnInterrupt += button_OnInterrupt;

      Timer periodTimer = new Timer(periodTimerProc, button, Timeout.Infinite, Timeout.Infinite);

      Timer startUpTImer = new Timer(startUpTimerProc, periodTimer, measurementDueTime.Milliseconds, Timeout.Infinite);

      Thread.Sleep(Timeout.Infinite);
   }

   static void startUpTimerProc(object status)
   {
      Timer periodTimer = (Timer)status;

      Debug.Print( DateTime.UtcNow.ToString("hh:mm:ss") + " -Startup complete");

      buttonLastPressedAtUtc = DateTime.UtcNow;
      periodTimer.Change(measurementDueTime, measurementperiodTime);
   }

   static void periodTimerProc(object status)
   {
      InterruptPort button = (InterruptPort)status;
      Debug.Print(DateTime.UtcNow.ToString("hh:mm:ss") + " -Period timer");

      if (button.Read())
      {
         TimeSpan duration = DateTime.UtcNow - buttonLastPressedAtUtc;

         buttonPressedDurationTotal += duration;
      }

      Debug.Print(buttonPressedDurationTotal.ToString());

      buttonPressedDurationTotal = new TimeSpan(0, 0, 0);
      buttonLastPressedAtUtc = DateTime.UtcNow;
   }

   static void button_OnInterrupt(uint data1, uint data2, DateTime time)
   {
      Debug.Print(DateTime.UtcNow.ToString("hh:mm:ss") + " -OnInterrupt");

      if (data2 == 0)
      {
         TimeSpan duration = time - buttonLastPressedAtUtc;

         buttonPressedDurationTotal += duration;

         Debug.Print(duration.ToString() + " " + buttonPressedDurationTotal.ToString());
      }
      else
      {
         buttonLastPressedAtUtc = time;
      }
   }
}

The debugging output looked positive, but more testing is required.

The thread ” (0x2) has exited with code 0 (0x0).
12:00:13 -Startup complete
12:01:13 -Period timer
00:00:00
12:01:43 -Period timer
00:00:00
12:01:46 -OnInterrupt
12:01:48 -OnInterrupt
00:00:01.2132510 00:00:01.2132510
12:01:49 -OnInterrupt
12:01:50 -OnInterrupt
00:00:01.3001240 00:00:02.5133750
12:01:53 -OnInterrupt
12:01:54 -OnInterrupt
00:00:01.1216510 00:00:03.6350260
12:02:13 -Period timer
00:00:03.6350260

Next steps – multi threading, extract code into a device driver and extend to support sensors like the SeeedStudio Smart dust Sensor which has two digital outputs, one for small particles (e.g. smoke) the other for larger particles (e.g. dust).