Netduino 3 Wifi Queued Azure Event Hub Field Gateway V1.0

My ADSL connection had been a bit flaky which had meant I had lost some sensor data with my initial Azure Event Hub gateway. In attempt make the solution more robust this version of the gateway queues unsent messages using the on-board MicroSD card support.

The code assumes that a file move is an “atomic operation”, so it streams the events received from the devices into a temporary directory (configurable) then moves them to the upload directory (configurable).

This code is proof of concept and needs to be soak tested, improved error handling and some additional multi threading locking added plus the magic constants refactored.

This code is called in the nRF24 receive messages handler

private void OnReceive(byte[] data)
{
   activityLed.Write(!activityLed.Read());

   // Ensure that we have a payload
   if (data.Length < 1 )
   {
      Debug.Print( "ERROR - Message has no payload" ) ;
      return ;
   }

   string message = new String(Encoding.UTF8.GetChars(data));
   Debug.Print("+" + DateTime.UtcNow.ToString("HH:mm:ss") + " L=" + data.Length + " M=" + message);

   string filename = DateTime.UtcNow.ToString("yyyyMMddhhmmssff") + ".txt";

   string tempDirectory = Path.Combine("\\sd", "temp");
   string tempFilePath = Path.Combine(tempDirectory, filename);

   string queueDirectory = Path.Combine("\\sd", "data");
   string queueFilePath = Path.Combine(queueDirectory, filename);

   File.WriteAllBytes(tempFilePath, data);

   File.Move(tempFilePath, queueFilePath);

   new Microsoft.SPOT.IO.VolumeInfo("\\sd").FlushAll();
}

A timer initiates the upload process which uses the AMQPNetlite library

bool UploadInProgress = false;

      
void uploaderCallback(object state)
{
   Debug.Print("uploaderCallback - start");

   if (UploadInProgress)
   {
      return;
   }
   UploadInProgress = true;

   string[] eventFilesToSend = Directory.GetFiles(Path.Combine("\\sd", "data")) ;

   if ( eventFilesToSend.Length == 0 )
   {
      Debug.Print("uploaderCallback - no files");
      UploadInProgress = false;
      return ;
   }

   try
   {
      Debug.Print("uploaderCallback - Connect");
      Connection connection = new Connection(new Address(serviceBusHost, serviceBusPort, serviceBusSasKeyName, serviceBusSasKey));

      Session session = new Session(connection);

      SenderLink sender = new SenderLink(session, "send-link", eventHubName);

      for (int index = 0; index < System.Math.Min(eventUploadBatchSizeMaximum, eventFilesToSend.Length); index++)
      {
         string eventFile = eventFilesToSend[ index ] ;

         Debug.Print("-" + DateTime.UtcNow.ToString("HH:mm:ss") + " " + eventFile ); ;

         Message message = new Message()
         {
            BodySection = new Data()
            {
               Binary = File.ReadAllBytes(eventFile),
            },
         ApplicationProperties = new Amqp.Framing.ApplicationProperties(),
         };

         FileInfo fileInfo = new FileInfo(eventFile);

         message.ApplicationProperties["AcquiredAtUtc"] = fileInfo.CreationTimeUtc;
         message.ApplicationProperties["UploadedAtUtc"] = DateTime.UtcNow;
         message.ApplicationProperties["GatewayId"] = gatewayId;
         message.ApplicationProperties["DeviceId"] = deviceId;
         message.ApplicationProperties["EventId"] = Guid.NewGuid();

         sender.Send(message);

         File.Delete(eventFile);

         new Microsoft.SPOT.IO.VolumeInfo("\\sd").FlushAll();
      }

      sender.Close();
      session.Close();
      connection.Close();
   }
   catch (Exception ex)
   {
      Debug.Print("ERROR: Upload failed with error: " + ex.Message);
   }
   finally
   {
      Debug.Print("uploaderCallback - finally");
      UploadInProgress = false;
   }
}

The timer period and number of files uploaded in each batch is configurable. I need to test the application to see how it handles power outages and MicroSD card corruption. The source is Netduino NRF24L01 AMQPNetLite Queued Azure EventHub Gatewaywith all the usual caveats.

This project wouldn’t have been possible without

One thought on “Netduino 3 Wifi Queued Azure Event Hub Field Gateway V1.0

  1. Pingback: Mikrobus.Net Quail, Weather & nRF-C clicks and xively | devMobile's blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s