Azure Event Hub Updates from a NetMF Device

I had read about how Azure Event Hubs supported both Advanced Message Queuing Protocol(AMQP) & Hypertext Transfer Protocol (HTTP) access and was keen to see how easy the REST API was to use from a .Net Microframework (NetMF) device.

My initial concept was an exercise monitoring system with a Global Positioning System (GPS) unit and a pulse oximeter connected to a FEZ Spider device. My posting GPS Tracker Azure Service Bus has more info about GPS Drivers  and Azure Service Bus connectivity.

FEZ Spider, GPS and PulseOximeter

Fez spider and sensors for exercise monitoring device

The software was inspired by the Service Bus Event Hubs Getting started, Scale Out Event Processing with Event Hubs,Service Bus Event Hubs Large Scale Secure Publishing and OBD Recorder for .Net Micro Framework with ServiceBus, AMQP (for IoT) samples. I created an Event Hub and associated device access keys and fired up Service Bus Explorer so I could monitor and tweak the configuration.

I started by porting the REST API SendMessage implementation of Service Bus Event Hubs Large Scale Secure Publishing sample to NetMF. My approach was to get the application into my local source control and then cut ‘n’ paste the code into a NetMF project and see what breaks. I then modified the code over several iterations so it ran on both the desktop and NetMF clients.

The next step was to download the HTTPS certificates and add them to the project as resources so the requests could be secured. See this post for more detail.

For the connection to be secured you need to set the local time (so the certificate valid to/from can be checked) and load the certificates so they can be attached to the HTTP requests

void ProgramStarted()
{
   ...
   Microsoft.SPOT.Hardware.Utility.SetLocalTime(NtpClient.GetNetworkTime());
   caCerts = new X509Certificate[] { new X509Certificate(Resources.GetBytes(Resources.BinaryResources.Baltimore)) };

I used the Network Time Protocol (NTP) library from the OBD Recorder for .Net Micro Framework sample to get the current time.

The Service Bus Event Hubs Large Scale Secure Publishing uses an asynchronous HTTP request which is not available on the NetMF platform. So I had to replace it with a synchronous version.

static void EventHubSendMessage(string eventHubAddressHttps, string token, string messageBody)
{
   try
   {
      HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(eventHubAddressHttps + "/messages" + "?timeout=60" + ApiVersion);
      {
         ...
         request.Headers.Add("Authorization", token);
         request.Headers.Add("ContentType", "application/atom+xml;type=entry;charset=utf-8");
         byte[] buffer = Encoding.UTF8.GetBytes(messageBody);
         request.ContentLength = buffer.Length;

         // request body
         using (Stream stream = request.GetRequestStream())
         {
            stream.Write(buffer, 0, buffer.Length);
         }
         using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
         {
            Debug.Print("HTTP Status:" + response.StatusCode + " : " + response.StatusDescription);
         }
      }
   }
   catch (WebException we)
   {
      Debug.Print(we.Message);
   }
}

The code to generate the SAS Token also required some modification as string.format, timespan, and SHA256 functionality are not natively available on the .NetMF platform. The GetExpiry, and SHA256 implementations were part of the OBD Recorder for .Net Micro Framework sample.

static string CreateSasToken(string uri, string keyName, string key)
{
   // Set token lifetime to 20 minutes. When supplying a device with a token, you might want to use a longer expiration time.
   uint tokenExpirationTime = GetExpiry(20 * 60);

   string stringToSign = HttpUtility.UrlEncode(uri) + "\n" + tokenExpirationTime;
   var hmac = SHA.computeHMAC_SHA256(Encoding.UTF8.GetBytes(key), Encoding.UTF8.GetBytes(stringToSign));

   string signature = Convert.ToBase64String(hmac);
   signature = Base64NetMf42ToRfc4648(signature);

   string token = "SharedAccessSignature sr=" + HttpUtility.UrlEncode(uri) + "&sig=" + HttpUtility.UrlEncode(signature) + "&se=" + tokenExpirationTime.ToString() + "&skn=" + keyName;
   return token;
}

static uint GetExpiry(uint tokenLifetimeInSeconds)
{
   const long ticksPerSecond = 1000000000 / 100; // 1 tick = 100 nano seconds</code>

   DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0);
   TimeSpan diff = DateTime.Now.ToUniversalTime() - origin;

   return ((uint)(diff.Ticks / ticksPerSecond)) + tokenLifetimeInSeconds;
}

private static string Base64NetMf42ToRfc4648(string base64netMf)
{
   var base64Rfc = string.Empty;
   for (var i = 0; i < base64netMf.Length; i++)
   {
      if (base64netMf[i] == '!')
      {
         base64Rfc += '+';
      }
      else if (base64netMf[i] == '*')
      {
         base64Rfc += '/';
      }
      else
      {
         base64Rfc += base64netMf[i];
      }
   }
return base64Rfc;
}

The HttpUtility class came from the OBD Recorder for .Net Micro Framework sample. The Base64NetMf42ToRfc4648 functionality is still necessary on NetMF 4.3.

After a couple of hours I had data upload working.(No GPS data as the device was running on my desk where GPS coverage is poor)

ServiceBusExplorerEventHub

NetMF Gadgeteer SeeedStudio GPS module driver oddness

A few months ago I built a proof of concept NetMF 4.2 application for a client which uploaded data to a Windows Azure Service. Recently we had been talking about extending the application to include support for tagging of the uploaded data with position information.

I purchased a Gadgeeter GPS Module from SeeedStudio and did a trial integration of the module into the client’s application. Initially it was working but I noticed that after a while (I monitored the application using MF Deploy) it would start displaying buffer overflow messages. I did the usual thing and went looking for “slow” code in the GPS events, removed debug print statements and made sure my application and the GPS were “playing nice” but the problem persisted.

My modifications seemed to make little difference so I downloaded the Microsoft Gadgeeter source code from codeplex so I could have a closer look at how the GPS driver worked. (I also personally would also like the driver to return the HDoP, VDoP or PDoP so there is an indication of the accuracy of the position data)

I noticed that there was a GPSTestApp and fired it up to see what would happen. I connected to my FEZ Spider device using MF Deploy and below is the output. The GPS Test application threw an exception shortly after I started it, then started displaying “Buffer OVFLW”. (Odd thing is I it must still be processing some NMEA1803 strings)

I need to do some digging to figure out what’s going on as initially thought it was my code but as the demo application fails I’m not so sure 🙂

Found debugger!
Create TS.
Loading start at a0e00000, end a0e1383c
Assembly: mscorlib (4.2.0.0)     Assembly: Microsoft.SPOT.Native (4.2.0.0)     Assembly: Microsoft.SPOT.Security.PKCS11 (4.2.0.0)     Assembly: System.Security (4.2.0.0)  Loading Deployment Assemblies.
Attaching deployed file.
Assembly: Microsoft.SPOT.IO (4.2.0.0)  Attaching deployed file.
Assembly: GTM.Seeed.GPS (1.6.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Graphics (4.2.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Hardware (4.2.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Hardware.PWM (4.2.0.1)  Attaching deployed file.
Assembly: GHI.Premium.Hardware (4.2.9.0)  Attaching deployed file.
Assembly: System (4.2.0.0)  Attaching deployed file.
Assembly: Gadgeteer.Serial (2.42.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.TinyCore (4.2.0.0)  Attaching deployed file.
Assembly: GHI.Premium.System (4.2.9.0)  Attaching deployed file.
Assembly: GPSTestApp (1.0.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Net (4.2.0.0)  Attaching deployed file.
Assembly: System.IO (4.2.0.0)  Attaching deployed file.
Assembly: GHI.Premium.IO (4.2.9.0)  Attaching deployed file.
Assembly: Gadgeteer (2.42.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Hardware.SerialPort (4.2.0.0)  Attaching deployed file.
Assembly: GHIElectronics.Gadgeteer.FEZSpider (1.1.1.0)  Resolving.
GC: 1msec 28272 bytes used, 7311396 bytes available
Type 0F (STRING              ):     24 bytes
Type 15 (FREEBLOCK           ): 7311396 bytes
Type 17 (ASSEMBLY            ):  28176 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
GC: performing heap compaction...
Ready.

Using mainboard GHI Electronics FEZSpider version 1.0
Program Started
#### Exception System.InvalidOperationException – 0x00000000 (4) ####
#### Message:
#### System.IO.Ports.SerialPort::set_ReadTimeout [IP: 0009] ####
#### Gadgeteer.Interfaces.Serial::ReadLineProcess [IP: 0015] ####
Uncaught exception
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW

95 x Buffer OVFLW

Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
Buffer OVFLW
Buffer OVFLW

NetMF HTTP Client Possible Memory Leak

As part of another project I was using the .NetMF HTTPClient functionality to initiate an HTTP GET.

I was having problems with the application crashing after a while so fired up the Visual Studio debugger. Due to a configuration problem the application was kicking of a WebRequest every 10 secs which was significantly faster than intended.

After some poking around I found the application was crashing due to a memory leak of roughly 45K per https call (http is okay). I tried several different ways of declaring/loading the x509 certificate to see if that impacted what the GC could collect but it made little or no difference. I tried forcing a GC after each webrequest but this also appeared to make little or no difference. Basically the BINARY_BLOB_HEAD value increases until the application runs out of memory and then crashes.

To help identify the problem I built a small test harness which reliably reproduces the problem with minimum overhead. The code is available here my HTTP Client Sample.

HTTPS + certificate loaded into a global variable

7120164 bytes
7074420
7029252
6984036
6938952
6893736
6848520
6803304
6758088
6712740

HTTP + certificate loaded into a global variable

7165980 bytes
7167840
7167840
7167708
7167840
7167840
7167840
7167708
7167708
7167840

HTTP – no certificate

7165980 bytes
7167840
7167840
7167840
7167840
7167840
7167840
7167708
7167840
7167840

HTTPS – certificate loaded for every webrequest

7118640 bytes
7073028
7027860
6982644
6937296
6892080
6846864
6801780
6756432
6711348

HTTPS + certificate loaded into a global readonly variable

7118664 bytes
7072920
7027752
6982668
6937452
6892236
6846888
6801804
6756588
6711372

HTTPS Memory info – Start

Type 0F (STRING ): 3084 bytes
Type 11 (CLASS ): 14352 bytes
Type 12 (VALUETYPE ): 1512 bytes
Type 13 (SZARRAY ): 7860 bytes
Type 03 (U1 ): 3192 bytes
Type 04 (CHAR ): 852 bytes
Type 07 (I4 ): 1044 bytes
Type 0F (STRING ): 72 bytes
Type 11 (CLASS ): 2616 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 7118664 bytes
Type 16 (CACHEDBLOCK ): 216 bytes
Type 17 (ASSEMBLY ): 32688 bytes
Type 18 (WEAKCLASS ): 96 bytes
Type 19 (REFLECTION ): 192 bytes
Type 1B (DELEGATE_HEAD ): 864 bytes
Type 1D (OBJECT_TO_EVENT ): 240 bytes
Type 1E (BINARY_BLOB_HEAD ): 152496 bytes
Type 1F (THREAD ): 1536 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 1632 bytes
Type 22 (TIMER_HEAD ): 216 bytes
Type 27 (FINALIZER_HEAD ): 144 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3552 bytes

HTTPS Memory info – Finish

Type 0F (STRING ): 3084 bytes
Type 11 (CLASS ): 14364 bytes
Type 12 (VALUETYPE ): 1512 bytes
Type 13 (SZARRAY ): 7860 bytes
Type 03 (U1 ): 3192 bytes
Type 04 (CHAR ): 852 bytes
Type 07 (I4 ): 1044 bytes
Type 0F (STRING ): 72 bytes
Type 11 (CLASS ): 2616 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 6711372 bytes
Type 16 (CACHEDBLOCK ): 96 bytes
Type 17 (ASSEMBLY ): 32688 bytes
Type 18 (WEAKCLASS ): 96 bytes
Type 19 (REFLECTION ): 192 bytes
Type 1B (DELEGATE_HEAD ): 828 bytes
Type 1D (OBJECT_TO_EVENT ): 240 bytes
Type 1E (BINARY_BLOB_HEAD ): 559476 bytes
Type 1F (THREAD ): 1920 bytes
Type 20 (SUBTHREAD ): 192 bytes
Type 21 (STACK_FRAME ): 1632 bytes
Type 22 (TIMER_HEAD ): 216 bytes
Type 27 (FINALIZER_HEAD ): 168 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3552 bytes


...
using (HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(@"https://www.google.co.nz"))
{
request.Method = "GET";

request.HttpsAuthentCerts = caCerts;

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
PulseDebugLED();
}
Debug.Print(response.StatusDescription);
}
}
Debug.Print("Memory: " + Microsoft.SPOT.Debug.GC(true).ToString());
...

Time to run Reflector over the system.http assembly and see what is going on. I will also post on tinyclr.com to see if anyone else has encountered anything like this. Without my typo I most probably never have noticed this possible problem before deployment.

NetMF cloud connectivity options

I do most of my dev work with NetMF devices and Windows Azure so I though it would be useful\interesting to explore the different cloud connectivity options.

I’ll start with the simplest possible HTTP based approach, look at how to reduce the device memory footprint, reduce traffic on the wire, secure the data, and take the solution mobile.

Then, as time allows I’ll build clients which use

The initial scenario is a GPS equiped NetMF device reporting position information to an application running in Windows Azure. Rather than reinventing the wheel I have used the NetMF Toolbox NMEA GPS module but have added HDoP reporting.

The samples will run on a Netduino, Netduino Plus or Fez Spider.

For the netduino based examples the BoM is

FEZ Spider NetMF 4.2 Networking revisited

I was determined to get the J11D Ethernet module working in the graphical Designer so after a lot of reading + trial and error….

http://www.tinyclr.com/forum/topic?id=9438
http://www.tinyclr.com/forum/topic?id=9661
http://www.tinyclr.com/forum/topic?id=9816
http://www.tinyclr.com/forum/topic?id=9845
http://www.tinyclr.com/forum/topic?id=10120

Making the J11D work after both reflashing the firmware & reboot was critical

void ProgramStarted()
{
Debug.Print("Program Started");

ethernet_J11D.Interface.Open();

if (!ethernet_J11D.Interface.IsActivated)
{
NetworkInterfaceExtension.AssignNetworkingStackTo(ethernet_J11D.Interface);
}
ethernet_J11D.Interface.NetworkAddressChanged += new NetworkInterfaceExtension.NetworkAddressChangedEventHandler(NetworkAddressChanged);

// The code below (Dynamic configuration or static configuration) could be removed once the MFDeploy fixed.
// This is a known issue see (http://wiki.tinyclr.com/index.php?title=NETMF4.2_developer)

// For networks with dynamic configuration
ethernet_J11D.Interface.NetworkInterface.EnableDhcp();
ethernet_J11D.Interface.NetworkInterface.EnableDynamicDns();

// For networks with static IP
//Ethernet.NetworkInterface.EnableStaticIP("x.x.x.x", "y.y.y.y", "z.z.z.z");
//Ethernet.NetworkInterface.EnableStaticDns(new[] { "x.x.x.x", "y.y.y.y" });

ipAddressReady.WaitOne();

// Display network config for debugging
Debug.Print("Network configuration");
Debug.Print(" Network interface type: " + ethernet_J11D.Interface.NetworkInterface.NetworkInterfaceType.ToString());
Debug.Print(" MAC Address: " + BytesToHexString(ethernet_J11D.Interface.NetworkInterface.PhysicalAddress));
Debug.Print(" DHCP enabled: " + ethernet_J11D.Interface.NetworkInterface.IsDhcpEnabled.ToString());
Debug.Print(" Dynamic DNS enabled: " + ethernet_J11D.Interface.NetworkInterface.IsDynamicDnsEnabled.ToString());
Debug.Print(" IP Address: " + ethernet_J11D.Interface.NetworkInterface.IPAddress.ToString());
Debug.Print(" Subnet Mask: " + ethernet_J11D.Interface.NetworkInterface.SubnetMask.ToString());
Debug.Print(" Gateway: " + ethernet_J11D.Interface.NetworkInterface.GatewayAddress.ToString());
foreach (string dnsAddress in ethernet_J11D.Interface.NetworkInterface.DnsAddresses)
{
Debug.Print(" DNS Server: " + dnsAddress.ToString());
}
}

void NetworkAddressChanged(object sender, EventArgs e)
{
Debug.Print("NetworkAddressChanged");
Debug.Print(" IP Address: " + ethernet_J11D.Interface.NetworkInterface.IPAddress.ToString());

if (ethernet_J11D.Interface.NetworkInterface.IPAddress != IPAddress.Any.ToString())
{
ipAddressReady.Set();
}
}

FEZ Spider MF 4.1 and 4.2 Networking differences

I upgraded one of the two FEZ Spider boards I have to Version 4.2 of the .NETMF and this has caused some problems with the MF Deploy Network Configuration and also broke some networking code.

The MFDeploy Network configuration save fails with “Invalid configuration data from Plug-In” which others in the GHI Forums have also encountered e.g. http://www.tinyclr.com/forum/topic?id=9661

When I ran up my code the client IP address, gateway address etc. weren’t getting setup properly, My home network uses DHCP and this was fixed with .EnableDhcp(). I then noticed that the DNS address was not what I was expecting. It was pointing to one of the OpenDNS project servers. This was fixed by the .EnableDynamicDns();

This code works with V4.1

// This could be configured via MFDeploy
networkInterface.EnableDhcp();
networkInterface.EnableDynamicDns();

// error checking etc. removed

Debug.Print("Client IP address:" + networkInterface.IPAddress.ToString());
Debug.Print("MAC Address: " + BytesToHexString(networkInterface.PhysicalAddress));
Debug.Print("IP Address: " + networkInterface.IPAddress.ToString());
Debug.Print("Subnet Mask: " + networkInterface.SubnetMask.ToString());
Debug.Print("Gateway: " + networkInterface.GatewayAddress.ToString());
foreach (string dnsAddress in networkInterface.DnsAddresses)
{
Debug.Print("DNS Server: " + dnsAddress.ToString());
}

After some mucking around I found this code works for a Fez SPider running V4.2
static readonly EthernetBuiltIn Ethernet = new EthernetBuiltIn();


void ProgramStarted()
{
Ethernet.Open();

if (!Ethernet.IsActivated)
{
NetworkInterfaceExtension.AssignNetworkingStackTo(Ethernet);
}

// This code could be removed once the MFDeploy issue is sorted
Ethernet.NetworkInterface.EnableDhcp();
Ethernet.NetworkInterface.EnableDynamicDns();

// error checking etc. removed

Debug.Print("Client IP address:" + Ethernet.NetworkInterface.IPAddress.ToString());
Debug.Print("MAC Address: " + BytesToHexString( Ethernet.NetworkInterface.PhysicalAddress));
Debug.Print("IP Address: " + Ethernet.NetworkInterface.IPAddress.ToString());
Debug.Print("Subnet Mask: " + Ethernet.NetworkInterface.SubnetMask.ToString());
Debug.Print("Gateway: " + Ethernet.NetworkInterface.GatewayAddress.ToString());
foreach( string dnsAddress in Ethernet.NetworkInterface.DnsAddresses )
{
Debug.Print("DNS Server: " + dnsAddress.ToString());
}

Fez Spider and Pulse Oximeter power problems

When I initially started writing code for the Seeed Studio Pulse Oximeter I found that I was getting no pulseOximeter_Heartbeat notifications and it seemed to take an awfully long time to get pulseOximeter_ProbeAttached or pulseOximeter_ProbeDetached notifications.

I was powering the kit from the USB port on my development desktop box via a USB Client SP Module. While doing some research I read this post. I then swapped to one of the USB ports on the mother board and the problems went away. I’m going to get a USB Client DP Module so this issue is permanently resolved.