Azure Event Hub Updates from a NetMF Device

I had read about how Azure Event Hubs supported both Advanced Message Queuing Protocol(AMQP) & Hypertext Transfer Protocol (HTTP) access and was keen to see how easy the REST API was to use from a .Net Microframework (NetMF) device.

My initial concept was an exercise monitoring system with a Global Positioning System (GPS) unit and a pulse oximeter connected to a FEZ Spider device. My posting GPS Tracker Azure Service Bus has more info about GPS Drivers  and Azure Service Bus connectivity.

FEZ Spider, GPS and PulseOximeter

Fez spider and sensors for exercise monitoring device

The software was inspired by the Service Bus Event Hubs Getting started, Scale Out Event Processing with Event Hubs,Service Bus Event Hubs Large Scale Secure Publishing and OBD Recorder for .Net Micro Framework with ServiceBus, AMQP (for IoT) samples. I created an Event Hub and associated device access keys and fired up Service Bus Explorer so I could monitor and tweak the configuration.

I started by porting the REST API SendMessage implementation of Service Bus Event Hubs Large Scale Secure Publishing sample to NetMF. My approach was to get the application into my local source control and then cut ‘n’ paste the code into a NetMF project and see what breaks. I then modified the code over several iterations so it ran on both the desktop and NetMF clients.

The next step was to download the HTTPS certificates and add them to the project as resources so the requests could be secured. See this post for more detail.

For the connection to be secured you need to set the local time (so the certificate valid to/from can be checked) and load the certificates so they can be attached to the HTTP requests

void ProgramStarted()
{
   ...
   Microsoft.SPOT.Hardware.Utility.SetLocalTime(NtpClient.GetNetworkTime());
   caCerts = new X509Certificate[] { new X509Certificate(Resources.GetBytes(Resources.BinaryResources.Baltimore)) };

I used the Network Time Protocol (NTP) library from the OBD Recorder for .Net Micro Framework sample to get the current time.

The Service Bus Event Hubs Large Scale Secure Publishing uses an asynchronous HTTP request which is not available on the NetMF platform. So I had to replace it with a synchronous version.

static void EventHubSendMessage(string eventHubAddressHttps, string token, string messageBody)
{
   try
   {
      HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(eventHubAddressHttps + "/messages" + "?timeout=60" + ApiVersion);
      {
         ...
         request.Headers.Add("Authorization", token);
         request.Headers.Add("ContentType", "application/atom+xml;type=entry;charset=utf-8");
         byte[] buffer = Encoding.UTF8.GetBytes(messageBody);
         request.ContentLength = buffer.Length;

         // request body
         using (Stream stream = request.GetRequestStream())
         {
            stream.Write(buffer, 0, buffer.Length);
         }
         using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
         {
            Debug.Print("HTTP Status:" + response.StatusCode + " : " + response.StatusDescription);
         }
      }
   }
   catch (WebException we)
   {
      Debug.Print(we.Message);
   }
}

The code to generate the SAS Token also required some modification as string.format, timespan, and SHA256 functionality are not natively available on the .NetMF platform. The GetExpiry, and SHA256 implementations were part of the OBD Recorder for .Net Micro Framework sample.

static string CreateSasToken(string uri, string keyName, string key)
{
   // Set token lifetime to 20 minutes. When supplying a device with a token, you might want to use a longer expiration time.
   uint tokenExpirationTime = GetExpiry(20 * 60);

   string stringToSign = HttpUtility.UrlEncode(uri) + "\n" + tokenExpirationTime;
   var hmac = SHA.computeHMAC_SHA256(Encoding.UTF8.GetBytes(key), Encoding.UTF8.GetBytes(stringToSign));

   string signature = Convert.ToBase64String(hmac);
   signature = Base64NetMf42ToRfc4648(signature);

   string token = "SharedAccessSignature sr=" + HttpUtility.UrlEncode(uri) + "&sig=" + HttpUtility.UrlEncode(signature) + "&se=" + tokenExpirationTime.ToString() + "&skn=" + keyName;
   return token;
}

static uint GetExpiry(uint tokenLifetimeInSeconds)
{
   const long ticksPerSecond = 1000000000 / 100; // 1 tick = 100 nano seconds</code>

   DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0);
   TimeSpan diff = DateTime.Now.ToUniversalTime() - origin;

   return ((uint)(diff.Ticks / ticksPerSecond)) + tokenLifetimeInSeconds;
}

private static string Base64NetMf42ToRfc4648(string base64netMf)
{
   var base64Rfc = string.Empty;
   for (var i = 0; i < base64netMf.Length; i++)
   {
      if (base64netMf[i] == '!')
      {
         base64Rfc += '+';
      }
      else if (base64netMf[i] == '*')
      {
         base64Rfc += '/';
      }
      else
      {
         base64Rfc += base64netMf[i];
      }
   }
return base64Rfc;
}

The HttpUtility class came from the OBD Recorder for .Net Micro Framework sample. The Base64NetMf42ToRfc4648 functionality is still necessary on NetMF 4.3.

After a couple of hours I had data upload working.(No GPS data as the device was running on my desk where GPS coverage is poor)

ServiceBusExplorerEventHub

GPS Tracker Azure Service Bus

After a break from the GPSTracker samples I dug out my FEZ Spider devices, upgraded them to .NetMF 4.3 and downloaded the discontinued module drivers so my SeeedStudio GPS would work.

 

GPS Tracker using FEZ Spider mainboard

GPS Tracker built using FEZ Spider mainboard

I updated the root certificates in the Microsoft.ServiceBus.Micro resources to the current “Baltimore CyberTrust Root” ones using the process described here

The code is based on the OBD Recorder for .Net Micro Framework with ServiceBus, AMQP (for IoT)

The GPS is initialised with handlers for valid & invalid positions.

gpsStatusLED.TurnRed();
gps.InvalidPositionReceived += gps_InvalidPositionReceived;
gps.PositionReceived += gps_PositionReceived;

void gps_InvalidPositionReceived(GPS sender, EventArgs e)
{
   gpsStatusLED.TurnRed();
}

<code>void gps_PositionReceived(GPS sender, GPS.Position e)
{
   gpsStatusLED.TurnGreen();
}

Once the network interface has an IP address, the time on the FEZ Spider is set (so the certificate from and until times can be checked) and then the ServiceBus connection is initialised

IPAddress ip = IPAddress.GetDefaultLocalAddress();

// Setup the device time
if (ip != IPAddress.Any)
{
   ....
   DateTime networkTime = NtpClient.GetNetworkTime();
   Microsoft.SPOT.Hardware.Utility.SetLocalTime(networkTime);
...

   SASTokenProvider tp = new SASTokenProvider("device", "YourTopSecretKey=");
   messagingClient = new MessagingClient(new Uri(@"https://YourEndpoint.servicebus.windows.net/YourQueueName"), tp);</code>

   Once the GPS returns a valid position every so often a message is sent to the service bus queue

   SimpleMessage message = new SimpleMessage()
   {
      BrokerProperties = { { "SessionId", Guid.NewGuid().ToString()}, { "Label", "NMEAPositionData" } },
      Properties =
      {
         { "Latitude", gps.LastPosition.Latitude.ToString("F4") },
         { "Longitude", gps.LastPosition.Longitude.ToString("F4") },
      },
   };
   try
   {
      Debug.Print("Message send");
      messagingClient.Send(message);
      Debug.Print("Message sent OK");
   }
   catch (Exception ex)
   {
      Debug.Print(ex.Message);
   }

The send appeared to be quite slow (even on my home LAN so some further investigation is required)

6462mSec
6399mSec
6471mSec
6346mSec
7403mSec

6325mSec
6188mSec
6426mSec
6493mSec
6555mSec

Average 6506mSec

HTTPS with NetMF calling an Azure Service Bus endpoint

Back in Q1 of 2013 I posted a sample application which called an Azure Service Bus end point just to confirm that the certificate configuration etc. was good.

Since I published that sample the Azure Root certificate has been migrated so I have created a new project which uses the Baltimore Cyber Trust Root certificate.

The sample Azure ServiceBus client uses a wired LAN connection (original one used wifi module) and to run it locally you will have to update the Date information around line 24.

NetMF Gadgeteer SeeedStudio GPS module driver oddness

A few months ago I built a proof of concept NetMF 4.2 application for a client which uploaded data to a Windows Azure Service. Recently we had been talking about extending the application to include support for tagging of the uploaded data with position information.

I purchased a Gadgeeter GPS Module from SeeedStudio and did a trial integration of the module into the client’s application. Initially it was working but I noticed that after a while (I monitored the application using MF Deploy) it would start displaying buffer overflow messages. I did the usual thing and went looking for “slow” code in the GPS events, removed debug print statements and made sure my application and the GPS were “playing nice” but the problem persisted.

My modifications seemed to make little difference so I downloaded the Microsoft Gadgeeter source code from codeplex so I could have a closer look at how the GPS driver worked. (I also personally would also like the driver to return the HDoP, VDoP or PDoP so there is an indication of the accuracy of the position data)

I noticed that there was a GPSTestApp and fired it up to see what would happen. I connected to my FEZ Spider device using MF Deploy and below is the output. The GPS Test application threw an exception shortly after I started it, then started displaying “Buffer OVFLW”. (Odd thing is I it must still be processing some NMEA1803 strings)

I need to do some digging to figure out what’s going on as initially thought it was my code but as the demo application fails I’m not so sure 🙂

Found debugger!
Create TS.
Loading start at a0e00000, end a0e1383c
Assembly: mscorlib (4.2.0.0)     Assembly: Microsoft.SPOT.Native (4.2.0.0)     Assembly: Microsoft.SPOT.Security.PKCS11 (4.2.0.0)     Assembly: System.Security (4.2.0.0)  Loading Deployment Assemblies.
Attaching deployed file.
Assembly: Microsoft.SPOT.IO (4.2.0.0)  Attaching deployed file.
Assembly: GTM.Seeed.GPS (1.6.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Graphics (4.2.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Hardware (4.2.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Hardware.PWM (4.2.0.1)  Attaching deployed file.
Assembly: GHI.Premium.Hardware (4.2.9.0)  Attaching deployed file.
Assembly: System (4.2.0.0)  Attaching deployed file.
Assembly: Gadgeteer.Serial (2.42.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.TinyCore (4.2.0.0)  Attaching deployed file.
Assembly: GHI.Premium.System (4.2.9.0)  Attaching deployed file.
Assembly: GPSTestApp (1.0.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Net (4.2.0.0)  Attaching deployed file.
Assembly: System.IO (4.2.0.0)  Attaching deployed file.
Assembly: GHI.Premium.IO (4.2.9.0)  Attaching deployed file.
Assembly: Gadgeteer (2.42.0.0)  Attaching deployed file.
Assembly: Microsoft.SPOT.Hardware.SerialPort (4.2.0.0)  Attaching deployed file.
Assembly: GHIElectronics.Gadgeteer.FEZSpider (1.1.1.0)  Resolving.
GC: 1msec 28272 bytes used, 7311396 bytes available
Type 0F (STRING              ):     24 bytes
Type 15 (FREEBLOCK           ): 7311396 bytes
Type 17 (ASSEMBLY            ):  28176 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
GC: performing heap compaction...
Ready.

Using mainboard GHI Electronics FEZSpider version 1.0
Program Started
#### Exception System.InvalidOperationException – 0x00000000 (4) ####
#### Message:
#### System.IO.Ports.SerialPort::set_ReadTimeout [IP: 0009] ####
#### Gadgeteer.Interfaces.Serial::ReadLineProcess [IP: 0015] ####
Uncaught exception
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW

95 x Buffer OVFLW

Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
Buffer OVFLW
GPS last position NO POSITION FIX age 10675199.02:48:05.4775807
Buffer OVFLW
Buffer OVFLW

FEZ Spider NetMF 4.2 Wifi with Designer Support

I now have a GHI Wifi RS21 Module which I needed get working with a FEZ Spider running NetMF 4.2 (I have upgraded both my FEZ Spiders to 4.2 so no 4.1 vs. 4.2 comparison this time). Other developers seemed to be struggling to get connectivity with various earlier versions of the NetMF going working robustly.

(http://www.tinyclr.com/forum/topic?id=9404)
(http://www.tinyclr.com/forum/topic?id=9388)
(http://www.tinyclr.com/forum/topic?id=9502)
(http://www.tinyclr.com/forum/topic?id=9538)

So after flashing my device to 4.2.9.0 I built a quick and dirty demo to illustrate how I got my RS9110 based Wifi module working with designer support. For a real application I would move the initialisation out of the ProgramStarted and improve the exception handling but this sample code should be enough to get you started. I have checked and this code works after both a device reset or re-flashing.

FezSpider + Wifi

NetMF HTTP Client Possible Memory Leak

As part of another project I was using the .NetMF HTTPClient functionality to initiate an HTTP GET.

I was having problems with the application crashing after a while so fired up the Visual Studio debugger. Due to a configuration problem the application was kicking of a WebRequest every 10 secs which was significantly faster than intended.

After some poking around I found the application was crashing due to a memory leak of roughly 45K per https call (http is okay). I tried several different ways of declaring/loading the x509 certificate to see if that impacted what the GC could collect but it made little or no difference. I tried forcing a GC after each webrequest but this also appeared to make little or no difference. Basically the BINARY_BLOB_HEAD value increases until the application runs out of memory and then crashes.

To help identify the problem I built a small test harness which reliably reproduces the problem with minimum overhead. The code is available here my HTTP Client Sample.

HTTPS + certificate loaded into a global variable

7120164 bytes
7074420
7029252
6984036
6938952
6893736
6848520
6803304
6758088
6712740

HTTP + certificate loaded into a global variable

7165980 bytes
7167840
7167840
7167708
7167840
7167840
7167840
7167708
7167708
7167840

HTTP – no certificate

7165980 bytes
7167840
7167840
7167840
7167840
7167840
7167840
7167708
7167840
7167840

HTTPS – certificate loaded for every webrequest

7118640 bytes
7073028
7027860
6982644
6937296
6892080
6846864
6801780
6756432
6711348

HTTPS + certificate loaded into a global readonly variable

7118664 bytes
7072920
7027752
6982668
6937452
6892236
6846888
6801804
6756588
6711372

HTTPS Memory info – Start

Type 0F (STRING ): 3084 bytes
Type 11 (CLASS ): 14352 bytes
Type 12 (VALUETYPE ): 1512 bytes
Type 13 (SZARRAY ): 7860 bytes
Type 03 (U1 ): 3192 bytes
Type 04 (CHAR ): 852 bytes
Type 07 (I4 ): 1044 bytes
Type 0F (STRING ): 72 bytes
Type 11 (CLASS ): 2616 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 7118664 bytes
Type 16 (CACHEDBLOCK ): 216 bytes
Type 17 (ASSEMBLY ): 32688 bytes
Type 18 (WEAKCLASS ): 96 bytes
Type 19 (REFLECTION ): 192 bytes
Type 1B (DELEGATE_HEAD ): 864 bytes
Type 1D (OBJECT_TO_EVENT ): 240 bytes
Type 1E (BINARY_BLOB_HEAD ): 152496 bytes
Type 1F (THREAD ): 1536 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 1632 bytes
Type 22 (TIMER_HEAD ): 216 bytes
Type 27 (FINALIZER_HEAD ): 144 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3552 bytes

HTTPS Memory info – Finish

Type 0F (STRING ): 3084 bytes
Type 11 (CLASS ): 14364 bytes
Type 12 (VALUETYPE ): 1512 bytes
Type 13 (SZARRAY ): 7860 bytes
Type 03 (U1 ): 3192 bytes
Type 04 (CHAR ): 852 bytes
Type 07 (I4 ): 1044 bytes
Type 0F (STRING ): 72 bytes
Type 11 (CLASS ): 2616 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 6711372 bytes
Type 16 (CACHEDBLOCK ): 96 bytes
Type 17 (ASSEMBLY ): 32688 bytes
Type 18 (WEAKCLASS ): 96 bytes
Type 19 (REFLECTION ): 192 bytes
Type 1B (DELEGATE_HEAD ): 828 bytes
Type 1D (OBJECT_TO_EVENT ): 240 bytes
Type 1E (BINARY_BLOB_HEAD ): 559476 bytes
Type 1F (THREAD ): 1920 bytes
Type 20 (SUBTHREAD ): 192 bytes
Type 21 (STACK_FRAME ): 1632 bytes
Type 22 (TIMER_HEAD ): 216 bytes
Type 27 (FINALIZER_HEAD ): 168 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3552 bytes


...
using (HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(@"https://www.google.co.nz"))
{
request.Method = "GET";

request.HttpsAuthentCerts = caCerts;

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
PulseDebugLED();
}
Debug.Print(response.StatusDescription);
}
}
Debug.Print("Memory: " + Microsoft.SPOT.Debug.GC(true).ToString());
...

Time to run Reflector over the system.http assembly and see what is going on. I will also post on tinyclr.com to see if anyone else has encountered anything like this. Without my typo I most probably never have noticed this possible problem before deployment.

HTTPS with NetMF HTTP Client managing certificates

One of the services I needed to call from my Fez Spider required an HTTPS connection. The HTTP client sample located at

C:\Users\…\Documents\Microsoft .NET Micro Framework 4.2\Samples\HttpClient

shows how to load a certificate and use it when making a request.

There wasn’t a lot of information about getting the required certificate so I decided to document how I did it. On my Windows Server 2k8 box I use either a web browser or the Certificate Manager for exporting certificates. The easiest way is to use your preferred browser to access the service endpoint (To enable the export functionality you need to “Run as administrator”).

1.IECertificate

View the certificate

2.CertificateDetails

Select the root certificate

3.CertificatePath

View the root certificate information

4.CertificateRoot

View the root certificate details

5.CertificateRootDetails

Export the certificate

6.CertificateRootExport

Save CER file in the resources directory of your NetMF Project and then add it to the application resources.

If you know the Root Certification Authority you can export the certificate using Certificate Manager

Certificate Manager

Don’t forget to Update the SSL Seed using MF Deploy and ensure that the device clock is correct.

I use either an Network Time Protocol (NTP) Client or an RTC (Realtime Clock) module to set the device clock.

Depending on the application and device you might need to set the device clock every so often.

HTTP Headers request duration Gadgeteer

I then ported the client application over to the Gadgeteer platform to see if that made a difference. I used a FEZSpider running .Net MF V4.2.

I then timed 10 requests to gpstrackerhttpheaders.cloudapp.net
1227,1153,1158,1156,1128,1130,1176,1224,1159,1225
Average 1174 mSec

IP Address
891,883,886,883,882,881,891,884,885,882
Average 884 mSec

These numbers are better than the Netduino+ but still not great. I need to go and take another look at my code to figure out what I am doing wrong.  For consistency with the Netduino+ I didn’t use the GHI premium Net library but will build a version of the client application which uses it in a future post.

FEZ Spider NetMF 4.2 Networking revisited

I was determined to get the J11D Ethernet module working in the graphical Designer so after a lot of reading + trial and error….

http://www.tinyclr.com/forum/topic?id=9438
http://www.tinyclr.com/forum/topic?id=9661
http://www.tinyclr.com/forum/topic?id=9816
http://www.tinyclr.com/forum/topic?id=9845
http://www.tinyclr.com/forum/topic?id=10120

Making the J11D work after both reflashing the firmware & reboot was critical

FEZ Spider MF 4.1 and 4.2 Networking differences

I upgraded one of the two FEZ Spider boards I have to Version 4.2 of the .NETMF and this has caused some problems with the MF Deploy Network Configuration and also broke some networking code.

The MFDeploy Network configuration save fails with “Invalid configuration data from Plug-In” which others in the GHI Forums have also encountered e.g. http://www.tinyclr.com/forum/topic?id=9661

When I ran up my code the client IP address, gateway address etc. weren’t getting setup properly, My home network uses DHCP and this was fixed with .EnableDhcp(). I then noticed that the DNS address was not what I was expecting. It was pointing to one of the OpenDNS project servers. This was fixed by the .EnableDynamicDns();

This code works with V4.1

// This could be configured via MFDeploy
networkInterface.EnableDhcp();
networkInterface.EnableDynamicDns();

// error checking etc. removed

Debug.Print("Client IP address:" + networkInterface.IPAddress.ToString());
Debug.Print("MAC Address: " + BytesToHexString(networkInterface.PhysicalAddress));
Debug.Print("IP Address: " + networkInterface.IPAddress.ToString());
Debug.Print("Subnet Mask: " + networkInterface.SubnetMask.ToString());
Debug.Print("Gateway: " + networkInterface.GatewayAddress.ToString());
foreach (string dnsAddress in networkInterface.DnsAddresses)
{
Debug.Print("DNS Server: " + dnsAddress.ToString());
}

After some mucking around I found this code works for a Fez SPider running V4.2
static readonly EthernetBuiltIn Ethernet = new EthernetBuiltIn();


void ProgramStarted()
{
Ethernet.Open();

if (!Ethernet.IsActivated)
{
NetworkInterfaceExtension.AssignNetworkingStackTo(Ethernet);
}

// This code could be removed once the MFDeploy issue is sorted
Ethernet.NetworkInterface.EnableDhcp();
Ethernet.NetworkInterface.EnableDynamicDns();

// error checking etc. removed

Debug.Print("Client IP address:" + Ethernet.NetworkInterface.IPAddress.ToString());
Debug.Print("MAC Address: " + BytesToHexString( Ethernet.NetworkInterface.PhysicalAddress));
Debug.Print("IP Address: " + Ethernet.NetworkInterface.IPAddress.ToString());
Debug.Print("Subnet Mask: " + Ethernet.NetworkInterface.SubnetMask.ToString());
Debug.Print("Gateway: " + Ethernet.NetworkInterface.GatewayAddress.ToString());
foreach( string dnsAddress in Ethernet.NetworkInterface.DnsAddresses )
{
Debug.Print("DNS Server: " + dnsAddress.ToString());
}