Low Power Payload (LPP) Encoder

I originally started building my own Low Power Protocol(LPP) encoder because I could only find one other Github repository with a C# implementation. There hadn’t been any updates for a while and I wasn’t confident that I could make the code work on my nanoFramework and TinyCLR devices.

I started with the sample Mbed C code and did a largely mechanical conversion to C#. I then revisited some of the mathematics where floating point values were converted to an integer.

The original C++ code (understandably) had some language specific approaches which didn’t map well into C#. I then translated the code to C#

public void TemperatureAdd(byte channel, float celsius)
{
   if ((index + TemperatureSize) > buffer.Length)
   {
      throw new ApplicationException("TemperatureAdd insufficent buffer capacity");
   }

   short val = (short)(celsius * 10);

   buffer[index++] = channel;
   buffer[index++] = (byte)DataType.Temperature;
   buffer[index++] = (byte)(val >> 8);
   buffer[index++] = (byte)val;
}

One of my sensors was sending values with more decimal places than LPP supported and I noticed the value was not getting rounded e.g. 2.99 ->2.9 not 3.0 etc. So I revised my implementation to use Math.Round (which is supported by the nanoFramework and TinyCLR).

public void DigitalInputAdd(byte channel, bool value)
{
   #region Guard conditions
   if ((channel < Constants.ChannelMinimum) || (channel > Constants.ChannelMaximum))
   {
      throw new ArgumentException($"channel must be between {Constants.ChannelMinimum} and {Constants.ChannelMaximum}", "channel");
   }

   if ((index + Constants.DigitalInputSize) > buffer.Length)
   {
      throw new ApplicationException($"Datatype DigitalInput insufficent buffer capacity, {buffer.Length - index} bytes available");
   }
   #endregion

   buffer[index++] = channel;
   buffer[index++] = (byte)Enumerations.DataType.DigitalInput;

   // I know this is fugly but it works on all platforms
   if (value)
   {
      buffer[index++] = 1;
   }
   else
   {
     buffer[index++] = 0;
   }
 }

I then extracted out the channel and buffer size validation but I’m not certain this makes the code anymore readable/understandable

public void DigitalInputAdd(byte channel, bool value)
{
   IsChannelNumberValid(channel);
   IsBufferSizeSufficient(Enumerations.DataType.DigitalInput);

   buffer[index++] = channel;
   buffer[index++] = (byte)Enumerations.DataType.DigitalInput;

   // I know this is fugly but it works on all platforms
   if (value)
   {
      buffer[index++] = 1;
   }
   else
   {
      buffer[index++] = 0;
   }
}

The code runs on netCore, nanoFramework, and TinyCLRV2 just needs a few more unit tests and it will be ready for production. I started with an LPP encoder which I needed for one of my applications. I’m also working an approach for a decoder which will run on all my target platforms with minimal modification or compile time directives.

HTTP Headers GPRS Modem HTTP Post

All my initial deployments used a CAT5 cable and my ADSL connection which was great for testing and debugging but not really representative of the connectivity a mobile solution would experience.

For this test I used a seeedstudio GPRS shield on top of a Netduino Plus.

Netduino + SeeedStudio GPRS Modem

SeedStudio GPRS Modem on top of Netduino Plus

The shield is based on a SIM900 module from SIMCom Wireless and can also initiate HTTP requests. This functionality looked useful as it could make my code a bit simpler and reduce the load on the Netduino CPU.

I initially looked at a the Netduino driver for Seeeduino GSM shield on codeplex but it appeared to only support the sending of SMS messages. (Feb2012)

After some more searching I stumbled across the CodeFreak Out SeeedStudio GPRS Driver which is available as a Nuget package or source code. I had to modify the code to allow me to pass a list of HTTP headers to be added into the request

var gsm = new GPRSShield("www.vodafone.net.nz", SerialPorts.COM1);
gsm.Post(@"gpstrackerhttpheaders.cloudapp.net", 80, @"/posV4.aspx", httpHeaders, "application/html", "");

My simulated data used the same header format as in my earlier testing

x-Pos: 5C-86-4A-00-3F-63 20130218081228 F -43.00000 172.00000 31.4 1.31 0 0

I timed 10 requests

4789,3778,3793,3756,3796,3825,3806,3817,3795,3877

Average 3903 mSec

This was a bit slower than I was expecting so i’ll have to do some digging into the code and see if anything looks a bit odd.

HTTP Headers Payload encryption

So far the content of the messages has been sent as clear text which would not be acceptable for many applications. The requirement for data privacy causes a problem on the Netduino+ (Nov 2012) as the standard NetMF crypto libraries are not baked into the platform.

I then set about finding some crypto libraries which were NetMF friendly. RSA and Xtea are included in some of other NetMF platforms in the Microsoft.SPOT.Cryptography assembly so Xtea seemed like a reasonable choice to ensure interoperability.

When looking for crypto implementations one of the best sources is the Legion of the Bouncy Castle which was where I started. I downloaded the the V17.7 zip file had a look at the size of the Xtea code & all the associated support libraries and then parked that approach as I was worried about the size and performance of the resulting binaries.

I then looked at other possible tea, xtea & xxtea implementations (including porting the original C code to C#)

I am not a cryptographer so I can’t personally confirm the quality and correctness of an implementation. So after having interop problems I went back to the Legion of the Bouncy Castle which has been peer reviewed by experts and had another look. To get an Xtea implementation working on a .NetMF platform like the Netduino+ you need to include the following files…

  • CryptoException.cs
  • DataLengthException.cs
  • IBlockCipher.cs
  • ICipherParameters.cs
  • KeyParameter.cs
  • XTEAEngine.cs

On the Azure side of things where size is not so critical I just added a reference to the Bouncy Castle main project.

Xtea requires 128 bit blocks so you need to pad out the data on the client, then trim off the padding on the server.
// Pad out the data to a multiple of 8 bytes with spaces
if (xPosition.Length % 8 != 0)
{
xPosition += new string(' ', 8 - xPosition.Length % 8);
}

The key and the data need to be converted to byte arrarys, the Xtea engine initialised and a buffer for storing the encrypted data created.

byte[] dataBytes = Encoding.UTF8.GetBytes(xPosition);
byte[] keyBytes = Encoding.UTF8.GetBytes(key);

xteaEngine.Init(true, new KeyParameter(keyBytes));

Then the data can be encrypted in 8 byte chunks
byte[] cryptoBytes = new byte[dataBytes.Length];
for (int i = 0; i < dataBytes.Length; i += 8)
{
xteaEngine.ProcessBlock(dataBytes, i, cryptoBytes, i);
}

I hex encoded the encrypted data for transmission. Downside to this was it doubled the size of the payload
string hexEncodedCryptoBytes = ByteArrayToHex(cryptoBytes);

I added a StopWatch so I could measure the time taken to encrypt the position data (roughly 72 chars) on my Netduino+
285,342,277,345,282,345,342,350,278,343
Average 318mSec

The size of the payload had grown a bit
Request - Bytes Sent: 262
POST http://gpstrackerhttpheaders.cloudapp.net/posV7.aspx HTTP/1.1
Host: gpstrackerhttpheaders.cloudapp.net
x-Pos: 693A7AC6EBF4E5848CE8ABBA2BC6CAC1ED20574C1B2384E7E246A202C8A67E3DE14EE5231A5DF98C211F64F8402547F8BFDCC2241AAE3782A820086E5EF37AA2C50744941F588442
Content-Length: 0
Connection: Close

Response - Bytes Received: 132
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html
Date: Sun, 03 Feb 2013 04:53:30 GMT
Content-Length: 0

This increase in size had an impact on the time taken to send the message

1123,1144,1122,1142,1125,1125,1138,1111,1099,1141
Average 1127mSec

The binary downloaded to the Netduino+ had grown to 28K which still left plenty of space for additional functionality.

HTTP Headers Request reduction

The requests generated by the HTTP_Client were a bit chunky with user-agent strings, content type etc.

POST http://gpstrackerhttpheaders.cloudapp.net/posV4.aspx HTTP/1.1
Accept: */*
Accept-Language: en
User-Agent: NETMFToolbox/0.1 (textmode; Netduino; IntegratedSocket; HTTP_Client)
Host: gpstrackerhttpheaders.cloudapp.net
x-Pos: 5C-86-4A-00-3F-63 20130130080711 F -43.00000 172.00000 16.9 1.28 0 0
Content-Type: application/x-www-form-urlencoded
Content-Length: 0
Connection: Close

I then modified the HTTP_Client so that the accept, accept language, user agent, and content Type headers could be removed if not necessary.

POST http://gpstrackerhttpheaders.cloudapp.net/posV4.aspx HTTP/1.1
Host: gpstrackerhttpheaders.cloudapp.net
x-Pos: 5C-86-4A-00-3F-63 20130130081228 F -43.00000 172.00000 31.4 1.31 0 0
Content-Length: 0
Connection: Close

This reduced the size of the request down to 186 bytes which was comparable with the smallest System.http HTTPRequest. But, the durations looked a bit odd..

1295,1381,1323,1347,1281,1264,1351,1305,1350,1269
Average duration 1317 mSec

This was roughly 200mSec slower than the larger request version. After some digging I think the TCP socket was buffering (Nagle algorithm send coalescing) the send.

I then modified IntegratedSocket.cs connect method to disable this buffering

EndPoint Destination = new IPEndPoint(address.AddressList[0], (int)this._Port);
// Connects to the socket
this._Sock.Connect(Destination);
this._Sock.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);

I then checked the request durations again

10 requests to gpstrackerhttpheaders.cloudapp.net
1026,1248,983,1027,966,843,1009,983,834,915
Average duration 983 mSec

10 requests to IP Address
584, 595,578,597,581,595,577,593,580,599
Average duration 588 mSec

These were faster than any of the results with the baked in HttpWebRequest in System.http.

So I fired up .Net Reflector and had a look at the decompiled System.http code and though there are calls socket.SetSocketOption in EstablishConnection I can’t see any way for a user to set the NoDelay socket option as allowWriteStreamBuffering appears not to be used.

HTTP Headers NetMF Toolbox

The system.http assembly adds roughly 40K to the size of an application download. The NetMF toolbox has an HTTP client implementation which should significantly reduce the size of the download.

For my first proof of concept attempt, I downloaded the latest version of the NetMF toolbox (21733) and

  • Removed system.http reference
  • Added Toolbox.NETMF.NET.HTTP_Client (4.2)
  • AddedToolbox.NETMF.NET.Core (4.2)
  • Added Toolbox.NETMF.NET.Integrated (4.2)
  • Added Toolbox.NETMF.Core (4.2)
  • Modified code to use HTTP_Client instead of HttpWebRequest

The standard NetMF Toolbox HTTP_Client didn’t support adding HTTP headers to a request so I modified HTTP_Client.cs adding my code in the style of the cookies implementation. After a look at the HTTP request generation code I was interested to see what the performance of the NetMF Toolbox HTTP client was compared to the Microsoft one with my payload reducing tweaks.

gpstrackerhttpheaders.cloudapp.net
1122, 1175, 1200, 1169, 1162, 1153, 1150, 1262, 1171, 1152
Average 1171 mSec

Which was a bit slower than I expected

I then hacked the HTTP client code so I could use Fiddler to inspect the request & response payloads
Request Bytes Sent:351
POST http://gpstrackerhttpheaders.cloudapp.net/posV4.aspx HTTP/1.1
Accept: */*
Accept-Language: en
User-Agent: NETMFToolbox/0.1 (textmode; Netduino; IntegratedSocket; HTTP_Client)
Host: gpstrackerhttpheaders.cloudapp.net
x-Pos: 5C-86-4A-00-3F-63 20000101025524 F -43.00000 172.00000 25.2 0.80 0 0
Content-Type: application/x-www-form-urlencoded
Content-Length: 0
Connection: Close


Bytes Received:132
HTTP/1.1 200 OK
Cache-Control: private
x-UpdMin: 30
Date: Tue, 29 Jan 2013 02:55:25 GMT
Connection: close
Content-Length: 0

The downloaded went from 57K to 25k which is roughly a 32K reduction in size. The HTTP Request was a bit more chunky at 351 chars vs. 192 chars, but in a future post I’ll look into reducing it.

HTTP Headers threading

Running the HTTP requests and the NMEA stack on the foreground thread was always going to be a bit marginal. There were regular buffer overflows and memory allocation issues which the code would not always recover from. The simplest approach was to modify the code so a new thread is spawned for each HTTP request.

static void Gps_PositionChangedV5(uint data1, uint data2, DateTime time)
{
...
Thread thread = new Thread(() => HandleRequestSyncV5(Gps.Fix3D, Gps.GPSTime, Gps.Latitude, Gps.Longitude, Gps.HDoP, Gps.Altitude, Gps.Kmh, Gps.TrackAngle));
Thread.Start();
}

static void HandleRequestSyncV5(bool fix3D, DateTime gpsTime, float latitude, float longitude, float hDop, float altitude, float kmh, float trackAngle)
{
try
{
Stopwatch stopwatch = Stopwatch.StartNew();

using (HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(CloudServerUrl + "posV4.aspx"))
{
stopwatch.Stop();
...

10 requests to gpstrackerhttpheaders.cloudapp.net
1172,1143,1182,1136,1178,1257,1110,1117,1099,1225
Average 1161 mSec

10 requests to IP Address
1031,834,1033,1086,1029,1108,1049,1072,1036,819
Average 1010 mSec

The application is now quite stable but at roughly 57K2 bytes there isn’t a lot of space for any business functionality…

HTTP Headers Request reduction

Approaching the minimal request & response payload sizes.

Request Bytes Sent: 192
POST http://gpstrackerhttpheaders.cloudapp.net/posV3.aspx HTTP/1.1
x-Pos: 5C-86-4A-00-3F-63 20130126054537 T -43.00000 172.00000 12.2 4.08 1 83
Content-Length: 0
Connection: Keep-Alive
Host: gpstrackerhttpheaders.cloudapp.net

Response Bytes Received:113
HTTP/1.1 200 OK
Cache-Control: private
x-UpdMin: 30
Date: Sat, 26 Jan 2013 05:45:37 GMT
Content-Length: 0

10 requests to gpstrackerhttpheaders.cloudapp.net
944,969,1246,1043,1282,948,940,968,980,968
Average 1029 mSec

10 requests to IP Address
692,696,691,702,693,684,691,692,689,693
Average 692 mSec

Removing anymore headers from the request or response could break could HTTP break compatibility.

I went back and timed a V1 request & response to see what impact the reduction in payload size was…

10 requests to gpstrackerhttpheaders.cloudapp.net
2534,1457,1426,1571,1453,1462,1516,1422,1416,1421
Average 1569 mSec

10 requests to IP Address
1222,1200,1221,1238,1210,1266,1199,1235,1218,1230,1215
Average 1345 mSec

V1 573 bytes – 1569mSec & 1345 mSec
V2 480 bytes
V3 376 bytes
V4 305 bytes – 1029 mSec & 692 mSec

Most probably another 20 or 30 bytes could be saved by removing the ‘-‘ from the MAC address, shorter url, fixed width fields with no delimiter, removing the keep alive etc.

During my testing the NMEA stack was quite unstable it would fail with buffer overflows and overruns. The HTTP client requests are blocking and sharing the main execution thread meant the NMEA stack was unable to keep up.

HTTP Headers request duration Gadgeteer

I then ported the client application over to the Gadgeteer platform to see if that made a difference. I used a FEZSpider running .Net MF V4.2.

I then timed 10 requests to gpstrackerhttpheaders.cloudapp.net
1227,1153,1158,1156,1128,1130,1176,1224,1159,1225
Average 1174 mSec

IP Address
891,883,886,883,882,881,891,884,885,882
Average 884 mSec

These numbers are better than the Netduino+ but still not great. I need to go and take another look at my code to figure out what I am doing wrong.  For consistency with the Netduino+ I didn’t use the GHI premium Net library but will build a version of the client application which uses it in a future post.

HTTP Headers request duration Netduino+

After the byte counting I figured it was time to check the duration of the requests. I used a stopwatch class which was an approximate equivalent of the full .Net BCL one from netduino.com.

My house is cabled with CAT5 and I have a patch panel in a hall cupboard with gigabit switch and ADSL modem (Fibre is coming to my street soon). The Netduino was plugged into one of the spare ports and was running MF V4.2.

I then timed 10 requests to gpstrackerhttpheaders.cloudapp.net
1780, 1512, 1621, 1516, 1617, 1544, 1672, 1618,1490, 1614
Average 1599 mSec

My initial thought was maybe name resolution was broken so I replaced the hostname in the MF client with its IP address

1099, 1006, 1182, 1006, 1008, 1104,1030, 1092,1002, 1170
Average 1070 mSec – faster but not great

I then built a quick ‘n’ dirty desktop console application for comparison purposes.

gpstrackerhttpheaders.cloudapp.net
965, 243, 246, 242, 242, 249, 240, 242, 241, 518
Average 343 mSec

IP Address
603, 247, 241, 240, 241, 240, 239, 253, 243, 251
Average 280 mSec

The desktop times tally with what I see with other client applications calling Azure instances in the Singapore, West US and South Central US data centres.

I then did some digging and found others with similar results. These durations would go some way to explaining why the NMEA stack was having problems with buffer over runs and data corruption.

Time to move the HTTP request code off to a separate thread. Then have a look at system.http with Redgate Reflector  and see if anything looks suspicious.

HTTP Headers response reduction

In the previous post the response was still looking a bit chunky

Response – Bytes Received: 217
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html
Server: Microsoft-IIS/7.0
x-UpdMin: 30
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Jan 2013 10:25:12 GMT
Content-Length: 0

The HTTP headers returned by IIS were a significant portion of the response. I found that there were several different approaches but the simplest to setup was discussed here.

Response – Bytes Received: 113
HTTP/1.1 200 OK
Cache-Control: private
x-UpdMin: 30
Date: Sun, 20 Jan 2013 01:51:37 GMT
Content-Length: 0

V1 573 bytes
V2 480 bytes
V3 376 bytes

With a little effort roughly 200 bytes or 1/3 reduction in payload size. Still some scope for further reductions…