Xively Personal is being retired

This is going to cause me a problem especially my Netduino based nRF24 Xively Field gateway which gets used in quite a few of my student projects. I’m looking for a replacement Internet of Things service which has http/s and/or mqtt, amqp support, C & C#  client libraries (which I can get to work on Windows 10 IoT Core & NetMF) would be a bonus.

From the Xively email

”After careful consideration, LogMeIn has made the decision to retire Xively Personal from its current line of products effective January 15, 2018 at 12:00PM ET . Please note that LogMeIn will continue to offer our Xively Enterprise edition – there is no change to that edition and we will continue to support that platform as part of our IoT business.

Retiring a product is never an easy decision, and we recognize it does introduce potential challenges to active users. So we want to make sure you have all the information you need to make as seamless a transition as possible.

Access to your account:
Your Xively Personal account will remain active until January 15th. Please note that devices will not be accessible via the Xively Personal service once it is retired.

Transferring your products to another IoT service:
Should you choose to switch to another service, there are essentially two options.

1) Migrate to Xively Enterprise: The latest Enterprise version of Xively is built on a more modern and reliable architecture, which brings the benefits of pre-built hardware integrations, identity and device management features, MQTT messaging, and best-in-class security, but it may require some reconfiguring of your current devices. We do offer a 30 day free trial of Xively Enterprise should you want to try it out for yourself.

2) Migrate to another free service: If your use is primarily for experimenting and personal projects, there are several free IoT platform options on the market, such as Adafruit, Thingspeak, or SparkFun.”

One of the suggestions – Sparkfun Phant has been retired

Some possible alternatives in no particular order (this list may grow)

AdaFruit.IO – The internet of things for everyone

Microsoft IoT Central – Enterprise-grade IoT SaaS

ThingSpeak – The open IoT platform with MATLAB analytics

Blynk – Democratizing the Internet of Things

Cayenne – Simplify the Connected World

Thinger.io platform

SenseIoT – Internet of Things Data Hosting Platform

Temboo – Tools for Digital Transformation

Carriots by Altair

Nearbus – An IoT Open Project

ubidots – An application Builder for the Internet of Things

Kii Cloud

Artik – End-to-end IoT Platform

goplusplatform – Connect your things with GO+

I’m initially looking for a platform which is the “least painful” transition from Xively.

Mikrobus.Net Quail, Weather & nRF-C clicks and xively

My next proof of concept uses a Weather click and nRF C click to upload temperature and humidity data to a Xively gateway running on a spare Netduino 2 Plus. I have a couple of Azure Event hub gateways (direct & queued) which require a Netduino 3 Wifi (for TLS/AMQPS support) and I’ll build a client for them in a coming post.

I initially purchased an nRF T click but something wasn’t quite right with its interrupt output. The interrupt line wasn’t getting pulled low at all so there were no send success/failure events. If I disabled the pull up resistor and strobed the interrupt pin on start-up the device would work for a while.


using (OutputPort Int = new OutputPort(socket.Int, true))
{
 Int.Write(true);
};

...

_irqPin = new InterruptPort(socket.Int, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeLow);

The code sends a reading every 10 seconds and has been running for a couple of days. It strobes Led1 for each successful send and turns on Led2 when a send fails.

private static readonly byte[] deviceAddress = Encoding.UTF8.GetBytes("Quail");
private static readonly byte[] gatewayAddress = Encoding.UTF8.GetBytes("12345");
private const byte gatewayChannel = 10;
private const NRFC.DataRate gatewayDataRate = NRFC.DataRate.DR1Mbps;
private const int XivelyUpdateDelay = 10000;
private const char XivelyGatewayChannelIdTemperature = 'J';
private const char XivelyGatewayChannelIdHumidity = 'K';

public static void Main()
{
   NRFC nRF24Click = new NRFC(Hardware.SocketFour);
   nRF24Click.Configure(deviceAddress, gatewayChannel, gatewayDataRate);
   nRF24Click.OnTransmitFailed += nRF24Click_OnTransmitFailed;
   nRF24Click.OnTransmitSuccess += nRF24Click_OnTransmitSuccess;
   nRF24Click.Enable();

   // Configure the weather click
   WeatherClick weatherClick = new WeatherClick(Hardware.SocketOne, WeatherClick.I2CAddresses.Address0);
   weatherClick.SetRecommendedMode(WeatherClick.RecommendedModes.WeatherMonitoring);

   Thread.Sleep(XivelyUpdateDelay);

   while (true)
   {
      string temperatureMessage = XivelyGatewayChannelIdTemperature + weatherClick.ReadTemperature().ToString("F1");
      Debug.Print(temperatureMessage);
      MBN.Hardware.Led1.Write(true);
      nRF24Click.SendTo(gatewayAddress, Encoding.UTF8.GetBytes(temperatureMessage));

      Thread.Sleep(XivelyUpdateDelay);

      string humidityMessage = XivelyGatewayChannelIdHumidity + weatherClick.ReadHumidity().ToString("F1");
      Debug.Print(humidityMessage);
      MBN.Hardware.Led1.Write(true);
      nRF24Click.SendTo(gatewayAddress, Encoding.UTF8.GetBytes(humidityMessage));

      Thread.Sleep(XivelyUpdateDelay);
   }
}

static void nRF24Click_OnTransmitSuccess()
{
   MBN.Hardware.Led1.Write(false);
   if (MBN.Hardware.Led2.Read())
   {
      MBN.Hardware.Led2.Write(false);
   }

   Debug.Print("nRF24Click_OnTransmitSuccess");
}

static void nRF24Click_OnTransmitFailed()
{
   MBN.Hardware.Led2.Write(true);

   Debug.Print("nRF24Click_OnTransmitFailed");
}

I need to have a look at interfacing some more sensors and soak testing the solution.

The MikroBus.Net team have done a great job with the number & quality of the drivers they have available.

Mikrobus.Net Quail, EthClick and xively

My second proof of concept application for the Mikrobus.Net Quail and EthClick uploads temperature and humidity data to Xively every 30 seconds for display and analysis.

Temperature and humidity Xively data stream

Temperature and humidity Xively data stream

The Xively REST API uses an HTTP PUT which initially didn’t work because the payload was not getting attached.

I patched the AssembleRequest method in the EtherClick driver to fix this issue.

private byte[] AssembleRequest()
{
   var a = RequestType;
   a += " " + Path + " " + Protocol + "\r\nHost: ";
   a += Host + "\r\n";

   foreach (object aHeader in Headers.Keys)
      a += (string)aHeader + ": " + (string)Headers[aHeader] + "\r\n";

   a += "\r\n"; // Cache-Control: no-cache\r\n  //Accept-Charset: utf-8;\r\n

   if (Content != null && Content != string.Empty && (RequestType == "POST" || RequestType == "PUT")) a += Content;

   return Encoding.UTF8.GetBytes(a);
}

The code reads the WeatherClick temperature and humidity values then assembles a CSV payload which it uploads with an HTTP PUT

</pre>
public class Program
{
   private const string xivelyHost = @"api.xively.com";
   private const string xivelyApiKey = @"YourAPIKey";
   private const string xivelyFeedId = @"YourFeedID";

   public static void Main()
   {
      WeatherClick weatherClick = new WeatherClick(Hardware.SocketOne, WeatherClick.I2CAddresses.Address0);
      weatherClick.SetRecommendedMode(WeatherClick.RecommendedModes.WeatherMonitoring);

      EthClick ethClick = new EthClick(Hardware.SocketTwo);
      ethClick.Start(ethClick.GenerateUniqueMacAddress("devMobileSoftware"), "QuailDevice");

      // Wait for an internet connection
      while (true)
      {
         if (ethClick.ConnectedToInternet)
         {
            Debug.Print("Connected to Internet");
            break;
         }
         Debug.Print("Waiting on Internet connection");
      }

      while (true)
      {
         Debug.Print("T " + weatherClick.ReadTemperature().ToString("F1") + " H " + weatherClick.ReadHumidity().ToString("F1") + " P " + weatherClick.ReadPressure(PressureCompensationModes.Uncompensated).ToString("F1"));

         HttpRequest request = new HttpRequest(@"http://" + xivelyHost + @"/v2/feeds/" + xivelyFeedId + @".csv");
         request.Host = xivelyHost;
         request.RequestType = "PUT";
         request.Headers.Add("Content-Type", "text/csv");
         request.Headers.Add("X-ApiKey", xivelyApiKey );

         request.Content = "OfficeT," + weatherClick.ReadTemperature().ToString("F1") + "\r\n" + "OfficeH," + weatherClick.ReadHumidity().ToString("F1") ;
         request.Headers.Add("Content-Length", request.Content.Length.ToString());

         var response = request.Send();
         if (response != null)
         {
            Debug.Print("Response: " + response.Message);
         }
         else
         {
            Debug.Print("No response");
         }
      Thread.Sleep(30000);
      }
   }
}
MikrobustNet Quail with Eth and Weather Clicks

MikrobustNet Quail with Eth and Weather Clicks

This proof of concept code appears to be reliable and has run for days at a time. The IP stack looks like it needs a bit more work.

Netduino 3 Wifi xively nRF24L01 Gateway

The first version of this code acquired data from a number of *duino devices and uploaded it to xively for a week without any problems(bar my ADSL modem dropping out every so often which it recovered from without human intervention). The data streams are the temperature and humidity for the three bedrooms in my house (the most reliable stream is Bedroom 1). Next version will use the new Netduino.IP stack and run on a Netduino 2 Plus

Netduino 3 Wifi with nRF24L01 shield

Netduino 3 Wifi + nRF24L01 shield

To make the software easy to setup all the gateway configuration is stored on a MicroSD and can be modified with a text editor. When the application starts it looks for a file in the root directory of the MicroSD card called app.config. If the file does not exist an empty template is created.

httprequestreadwritetimeoutmsec=2500
httprequesttimeoutmsec=2500
webproxyaddress=
webproxyport=
xivelyapibaseurl=http://api.xively.com/v2/feeds/
xivelyapikey=XivelyAPIKeyGoesHere
xivelyapifeedid=XivelyFeedIDGoesHere
xivelyapicontenttype=text/csv
xivelyapiendpoint=.csv
nrf2l01address=AddressGoesHere
nrf2l01channel=ChannelGoesHere
nrf2l01datarate=0
channel1=Sensor1
channel2=Sensor2
channel3=Sensor3
channel4=Sensor4
channel5=Sensor5
...
...

The first byte of each (upto 32 byte) nRF24L01 message is used to determine the Xively channel.

For testing I used a simple *duino program which uploads temperature and humidity readings every 5 seconds. It’s not terribly efficient or elegant and is just to illustrate how to package up the data.

#include <RF24_config>
#include <nRF24L01.h>
#include <SPI.h>
#include <RF24.h>
#include "Wire.h"
#include <TH02_dev.h>

//UNO R3 with embedded coolness board
//RF24 radio(3, 7);
//devDuino  with onboard
RF24 radio(8, 7);

char payload[32] = "";
const uint64_t pipe = 0x3165736142LL; // Base1 pay attention to byte ordering and address length

void setup()
{
  Serial.begin(9600);

  radio.begin();
  radio.setPALevel(RF24_PA_MAX);
  radio.setChannel(10);
  radio.enableDynamicPayloads();
  radio.openWritingPipe(pipe);

  radio.printDetails();

  /* Power up,delay 150ms,until voltage is stable */
  delay(150);

  TH02.begin();

  delay(1000);
}

void loop()
{
  float temperature = TH02.ReadTemperature();
  float humidity = TH02.ReadHumidity();

  radio.powerUp();

  payload[0] = 'A';
  dtostrf(temperature, 5, 1, &payload[1]);
  Serial.println(payload);
  boolean result = radio.write(payload, strlen(payload));
  if (result)
    Serial.println("T Ok...");
  else
    Serial.println("T failed.");

  payload[0] = 'B';
  dtostrf(humidity, 5, 1, &payload[1]);
  Serial.println(payload);
  result = radio.write(payload, strlen(payload));
  if (result)
    Serial.println("H Ok...");
  else
    Serial.println("H failed.");

  radio.powerDown();

  delay(5000);
}

The gateway code creates a thread for each call to the Xively REST API. (In future the code may need to limit the number of concurrent requests)

private void OnReceive(byte[] data)
{
   activityLed.Write(!activityLed.Read());

   // Ensure that we have a valid payload
   if ( data.Length == 0 )
   {
      Debug.Print( "ERROR - Message has no payload" ) ;
      return ;
   }

   // Extract the device id
   string deviceId = xivelyApiChannleIDPrefix + data[0].ToString();
   string message = new String(Encoding.UTF8.GetChars(data, 1, data.Length - 1));

   string xivelyApiChannel = appSettings.GetString( deviceId, string.Empty ) ;
   if ( xivelyApiChannel.Length == 0 )
   {
      Debug.Print("ERROR - Inbound message has unknown channel " + deviceId);
      return ;
   }
   Debug.Print(DateTime.Now.ToString("HH:mm:ss") + " " + xivelyApiChannel + " " + message); ;

   Thread thread = new Thread(() =&gt; xivelyFeedUpdate(xivelyApiChannel, message ));
   thread.Start();
   }

private void xivelyFeedUpdate( string channel, string value)
{
   #region Assertions
   Debug.Assert(channel != null);
   Debug.Assert(channel != string.Empty );
   Debug.Assert(value != null);
   #endregion

   try
   {
      WebProxy webProxy = null;

      if (webProxyAddress.Length &gt; 1)
      {
         webProxy = new WebProxy(webProxyAddress, webProxyPort);
      }

      using (HttpWebRequest request = (HttpWebRequest)WebRequest.Create(xivelyApiBaseUrl + xivelyApiFeedID + xivelyApiEndpoint))
      {
         byte[] buffer = Encoding.UTF8.GetBytes(channel + "," + value);

         DateTime httpRequestedStartedAtUtc = DateTime.UtcNow;

         if (webProxy != null)
         {
            request.Proxy = webProxy;
         }
         request.Method = "PUT";
         request.ContentLength = buffer.Length;
         request.ContentType = xivelyApiContentType;
         request.Headers.Add("X-ApiKey", xivelyApiKey);
         request.KeepAlive = false;
         request.Timeout = httpRequestTimeoutmSec;
         request.ReadWriteTimeout = httpRequestReadWriteTimeoutmSec;

         // request body
         Debug.Print("HTTP request");
         using (Stream stream = request.GetRequestStream())
         {
            stream.Write(buffer, 0, buffer.Length);
         }

         using (var response = (HttpWebResponse)request.GetResponse())
         {
            Debug.Print(" Status: " + response.StatusCode + " : " + response.StatusDescription);
         }

         TimeSpan duration = DateTime.UtcNow - httpRequestedStartedAtUtc;
         Debug.Print(" Duration: " + duration.ToString());
      }
   }
   catch (Exception ex)
   {
      Debug.Print(ex.Message);
   }
}

To use this code download the Nordic nRF24L01 library from Codeplex then include that plus my Netduino NRF24L01 Xively Gateway in a new solution and it should just work.

Deploy the application to a Netduino 2 Plus or Netduino 3 Wifi device and run it to create the app.config file, then use a text editor to update the file with your Xively & device settings.

I’ll upload this and a couple of other projects to GitHub shortly.

Bill of materials (prices as at July 2015)

Netduino 3 Wifi xively nRF24L01 Gateway data stream live

The gateway is now live, I’m regularly updating the Netduino 3 wifi code and the client arduino, devDuino + netduino devices so there maybe short periods of downtime and/or missing data points.

The stream is available here and is currently just temperature and humidity readings from two bedrooms updating roughly once a minute.

I live in New Zealand which is currently UTC + 12.