Ubidots with MQTTnet

As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.

This PoC was to confirm that I could connect to the ubidots MQTT API then format the topics and payloads correctly. The ubidots screen designer has “variables” (both actual sensors & synthetic calculated ones) which present as topics so I built a client which could subscribe to these.

.Net Core V2 MQTTnet client

The MQTT broker, username, password, and client ID are command line options.

class Program
{
	private static IMqttClient mqttClient = null;
	private static IMqttClientOptions mqttOptions = null;
	private static string server;
	private static string username;
	private static string deviceLabel;

	static void Main(string[] args)
	{
		MqttFactory factory = new MqttFactory();
		mqttClient = factory.CreateMqttClient();
		bool heatPumpOn = false;

		if (args.Length != 3)
		{
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID]");
			Console.WriteLine("Press <enter> to exit");
			Console.ReadLine();
			return;
		}

		server = args[0];
		username = args[1];
		deviceLabel = args[2];

		Console.WriteLine($"MQTT Server:{server} Username:{username} DeviceLabel:{deviceLabel}");

		mqttOptions = new MqttClientOptionsBuilder()
			.WithTcpServer(server)
			.WithCredentials(username, "NotVerySecret")
			.WithClientId(deviceLabel)
			.WithTls()
			.Build();

		mqttClient.ApplicationMessageReceived += MqttClient_ApplicationMessageReceived;
		mqttClient.Disconnected += MqttClient_Disconnected;
		mqttClient.ConnectAsync(mqttOptions).Wait();

		// Setup a subscription for commands sent to client
		string commandTopic = $"/v1.6/devices/{deviceLabel}/officetemperaturedesired/lv";
		mqttClient.SubscribeAsync(commandTopic).GetAwaiter().GetResult();

		//// Ubidots formatted client state update topic
		string stateTopic = $"/v1.6/devices/{deviceLabel}";

		while (true)
		{
			string payloadText;
			double temperature = 22.0 + (DateTime.UtcNow.Millisecond / 1000.0);
			double humidity = 50 + (DateTime.UtcNow.Millisecond / 100.0);
			double speed = 10 + (DateTime.UtcNow.Millisecond / 100.0);
			Console.WriteLine($"Topic:{stateTopic} Temperature:{temperature:0.00} Humidity:{humidity:0} HeatPumpOn:{heatPumpOn}");

			// First attempt which worked
			//payloadText = @"{""OfficeTemperature"":22.5}";

			// Second attempt to work out data format with "real" values injected
			//payloadText = @"{ ""officetemperature"":"+ temperature.ToString("F2") + @",""officehumidity"":" + humidity.ToString("F0") + @"}";

			// Third attempt with Jobject which sort of worked but number serialisation was sub optimal
			JObject payloadJObject = new JObject(); 
			payloadJObject.Add("OfficeTemperature", temperature.ToString("F2"));
			payloadJObject.Add("OfficeHumidity", humidity.ToString("F0"));

			if (heatPumpOn)
			{
				payloadJObject.Add("HeatPumpOn", 1);
			}
			else
			{
				payloadJObject.Add("HeatPumpOn", 0);
			}
			heatPumpOn = !heatPumpOn;
			payloadText = JsonConvert.SerializeObject(payloadJObject);

			/*
			// Forth attempt with JOBject, timestamps and gps 
			JObject payloadJObject = new JObject();
			JObject context = new JObject();
			context.Add("lat", "-43.5309325");
			context.Add("lng", "172.637119");// Christchurch Cathederal
			//context.Add("timestamp", ((DateTimeOffset)(DateTime.UtcNow)).ToUnixTimeSeconds()); // This field is optional and can be commented out
			JObject position = new JObject();
			position.Add("context", context);
			position.Add("value", "0");
			payloadJObject.Add("postion", position);
			payloadText = JsonConvert.SerializeObject(payloadJObject);
			*/

			var message = new MqttApplicationMessageBuilder()
				.WithTopic(stateTopic)
				.WithPayload(payloadText)
				.WithQualityOfServiceLevel(global::MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
			//.WithExactlyOnceQoS()// With ubidots this caused the publish to hang
			.WithAtLeastOnceQoS()
			.WithRetainFlag() 
			.Build();

			Console.WriteLine("PublishAsync start");
			mqttClient.PublishAsync(message).Wait();
			Console.WriteLine("PublishAsync finish");

			Thread.Sleep(30100);
		}
	}

	private static void MqttClient_ApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e)
	{
		Console.WriteLine($"ClientId:{e.ClientId} Topic:{e.ApplicationMessage.Topic} Payload:{e.ApplicationMessage.ConvertPayloadToString()}");
	}

	private static async void MqttClient_Disconnected(object sender, MqttClientDisconnectedEventArgs e)
	{
		Debug.WriteLine("Disconnected");
		await Task.Delay(TimeSpan.FromSeconds(5));

		try
		{
			await mqttClient.ConnectAsync(mqttOptions);
		}
		catch (Exception ex)
		{
			Debug.WriteLine("Reconnect failed {0}", ex.Message);
		}
	}
}

For this PoC I used the MQTTnet package which is available via NuGet. It appeared to be reasonably well supported and has had recent updates.

Variable configuration with device location map

Overall the initial configuration went smoothly, I found the dragging of blocks onto the dashboard and configuring them worked as expected.

The configuration of a “synthetic” variable (converting a temperature to Fahrenheit for readers from the Unites States of America, Myanmar & Liberia ) took a couple of goes to get right.

I may have missed something (April 2019) but the lack of boolean datatype variables was a bit odd.

Synthetic (calculated) variable configuration

I put a slider control on my test dashboard, associated it with a variable and my client reliably received messages when the slider was moved.

Dashboard with slider for desired temperature

Overall the Ubidots experience was pretty good and I’m going to spend some more time working with the device, data, users configurations to see how well it works for a “real-world” project.

I found (April 2019) that to get MQTTS going I had to install a Ubidots provided certificate

MQTT with TLS guidance and certificate download link

When my .Net Core application didn’t work I tried one my MQTT debugging tools and they didn’t work either with the Ubitdots MQTT brokers. The Ubidots forum people were quite helpful, but making it not necessary to install a certificate or making it really obvious in the documentation that this was required would be a good thing.

Losant IoT with MQTTnet

As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.

This PoC was to confirm that I could connect to the Losant MQTT API then format the topics and payloads correctly. The Losant screen designer has “Blocks” which generate commands for devices so I extended the test client to see how well this worked.

The MQTT broker, username, password, and client ID are command line options.

class Program
{
	private static IMqttClient mqttClient = null;
	private static IMqttClientOptions mqttOptions = null;
	private static string server;
	private static string username;
	private static string password;
	private static string clientId;

	static void Main(string[] args)
	{
		MqttFactory factory = new MqttFactory();
		mqttClient = factory.CreateMqttClient();
		bool heatPumpOn = false;

		if (args.Length != 4)
		{
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID]");
			Console.WriteLine("Press <enter> to exit");
			Console.ReadLine();
		}

		server = args[0];
		username = args[1];
		password = args[2];
		clientId = args[3];

		Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId}");

		mqttOptions = new MqttClientOptionsBuilder()
			.WithTcpServer(server)
			.WithCredentials(username, password)
			.WithClientId(clientId)
			.WithTls()
			.Build();

		mqttClient.ApplicationMessageReceived += MqttClient_ApplicationMessageReceived;
		mqttClient.Disconnected += MqttClient_Disconnected;
		mqttClient.ConnectAsync(mqttOptions).Wait();

		// Setup a subscription for commands sent to client
		string commandTopic = $"losant/{clientId}/command";
		mqttClient.SubscribeAsync(commandTopic);

		// Losant formatted client state update topic
		string stateTopic = $"losant/{clientId}/state";

		while (true)
		{
			string payloadText;
			double temperature = 22.0 + +(DateTime.UtcNow.Millisecond / 1000.0);
			double humidity = 50 + +(DateTime.UtcNow.Millisecond / 1000.0);
			Console.WriteLine($"Topic:{stateTopic} Temperature:{temperature} Humidity:{humidity} HeatPumpOn:{heatPumpOn}");

			// First attempt which worked
			//payloadText = @"{""data"":{ ""OfficeTemperature"":22.5}}";

			// Second attempt to work out data format with "real" values injected
			payloadText = @"{""data"":{ ""OfficeTemperature"":"+ temperature.ToString("f1") + @",""OfficeHumidity"":" + humidity.ToString("F0") + @"}}";

			// Third attempt with Jobject which sort of worked but number serialisation is sub optimal
			//JObject payloadJObject = new JObject(); 
			//payloadJObject.Add("time", DateTime.UtcNow.ToString("u")); // This field is optional and can be commented out

			//JObject data = new JObject();
			//data.Add("OfficeTemperature", temperature.ToString("F1"));
			//data.Add("OfficeHumidity", humidity.ToString("F0"));

			//data.Add("HeatPumpOn", heatPumpOn);
			//heatPumpOn = !heatPumpOn;
			//payloadJObject.Add( "data", data);

			//payloadText = JsonConvert.SerializeObject(payloadJObject);

			// Forth attempt with JOBject and gps info https://docs.losant.com/devices/state/
			//JObject payloadJObject = new JObject(); 
			//payloadJObject.Add("time", DateTime.UtcNow.ToString("u")); // This field is optional and can be commented out
			//JObject data = new JObject();
			//data.Add("GPS", "-43.5309325, 172.637119"); // Christchurch Cathederal
			//payloadJObject.Add("data", data);
			//payloadText = JsonConvert.SerializeObject(payloadJObject);

			var message = new MqttApplicationMessageBuilder()
				.WithTopic(stateTopic)
				.WithPayload(payloadText)
				.WithQualityOfServiceLevel(global::MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
				//.WithExactlyOnceQoS() With Losant this caused the publish to hang
				.WithAtLeastOnceQoS()
				//.WithRetainFlag() Losant doesn't allow this flag
				.Build();

			Console.WriteLine("PublishAsync start");
				mqttClient.PublishAsync(message).Wait();
			Console.WriteLine("PublishAsync finish");

			Thread.Sleep(30100);
		}
	}

	private static void MqttClient_ApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e)
	{
		Console.WriteLine($"ClientId:{e.ClientId} Topic:{e.ApplicationMessage.Topic} Payload:{e.ApplicationMessage.ConvertPayloadToString()}");
	}

	private static async void MqttClient_Disconnected(object sender, MqttClientDisconnectedEventArgs e)
	{
		Debug.WriteLine("Disconnected");
		await Task.Delay(TimeSpan.FromSeconds(5));

		try
		{
			await mqttClient.ConnectAsync(mqttOptions);
		}
		catch (Exception ex)
		{
			Debug.WriteLine("Reconnect failed {0}", ex.Message);
		}
	}
}

For this PoC I used the MQTTnet package which is available via NuGet. It appeared to be reasonably well supported and has had recent updates.

Overall the initial configuration went really smoothly, I found the dragging of blocks onto the dashboard and configuring them worked well.

Losant device configuration screen with trace logging

Losant .Net Core V2 client uploading simulated sensor readings

The device log made bringing up a new device easy and the error messages displayed when I had badly formatted payloads were helpful (unlike many other packages I have used).

I put a button block on the overview screen, associated it with a command publication and my client reliably received messages when the button was pressed

Losant .Net Core V2 client processing command

Overall the Losant experience was pretty good and I’m going to spend some more time working with the application designer, devices recipes, webhooks, integrations and workflows etc. to see how well it works for a “real-world” project.

myDevices Cayenne with MQTTnet

As I’m testing my Message Queue Telemetry Transport(MQTT) LoRa gateway I’m building a proof of concept(PoC) .Net core console application for each IoT platform I would like to support.

This PoC was to confirm that I could connect to the myDevices Cayenne MQTT API and format the topics and payloads correctly. The myDevices team have built many platform specific libraries that wrap the MQTT platform APIs to make integration for first timers easier (which is great). Though, as an experienced Bring Your Own Device(BYOD) client developer, I did find myself looking at the C/C++ code to figure out how to implement parts of my .Net test client.

The myDevices screen designer had “widgets” which generated commands for devices so I extended the test client implementation to see this worked.

The MQTT broker, username, password, client ID, channel number and optional subscription channel number are command line options.

class Program
{
	private static IMqttClient mqttClient = null;
	private static IMqttClientOptions mqttOptions = null;
	private static string server;
	private static string username;
	private static string password;
	private static string clientId;
	private static string channelData;
	private static string channelSubscribe;

	static void Main(string[] args)
	{
		MqttFactory factory = new MqttFactory();
		mqttClient = factory.CreateMqttClient();

		if ((args.Length != 5) && (args.Length != 6))
		{
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID] [Channel]");
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID] [ChannelData] [ChannelSubscribe]");
			Console.WriteLine("Press <enter> to exit");
			Console.ReadLine();
			return;
		}

		server = args[0];
		username = args[1];
		password = args[2];
		clientId = args[3];
		channelData = args[4];

		if (args.Length == 5)
		{
			Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId} ChannelData:{channelData}");
		}

		if (args.Length == 6)
		{
			channelSubscribe = args[5];
			Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId} ChannelData:{channelData} ChannelSubscribe:{channelSubscribe}");
		}

		mqttOptions = new MqttClientOptionsBuilder()
			.WithTcpServer(server)
			.WithCredentials(username, password)
			.WithClientId(clientId)
			.WithTls()
			.Build();

		mqttClient.ConnectAsync(mqttOptions).Wait();

		if (args.Length == 6)
		{
			string topic = $"v1/{username}/things/{clientId}/cmd/{channelSubscribe}";

			Console.WriteLine($"Subscribe Topic:{topic}");
			mqttClient.SubscribeAsync(topic).Wait();
			// mqttClient.SubscribeAsync(topic, global::MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce).Wait(); 
			// Thought this might help with subscription but it didn't, looks like ACK might be broken in MQTTnet
			mqttClient.ApplicationMessageReceived += MqttClient_ApplicationMessageReceived;
		}
		mqttClient.Disconnected += MqttClient_Disconnected;

		string topicTemperatureData = $"v1/{username}/things/{clientId}/data/{channelData}";

		Console.WriteLine();

		while (true)
		{
			string value = "22." + DateTime.UtcNow.Millisecond.ToString();
			Console.WriteLine($"Publish Topic {topicTemperatureData}  Value {value}");

			var message = new MqttApplicationMessageBuilder()
				.WithTopic(topicTemperatureData)
				.WithPayload(value)
				.WithQualityOfServiceLevel(global::MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
				//.WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.ExactlyOnce) // Causes publish to hang
				.WithRetainFlag()
				.Build();

			Console.WriteLine("PublishAsync start");

			mqttClient.PublishAsync(message).Wait();
			Console.WriteLine("PublishAsync finish");
			Console.WriteLine();

			Thread.Sleep(30100);
		}
	}

	private static void MqttClient_ApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e)
	{
		Console.WriteLine($"ApplicationMessageReceived ClientId:{e.ClientId} Topic:{e.ApplicationMessage.Topic} Qos:{e.ApplicationMessage.QualityOfServiceLevel} Payload:{e.ApplicationMessage.ConvertPayloadToString()}");
		Console.WriteLine();
	}

	private static async void MqttClient_Disconnected(object sender, MqttClientDisconnectedEventArgs e)
	{
		Debug.WriteLine("Disconnected");
		await Task.Delay(TimeSpan.FromSeconds(5));

		try
		{
			await mqttClient.ConnectAsync(mqttOptions);
		}
		catch (Exception ex)
		{
			Debug.WriteLine("Reconnect failed {0}", ex.Message);
		}
	}
}

For this PoC I used the MQTTnet package which is available via NuGet. It appeared to be reasonably well supported and has had recent updates. There did appear to be some issues with myDevices Cayenne default quality of service (QoS) and the default QoS used by MQTTnet connections and also the acknowledgement of the receipt of published messages.

myDevices Cayenne .Net Core 2 client
Cayenne UI with graph, button and value widgets

Overall the initial configuration went ok, I found the dragging of widgets onto the overview screen had some issues (maybe the caching of control settings (I found my self refreshing the whole page every so often) and I couldn’t save a custom widget icon at all.

I put a button widget on the overview screen and associated it with a channel publication. The client received a message when the button was pressed

myDevices .Net Core 2 client displaying a received command message

But the button widget was disabled until the overview screen was manually refreshed.

Cayenne UI after button press

The issue with the subscription maybe an issue with the MQTTnet library so I will build another client with the Eclipse Paho project .net client.

Overall the myDevices Cayenne experience (April 2018) was a bit flaky with basic functionality like the saving of custom widget icons broken, updates of the real-time data viewer didn’t occur or were delayed, and there were other configuration screen update issues.

Adafruit MQTT with MQTTnet

Before building the Message Queue Telemetry Transport(MQTT) gateway I built a proof of concept(PoC) .Net core console application. This was to confirm that I could connect to the Adafruit.IO MQTT broker and format the topic (with and without group name) and payload correctly. The Adafruit IO MQTT documentation suggests an approach for naming topics which allows a bit more structure for feed names than the REST API.

The MQTT broker, username, API key, client ID, optional group name (to keep MQTT aligned with REST API terminology) and feed name are command line options.

class Program
{
	private static IMqttClient mqttClient = null;
	private static IMqttClientOptions mqttOptions = null;
	private static string server;
	private static string username;
	private static string password;
	private static string clientId;
	private static string groupname;
	private static string feedname;

	static void Main(string[] args)
	{
		MqttFactory factory = new MqttFactory();
		mqttClient = factory.CreateMqttClient();

		if ((args.Length != 5) && (args.Length != 6))
		{
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID] [GroupName] [FeedName]");
			Console.WriteLine("[MQTT Server] [UserName] [Password] [ClientID] [FeedName]");
			Console.WriteLine("Press <enter> to exit");
			Console.ReadLine();
			return;
		}

		server = args[0];
		username = args[1];
		password = args[2];
		clientId = args[3];
		if (args.Length == 5)
		{
			feedname = args[4].ToLower();
			Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId} Feedname:{feedname}");
		}

		if (args.Length == 6)
		{
			groupname = args[4].ToLower();
			feedname = args[5].ToLower();
			Console.WriteLine($"MQTT Server:{server} Username:{username} ClientID:{clientId} Groupname:{groupname} Feedname:{feedname}");
		}

		mqttOptions = new MqttClientOptionsBuilder()
			.WithTcpServer(server)
			.WithCredentials(username, password)
			.WithClientId(clientId)
			.WithTls()
			.Build();

		mqttClient.Disconnected += MqttClient_Disconnected;
		mqttClient.ConnectAsync(mqttOptions).Wait();

		// Adafruit.IO format for topics which are called feeds
		string topic = string.Empty;

		if (args.Length == 5)
		{
			topic = $"{args[1]}/feeds/{feedname}";
		}

		if (args.Length == 6)
		{
			topic = $"{args[1]}/feeds/{groupname}.{feedname}";
		}

		while (true)
		{
			string value = "22." + DateTime.UtcNow.Millisecond.ToString();
			Console.WriteLine($"Topic:{topic} Value:{value}");

			var message = new MqttApplicationMessageBuilder()
				.WithTopic(topic)
				.WithPayload(value)
				.WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
				.WithExactlyOnceQoS()
				.WithRetainFlag()
				.Build();

			Console.WriteLine("PublishAsync start");
			mqttClient.PublishAsync(message).Wait();
			Console.WriteLine("PublishAsync finish");

			Thread.Sleep(30100);
		}
	}

	private static async void MqttClient_Disconnected(object sender, MqttClientDisconnectedEventArgs e)
	{
		Debug.WriteLine("Disconnected");
		await Task.Delay(TimeSpan.FromSeconds(5));

		try
		{
			await mqttClient.ConnectAsync(mqttOptions);
		}
		catch (Exception ex)
		{
			Debug.WriteLine("Reconnect failed {0}", ex.Message);
		}
	}
}

For this PoC I used the MQTTnet package which is available via NuGet. It appeared to be reasonably well supported and has had recent updates.

Overall the process went pretty well, I found that looking at the topic names in the Adafruit IO feed setup screens helped a lot. A couple of times I was tripped up by mixed case in my text fields.

.Net Core 2 client with group name
Adafruit IO feed setup with group name
Console client without group name
Adafruit IO feed setup without group name

I am also going to try building some clients with the Eclipse Paho project .net client so I can compare a couple of different libraries.

ASP MVC Core V2.1 and Cross-Origin Resource Sharing

I’m working on an project for a customer which implements a number of application programming Interfaces(API) for a Single Page Application(SPA) and other clients. We are using entity tags (ETags) for versioning and the front end developers found the couldn’t access them from javascipt running in mainstream browser clients (June 2018).

The problems was understanding how Cross-Origin Resource Sharing (CORS) worked and how it interacted with our security model (API key and OAuth2.0 depending on the client)

In our scenario we first found the pre-flight check wasn’t working because in the HyperText Transfer Protocol (HTTP) OPTIONS method our X-API-KEY check was failing

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key
Access-Control-Request-Method: GET
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Server: Kestrel
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:48:30 GMT

13
API Key is invalid.
0

So I disabled X-API-KEY validation in startup.cs

public async Task Invoke(HttpContext context)
{
   if (context.Request.Method == "OPTIONS")
   {
      await this.next.Invoke(context);
      return;
   }

   var claims = new List();
…

OPTIONS then worked

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key
Access-Control-Request-Method: GET
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 404 Not Found
Server: Kestrel
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:52:20 GMT
Content-Length: 0

I then turned on CORS allowing pretty much anything

public void ConfigureServices(IServiceCollection services)
{
   services.AddCors(options =>
   {
      options.AddPolicy("CorsPolicy",
      builder => builder.AllowAnyOrigin()
         .AllowAnyMethod()
         .AllowAnyHeader()
         .AllowCredentials());
   });
   services.AddMvc();
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
   if (env.IsDevelopment())
   {
      app.UseDeveloperExceptionPage();
   }

   TelemetryConfiguration.Active.InstrumentationKey = this.configuration.GetSection("ApplicationInsights").GetSection("InstrumentationKey").Value;

   loggerFactory.AddLog4Net();
   this.log.Info("Startup.Configure called");

   app.ApplyUserKeyValidation();
   app.UseCors("CorsPolicy");
   app.UseMvc();
   }
}

OPTIONS then worked

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key
Access-Control-Request-Method: GET
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 204 No Content
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: x-api-key
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:57:33 GMT

GET then worked

GET http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
If-None-Match: 00-00-00-00-00-00-00-76
Host: xyz.azurewebsites.net
Connection: Keep-Alive

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
ETag: 00-00-00-00-00-00-00-76
Vary: Origin,Accept-Encoding
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:57:34 GMT
Content-Length: 2216

[{"...."}}

But HEAD didn’t work

HEAD http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
If-None-Match: 00-00-00-00-00-00-00-76
Host: xyz.azurewebsites.net
Connection: Keep-Alive

HTTP/1.1 400 Bad Request
Content-Length: 0
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 05:59:55 GMT

From the Application Insights logging and RestTest client (which I ran locally and remotely) I could see that the client side code couldn’t access the value of our eTag.  It had to be “exposed”

public void ConfigureServices(IServiceCollection services)
{
   services.AddCors(options =>
   {
      options.AddPolicy("CorsPolicy",
            builder => builder.AllowAnyOrigin()
            .AllowAnyMethod()
            .AllowAnyHeader()
            .WithExposedHeaders("etag")
            .AllowCredentials()
         );
      });
      services.AddMvc();
   }
...

GET then worked

GET http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
ETag: 00-00-00-00-00-00-00-76
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
If-None-Match: 00-00-00-00-00-00-00-76
Host: xyz.azurewebsites.net
Connection: Keep-Alive

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
ETag: 00-00-00-00-00-00-00-76
Vary: Origin,Accept-Encoding
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 07:53:41 GMT

[{"...."}}

HEAD then worked

OPTIONS http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
Access-Control-Request-Headers: x-api-key,etag
Access-Control-Request-Method: HEAD
Accept-Encoding: gzip, deflate
Content-Length: 0
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 204 No Content
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: x-api-key,etag
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 07:57:31 GMT

HEAD http://xyz.azurewebsites.net/api/portfolio HTTP/1.1
...
X-API-KEY: ABCDEFGHIJKLMNOPQRSTUVWXYZ
ETag: 00-00-00-00-00-00-00-76
Accept-Language: en-NZ
Accept-Encoding: gzip, deflate
Host: xyz.azurewebsites.net
Connection: Keep-Alive
Pragma: no-cache

HTTP/1.1 304 Not Modified
Vary: Origin
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: file://
X-Powered-By: ASP.NET
...
Date: Sun, 24 Jun 2018 07:57:31 GMT

I had some oddness with releasing code updates which I think was down to caching of pre-flight request responses.
Next steps tidy up the headers etc. and lock the CORS configuration down to expose the minimum necessary required for the application to work.

.Net Core & WCF TransportWithMessageCredential

In one of my day jobs I look after a system which has been around since 2010 (Early adopter of Microsoft Azure, developement started on .Net 3.5). The product has a number of Windows Communication Foundation(WCF) services hosted in an Azure CloudService.

A client built with .Net Core wanted to be able to call one of the services which was implemented using wsHttpBinding and TransportWithMessageCredential and this proved a bit more painful than expected…

I first tried the Visual Studio 2017 Microsoft WCF Web Service Reference Provider fromt the WCF Core Team.

The “add connected service” extension dialog allowed me to select an endpoint

ConfigureWCFWebSeriveReference

But the code generation process failed

WCFWebServiceReferenceError.png

The error message wasn’t particularly helpful so I used the command line utility svcutil to generate client classes. Which I used to built a .net core client with and the associated .Net Core WCF NuGet packages.

The console application failed when I called the service with a “PlatformNotSupportedException”. After some searching I found that the .Net Core WCF libraries don’t support TransportWithMessageCredential (September 2017).

Some more searching lead to a StackOverflow article where an answer suggested using the SimpleSOAPClient NuGet package. I then created a new client using the generated classes as the basis for the ones used in my SimpleSOAPClient proof of concept(PoC)

[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.ServiceModel", "4.0.0.0")]
[System.ServiceModel.MessageContractAttribute(WrapperName="Redeem", WrapperNamespace="http://qwertyuiop.com/services2011/08", IsWrapped=true)]
public partial class RedeemRequest
{
    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=1)]
    public string voucherCode;

    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=2)]
    public string merchantId;

    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=3)]
    public string merchantReference;

    [System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://qwertyuiop.com/services2011/08", Order=4)]
    public string terminalId;

    public RedeemRequest()
    {
    }

    public RedeemRequest(string voucherCode, string merchantId, string merchantReference, string terminalId)
    {
        this.voucherCode = voucherCode;
        this.merchantId = merchantId;
        this.merchantReference = merchantReference;
        this.terminalId = terminalId;
    }
}

became

[XmlRoot("Redeem", Namespace = "http://qwertyuiop.com/services2011/08")]
public partial class RedeemRequest
{
   [XmlElement("voucherCode")]
   public string voucherCode;
   [XmlElement("transactionAmount")]
   public decimal transactionAmount;
   [XmlElement("merchantId")]
   public string merchantId;
   [XmlElement("merchantReference")]
   public string merchantReference;
   [XmlElement("terminalId")]
   public string terminalId;
}

This client failed with a SOAPAction related exception so I fired up Telerik Fiddler and found that the header was missing. When I manually added the header in the request composer (after dragging one of my failed requests onto the composer tab) it worked.

I had a look at the code in the SimpleSOAPClient repository to see how to add a custom HTTP Header to a request.

RedeemRequest redeemRequest = new RedeemRequest()
{
   merchantId = "......",
   merchantReference = "......",
   terminalId = "......",
   voucherCode = "......",
};

using (var client = SoapClient.Prepare())
{
   client.HttpClient.DefaultRequestHeaders.Add("SOAPAction", "http://qwertyuiop.com/services2011/08/IRedemptionProxyServiceV1/Redeem");
   var responseEnvelope = await client.SendAsync(
      "https://qwertyuiop.com/RedemptionProxy.svc",
      "https://qwertyuiop.com/services2011/08/IRedemptionProxyServiceV1/Redeem",
      SoapEnvelope.Prepare()
      .WithHeaders(KnownHeader.Oasis.Security.UsernameTokenAndPasswordText(".....", "......"))
      .Body(redeemRequest), ct);

      var response = responseEnvelope.Body<RedeemResponse>();

      Console.WriteLine("Redeem Result:{0}  Message:{1}", response.Result, response.messageText);
   }
}

After sorting out a few typos my request worked as expected. Only a couple of hours lost from my life, hopefully this post will help someone else.