TTN V3 Connector Revisited

Earlier in the year I built Things Network(TTN) V2 and V3 connectors and after using these in production applications I have learnt a lot about what I had got wrong, less wrong and what I had got right.

Using a TTN V3 MQTT Application integration wasn’t a great idea. The management of state was very complex. The storage of application keys in a app.settings file made configuration easy but was bad for security.

The use of Azure Key Vault in the TTNV2 connector was a good approach, but the process of creation and updating of the settings needs to be easier.

Using TTN device registry as the “single source of truth” was a good decision as managing the amount of LoRaWAN network, application and device specific configuration in an Azure IoT Hub would be non-trivial.

Using a Webhooks Application Integration like the TTNV2 connector is my preferred approach.

The TTNV2 Connector’s use of Azure Storage Queues was a good idea as they it provide an elastic buffer between the different parts of the application.

The use of Azure Functions to securely ingest webhook calls and write them to Azure Storage Queues with output bindingts should simplify configuration and deployment. The use of Azure Storage Queue input bindings to process messages is the preferred approach.

The TTN V3 processing of JSON uplink messages into a structure that Azure IoT Central could ingest is a required feature

The TTN V2 and V3 support for the Azure Device Provisioning Service(DPS) is a required feature (mandated by Azure IoT Central). The TTN V3 connector support for DTDLV2 is a desirable feature. The DPS implementation worked with Azure IoT Central but I was unable to get the DeviceClient based version working.

Using DPS to pre-provision devices in Azure IoT Hubs and Azure IoT Central by using the TTN Application Registry API then enumerating the TTN applications, then devices needs to be revisited as it was initially slow then became quite complex.

The support for Azure IoT Hub connection strings was a useful feature, but added some complexity. This plus basic Azure IoT Hub DPS support(No Azure IoT Central support) could be implemented in a standalone application which connects via Azure Storage Queue messages.

The processing of Azure IoT Central Basic, and Request commands then translating the payloads so they work with TTN V3 is a required feature. The management of Azure IoT Hub command delivery confirmations (abandon, complete and Reject) is a required feature.

I’m considering building a new TTN V3 connector but is it worth the effort as TTN has one now?

TTI V3 Gateway provisioning Dragino LHT65 Uplink

This very long post is about how to connect a Dragino LHT65 Temperature and Humidity sensor to Azure IoT Central using my TTI/TTN V3Azure IoT Connector and the Digital Twin Definition Language (DTDL).

Dragino LHT65 temperature and Humidity sensor

The first step was to add an application(dragino-lht65) in my The Things Industries(TTI) tenant

TTI/TTN application for my Dragino LHT65 devices
Adding devMobile as a collaborator on the new application
TTI Application API Key configuration

The new Application API Key used by the MQTTnet managed client only needs to have write downlink and read uplink traffic enabled.

FTDI Adapter and modified LHT64 cable

So I could reliably connect to my LHT65 devices to configure them I modified a programming cable so I could use it with a spare FTDI adaptor without jumper wires. Todo this I used a small jewelers screwdriver to “pop” out the VCC cable and move the transmit data line.

After entering the device password and checking the firmware version I used the AT+CFG command to display the device settings

AT+CFG: Print all configurations

[334428]***** UpLinkCounter= 0 *****
[334430]TX on freq 923200000 Hz at DR 2
[334804]txDone
[339807]RX on freq 923200000 Hz at DR 2
[339868]rxTimeOut
[340807]RX on freq 923200000 Hz at DR 2
[340868]rxTimeOut

Correct Password

Stop Tx events,Please wait for all configurations to print
Printf all config...
AT+DEUI=a8 .. .. .. .. .. .. d6
AT+DADDR=01......D6

AT+APPKEY=9d .. .. .. .. .. .. .. .. .. .. .. .. .. .. 2e
AT+NWKSKEY=f6 .. .. .. .. .. .. .. .. .. .. .. .. .. .. 69
AT+APPSKEY=4c 35 .. .. .. .. .. .. .. .. .. .. .. .. .. 3d
AT+APPEUI=a0 .. .. .. .. .. .. 00
AT+ADR=1
AT+TXP=0
AT+DR=0
AT+DCS=0
AT+PNM=1
AT+RX2FQ=923200000
AT+RX2DR=2
AT+RX1DL=1000
AT+RX2DL=2000
AT+JN1DL=5000
AT+JN2DL=6000
AT+NJM=1
AT+NWKID=00 00 00 00
AT+FCU=0
AT+FCD=0
AT+CLASS=A
AT+NJS=0
AT+RECVB=0:
AT+RECV=0:
AT+VER=v1.7 AS923

AT+CFM=0
AT+CFS=0
AT+SNR=0
AT+RSSI=0
AT+TDC=1200000
AT+PORT=2
AT+PWORD=123456
AT+CHS=0
AT+DATE=21/3/26 07:49:15
AT+SLEEP=0
AT+EXT=4,2
AT+RTP=20
AT+BAT=3120
AT+WMOD=0
AT+ARTEMP=-40,125
AT+CITEMP=1
Start Tx events OK


[399287]***** UpLinkCounter= 0 *****

[399289]TX on freq 923400000 Hz at DR 2

[399663]txDone

[404666]RX on freq 923400000 Hz at DR 2

[404726]rxTimeOut

[405666]RX on freq 923200000 Hz at DR 2

[405726]rxTimeOut

I copied the AppEUI and DevEUI for use on the TI Dragino LHT65 Register end device form provided by the TTI/TTN.

TTYI/TTN Dragino LHT65 Register end device

The Dragino LHT65 uses the DeviceEUI as the DeviceID which meant I had todo more redaction in my TTI/TTN and Azure Application Insights screen captures. The rules around the re-use of EndDevice ID were a pain in the arse(PITA) in my development focused tenant.

Dragino LHT 65 Device uplink payload formatter

The connector supports both uplink and downlink messages with JSON encoded payloads. The Dragino LHT65 has a vendor supplied formatter which is automatically configured when an EndDevice is created. The EndDevice formatter configuration can also be overridden at the Application level in the app.settings.json file.

Device Live Data Uplink Data Payload

Once an EndDevice is configured in TTI/TTN I usually use the “Live data Uplink Payload” to work out the decoded payload JSON property names and data types.

LHT65 Uplink only Azure IoT Central Device Template
LHT65 Device Template View Identity

For Azure IoT Central “automagic” provisioning the DTDLModelId has to be copied from the Azure IoT Central Template into the TTI/TTN EndDevice or app.settings.json file application configuration.

LHT65 Device Template copy DTDL @ID
TTI EndDevice configuring the DTDLV2 @ID at the device level

Configuring the DTDLV2 @ID at the TTI application level in the app.settings.json file

{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "Microsoft": "Debug",
      "Microsoft.Hosting.Lifetime": "Debug"
    },
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Debug"
      }
    }
  },

  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0ne...DD9",
            "GroupEnrollmentKey": "eFR...w=="
          }
        },
        "DTDLModelId": "dtmi:ttnv3connectorclient:FezduinoWisnodeV14x8;4",
        "MQTTAccessKey": "NNSXS.HCY...RYQ",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_0": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_1": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "TemperatureOOBAlertMinimumAndMaximum": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          }
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0ne...DD9",
            "GroupEnrollmentKey": "AtN...g=="
          },
        },
        "DTDLModelId": "dtmi:ttnv3connectorclient:SeeeduinoLoRaWAN4cz;1",
        "MQTTAccessKey": "NNSXS.V44...42A",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      },
      "dragino-lht65": {
        "AzureSettings": {
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0ne...DD9",
            "GroupEnrollmentKey": "SLB...w=="
          }
        },
        "DTDLModelId": "dtmi:ttnv3connectorclient:DraginoLHT656w6;1",
        "MQTTAccessKey": "NNSXS.RIJ...NZQ",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
      "MqttServerName": "eu1.cloud.thethings.industries",
      "MqttClientId": "MQTTClient",
      "MqttAutoReconnectDelay": "00:00:05",
      "Tenant": "...-test",
      "ApiBaseUrl": "https://...-test.eu1.cloud.thethings.industries/api/v3",
      "ApiKey": "NNSXS.NR7...ZSA",
      "Collaborator": "devmobile",
      "DevicePageSize": 10,
      "DeviceIntegrationDefault": true
    }
  }
}

The Azure Device Provisioning Service(DPS) is configured at the TTI application level in the app.settings.json file. The IDScope and one of the Primary or Secondary Shared Access Signature(SAS) keys should be copied into DeviceProvisioningServiceSettings of an Application in the app.settings.json file. I usually set the “Automatically connect devices in this group” flag as part of the “automagic” provisioning process.

Azure IoT Central Group Enrollment Key
Then device templates need to be mapped to an Enrollment Group then Device Group.

For testing the connector application can be run locally with diagnostic information displayed in the application console window as it “automagically’ provisions devices and uploads telemetry data.

Connector application Diagnostics
Azure IoT Central Device list before my LHT65 device is “automagically” provisioned
Azure IoT Central Device list after my LHT65 device is “automagically” provisioned

One a device has been provisioned I check on the raw data display that all the fields I configured have been mapped correctly.

Azure IoT Central raw data display

I then created a dashboard to display the telemetry data from the LHT65 sensors.

Azure IoT Central dashboard displaying LHT65 temperature, humidity and battery voltage graphs.

The dashboard also has a few KPI displays which highlighted an issue which occurs a couple of times a month with the LHT65 onboard temperature sensor values (327.7°). I have connected Dragino technical support and have also been unable to find a way to remove the current an/or filter out future aberrant values.

Azure Application Insights logging

I also noticed that the formatting of the DeviceEUI values in the Application Insights logging was incorrect after trying to search for one of my Seeedstudio LoRaWAN device with its DeviceEUI.

TTI V3 Gateway Azure IoT Central Digital Twin Definition Language(DTDL) support

Over the last couple of days I have added limited Digital Twin Definition Language(DTDLV2) support to my The Things Industries(TTI) V3 connector so that Azure IoT Central devices can be “zero touch” provisioned. For this blog post I used five Seeeduino LoRaWAN devices left over from another abandoned project.

The first step was to configure and Azure IoT Central enrollment group (ensure “Automatically connect devices in this group” is on) and copy the IDScope and Group Enrollment key to the appsettings.json file (see sample file below for more detail)

Azure IoT Central Enrollment Group configuration

Then I created an Azure IoT Central template for the seeeduino LoRAWAN devices which are running software (developed with the Arduino tooling) that read values from a Grove – Temperature&Humidity sensor. The naming of telemetry properties in specified by the Cayenne Low Power Protocol(LPP) encoder/decoder (I check the decoded payload in TTI EndDevice “Live Data” tab).

Configuring Seeeduino LoRaWAN device template

Then I mapped the Azure IoT Central Device Group to my Azure IoT Central Enrollment Group

Associating Device Group with Group Enrollment configuration

The Device Template @Id can be configured as the “default” template for all the devices in a TTI application in the app.settings.json file.

{
...
   "ProgramSettings": {
      "Applications": {
...
      "seeeduinolorawan": {
        "AzureSettings": {
           "DeviceProvisioningServiceSettings": {
              "IdScope": "...",
              "GroupEnrollmentKey": "..."
            }
         },
         "DTDLModelId": "dtmi:ttnv3connectorclient:SeeeduinoLoRaWAN4cz;1",
         "MQTTAccessKey": "...",
         "DeviceIntegrationDefault": true,
         "DevicePageSize": 10
      }
   }.
...

The Device Template @Id can also be set using a dtdlmodelid attribute in a TTI end device settings so devices can be individually configured.

TTI Application EndDevice dtdlmodelid attribute usage

At startup the TTI Gateway enumerates through the devices in each application configured in the app.settings.json. The Azure Device Provisioning Service(DPS) is used to retrieve each device’s connection string and configure it in Azure IoT Central if required.

Azure IoT Central Device Group with no provisioned Devices
TTI Connector application connecting and provisioning EndDevices
Azure IoT Central devices mapped to an Azure IoT Central Template via the modelID

The ProvisioningRegistrationAdditionalData optional parameter of the DPS RegisterAsync method has a JSON property which is used to the specify the device ModelID.

using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
{
	ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create( 
		Constants.AzureDpsGlobalDeviceEndpoint,
		deviceProvisiongServiceSettings.IdScope,
		securityProvider,
		transport);

	DeviceRegistrationResult result;

	if (!string.IsNullOrEmpty(modelId))
	{
		ProvisioningRegistrationAdditionalData provisioningRegistrationAdditionalData = new ProvisioningRegistrationAdditionalData()
		{
			JsonData = $"{{\"modelId\": \"{modelId}\"}}"
		};

		result = await provClient.RegisterAsync(provisioningRegistrationAdditionalData, stoppingToken);
	}
	else
    {
		result = await provClient.RegisterAsync(stoppingToken);
	}

	if (result.Status != ProvisioningRegistrationStatusType.Assigned)
	{
		_logger.LogError("Config-DeviceID:{0} Status:{1} RegisterAsync failed ", deviceId, result.Status);

		return false;
	}

	IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

	deviceClient = DeviceClient.Create(result.AssignedHub, authentication, transportSettings);
}

My implementation was “inspired” by TemperatureController project in the PnP Device Samples.

Azure IoT Central Dashboard with Seeeduino LoRaWAN devices around my house that were “automagically” provisioned

I need to do some testing to confirm my code works reliably with both DPS and user provided connection strings. The RegisterAsync call is currently taking about four seconds which could be an issue for TTI applications with many devices.

TTI V3 Gateway Azure IoT Hub Support

After a couple of weeks work my The Things Industries(TTI) V3 gateway is in beta testing. For this blog post I have configured five Seeeduino LoRaWAN devices. My sensor nodes connect to an Azure IoT Hub with a Shared Access Signature(SAS) device policy connection string. I’m using Device Twin Explorer to display Telemetry from and send messages to the sensor nodes. I have also configured Azure Stream Analytics and PowerBI to graph telemetry from the sensor nodes.

Device Twin Explorer displaying telemetry from one of the Seeeduino devices

My integration uses only queued messages as often they won’t be delivered to the sensor node immediately, especially if the sensor node only sends an uplink message every 30 minutes/hour/day.

The confirmed flag should be used with care as the Azure IoT Hub messages may expire before a delivery Ack/Nack/Failed is received from the TTI.

PowerBI graph of temperature and humidity in my garage over 24 hours

To send a downlink message, TTI needs a LoRaWAN port number (plus optional queue, confirmed and priority values) which is specified in the Azure IoT Hub message custom properties.

Device explorer displaying a raw payload message which has been confirmed delivered
TTI device live data tab displaying raw payload in downlink message information tab
Azure IoT Connector console application sending raw payload to sensor node with confirmation ack
Arduino monitor displaying received raw payload from TTI

If the Azure IoT Hub message payload is valid JSON it is copied into the payload decoded downlink message property. and if it is not valid JSON it assumed to be a Base64 encoded value and copied into the payload raw downlink message property.

try
{
	// Split over multiple lines in an attempt to improve readability. A valid JSON string should start/end with {/} for an object or [/] for an array
	if (!(payloadText.StartsWith("{") && payloadText.EndsWith("}"))
										&&
		(!(payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
	{
		throw new JsonReaderException();
	}

	downlink.PayloadDecoded = JToken.Parse(payloadText);
}
catch (JsonReaderException)
{
	downlink.PayloadRaw = payloadText;
}

Like the Azure IoT Central JSON validation I had to add a check that the string started with a “{” and finished with a “}” (a JSON object) or started with a “[” and finished with a “]” (a JSON array) as part of the validation process.

Device explorer displaying a JSON payload message which has been confirmed delivered

I normally wouldn’t use exceptions for flow control but I can’t see a better way of doing this.

TTI device live data tab displaying JSON payload in downlink message information tab
Azure IoT Connector console application sending JSON payload to sensor node with confirmation ack
Arduino monitor displaying received JSON payload from TTI

The build in TTI decoder only supports downlink decoded payloads with property names “value_0” through “value_x” custom encoders may support other property names.

TTI V3 Gateway Azure IoT Central Support

After a couple of weeks work my The Things Industries(TTI) V3 gateway is in beta testing. For this blog post the client is a GHI Electronics Fezduino with a RAK811 LPWAN Evaluation Board(EVB). My test device was configured in Azure IoT Central by the Device Provisioning Service(DPS) and I then manually migrated the device to each of the four templates used in this post.

The first step was to display the temperature and barometric pressure values from the Seeedstudio Grove BMP180 attached to my sensor node.

Sensor node displaying temperature and barometric pressure values
Azure IoT Central temperature and barometric pressure telemetry configuration
Azure IoT Central Telemetry Dashboard displaying temperature and barometric pressure values

The next step was to configure a simple Azure IoT Central command to send to the sensor node. This was a queued request with no payload. An example of this sort of command would be a request for a sensor node to reboot or turn on an actuator.

My integration uses only offline queued commands as often messages won’t be delivered to the sensor node immediately, especially if the sensor node only sends a message every half hour/hour/day. The confirmed flag should be used with care as the Azure IoT Hub messages may expire before a delivery Ack/Nack/Failed is received from the TTI and it consumes downlink bandwidth.

if (message.Properties.ContainsKey("method-name"))
{
}

I determine an Azure IoT Hub message is an Azure IoT Central command by the presence of the “method-name” property. If the Azure IoT Central command does not have a request payload the Azure IoT Hub message payload will contain a single “@” character so the Azure IoT Connector sends a TTI downlink message with an empty raw payload via the TTI Data API(MQTT).

if (payloadText.CompareTo("@") != 0)
{
   .
}
else
{
   downlink.PayloadRaw = "";
}
Azure IoT Central command with out a request payload value command configuration

To send a downlink message, TTI needs a LoRaWAN port number (plus optional queue, confirmed and priority values) which can’t be provided via the Azure IoT Central command setup so these values are configured in the app.settings file.

Each TTI application has zero or more Azure IoT Central command configurations which supply the port, confirmed, priority and queue settings.

  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          ...
          }
        },
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
        }
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
...
   }
}
Azure IoT Central simple command dashboard
Azure IoT Central simple command initiation
Azure IoT TTI connector application sending a simple command to my sensor node
Sensor node display simple command information. The note message payload is empty

The next step was to configure a more complex Azure IoT Central command to send to the sensor node. This was a queued request with a single value payload. An example of this sort of command could be setting the speed of a fan or the maximum temperature of a freezer for an out of band (OOB) notification to be sent.

Azure IoT Central single value command configuration
  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          ...
          }
        },
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_0": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_1": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
        }
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
...
   }
}

The value_0 settings are for the minimum temperature the value_1 settings are for the maximum temperature value.

Azure IoT Central single value command initiation
Azure IoT TTI connector application sending a single value command to my sensor node
Sensor node displaying single value command information. There are two downlink messages and each payload contains a single value

The single value command payload contains the textual representation of the value e.g. “true”/”false” or “1.23” which are also valid JSON. This initially caused issues as I was trying to splice a single value into the decoded payload.

I had to add a check that the string started with a “{” and finished with a “}” (a JSON object) or started with a “[” and finished with a “]” (a JSON array) as part of the validation process.

For a single value command the payload decoded has a single property with the method-name value as the name and the payload as the value. For a command with a JSON payload the message payload is copied into the PayloadDecoded.

I normally wouldn’t use exceptions for flow control but I can’t see a better way of doing this.

	try
	{
		// Split over multiple lines to improve readability
		if (!(payloadText.StartsWith("{") && payloadText.EndsWith("}"))
									&&
			(!(payloadText.StartsWith("[") && payloadText.EndsWith("]"))))
		{
			throw new JsonReaderException();
		}

		downlink.PayloadDecoded = JToken.Parse(payloadText);
	}
	catch (JsonReaderException)
	{
		try
		{
			JToken value = JToken.Parse(payloadText);

			downlink.PayloadDecoded = new JObject(new JProperty(methodName, value));
		}
		catch (JsonReaderException)
		{
			downlink.PayloadDecoded = new JObject(new JProperty(methodName, payloadText));
		}
	}

The final step was to configure an another Azure IoT Central command with a JSON payload to send to the sensor node. A “real-world” example of this sort of command would be setting the minimum and maximum temperatures of a freezer in a single downlink message.

Azure IoT Central JSON payload command setup
Azure IoT Central JSON payload command payload configuration
  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          ...
          }
        },
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": false,
        "MethodSettings": {
          "Reboot": {
            "Port": 21,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_0": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "value_1": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          },
          "TemperatureOOBAlertMinimumAndMaximum": {
            "Port": 30,
            "Confirmed": true,
            "Priority": "normal",
            "Queue": "push"
          }
        }
      },
      "seeeduinolorawan": {
        "AzureSettings": {
        }
        "MQTTAccessKey": "...",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },
    "TheThingsIndustries": {
...
   }
}
Azure IoT Central JSON payload command initiation

Azure IoT TTI connector application sending a JSON payload command to my sensor node
Sensor node displaying JSON command information. There is a single payload which contains a two values

The build in TTI decoder only supports downlink decoded payloads with property names “value_0” through “value_x” which results in some odd command names and JSON payload property names. (Custom encoders may support other property names). Case sensitivity of some configuration values also tripped me up.

TTN V3 Gateway Configuration, Deployment and Operation

After configuring, deploying and then operating my The Things Network(TTN) V2 gateway I have made some changes to my The Things Industries(TTI) V3 gateway.

TTI V3 Gateway running as a console application on my desktop

Azure IoT integration can be configured at the Device (TTN Device “azureintegration” attribute).

TTN Device AzureIntegration Attribute

Then falls back to the Application default (TTN application “azureintegrationdevicedefault” attribute).

TTN Application AzureIntegrationDeviceDefault attribute.

Then falls back to the “DeviceIntegrationDefault” setting for the Application then finally “DeviceIntegrationDefault” setting for the webjob the in the app.settings.json file

{
  ...
  "ProgramSettings": {
    "Applications": {
      "application1": {
        "AzureSettings": {
          "IoTHubConnectionString": "HostName=TT...n1.azure-devices.net;SharedAccessKeyName=device;SharedAccessKey=Am...M=",
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0n...3B",
            "GroupEnrollmentKey": "Kl...Y="
          }
        },
        "MQTTAccessKey": "NNSXS.HC...YQ",
        "DeviceIntegrationDefault": false,
        "DevicePageSize": 10
      },
      "seeeduinolorawan": {
        "AzureSettings": {
          "IoTHubConnectionString": "HostName=TT...n2.azure-devices.net;SharedAccessKeyName=device;SharedAccessKey=D2q...L8=",
          "DeviceProvisioningServiceSettings": {
            "IdScope": "0n...3B",
            "GroupEnrollmentKey": "Kl...Y="
          }
        },
        "MQTTAccessKey": "NNSXS.V44...42A",
        "DeviceIntegrationDefault": true,
        "DevicePageSize": 10
      }
    },

    "AzureSettingsDefault": {
      "IoTHubConnectionString": "HostName=TT...ors.azure-devices.net;SharedAccessKeyName=device;SharedAccessKey=yd...k=",
      "DeviceProvisioningServiceSettings": {
        "IdScope": "0n...3B",
        "GroupEnrollmentKey": "Kl...Y="
      }
    },

    "TheThingsIndustries": {
      "MqttServerName": "eu1.cloud.thethings.industries",
      "MqttClientId": "MQTTClient",
      "MqttAutoReconnectDelay": "00:00:05",
      "Tenant": "br...st",
      "ApiBaseUrl": "https://br..st.eu1.cloud.thethings.industries/api/v3",
      "ApiKey": "NNSXS.NR...SA",
      "Collaborator": "de...le",
      "DevicePageSize": 10,
      "DeviceIntegrationDefault": true
    }
  }
}

This approach is now used for most of the application settings to recue the amount of configuration required for a small scale deployment.

To reduce complexity the initial version of the V3 TTI gateway doesn’t support Azure IoT Central and the Device Provisioning Service(DPS).

Azure IoT Central Connectivity Part4

The Things Network(TTN) Friendly Commands

I have built a several Proof of Concept(PoC) applications (Azure IoT Central Basic Telemetry, Basic Commands, and Request Commands) to explore to how an Azure IoT Central integration with TTN could work. This blog post is about how to configure queued and non queued Cloud to Device(C2D) Commands with request parameters so they should work with my TTN Message Queue Telemetry Transport(MQTT) Data API connector.

I have focused on commands with Analog values but the same approach should be valid for other parameter types like Boolean, Date, DateTime, Double, Duration, Enumeration, Float, Geopoint, Vector, Integer, Long, String, and Time.

Multiple versions of my Azure IoT Central templates

There was a lot of “trial and error” (26 template versions) required to figure out how to configure commands and queued commands so they can and used in TTN downlink payloads.

{
  "end_device_ids": {
    "device_id": "dev1",
    "application_ids": {
      "application_id": "app1"
    },
    "dev_eui": "4200000000000000",
    "join_eui": "4200000000000000",
    "dev_addr": "00E6F42A"
  },
  "correlation_ids": [
    "my-correlation-id",
    "..."
  ],
  "downlink_ack": {
    "session_key_id": "AWnj0318qrtJ7kbudd8Vmw==",
    "f_port": 15,
    "f_cnt": 11,
    "frm_payload": "....",
    "decoded_payload": 
    {
      "Value_0":"1.23"
      ...
    }
    "confirmed": true,
    "priority": "NORMAL",
    "correlation_ids": [
      "my-correlation-id",
      "..."
    ]
  }
}

My Azure IoT Central client application displays the generated message message including the decoded_payload field which is used by the built in Cayenned Low Power Protocol(LPP) decoder/encoder and other custom encoders/decoders.

Azure IoT Central commands for TTN/TTI integration

From the “Device Commands” form I can send commands and a queued commands which have float parameters or object parameters which contain one or more float values in a JSON payload.

For commands which call the methodHander which was been registered by calling SetMethodDefaultHandlerAsync the request payload can be JSON or plain text. If the payload is valid JSON it is “grafted”(couldn’t think of a better word) into the decoded_payload field. If the payload is not valid a JSON object with the method name as the “name” and the text payload as the value is added the decoded_payload.

private static async Task<MethodResponse> MethodCallbackDefaultHandler(MethodRequest methodRequest, object userContext)
{
   AzureIoTMethodHandlerContext receiveMessageHandlerConext = (AzureIoTMethodHandlerContext)userContext;

   Console.WriteLine($"Default handler method {methodRequest.Name} was called.");

   Console.WriteLine($"Payload:{methodRequest.DataAsJson}");
   Console.WriteLine();

   if (string.IsNullOrWhiteSpace(methodRequest.Name))
   {
      Console.WriteLine($"   Method Request Name null or white space");
      return new MethodResponse(400);
   }

   string payloadText = Encoding.UTF8.GetString(methodRequest.Data);
   if (string.IsNullOrWhiteSpace(payloadText))
   {
       Console.WriteLine($"   Payload null or white space");
       return new MethodResponse(400);
   }

   // At this point would check to see if Azure DeviceClient is in cache, this is so nasty
   if ( String.Compare( methodRequest.Name, "Analog_Output_1", true) ==0 )
   {
      Console.WriteLine($"   Device not found");
      return new MethodResponse(UTF8Encoding.UTF8.GetBytes("Device not found"), 404);
   }

   JObject payload;

   if (IsValidJSON(payloadText))
   {
      payload = JObject.Parse(payloadText);
   }
   else
   {
      payload = new JObject
      {
         { methodRequest.Name, payloadText }
      };
   }

   string downlinktopic = $"v3/{receiveMessageHandlerConext.ApplicationId}@{receiveMessageHandlerConext.TenantId}/devices/{receiveMessageHandlerConext.DeviceId}/down/push";

   DownlinkPayload downlinkPayload = new DownlinkPayload()
   {
      Downlinks = new List<Downlink>()
      {
         new Downlink()
         {
            Confirmed = false,
            //PayloadRaw = messageBody,
            PayloadDecoded = payload,
            Priority = DownlinkPriority.Normal,
            Port = 10,
            /*
            CorrelationIds = new List<string>()
            {
               methodRequest.LockToken
            }
            */
         }
      }
   };

   Console.WriteLine($"TTN Topic :{downlinktopic}");
   Console.WriteLine($"TTN downlink JSON :{JsonConvert.SerializeObject(downlinkPayload, Formatting.Indented)}");

   return new MethodResponse(200);
}
Configuration of unqueued Commands with a typed payload
The output of my test harness for a Command for a typed payload
Configuring fields of object payload(JSON)

A JSON request payload also supports downlink messages with more that one value.

The output of my test harness for a Command with an object payload(JSON)

For queued commands which call the ReceiveMessageHandler which has was registered by calling SetReceiveMessageHandler the request payload is JSON or plain text.

private async static Task ReceiveMessageHandler(Message message, object userContext)
{
   AzureIoTMessageHandlerContext receiveMessageHandlerConext = (AzureIoTMessageHandlerContext)userContext;

   Console.WriteLine($"ReceiveMessageHandler handler method was called.");

   Console.WriteLine($" Message ID:{message.MessageId}");
   Console.WriteLine($" Message Schema:{message.MessageSchema}");
   Console.WriteLine($" Correlation ID:{message.CorrelationId}");
   Console.WriteLine($" Lock Token:{message.LockToken}");
   Console.WriteLine($" Component name:{message.ComponentName}");
   Console.WriteLine($" To:{message.To}");
   Console.WriteLine($" Module ID:{message.ConnectionModuleId}");
   Console.WriteLine($" Device ID:{message.ConnectionDeviceId}");
   Console.WriteLine($" User ID:{message.UserId}");
   Console.WriteLine($" CreatedAt:{message.CreationTimeUtc}");
   Console.WriteLine($" EnqueuedAt:{message.EnqueuedTimeUtc}");
   Console.WriteLine($" ExpiresAt:{message.ExpiryTimeUtc}");
   Console.WriteLine($" Delivery count:{message.DeliveryCount}");
   Console.WriteLine($" InputName:{message.InputName}");
   Console.WriteLine($" SequenceNumber:{message.SequenceNumber}");

   foreach (var property in message.Properties)
   {
      Console.WriteLine($"   Key:{property.Key} Value:{property.Value}");
   }

   Console.WriteLine($" Content encoding:{message.ContentEncoding}");
   Console.WriteLine($" Content type:{message.ContentType}");
   string payloadText = Encoding.UTF8.GetString(message.GetBytes());
   Console.WriteLine($" Content:{payloadText}");
   Console.WriteLine();

   if (!message.Properties.ContainsKey("method-name"))
   {
      await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message);
      Console.WriteLine($"   Property method-name not found");
      return;
   }

   string methodName = message.Properties["method-name"];
   if (string.IsNullOrWhiteSpace( methodName))
   {
      await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message);
      Console.WriteLine($"   Property null or white space");
      return;
   }

   if (string.IsNullOrWhiteSpace(payloadText))
   {
      await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message);
      Console.WriteLine($"   Payload null or white space");
      return;
   }

   JObject payload;

   if (IsValidJSON(payloadText))
   {
      payload = JObject.Parse(payloadText);
   }
   else
   {
      payload = new JObject
      {
         { methodName, payloadText }
      };
   }

   string downlinktopic = $"v3/{receiveMessageHandlerConext.ApplicationId}@{receiveMessageHandlerConext.TenantId}/devices/{receiveMessageHandlerConext.DeviceId}/down/push";

   DownlinkPayload downlinkPayload = new DownlinkPayload()
   {
      Downlinks = new List<Downlink>()
      {
         new Downlink()
         {
            Confirmed = false,
            //PayloadRaw = messageBody,
            PayloadDecoded = payload,
            Priority = DownlinkPriority.Normal,
            Port = 10,
            CorrelationIds = new List<string>()
            {
               message.LockToken
            }
         }
      }
   };

   Console.WriteLine($"TTN Topic :{downlinktopic}");
   Console.WriteLine($"TTN downlink JSON :{JsonConvert.SerializeObject(downlinkPayload, Formatting.Indented)}");

   //await receiveMessageHandlerConext.AzureIoTHubClient.AbandonAsync(message); // message retries
   //await receiveMessageHandlerConext.AzureIoTHubClient.CompleteAsync(message);
   await receiveMessageHandlerConext.AzureIoTHubClient.CompleteAsync(message.LockToken);
   //await receiveMessageHandlerConext.AzureIoTHubClient.RejectAsync(message); // message gone no retry
}

When I initiated an Analog queued command the message handler was invoked with the name of the command capability (Analog_Output_2) in a message property called “method-name”. For a typed parameter the message content was a string representation of the value. For an object parameter the payload contains a JSON representation of the request field(s)

The output of my test harness for a Queued Command with a typed payload

A JSON request payload supports downlink message with more that one value.

The output of my test harness for a Queued Command with an object payload(JSON)

The choice of Value_0, Value_1 (I think they are float64 type) etc. for the decoded_payload is specified in the Cayenne LPP downlink decode/encoder source code.

The context information for both comments and queued commands provides additional information required to construct the MQTT topic for publishing the downlink messages.

For queued commands the correlation_id will contain the message.LockToken so that messages can be Abandoned, Completed or Rejected. The MQTT broker publishes a series of topics so the progress of the transmission of downlink message can be monitored.

If the device is not known the Abandon method will be called immediately. For command messages Completed will be called as soon as the message is “sent”

  • v3/{application id}@{tenant id}/devices/{device id}/down/queued
  • v3/{application id}@{tenant id}/devices/{device id}/down/sent
  • v3/{application id}@{tenant id}/devices/{device id}/down/ack
  • v3/{application id}@{tenant id}/devices/{device id}/down/nack
  • v3/{application id}@{tenant id}/devices/{device id}/down/failed

For queued messages the point in the delivery process where the Abandoned, Completed and Rejected methods will be called will be configurable.

Azure IoT Central Connectivity Part3

Request Commands

I have built a couple of proof of Concept(PoC) applications to explore the Basic Telemetry and Basic Command functionality of Azure IoT Central. This blog post is about queued and non queued Cloud to Device(C2D) Commands with request parameters.

I initially created an Azure IoT Central Device Template with command and telemetry device capabilities.

“Collapsed” Command Request template
Command Request Template digital commands

I tried typed request and object based parameters to explorer how an integration with The Things Network(TTN)/The Things Industries(TTI) using the Message Queue Telemetry Transport(MQTT) interface could work.

Object parameter schema designer

With object based parameters the request JSON could contain more than one value though the validation of user provided information didn’t appear to be as robust.

Object parameter schema definition

I “migrated” my third preconfigured device to the CommandRequest template to see how the commands with Request parameters interacted with my PoC application.

After “migrating” my device I went back and created a Template view so I could visualise the simulated telemetry from my PoC application and provide a way to initiate commands (Didn’t really need four command tiles as they all open the Device commands form).

CommandRequest device template default view

From the Device Commands form I could send commands and a queued commands which had analog or digital parameters.

Device Three Command Tab

When I initiated an Analog non-queued command the default method handler was invoked with the name of the command capability (Analog_Output_1) as the method name and the payload contained a JSON representation of the request values(s). With a typed parameter a string representation of the value was in the message payload. With a typed parameter a string representation of the value was in the message payload rather than JSON.

Console application displaying Analog request and Analog Request queued commands

When I initiated an Analog queued command the message handler was invoked with the name of the command capability (Analog_Output_2) in a message property called “method-name” and the payload contained a JSON representation of the request value(s). With a typed parameter a string representation of the value was in the message payload rather than JSON.

When I initiated a Digital non-queued command the default method handler was invoked with the name of the command capability (Digital_Output_1) as the method name and the payload contained a JSON representation of the request values(s). With a typed parameter a string representation of the value was in the message payload rather than JSON.

Console application displaying Digital request and Digital Request queued commands

When I initiated a Digital queued command the message handler was invoked with the name of the command capability(Digital_Output_2) in a message property called “method-name” and the payload contained a JSON representation of the request value(s). With a typed parameter a string representation of the value was in the message payload rather than JSON.

The validation of user input wasn’t as robust as I expected, with problems selecting checkboxes with a mouse when there were several Boolean fields. I often had to click on a nearby input field and use the TAB button to navigate to the desired checkbox. I also had problems with ISO 8601 format date validation as the built in Date Picker returned a month, day, year date which was not editable and wouldn’t pass validation.

The next logical step would be to look at commands with a Response parameter but as the MQTT interface is The Things Network(TTN) and The Things Industries(TTI) is asynchronous and devices reporting every 5 minutes to a couple of times a day there could be a significant delay between sending a message and receiving an optional delivery confirmation or response.

Azure IoT Central Connectivity Part2

Basic Commands

I have been struggling with making The Things Network(TTN) and The Things Industries(TTI) uplink/downlink messages work well Azure IoT Central. To explore different messaging approaches I have built a proof of Concept(PoC) application which simulates TTN/TTI connectivity to an Azure IoT Hub, or Azure IoT Central.

This blog post is about queued and non queued Cloud to Device(C2D) commands without request or response parameters. I have mostly used non queued commands in other projects (my Azure IoT Hub LoRa and RF24L01 gateways) to “Restart” devices etc..

The first step was to create an Azure IoT Central Device Template with command and telemetry device capabilities.

CommandBasic device template device with command & telemetry capabilities

I then “migrated” my second preconfigured device to the CommandBasic template.

Migrating a device to TelemetryBasic template

I then went back and created a Template view to visualise the telemetry from my console application and initiate commands.

CommandBasic device template default view

I modified the PoC application adding handlers for Methods (SetMethodDefaultHandlerAsync) and Messages (SetReceiveMessageHandlerAsync).

private static async Task ApplicationCore(CommandLineOptions options)
{
   DeviceClient azureIoTHubClient;
   Timer MessageSender;

   try
   {
      // Open up the connection
      azureIoTHubClient = DeviceClient.CreateFromConnectionString(options.AzureIoTHubconnectionString, TransportType.Amqp_Tcp_Only);

      await azureIoTHubClient.OpenAsync();
      await azureIoTHubClient.SetReceiveMessageHandlerAsync(ReceiveMessageHandler, azureIoTHubClient);
      await azureIoTHubClient.SetMethodHandlerAsync("Named", MethodCallbackNamedHandler, null);
      await azureIoTHubClient.SetMethodDefaultHandlerAsync(MethodCallbackDefaultHandler, null);

      MessageSender = new Timer(TimerCallbackAsync, azureIoTHubClient, new TimeSpan(0, 0, 10), new TimeSpan(0, 2, 0));

      Console.WriteLine("Press any key to exit");
      while (!Console.KeyAvailable)
      {
         await Task.Delay(100);
      }
   }
   catch (Exception ex)
   {
      Console.WriteLine($"Main {ex.Message}");
      Console.WriteLine("Press <enter> to exit");
      Console.ReadLine();
   }
}

The method handler displays the method name and the message payload.

private static async Task<MethodResponse> MethodCallbackDefaultHandler(MethodRequest methodRequest, object userContext)
{
   Console.WriteLine($"Default handler method {methodRequest.Name} was called.");

   Console.WriteLine($"Payload:{methodRequest.DataAsJson}");
   Console.WriteLine();

   //return new MethodResponse(400);
   //return new MethodResponse(404);
   return new MethodResponse(200);
}

The message handler displays a selection the message properties, any attributes and the message payload.

 private async static Task ReceiveMessageHandler(Message message, object userContext)
{
   DeviceClient azureIoTHubClient = (DeviceClient)userContext;

   Console.WriteLine($"ReceiveMessageHandler handler method was called.");

   Console.WriteLine($" Message ID:{message.MessageId}");
   Console.WriteLine($" Message Schema:{message.MessageSchema}");
   Console.WriteLine($" Correlation ID:{message.CorrelationId}");
   Console.WriteLine($" Component name:{message.ComponentName}");
   Console.WriteLine($" To:{message.To}");
   Console.WriteLine($" Module ID:{message.ConnectionModuleId}");
   Console.WriteLine($" Device ID:{message.ConnectionDeviceId}");
   Console.WriteLine($" CreatedAt:{message.CreationTimeUtc}");
   Console.WriteLine($" EnqueuedAt:{message.EnqueuedTimeUtc}");
   Console.WriteLine($" ExpiresAt:{message.ExpiryTimeUtc}");
   Console.WriteLine($" Delivery count:{message.DeliveryCount}");
   Console.WriteLine($" InputName:{message.InputName}");
   Console.WriteLine($" SequenceNumber:{message.SequenceNumber}");

   foreach (var property in message.Properties)
   {
     Console.WriteLine($"   Key:{property.Key} Value:{property.Value}");
   }

   Console.WriteLine($" Content encoding:{message.ContentEncoding}");
   Console.WriteLine($" Content type:{message.ContentType}");
   Console.WriteLine($" Content:{Encoding.UTF8.GetString(message.GetBytes())}");
   Console.WriteLine();

   //await azureIoTHubClient.AbandonAsync(message); // message retries
   await azureIoTHubClient.CompleteAsync(message);
   //await azureIoTHubClient.RejectAsync(message); // message gone no retry
}

From the Device Commands tab I can could non queued and a queued commands

Device Two Commands tab

When I sent a non-queued command the default method handler was invoked with the name of the command capability (Digital_Output_0) as the method name and an empty payload. In the Azure IoT Central interface I couldn’t see any difference for successful (HTTP 200 OK) or failure (HTTP 400 Bad Request or HTTP 404 Not Found) responses. If the application was not running the command failed immediately.

Console application displaying non-queued call

With Azure IoT Explorer failure responses were visible.

Azure IoT Explorer show message with 404 response

When I sent a queued command the message handler was invoked with the name of the command capability(Digital_Output_1) in a message property called “method-name” and the payload contained only an “@” character.

Console application displaying queued call

If the application was not running the command was queued until the Console application was started. When the console application was running and AbandonAsync was called rather than CompleteAsync the message was retried 10 times. If RejectAsync was called rather than CompleteAsync the message was deleted from the queue and not retried. There didn’t appear to be any difference for the displayed Azure IoT Central or Azure IoT Hub explorer results when AbandonAsync or RejectAsync were called.

I also created a personal dashboard to visualise the telemetry data and initiate commands. The way the two commands were presented on the dashboard was quite limited so I will go back to the documentation and see what I missed

Azure IoT Central Connectivity Part1

Basic Telemetry

I have been struggling with making The Things Network(TTN) and The Things Industries(TTI) uplink/downlink messages Azure IoT Central compatible. To explore the messaging approaches used I have built a proof of Concept(PoC) application which simulates TTN/TTI connectivity to an Azure IoT Hub, or Azure IoT Central.

My “nasty” console application uses the Azure DeviceClient library (Advanced Message Queuing Protocol(AMQP) connectivity) to explore how to interface with Azure IoT Central. This first blog post is about to Device Cloud(D2C) telemetry

The first step was to create an Azure IoT Central Device Template with a selection of telemetry capabilities.

TelemetryBasic device template device capabilities

I then created a Plain old Common Language Runtime(CLR) object(PoCo) with Newtonsoft JSON library attributes to fine tune the serialisation/deserialation.

public class GPSPosition
{    
   [JsonProperty("lat")]
   public float Latitude { get; set; }
   [JsonProperty("lon")]
   public float Longitude { get; set; }
   [JsonProperty("alt")]
   public float Altitude { get; set; }
}

public class DigitialTelemetryPayload
{
   [JsonProperty("Digital_Input_0")]
   public bool DigitalInput { get; set; }

   [JsonProperty("Analog_Input_0")]
   public float AnalogInput { get; set; }

   [JsonProperty("GPS_0")]
   public GPSPosition GPSPosition { get; set; }
 }

I created five devices and generated their connection strings using the DPS individual enrollment functionality of one my other sample applications.

I then “migrated” the first device to my BasicTelemetry template

Migrating a device to TelemetryBasic template

I then went back and created a Template view to visualise the telemetry from my console application.

TelemetryBasic device template default view

Then I configured a preview device so the template view was populated with “realistic” data.

TelemetryBasic device template default view configuring a device as data source

The console application simulates a digital input (random true/false), analog input (random value between 0.0 and 1.0) and a Global Positioning System(GPS) location (Christchurch Anglican Cathedral with a random latitude, longitude and altitude offset) .

Basic Telemetry Console Application

The final step was to create an Azure IoT Central Personal dashboard to visualise the data from my simulated device.

Basic Telemetry Dashboard

Connecting a Device, creating a Device Template, Migrating the Device, and then displaying telemetry on a personal dashboard was a good introduction to interfacing with and configuring Azure IoT Central devices.

In other applications I have mapped “payload_fields” to an Azure IoT Central telemetry payload with minimal code.

{
   "app_id": "rak811wisnodetest",
   "dev_id": "seeeduinolorawan4",
   "hardware_serial": "1234567890123456",
   "port": 10,
   "counter": 1,
   "is_retry": true,
   "payload_raw": "AWcBEAFlAGQBAAEBAgAyAYgAqYgGIxgBJuw=",
   "payload_fields": {
      "analog_in_1": 0.5,
      "digital_in_1": 1,
      "gps_1": {
         "altitude": 755,
         "latitude": 4.34,
         "longitude": 40.22
      },
      "luminosity_1": 100,
      "temperature_1": 27.2
   },
   "metadata": {
      "time": "2020-08-28T10:41:04.496594225Z",
      "frequency": 923.4,
      "modulation": "LORA",
      "data_rate": "SF12BW125",
      "coding_rate": "4/5",
      "gateways": [
         {
            "gtw_id": "eui-b827ebfffe6c279d",
            "timestamp": 3971612260,
            "time": "2020-08-28T10:41:03.313471Z",
            "channel": 1,
            "rssi": -53,
            "snr": 11.2,
            "rf_chain": 0,
            "latitude": -43.49885,
            "longitude": 172.60095,
            "altitude": 25
         }
      ]
   },
   "downlink_url": "https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/rak811wisnodetest/azure-webapi-endpoint?key=ttn-account-v2.12345678901234567_12345_1234567-dduo"
}

This was a longish post with lots of screen shots so I don’t have to repeat core setup instructions in future posts.