Myriota Connector – Azure IoT Hub DTDL Support

The Myriota connector supports the use of Digital Twin Definition Language(DTDL) for Azure IoT Hub Connection Strings and the Azure IoT Hub Device Provisioning Service(DPS).

{
  "ConnectionStrings": {
    "ApplicationInsights": "...",
    "UplinkQueueStorage": "...",
    "PayloadFormattersStorage": "..."
  },
  "AzureIoT": {
   ...
 "ApplicationToDtdlModelIdMapping": {
   "tracker": "dtmi:myriotaconnector:Tracker_2lb;1",
     }
  }
 ...    
}

The Digital Twin Definition Language(DTDL) configuration used when a device is provisioned or when it connects is determined by the payload application which is based on the Myriota Destination endpoint.

The Azure Function Configuration of Application to DTDL Model ID

BEWARE – They application in ApplicationToDtdlModelIdMapping is case sensitive!

Azure IoT Central Device Template Configuration

I used Azure IoT Central Device Template functionality to create my Azure Digital Twin definitions.

Azure IoT Hub Device Connection String

The DeviceClient CreateFromConnectionString method has an optional ClientOptions parameter which specifies the DTLDL model ID for the duration of the connection.

private async Task<DeviceClient> AzureIoTHubDeviceConnectionStringConnectAsync(string terminalId, string application, object context)
{
    DeviceClient deviceClient;

    if (_azureIoTSettings.ApplicationToDtdlModelIdMapping.TryGetValue(application, out string? modelId))
    {
        ClientOptions clientOptions = new ClientOptions()
        {
            ModelId = modelId
        };

        deviceClient = DeviceClient.CreateFromConnectionString(_azureIoTSettings.AzureIoTHub.ConnectionString, terminalId, TransportSettings, clientOptions);
    }
    else
    { 
        deviceClient = DeviceClient.CreateFromConnectionString(_azureIoTSettings.AzureIoTHub.ConnectionString, terminalId, TransportSettings);
    }

    await deviceClient.OpenAsync();

    return deviceClient;
}
Azure IoT Explorer Telemetry message with DTDL Model ID

Azure IoT Hub Device Provisioning Service

The ProvisioningDeviceClient RegisterAsync method has an optional ProvisionRegistrationAdditionalData parameter. The PnpConnection CreateDpsPayload is used to generate the JsonData property which specifies the DTLDL model ID used when the device is initially provisioned.

private async Task<DeviceClient> AzureIoTHubDeviceProvisioningServiceConnectAsync(string terminalId, string application, object context)
{
    DeviceClient deviceClient;

    string deviceKey;
    using (var hmac = new HMACSHA256(Convert.FromBase64String(_azureIoTSettings.AzureIoTHub.DeviceProvisioningService.GroupEnrollmentKey)))
    {
        deviceKey = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(terminalId)));
    }

    using (var securityProvider = new SecurityProviderSymmetricKey(terminalId, deviceKey, null))
    {
        using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
        {
            DeviceRegistrationResult result;

            ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
                _azureIoTSettings.AzureIoTHub.DeviceProvisioningService.GlobalDeviceEndpoint,
                _azureIoTSettings.AzureIoTHub.DeviceProvisioningService.IdScope,
                securityProvider,
            transport);

            if (_azureIoTSettings.ApplicationToDtdlModelIdMapping.TryGetValue(application, out string? modelId))
            {
                ClientOptions clientOptions = new ClientOptions()
                {
                    ModelId = modelId
                };

                ProvisioningRegistrationAdditionalData provisioningRegistrationAdditionalData = new ProvisioningRegistrationAdditionalData()
                {
                    JsonData = PnpConvention.CreateDpsPayload(modelId)
                };
                result = await provClient.RegisterAsync(provisioningRegistrationAdditionalData);
            }
            else
            {
                result = await provClient.RegisterAsync();
            }
  
            if (result.Status != ProvisioningRegistrationStatusType.Assigned)
            {
                _logger.LogWarning("Uplink-DeviceID:{0} RegisterAsync status:{1} failed ", terminalId, result.Status);

                throw new ApplicationException($"Uplink-DeviceID:{0} RegisterAsync status:{1} failed");
            }

            IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

            deviceClient = DeviceClient.Create(result.AssignedHub, authentication, TransportSettings);
        }
    }

    await deviceClient.OpenAsync();

    return deviceClient;
}
Azure IoT Central Device Connection Group configuration

An Azure IoT Central Device connection groups can be configured to “automagically” provision devices.

Myriota Connector – Azure IoT Hub Connectivity

The Myriota connector supports the use of Azure IoT Hub Connection Strings and the Azure IoT Hub Device Provisioning Service(DPS) for device management. I use Alastair Crabtree’s LazyCache to store Azure IoT Hub connections which are opened the first time they are used.

 public async Task<DeviceClient> GetOrAddAsync(string terminalId, object context)
 {
     DeviceClient deviceClient;

     switch (_azureIoTSettings.AzureIoTHub.ConnectionType)
     {
         case Models.AzureIotHubConnectionType.DeviceConnectionString:
             deviceClient = await _azuredeviceClientCache.GetOrAddAsync(terminalId, (ICacheEntry x) => AzureIoTHubDeviceConnectionStringConnectAsync(terminalId, context));
             break;
         case Models.AzureIotHubConnectionType.DeviceProvisioningService:
             deviceClient = await _azuredeviceClientCache.GetOrAddAsync(terminalId, (ICacheEntry x) => AzureIoTHubDeviceProvisioningServiceConnectAsync(terminalId, context));
             break;
         default:
             _logger.LogError("Uplink- Azure IoT Hub ConnectionType unknown {0}", _azureIoTSettings.AzureIoTHub.ConnectionType);

             throw new NotImplementedException("AzureIoT Hub unsupported ConnectionType");
     }

     return deviceClient;
 }

The IAzureDeviceClientCache.GetOrAddAsync method returns an open Azure IoT Hub DeviceClient connection or uses the method specified in the application configuration.

Azure IoT Hub Device Connection String

The Azure IoT Hub delegate uses a Device Connection String which is retrieved from the application configuration.

{
  "ConnectionStrings": {
    "ApplicationInsights": "...",
    "UplinkQueueStorage": "...",
    "PayloadFormattersStorage": "..."
  },
  "AzureIoT": {
    "AzureIoTHub": {
      "ConnectionType": "DeviceConnectionString",
      "connectionString": "HostName=....azure-devices.net;SharedAccessKeyName=device;SharedAccessKey=...",
        }
   }
 ...    
}
Azure Function with IoT Hub Device connection string configuration
private async Task<DeviceClient> AzureIoTHubDeviceConnectionStringConnectAsync(string terminalId, object context)
{
    DeviceClient deviceClient = DeviceClient.CreateFromConnectionString(_azureIoTSettings.AzureIoTHub.ConnectionString, terminalId, TransportSettings);

    await deviceClient.OpenAsync();

    return deviceClient;
 }
Azure IoT Hub Device Shared Access Policy for Device Connection String

One of my customers uses an Azure Logic Application to manage Myriota and Azure IoT Connector configuration.

Azure IoT Hub manual Device configuration

Azure IoT Hub Device Provisioning Service

The Azure IoT Hub Device Provisioning Service(DPS) delegate uses Symmetric Key Attestation with the Global Device Endpoint, ID Scope and Group Enrollment Key retrieved from the application configuration.

{
  "ConnectionStrings": {
    "ApplicationInsights": "...",
    "UplinkQueueStorage": "...",
    "PayloadFormattersStorage": "..."
  },
  "AzureIoT": {
      "ConnectionType": "DeviceProvisioningService",
      "DeviceProvisioningServiceIoTHub": {
        "GlobalDeviceEndpoint": "global.azure-devices-provisioning.net",
        "IDScope": ".....",
        "GroupEnrollmentKey": "...."
      }
   }
}
Azure IoT Function with Azure IoT Hub Device Provisioning Service(DPS) configuration

Symmetric key attestation with the Azure IoT Hub Device Provisioning Service(DPS) is performed using the same security tokens supported by Azure IoT Hubs to securely connect devices. The symmetric key of an enrollment group isn’t used directly by devices in the provisioning process. Instead, devices that provision through an enrollment group do so using a derived device key.

private async Task<DeviceClient> AzureIoTHubDeviceProvisioningServiceConnectAsync(string terminalId, object context)
{
    DeviceClient deviceClient;

    string deviceKey;
    using (var hmac = new HMACSHA256(Convert.FromBase64String(_azureIoTSettings.AzureIoTHub.DeviceProvisioningService.GroupEnrollmentKey)))
    {
        deviceKey = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(terminalId)));
    }

    using (var securityProvider = new SecurityProviderSymmetricKey(terminalId, deviceKey, null))
    {
        using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
        {
            DeviceRegistrationResult result;

            ProvisioningDeviceClient provClient = ProvisioningDeviceClient.Create(
                _azureIoTSettings.AzureIoTHub.DeviceProvisioningService.GlobalDeviceEndpoint,
                _azureIoTSettings.AzureIoTHub.DeviceProvisioningService.IdScope,
                securityProvider,
                transport);

            result = await provClient.RegisterAsync();
  
            if (result.Status != ProvisioningRegistrationStatusType.Assigned)
            {
                _logger.LogWarning("Uplink-DeviceID:{0} RegisterAsync status:{1} failed ", terminalId, result.Status);

                throw new ApplicationException($"Uplink-DeviceID:{0} RegisterAsync status:{1} failed");
            }

            IAuthenticationMethod authentication = new DeviceAuthenticationWithRegistrySymmetricKey(result.DeviceId, (securityProvider as SecurityProviderSymmetricKey).GetPrimaryKey());

            deviceClient = DeviceClient.Create(result.AssignedHub, authentication, TransportSettings);
        }
    }

    await deviceClient.OpenAsync();

    return deviceClient;
}

The derived device key is a hash of the device’s registration ID and is computed using the symmetric key of the enrollment group. The device can then use its derived device key to sign the SAS token it uses to register with DPS.

Azure Device Provisioning Service Adding Enrollment Group Attestation
Azure Device Provisioning Service Add Enrollment Group IoT Hub(s) selection.
Azure Device Provisioning Service Manager Enrollments

For initial development and testing I ran the function application in the desktop emulator and simulated Myriota Device Manager webhook calls with Azure Storage Explorer and modified sample payloads.

Azure Storage Explorer Storage Account Queued Messages

I then used Azure IoT Explorer to configure devices, view uplink traffic etc.

Azure IoT Explorer Devices

When I connected to my Azure IoT Hub shortly after starting the Myriota Azure IoT Connector Function my test devices started connecting as messages arrived.

Azure IoT Explorer Device Telemetry

I then deployed my function to Azure and configured the Azure IoT Hub connection string, Azure Application Insights connection string etc.

Azure Portal Myriota Resource Group
Azure Portal Myriota IoT Hub Metrics

There was often a significant delay for the Device Status to update. which shouldn’t be a problem.

Azure Functions Isolated Worker support for VB.Net 4.8

As part of my “day job” I spend a bit of time working with VB.Net 4.X “legacy” projects doing upgrades, and bug fixes. Currently I am updating a number of Windows Service applications to run as Microsoft Azure Functions. With the release of the Azure functions runtime V4 Isolated Worker Processes with .NET Framework V4.8 support this is the last post in my Azure Functions with VB.Net 4.X and Azure Functions with VB.Net on .NET Core V6 series.

I have published source code for Azure Storage BlobTrigger, Azure Storage QueueTrigger, and TimerTriggers.

Visual Studio Solution explorer Azure Functions projects

All of the examples now have a program.vb which initialises the Trigger.

Namespace VBNet....TriggerIsolated
    Friend Class Program
        Public Shared Sub Main(ByVal args As String())
            Call FunctionsDebugger.Enable()

            Dim host = New HostBuilder().ConfigureFunctionsWorkerDefaults().Build()

            host.Run()
        End Sub
    End Class
End Namespace

All of the Isolated worker process Triggers displayed this message which appeared to be benign.

Csproj not found in C:\Users\..\VBNetHttpTriggerIsolated\bin\Debug\net48 directory tree. Skipping user secrets file configuration.

There were a lot of articles about problems building Docker images but the only relevant ones appeared to talk about getting F# and other .NET Core languages to work in Azure Functions.

Namespace devMobile.Azure.VBNetBlobTriggerIsolated
    Public Class BlobTrigger
        Private ReadOnly _logger As ILogger

        Public Sub New(ByVal loggerFactory As ILoggerFactory)
            _logger = loggerFactory.CreateLogger(Of BlobTrigger)()
        End Sub

        <[Function]("vbnetblobtriggerisolated")>
        Public Sub Run(
        <BlobTrigger("vbnetblobtriggerisolated/{name}", Connection:="blobendpoint")> ByVal myBlob As String, ByVal name As String)

            _logger.LogInformation($"VB.Net NET 4.8 Isolated Blob trigger function Processed blob Name: {name}  Data: {myBlob}")
        End Sub
    End Class
End Namespace

I used Azure Storage Explorer to upload files containing Lorem Ipsum for testing the BlobTrigger.

Azure BlobTrigger function running in the desktop emulator
Azure BlobTrigger Function logging in Application Insights

I used Telerik Fiddler to POST messages to the desktop emulator and Azure endpoints.

Namespace VBNetHttpTriggerIsolated
    Public Class HttpTrigger
        Private Shared executionCount As Int32
        Private ReadOnly _logger As ILogger

        Public Sub New(ByVal loggerFactory As ILoggerFactory)
            _logger = loggerFactory.CreateLogger(Of HttpTrigger)()
        End Sub

        <[Function]("Notifications")>
        Public Function Run(
        <HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")> ByVal req As HttpRequestData) As HttpResponseData
            Interlocked.Increment(executionCount)

            _logger.LogInformation("VB.Net NET 4.8 Isolated HTTP trigger Execution count:{executionCount} Method:{req.Method}", executionCount, req.Method)

            Dim response = req.CreateResponse(HttpStatusCode.OK)
            response.Headers.Add("Content-Type", "text/plain; charset=utf-8")

            Return response
        End Function
    End Class
End Namespace
Azure HttpTrigger Function running in the desktop emulator
Azure HttpTrigger Function logging in Application Insights

I used Azure Storage Explorer to create messages for testing the QueueTrigger

Namespace devMobile.Azure.VBNetQueueTriggerIsolated

    Public Class QueueTrigger
        Private Shared _logger As ILogger
        Private Shared _concurrencyCount As Integer = 0
        Private Shared _executionCount As Integer = 0

        Public Sub New(ByVal loggerFactory As ILoggerFactory)
            _logger = loggerFactory.CreateLogger(Of QueueTrigger)()
        End Sub

        <[Function]("VBNetQueueTriggerIsolated")>
        Public Sub Run(
        <QueueTrigger("vbnetqueuetriggerisolated", Connection:="QueueEndpoint")> ByVal message As String)
            Interlocked.Increment(_concurrencyCount)
            Interlocked.Increment(_executionCount)

            _logger.LogInformation("VB.Net .NET 4.8 Isolated Queue Trigger Concurrency:{_concurrencyCount} ExecutionCount:{_executionCount} Message:{message}", _concurrencyCount, _executionCount, message)

            Interlocked.Decrement(_concurrencyCount)
        End Sub
    End Class
End Namespace
Azure QueueTrigger Function running in the desktop emulator
Azure QueueTrigger Function logging in Application Insights
Namespace devMobile.Azure.VBNetTimerTriggerIsolated
    Public Class TimerTrigger
        Private Shared _logger As ILogger
        Private Shared _executionCount As Integer = 0

        Public Sub New(ByVal loggerFactory As ILoggerFactory)
            _logger = loggerFactory.CreateLogger(Of TimerTrigger)()
        End Sub

        <[Function]("Timer")>
        Public Sub Run(
        <TimerTrigger("0 */1 * * * *")> ByVal myTimer As MyInfo)

            Interlocked.Increment(_executionCount)
            _logger.LogInformation("VB.Net Isolated TimerTrigger next trigger:{0} Execution count:{1}", myTimer.ScheduleStatus.Next, _executionCount)
        End Sub
    End Class
Azure TimerTrigger Function running in the desktop emulator
Azure TimerTrigger Function logging in Application Insights

The development, debugging and deployment of these functions took a lot of time. Initially Azure Application Insights didn’t work when the Azure Isolated Worker triggers were deployed to Azure. After some experimentation I found that Application Insights Connection Strings worked and Application Instrumentation Keys did not.

With the Microsoft: ‘We Do Not Plan to Evolve Visual Basic as a Language this should hopefully be my last post about VB.Net ever.

Azure Functions with VB.Net on .NET Core V6

A year and a half ago I wrote a post about how to build Azure functions with VB.Net and the .NET Framework 4.X. The Microsoft VB team posted about Visual Basic Support for .NET 5.0 in March 2020 then went quiet, so my customer put the project on hold. Since then, a lot has changed .NET Core 3.1 LTS ends December 12, 2022, and .NET Core 5.0 support (no LTS) ended May 10, 2022 so I have ported the samples to .NET Core V6.

The process is similar (but different) to the original approach

The VB.Net Solution from June 2021

First step is to create a Visual Basic .NET Core V6 console application

Visual Studio 2022 “Add a new project”

The specify a name for the new project.

Visual Studio 2022 Add Project “Configure your new project”

Then select the version of .NET Core used

Visual Studio 2022 Add Project “Additional information”

Then rename program.cs to a name which highlights that it is a trigger

Visual Studio 2022 rename program.vb to TimerTrigger.vb

The initial version of the TimerTrigger code was “inspired” by the VB.Net 4.8 version.

'---------------------------------------------------------------------------------
' Copyright (c) November 2022, devMobile Software
'
' Licensed under the Apache License, Version 2.0 (the "License");
' you may Not use this file except in compliance with the License.
' You may obtain a copy of the License at
'
'     http://www.apache.org/licenses/LICENSE-2.0
'
' Unless required by applicable law Or agreed to in writing, software
' distributed under the License Is distributed on an "AS IS" BASIS,
' WITHOUT WARRANTIES Or CONDITIONS OF ANY KIND, either express Or implied.
' See the License for the specific language governing permissions And
' limitations under the License.
'
'---------------------------------------------------------------------------------
Imports System.Threading

Imports Microsoft.Azure.WebJobs
Imports Microsoft.Extensions.Logging


Public Class TimerTrigger
    Shared executionCount As Int32

    <FunctionName("Timer")>
    Public Shared Sub Run(<TimerTrigger("0 */1 * * * *")> myTimer As TimerInfo, log As ILogger)
        Interlocked.Increment(executionCount)

        log.LogInformation("VB.Net .NET V6 TimerTrigger next trigger:{0} Execution count:{1}", myTimer.ScheduleStatus.Next, executionCount)

    End Sub
End Class

Visual Studio 2022 highlighting missing libraries
Visual Studio 2022 with additional function SDK references

The next step is to add the hosts.json(empty for timer tigger) and localsettings.json to configure the function

Visual 2022 Hosts.json file
Visual Studio 2022 showing hosts.json & local.settings.json

Then I could run the function in the Azure Functions runtime emulator and “single step” in the Visual Studio 2022 Debugger.

VB.Net .NET Core V6 Timer Trigger running in emulator

For completeness I also built sample BlobTrigger, HttpTrigger and QueueTrigger versions

VB.Net .NET Core V6 Blob Trigger running in emulator
VB.Net .NET Core V6 HTTP Trigger running in emulator
VB.Net .NET Core V6 Queue Trigger running in emulator

I also deployed the Azure Storage QueueTrigger to Microsoft Azure, configured it, and then stress tested it with multiple instances of my QueueMessageGenerator.

Queue Trigger Function deployment
Queue Trigger configuration
Queue Trigger Throughput 48K messages

What if it goes wrong…

“Can’t determine project language from files. Please add one of [–csharp, –javascript, –typescript, –java, –powershell, –customer]

Check “FUNCTIONS_WORKER_RUNTIME” in the local.settings.json file.

The baked in error logging doesn’t handle broken message formats very well. Look at the call stack or single step through the application to find the message format that is broken

Visual Studio 2022 editor with malformed message highlighted

WARNING

I assume this is not a supported approach so use

“at your own risk”

.NET Core web API + Dapper – Authorisation of Data Access

The theme of this post is controlling users’ ability to read and write rows in a table. The best scenario I could come up with using the World Wide Importers database was around controlling access to Customer information.

This would be a representative set of “project requirements”…

  • Salespeople tend to look after categories of Customers
    • Kayla – Novelty Shops
    • Hudson – Supermarkets
    • Issabella – Computer Stores
    • Sophia – Gift Stores, Novelty Shops
    • Amy – Corporates
    • Anthony – Novelty Stores
    • Alica – Coporates
    • Stella – Supermarkets
  • But some Salespeople have direct relationships with Customers
    • Kayla – Corporate customers Eric Torres & Cosmina
    • Hudson – Tailspin Toys Head Office
    • Issabell – Tailspin Toys (Sylvanite, MT), Tailspin Toys (Sun River, MT), Tailspin Toys (Sylvanite, MT)
  • No changes to the database which could break the existing solution

In a previous engagement we added CustomerCategoryPerson and CustomerPerson like tables to the database to control read/write access to Customers’ information.

The CustomerCategoryPerson table links the CustomerCategory and Person tables with a flag (IsWritable) which indicates whether the Person can read/write Customer information for all the Customers in a CustomerCategory.

CREATE TABLE [Sales].[CustomerCategoryPerson](
	[CustomerCategoryPersonID] [int] IDENTITY(1,1) NOT NULL,
	[CustomerCategoryId] [int] NOT NULL,
	[PersonId] [int] NOT NULL,
	[IsWritable] [bit] NOT NULL,
	[LastUpdatedBy] [int] NOT NULL,
 CONSTRAINT [PK_CustomerCategoryPerson] PRIMARY KEY CLUSTERED 
(
	[CustomerCategoryPersonID] ASC
)...

The CustomerPerson table links the Customer and Person tables with a flag (IsWritable) which indicates whether a Person can read/write a Customer’s information.

CREATE TABLE [Sales].[CustomerPerson](
	[CustomerPersonId] [int] IDENTITY(1,1) NOT NULL,
	[CustomerID] [int] NOT NULL,
	[PersonId] [int] NOT NULL,
	[IsWritable] [bit] NOT NULL,
	[LastEditedBy] [int] NOT NULL,
 CONSTRAINT [PK_CustomerPerson] PRIMARY KEY CLUSTERED 
(
	[CustomerPersonId] ASC
)...

Users can do “wildcard” searches for Customers and the results set has to be limited to “their” customers and customers in the customer categories they are assigned too.

ALTER PROCEDURE [Sales].[CustomersNameSearchUnionV1]
@UserId as int,
@SearchText nvarchar(20),
@MaximumRowsToReturn int
AS
BEGIN
	-- Individual assignment
    SELECT TOP(@MaximumRowsToReturn) [Customers].[CustomerID] as "ID", [Customers].[CustomerName] as "Name", [Customers].[IsOnCreditHold] as "IsOnCreditHold"
    FROM Sales.Customers
	INNER JOIN [Sales].[CustomerPerson] ON ([Sales].[Customers].[CustomerId] = [Sales].[CustomerPerson].[CustomerId])
    WHERE ((CustomerName LIKE N'%' + @SearchText + N'%')
		AND ([Sales].[CustomerPerson].PersonId = @UserId))
    --ORDER BY [CustomerName]

	UNION 
	
	-- group assignment
   SELECT TOP(@MaximumRowsToReturn) [Customers].[CustomerID] as "ID", [Customers].[CustomerName] as "Name", [Customers].[IsOnCreditHold] as "IsOnCreditHold"
   FROM [Sales].[Customers]
      INNER JOIN [Sales].[CustomerCategories] ON ([Sales].[Customers].[CustomerCategoryID] = [Sales].[CustomerCategories].[CustomerCategoryID])
      INNER JOIN [Sales].[CustomerCategoryPerson] ON ([Sales].[Customers].[CustomerCategoryID] = [CustomerCategoryPerson].[CustomerCategoryID])
    WHERE ((CustomerName LIKE N'%' + @SearchText + N'%')
		AND ([Sales].[CustomerCategoryPerson].PersonId = @UserId))

END;

This approach increases the complexity and reduces the maintainability of stored procedures which have to control the reading/writing of Customer information. Several times I have extracted customer information read\write controls to a couple of database views, one for controlling read access.

CREATE VIEW [Sales].[CustomerPersonReadV1]
AS
-- Individual assignment
   SELECT [Sales].[Customers].[CustomerID], [Sales].[CustomerPerson].[PersonID], [Sales].[Customers].[CustomerCategoryID]
   FROM [Sales].[Customers]
      INNER JOIN [Sales].[CustomerPerson] ON ( [Sales].[Customers].[CustomerID] = CustomerPerson.CustomerID)

UNION -- Takes care of duplicates

-- Group assignment
   SELECT [Sales].[Customers].[CustomerID], [Sales].[CustomerCategoryPerson].[PersonID], [Sales].[Customers].[CustomerCategoryID]
   FROM [Sales].[Customers]
      --INNER JOIN [Sales].[CustomerCategories] ON ([Sales].[Customers].[CustomerCategoryID] = [Sales].[CustomerCategories].[CustomerCategoryID])
      INNER JOIN [Sales].[CustomerCategoryPerson] ON ([Sales].[Customers].[CustomerCategoryID] = [CustomerCategoryPerson].[CustomerCategoryID])

The other database for controlling write access

CREATE VIEW [Sales].[CustomerPersonWriteV1]
AS
-- Individual assignment
   SELECT [Sales].[Customers].[CustomerID], [Sales].[CustomerPerson].[PersonID], [Sales].[Customers].[CustomerCategoryID]
   FROM [Sales].[Customers]
      INNER JOIN [Sales].[CustomerPerson] ON (([Sales].[Customers].[CustomerID] = [CustomerPerson].[CustomerID]) AND ([Sales].[CustomerPerson].[IsWritable] = 1))

UNION -- Takes care of duplicates

-- Group assignment
   SELECT [Sales].[Customers].[CustomerID], [Sales].[CustomerCategoryPerson].[PersonID], [Sales].[Customers].[CustomerCategoryID]
   FROM [Sales].[Customers]
      INNER JOIN [Sales].[CustomerCategories] ON ([Sales].[Customers].[CustomerCategoryID] = [Sales].[CustomerCategories].[CustomerCategoryID])
      INNER JOIN [Sales].[CustomerCategoryPerson] ON ([Sales].[Customers].[CustomerCategoryID] = [CustomerCategoryPerson].[CustomerCategoryID] AND ([Sales].[CustomerCategoryPerson].[IsWritable] = 1))

The versioning of database views uses the same approach as stored procedures. When a view is updated (the columns returned changes , updated constraints etc.) the version number is incremented. Then we work through the dependencies list checking and updating the view version used and re-testing.

SQL Server Management Studio displaying objects which depend on the view

These two views are the UNION of the users individual and group access permissions. (If a user has Write they also have Read access). This reduces the complexity of stored procedures used for reading from and writing to the Customer table.

CREATE PROCEDURE [Sales].[CustomersListV1]
@UserId as int
AS
BEGIN
SELECT [Customers].[CustomerID] as "ID", [Customers].[CustomerName] as "Name", [Customers].[IsOnCreditHold] as "IsOnCreditHold"
	FROM [Sales].[Customers]
		INNER JOIN [Sales].[CustomerPersonReadV1] ON ([Sales].[Customers].[CustomerID] = [Sales].[CustomerPersonReadV1].CustomerID)
    WHERE ([Sales].[CustomerPersonReadV1].PersonId = @UserId)
	ORDER BY Name
END

The GET method of the Customer controller returns a list of all the Customers the current user has read only access to using their individual and group assignment.

[HttpGet(), Authorize(Roles = "SalesPerson,SalesAdministrator")]
[ProducesResponseType(StatusCodes.Status200OK, Type = typeof(List<Models.CustomerListDtoV1>))]
public async Task<ActionResult<IEnumerable<Models.CustomerListDtoV1>>> Get()
{
      IEnumerable<Models.CustomerListDtoV1> response;

      using (SqlConnection db = new SqlConnection(this.connectionString))
      {
         response = await db.QueryWithRetryAsync<Models.CustomerListDtoV1>(sql: "[Sales].[CustomersListV1]", param: new { userId = HttpContext.PersonId() }, commandType: CommandType.StoredProcedure);
      }

   return this.Ok(response);
}

The CustomerPersonWriteV1 view is used to stop users without IsWritable set (individual or group) updating a Customers IsOnCreditHold flag.

CREATE PROCEDURE [Sales].[CustomerCreditHoldStatusUpdateV1]
@UserID int,
@CustomerId int,
@IsOnCreditHold Bit
AS
BEGIN
    UPDATE [Sales].[Customers]
    SET IsOnCreditHold = @IsOnCreditHold, LastEditedBy = @UserID
	FROM [Sales].[Customers]
		INNER JOIN [Sales].[CustomerPersonWriteV1] ON ([Sales].[Customers].[CustomerID] = [Sales].[CustomerPersonWriteV1].CustomerID)
    WHERE (([Sales].[CustomerPersonWriteV1].PersonId = @UserId) 
		AND ([Sales].[Customers].[CustomerID] = @CustomerId )
		AND (IsOnCreditHold <> @IsOnCreditHold))
	
END

The PUT CreditHold method uses a combination of roles (Aministrator,SalesAdministrator,SalesPerson) and database views (CustomerPersonWriteV1) to control the updating of customer data.

[HttpPut("{customerId}/CreditStatus", Name ="CreditHold")]
[Authorize(Roles = "Aministrator,SalesAdministrator,SalesPerson")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status409Conflict)]
public async Task<IActionResult> CustomerCeditHold(int customerId, [FromBody] Models.CustomerCreditHoldUpdateV1 request )
{
    request.UserId = HttpContext.PersonId();
    request.CustomerId = customerId;

    using (SqlConnection db = new SqlConnection(connectionString))
    {
        if (await db.ExecuteWithRetryAsync("[Sales].[CustomerCreditHoldStatusUpdateV1]", param: request, commandType: CommandType.StoredProcedure) != 1)
        {
            logger.LogWarning("Person {UserId} Customer {CustomerId} IsOnCreditHold {IsOnCreditHold} update failed", request.UserId, request.CustomerId, request.IsOnCreditHold);

            return this.Conflict();
        }
    }

    return this.Ok();
}

My customers usually don’t have a lot of automated testing so minimising the impact of changes across the database and codebase is critical. Sometimes we duplicate code (definitely not DRY) so that the amount of functionality that has to be retested is reduced. We ensure this is time allocated for revisiting these decisions and remediating as required.

.NET Core web API + Dapper – Authorisation Permissions

The permissions required for an on-premises system running in a trusted environment are often minimalist. The World Wide Importers database People table has IsSystemUser, IsEmployee, IsSalesperson which is representative of the granularity of permissions I have encountered in Windows Forms .NET, ASP.NET Web Forms and other “legacy” applications.

CREATE TABLE [Application].[People](
    [PersonID] [int] NOT NULL,
    [FullName] nvarchar NOT NULL,
    [PreferredName] nvarchar NOT NULL,
    [SearchName] AS (concat([PreferredName],N' ',[FullName])) PERSISTED NOT NULL,
    [IsPermittedToLogon] [bit] NOT NULL,
    [LogonName] nvarchar NULL,
    [IsExternalLogonProvider] [bit] NOT NULL,
    [HashedPassword] varbinary NULL,
    [IsSystemUser] [bit] NOT NULL,
    [IsEmployee] [bit] NOT NULL,
    [IsSalesperson] [bit] NOT NULL,
    [UserPreferences] nvarchar NULL,
    [PhoneNumber] nvarchar NULL,
    [FaxNumber] nvarchar NULL,
    [EmailAddress] nvarchar NULL,
    [Photo] varbinary NULL,
    [CustomFields] nvarchar NULL,
    [OtherLanguages] AS (json_query([CustomFields],N'$.OtherLanguages')),
    [LastEditedBy] [int] NOT NULL,
    [ValidFrom] datetime2 GENERATED ALWAYS AS ROW START NOT NULL,
    [ValidTo] datetime2 GENERATED ALWAYS AS ROW END NOT NULL,
CONSTRAINT [PK_Application_People] PRIMARY KEY CLUSTERED
(
    [PersonID] ASC
)

The existing application appears to have a method for a Person to change their password which calls the [Website].[ChangePassword] stored procedure (I was surprised that the stored procedure didn’t set the LastEditedBy value).

CREATE PROCEDURE [Website].[ChangePassword]
@PersonID int,
@OldPassword nvarchar(40),
@NewPassword nvarchar(40)
WITH EXECUTE AS OWNER
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;

    UPDATE [Application].People
    SET IsPermittedToLogon = 1,
        HashedPassword = HASHBYTES(N'SHA2_256', @NewPassword + FullName)
    WHERE PersonID = @PersonID
    AND PersonID <> 1
    AND HashedPassword = HASHBYTES(N'SHA2_256', @OldPassword + FullName);

    IF @@ROWCOUNT = 0
    BEGIN
        PRINT N'The PersonID must be valid, and the old password must be valid.';
        PRINT N'If the user has also changed name, please contact the IT staff to assist.';
        THROW 51000, N'Invalid Password Change', 1;
        RETURN -1;
    END;
END;

The new version removes the PersonId special case (Assumed that PersonId 1 can’t logon and the use of Throw). I think the use of Throw can add significant complexity to the exception handling of the WebAPI controller that calls the stored procedure

ALTER PROCEDURE [Website].[PersonPasswordChangeV1]
	@UserID int,
	@PasswordOld nvarchar(40),
	@PasswordNew nvarchar(40)
WITH EXECUTE AS OWNER
AS
BEGIN
    UPDATE [Application].People
    SET IsPermittedToLogon = 1
        ,HashedPassword = HASHBYTES(N'SHA2_256', @PasswordNew + FullName)
		,LastEditedBy = @UserID
    WHERE ((PersonID = @UserID )
		AND (HashedPassword = HASHBYTES(N'SHA2_256', @PasswordOld + FullName)))
END;

The PasswordChange method of the Person Controller only requires the caller to be authenticated.

/// <summary>
/// Changes current user's password.
/// </summary>
/// <param name="request">Current password and new password</param>
/// <response code="200">Password changed.</response>
/// <response code="401">Unauthorised, bearer token missing or expired.</response>
/// <response code="409">Previous password invalid or User name has changed.</response>
[Authorize()]
[HttpPut(Name = "PasswordChange")]
public async Task<ActionResult> PasswordChange([FromBody] Models.PersonPasswordChangeRequest request)
{
    request.UserID = HttpContext.PersonId();

    using (SqlConnection db = new SqlConnection(connectionString))
    {
        if (await db.ExecuteWithRetryAsync("[WebSite].[PersonPasswordChangeV1]", param: request, commandType: CommandType.StoredProcedure) != 1)
        {
            logger.LogWarning("Person {0} password change failed", request.UserID);

            return this.Conflict();
        }
    }

    return this.Ok();
}

The new application will have functionality for resetting a Person’s password. Access to this functionality will be restricted to people with the “Administrator” and “PasswordReset” roles.

CREATE PROCEDURE [Website].[PersonPasswordResetV1]
@UserID int,
@PersonID int,
@Password nvarchar(40)
WITH EXECUTE AS OWNER
AS
BEGIN
    UPDATE [Application].People
    SET IsPermittedToLogon = 1
        ,HashedPassword = HASHBYTES(N'SHA2_256', @Password + FullName)
		,LastEditedBy = @UserID
    WHERE PersonID = @PersonID
END;

One of the conventions we often use, is that the first parameter of any stored procedure that is called once a User has logged on is their unique identifier which is used for data access permissions and change tracking.

[Authorize(Roles = "Administrator")]
[HttpPut("{personId:int}", Name = "PasswordReset")]
public async Task<ActionResult> PasswordReset([Range(1, int.MaxValue, ErrorMessage = "Person id must greater than 1")] int personId, [FromBody] Models.PersonPasswordResetRequest request)
{
    request.UserId = HttpContext.PersonId();
    request.PersonID = personId;

    using (SqlConnection db = new SqlConnection(connectionString))
    {
        if (await db.ExecuteWithRetryAsync("[WebSite].[PersonPasswordResetV1]", param: request, commandType: CommandType.StoredProcedure) != 1)
        {
            logger.LogWarning("Person {0} password change failed", request.PersonID);

            return this.Conflict();
        }
    }

    return this.Ok();
}

For a couple of applications, we have added “Permissions” and “PersonPermissions” tables alongside the existing authorisation functionality to reduce the likely hood of any unintended side effects.

CREATE TABLE [Application].[Permissions](
	[PermissionID] [int] IDENTITY(1,1) NOT NULL,
	[Name] [nvarchar](20) NOT NULL,
	[Description] [nvarchar](50) NOT NULL,
	[LastEditedBy] [int] NOT NULL,
	[ValidFrom] [datetime2](7) GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
	[ValidUntil] [datetime2](7) GENERATED ALWAYS AS ROW END HIDDEN NOT NULL,
 CONSTRAINT [PK_Permissions] PRIMARY KEY CLUSTERED 
(
	[PermissionID] ASC
)...

We try and keep the names of the permissions short, so the token doesn’t get too large.

CREATE TABLE [Application].[PersonPermissions](
	[PersonPermissionId] [int] IDENTITY(1,1) NOT NULL,
	[PersonId] [int] NOT NULL,
	[PermIssionId] [int] NOT NULL,
	[Active] [bit] NOT NULL,
	[ValidFrom] [datetime2](7) GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
	[ValidUntil] [datetime2](7) GENERATED ALWAYS AS ROW END HIDDEN NOT NULL,
 CONSTRAINT [PK_PersonPermissions] PRIMARY KEY CLUSTERED 
(
	[PersonPermissionId] ASC
)...
Permissions, PersonPermissions and People with Foreign Keys

The additional permissions (from the Person record) and the PersonPermissions table required some modifications to the PersonAuthenticateLookupByLogonNameV1 stored procedure and the addition of the PersonPermissionsByPersonIdV1 stored procedure.

ALTER PROCEDURE [Website].[PersonAuthenticateLookupByLogonNameV2]
@LogonName nvarchar(50),
@Password nvarchar(40)
AS
BEGIN
	SELECT PersonID
		,FullName
		,EmailAddress
		,IsSystemUser
		,IsEmployee
		,IsSalesPerson
	FROM [Application].[People]
	WHERE (( LogonName = @LogonName)
		AND (IsPermittedToLogon = 1)
		AND (HASHBYTES(N'SHA2_256', @Password + FullName) = HashedPassword))

The IsSystemUser, IsEmployee and IsSalesPerson bit flags were added to the stored procedure and Data Transfer Object(DTO)

private class PersonAuthenticateLogonDetailsDto
{
    public int PersonID { get; set; }    

    public string FullName { get; set; }

    public string EmailAddress { get; set; }

    public bool IsSystemUser { get; set; }

    public bool IsEmployee { get; set; }

    public bool IsSalesPerson { get; set; }
}

The PersonPermissionsByPersonIdV1 retrieves a list of the permissions of the User who has been authenticated.

ALTER PROCEDURE [Website].[PersonPermissionsByPersonIdV1]
	@PersonId AS int
AS
BEGIN

	SELECT [Application].[Permissions].[Name]
	FROM [Application].[Permissions]
		INNER JOIN [Application].[PersonPermissions] ON ([Application].[Permissions].PermissionID = [Application].[PersonPermissions].[PermissionId] )
	WHERE [Application].[PersonPermissions].[PersonId] = @PersonId
	ORDER BY [Application].[Permissions].[Name]

END

The Person’s permissions(effectively roles) are added as claims, the IsSystemUser, IsEmployee and IsSalesPerson flags are also added to the list of claims so they can be used in the new application.

[HttpPost("logon")]
public async Task<ActionResult> Logon([FromBody] Models.LogonRequest request )
{
    PersonAuthenticateLogonDetailsDto userLogonUserDetails;
    IEnumerable<string> permissions;
    var claims = new List<Claim>();

    using (SqlConnection db = new SqlConnection(configuration.GetConnectionString("WorldWideImportersDatabase")))
    {
        userLogonUserDetails = await db.QuerySingleOrDefaultWithRetryAsync<PersonAuthenticateLogonDetailsDto>("[Website].[PersonAuthenticateLookupByLogonNameV2]", param: request, commandType: CommandType.StoredProcedure);
        if (userLogonUserDetails == null)
        {
            logger.LogWarning("Login attempt by user {0} failed", request.LogonName);

           return this.Unauthorized();
        }

        // Lookup the Person's permissions
        permissions = await db.QueryWithRetryAsync<string>("[Website].[PersonPermissionsByPersonIdV1]", new { userLogonUserDetails.PersonID }, commandType: CommandType.StoredProcedure);
    }

    // Setup the primary SID + name info
    claims.Add(new Claim(ClaimTypes.PrimarySid, userLogonUserDetails.PersonID.ToString()));
    if (userLogonUserDetails.IsSystemUser)
    {
       claims.Add(new Claim(ClaimTypes.Role, "SystemUser"));
    }
    if (userLogonUserDetails.IsEmployee)
    {
       claims.Add(new Claim(ClaimTypes.Role, "Employee"));
    }
    if (userLogonUserDetails.IsSalesPerson)
    {
        claims.Add(new Claim(ClaimTypes.Role, "SalesPerson"));
    }

    foreach(string permission in permissions)
    {
        claims.Add(new Claim(ClaimTypes.Role, permission));
    }

    var authSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(jwtIssuerOptions.SecretKey));

    var token = new JwtSecurityToken(
         issuer: jwtIssuerOptions.Issuer,
          audience: jwtIssuerOptions.Audience, 
          expires: DateTime.UtcNow.Add(jwtIssuerOptions.TokenExpiresAfter),
          claims: claims,
          signingCredentials: new SigningCredentials(authSigningKey, SecurityAlgorithms.HmacSha256));
                
    return this.Ok(new
   {
        token = new JwtSecurityTokenHandler().WriteToken(token),
        expiration = token.ValidTo,
    });
}

We try to reduce the number of roles a User requires by having core roles (Administrator, Sales consultant, Warehouse administrator etc.) with additional roles for each task that can be added as required (ResetPassword, CustomerIsOnCreditHold Set/Clear etc.)

The Get Method of Authorisation controller returns a list of the User’s Roles which can be used to enable/disable functionality of the user interface.

/// <summary>
/// Gets a list of the current User's roles.
/// </summary>
/// <response code="200">List of claims returned.</response>
/// <response code="401">Unauthorised, bearer token missing or expired.</response>
/// <returns>list of claims.</returns>
[HttpGet]
[ProducesResponseType(StatusCodes.Status200OK, Type = typeof(List<string>))]
public List<string> Get()
{
    List<string> claimNames = new List<string>();

    foreach (var claim in this.User.Claims.Where(c => c.Type == ClaimTypes.Role))
    {
        claimNames.Add(claim.Value);
    }

    return claimNames;
}

We have found this approach to be a robust way to add granular authorisation for new functionality to a “legacy’ application without breaking the existing solution. I have ignored a user being disabled after a number of failed logins, password complexity rules etc. as these tend to be application specific and not really related to the use of Dapper.

Most blog posts talk about building “green fields” applications, I have found hardly any cover “muddy fields” development where you have to deal with “legacy” code.

Not all “legacy” code is bad, I work on one code base which is nearly 20years old. It started as a spreadsheet plug-in and has grown of time to a SaaS application. There is very little of the original code left it has just been carefully re-factored over the years with time allocated to chip away at technical debt.

.NET Core web API + Dapper – Authentication

A couple of recent contracts have been maintaining and “remediating” legacy codebases which have been in production for upto a decade. The applications are delivering business value (can’t stop working) and the customer’s budgets are limited (they can only afford incremental change). As a result of this we end up making tactical decisions to keep the application working and longer-term ones to improve the “ilities“(taking into account the customer’s priorities).

It is rare to have a “green fields” project, so my plan was to use the next couple of WebAPI + Dapper posts to illustrate the sort of challenges we have encountered.

Customer: “we want to put a nice webby frontend on our existing bespoke solution” (“putting lipstick on a pig”).

Customer: The user’s login details are stored in the database and we can’t change the login process as the help desk won’t cope.

The remediation of Authentication and Authorisation(A&A) functionality can be particularly painful and is often driven by compliance issues e.g. EU GDPR, The Privacy Act 202 etc.

The World Wide Importers database would be a representative example of databases we have worked with.

CREATE TABLE [Application].[People](
	[PersonID] [int] NOT NULL,
	[FullName] [nvarchar](50) NOT NULL,
	[PreferredName] [nvarchar](50) NOT NULL,
	[SearchName]  AS (concat([PreferredName],N' ',[FullName])) PERSISTED NOT NULL,
	[IsPermittedToLogon] [bit] NOT NULL,
	[LogonName] [nvarchar](50) NULL,
	[IsExternalLogonProvider] [bit] NOT NULL,
	[HashedPassword] [varbinary](max) NULL,
	[IsSystemUser] [bit] NOT NULL,
	[IsEmployee] [bit] NOT NULL,
	[IsSalesperson] [bit] NOT NULL,
	[UserPreferences] [nvarchar](max) NULL,
	[PhoneNumber] [nvarchar](20) NULL,
	[FaxNumber] [nvarchar](20) NULL,
	[EmailAddress] [nvarchar](256) NULL,
	[Photo] [varbinary](max) NULL,
	[CustomFields] [nvarchar](max) NULL,
	[OtherLanguages]  AS (json_query([CustomFields],N'$.OtherLanguages')),
	[LastEditedBy] [int] NOT NULL,
	[ValidFrom] [datetime2](7) GENERATED ALWAYS AS ROW START NOT NULL,
	[ValidTo] [datetime2](7) GENERATED ALWAYS AS ROW END NOT NULL,
 CONSTRAINT [PK_Application_People] PRIMARY KEY CLUSTERED 
(
	[PersonID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [USERDATA],
	PERIOD FOR SYSTEM_TIME ([ValidFrom], [ValidTo])
) ON [USERDATA] TEXTIMAGE_ON [USERDATA]
WITH
(
SYSTEM_VERSIONING = ON (HISTORY_TABLE = [Application].[People_Archive])
)

Initial observations

  • The LogonName column doesn’t have an index and uniqueness is not enforced which is a bit odd.
  • The table has SYSTEM_VERSIONING enabled so any structural changes are going to be hard work.
  • There are a couple of computed columns, so we need to be careful with any changes.
  • A password hash is a varbinary max column, so we need to figure out how this is generated and updated
  • Surprising number of nullable columns
  • The code associated with IsExternalLogonProvider needs to be investigated.
  • Looks like the granularity of permissions i.e. IsSystemUser, IsEmployee, IsSalesperson is low.
  • The database must be old there is a FaxNumber column.
  • Looks like there are internal and external (maybe IsSystemUser, IsEmployee, IsSalesperson are all false) people.
  • The PersonId is a Sequence rather than an Identity column which is unusual.
  • IsPermittedToLogin indicates that login process might be a bit more complex than expected
  • The terms Login and Logon appear to be used interchangeably.
  • No lockout after several failed logon attempts, lockout until etc. functionality.
  • No concurrency control (optimistic or pessimistic) for updates (with TimeStamp or Version column) so last update wins.

The next step would be to have a look at the contents of the People table with SQL Server Management Studio(SSMS)

SQL Server Management Studio [Application].[Person] table

Initial observations

  • Looks like the system administrators are in the first couple of rows. (makes Indirect Object Reference Attack easier).
  • Lots of NULL values which often makes application code more complex
  • Duplicates in LoginName column e.g. “NO LOGON”
  • Some “magic” values e.g. “NO LOGON”
  • Why does the “Data Conversion Only” person have a photo?
  • The IsExternalLogonProvider is always false.
  • The UserPreferences, CustomFields and OtherLanguages columns contain Java Script Object Notation(JSON), need to see how these a generated, updated and used.

The application must have an existing external user provisioning process. It looks like a Person record is created by an Administrator then the User sets their password on first Logon. (not certain if there are any password complexity rules enforced)

An application I worked on didn’t have any enforcement of password complexity and minimum length in earlier versions. This caused issues when a number of their clients couldn’t logon to the new application because their existing password was too short. We updated the logon field rules and retained minimum complexity and length rules on change and reset password field validation. We then forced 5-10% of users per month (so the helpdesk wasn’t overwhelmed by support calls) to update their passwords.

ALTER PROCEDURE [Website].[ActivateWebsiteLogon]
@PersonID int,
@LogonName nvarchar(50),
@InitialPassword nvarchar(40)
WITH EXECUTE AS OWNER
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;

    UPDATE [Application].People
    SET IsPermittedToLogon = 1,
        LogonName = @LogonName,
        HashedPassword = HASHBYTES(N'SHA2_256', @InitialPassword + FullName),
        UserPreferences = (SELECT UserPreferences FROM [Application].People WHERE PersonID = 1) -- Person 1 has User Preferences template
    WHERE PersonID = @PersonID
    AND PersonID <> 1
    AND IsPermittedToLogon = 0;

    IF @@ROWCOUNT = 0
    BEGIN
        PRINT N'The PersonID must be valid, must not be person 1, and must not already be enabled';
        THROW 51000, N'Invalid PersonID', 1;
        RETURN -1;
    END;
END;

Initial observations

  • The password hashing uses SHA2_256 which is good
  • The password hash is seeded with the persons FullName, this could be a problem if a user changes their name.
  • The user selects their LoginName, this needs further investigate as duplicates could be an issue
  • XACT_ABORT ON + THROW for validation and state management is odd, need to check how SqlExceptions are handled in application code.
  • PersonID magic number handling adds complexity and needs further investigation.
  • EXECUTE AS OWNER caught my attention, checked only one Database user for application.
  • The LastEditBy isn’t set to the PersonID which seems a bit odd.

Based on the [Website].[ActivateWebsiteLogon] (couldn’t find Logon so reviewed ChangePassword) this is my first attempt at a stored procedure which validates a user’s LogonName and Password.

CREATE PROCEDURE [Application].[PersonAuthenticateLookupByLogonNameV1]
@LogonName nvarchar(50),
@Password nvarchar(40)
AS
BEGIN
	SELECT PersonID, FullName, EmailAddress
	FROM [Application].[People]
	WHERE (( LogonName = @LogonName)
		AND (IsPermittedToLogon = 1)
		AND (HASHBYTES(N'SHA2_256', @Password + FullName) = HashedPassword))
END
GO

We use the ..VX approach to reduce issues when doing canary and rolling deployments. If the parameter list or return dataset of a stored procedure changes (even if we think it is backwards compatible) the version number is incremented so that different versions of the application can be run concurrently and backing out application updates is less fraught.

The next step was to create a Data Transfer Object(DTO) for the Logon request payload

public class LogonRequest
{
    [JsonRequired]
    [MinLength(Constants.LogonNameLengthMinimum)]
    [MaxLength(Constants.LogonNameLengthMaximum)]
    public string LogonName { get; set; }

    [JsonRequired]
    [MinLength(Constants.PasswordLengthMinimum)]
    [MaxLength(Constants.PasswordLengthMaximum)]
    public string Password { get; set; }
}

Then a Proof of Concept (PoC) AuthenticationController to process a login request.

Swagger Docs Authentication controller Login

The Logon method calls the PersonAuthenticateLookupByLogonNameV1 stored procedure to validate the LoginName and password. In this iteration the only claim added to the JSON Web Token(JWT) is the PersonId. We try and keep the JWTs small as possible as one customer’s application failed randomly because a couple of user’s JWTs were so large (lots of roles) that some versions of browsers choked.

public AuthenticationController(IConfiguration configuration, ILogger<AuthenticationController> logger, IOptions<Model.JwtIssuerOptions> jwtIssuerOptions)
{
    this.configuration = configuration;
    this.logger = logger;
    this.jwtIssuerOptions = jwtIssuerOptions.Value;
 }

[HttpPost("login")]
public async Task<ActionResult> Login([FromBody] Model.LoginRequest request )
{
    var claims = new List<Claim>();

    using (SqlConnection db = new SqlConnection(configuration.GetConnectionString("WorldWideImportersDatabase")))
    {
        UserLogonUserDetailsDto userLogonUserDetails = await db.QuerySingleOrDefaultWithRetryAsync<UserLogonUserDetailsDto>("[Application].[PersonAuthenticateLookupByLogonNameV1]", param: request, commandType: CommandType.StoredProcedure);
        if (userLogonUserDetails == null)
        {
             logger.LogWarning("Login attempt by user {0} failed", request.LogonName);

             return this.Unauthorized();
        }

        // Setup the primary SID + name info
        claims.Add(new Claim(ClaimTypes.PrimarySid, userLogonUserDetails.PersonID.ToString()));
    }

    var authSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(jwtIssuerOptions.SecretKey));

    var token = new JwtSecurityToken(
              issuer: jwtIssuerOptions.Issuer,
              audience: jwtIssuerOptions.Audience, 
              expires: DateTime.UtcNow.Add(jwtIssuerOptions.TokenExpiresAfter),
              claims: claims,
              signingCredentials: new SigningCredentials(authSigningKey, SecurityAlgorithms.HmacSha256));
                
        return this.Ok(new
        {
            token = new JwtSecurityTokenHandler().WriteToken(token),
            expiration = token.ValidTo,
        });
    }
}

After a successful logon the Token has to be copied (I regularly miss the first or the last character) from the response payload to the Authorisation form.

The Swagger UI Authentication method after a successful Logon with the bearer token highlighted

I decorated the SystemController DeploymentVersion deployment with the [Authorize] attribute to force a check that the user is authenticated.

/// <summary>
/// WebAPI controller for handling System Dapper functionality.
/// </summary>
[Route("api/[controller]")]
[ApiController]
public class SystemController : ControllerBase
{
    /// <summary>
    /// Returns the Application version in [Major].[Minor].[Build].Revision] format.
    /// </summary>
    /// <response code="200">List of claims returned.</response>
    /// <response code="401">Unauthorised, bearer token missing or expired.</response>
    /// <returns>Returns the Application version in [Major].[Minor].[Build].Revision] format.</returns>
    [HttpGet("DeploymentVersion"), Authorize]
    public string DeploymentVersion()
    {
        return Assembly.GetExecutingAssembly().GetName().Version.ToString();
    }
}

If the bearer token is missing, invalid (I accidentally didn’t copy either the first or last character) or expired the method call will fail with a 401 Unauthorized error.

Swagger UI System Controller DeploymentVersion method failing because the JWT is missing, invalid or expired

Controlling access to controllers and methods of controllers is probably not granular for most applications so adding “coarse” and “fine grained” authorisation to an existing application and the configuration of Swashbuckle and application request processing middleware to support JWTs will be covered in a couple of future posts.

For remediation projects we try to keep the code as simple as possible (but no simpler), by minimising the plumbing and only using advanced language features etc. where it adds value.

I’m leaning towards using Dependency Injection for configuration information, so the way connection strings and jwtIssuerOptions is going to be harmonised. In a future version of the application the jwtIssuerOptions will be migrated to an Azure Key Vault.

.NET nanoFramework RAK11200 – Azure IoT Hub HTTP SAS Tokens – Revisited

Several times my client apps inspired by Azure IoT Hub HTTP Basic have not worked because I have failed to correctly trim the Azure IoT Hub Shared Access Signature(SAS) generated with tools like Azure Command Line az iot hub generate-sas-token, Azure IoT Tools for Visual Studio Code or Azure IoT Explorer.

The tokens are quite long but “the only “important” parts are the resource(sr), signature(sig) and expiry(se) values. If the connection string is generated

HostName=01234567890123456789.azure-devices.net;DeviceId=RAK11200-RAK19001;SharedAccessSignature=SharedAccessSignature sr=01234567890123456789.azure-devices.net%2Fdevices%2FRAK11200-RAK19001&sig=ABCDEFGHIJLMNOPQRSTUVWXYZ1234567890abcdefghijklmnopqrs&se=1663810576

The final version of the application constructs the Azure IoT Hub Shared Access Signature(SAS) with the AzureIoTHubHostName, DeviceID, signature(sig) & expiry(se) values in the config.cs file.

public class Config
{
   public const string DeviceID = "RAK11200-RAK19001";
   public const string SasSignature = "..."; // sig
   public const string SasExpiryTime = "..."; // se

   public const string AzureIoTHubHostName = "..";
   public const string Ssid = "...";
   public const string Password = "..";
   ...
}
 _httpClient = new HttpClient
{
   SslProtocols = System.Net.Security.SslProtocols.Tls12,
   HttpsAuthentCert = new X509Certificate(Config.DigiCertBaltimoreCyberTrustRoot),
   BaseAddress = new Uri($"https://{Config.AzureIoTHubHostName}.azure-devices.net/devices/{Config.DeviceID}/messages/events?api-version=2020-03-13"),
};

string sasKey = $"SharedAccessSignature sr={Config.AzureIoTHubHostName}.azure-devices.net%2Fdevices%2F{Config.DeviceID}&sig={Config.SasSignature}&se={Config.SasExpiryTime}";

_httpClient.DefaultRequestHeaders.Add("Authorization", sasKey);

.NET nanoFramework RAK11200 – Azure IoT Hub HTTP SAS Tokens

This is the simplest .NET nanoFramework Azure IoT Hub client I could come up with (inspired by this nanoFramework sample).

My test setup was a RAKwireless RAK11200 WisBlock WiFi Module, RAK5005 WisBlock Base Board or RAK19001 WisBlock Dual IO Base Board and RAK1901 WisBlock Temperature and Humidity Sensor

RAK112000+RAK5005-O+RAK1901 Test rig
RAK112000+RAK19001+RAK1901 Test rig

I used a RAK1901 WisBlock Temperature and Humidity Sensor because it has nanoFramework.IoTDevice library support

public class Program
{
    private static TimeSpan SensorUpdatePeriod = new TimeSpan(0, 30, 0);

    private static HttpClient _httpClient;

    public static void Main()
    {
        Debug.WriteLine("devMobile.IoT.RAK.Wisblock.AzureIoHub.RAK1901 starting");

        Configuration.SetPinFunction(Gpio.IO04, DeviceFunction.I2C1_DATA);
        Configuration.SetPinFunction(Gpio.IO05, DeviceFunction.I2C1_CLOCK);

        if (!WifiNetworkHelper.ConnectDhcp(Config.Ssid, Config.Password, requiresDateTime: true))
        {
            if (NetworkHelper.HelperException != null)
            {
                Debug.WriteLine($"WifiNetworkHelper.ConnectDhcp failed {NetworkHelper.HelperException}");
            }

            Thread.Sleep(Timeout.Infinite);
        }

        _httpClient = new HttpClient
        {
            SslProtocols = System.Net.Security.SslProtocols.Tls12,
            HttpsAuthentCert = new X509Certificate(Config.DigiCertBaltimoreCyberTrustRoot),
            BaseAddress = new Uri($"https://{Config.AzureIoTHubHostName}.azure-devices.net/devices/{Config.DeviceID}/messages/events?api-version=2020-03-13"),
        };
        _httpClient.DefaultRequestHeaders.Add("Authorization", Config.SasKey);

        I2cConnectionSettings settings = new(1, Shtc3.DefaultI2cAddress);
        I2cDevice device = I2cDevice.Create(settings);
        Shtc3 shtc3 = new(device);

        while (true)
        {
            if (shtc3.TryGetTemperatureAndHumidity(out var temperature, out var relativeHumidity))
            {
                Debug.WriteLine($"Temperature {temperature.DegreesCelsius:F1}°C  Humidity {relativeHumidity.Value:F0}%");

                string payload = $"{{\"RelativeHumidity\":{relativeHumidity.Value:F0},\"Temperature\":{temperature.DegreesCelsius.ToString("F1")}}}";

                try
                {
                    using (HttpContent content = new StringContent(payload))
                    using (HttpResponseMessage response = _httpClient.Post("", content))
                    {
                        Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} Response code:{response.StatusCode}");

                        response.EnsureSuccessStatusCode();
                    }
                }
                catch(Exception ex)
                {
                    Debug.WriteLine($"Azure IoT Hub POST failed:{ex.Message}");
                }
            }

            Thread.Sleep(SensorUpdatePeriod);
        }
    }
}

I generated the Azure IoT Hub Shared Access Signature(SAS) Tokens (10800 minutes is 1 week) with Azure IoT Explorer (Trim the SAS key so it starts with SharedAccessSignature sr=….)

Azure IoT Explorer SAS Token Generation

I was using Azure IoT Explorer to monitor the telemetry and found that the initial versions of the application would fail after 6 or 7 hours. After reviewing the code I added a couple of “using” statements which appear to have fixed the problem as the soak test has been running for 12hrs, 24hrs, 36hrs, 48hrs, 96hrs

Smartish Edge Camera – Azure IoT Updateable Properties (not persisted)

This post builds on my Smartish Edge Camera -Azure IoT Direct Methods post adding two updateable properties for the image capture and processing timer the due and period values. The two properties can be updated together or independently but the values are not persisted.

When I was searching for answers I found this code in many posts and articles but it didn’t really cover my scenario.

private static async Task OnDesiredPropertyChanged(TwinCollection desiredProperties, 
  object userContext)
{
   Console.WriteLine("desired property chPleange:");
   Console.WriteLine(JsonConvert.SerializeObject(desiredProperties));
   Console.WriteLine("Sending current time as reported property");
   TwinCollection reportedProperties = new TwinCollection
   {
       ["DateTimeLastDesiredPropertyChangeReceived"] = DateTime.Now
   };

    await Client.UpdateReportedPropertiesAsync(reportedProperties).ConfigureAwait(false);
}

When AZURE_DEVICE_PROPERTIES is defined in the SmartEdgeCameraAzureIoTService project properties the device reports a number of properties on startup and SetDesiredPropertyUpdateCallbackAsync is used to configure the method called whenever the client receives a state update(desired or reported) from the Azure IoT Hub.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
	_logger.LogInformation("Azure IoT Smart Edge Camera Service starting");

	try
	{
#if AZURE_IOT_HUB_CONNECTION
		_deviceClient = await AzureIoTHubConnection();
#endif
#if AZURE_IOT_HUB_DPS_CONNECTION
		_deviceClient = await AzureIoTHubDpsConnection();
#endif

#if AZURE_DEVICE_PROPERTIES
		_logger.LogTrace("ReportedPropeties upload start");

		TwinCollection reportedProperties = new TwinCollection();

		reportedProperties["OSVersion"] = Environment.OSVersion.VersionString;
		reportedProperties["MachineName"] = Environment.MachineName;
		reportedProperties["ApplicationVersion"] = Assembly.GetAssembly(typeof(Program)).GetName().Version;
		reportedProperties["ImageTimerDue"] = _applicationSettings.ImageTimerDue;
		reportedProperties["ImageTimerPeriod"] = _applicationSettings.ImageTimerPeriod;
		reportedProperties["YoloV5ModelPath"] = _applicationSettings.YoloV5ModelPath;

		reportedProperties["PredictionScoreThreshold"] = _applicationSettings.PredictionScoreThreshold;
		reportedProperties["PredictionLabelsOfInterest"] = _applicationSettings.PredictionLabelsOfInterest;
		reportedProperties["PredictionLabelsMinimum"] = _applicationSettings.PredictionLabelsMinimum;

		await _deviceClient.UpdateReportedPropertiesAsync(reportedProperties, stoppingToken);

		_logger.LogTrace("ReportedPropeties upload done");
#endif

		_logger.LogTrace("YoloV5 model setup start");
		_scorer = new YoloScorer<YoloCocoP5Model>(_applicationSettings.YoloV5ModelPath);
		_logger.LogTrace("YoloV5 model setup done");

		_ImageUpdatetimer = new Timer(ImageUpdateTimerCallback, null, _applicationSettings.ImageTimerDue, _applicationSettings.ImageTimerPeriod);

		await _deviceClient.SetMethodHandlerAsync("ImageTimerStart", ImageTimerStartHandler, null);
		await _deviceClient.SetMethodHandlerAsync("ImageTimerStop", ImageTimerStopHandler, null);
		await _deviceClient.SetMethodDefaultHandlerAsync(DefaultHandler, null);

		await _deviceClient.SetDesiredPropertyUpdateCallbackAsync(OnDesiredPropertyChangedAsync, null);

		try
		{
			await Task.Delay(Timeout.Infinite, stoppingToken);
		}
		catch (TaskCanceledException)
		{
			_logger.LogInformation("Application shutown requested");
		}
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "Application startup failure");
	}
	finally
	{
		_deviceClient?.Dispose();
	}

	_logger.LogInformation("Azure IoT Smart Edge Camera Service shutdown");
}

// Lots of other code here

private async Task OnDesiredPropertyChangedAsync(TwinCollection desiredProperties, object userContext)
{
	TwinCollection reportedProperties = new TwinCollection();

	_logger.LogInformation("OnDesiredPropertyChanged handler");

	// NB- This approach does not save the ImageTimerDue or ImageTimerPeriod, a stop/start with return to appsettings.json configuration values. If only
	// one parameter specified other is default from appsettings.json. If timer settings changed I think they won't take
	// effect until next time Timer fires.

	try
	{
		// Check to see if either of ImageTimerDue or ImageTimerPeriod has changed
		if (!desiredProperties.Contains("ImageTimerDue") && !desiredProperties.Contains("ImageTimerPeriod"))
		{
			_logger.LogInformation("OnDesiredPropertyChanged neither ImageTimerDue or ImageTimerPeriod present");
			return;
		}

		TimeSpan imageTimerDue = _applicationSettings.ImageTimerDue;

		// Check that format of ImageTimerDue valid if present
		if (desiredProperties.Contains("ImageTimerDue"))
		{
			if (TimeSpan.TryParse(desiredProperties["ImageTimerDue"].Value, out imageTimerDue))
			{
				reportedProperties["ImageTimerDue"] = imageTimerDue;
			}
			else
			{
				_logger.LogInformation("OnDesiredPropertyChanged ImageTimerDue invalid");
				return;
			}
		}

		TimeSpan imageTimerPeriod = _applicationSettings.ImageTimerPeriod;

		// Check that format of ImageTimerPeriod valid if present
		if (desiredProperties.Contains("ImageTimerPeriod"))
		{
			if (TimeSpan.TryParse(desiredProperties["ImageTimerPeriod"].Value, out imageTimerPeriod))
			{
				reportedProperties["ImageTimerPeriod"] = imageTimerPeriod;
			}
			else
			{
				_logger.LogInformation("OnDesiredPropertyChanged ImageTimerPeriod invalid");
				return;
			}
		}

		_logger.LogInformation("Desired Due:{0} Period:{1}", imageTimerDue, imageTimerPeriod);

		if (!_ImageUpdatetimer.Change(imageTimerDue, imageTimerPeriod))
		{
			_logger.LogInformation("Desired Due:{0} Period:{1} failed", imageTimerDue, imageTimerPeriod);
		}

		await _deviceClient.UpdateReportedPropertiesAsync(reportedProperties);
	}
	catch (Exception ex)
	{
		_logger.LogError(ex, "OnDesiredPropertyChangedAsync handler failed");
	}
}

The TwinCollection desiredProperties is checked for ImageTimerDue and ImageTimerPeriod properties and if either of these are present and valid the Timer.Change method is called.

The AzureMLMetSmartEdgeCamera supports both Azure IoT Hub and Azure IoT Central so I have included images from Azure IoT Explorer and my Azure IoT Central Templates.

SmartEdge Camera Device Twin properties in Azure IoT Explorer

When I modified, then saved the Azure IoT Hub Device Twin desired properties JavaScript Object Notation(JSON) in Azure IoT Hub Explorer the method configured with SetDesiredPropertyUpdateCallbackAsync was invoked on the device.

In Azure IoT Central I added two Capabilities to the device template, the time properties ImageTimerDue, and ImageTimerPeriod.

Azure IoT Central SmartEdgeCamera Device template capabilities

I added a View to the template so the two properties could be changed (I didn’t configure either as required)

Azure IoT Central SmartEdgeCamera Device Default view designer

In the “Device Properties”, “Operation Tab” when I changed the ImageTimerDue and/or ImageTimerPeriod there was visual feedback that there was an update in progress.

Azure IoT Central SmartEdgeCamera Device Properties update start

Then on the device the SmartEdgeCameraAzureIoTService the method configured with SetDesiredPropertyUpdateCallbackAsync was invoked on the device.

SmartEdge Camera Console application displaying updated properties

Once the properties have been updated on the device the UpdateReportedPropertiesAsync method is called

Then a message with the updated property values from the device was visible in the telemetry

Azure IoT Central SmartEdgeCamera Device Properties update done

Then finally the “Operation Tab” displayed a visual confirmation that the value(s) had been updated.