libcamera-jpeg on Raspberry Pi OS Bullseye Duration

The image capture process was taking about 5 seconds which a bit longer than I was expecting.

libcamera-jpeg -o rotated.jpg --rotation 180

The libcamera-jpeg program has a lot of command line parameters.

pi@raspberrypi4a:~ $ libcamera-jpeg --help
Valid options are:
  -h [ --help ] [=arg(=1)] (=0)         Print this help message
  --version [=arg(=1)] (=0)             Displays the build version number
  -v [ --verbose ] [=arg(=1)] (=0)      Output extra debug and diagnostics
  -c [ --config ] [=arg(=config.txt)]   Read the options from a file. If no filename is specified, default to
                                        config.txt. In case of duplicate options, the ones provided on the command line
                                        will be used. Note that the config file must only contain the long form
                                        options.
  --info-text arg (=#%frame (%fps fps) exp %exp ag %ag dg %dg)
                                        Sets the information string on the titlebar. Available values:
                                        %frame (frame number)
                                        %fps (framerate)
                                        %exp (shutter speed)
                                        %ag (analogue gain)
                                        %dg (digital gain)
                                        %rg (red colour gain)
                                        %bg (blue colour gain)
                                        %focus (focus FoM value)
                                        %aelock (AE locked status)
  --width arg (=0)                      Set the output image width (0 = use default value)
  --height arg (=0)                     Set the output image height (0 = use default value)
  -t [ --timeout ] arg (=5000)          Time (in ms) for which program runs
  -o [ --output ] arg                   Set the output file name
  --post-process-file arg               Set the file name for configuring the post-processing
  --rawfull [=arg(=1)] (=0)             Force use of full resolution raw frames
  -n [ --nopreview ] [=arg(=1)] (=0)    Do not show a preview window
  -p [ --preview ] arg (=0,0,0,0)       Set the preview window dimensions, given as x,y,width,height e.g. 0,0,640,480
  -f [ --fullscreen ] [=arg(=1)] (=0)   Use a fullscreen preview window
  --qt-preview [=arg(=1)] (=0)          Use Qt-based preview window (WARNING: causes heavy CPU load, fullscreen not
                                        supported)
  --hflip [=arg(=1)] (=0)               Request a horizontal flip transform
  --vflip [=arg(=1)] (=0)               Request a vertical flip transform
  --rotation arg (=0)                   Request an image rotation, 0 or 180
  --roi arg (=0,0,0,0)                  Set region of interest (digital zoom) e.g. 0.25,0.25,0.5,0.5
  --shutter arg (=0)                    Set a fixed shutter speed
  --analoggain arg (=0)                 Set a fixed gain value (synonym for 'gain' option)
  --gain arg                            Set a fixed gain value
  --metering arg (=centre)              Set the metering mode (centre, spot, average, custom)
  --exposure arg (=normal)              Set the exposure mode (normal, sport)
  --ev arg (=0)                         Set the EV exposure compensation, where 0 = no change
  --awb arg (=auto)                     Set the AWB mode (auto, incandescent, tungsten, fluorescent, indoor, daylight,
                                        cloudy, custom)
  --awbgains arg (=0,0)                 Set explict red and blue gains (disable the automatic AWB algorithm)
  --flush [=arg(=1)] (=0)               Flush output data as soon as possible
  --wrap arg (=0)                       When writing multiple output files, reset the counter when it reaches this
                                        number
  --brightness arg (=0)                 Adjust the brightness of the output images, in the range -1.0 to 1.0
  --contrast arg (=1)                   Adjust the contrast of the output image, where 1.0 = normal contrast
  --saturation arg (=1)                 Adjust the colour saturation of the output, where 1.0 = normal and 0.0 =
                                        greyscale
  --sharpness arg (=1)                  Adjust the sharpness of the output image, where 1.0 = normal sharpening
  --framerate arg (=30)                 Set the fixed framerate for preview and video modes
  --denoise arg (=auto)                 Sets the Denoise operating mode: auto, off, cdn_off, cdn_fast, cdn_hq
  --viewfinder-width arg (=0)           Width of viewfinder frames from the camera (distinct from the preview window
                                        size
  --viewfinder-height arg (=0)          Height of viewfinder frames from the camera (distinct from the preview window
                                        size)
  --tuning-file arg (=-)                Name of camera tuning file to use, omit this option for libcamera default
                                        behaviour
  --lores-width arg (=0)                Width of low resolution frames (use 0 to omit low resolution stream
  --lores-height arg (=0)               Height of low resolution frames (use 0 to omit low resolution stream
  -q [ --quality ] arg (=93)            Set the JPEG quality parameter
  -x [ --exif ] arg                     Add these extra EXIF tags to the output file
  --timelapse arg (=0)                  Time interval (in ms) between timelapse captures
  --framestart arg (=0)                 Initial frame counter value for timelapse captures
  --datetime [=arg(=1)] (=0)            Use date format for output file names
  --timestamp [=arg(=1)] (=0)           Use system timestamps for output file names
  --restart arg (=0)                    Set JPEG restart interval
  -k [ --keypress ] [=arg(=1)] (=0)     Perform capture when ENTER pressed
  -s [ --signal ] [=arg(=1)] (=0)       Perform capture when signal received
  --thumb arg (=320:240:70)             Set thumbnail parameters as width:height:quality
  -e [ --encoding ] arg (=jpg)          Set the desired output encoding, either jpg, png, rgb, bmp or yuv420
  -r [ --raw ] [=arg(=1)] (=0)          Also save raw file in DNG format
  --latest arg                          Create a symbolic link with this name to most recent saved file
  --immediate [=arg(=1)] (=0)           Perform first capture immediately, with no preview phase
pi@raspberrypi4a:~ $

My libcamera-jpeg application is run “headless” so I tried turning off the image preview functionality.

libcamera-jpeg -o rotatednopreview.jpg --nopreview

When I ran libcamera-jpeg in a console windows or my application this didn’t appear to make any noticeable difference.

libcamera-jpeg run from the command line with –nopreview

libcamera-jpeg run by my application with –nopreview

I then had another look at the libcamera-jpeg command line parameters to see if any looked useful for reducing the time that it took to take a save an image and this one caught my attention.

I had assumed the delay was related to how long the preview window was displayed.

libcamera-jpeg run from the command line with –nopreview –t1

I modified the application (V5) then ran it from the command line and the time reduced to less than a second.

private static void ImageUpdateTimerCallback(object state)
{
	try
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image update start");

		// Just incase - stop code being called while photo already in progress
		if (_cameraBusy)
		{
			return;
		}

		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image capture start");

		using (Process process = new Process())
		{
			process.StartInfo.FileName = @"libcamera-jpeg";
			// V1 it works
			//process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal}";
			// V3a Image right way up
			//process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal} --vflip --hflip";
			// V3b Image right way up
			//process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal} --rotation 180";
			// V4 Image no preview
			//process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal} --rotation 180 --nopreview";
			// V5 Image no preview, no timeout
			process.StartInfo.Arguments = $"-o {_applicationSettings.ImageFilenameLocal} --nopreview -t1 --rotation 180";
			//process.StartInfo.RedirectStandardOutput = true;
			// V2 No diagnostics
			process.StartInfo.RedirectStandardError = true;
			//process.StartInfo.UseShellExecute = false;
			//process.StartInfo.CreateNoWindow = true; 

			process.Start();

			if (!process.WaitForExit(10000) || (process.ExitCode != 0))
			{
				Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image update failure {process.ExitCode}");
			}
		}

		Console.WriteLine($" {DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image capture done");
	}
	catch (Exception ex)
	{
		Console.WriteLine($"{DateTime.UtcNow:yy-MM-dd HH:mm:ss} Image update error {ex.Message}");
	}
	finally
	{
		_cameraBusy = false;
	}
}
libcamera-jpeg run by my application with –nopreview -t1

The image capture process now takes less that a second which is much better (but not a lot less than retrieving an image from one of my security cameras).

Security Camera ONVIF Discovery

The ONVIF specification standardises the network interface (the network layer) of network video products. It defines a communication framework based on relevant IETF and Web Services standards including security and IP configuration requirements. ONVIF uses Web Services Dynamic Discovery (WS-Discovery) to locate devices on the local network which operates over UDP port 3702 and uses IP multicast address 239.255.255.250.

The first issue was that WS-Discovery is not currently supported by the .Net Core Windows Communication Foundation(WCF) implementation CoreWCF(2021-08). So I built a proof of concept(PoC) client which used UDP to send and receive XML messages (WS-Discovery specification) to “probe” the local network.

My .Net Core 5 console application enumerates the host device’s network interfaces, then sends a “probe” message and waits for responses. The ONVID application programmers guide specifies the format of the “probe” request and response messages (One of the namespace prefixes in the sample is wrong). The client device can return its name and details of it’s capabilities in the response. Currently I only need the IP addresses of the cameras but if more information was required I would use the XML Serialisation functionality of .Net Core to generate the requests and unpack the responses.

class Program
{
	// From https://specs.xmlsoap.org/ws/2005/04/discovery/ws-discovery.pdf & http://www.onvif.org/wp-content/uploads/2016/12/ONVIF_WG-APG-Application_Programmers_Guide-1.pdf
	const string WSDiscoveryProbeMessages =
		"<?xml version = \"1.0\" encoding=\"UTF-8\"?>" +
		"<e:Envelope xmlns:e=\"http://www.w3.org/2003/05/soap-envelope\" " +
			"xmlns:w=\"http://schemas.xmlsoap.org/ws/2004/08/addressing\" " +
			"xmlns:d=\"http://schemas.xmlsoap.org/ws/2005/04/discovery\" " +
			"xmlns:dn=\"http://www.onvif.org/ver10/network/wsdl\"> " +
				"<e:Header>" +
					"<w:MessageID>uuid:{0}</w:MessageID>" +
					"<w:To e:mustUnderstand=\"true\">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To> " +
					"<w:Action mustUnderstand=\"true\">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action> " +
				"</e:Header> " +
				"<e:Body> " +
					"<d:Probe> " +
						"<d:Types>dn:NetworkVideoTransmitter</d:Types>" +
					"</d:Probe> " +
				"</e:Body> " +
		"</e:Envelope>";

	static async Task Main(string[] args)
	{
		List<UdpClient> udpClients = new List<UdpClient>();

		foreach (var networkInterface in NetworkInterface.GetAllNetworkInterfaces())
		{
			Console.WriteLine($"Name {networkInterface.Name}");
			foreach (var unicastAddress in networkInterface.GetIPProperties().UnicastAddresses)
			{
				if (unicastAddress.Address.AddressFamily == AddressFamily.InterNetwork)
				{
					var udpClient = new UdpClient(new IPEndPoint(unicastAddress.Address, 0)) { EnableBroadcast = true };

					udpClient.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveTimeout, 5000);

					udpClients.Add(udpClient);
				}
			}
		}

	var multicastEndpoint = new IPEndPoint(IPAddress.Parse("239.255.255.250"), 3702);

		foreach (UdpClient udpClient in udpClients)
		{
			byte[] message = UTF8Encoding.UTF8.GetBytes(string.Format(WSDiscoveryProbeMessages, Guid.NewGuid().ToString()));

			try
			{
				await udpClient.SendAsync(message, message.Length, multicastEndpoint);

				IPEndPoint remoteEndPoint = null;

				while(true)
				{				
					message = udpClient.Receive(ref remoteEndPoint);

					Console.WriteLine($"IPAddress {remoteEndPoint.Address}");
					Console.WriteLine(UTF8Encoding.UTF8.GetString(message));

					Console.WriteLine();
				}
			}
			catch (SocketException sex)
			{
				Console.WriteLine($"Probe failed {sex.Message}");
			}
		}

		Console.WriteLine("Press enter to <exit>");
		Console.ReadKey();
	}
}

After confirming the program was working I used the excellent RaspberryDebugger to download the application and debug it on a Raspberry PI 3 running the Raspberry PI OS.

Windows 10 IoT Core TPM SAS Token Expiry

This is for people who were searching for why the SAS token issued by the TPM on their Windows 10 IoT Core device is expiring much quicker than expected or might have noticed that something isn’t quite right with the “validity” period. (as at early May 2019). If you want to “follow along at home” the code I used is available on GitHub.

I found the SAS key was expiring in roughly 5 minutes and the validity period in the configuration didn’t appear to have any effect on how long the SAS token was valid.

10:04:16 Application started
...
10:04:27 SAS token needs renewing
10:04:30 SAS token renewed 
 10:04:30.984 AzureIoTHubClient SendEventAsync starting
 10:04:36.709 AzureIoTHubClient SendEventAsync starting
The thread 0x1464 has exited with code 0 (0x0).
 10:04:37.808 AzureIoTHubClient SendEventAsync finished
 10:04:37.808 AzureIoTHubClient SendEventAsync finished
The thread 0xb88 has exited with code 0 (0x0).
The thread 0x1208 has exited with code 0 (0x0).
The thread 0x448 has exited with code 0 (0x0).
The thread 0x540 has exited with code 0 (0x0).
 10:04:46.763 AzureIoTHubClient SendEventAsync starting
 10:04:47.051 AzureIoTHubClient SendEventAsync finished
The thread 0x10d8 has exited with code 0 (0x0).
The thread 0x6e0 has exited with code 0 (0x0).
The thread 0xf7c has exited with code 0 (0x0).
 10:04:56.808 AzureIoTHubClient SendEventAsync starting
 10:04:57.103 AzureIoTHubClient SendEventAsync finished
The thread 0xb8c has exited with code 0 (0x0).
The thread 0xc60 has exited with code 0 (0x0).
 10:05:06.784 AzureIoTHubClient SendEventAsync starting
 10:05:07.057 AzureIoTHubClient SendEventAsync finished
...
The thread 0x4f4 has exited with code 0 (0x0).
The thread 0xe10 has exited with code 0 (0x0).
The thread 0x3c8 has exited with code 0 (0x0).
 10:09:06.773 AzureIoTHubClient SendEventAsync starting
 10:09:07.044 AzureIoTHubClient SendEventAsync finished
The thread 0xf70 has exited with code 0 (0x0).
The thread 0x1214 has exited with code 0 (0x0).
 10:09:16.819 AzureIoTHubClient SendEventAsync starting
 10:09:17.104 AzureIoTHubClient SendEventAsync finished
The thread 0x1358 has exited with code 0 (0x0).
The thread 0x400 has exited with code 0 (0x0).
 10:09:26.802 AzureIoTHubClient SendEventAsync starting
 10:09:27.064 AzureIoTHubClient SendEventAsync finished
The thread 0x920 has exited with code 0 (0x0).
The thread 0x1684 has exited with code 0 (0x0).
The thread 0x4ec has exited with code 0 (0x0).
 10:09:36.759 AzureIoTHubClient SendEventAsync starting
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Net.Requests.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'backgroundTaskHost.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.27505.2_arm__8wekyb3d8bbwe\System.Net.WebSockets.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Sending payload to AzureIoTHub failed:CONNECT failed: RefusedNotAuthorized

I went and looked at the NuGet package details and it seemed a bit old.

I have the RedGate Reflector plugin installed on my development box so I quickly disassembled the Microsoft.Devices.TPM assembly to see what was going on. The Reflector code is pretty readable and it wouldn’t take much “refactoring” to get it looking like “human” generated code.

public string GetSASToken(uint validity = 0xe10)
{
    string deviceId = this.GetDeviceId();
    string hostName = this.GetHostName();
    long num = (DateTime.get_Now().ToUniversalTime().ToFileTime() / 0x98_9680L) - 0x2_b610_9100L;
    string str3 = "";
    if ((hostName.Length > 0) && (deviceId.Length > 0))
    {
        object[] objArray1 = new object[] { hostName, "/devices/", deviceId, "\n", (long) num };
        byte[] bytes = new UTF8Encoding().GetBytes(string.Concat((object[]) objArray1));
        byte[] buffer2 = this.SignHmac(bytes);
        if (buffer2.Length != 0)
        {
            string str5 = this.AzureUrlEncode(Convert.ToBase64String(buffer2));
            object[] objArray2 = new object[] { "SharedAccessSignature sr=", hostName, "/devices/", deviceId, "&sig=", str5, "&se=", (long) num };
            str3 = string.Concat((object[]) objArray2);
        }
    }
    return str3;
}

The validity parameter appears to not used. Below is the current code from the Azure IoT CSharp SDK on GitHub repository and they are different, the validity is used.

public string GetSASToken(uint validity = 3600)
{
   const long WINDOWS_TICKS_PER_SEC = 10000000;
   const long EPOCH_DIFFERNECE = 11644473600;
   string deviceId = GetDeviceId();
   string hostName = GetHostName();
   long expirationTime = (DateTime.Now.ToUniversalTime().ToFileTime() / WINDOWS_TICKS_PER_SEC) - EPOCH_DIFFERNECE;
   expirationTime += validity;
   string sasToken = "";
   if ((hostName.Length > 0) && (deviceId.Length > 0))
   {
      // Encode the message to sign with the TPM
      UTF8Encoding utf8 = new UTF8Encoding();
      string tokenContent = hostName + "/devices/" + deviceId + "\n" + expirationTime;
      Byte[] encodedBytes = utf8.GetBytes(tokenContent);

      // Sign the message
      Byte[] hmac = SignHmac(encodedBytes);

      // if we got a signature foramt it
      if (hmac.Length > 0)
      {
         // Encode the output and assemble the connection string
         string hmacString = AzureUrlEncode(System.Convert.ToBase64String(hmac));
         sasToken = "SharedAccessSignature sr=" + hostName + "/devices/" + deviceId + "&sig=" + hmacString + "&se=" + expirationTime;
         }
   }
   return sasToken;
}

I went back and look at the Github history and it looks like a patch was applied after the NuGet packages were released in May 2016.

If you read from the TPM and get nothing make sure you’re using the right TPM slot number and have “System Management” checked in the capabilities tab of the application manifest.

I’m still not certain the validity is being applied correctly and will dig into in a future post.