💾 Archived View for gemini.complete.org › verifying-filespooler-job-integrity captured on 2024-08-31 at 12:09:21. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-07-09)
-=-=-=-=-=-=-
Sometimes, one wants to verify the integrity and authenticity of a Filespooler[1] job file before processing it.
This poses a particular challenge, because most of these verification tools (including GnuPG (GPG)[2] have a weakness: even though some of them can work in a pipe, most of them are not able to establish whether the data was authentic or not until they have processed the *entire* file. So even when Filespooler may only want to read 100 bytes to get to the header, the verifier may emit that data without having validated it.
So the usual Filespooler trick of using a `--decoder` isn't applicable here.
Instead, we really need to pre-process the data. Since one can Process Filespooler Queues Without Filespooler[3], we can have a two-step approach:
3: /processing-filespooler-queues-without-filespooler/
1. Incoming data is written to an "incoming" queue, which will never actually be processed by Filespooler.
2. When it is verified, it is moved to the real queue to be processed.
Cryptographic signatures (or, really, simple hashes) come in two flavors: ones that embed the signature in the file itself, and ones that put it in a separate file. This example focuses on the former, but the latter is easy to handle as well; you can just reference the signature file and delete it once verified.
GPG is a particularly interesting problem, because it has several options:
This example will assume data was encrypted and signed. We will put it into the incoming queue, process from there without using Filespooler, and then move it to the "verified" queue.
First, let's create the queues:
fspl queue-init -q incoming fspl queue-init -q verified
We're never going to actually use Filespooler to *read* the incoming queue, but by setting it up this way, we can use `fspl queue-write` to add things to it.
For more detail, please refer to Encrypting Filespooler Jobs with GPG[4]. This assumes a file synchronization transport as in Using Filespooler over Syncthing[5]; of course, you can adapt it to whatever transport you like.
4: /encrypting-filespooler-jobs-with-gpg/
5: /using-filespooler-over-syncthing/
You can create a signed and encrypted packet with something like this:
echo hi | \ fspl prepare -s state -i - | \ gpg -e -r 0x1234567891123456789212345678931234567894 -u 0x1234567891123456789212345678931234567894 --sign - | \ fspl queue-write -q incoming
I always like to give the full 40-digit hex ID for a key to GPG for maximum security. Here, we:
* `-e` says to encrypt the data
* `-r` sets the recipient for the encryption
* `-u` sets the key ID to sign with
* `--sign` says to sign it
* `-` says to process stdin
Now, we need to verify the signed packets. As we've established before, using Filespooler to do this directly has some obstacles, but it is easy to do ourselves. First, we write a script to verify it:
#!/usr/bin/env bash
set -euo pipefail
if gpg --status-fd 1 -o /dev/null -d < "$2" | \
grep -q 'VALIDSIG 1234567891123456789212345678931234567894'; then
echo ln "$2" "$1/$2"
echo rm "$2"
else
echo "Failed validation of $1"
exit 1
fi
Save this as `queue-verify` somewhere on your path and mark it executable.
Let's walk through that script and how it works:
* `$1` is the path to the verified queue
* `$2` is the file to verify
* `--status-fd 1` tells gpg to write status information in a machine-parsable way to stdout.
* `-o /dev/null` tells gpg to discard the decrypted data.
* First, create a hardlink to add it there
* Then, delete it from the incoming queue.
6: /guidelines-for-writing-to-filespooler-queues-without-using-filespooler/
We will call this script from find. Due to the way it handles paths, the directory it's called from is important:
cd incoming
find jobs -name 'fspl-*.fspl' -exec queue-verify ../verified {} \;
This is the same as in Encrypting Filespooler Jobs with GPG[7]. For instance:
7: /encrypting-filespooler-jobs-with-gpg/
fspl queue-ls -d 'gpg -q -d - 2>/dev/null' -q verified
Of course, you could use `queue-process` or whatever here also.
Here are some other ideas for verification tools:
--------------------------------------------------------------------------------
GnuPG (also known by its command name, gpg) is a tool primarily for public key Encryption[11] and cryptographic authentication.
12: /encrypting-filespooler-jobs-with-gpg/
Thanks to Filespooler[13]'s support for decoders, data for filespooler can be Encrypted[14] at rest and only decrypted when Filespooler needs to scan or process a queue.
Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing[16], NNCP[17], ssh, UUCP[18], USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler.
Filespooler is particularly suited to distributed and Asynchronous Communication[19].
19: /asynchronous-communication/
(c) 2022-2024 John Goerzen