💾 Archived View for jsreed5.org › log › 2023 › 202309 › 20230902-a-practical-use-for-nncp.gmi captured on 2023-11-14 at 07:55:16. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-09-08)
-=-=-=-=-=-=-
---
I've used AWS for quite some time before moving my capsule into EC2: my primary use case for it is off-site archival by backing up my files to S3. Using a shell script and the AWS CLI, I encrypt the files in my NAS, tag them with a checksum to detect file changes, and upload them in bulk, storing them in the Glacier Flexible Retrieval tier to save on storage costs.
I recently rewrote my backup script to be able to exclude directories from archival. The mechanism by which I do this is including a ".noupload" dotfile in any directory I want to exclude. When the script sees this file, it archives a list of the directory's contents rather than the directory itself, and it ignores all of its subdirectories. This allows me to save even more by excluding files that are easy to replace or don't mean much if I lose them.
We have a few terabytes of data of all kinds, and it's been several months since I last ran a full backup job. I like to run the backup script on only a few directories at a time, letting each one complete successfully and update a master list of all the files in S3 in chunks. This means I run the script relatively frequently over the course of a complete backups.
My family is about to leave on a trip, and our home network will be idle while we're gone: a great opportunity to do another full archive. However, since I run my script in several small batches, I'll need to complete multiple runs to finish the entire thing--and since I won't be home, I'll have to check on the script manually every now and then to see if it's completed.
I wanted some kind of notification that the archiving machine could send to my smartphone. E-mail is the standard tool for this kind of thing: thanks to running yggdrasil on all my devices, they can behave as if they're all on the same network even when they're not. Unfortunately, I know absolutely nothing about setting up SMTP or a mail server, and I don't have much time to research now to do it before the trip.
I then realized that I'd already set up an application that was perfect for this use case: NNCP. NNCP is a spiritual successor to UUCP: a suite of store-and-forward programs, allowing for asynchronous file transfers and even command execution. It works over almost any transmission protocol imaginable, from TCP to sneakernets, and features encryption for every packet including onion encryption for relaying. I had set up NNCP on all my devices that were running yggdrasil, but at the time I didn't have a particular use case; I just thought it would be something fun to play with.
Now a use case was clear: I could have my script create a text file with a summary of the backup job, enqueue it with NNCP, and relay it to the NNCP node running on my capsule. That node auto-tosses NNCP packets as they come in, and all I need to do on my smartphone is set up a job in Termux to occasionally call the capsule node for packets.
I set up the process in my backup script last night and tested it with some small archival jobs. Sure enough, a summary file sat on my phone a few minutes later with all the info I needed. If I got a summary file, the backup job had finished successfully, and I was ready to start another one.
This opens up a whole realm of possibilities for me. Termux can write files to common storage, which means I can configure an app to create a notification in my status bar when a summary file arrives and have near real-time pings about my backups. NNCP offers remote execution as well; I can configure the node on my archival machine to kick off a job simply by requesting one via an NNCP packet. I can finally start really exploring the power of automation, without needing SSH, Ansible, Salt or other complex tools--as great as those tools are.
I posted only a broad overview of the installation process for yggdrasil on my capsule, and I haven't talked about configuring NNCP at all. Soon I'll write up a technical guide and share it here. These programs are fantastically powerful tools and they cater well to the DIY spirit of many of us here on Gemini.
---
[Last updated: 2023-09-02]