💾 Archived View for gemini.complete.org › nncp-concepts captured on 2024-07-09 at 00:59:43. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
This page describes some basic concepts of NNCP[1].
NNCP is a versatile communications layer; think of it as an Asynchronous Communication[2] version of ssh. See my NNCP[3] page for ideas and links. NNCP can effectively route over anything from AWS instances to USB sticks and CD-Rs, and supports file transport and remote execution.
2: /asynchronous-communication/
Everything in NNCP is protected by end-to-end encryption and good security.
A key property of an NNCP network is that the two endpoints of communication need not be online simultaneously.
NNCP works with "packets". A packet is something that carries data. It is stored as a file in a queue on disk in your local NNCP instance. A packet can be an arbitrary size; from just a few bytes to multiple TB.
So, the NNCP commands are concerned with:
* And, as a subset of processing, routing packets.
I will go into the details of how packets are processed in a bit, but first I need to give you some background on the big picture of the concepts these packets provide. After that, I'll explain how these concepts map to the three categories of commands I outlined above.
The basic operations within NNCP are file transmission and remote command execution.
With file transmission, you can either send a file to a remote node (using nncp-file[4]) or request a remote node send a file to you (nncp-freq[5]). For instance:
4: https://nncp.mirrors.quux.org/nncp_002dfile.html
5: https://nncp.mirrors.quux.org/nncp_002dfreq.html
`nncp-file /bin/bash remotenode:bash2`
This command creates a packet, encrypted to remotenode, using `/bin/bash` as the content. It adds some metadata specifying that the file should be named `bash2` on the remote end. It writes this packet as a file in the local queue bound for transmission to remotenode.
The remotenode's configuration specifies whether this node is allowed to send it files, and if so, where to put them.
To request a file from the remote node:
`nncp-freq remotenode:README.txt`
This command creates a (very small) packet, encrypted to remotenode, saying "Hey, send me README.txt". remotenode checks to see if the request is authorized, and if so, what directory to find the file in. If it finds one, it does something very similar to the nncp-file command above, generating a reply packet that is basically a file transmission.
Let's say that on node bob, I authorized node alice to run a command I call "tarextract", defined like this:
`"tarextract": ["/usr/bin/tar", "-xpf", "-"]`
If you think this has a parallel to ssh's `authorized_keys`, you're absolutely right.
Now on alice, I can say this:
`tar -cpf - /usr/local | nncp-exec bob tarextract -C /var/local/backups`
So what does nncp-exec[6] do? It reads stdin and encrypts it to node bob. It adds metadata, saying it's requesting the "tarextract" command, and adding two arguments to it (the `-C /var/local/backups`). It writes this all into a packet file in the transmission queue for bob.
6: https://nncp.mirrors.quux.org/nncp_002dexec.html
When bob processes the packet, it spins up `/usr/bin/tar -xpf - -C /var/local/backups`, piping to it the data that had originally been piped to `nncp-exec`.
Email can be sent over NNCP; for instance, to use tools originating in the UUCP[7] days and which still work today, one might pipe an email message to `nncp-exec mailserver rmail foo@example.com`. The NNCP manual has details on how to set this up.
Now let's talk about the NNCP commands and how they relate to the operations we've identified so far.
We've already covered the three main commands to create packets: nncp-file, nncp-freq, and nncp-exec. All of these commands take some sort of input -- a file, a command to run, or just the name of a file to request -- and create a packet, encrypted with the destination's public key, which they write into the outgoing queue for the destination's node. (Except for when we route via an intermediary; more on that below.)
Now, there are three main ways to transport packets between NNCP nodes:
Let's take a look at these in more detail.
Since NNCP packets are fundamentally just files, these files can be transported to the remote over anything that can store... well, files. nncp-xfer[9] does just that. You can specify a destination node, and it will extract packets destined for that node from the NNCP queue, and copy (or move) them to the target path. It makes a simple directory structure on the target path.
9: https://nncp.mirrors.quux.org/nncp_002dxfer.html
Then, at the remote, you run nncp-xfer in receive mode. It does just the opposite; scans the target path for packets meant for it, and moves those into the incoming queue for later processing.
Since the directory tree is very basic, it can be just about anything. A USB stick is common, or a shared folder with Syncthing or Dropbox, or a mounted S3 folder -- whatever you like.
You can, of course, use the same filesystem to both send and receive packets from a given node.
Sometimes it is handy to be able to pipe a stream to somewhere. The target might be a tape drive, a process that burns the stream to a DVD+R, an ssh pipe to a remote, or all sorts of other things you can pipe stuff to in Unix.
That's what nncp-bundle[10] does. Conceptually, it's very similar to nncp-xfer; it's just that the interchange format is a stream (or single file) instead of a directory tree populated with files. nncp-bundle in transmission mode generates a stream for a remote, filled with all the packets in the transmission queue for that node. In receive mode, it receives a stream, and copies any packets meant for the local node into its incoming queue.
10: https://nncp.mirrors.quux.org/nncp_002dbundle.html
Most people will probably have a transmission over the network at some point. Although all NNCP nodes are peers, in TCP, you need someone to be the server and someone to be a client. So, the server is `nncp-daemon` and the client is `nncp-call` or `nncp-caller`.
Why two clients? Both do conceptually the same thing, and speak the same protocol. The difference is that nncp-call is designed to be run as a standalone command ("connect to the server for node foo RIGHT NOW"), while nncp-caller is designed to be a long-running service, that either maintains long-lasting connections to the remote nncp-daemon, or periodically establishes connections to it, or both. Functionally they are the same.
The network communication with NNCP is quite versatile. It is bidirectional, can send packets in an order given by their user-defined prioritization, can resume interrupted transfers, and can even auto-detect peers on the local network. It is also, of course, encrypted and authenticated.
My page at Tunneling NNCP over other transports[11] shows some more advanced ways to integrate these fundamental commands with things like ssh, Syncthing[12], and tor.
11: /tunneling-nncp-over-other-transports/
So far, I've talked a lot about queues. The commands that create packets (nncp-file, freq, and exec) put the packets in an outbound queue. There is also an inbound queue. The commands to transport packets exchange data from those queues. That leaves us with one more piece: how do we handle packets in the inbound queue?
That's the job of nncp-toss[13]. It literally processes packets from incoming queues. It verifies their integrity, decrypts them, and handles them as needed: running commands for packets generated by nncp-exec, depositing files sent by nncp-file, answering requests from nncp-freq, and also routing packets on to their destination (more on that below).
13: https://nncp.mirrors.quux.org/nncp_002dtoss.html
nncp-toss can be run as a standalone command, from a cron job, or as a daemon with the `-cycle` parameter.
There is also a command `nncp-stat` that shows you the status of all your queues, incoming and outgoing.
Take a look at my other NNCP[14] resources, particularly Getting Started with NNCP[15].
15: /getting-started-with-nncp/
You can skip this section for now; it covers some more advanced topics.
Thus far, I have made an assumption: that when you send a packet from node alice to node bob, alice and bob are "directly" connected. That is, they can speak to each other, or you can move a USB stick between them, or some such scenario.
Now imagine this scenario: you have three nodes. Alice can communicate with bob, and bob can communicate with charlie, but alice and charlie have no way of communicating directly. Wouldn't it be nice to use bob as an intermediary between alice and charlie? In fact, with NNCP, you can do just that!
For instance, on alice you could say:
`nncp-file -via bob /bin/bash charlie:bash2`
So, here alice sends a packet to charlie via bob! What exactly does that do? It uses onion encryption, like Tor. Conceptually, it works like this:
(In practice, this is much more efficient and optimized than you might guess from reading this description.)
You can, in fact, have a via path that's arbitrarily long; for instance, `-via alpha,bravo,charlie`. This kind of via routing is used in a number of scenarios: for instance, the quux.org NNCP public relay[16] is using it. Also note that each hop along the via path can transport packets using any of NNCP's methods. So you could, for instance, build a very secure machine to receive backups. So secure, in fact, that it has no network access.
16: /quux-org-nncp-public-relay/
You could have one of the machines on your network as the dedicated "hub"; all the machines could send their backups to the backuphost via the hub. Then periodically you write the backups to a USB drive, unplug it from the hub, plug it into the backup server, and ingest them there.
You can specify a default via path for each of the nodes you communicate with in the configuration file.
NNCP also supports what it calls multicast areas[17]. The idea here is when you want to send packet to more than one destination; for instance, to a whole handful of nodes in your NNCP network. It facilitates what amounts to an Asynchronous[18] Mesh Network[19]. I use it in my gitsync-nncp[20] software.
17: https://nncp.mirrors.quux.org/Multicast.html
18: /asynchronous-communication/
20: https://changelog.complete.org/archives/10274-distributed-asynchronous-git-syncing-with-nncp
--------------------------------------------------------------------------------
21: /using-filespooler-over-nncp/
NNCP[22] is a powerful tool for building Asynchronous Communication[23] networks. It features end-to-end Encryption[24] as well as all sorts of other features; see my NNCP Concepts[25] page for some more ideas.
23: /asynchronous-communication/
I sometimes see people read about NNCP[27] and wonder "This sounds great! But... what can I do with it?" This page aims to answer those questions.
The care and feeding of an NNCP[29] installation.
30: /getting-started-with-nncp/
This page describes the basic installation and configuration of NNCP[31].
Usenet[33], of course, originally ran over UUCP[34] in quite a few cases. Since NNCP[35] is quite similar to UUCP -- in fact, you can map UUCP commands to NNCP ones[36] -- it is quite possible, and not all that hard, to run Usenet over NNCP. In fact, in a number of ways, it works better than Usenet over UUCP!
36: https://nncp.mirrors.quux.org/Comparison.html
37: /quux-org-nncp-public-relay/
According to the NNCP documentation[38], NNCP[39] is intended to help build up small size ad-hoc friend-to-friend (F2F) statically routed darknet delay-tolerant[40] networks for fire-and-forget secure reliable files, file requests, Internet Email[41] and commands transmission. All packets are integrity checked, end-to-end Encrypted[42], explicitly authenticated by known participants public keys. Onion encryption is applied to relayed packets. Each node acts both as a client and server, can use push and poll behaviour model. Also there is multicasting area support.
38: https://nncp.mirrors.quux.org/
40: /asynchronous-communication/
NNCP lets you securely send files, or request remote execution, between systems. It uses asynchronous communication[44], so the source and destination need never be online simultaneously. NNCP can route requests via intermediate devices -- other NNCP nodes, USB sticks, tapes, radios, phones, cloud services, whatever -- leading to a network that is highly resilient and flexible. NNCP makes it much easier to communicate with devices that lack Internet connectivity, or have poor Internet.
44: /asynchronous-communication/
(c) 2022-2024 John Goerzen