💾 Archived View for gemini.complete.org › usenet-over-nncp captured on 2024-08-31 at 12:03:29. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-07-09)
-=-=-=-=-=-=-
Usenet[1], of course, originally ran over UUCP[2] in quite a few cases. Since NNCP[3] is quite similar to UUCP -- in fact, you can map UUCP commands to NNCP ones[4] -- it is quite possible, and not all that hard, to run Usenet over NNCP. In fact, in a number of ways, it works better than Usenet over UUCP!
4: https://nncp.mirrors.quux.org/Comparison.html
According to the NNCP documentation[6], NNCP is intended to help build up small size ad-hoc friend-to-friend (F2F) statically routed darknet delay-tolerant networks for fire-and-forget secure reliable files, file requests, Internet mail and commands transmission. All packets are integrity checked, end-to-end encrypted, explicitly authenticated by known participants public keys. Onion encryption is applied to relayed packets. Each node acts both as a client and server, can use push and poll behaviour model. Also there is multicasting area support. NNCP can operate over a lot of transports: Internet, USB sticks, tapes, CD-ROMs, and some more[7] such as ssh, socat, Dropbox, and so forth.
6: https://nncp.mirrors.quux.org/
7: /tunneling-nncp-over-other-transports/
The NNCP homepage is http://www.nncpgo.org[8]. A mirror using TLS and Let's Encrypt is at https://nncp.mirrors.quux.org[9]. See NNCP Concepts[10] for an introduction to NNCP if you aren't already familiar with it.
9: https://nncp.mirrors.quux.org
Usenet[12] is sometimes said to be the world's oldest social network. Since 1980, Usenet has been a massive, global discussion system. Participants can read and post messages (called articles) in discussion forums (called newsgroups). Unlike web forums, Usenet newsgroups are available from thousands of independently-operated servers worldwide (instead of just one particular site). You can also use the client of your choice to access them.
An NNCP feed is ideal for people that wish to run a smaller server; since the tools that operate Usenet come from an era in which even a Raspberry Pi would have been an unimaginably high level of computing power, the resource requirements for a small site are quite minimal
This document is primarily intended to let you operate as a leaf node for news.quux.org, the Usenet transit server operated principally by John Goerzen[14]. The quux.org Usenet NNTP and NNCP peer[15] (news.quux.org) has numerous peers with which it exchanges Usenet articles, and is able to offer you a full (basically every non-binary newsgroup) or partial (the newsgroups you select) feed.
15: /quux-org-usenet-nntp-and-nncp-peer/
Secondary intents are to:
Demonstrate how to integrate INN with NNCP
Explain how you might operate a standalone (not connected with the global Usenet) INN network using NNCP. This scenario is actually simpler than participating in the public Usenet, so much of what is said here may not apply to you.
Here are some examples:
* You can also do the reverse; feed posts your friends send you via NNCP to quux and thus to the global Usenet.
Before continuing, you must already:
If you are not already familiar with Usenet, it would probably be a good idea to dive in!
Start with a newsreader program. Mozilla Thunderbird[17] or PAN[18] may be good places to start. (For Thunderbird, add a new "other" account, for Usenet/newsgroup access). You can get a free Usenet account from AIOE[19] or Eternal September[20] to use with it. Don't skip this step!
17: https://www.thunderbird.net/en-US/
20: https://www.eternal-september.org/
Because news.quux.org is capable of feeding your Usenet posts to thousands of Usenet servers across the globe, some additional requirements apply:
Your NNCP data will be exchanged using the quux.org NNCP public relay[21]. So, you will need to establish yourself as an NNCP peer with it according to its documentation[22]. (This is a quick and simple process.)
21: /quux-org-nncp-public-relay/
22: /quux-org-nncp-public-relay/
NNCP already set up to call quux, and to run the tosser in some way.
If you discontinue using the quux NNCP gateway -- and that's fine -- you need to let me know, so we don't build up an endless set of articles for a system that will never talk to us again.
You must not operate an open or public relay. Generally, that means "don't expose any ports on your Usenet server to the Internet."
* If you must expose the NNTP port to the Internet - which is discouraged in this situation - it must require authentication and you must not hand out accounts to the public. As with an email service, operating a Usenet service that is open to the world would be ripe for abuse. Those at AIOE and Eternal September have years of experience preventing that, and running a public relay from quux over NNCP is just going to be more pain than any of us have time to deal with.
news.quux.org is a well-connected transit system, and as such will send you articles from the public Internet. While measures are taken to filter at quux, you must understand and agree that, like much of the public Internet, Usenet has no central authority and you may encounter content you find objectionable. Content policing is up to each individual server operator in Usenet, which will now include YOU. Be familiar with what's out there before, for instance, you offer it to children.
Similarly, it is your duty to not spread objectionable content to the Internet via quux. Examples of objectionable content include that which is illegal, spam, abusive or exploitative, "doxxing", harassing, impersonating others with fake headers or the like, threatening or causing harm (including by spreading health misinformation), and so forth. While there are corners of the global Usenet that have ugly behavior, and news.quux.org being fundamentally a dumb machine that just copies data from point A to B means that you may encounter this, there is a also an effort to make more parts of the Internet a positive environment, and quux.org is part of that effort. https://floss.social/terms[23] is an example of the kind of conduct we would like to see from articles you feed to us. **We want to be part of the solution, not part of the problem.**
23: https://floss.social/terms
* If you provide access to others in any way, whether by letting them access your server or by becoming NNCP peers with them, you need to make sure they understand their responsibilities on what gets injected to the globe via you, and be prepared to enforce standards of behavior. You may, for instance, need to cut them off if they engage in objectionable behavior via your system. Keep this in mind.
* If persistent abuse occurs from your site which you are unable or unwilling to correct, or particularly egregious abuse occurs, your feed may be canceled.
* You must have up-to-date contact information on file with me at quux, so that you can be reached in the event complaints are received about your site at quux. The idea is to resolve things collaboratively so that feed interruption need not occur -- but moreover, to prevent issues from occurring in the first place.
* Also this service is all-volunteer without guarantee of correct operation or the continuing operation of any feed.
You should also be aware, in case you are planning to take your solar-powered Raspberry Pi with INN and NNCP with you on a 3-week hike across the remotest parts of Canada, writing articles to pass the time and posting them when you reach civilization, that most Usenet sites reject articles that reach them if they are older than a certain number of days. For most, that number is 10, but it may be as little as 1 or 2 on some. So if you are feeding data TO the Internet via quux, keep that in mind. For data coming to YOU from quux, you can set the limit whatever you like (but the quux public relay may also auto-reap packets of advanced age that are bound to you). So basically if you want to use this, checking in daily will probably work; weekly will probably usually work, and monthly will not.
There are various Usenet servers; for this, we will use INN 2.6 (if you're using Debian, the package name is inn2).
INN still has UUCP support. NNCP presents an interface that is almost identical to UUCP, so we will be generally configuring INN as if it's exchanging articles by UUCP.
Here's how INN generally works in a UUCP/NNCP scenario:
This is basically using a tried-and-true method that has worked with INN's UUCP support for decades. All we are doing is substituting NNCP for the UUCP layer. We are not using NNCP's multicast areas for this; see the conversation below about this.
Before getting started, you need to think about a few things.
The Usenet hostname you set is a vital part of preventing routing loops and duplicate messages on Usenet. It does not have to be a valid Internet hostname (though it often is). It does have to be globally unique, and it also ought to serve to identify your system.
If you already own a DNS domain, you could use something under that. For instance, if you have registered example.com, you might consider news.nncp.example.com. It is fine if `news.nncp.example.com` doesn't resolve in DNS; it just needs to not be used for something else.
If you don't own a DNS domain, then you may have to make something up, perhaps ending in `.nncp`. For instance, `news.smith.newark.nj.us.nncp` or something. Visit with me about this if you like. The key is that there must be no chance of an accidental use of the same name by some other system anywhere on the planet.
You will add this to the "pathhost" line in inn.conf, and also to the "ME" line in newsfeeds.
You can opt to carry a "full feed" from quux - that is, basically everything that transits quux will be sent to you. The concept of "everything" is a little different at each site due to things like spam filters.
Or, you can opt to carry a handful of specific newsgroups -- say, `comp.mail.uucp`. Or an entire hierarchy; say, `comp.*` or `comp.mail.*`. These can be changed later, but do require manual config file editing on my part, so be aware that a change doesn't happen instantly.
New newsgroups are often created (or added to quux because a new feed is added which carries them).
You can get a sense of what newsgroups are out there by looking at the ISC's files at ftp://ftp.isc.org/pub/usenet/CONFIG[24] . The file `active` in that directory contains a list of every newsgroup believed to be active at ISC. The file `newsgroups` contains descriptions for many, but not all, newsgroups. Be aware that the actual set of carried newsgroups varies from site to site, particularly for `alt.*` which lacks a formal process for authorizing new groups.
24: ftp://ftp.isc.org/pub/usenet/CONFIG
Some newsgroups are *moderated*; that is, you can't post directly to them. When you attempt to post to a moderated group, INN will actually submit your post via email to the moderator, rather than post it to Usenet. Therefore, if you intend to post to a moderated group, you will require working email service on your Usenet server. Moderated newsgroups today are often quite low-traffic and rarely posted to by the public. You can identify a moderated group because, in the active file from ISC, the line ends with `m` instead of `y`.
Send your peering request to me, jgoerzen@complete.org. If you haven't already, make sure you include your information for the public relay nodelist. Also include your selected hostname and the names of the newsgroups you want to use.
OK, let's begin! This is the easy part. First, you need to have done the setup for the quux.org NNCP public relay[25], which will have included adding the quux definition to your nncp.hjson.
25: /quux-org-nncp-public-relay/
Then, you need to note the quux news server in your nncp.hjson's neigh section:
news.quux.org: {
id: 6QVFU7QIHQNI6TWNRK7OAJUUBDZT262XX3I5YHGZOGLTVYOQM47Q
exchpub: DBHZYJFEXNGJ5UFC6TZVYRAAAKKKVLT3DAAXCQXZZYERXPLLEQIQ
signpub: OKFURSS5YFXUEEPYCAEGPBIPZDHDBISNJA47LHGMVECH6KC5PEWA
noisepub: ZXEYBPZN5WAI4674NBF5ZRX6ZG7YEILVMRDGQT3NWHA7Q5ZMXQXQ
via: ["quux"]
incoming: "/tmp" # or more appropriate path; only if you intend to freq active from news.quux.org
exec: {
rnews: ["/usr/bin/rnews", "-h", "news.quux.org"]
}
}
Here you have set up NNCP so that rnews can be executed to process incoming packets from quux. As a further exercise for added security, you should wrap it with a script that prevents any further arguments to rnews (since NNCP otherwise permits that), but this simple example gets you started.
Now, we need a script to act as a plug-in replacement for `uux`. This is what INN will call to submit the articles you write back to quux. Put this in `/usr/local/bin/nncp-uux`:
#!/usr/bin/env bash
set -eo pipefail
while [ -n "$1" ]; do
if echo "$1" | grep -q "!rnews"; then
HOST="$(echo "$1" | sed 's/!rnews//')"
# This exec will terminate the script here
exec /usr/local/bin/nncp-exec -quiet "$HOST" rnews
fi
shift
done
echo "Couldn't find valid host"
exit 5
And, of course, `chmod a+x /usr/local/bin/nncp-uux` and, if your nncp-exec isn't in `/usr/local/bin`, fix that path. INN's `send-uucp` process will cause this thing to be invoked with a bunch of parameters, but most particularly one that's `HOSTNAME!rnews`. We extract that HOSTNAME as our nncp node name, and form the nncp-exec command from there.
The last remaining part is to allow the `news` user to queue up packets using `nncp-exec`. There are three ways you can do that:
26: https://nncp.mirrors.quux.org/Administration.html#Shared-spool
There is a similar part to allow NNCP to run rnews; you need to add nncp to the uucp group; `adduser nncp uucp` will do the trick on Debian.
What you do is up to you. Allowing the news user permissions to NNCP does mean that it could access the NNCP private keys if there was a security breach in the INN process or something, but whether you care about that possibility is up to you.
It is not necessary to run `rnews` as the news user, since it feeds to INN by connecting to localhost:119. But, by default it is only executable by the news user or uucp group, which is why we need NNCP to be running as either that user or that group.
That's it. You're done configuring NNCP!
For the purposes of easy conversation, this document will cover INN 2.6.4 as distributed in Debian 11 (bullseye). Path names will differ on other systems, but the concepts should be the same.
This document is not a substitute for the proper INN documentation[28]. In particular, consult the INSTALL[29] file, the checklist[30], and the documentation for each config file and program[31]. I am going to highlight some steps, but this is *not* a comprehensive INN installation document!
28: https://www.eyrie.org/~eagle/software/inn/
29: https://www.eyrie.org/~eagle/software/inn/docs-2.6/install.html
30: https://www.eyrie.org/~eagle/software/inn/docs-2.6/checklist.html
31: https://www.eyrie.org/~eagle/software/inn/docs-2.6/
First, install INN: `apt-get install inn2 inn2-inews`. (the inn2-inews package includes rnews)
The INN configuration files live in `/etc/news`. You'll usually edit them as root, but all other actions should be run as the `news` user, which you can enter with `su -s /bin/bash - news`.
Although you can reload certain INN config files via `ctlinnd`, the easiest way to make changes take effect is with `systemctl restart inn2`.
inn.conf[32] has some critical settings. You'll want to set something sensible for organization. The hostname you've chosen goes in pathhost.
32: https://www.eyrie.org/~eagle/software/inn/docs-2.6/inn.conf.html
You will probably want to set `innflags: "-C"` to disable processing of cancel messages unless you obtain more advanced processing (which is outside the scope of this document).
Depending on your circumstances, you may wish to set `addinjectionpostinghost` to false for privacy reasons.
You should put your valid e-mail address for `complaints`, which is part of the `Injection-Info` header.
Many sites will also need to set `domain:` to be the domain part of your pathhost. That is, if your pathhost is "foo.bar.baz.example.com", then domain is "bar.baz.example.com".
expire.ctl[33] defines how long old articles are kept around. By default, they're generally kept for 15 days. See the comments in this file if you want to keep them for a longer or shorter time. It is not necessary to modify this file.
33: https://www.eyrie.org/~eagle/software/inn/docs-2.6/expire.ctl.html
In newsfeeds[34], you need to edit the "ME" line to exclude your own hostname (pathhost). Add `/PATHHOST` after "ME", so it reads like this:
34: https://www.eyrie.org/~eagle/software/inn/docs-2.6/newsfeeds.html
ME/my.pathhost.value.goes.here:!*/!local,!collabra-internal::
We will do some more work in this file later.
Now that you have done the basics, it's time to create some newsgroups. How you will do that will depend on whether you are grabbing a few newsgroups or thousands.
In this scenario, you can manually create each relevant newsgroup. Say you wanted to add `comp.misc.fakegroup`, as unmoderated. You would run:
`/usr/lib/news/bin/ctlinnd newgroup comp.misc.fakegroup y`
If it was moderated, you'd use `m` instead of `y`. That's it. You're done.
In this case, you will need to use some automated mechanism to create your local newsgroups. First, request `active` from news.quux.org:
nncp-freq news.quux.org:active
Once that file is delivered, you can use actsync to process it.
Edit actsync.ign. By default, it's set up to just bring in the 8 "major" news hierarchies. That's probably reasonable to start. If you want to just take everything quux has, add `c *` after the example lines. Then run:
/usr/lib/news/bin/actsync -p 0 -v 2 -i /etc/news/actsync.ign localhost \ /var/local/nncp-incoming/news.quux.org/active > /tmp/foo
This `nncp-incoming` path is the location where the active file you freq'd appears (don't forget to chown it to news before you run this command, if needed). Now, inspect /tmp/foo. You might make sure there are no unexpected `rmgroup` commands (which would remove groups). If you're satisfied, then run:
/usr/lib/news/bin/mod-active /tmp/foo
Remember, the INN configuration is in `/etc/news`.
We're going to set up INN's send-uucp[35] for our purposes. This is what configures INN for sending outbound articles you write to quux. Add a line for quux like this:
35: https://www.eyrie.org/~eagle/software/inn/docs-2.6/send-uucp.html
news.quux.org none 52428800
This defines that you will send to quux, without compression, with a maximum size of 50MB per batch. We turn off compression here because NNCP has built-in compression that is better anyway, and running through two compressors just wastes CPU power. send-uucp will start a new batch if any one batch exceeds 50MB in size. A batch is one or more articles that are combined into a single nncp-exec invocation.
Now, in `newsfeeds`, you need to add an entry for quux like so:
news.quux.org\ :*,\ !*.bina*,!*.bain*,!*.dateien*,!*.pictures*,!junk/!local\ :Tf,Wnb,B4096/1024:
Here's a **very important note**: make sure there isn't a space at the end of that line. If there is, it will take the space to be the name of the file it should create in `/var/spool/news/outgoing` and that will mess everything up.
If you are taking only a subset of newsgroups, you will want to modify this accordingly to send only the desired messages to quux.
Also, remember to modify the "ME" line as described above.
Next, we're going to tell INN to use our fake uux instead of UUCP's. Add this to innshellvars.pl.local:
$uux = '/usr/local/bin/nncp-uux';
Now, INN's Perl config system in `/usr/share/perl5/INN/Config.pm` won't actually use this file unless it's marked executable. So:
`chmod a+x innshellvars.pl.local`
Now, make a test post somewhere. Then let's see if we can generate outbound packets.
First, become the news user: `su -s /bin/bash - news`.
Then run `/usr/lib/news/bin/send-uucp`. With any luck, `nncp-stat` should show an outbound packet generated!
Now you can set up cron for send-uucp. Something like this in /etc/cron.d/local-send-uucp will do the trick:
PATH=/usr/lib/news/bin:/sbin:/bin:/usr/sbin:/usr/bin 20,50 * * * * news /usr/lib/news/bin/send-uucp
The Usenet Top 1000[36] project tracks the top 1000 Usenet servers on the Internet. It does this by receiving automated reports from participating administrators of the articles that pass through their site - and, critically, information about the Path header in each article that indicates where in the globe it traversed.
36: http://top1000.anthologeek.net/
Your participation helps in several ways:
Once INN is set up and working, the only requirement for participation is a working email system. Reports are (these days) sent out daily.
To enable, first edit `newsfeeds` and add this line:
inpaths!:*:Tc,WP:/usr/lib/news/bin/ginpaths2
Don't forget to restart INN or issue `ctlinnd reload newsfeeds reason`.
Now, edit `/etc/cron.d/inn2` and find the "NINPATHS" section at the bottom. Uncomment the two lines, and in the second line, change `1 * *` to `* * *` (that line dates from a time when people made monthly instead of daily reports). The lines should now read:
6 6 * * * news ctlinnd -s -t 60 flush inpaths! 8 6 * * * news sendinpaths
That's it. Done!
By now, you should have your system fully armed and operational! It should be able to send and receive news. Except... you have no way to read it. So let's do that.
The file you will be interested in is readers.conf[37]. By default, it will let you connect from localhost. So, that may be enough for you! You could install a reader like slrn over there and just ssh into your newsbox and read news there.
37: https://www.eyrie.org/~eagle/software/inn/docs-2.6/readers.conf.html
But let's say you want to permit access from your LAN. Then you need to support passwords. Add this after the `auth "localhost"` block:
auth "authenticated" {
hosts: "192.168.0.0/16"
auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f /var/lib/news/newsusers"
}
access "authenticated" {
newsgroups: "*"
}
Of course, use the relevant netmask there.
Now you need to create the `/var/lib/news/newsusers` file. On Debian, you will need to `apt-get install apache2-utils` to do this.
Now, `su -s /bin/bash - news` and run `htpasswd -c -d /var/lib/news/newsusers username`. You can add more users also; just leave off the `-c` (since it means "create").
If you're going to use this on the public Internet -- and I recommend you do **NOT** - but if you must, then you will certainly want to arrange to run a nnrpd with SSL and add `require_ssl: true` to the auth group.
NNCP permits the caller to specify parameters to the command lines of executed programs in nncp-exec. To prevent callers from adding additional parameters to rnews - perhaps maliciously targeting other NNTP servers for instance - you could lock things down. First, create this at `/usr/local/bin/restricted-rnews`:
#!/usr/bin/env bash set -euo pipefail exec /usr/bin/rnews -h "$1"
Now, your exec clause in nncp.hjson changes to:
exec: {
rnews: ["/usr/local/bin/restricted-rnews", "news.quux.org"]
}
Since any other parameters would be added after the first one -- the hostname you've given -- they will be ignored here.
Indeed, NNCP's multicast areas[38] feature is quite appealing. You might rightly conclude that its algorithm for asynchronous distribution is quite similar to Usenet's, but made more general so it can be used with anything.
38: https://nncp.mirrors.quux.org/Multicast.html
It would, in fact, be quite possible to handle distribution via either a multicast area or direct feeds from INN like this document discusses. I opted for direct feeds for two reasons:
This is unlikely to be a great idea, because it would mostly be two systems sending each other articles that the other has already seen. I'm willing to explore, but would probably prefer to peer with you over NNTP.
This information is subject to change.
At the moment, I am trying to strike a balance between quick turnaround, and not giving those that process news daily thousands of tiny packets to process. Right now:
For most, calling the quux public relay at :20 and :50 would be the most expeditious way to get your local posts out.
If two NNCP peers of news.quux.org carry a newsgroup, and one has a new post, it will be processed like this:
1. The post flows to news.quux.org via NNCP.
2. When it is ingested into quux, it will be queued for transmission to all of quux's peers, Internet and NNCP.
3. When the next set of outbound NNCP batches is run -- generally 5 minutes after the incoming post is processed -- the post will be included, along with a Path: entry reflecting that it passed through news.quux.org. It is not necessary for the post to traverse the Internet to be included in packets for other quux NNCP peers.
From ftp.isc.org in /pub/usenet/CONFIG.
quux.org[39] is one of two domains used for services I have long provided; the other, primary one, is complete.org. They both run on the same hardware. complete.org has had an Internet presence since 1995, but existed as a BBS before then. The quux.org domain was registered in 2000.
Because:
40: /old-and-small-technology/
A few more words on that last one. Today we see a lot of trouble with the giants that control the Internet: the Facebooks and the Youtubes of our time. We see them promoting anything that drives people to spend time on their site -- even if it is false and harmful -- because spending time on their site is how they make money. This has had real-world consequences, including deaths.
It's not that Usenet was or is perfect. After all, spam sort of started on Usenet. Usenet has no central authority and is therefore anarchic in some ways. This carries its own set of problems, but those problems are *different* than the ones at Facebook and Twitter. At Facebook and Twitter, you get a highly curated experience -- curated in a way that benefits the social media companies, not you. With Usenet, you get an uncurated experience. It may be better, it may be worse, but it is most definitely not exploiting your attention[41] for somebody else's profit.
I hope to operate a useful service for a long time, but no, there are no guarantees. This service is powered by goodwill and volunteer effort. Part of the reason for the careful ToS for posts is because large corporations have the staff to deal with numerous complaints, but volunteers don't.
But let's have fun with this and do our part to make a better Internet and community!
--------------------------------------------------------------------------------
I sometimes see people read about NNCP[43] and wonder "This sounds great! But... what can I do with it?" This page aims to answer those questions.
44: /quux-org-usenet-nntp-and-nncp-peer/
At quux.org[45], I[46] operate a heavily-peered Usenet[47] server. It peers with others on the Internet using conventional NNTP. Moreover, I also offer partial and full Usenet over NNCP[48] feeds. quux.org carries a full set of text newsgroups, and no binaries.
This page gives you references to software by John Goerzen[50].
51: /the-pc-internet-revolution-in-rural-america/
Inspired by several others (such as Alex Schroeder's post[52] and Szczeżuja's prompt[53]), as well as a desire to get this down for my kids, I figure it's time to write a bit about living through the PC and Internet revolution where I did: outside a tiny town in rural Kansas. And, as I've been back in that same area for the past 15 years, I reflect some on the challenges that continue to play out.
52: https://alexschroeder.ch/wiki/2021-11-14_The_early_years_on_the_net
53: https://mastodon.online/@szczezuja/108902027541781265
NNCP lets you securely send files, or request remote execution, between systems. It uses asynchronous communication[55], so the source and destination need never be online simultaneously. NNCP can route requests via intermediate devices -- other NNCP nodes, USB sticks, tapes, radios, phones, cloud services, whatever -- leading to a network that is highly resilient and flexible. NNCP makes it much easier to communicate with devices that lack Internet connectivity, or have poor Internet.
55: /asynchronous-communication/
56: /sites-and-services-hosted-at-complete-org/
These sites are hosted on the complete.org server. Some are hosted with resources donated to non-profit organizations.
(c) 2022-2024 John Goerzen