This document describes various conventions and snippets of advice for implementing and using the Gemini protocol which, while not mandated by the protocol specification, are generally considered a good idea. If you're writing Gemini software or building a Gemini site, you should generally follow the advice given here unless you have good reasons not to.
Gemini servers need to inform clients of the MIME type of the files they are serving. The most convenient way for servers to figure out the MIME type of files is via the extension of the filename. These mappings are mostly well-standardised (and unix systems often have an /etc/mime.types file full of them), but the question remains as to how servers should recognise files to be served with the text/gemini type defined by Gemini.
Current Gemini servers seem to use .gmi or .gemini extensions for this purpose, and new servers are strongly encouraged to support one or both of these options instead of adding a new one to the mix.
Following the convention for webservers, if a request is received for a path which maps to a directory in the server's filesystem and a file named index.gmi or index.gemini exists in that directory, it is served up for that path.
Gemini servers do not inform clients of the size of files they are serving, which can make it difficult to detect if a connection is closed prematurely due to a server fault. This risk of this happening increases with file size.
Gemini also has no support for compression of large files, or support for checksums to enable detection of file corruption, the risk of which also increases with file size.
For all of these reasons, Gemini is not well suited to the transfer of "very large" files. Exactly what counts as "very large" depends to some extent on the speed and reliability of the internet connections involved, and the patience of the users. As a rule of thumb, files larger than 100MiB might be thought of as best served some other way.
Of course, because Gemini supports linking to other online content via any protocol with a URL scheme, it's still possible to link from a Gemini document to a large file served via HTTPS, BitTorrent, IPFS or whatever else tickles your fancy.
Gemini supports any text encoding you like via the "charset" parameter of text/* MIME types. This allows serving "legacy" text content in obscure regional encoding schemes.
For new content, please, please, please just use UTF-8. The Gemini specification mandates that clients be able to handle UTF-8 text. Support for any other encoding is up to the client and is not guaranteed. Serving your content as UTF-8 maximises its accessibility and maximises the utility of simple clients which support only UTF-8.
Redirects were included in Gemini primarily to permit the restructuring of sites or the migration of sites between servers without breaking existing links. A large, interconnected space of documents without such a facility inevitably becomes "brittle".
However, redirects are, generally speaking, nasty things. They reduce the transparency of a protocol and make it harder for people to make informed choices about which links to follow, and they can leak information about people's online activity to third parties. They are not as bad in Gemini as in HTTP (owing to the lack of cookies, referer headers, etc.), but they remain at best a necessary evil.
As such, please refrain from using redirects frivolously! Things like URL-shorteners are almost totally without merit. In general, think long and hard about using redirects to do anything other than avoid link breakage.
Clients may prompt their users for decisions as to whether or not to follow a redirect, or they may follow redirects automatically. If you write a client which follows redirects automatically, you should keep the following issues in mind.
Misconfigured or malicious Gemini servers may serve redirects in such a way that a client which follows them blindly gets trapped in an infinite loop of redirects, or otherwise has to complete a very long chain of redirects. Robust clients will need to be smart enough to detect these conditions and act accordingly. The simplest implementation is to refuse to follow more than N consecutive redirects. It is recommended that N be set no higher than 5. This is inline with the original recommenation for HTTP (see RFC-2068).
Cross-protocol redirects (i.e. redirects from Gemini to something else, like Gopher) are possible within Gemini, but are very heavily discouraged. However, misconfigured or malicious servers will always be able to serve such redirects, so well-written clients should be ready to detect them and respond accordingly.
It is strongly recommended that even clients which generally follow redirects automatically alert the user and ask for explicit confirmation when served a redirect to a non-TLS-secured protocols like HTTP or Gopher, assuming the client implements support for these protocols. This avoids unintentional plaintext transfers.
TLS 1.2 is reluctantly permitted in Gemini despite TLS 1.3 being drastically simpler and removing many insecure cryptographic primitives. This is because only OpenSSL seems to currently have good support for TLS 1.3 and so requiring TLS 1.3 or higher would discourage the use of libraries like LibreSSL or BearSSL, which otherwise have much to recommend them over OpenSSL.
Client and server authors who choose to support TLS 1.2 should ideally only permit the use of ciphersuites which offer similar security to TLS 1.3. In particular, such software should:
As a server author, you need to expect bots to visit your site. Some of them do not obey your /robots.txt file. Some of them relentlessly hammer your site. Some of them get trapped in endless loops in an attempt to index dynamic sites (games, for example).
Therefore, either make sure your server only serves a number of static files so that it can handle the bots, or hide the dynamic parts of your site behind a client certificate requirement, or block bots.
One way to deal with bots is to log the IP numbers of visitors to a file and let a separate process like fail2ban handle the blocking: write your fail2ban rules such that every access counts as a failed login attempt, and then pick a reasonable limit like 30 requests in 60 seconds (averaging 2/s but allowing for peaks). fail2ban now blocks any visitor exceeding that rate using the firewall. By default, the block is for 10 minutes; if you activate the “recidive” setup, then such blocks now trigger another rule, where you can define yet another number of hits, like 3 blocks in 1 hour. This second block might extend for a week, for example.
For visitors, getting blocked by the firewall means that their connections are simply refused. There is no explanation shown on their end. It’s definitely going to save you resources since your server doesn’t have to serve these requests, but occasionally innocent people are blocked without understanding what is happening to them.
Logging IP numbers is a bit of an issue since under the General Data Protection Regulation (GDPR) they count as personal information. You can of course argue that you really need them to keep the site up, but perhaps there are other options.
A solution without logging IP numbers means that the server needs to keep a list of most recent visitors in memory. Pick a reasonable limit like 30 requests in 60 seconds. In this case, the server needs 30 timestamps for every visitor. If there are 30 timestamps, it compares the oldest one to the current time and if they are less than 60 seconds apart, the server replies with a status 44 “Slow Down!” and a number of seconds that seems reasonable to you.
One way of handling bots that are forever patient and ruthless, is to increase the ban every time it is triggered. That is, for ever ban, remember for how many seconds you banned the IP number and when it violates the ban (ignoring the status 44), double the remaining time. So if you ban a visitor for 60 seconds, and they make another request within 60 seconds, ban them for 120 seconds, and so on. Maybe have an upper limit, say four weeks. Just in case somebody else ends up with that IP number.
It’s true. Perhaps there’s a shared server at that IP number. One of the users on that server writes a misbehaving bot and all are punished. If you are concerned about that, your server needs to move the dynamic content behind a client certificate requirement. There is no other way to identify particular users using Gemini.
Some bot authors have rented cloud servers using IP numbers from Amazon, OVH, Hetzner, and others. If your server blocks one of the IP numbers, they’ll switch to another one. How do we handle that?
If you can determine that a bunch of them belong to the same network, you can block the entire network. Switching IP numbers is now much less effective. This works quite well because there are “residential” networks for ordinary people, and “cloud server” networks, so chances are you’ll only be blocking cloud servers, not ordinary people like you an me.
One way of dealing with this is to make a reverse lookup. This only works for IPv4, for now. Assume that the offending IP number is 178.209.50.237, but you have other IP numbers currently blocked as well. You need to reverse that IP number, add it to “asn.routeviews.org”, and make a DNS query. You’ll get back a TXT record telling you which IP to ban.
The answer section:
This tells you that you can ban the entire network by banning the IP range 178.209.32.0/19.
Needless to say, if banning IP numbers is problematic, then banning entire IP ranges is even more problematic. Then again, an abundance of unsupervised bots is just as problematic.