💾 Archived View for skyjake.fi › gemlog › 2020-12_hooks-and-pipes.gmi captured on 2023-04-19 at 23:02:31. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Hooks and Pipes

There is a conflict between aspiring to stay true to Gemini's simplicity and adding support for all the real-world use cases plus the myriad of file/data formats out there. Lagrange v0.12 offers a mechanism that tries to strike a balance between extensibility and keeping things simple.

Keeping it simple and only supporting gemtext feels "pure", but it incurs a cost as the user may be frustrated to find out their favorite format can't be used. On the other hand, supporting "non-smol" formats — such as ones based on XML — undermines the cohesion and accessibility of the Geminisphere since a subset of clients won't be able to handle them.

One shouldn't ignore the technical implementation burden either. Every additional feature and format requires code, either written from scratch or sourced from third parties. The maintenance costs grow over time.

Hook me up

Fortunately, we can look to the wisdom of UNIX for a solution. According to the highest ideals of UNIX philosophy, a set of small co-operating programs are used to achieve complex objectives. Each small tool can focus on and master its well-defined responsibilities, and they can exchange streams of data with each other.

This is what I've implemented in Lagrange version 0.12. It is now possible to configure a set of external programs for preprocessing Gemini request responses. In a nutshell, a response is piped to a program chosen by its MIME type, and that program is free to rewrite the entire response (including the MIME type) as it sees fit. The feature is called "MIME hooks".

This is quite a long-winded way of saying that if you want to subscribe to RSS/Atom feeds, here, go do it yourself. 😁 Parsing XML is certainly infinitely more comfortable in, say, a Python script than in plain C. As long as you output a Gemini feed index page, Lagrange can use it for subscriptions and register the feed entries.

This feature goes much beyond RSS feeds, though. Another file format I've been wanting to support is MIDI music, but now you can just pipe it through Timidity via a simple shell script:

echo "20 audio/wave\r"
/usr/local/bin/timidity -Ow -o - -

Lagrange then plays back the MIDI file as if it actually contained PCM WAV audio. The fact that the data grows by a few orders of magnitude doesn't really matter since it's all in a local buffer in memory, and discarded once the audio player is closed. The possibilities don't end there: you can output an image, convert Markdown or HTML to Gemtext, do language translations, or scan the contents for keywords and perform actions accordingly. Let your imagination run wild.

Is privacy a concern here? No data leaves your computer unless you want it to, as there are no proxy servers you need to trust, and no hooks are configured by default so you are in full control of what happens. The external programs you choose to use receive only the response body and its MIME type and parameters. This is essentially automating what you'd have to do manually if the feature wasn't available.

Pipes all the way down

As features go this one is pretty experimental. Taken to its extreme, one could design the entire architecture of the browser around tiny programs that each perform a small task, but such a system would be difficult to maintain and optimize — never mind to debug. Not really a design where simplicity is valued.

MIME hooks provide a bloat-free avenue for client-side scripting. There's nothing particularly Lagrange-specific to these preprocessors so they should be generally useful for bringing content into the Geminisphere when run independently or on a proxy server. Of course, existing conversion tools should be quite easy to plug in: just read from `stdin` and write a Gemini response to `stdout`.

I would like to continue shipping Lagrange without any extra add-ons and tools. It should be useful on its own. This mechanism enables extensibility and adaptivity beyond my resources and abilities. Should some use case emerge as "too popular", well that is a prime candidate for integration into the client — after carefully considering if it's worth sacrificing some simplicity.

An Atom example

For the time being there is no GUI provided for configuration. You'll need to create a file called "mimehooks.txt" in Lagrange's config directory. For example, "~/.config/lagrange/mimehooks.txt":

Convert Atom to Gemini feed
application/xml
/usr/bin/python3;/home/jaakko/atomconv.py

Each hook is specified as three lines:

A human-readable label
MIME type/parameter regular expression
Command to execute, plus additional arguments each separated with semicolons

MIME hooks can be as simple as shell scripts, but in the case of an Atom feed it's good to use a more capable language. Here we use Python. The contents of "atomconv.py" are below (a naïve proof-of-concept):

import sys
import xml.etree.ElementTree as ET

def atomtag(n):
    return '{http://www.w3.org/2005/Atom}' + n

root = ET.fromstring(sys.stdin.read())
if root.tag != atomtag('feed'): 
    sys.exit(0)
feed_title = ''
feed_author = ''
feed_entries = []
for child in root:
    if child.tag == atomtag('title'):
        feed_title = child.text
    elif child.tag == atomtag('entry'):
        feed_entries.append(child)        
print("20 text/gemini\r")
print(f'# {feed_title}')
for entry in feed_entries:   
    entry_date = ''
    entry_title = ''
    entry_link = ''
    for child in entry:        
        if child.tag == atomtag('updated'): 
            entry_date = child.text[:10]
        elif child.tag == atomtag('title'):
            entry_title = child.text
        elif child.tag == atomtag('link'): 
            entry_link = child.attrib['href']
    print(f'=> {entry_link} {entry_date} {entry_title}')

With these in place, opening an Atom XML document in Lagrange will appear as if a 'text/gemini' page was loaded instead.

📅 2020-12-04

CC-BY-SA 4.0