💾 Archived View for mediocregopher.com › posts › micropelago-2023.gmi captured on 2024-08-18 at 17:23:50. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-05-26)
-=-=-=-=-=-=-
It's the end of the year, which means it's time for everyone's favorite category of posts: year-end reviews!
This year my primary focus has been Micropelago, a collection of open-source projects centered around the idea that it should be possible for a community of individuals to host their own online space using off-the-shelf hardware and residential internet.
Isle is the server component of the project, and is currently receiving the most attention.
It's been a big year for Micropelago, one worth commemorating with a wrap-up post like this one. At the end I'll also talk about where the project is headed, and how people might be able to get more involved.
As mentioned, 2023 has been pretty big for Micropelago. Here's some of the most notable steps forward.
Fun fact: at the beginning of this year the "Micropelago" project didn't actually exist! The project was called "cryptic-net" then, but I'd known for a while that it deserved a less insidious name. After quite a bit of brainstorming with betamike we eventually came up with Micropelago, a play on "archipelago", which is a term for a cluster of small islands in an ocean. The name conveys the idea of a connected cluster of individuals amidst a vast and chaotic expanse, but also of those individuals working together to support each other in a bright and fun way.
We have registered micropelago.net as the primary entrypoint for the project, but so far haven't used it for much.
Given that a network of islands forms an archipelago, it was a short mental leap to name the individual server component Isle ("island" being too commonly used). Upcoming projects like a GUI, mobile app, etc... will receive names with appropriately analogous meanings as well.
The MVP of the server component was actually finished in 2022, and if you were to look at the changelog since then it would seem that not much has happened. In fact most of my work has been in the nix-based build system, as I've been working to get Isle cross-compiling to other architectures. This proved to be quite a challenge for me, as I had to first learn quite a bit about nix, CPU architectures, GNU linker shenanigans, etc...
I did, in the end, manage to expand the supported build targets to 32 and 64-bit Intel/AMD CPUs, as well as 64-bit ARM, all for the linux operating system.
This work culminated in an official tagged v0.0.1 release. Woo!
If I'm honest I haven't properly _tested_ these builds on these architectures yet, but getting them to build is like 90% of the effort. This does, however, bring me to the next major milestone which was reached in 2023...
Isle now has a basic integration testing framework included in its codebase, with a handful of tests already specified for basic functionality. The major challenges here were centered around the fact that setting up an Isle cluster takes a handful of seconds, while at the same time it's helpful to have many granular tests which target one specific piece of functionality, leading to many defined tests. We don't want each of those tests to take a handful of seconds.
So the framework uses a basic dependency system where each test declares what kind of cluster it needs, and that cluster is only spun up if some previous test hadn't already needed it and done so itself. Similarly the framework handles tear-down of these clusters in a clean way.
With a real testing framework it should become much easier to properly test releases against all required architectures. The fact that I haven't done so yet is more a function of not having a usable 64-bit ARM machine than anything. I'm eyeing getting a PineBook in 2024, which will solve that problem.
EDIT(2024-08-02): the domani project frontend has been turned off. In the end I've decided it makes more sense to develop plugins to existing projects like Caddy rather than to re-invent the wheel. I've noted places where a link used to be but has been removed. The domani source-code can still be found at the link below, but I wouldn't expect it to be developed further.
While Isle is the most prominent project in Micropelago, I also spent a decent chunk of time working on another project in the collection:
Domani - The universal, zero-authentication hosting service (link removed)
The premise of Domani is that it's quite a bit of work to make a service publicly available via a domain name. You have to set up DNS properly, get a static IP or set up dDNS, figure out hosting, deal with TLS certificates, etc... Domani allows someone to take this sunk-cost and share its benefits with others.
Domani is essentially a reverse-proxy which can be configured via its web interface. When someone wants to add a new proxy target all they have to do is to point their DNS domain to Domani, as well as set a TXT DNS record with a specific hash of their desired configuration. From there Domani ensures that everything checks out, pulls down whatever content is being hosted (currently only git repos are supported), sets up TLS, and hosts the desired domain. The web interface of Domani guides the user through the required steps as simply as possible.
Domani supports both HTTPS and Gemini, and could support other frontends in the future as well.
It may seem that Domani is a bit orthogonal to Isle, but actually it will eventually be bundled into it. As noted already, it's not a trivial thing to make a service publicly accessible, and someone within an Isle cluster will have to have done so in order for the cluster to function (otherwise how will members access it?). Domani can take advantage of this work and allow members to host their own publicly-facing websites/gemini capsules via the public access point of the cluster.
But why create a new project for that in the first place? Since an Isle cluster can (and should) have multiple public nodes, a reverse proxy which is able to manage TLS certificates in a distributed way is required. Isle uses garage, an S3-compatible distributed storage service, as its data layer, which means that these TLS certificates would need to be shared via S3, and also deal with cache invalidation and other concerns using the tools at hand in Isle. I'm not aware of any existing projects which could fit these requirements (not to mention the soft requirement of gemini support), while Domani has been designed since its inception to eventually do so.
There have been a few small improvements made to Isle throughout the year as well, none of them deserving their own section.
With 2023 over and done with, I've got a lot that I'm looking forward to working on in 2024. In this section I'll list some high level items, roughly in the order I intend to complete them. The Isle project contains a roadmap which is much more detailed, for those interested.
Isle roadmap at time of writing.
Probably _the_ highest level goal in Micropelago is to make these tools available for average users. And on average users use Windows. Digging into this further: given that Micropelago is designed to work using computers hosted in users' homes, and that most users who have desktop computers which would be up for this task are gamers, and most gamers still use Windows, a Windows target is a hard requirement.
With this in mind I've been working hard to get Isle cross-compiled to Windows.
The first step in this journey has been to get garage cross-compiling to Windows, a step I've already begun this last month. It hasn't been easy. While I have finally managed to get a binary to be emitted, in testing the binary just silently fails without any output. So more work is needed there.
Depending on how painful this part is I will consider Darwin (Mac OS) support. While Mac users are generally more technically inclined, they are not the majority, and usually they have laptops. If anything Mac would be a target for a more slimmed down version of Isle which only accesses a cluster, but which can't be used for contributing resources (data persistence, public IPs) towards it.
While a windows command-line app is a required first-step, the ultimate goal here is a GUI. Average users can't use a command-line. The GUI MVP is going to be extremely simple, not much more than a frontend on the existing YAML configuration file which Isle uses. From there I would like to use it to streamline the process of joining a cluster, possibly introducing some kind of invitation string mechanism rather than the existing process of copying bootstrap files around manually.
A simple mobile app is probably _not_ in the cards for this year, but it's something on my mind anyway. I discovered this past year that the nebula mobile app is not actually open-source.
Nebula is the VPN component used by Isle.
But the mobile app is deliberately not open-source.
So I won't be able to build directly upon that. But it should be straightforward to build something basic from scratch which embeds the nebula binary (which _is_ open-source) and uses it to connect the phone to the Isle cluster. An improved invitation mechanism would go a long way here.
While we settled on a name this year, not much has really been done with it. Nor do we have a proper logo for the whole project, which is an obvious requirement. So far I've just been using the Teletubby baby.
While kind of a joke, I do think a sun graphic is probably what it's going to end up being. It's the only thing that feels right. I just have to actually do it.
With a logo done it would be good to set up a real website at micropelago.net, if only to briefly introduce the project and direct folks to the latest release page for now, but ideally also to host the documentation, which should get expanded to include text and video tutorials.
Another idea I've been toying with is that of hosting some kind of regular virtual meetup for the project. I know there are a few people who have expressed interest, but maybe don't realize the project isn't dead. These meetups would be probably quarterly to start with, with very informal content. Mostly chatting, getting people involved, figuring out how people who want to contribute can do so.
This has been the past, present, and future of the Micropelago project as of EOY 2023. At time of writing I want to make this wrap-up a yearly thing, but we'll see if I remember that want at the end of next year :) In the meantime I wish everyone reading this a happy holidays and new year!
-----
Published 2023-12-22
