💾 Archived View for rosenzweig.io › gemlog › the-federation-fallacy.gmi captured on 2021-12-17 at 13:26:06. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-04-08)
-=-=-=-=-=-=-
Throughout the free software community, an unbridled aura of justified mistrust fills the air: mistrust of large corporations, mistrust of governments, and of course, mistrust of proprietary software. Each mistrust is connected by a critical thread: centralisation.
Thus, permeating the community are calls for decentralisation. To attack the information silos, corporate conglomerates, and governmental surveillance, decentralisation calls for _individuals_ to host servers for their own computing, rather than defaulting on the servers of those rich in data.
In the decentralised dream, every user hosts their own server. Every toddler and grandmother is required to become their own system administrator. This dream is an accessibility nightmare, for if advanced technical skills are the price to privacy, all but the technocratic elite are walled off from freedom.
Federation is a compromise. Rather than everyone hosting their own systems, ideally every technically able person would host a system for themselves and for their friends, and everyone's systems could connect. If I'm technically able, I can host an "instance" not only for myself but also my loved ones around me. In theory, through federation my friends and family could take back their computing from the conglomerates, by trusting me and ceding power to me to cover the burden of their system administration.
What a dream.
Federated systems are all around us. The classic example is e-mail. I host my own email server, so I have the privilege of managing my own email address. According to the ideas of decentralisation, for an e-mail user to be fully free, they should host their own e-mail server.
But do _you_ host your own mail server? Do your friends? Does your _grandmother_? Setting up a mail server often is time-consuming, ad hoc, and brittle; despite technical literacy and the hours I poured in, I continue to have problems with my e-mail delivery. Of course, e-mail is _technically_ federated, but for pragmatic reasons, most people's personal e-mail is controlled by a centralised service like Google Gmail. Is Google a small individual you know personally?
There's no surprise large companies administer most e-mail accounts; it is the expected consequence of _economies of scale_. Due to the overhead of running a mail server, it makes economic sense to centralise. Decentralisation, while certainly possible, is impractical for email; accordingly, there are considerable privacy and censorship risks for many e-mail users, submitting to the rules of a massive service provider.
But maybe, due to its configuration complexity, e-mail could be an outlier. What about the federated chat protocol, XMPP?
Well? How many people do _you_ know who run their own XMPP server?
The protocol family claims over one billion users; you may unwittingly be one of them. But almost every one of those users is connecting not to the federated, open paradise, but rather to a walled garden. Of course, a few users connect to their personal server via a free software client, but most connect to _Facebook's_ walled garden, via _Facebook_'s app: WhatsApp.
Yes, _internally_ the ever-popular, ever-proprietary WhatsApp was once a federated protocol. Inside WhatsApp is XMPP, but WhatsApp users are isolated from non-WhatsApp users. Likewise, users preferring free software for practical or ideological reasons are isolated from their friends on WhatsApp. Federation was baked into the genes of this protocol, but while the original "freedom-preserving" chat system claims a few technically advanced adherents, the freedom of the masses was unfortunately lost to corporate interests.
Chat protocols aside, arguably the web itself is a classic, if overlooked, example of a federated system. The web, as a collection of interlinked websites backed by the Hypertext Transfer Protocol, contains the two essential features of a federated, decentralised system. Every author is their own publisher with their own web server, decentralising the web. Similarly, authors are encouraged to link their pages to other pages on other parts of the Internet, hosted by other authors, and in exchange, other pages tend to link to them, creating an interconnected -- federated? -- system. The fabric of the Internet itself follows the decentralised technical structure.
Yet the web is no friend of freedom. The overwhelming majority of web traffic is to commercial silos without users' best interests at heart. It is certainly possible, if complicated, to host a web server oneself. But practically, the English-speaking web is undeniably centralised around Silicon Valley enterprises, the same enterprises that pose considerable threats to user freedom without the possibility of informed consent. Decentralisation specifically seeks to push _back_ against these Internet masters, yet these centralised giants are operating within the technical framework of a decentralised system.
Admittedly, the web is nebulous as an example of federation. What do we make of Mastodon, a freedom-respecting federated alternative to Twitter, whose network of servers hosted by diverse individuals and organizations is lovingly known as the "fediverse"?
Mastodon is perhaps the most promising example of a federated system staying true to its grassroots ideals, so far lacking the tell-tale signs of large-scale corporate tampering. If any federated system can work, let it be Mastodon.
But let's look at the data first. In comparison to some other federated systems, a comprehensive list of federated Mastodon instances (servers) is available for browsing online at instances.social. Some non-Mastodon microblogging services are mixed in, as they federate with Mastodon, knitting together the larger "fediverse". Nevertheless, some simple scripting allows the machine-readable list to be downloaded and processed.
Analysing the data, after filtering out instances with no users, we see there are 3070 listed instances. So far, sounds good -- that is 3070x more theoretically independent instances than Twitter.
Looking at user count, we see there are just under two-million user accounts registered. In other words, there is an average of 642 users per instance. Already the federation fantasy is feeling shaky. How could one system administrator maintain close friendships with over six-hundred people? How could so many people trust a single administrator with their private status updates and direct messages? It seems improbable, if not impossible. With the amount of time required to maintain intimacy with 640 people, there would be no time left in a day to maintain a large Mastodon instance!
But what's really striking about the data is the _distribution_ of users across the instances. In the federation fantasy, given 3,070 instances, we hope that 50% of the user base is spread across 1,535 instances, and the other 50% across the other 1535 instances. That is, in fantasy land, every instance would host an equal number of users (642).
In reality, guess how many instances encompass half of the user base. Maybe 1,000? Alright, there are some big instances in there, so perhaps 100? Well, there are a lot of really tiny instances mixed in, so possibly only 20?
The answer?
Just *three* instances encompass 50.8% of users.
The most popular instance in the Mastodon universe sports over a _half a million users_. The runner up is mastodon.social, the flagship instance run by the developer, clocking in at just over three-hundred thousand. These two instances alone encompass 41.2% of users.
It is true that many Mastodon users have multiple accounts spread out across different instances, so it may be more reliable to check by a metric like number of status updates. Unfortunately, results there are similar: half of reported status updates correspond to a mere five instances.
Overall, Mastodon certainly encourages people to use its _technical_ ability to host their own instances for themselves, their friends, and their shared interest groups. True, federation technically allows all 3,070 instances to connect to each other, subject to the policies of each instance owner. But practically, what have we achieved when the typical Mastodon user has an account on one of just _three_ instances?
If we define its success by decentralisation, unfortunately -- unsurprisingly -- Mastodon has failed. Unless you use an obscure micro instance, what good is decentralisation anyway?
Visually, the above graph shows user count on the Y-axis, where the X-axis corresponds to the instance popularity ranking. So, the most popular instance is the left-most point, the second most popular instance is a hair to the right of it, and all the way on the right is the tiniest instance. In total, the plot shows the distribution of users on (primarily Mastodon) instances
In the federated ideal, where all instances are created equal, the graph should be a horizontal line showing each instance with 642 people.
Of course, due to substantial inequality between instance size, we expect to see a power distribution, with a spike on the left that quickly falls and tapers out. Power laws govern much of the real world; many phenomena behave according to this unequal distribution.
But look at that graph. Calling this distribution a power law would be generous to say the least. There is a _massive_ spike corresponding to just a few instances, and the rest of the graph is nearly invisible to the naked eye, so tiny and so overshadowed by just a few giants. Frankly, this distribution is closer to the Dirac delta function than a power law.
Ultimately, there are two types of production-quality networked systems: those designed for centralisation, and those designed for federation. But from the tales of e-mail, XMPP, the web, and Mastodon, it is clear that _federation does not mean decentralisation_. Each federated system analyzed above only became production-quality and accessible to the masses at the immense cost of _centralisation_. Each service retains the theoretical ability to federate with tiny self-hosted servers, but the vast majority of users are _de facto_ concentrated about a few major servers. E-mail concentrated on Google, XMPP concentrated on Facebook, the web concentrated on Silicon Valley, and Mastodon concentrated onto a few flagship instances.
Indeed, it seems all networked systems tend towards centralisation as the natural consequence of growth. Some systems, both legitimate and illegitimate, are intentionally designed for centralisation. Other systems, like those in the Mastodon universe, are specifically designed to avoid centralisation, but even these succumb to the centralised black hole as their user bases grow towards the event horizon.
Consequently, it is not enough to build systems that "can" federate: all four case studies above _can_ federate but de facto stay centralised. Nor is it enough to "save ourselves", self-hosting our own decentralised digital islands, while ignoring the reality of the masses. We cannot close our eyes and rest, content with freedom in our personal bubble, ignoring the reality of our non-technical friends and family who do not enjoy the same luxuries of privacy and free speech. We cannot ignore their struggles after resolving our own, justifying our behaviour to ourselves given that there is no technical obstacle to their digital freedom "if only" they abandoned their convenience and dedicated themselves to learning system administration and software debugging. With or without decentralised free software, within the technocracy our non-technical loved ones are barred from liberty over their own lives. Decentralisation is certainly better than dependence on centralised corporate conglomerates, but _for whom?_ A society free for the few is a society in chains.
Nevertheless, as we mourn the centralised fates of these promising systems and yearn to improve the future, we must understand that _centralisation is inevitable_. For those of us immersed in the decentralised dream, this fact is uncomfortable to face, but until we do, we will never move on to build systems that are truly free. The fact is, as networked systems designers our choice is rarely "concentrated power or dispersed power?", but rather "where will power centralise?" If we quixotically opt to cede power over our creations via decentralisation, inevitably someone else will fill the power gap.
Alas, naive decentralisation is an experiment in anarchy. Anarchy, whether real or cyber, is a zone where life is "nasty, brutish, and short"; there is no guarantee of true freedom. In any anarchy, soon the power hungry will fill in the void. Remember, history teaches these new despots often establish totalitarian regimes no better than those originally overturned. Did microblogging suffer a parallel fate? If the explosion of Mastodon dethroned Twitter, it did so at the cost of establishing a new instance as the new king.
For freedom's sake, face the facts: federation is dead.
But there is hope. In political history, the birth of free nations typically contains three stages: dictatorship, briefly overthrown to anarchy, reformed to democracy. Democracy balances the will of the people with the efficiency of centralisation. Democratic freedom is incompatible with an oligarchic authority, accumulating power imposing their will on others. But neither is democracy a free-for-all; some degree of centralisation is acceptable and even useful for protecting liberty, provided it's centralised around a legitimate, democratic institution.
So it is in cyberspace. As the Internet blossomed into chains, the lucrative, exploitative practices of the Silicon Valley giants created digital totalitarianism, a system offering profits for the few at the expense of freedom of the many: **information dictatorship**.
The decentralisation movement understands this oppression as a consequence of unjust organized power, and platforms like Mastodon are successfully overthrowing this information dictatorship, in its absence creating **information anarchy**.
While the story does not end with anarchy, it need not end with a return to the status quo or to the aristocracy of the technical elite. No, in the power vacuum left by the victory of decentralisation lies the opportunity to create something new, something beautiful, something promising true digital freedom for everyone. After the fall of the information dictatorship, balancing in our fingertips is the precious opportunity to create **information democracy**.
Like its real world counterpart, information democracy is partially centralised for efficiency but encourages community participation for legitimacy. This primarily centralised democratic model is the only realistic option, evident from the structure of free nations in the real world. Yes, there is centralisation required, but we cannot continue to paint centralisation as the virtual bogeyman. Spreading fear is a natural knee jerk reaction to the illegitimate corporatocracy, but we cannot succumb to fear. Whether physical or virtual, centralisation is _not evil_; it is merely morally neutral.
That said, when we ultimately do centralise, we must ensure that the central organization belongs to us, the users, not to special interests. On this point, both corporate and decentralised services alike fail. Yes, corporate interests are oligarchies, bowing to money, but so too is the decentralised technocracy, bowing to arcane technical know-how. In a truly democratic digital space, participation must be accessible to _everyone_, not just those with money or expertise. If my grandmother cannot participate in the administration of the technology she uses, her best interests may not be adequately represented by the technology. Whether she is beholden to a corporation or to a system administrator, that is not digital freedom.
These features distinguish information democracy from its predecessors, creating a system not only more practical but also freer than the federated anarchy. In any democracy, centralised power is wielded to protect freedom, but in decentralised anarchy, no power is wielded at all. Anarchy fantasizes that freedom protects itself; inevitably, life in anarchy is life in chains, for any freedom gained is temporary as the system collapses under the tragedy of the commons and the paradox of tolerance.
Granted, information democracy is not a perfect system; in a virtual world composed primarily of information oligarchies, it is natural to be wary of its potential to stray from its ideals. Nevertheless, the few flaws of information democracy directly mirror the flaws of physical democracy; most objections to information democracy were raised centuries ago during the spread of real-world democracies. Indeed, democracy is not perfect, but it is the only just system we have. Fear cannot blind us, painting perfect as the enemy of good. However flawed, democracy is the only option for a free world.
Without further ado, if proprietary services are dictatorships and decentralised services oligarchic anarchies, imagine democratic microblogging: a flagship instance with a clear set of liberal rules, reached by mutual consensus in the community. When necessary, these rules are enforced by an elected moderation team voted in by the community, though generally bureaucracy should be minimized. The platform is entirely free software, so anyone technical can contribute via code. The server exposes a public API, allowing third-party clients to flourish without encumbering progress on the "official" client. Similarly, although the service is primarily centralised, the servers may permit federation if relevant. Ideally, when sensitive data like private messages is involved, communications are end-to-end encrypted and possibly even peer-to-peer, minimizing the centralised server's power.
Sound good? These criteria are non-exhaustive but illustrate the potential for information democracy. More importantly, sound familiar? Many of these ideals are _already embodied by Mastodon_. Mastodon may not have lived up to its federated fantasy, and judging it by its own meter stick of decentralisation, it failed. But judging it by the standards of an information democracy, regardless of its inadvertent centralisation, Mastodon is _a success story_.
True, Mastodon is de facto centralised, but despite the size of the largest instances, it retains the ability to federate with other Mastodon instances. Further, Mastodon is able to federate with other free software friendly networks via a pair of common protocols, creating the familial fabric of the "fediverse". Centralization and federation can certainly co-exist in harmony to improve efficiency while retaining user choice.
Nevertheless, the most successful information democracy is orders of magnitude larger than Mastodon. A centralised site with a documented set of rules reached by community consensus, hosted by a non-profit funded by donations from the user base itself. A community governed by administrators elected by participants in the community. A site whose source code is licensed as free software, allowing specialized adaptations to flourish, complementing rather than harming the main instance. A site that embraces free culture to ensure its fruits are accessible to all of humanity in perpetuity.
Wikipedia.
The Wikipedia projects embody the ideals of information democracy. Yes, Wikipedia is centralised; indeed, its software does not directly federate with other smaller wikis, many of which mirror Wikipedia articles -- and that's okay. Wikipedia derives its user freedom not from decentralisation, but from participatory democracy structures. Anyone can edit; with some exceptions to keep logistics manageable, a new user's first edit is valued as much an administrator's one-millionth. Typically, conflicts are resolved not from a top-down dictatorship, despite the centralisation, but rather from bottom-up consensus seeking. Wikipedia is not an experiment in democracy, but it nevertheless operates as an information democracy, no experimentation needed.
To protect the digital freedom of _everyone_, not just the wealthy or the technically inclined, we need **information democracy**. To see the future of Internet liberty for the few, look not to Silicon Valley, for overwhelming commercial interests can never adequately protect the user. To see the future of liberty for the many, look not to the obscurity of XMPP, for arcane technical voodoo can never be wielded by those who need it most. To see the free future, look to Wikipedia.
Whether online or in the real world, rejecting dictatorships is not enough for freedom.
We must endorse democracy.
_Thank you to April, Connor, Florrie, and Natalia for soundboarding and reading early drafts._
