Authors: Ben <benulo@systemli.org>
Dated: 2019-10-16
Like many people, I recently have been re-discovering the XMPP messaging protocol and ended up running my own server. In fact, this blog is essentially a result of that effort, though it functions independently from the XMPP service.
You see, I had used XMPP back in my college days when we used to call it Jabber and before today's most popular messaging platforms based their protocols on it. To me and many others, XMPP just meant jabber.org. (Sure, there were other servers, but one would expect the original implementation to be the leader and cutting edge.)
I kept using the protocol as much as was practical, though its practicality diminished greatly over the years. Over a decade passed where logging into jabber.org with Psi or maybe Pidgin was a pointless affair. The protocol used to interoperate with Google Talk, but that eventually came to an end when Google gradually became more and more evil. (And this is a true story: They literally removed the "don't be evil" clause from their official corporate policy.)
So, what was the point of using XMPP anymore? Nobody used it, and messaging had revolutionized and gone mobile; nobody needed desktop messengers, but mobile phone ones. Even an app like Skype could not compete with services like WhatsApp and Telegram. XMPP would be a joke by comparison, with a handful of Android clients of abysmal quality and not suited for mobile use and mobile connections.
However, over the years XMPP silently evolved and grew powerful. Someone randomly mentioned to me online that they not only still use it, but that it's awesome thanks to modern clients like Conversations. Eventually I too decided to use Conversations, but for that I needed a server.
As it turned out, jabber.org had dropped the ball. My account on their server still works even, but their server is not modern in the sense that it doesn't even support half the features Conversations needs to provide a great mobile experience on par with WhatsApp. (And yes, it really does this.)
I figured OK, I have what I need to run my own server. I just need to install the free software and let it do its thing. I think I had even played around with ejabberd once many years ago.
And that's where I fell into the trap that you might fall into as well. I knew that ejabberd existed and was available, but I thought, "Gosh, ejabberd is ancient! It's like the original XMPP server. And it's written in that probably obsolete language, Erlang." If jabber.org is obsolete, what would ejabberd be? jabber.org even stopped using ejabberd long ago in favor of a newer server.
That's why I did a quick search around, and Prosody caught my attention. I had heard of it once before, and at the time it had a pretty good reputation, and now it seemed to me like it was the most cutting edge and modern server. It certainly seemed to be the most currently popular, so people who know their stuff must all be using Prosody, right?
Well, Prosody got installed on my server the day I downloaded Conversations, and setting it up was indeed painless at first. It was simple enough to install and get up and running, and the base configuration was a breeze. I was able to log in right away and things were looking alright.
However, it didn't take long for me to realize that something was amiss. First of all, Conversations supports all these fantastic features that Prosody also supports, but they aren't ready out of the box. Some of the advanced features have to be enabled manually by loading "community" modules. That is, these are modules that implement some tricky features in the protocol that are not officially part of Prosody or supported by its devs, but are provided as a separate thing you can install made by community members who contribute them. Not to say that community modules are bad or anything like that, but the fact that they're separated is not totally desirable.
Now, Conversations is nice enough to tell you for the most part what is working on your server and what isn't. So the first step of enabling the modules I needed in order to have all the features I wanted was not too difficult. However, then problems started creeping in.
Some modules had dependencies to other modules that were not entirely clear. Some modules have zero documentation. Modules have their dependencies automatically loaded if you forget one, so that's not such a big deal, but some modules can conflict with each other, and you have to make choices as to which module is right for you. (For example, stream management for mobile devices.) But how do you make that choice when the models that you apparently have to choose between may not be sufficiently documented, or one is beta quality and may not work?
I spent hours and days tweaking my Prosody configuration to make Conversations happy. I talked to the Prosody people, Googled as many tutorials and configuration examples as I could find, and there was never any clear known-good setup. It was always just, "Well, maybe if you load this or that module it'll work this way." Every example config I looked at recommended different modules apparently without anyone really being sure which ones are needed and which ones aren't.
My hard work did yield some progress, and I got to the point where I had a config or two that should be perfect for my client, but there was just one show-stopper:
Without getting too technical, a mobile phone can't maintain a solid connection to the server all the time because mobile phones are very touch-and-go. The app gets backgrounded and put to sleep, the Internet comes and goes depending on your signal, and so on.
What Conversations is designed to do is work together with the server to understand that these things happen and not terminate your session and break your communications at the slightest hiccup. That means the client can lose connectivity and regain the original connection by letting the server know it's still there.
This is exactly what wasn't working with my Prosody server, and when Conversations tried to resume a preestablished connection it would just hang and never connect. This meant that it would frustratingly stop working after maybe half an hour from the time of connecting until I manually reset the connection. I could fix it by disabling that feature on the server, but then Conversations would always start a new connection to the server every time it needed to do something. That kind of inefficiency is bad for the phone's battery life, as it's working too hard on something it shouldn't have to be doing.
In the mean time, some of the connectivy problems manifested as message delivery errors, where Prosody would tell my client that messages had failed to deliver because of timeouts, but the other person actually was getting them. This is also a known issue with mobile configurations that a certain module can address, allowing Prosody to ignore apparent delivery failtures.
As it stood, my server wasn't cutting it, and after talking to the nice Prosody guys, the eventual conclusion that we were able to come to was that my modules were outdated, and I was encountering a bug that had already been fixed. The frustrating part there was that when I installed Prosody in the first place I made sure to check its version, as the whole point was to use the latest tech. My Prosody version was shown as the last minor release, so the package was fairly recent.
So in the end, I could not use my distro's Prosody packages if I wanted it to work, and I didn't really feel like manually installing things outside of my package manager, possibly experimenting with nightly builds or something like that. In the mean time I had heard a curious thing, which was that actually ejabberd was not a dead project, but in fact had lept ahead and was one of the most advanced servers out there.
I was a little skeptical, still clinging to the idea that ejabberd was outdated or at best inefficient, as Prosody claims to be faster or lighter on resources. However, after some discussion with others, I also learned that the Conversations project openly prefers and recommends ejabberd. But why?
Well, at this point I had to try it. It seems like a scary program that's difficult to set up, but at least I could tell my package manager to install it, so at worst I might waste a couple hours attempting to run it, let alone configure it. In a single afternoon I had Prosody fully replaced with a working ejabberd install, with virtually all features working out of the box.
You see, ejabberd is a very sophisticated piece of software. What it does is actually quite amazing (thanks in no small part to features built into Erlang), and the effectiveness of its design has not diminished with age. All the advanced features that a client like Conversations needs are built into the server and available out of the box. They're still modular by design, but they are included in the base package, and the integration is tight. Its cohesive design saves you a lot of pain.
Was it as easy to set up as Prosody? Not exactly, but I'd say it's only marginally more difficult. After installing the ejabberd package on my system, I had to resolve a slight misconfiguration in the server's config file that wasn't obvious at first. (It was a bind problem that may have been an oversight by the package maintainer.) The log output was helpful to that end, and once the server was able to load and run, adding user accounts was simple.
Over time, what I learned to appreciate about ejabberd is how easy it is to work with. There might be a bit of a learning curve at first, and its many features and endlessly configurable behavior is truly daunting, but it mainly just works as-is and needs only a light touch to fit your needs perfectly.
Contrary to what I was led to believe, ejabberd is definitely fast and efficient. It is also stable and reliable, and seemingly handles whatever gets thrown at it. It can do some things that no other server can, and it's flexible and straightforward. My personal opinion is that it's the opposite of Prosody in the sense that Prosody starts out easy but gets harder when you need to tweak it, but ejabberd is a little harder to start out with but makes your life much easier down the line.
ejabberd can also do incredible things like hot-reload the server's configuration while the server is running, and there's no break in service. You can literally reconfigure almost all of the server's essential functionality with no downtime. (Want to reconfigure ports? Change modules? No problem!) Less impressive but greatly appreciated is the server's web interface, which makes user management and status monitoring as simple as possible.
On top of all this, my version of ejabberd was not even the latest one at the time I installed it. (It has since been updated.) Some major XMPP servers are using an even older version of it and still do everything Conversations needs. I also have since customized the server to provide the exact functionality I want, and I'm completely satisfied. There's maybe only one or two features left that I could enable, and I technically don't need them. (For example, XMPP over HTTP if I wanted to offer a web client.)
Any downside? Well, my little server has severely limited storage, and ejabberd likes to use storage space. For one thing, message history never expires, so that'll grow indefinitely unless you do something about it. A simple command to the server in a cron job handles that nicely. The other thing is that the server by default logs basically all network activity, which will get out of hand even with normal activity levels. Knocking the log level down a notch solves this easily.
Is Prosody bad? Not at all. Just make sure you're using the latest version possible, and keep it up to date as well. ejabberd and Prosody are probably the two best XMPP servers out there, though to be fair I have not tried the others to see how they truly compare. However, if you're seriously thinking of running an XMPP server of your own, just use ejabberd. I really can't recommend it highly enough, and I'm no stranger to running network servers. It's really not every day that you get a chance to discover software as great as this.