💾 Archived View for colincogle.name › blog › dovecot-with-apple-push captured on 2024-12-17 at 10:08:37. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
Colin Cogle's Blog
==================
By adding a few things to your Dovecot IMAP server, you can have instant new mail notifications on your Apple devices.
Written October 19, 2024.
Updated November 14, 2024 with corrected GitHub links.
Self-hosting your email can be a challenge. It's also a great learning experience if you want to understand system administration and see firsthand how email works. There are many tutorials that show you how to set up your own email server, but if you can follow the directions to install the apps, configure your DNS zone, set up the big three acronyms (SPF, DKIM, and DMARC), and get your IPv4 and IPv6 address not frowned upon by the major email providers, then you're left with a fully-working email server that works great with desktop apps (and you can install your own webmail, if you're into that).
However, I have an iPhone, and I've found that my email doesn't always update as quickly as I'd like. Having been in the Apple ecosystem for a while, I know that Apple is a big fan of the IMAP IDLE extension, which lets your mail client keep a persistent connection open to your mail server. When a new message arrives, your mail app finds out instantly and downloads it immediately. Research shows that Apple's three big mobile operating systems — iOS, iPadOS, and watchOS — do support IMAP IDLE, but only when plugged in and on Wi-Fi. When they are unplugged or on cellular data, that persistent TCP connection to your email server becomes a battery drain and a needless (albeit minuscule) consumer of metered data plans. Thus, iOS (and its variants) will refuse to use IMAP IDLE most of the time, falling back to fetching new mail a few times per hour or whenever you open the Mail app. This is okay, but what if you're waiting for those stupid, less-secure emailed MFA codes? You might be waiting a while.
Despite this, Apple's own iCloud uses IMAP, yet it doesn't suffer from this limitation. Why is that?
To conserve battery, minimize cellular data usage, and maintain performance, iOS devices (with few exceptions) do not allow apps to keep persistent background connections to servers. Instead, Apple created the Apple Push Notification Service. An iOS device keeps a singular connection open to the APNs, and all app developers funnel their notifications through there. When the phone gets a notification for an app, the app is allowed to wake up and perform a little bit of related work.
Apple wisely used this feature for their own email servers. Fortunately for us, they didn't follow Microsoft's lead and invent a monstrosity; rather, Apple decided to use the open-source Dovecot POP3/IMAP server to power their infrastructure. To work around their own self-inflicted problem for iOS, they wrote a custom plugin for Dovecot that advertises a new feature called **XAPPLEPUSHSERVICE**. When someone receives a new email, Dovecot takes the message and puts it in the user's inbox folder, then sends a push notification via APNs. The user's iOS devices receive this push, then make an IMAP connection to iCloud's servers to download that new message.
It's absurdly brilliant. If you don't get a lot of email, then there's no reason for your phone to keep wasting power and data to log onto the mail server for no reason. If you do get a lot of email, well, you'll get a notification almost instantly. However, what isn't brilliant is that Apple doesn't seem to have released the full custom software suite as open-source. Perhaps it never occurred to them, or perhaps it's an invisible impetus to get you to use iCloud for your mail, contacts, and calendars. Fortunately, a dedicated programmer, Stefan Arentz, was able to reverse-engineer all of this, and has released his own Dovecot plugins that you can add onto your own email server. Forks by Frederik Schwan worked better for me, and those are what I recommend.
It's worth noting that recent versions of macOS (since 10.7, “Lion”) also connect and listen to the APNs, though without the network limitations of iOS, it's merely a value-add for macOS.
Yet, following the documentation on GitHub didn't yield a working product for me, so that's why I'm writing this tutorial. May you learn from my mistakes.
This article assumes that you've got a fully-functioning Dovecot server somewhere, version 2.2.19 or newer. Some of these extra apps will need to be built from source, so make sure that you have a working C compiler and a copy of Git, as well as CMake and the Dovecot development libraries. If you're using a Debian-based Linux, you should be able to do something like this:
apt build-dep dovecot-core apt install build-essential cmake dovecot-dev git golang-go
Download and install the Dovecot development libraries, as well as all the tools we'll need to build some code. This shouldn't take too long, and if you're tight on space, you should be able to apt remove these after we're done.
To start, we will need a copy of a server app called `dovecot-xapsd-daemon`. There are a few forks out there, so make sure we're using the one by Frederik Schwan (freswa). Be sure to use his app; there are some forks, but we need both of the “official” versions.
GitHub - dovecot-xaps-daemon by freswa
Clone the repository and build it. Finally, we'll need to install its various pieces by hand, as unlike a lot of open-source projects, there doesn't seem to be an install script.
git clone https://github.com/freswa/dovecot-xaps-daemon.git cd dovecot-xapsd-daemon go build -o xapsd # Install the app. sudo ln xapsd /usr/bin/xapsd # Create the app's user account and home folder. sudo useradd --create-home --home-dir /var/lib/xapsd --shell /bin/false --user-group xapsd # Install the configuration file. mkdir /etc/xapsd/ /var/lib/xapsd cp configs/xapsd/xapsd.yaml /etc/xapsd/xapsd.yaml # Install the systemd unit file. cp configs/xapsd/xapsd.service /etc/systemd/system/ sudo systemctl daemon-reload # Install a systemd-tmpfiles configuration. # If you don't have this installed, that's okay. cp configs/xapsd/xapsd.tmpfiles /etc/tmpfiles.d/ 2> /dev/null
We've copied over a sample configuration file, `/etc/xapsd/xapsd.yaml`. Go ahead and open that up in your favorite editor. We'll need to edit one file — but first, we need to put our hashed password into that file. You can't just type it in cleartext (thankfully!) nor can you hash it yourself. Use the xapsd app itself to hash your password.
xapsd --pass | tee --append /etc/xapsd/xapsd.yaml
Please enter the password -> passw0rd This is the hash -> 8f0e2f76e22b43e2855189877e7dc1e1e7d98c226c95db247cd1d547928334a9 For security reasons, we don't fill in the hash automagically. Please do so yourself.
(The unadulterated password is printed to the screen, so don't type it while anyone is peeking over your shoulder. But, the tee command will make sure it winds up in our configuration file.)
Now, open the `/etc/xapsd/xapsd.yaml` file in your favorite editor. We'll need to put the password hash in the right place, and add our username, too.
# set the loglevel to either trace, debug, error, fatal, info, panic or warn # Default: info loglevel: info # xapsd creates a json file to store the registration persistent on disk. This sets the location of the file. databaseFile: /var/lib/xapsd/database.json # xapsd listens on a socket for http/https requests from the dovecot plugin. This sets the address and port number of the listen socket. listenAddr: '[::1]' port: 11619 # […] # Notifications that are not initiated by new messages are not sent immediately for two reasons: # 1. When you move/copy/delete messages you most likely move/copy/delete more messages # within a short period of time. # 2. You don't need your mailboxes to synchronize immediately since they are automatically synchronized when opening the app # If a new message comes and the move/copy/delete notification is still on hold it will be sent with the notification for the new message. This sets the interval to check for delayed messages. checkInterval: 20 # Set the time how long notifications for not-new messages should be delayed until they are sent. Whenever checkInterval runs, it checks if "delay" <= "waiting time" and sends the notification if the expression is true. delay: 30 # To retrieve certificates from Apple, we need to login with a valid Apple Account. The account's email must be given in cleartext, but the password has to be hashed before sending it. To not leak working credentials on running servers, we do not accept the cleartext password here. appleId: colin@colincogle.name # use `xaps -pass` to calculate the hash of the Apple Account password. appleIdHashedPassword: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
(You will need to fill in your Apple Account username (email address) and password. Fortunately, whether you use Apple's basic MFA or you have Advanced Protection enabled, this single-factor usage will work just fine. It'll get a certificate that will function as an additional factor in the future.)
Once you've got the text file configured, it's time to start the daemon.
sudo systemctl --enable now xapsd.service
(If you're one of those anti-`systemd` people, that's fine, but you'll need to make your own init script.)
As soon as you start the app, it will connect to the Apple Push Notification Service and request a client certificate automatically. You should get an email confirming this (but you won't get a push notification because there are more steps). If you do not get the email, check the logs for errors and troubleshoot as needed.
Next, you will need to download and compile the Dovecot plugin `dovecot-xaps-plugin`.
GitHub - dovecot-xaps-plugin by freswa
git clone https://github.com/freswa/dovecot-xaps-plugin.git cd dovecot-xaps-plugin/ mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=Release sudo make install sudo cp xaps.conf /etc/dovecot/conf.d/95-xaps.conf
(Download, build, and install the Dovecot plugin. Then, we'll copy over a sample configuration file. We will need to make some edits to it, though.)
Now, let's edit the configuration file. This file assumes that the daemon is listening on a UNIX socket, but it doesn't do that anymore. It binds to IPv6 localhost, so we need to set the `/etc/dovecot/conf.d/95-xaps.conf` file as I've done below.
protocol imap {
mail_plugins = $mail_plugins notify push_notification xaps_push_notification xaps_imap
}
protocol lda {
mail_plugins = $mail_plugins notify push_notification xaps_push_notification
}
protocol lmtp {
mail_plugins = $mail_plugins notify push_notification xaps_push_notification
}
plugin {
# Defaults to /var/run/dovecot/xapsd.sock
xaps_config = url=http://[::1]:11619
# Defaults to NULL. Use if you want to determine the username used for PNs from environment variables provided by login mechanism. Value is variable name to look up.
# xaps_user_lookup = "colin@colincogle.name"
push_notification_driver = xaps
}
(You only need to change the xaps_config line. Depending on your Dovecot configuration, you might have to set the xaps_user_lookup option as well.)
Save that file and restart Dovecot! The next time your phone connects to your email server, it'll notice that Dovecot is advertising the **XAPPLEPUSHSERVICE** feature and act accordingly. The only thing to do now is to unplug your phone, lock it, send yourself an email, and see how little time it takes to get that pop-up on your screen.
One thing I learned during my testing is that, unlike the kind of push notifications you're used to getting, these ones are invisible. They do not contain the subject line, the sender name, or anything about the new email you've received. None of that information gets relayed through Apple. Your server is merely sending a notification to your phone, via APNs, that it should check for new email. As Stefan writes:
Each time a message is received, `dovecot-xaps-daemon` sends Apple a TLS-secured HTTP request, which Apple uses to send a notification over a persistent connection maintained […] between the user's device and Apple's push notification servers.
The request contains the following information: a device token (used by Apple to identify which device should be sent a push notification), an account ID (used by the user's device to identify which account it should poll for new messages), and a certificate topic. The certificate topic identifies the server to Apple and is hardcoded in the certificate issued by Apple and setup in the configuration for `dovecot-xaps-daemon`.
By virtue of having made the request, Apple also learns the IP address of the server sending the push notification, and the time at which the push notification is sent by the server to Apple.
While no information typically thought of as private is directly exposed to Apple, some difficult to avoid leaks still occur. For example, Apple could correlate that two or more users frequently receive a push notification at almost the exact same time. From this, Apple could potentially infer that these users are receiving the same message. For most users this may not be a significant new loss of privacy.
Thus, the invisible notification says little more than, “Check your email.” When your phone receives the notification, it logs onto the IMAP server and downloads the message. Once it has that, iOS takes the message metadata and creates the notification that you will see. If your phone can't contact your email server (for example, if IMAP is only available over a VPN that's not connected, or only over IPv6 while you're on an IPv4 “island”), then you will not see anything on your phone at all.
If you've read this far, and followed along at home, you have now added the **XAPPLEPUSHSERVICE** feature to your Dovecot IMAP server. The XAPS daemon and plugin are both running and communicating with the Apple Push Notification Service, and as far as I can tell, the certificate Apple minted for you will be renewed automatically. While I will laud Apple for making their own private, secure, and efficient version of one of Microsoft Exchange's favorite features, they failed to go as far as properly releasing this plugin to the open-source community. I'd also like to see this become an official Dovecot plugin, but I hope that the trial and error I put into this article has helped you to make your own email server just a little bit better.
==================================================
An earlier version of this article included links to the wrong repositories. Those two apps have been forked so many times that I saved the wrong ones. I'd shared the originals by Stefan Arentz, when I meant to share the forks by Frederik Schwan. Oops.