💾 Archived View for lyk.so › systems › e-mail captured on 2021-11-30 at 20:18:30. Gemini links have been rewritten to link to archived content

View Raw

More Information

➡️ Next capture (2023-01-29)

-=-=-=-=-=-=-

e-mail

Updated 2021-09-13

latest

I use mblaze, isync/mbsync, msmtp, and some custom scripts for handling e-mail.

mblaze is a suite of utilities for managing maildir mailboxes, in the Unix spirit of simple, combinable tools.

mbsync synchronizes IMAP4 and maildir mailboxes.

msmtp is a simplified sendmail alternative.

I also secure the passwords for my e-mail using two hardware devices: a physical password manager (the Mooltipass) and a PGP-capable crypto fob (the Ledger Nano S), which I also use for managing my SSH keys.

In short, I:

1. Set up trezor-agent and created a GPG key for encrypting the passwords file on the hard disk.

2. Used my gpgkv script to create an encrypted TSV file with my e-mail addresses and their passwords at ~/.mbsync-passwords.

3. Created a dedicated "mbsync" user.

4. Set up /etc/sudoers to allow my user to run mbsync and msmtp as mbsync without a password.

5. Ran my load-mbsync-secrets script, which decrypts .mbsync-passwords using my Ledger and loads them into mbsync's kernel keyring. (I have to run this after every reboot as well.)

6. Set up .msmtprc and .mbsyncrc files in /home/mbsync with password commands allowing them to load the passwords from the user's keyring.

7. Set up my mblaze configuration in ~/.mblaze/profile.

8. Made my own user a member of the mbsync group, gave full, recursive permissions to the group on /home/mbsync/mail, and symlinked ~/mail to that directory.

9. Created a cronjob to run my check-mail script every ten minutes.

10. Wrote some extra scripts to make my daily e-mailing life easier.

securing passwords

My e-mail account passwords are auto-generated and stored on a hardware device that can act as a keyboard. This is about as secure as passwords get, as far as I can tell. This is not practical for checking e-mail periodically in the background, however.

The Ledger allows me to generate cryptographic keys deterministically on the device itself based on a randomly-generated string of 24 words. The algorithm is open-source, so as long as I can keep track of my 24 words, all the keys I've ever generated with it should be recoverable. I use trezor-agent to integrate the device with GPG and SSH. This allows me to keep my e-mail passwords encrypted on disk, decrypting them only to put them into a dedicated user account's kernel keyring once per boot.

Setting up trezor-agent for use with the Ledger and GPG is outside the scope of this document, but the process is described in that project's source code repository under the docs folder. Once set up, I used the following AGPL-3 licensed script (with the -p flag set) to create a store of e-mail address keys to password values at ~/.mbsync-passwords:

#!/usr/bin/env bash

# <AGPL-3 license omitted here for brevity>

# Has no way to handle tabs in keys or values, so don't try it!

set -e
set -o pipefail # This is the one line that keeps this a bash script.

[ ! -e "$HOME/.gpgkvrc" ] || . "$HOME/.gpgkvrc"

export GPGBIN="${GPGBIN:-gpg2 -q}"
export GPGKV_IDENT="${GPGKV_IDENT:-$USER}"
export GPGKV_STORE="${GPGKV_STORE:-$HOME/.gpgkv-store}"

case "$1" in
        add)
                if [ "$2" = "-p" ]; then
                        valopt="-s"
                        valprompt="Password"
                        shift
                else
                        valprompt="Value"
                fi

                key="$2"
                val="$3"

                [ "$key" ] || { echo -n "Key: " && read key; }
                [ "$val" ] || { echo -n "$valprompt: " && read $valopt val; }

                if [ ! -e "$GPGKV_STORE" ]; then
                        printf "%s\t%s\n" "$key" "$val" \
                        | $GPGBIN --armour --encrypt -r "$GPGKV_IDENT" \
                        | tee "$GPGKV_STORE" >/dev/null
                else
                        echo "$($GPGBIN --decrypt "$GPGKV_STORE" \
                                                | xargs -0 printf "%s\t%s\n%s" "$key" "$val" \
                                                | $GPGBIN --armour --encrypt -r "$GPGKV_IDENT")" \
                        > "$GPGKV_STORE"
                fi
                echo # in case it used a password prompt
                ;;

        get)
                $GPGBIN --decrypt "$GPGKV_STORE" \
                | grep -o "^$2\t" \
                | cut -f2
                ;;

        del)
                $GPGBIN --decrypt "$GPGKV_STORE" \
                | grep -v "^$2\t" \
                | $GPGBIN --armour --encrypt -r "$GPGKV_IDENT" \
                | tee "$GPGKV_STORE" >/dev/null
                ;;

        dump)
                $GPGBIN --decrypt "$GPGKV_STORE"
                ;;

        *)
                echo "Usage: $0 <command> [args]"
                echo "Commands:"
                echo "  add [-p] [<key> <value>]"
                echo "  get <key>"
                echo "  del <key>"
                echo "  dump"
esac

gpgkv

To write to ~/.mbsync-passwords instead of the default ~/.gpgkv-store, set the GPGKV_STORE variable when invoking gpgkv. E.g., "GPGKV_STORE='~/.mbsync-passwords' gpgkv ..."

editing sudoers

Creating a dedicated user for the purpose of running msmtp and mbsync and keeping the passwords in their kernel keyring seemed to me like the easiest, safest way to keep the passwords in memory. In order to avoid having to type my user password during the normal execution of this user's duties, I made these entries in my sudoers file:

lykso ALL=(mbsync) NOPASSWD: /bin/keyctl padd user *@* @u
lykso ALL=(mbsync) NOPASSWD: /bin/keyctl link @us @s
lykso ALL=(mbsync) NOPASSWD: /usr/bin/mbsync -a
lykso ALL=(mbsync) NOPASSWD: /usr/bin/msmtp
lykso ALL=(mbsync) NOPASSWD: /bin/chmod -R g=u /home/mbsync/mail/*@*

If you copy this, be sure to verify the locations of keyctl, mbsync, msmtp, and chmod. I moved this to another distribution of Linux recently and had some trouble when, e.g., mbsync moved from /bin/mbsync to /usr/bin/mbsync.

loading passwords

To load the passwords, I wrote a simple script that iterates over every entry in ~/.mbsync-passwords and puts it in mbsync's keyring:

#!/usr/bin/env bash

# Opens the .mbsync-passwords file with gpgkv and loads each key/value pair into
# the kernel's keystore for the mbsync user.

set -e
set -o pipefail

export GPGKV_STORE="$HOME/.mbsync-passwords"

# Make sure the session and user session keyrings are linked.
# User keys cannot be found otherwise.
# Some distributions do this by default, but others don't.
sudo -u mbsync keyctl link @us @s

gpgkv dump | while read line; do
	# Doing it this way to keep the credentials out of the process list.
	IFS=


\t' read -r -a credentials <<<"$line"
	sudo -u mbsync keyctl padd user "${credentials[0]}" @u <<<"${credentials[1]}"
done

load-mbsync-secrets

As noted before, this script has to be re-run at every boot. The integration with my Ledger is seamless, so this script can be used as-is regardless of whether or not you have such a thing.

configuring msmtp, mbsync, and mblaze.

.msmtprc and .mbsyncrc both are kept in /home/mbsync and belong to the mbsync user and group.

defaults
auth on
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.config/msmtp/msmtp.log

account lykso@lyk.so
host mail.privateemail.com
port 465
tls_starttls off
from lykso@lyk.so
user lykso@lyk.so
passwordeval "keyctl request user lykso@lyk.so | xargs keyctl pipe"

/home/mbsync/.msmtprc

IMAPAccount lykso@lyk.so
Host mail.privateemail.com
Port 993
User lykso@lyk.so
PassCmd "keyctl request user lykso@lyk.so | xargs keyctl pipe"
SSLType IMAPS
SSLVersions TLSv1.2

IMAPStore lykso@lyk.so-remote
Account lykso@lyk.so

MaildirStore lykso@lyk.so-local
Path   ~/mail/lykso@lyk.so/
Inbox  ~/mail/lykso@lyk.so/Inbox
Trash  ~/mail/lykso@lyk.so/Trash
SubFolders Verbatim

Channel lykso@lyk.so
Master :lykso@lyk.so-remote:
Slave :lykso@lyk.so-local:
Patterns *
Create Both
SyncState *

/home/mbsync/.mbsyncrc

.mblaze/profile is kept in my home directory and belongs to my user and group.

Local-Mailbox: lykso@lyk.so
Sendmail: sudo -u mbsync msmtp
Sendmail-Args: --read-recipients --read-envelope-from

~/.mblaze/profile

custom scripts

After adding my user to the mbsync group, symlinking ~/mail to /home/mbsync/mail, and running `chmod g+rw /home/mbsync/mail`, I wrote a couple scripts to make using all these pieces togther easier. First was check-mail, for which I created a cronjob that runs every 10 minutes:

#!/usr/bin/env sh

# Dependencies:
# mblaze
# mbsync
# notify-send

set -e

sudo -u mbsync mbsync -a
sudo -u mbsync chmod -R g=u /home/mbsync/mail/*@*
new="$(find /home/mbsync/mail -path '**/new/**' | wc -l)"

[ "$new" != "0" ] || exit 0

# Void doesn't set this, but Debian does.
## notify-send needs this set.
#export XDG_RUNTIME_DIR="$HOME/.service/xdg"

[ "$new" = 1 ] || plural="s"
notify-send "You've got mail!" "$new new message${plural}!"

mdirs /home/mbsync/mail | while read mdir; do
  minc "$mdir" 2>&1 > /dev/null
done

echo "New messages: $new"

check-mail

A script for listing all mail in my inboxes, and optionally searching via arguments to mpick:

#!/usr/bin/env sh

# Show all messages in the inboxes not tagged as trash.
mdirs ~/mail \
  | grep '/Inbox\(/\|$\)' \
  | mlist \
  | mpick "$@" \
  | msort -d \
  | mseq -S \
  | mscan

list-inbox

Same idea, but trimmed down and focused on trash:

#!/usr/bin/env sh

# Show all messages in the trash
mdirs $1 ~/mail \
  | grep '/Trash\(/\|$\)' \
  | mlist \
  | msort -d \
  | mscan

list-trash

The shortest script, the one I use to read my mail:

#!/usr/bin/env sh

mless $1 && mflag -S $1

read-mail

This only works for plaintext e-mail. For HTML e-mails, I haven't bothered writing a script yet. Any time I get an important e-mail that doesn't include a plaintext version, I just run:

mshow <e-mail number> | lynx -stdin

If it contains images that need to be seen, then I suppose I could just pipe mshow to a file and view it in Firefox or Netsurf, but that hasn't happened yet.

For archiving e-mail:

#!/usr/bin/env sh

set -e

message="$(mpick $1)"
mrefile $1 "$(echo "$message" | sed 's/\/Inbox\/.\+/\/Archives/')"

arch-mail

For trashing mail:

#!/usr/bin/env sh

set -e

[ "$1" ] || { >&2 echo "Need to specify at least one test!"; exit 1; }
mdirs ~/mail \
  | mlist \
  | mpick "$@" \
  | mflag -T \
  | while read message; do
    mrefile "$message" "$(echo "$message" | sed 's/\/Inbox\/.\+/\/Trash/')"
  done

trash-mail

For trashing things that get through my spam filter:

#!/usr/bin/env sh

# Spam
trash-mail '-t from =~ "@winsonsoloads.com" || from =~ "@welcometoterizin.org"'

auto-trash-mail

Probably ought to be marking those as spam instead, but this is what I'm doing as of this writing, so there you are.

Finally, for moving mail between folders in the same mailbox:

#!/usr/bin/env sh

set -e

dest="$1"
shift

[ "$1" ] || { >&2 echo "Please specify at least one test or message!"; exit 1; }

mdirs ~/mail \
  | mlist \
  | mpick "$@" \
  | while read message; do
    account="$(echo "${message#*/mail/}" | cut -d/ -f1)"
    mrefile -v "$message" "$HOME/mail/$account/$dest"
  done

mv-mail

Hopefully that gives you an idea of how mblaze's utilities might be recombined to create whatever experience you're after. I very much appreciate the adherence to the Unix philosophy, as it allows me to craft whatever workflow I like. I also very much like that this keeps my e-mails in a greppable, human-friendly, flatfile format rather than in some database that needs special tools to be interacted with.

debian addendum

After migrating from Void to Debian, I found that msmtp had stopped working. I was getting permission errors from keyctl and xargs. After some searching, I found someone else with the same problem:

Stack Overflow: awk permission denied when run through msmtp

To save you a click, the trouble was that Debian comes with a restrictive AppArmor profile for msmtp. Rather than learn how to use AppArmor and edit the profile, I followed the advice on the StackOverflow page and just disabled it for msmtp:

sudo ln -s /etc/apparmor.d/usr.bin.msmtp /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.bin.msmtp 

relevant links

mblaze

isync, which provides the mbsync command.

msmtp

Mooltipass

Ledger

trezor-agent