💾 Archived View for republic.circumlunar.space › users › johngodlee › posts › 2018-07-15-mutt.gmi captured on 2023-09-08 at 16:22:11. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-04-19)
-=-=-=-=-=-=-
DATE: 2018-07-15
AUTHOR: John L. Godlee
I've been experimenting with a new email client on the command line. I used alpine for a long time, but I've heard a lot about mutt as well. It seems to be the standard terminal email program, but my past attempts to set it up have always failed, so I set aside a couple of hours over the last few days to really get into the subject. This is the setup I have so far, which uses the neomutt fork. It's an amalgamation of lots of different guides and snippets that I've found scattered around on the internet. I've referenced the important ones at the end.
y setup is fairly particular to me and my needs. I use macOS rather than Linux which produces some interesting idiosyncrasies, and I wanted to integrate my existing command line programs where possible, notably vim (text editor) and pass (GPG enabled password manager), and w3m (web browser). I also use Gmail, so interacting with that requires understanding their slightly odd IMAP configuration.
For now we won't even install neomutt, there isn't anything to see yet anyway and much of the config won't make sense until we've set up the back end stuff. The first thing to set up is the syncing of emails with Gmail's IMAP system.
neomutt can handle IMAP email, but grabbing from a remote server to read an email requires that there be an internet connection everytime you open neomutt. To keep an offline database of my Gmail account, so I can search it when travelling, I use offlineimap, which can also be installed using Homebrew[1]:
:
brew install offlineimap
Steve Losh
offlineimap takes its configuration from ~/.offlineimaprc. This is my config:
general
ui = TTY.TTYUI
accounts = johngodlee@gmail.com
pythonfile = ~/.offlineimap.py
fsync = False
ssl = False
Account johngodlee@gmail.com
localrepository = johngodlee-local
remoterepository = johngodlee-remote
Repository johngodlee-local
type = Maildir
localfolders = ~/.mail/johngodlee@gmail.com
nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
'sent': '[Google Mail]/Sent Mail',
'flagged': '[Google Mail]/starred',
'trash': '[Google Mail]/Bin',
}.get(folder, folder)
Repository johngodlee-remote
maxconnections = 3
type = Gmail
remoteuser = johngodlee@gmail.com
remotepasseval = get_pass()
realdelete = no
ssl=true
sslcacertfile = /usr/local/etc/openssl/cert.pem
nametrans = lambda folder: {'[Google Mail]/Drafts': 'drafts',
'[Google Mail]/Sent Mail': 'sent',
'[Google Mail]/Starred': 'starred',
'[Google Mail]/Bin': 'trash',
}.get(folder, folder)
folderfilter = lambda folder: folder not in ['[Google Mail]/Bin',
'[Google Mail]/Important',
'[Google Mail]/Spam',
'[Google Mail]/Chats',
'[Google Mail]/All Mail',
]
There is a lot there, so here is a line by line breakdown:
ui = TTY.TTYUI signals which user interface to use. TTY.TTYUI makes sure that the process runs quietly in the background, there are also other options which may be better, but this one works for me.
accounts = johngodlee@gmail.com is the email account to be checked.
pythonfile = ~/.offlineimap.py is a python script which offlineimap calls to get the password for my email account from pass. I'll go through .offlineimap.py later.
fsync = False just tells offlineimap that it doesn't need to make sure a full sync is completed every time, if a few emails get lost, no bother, they will just get synced next time
Account johngodlee@gmail.com localrepository = johngodlee-local remoterepository = johngodlee-remote
The section above defines the names of repositories to use for this email account.
Repository johngodlee-local type = Maildir localfolders = ~/.mail/johngodlee@gmail.com
The section above defines the location and mailbox type of the local repository for the email from my account
nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts', ...}.get(folder, folder) translates Gmail's IMAP folders into folders in my local mail directory. I think this is necessary as Gmail's folders contain forward slashes, which would mess up a normal file system. I only translated the system IMAP folders, as custom folders (labels) don't use the [Google Mail]/ prefix and are synced automatically as is. I had some trouble getting offlineimap to recognise my system folders, as many other guides recommend using the [Gmail]/ prefix, but for whatever reason my account uses [Google Mail]/.
Repository johngodlee-remote maxconnections = 3 type = Gmail remoteuser = johngodlee@gmail.com remotepasseval = get_pass() realdelete = no ssl=true sslcacertfile = /usr/local/etc/openssl/cert.pem
This section configures the remote repository and how to interact with it.
maxconnections = 3 defines the number of parallel connections offlineimap can make when syncing emails. 3 is low enough that Gmail won't enforce rate limits and break the connection.
type = Gmail tells offlineimap that the account is a Gmail account, and so it should take that into account when dealing with their weird IMAP setup.
remoteuser = johngodlee@gmail.com simply tells us what account is being accessed.
remotepasseval = get_pass() tells offlineimap where to get the password for the account. get_pass() is the python function that is created in ~/.offlineimap.py
realdelete = no tells offlineimap not to totally delete an email when you press delete in neomutt, instead it will keep it in All Mail.
ssl = true says to always use ssl encryption when syncing
sslcacertfile = /usr/local/etc/openssl/cert.pem gives the location of a security certificate, which I think helps ssl to further prevent man in the middle attacks when syncing.
nametrans = lambda folder: {'[Google Mail]/Drafts': 'drafts', ...}.get(folder, folder) is just like the nametrans function earlier, only it goes in the other direction.
folderfilter = lambda folder: folder not in ['[Google Mail]/Bin', ...] gives a list of folders that should not be synced. In my case, I don't want to sync the trash (Bin because British), Spam, Chats, and All Mail.
Later on I'll look at how to get offlineimap to run in the background using launchd, a macOS alternative to crontab.
#!/usr/bin/env python
from subprocess import check_output
def get_pass():
return check_output("/usr/local/bin/pass email/johngodlee@gmail.com", shell=True).splitlines()[0]
This script looks in pass for the entry email/johngodlee@gmail.com, which contains the password for my email account, and then takes the first line (splitlines()[0]) of that entry to store it in get_pass(). This function is then called by remotepasseval in .offlineimaprc. The script should be saved as ~/.offlineimap.py.
notmuch provides very fast email searching. It has other capabilities such as tagging and indexing email, but I use Gmail's IMAP folders (labels) for this, so I only use it for searching. notmuch can be installed via Homebrew:
brew install notmuch
When installed, neomutt allows searching using notmuch by defining a custom macro. You can then use all the notmuch search syntax to quickly find emails within neomutt. I'll go through that in the next section where I define the muttrc.
Neomutt doesn't know how to display emails that are encoded as HTML only. To render HTML as something legible in the terminal I use w3m. This is a text based web browser. Once again, this can be installed using Homebrew:
brew install w3m
To tell neomutt to forward HTML emails to w3m for plain text encoding, then send it back to neomutt, we need to create a file called .mailcap. Keep it in the home directory as convention:
touch ~/.mailcap
Ten add the following snippet to the file:
text/html; w3m -dump -o document_charset=%{charset} '%s'; nametemplate=%s.html; copiousoutput
.mailcap is then referenced later in the muttrc.
Now that we've set up all the background, we can install the central part of the workflow, mutt. Specifically, I'm using neomutt, which is a fork of the original mutt which incorporates some of the most widely used plugins.
neomutt can be installed using Homebrew:
brew install neomutt
Then we need to make a configuration file, I like to keep mine in ~/.mutt/muttrc but see man neomutt for more locations where neomutt will look for a config file:
mkdir ~/.mutt touch ~/.mutt/muttrc
The first thing to add to the configuration is some IMAP settings:
set imap_user = "johngodlee@gmail.com" ## Call pass from within set my_pass = "`pass email/johngodlee@gmail.com`" set imap_pass = $my_pass set from = "johngodlee@gmail.com" set realname = "John Godlee" set folder = "~/.mail/johngodlee@gmail.com" set spoolfile = +INBOX set postponed = +drafts set record = +sent ## Don't automatically move messages after reading set move = no ## Max time mutt should wait before polling IMAP connections, in minutes set imap_keepalive = 30 ## Set smtp URL for replying set smtp_url = "smtp://johngodlee@smtp.gmail.com:587/" ## Set the login method for smtp replying set smtp_pass = $my_pass set smtp_authenticators = 'login' ## Allow mutt to open new imap connections to test for new mail unset imap_passive ## Set cache locations so IMAP polling is quicker on startup set header_cache = ~/.mutt/johngodlee/headers set message_cachedir = ~/.mutt/johngodlee/bodies set certificate_file = ~/.mutt/certificates ## If stuck in a prompt, abort after n minutes to check IMAP (timeout), n minutes between checks (mail_check) set timeout = 10 set mail_check = 5
Some of this is self explanatory, so I'll only go through the important bits
set my_pass ... calls pass to create the variable my_pass which is then sent to imap_pass to set the password for my Gmail account, which is needed for sending mail.
set folder ... sets the location of the mailbox which offlineimap syncs email to. The value of folder also acts as the prefix for set spoolfile, set postponed and set record, as indicated by putting + before the folder names.
set spoolfile ... should be the location of your inbox, where new mail arrives
set postponed ... is where unfinished new emails are kept if they are aborted before being sent
set record ... should be a folder where a copy of sent messages are kept
Note that set smtp_pass ... also calls the my_pass variable
As neomutt comes with the sidebar patch, I wanted to make use of it by displaying any folders that are synced from offlineimap:
## Mailboxes to display in sidebar and check regularly mailboxes +INBOX +'starred' +'trash' +'drafts' +'sent' +'Archived' +'coding_club' +'diss_manuscript' +'personal' +'PhD' +'SEOSAW_contacts' +'SEOSAW_logos' +'STEB_2018' +'urgent' +'hemi_lens_proposal' +kew_taxonomy_course +'angola_botanic_garden' ## Sort sidebar folders alphabetically set sidebar_sort_method = name ## Sidebar visible by default set sidebar_visible = yes ## Don't abbreviate folders in sidebar set sidebar_short_path = no
I use a stripped down .vimrc for writing email, as I don't need a lot of the plugins that I use for programming. I can tell neomutt to write emails using vim with this alternative .vimrc (.vimrc_alpine) by adding the following to the neomuttrc:
set editor = 'vim -u ~/.vimrc_alpine'
Here is my .vimrc_alpine:
set nocompatible " be iMproved, required
filetype off " required
" enable syntax highlighting
syntax on
" Stop creating swp and ~ files
set nobackup
set noswapfile
" Ignore case of searches
set ignorecase
" Don’t reset cursor to start of line when moving around
set nostartofline
" Preserve indentation on wrapped lines
set breakindent
" Disable folding in markdown
let g:vim_markdown_folding_disabled = 1
" Disable syntax conceal in markdown
let g:vim_markdown_conceal = 0
" Normal backspace behaviour
set backspace=2
" map A (append at end of line) to a (append in place)
nnoremap a A
" Move by visual lines rather than actual lines with k,j
nnoremap k gk
nnoremap j gj
nnoremap gk k
nnoremap gj j
" Easier save and quit with ;
noremap ;w :w<CR>
noremap ;q :q<CR>
" Copy and paste from `+` register for interacting with mac clipboard
vnoremap y "+y
vnoremap p "+p
nnoremap p "+gp
vnoremap d "+d
nnoremap dd "+dd
" Map of modes and their codes for statusline
let g:currentmode={
\ 'n' : 'N ',
\ 'no' : 'N·Operator Pending ',
\ 'v' : 'V ',
\ 'V' : 'V·Line ',
\ '^V' : 'V·Block ',
\ 's' : 'Select ',
\ 'S' : 'S·Line ',
\ '^S' : 'S·Block ',
\ 'i' : 'I ',
\ 'R' : 'R ',
\ 'Rv' : 'V·Replace ',
\ 'c' : 'Command ',
\ 'cv' : 'Vim Ex ',
\ 'ce' : 'Ex ',
\ 'r' : 'Prompt ',
\ 'rm' : 'More ',
\ 'r?' : 'Confirm ',
\ '!' : 'Shell ',
\ 't' : 'Terminal '
\}
" Change statusline based on colour
function! ChangeStatuslineColor()
if (mode() =~# '\v(n|no)')
exe 'hi! StatusLine ctermfg=112'
elseif (mode() =~# '\v(v|V)' || g:currentmode[mode()] ==# 'V·Block' || get(g:currentmode, mode(), '') ==# 't')
exe 'hi! StatusLine ctermfg=172'
elseif (mode() ==# 'i')
exe 'hi! StatusLine ctermfg=044'
else
exe 'hi! StatusLine ctermfg=007'
endif
return ''
endfunction
" Make statusline always show
set laststatus=2
" statusline
" left side
set statusline=%{ChangeStatuslineColor()} " Change colour
set statusline+=%0*\ %{toupper(g:currentmode[mode()])} " Current mode
set statusline+=\
set statusline+=%1*%m%r " Modified and read only flags
set statusline+=%1*%= " Switch to right side
" right side
set statusline+=%0*\ "Space
set statusline+=%0*%y " File syntax
set statusline+=%0*\| " Vert-line
set statusline+=%0*%p%% " Percentage through file
set statusline+=%0*\|\ " Vert-line and Space
set statusline+=%0*%l:%c " Line and column number
set statusline+=%0*\ " Space
" Set colours for statusline middle section
hi User1 ctermfg=255 ctermbg=240
" Ragged right line breaks
set linebreak
I want to read emails in plain text if possible, with the provided plain text being used preferentially, and a converted HTML version if plain text isn't available, which I can set using alternative_order .... I also want to tell neomutt how to deal with forced HTML emails, which is to automatically view them and also to give the path to the .mailcap file we created earlier, which uses w3m to parse HTML as plain text:
# Read in plain text if possible alternative_order text/plain text/html auto_view text/html set mailcap_path = ~/.mailcap # Enforce encoding in utf8 set send_charset="utf-8"
To allow notmuch to work I need to allow neomutt to create virtual folders when searching and also to give the location of the mailbox to search (set nm_default_uri ...)
I also set up a keybinding to initiate the notmuch search from within neomutt as \\
set virtual_spoolfile = yes set nm_default_uri = "notmuch:///Users/johngodlee/.mail/johngodlee@gmail.com" macro index,pager \\ "<vfolder-from-query>"
Other guides that I read recommended that I use crontab to schedule offlineimap updates. But on macOS launchd is the proper way to do things. macOS uses .plist scripts stored in ~/Library/LaunchAgents to designate jobs, which use an XML format. Helpfully, Homebrew services can generate .plist scripts for many programs, which can then be amended later.
To start running Homebrew services, first see what services are listed:
brew services list
offlineimap should be listed in the output.
To generate the script to start the service at login:
brew services start offlineimap
To ensure that notmuch follows behind offlineimap and updates its database after every pull from Gmail, I added some extra commands to the .plist file which Homebrew services generated:
vim ~/Library/LaunchAgents/homebrew.mxcl.offlineimap.plist
Then amend the <array> section so it looks like this:
<array>
<string>/usr/local/opt/offlineimap/bin/offlineimap</string>
<string>-u</string>
<string>quiet</string>
<string>;</string>
<string>notmuch</string>
<string>new</string>
</array>
This runs a notmuch new everytime offlineimap runs. The other flags -u and quiet tell offlineimap not to print anything to the terminal which could disrupt the job running in the background.
With all this setup you should be able to run offlineimap, then open neomutt and start reading and writing email, though it's entirely possible that I may have missed something. In any case, here are the guides that I learnt from and used to write this setup:
y full setup can be explored in my dotfiles[11]
2: https://wiki.archlinux.org/index.php/mutt
3: http://stevelosh.com/blog/2012/10/the-homely-mutt/
4: https://smalldata.tech/blog/2016/09/10/gmail-with-mutt
5: https://gist.github.com/amandabee/cf7faad0a6f2afc485ee
6: https://github.com/cbracken/mutt
7: https://www.farces.com/wikis/naked-server/homebrew/brew-services/
8: https://bbs.archlinux.org/viewtopic.php?id=142377
9: https://pbrisbin.com/posts/mutt_gmail_offlineimap/
10: https://baptiste-wicht.com/posts/2014/07/a-mutt-journey-my-mutt-configuration.html
