As a recent Qubes OS user, but also a NixOS user, I want to be able to reproduce my system configuration instead of fiddling with files everywhere by hand and being clueless about what I changed since the installation time.
Fortunately, Qubes OS is managed internally with Salt Stack (it's similar to Ansible if you didn't know about Salt), so we can leverage salt to modify dom0 or Qubes templates/VMs.
Qubes OS official project website
In this example, I'll show how to write a simple Salt state files, allowing you to create/modify system files, install packages, add repositories etc...
Everything will happen in dom0, you may want to install your favorite text editor in it. Note that I'm still trying to figure a nice way to have a git repository to handle this configuration, and being able to synchronize it somewhere, but I still can't find a solution I like.
The dom0 salt configuration can be found in `/srv/salt/`, this is where we will write:
Quick extra explanation: there is a directory `/srv/pillar/`, where you store things named "pillars", see them as metadata you can associate to remote hosts (AppVM / Templates in the Qubes OS case). We won't use pillars in this guide, but if you want to write more advanced configurations, you will surely need them.
Let's use dom0 to manage itself 🙃.
Create a text file `/srv/salt/custom.top` with the content (YAML format):
base:
'dom0':
- dom0
This tells that hosts matching `dom0` (2nd line) will use the state named `dom0`.
We need to enable that .top file so it will be included when salt is applying the configuration.
qubesctl top.enable custom
Now, create the file `/srv/salt/dom0.sls` with the content (YAML format):
my packages:
pkg.installed:
- pkgs:
- kakoune
- git
This uses the salt module named `pkg`, and pass it options in order to install the packages "git" and "kakoune".
Salt Stack documentation about the pkg module
On my computer, I added the following piece of configuration to `/srv/salt/dom0.sls` to automatically assign the USB mouse to dom0 instead of being asked every time, this implements the instructions explained in the documentation link below:
Qubes OS documentation: USB mice
/etc/qubes-rpc/policy/qubes.InputMouse:
file.line:
- mode: ensure
- content: "sys-usb dom0 allow"
- before: "^sys-usb dom0 ask"
Salt Stack documentation: file line
This snippet makes sure that the line `sys-usb dom0 allow` in the file `/etc/qubes-rpc/policy/qubes.InputMouse` is present above the line matching `^sys-usb dom0 ask`. This is a more reproducible way of adding lines to configuration file than editing by hand.
Now, we need to apply the changes by running salt on dom0:
qubesctl --target dom0 state.apply
You will obtain a list of operations done by salt, with a diff for each task, it will be easy to know if something changed.
Note: state.apply used to be named state.highstate (for people who used salt a while ago, don't be confused, it's the same thing).
Using the same method as above, we will add a match for the fedora templates in the custom top file:
In `/srv/salt/custom.top` add:
'fedora-*':
- globbing: true
- fedora
This example is slightly different than the one for dom0 where we matched the host named "dom0". As I want my salt files to require the least maintenance possible, I won't write the template name verbatim, but I'd rather use a globbing (this is the name for simpler wildcard like `foo*`) matching everything starting by `fedora-`, I currently have fedora-37 and fedora-38 on my computer, so they are both matching.
Create `/srv/salt/fedora.sls`:
custom packages:
pkg.installed:
- pkgs:
- borgbackup
- dino
- evolution
- fossil
- git
- pavucontrol
- rsync
- sbcl
- tig
In order to apply, we can type `qubesctl --all state.apply`, this will work but it's slow as salt will look for changes in each VM / template (but we only added changes for fedora templates here, so nothing would change except for the fedora templates).
For a faster feedback loop, we can specify one or multiple targets, for me it would be `qubesctl --targets fedora-37,fedora-38 state.apply`, but it's really a matter of me being impatient.
An interesting setup with Qubes OS is to have your SSH key in a separate VM, and use Qubes OS internal RPC to use the SSH from another VM, with a manual confirmation on each use. However, this setup requires modifying files at multiple places, let's see how to manage everything with salt.
Qubes OS community documentation: Split SSH
Reusing the file `/srv/salt/custom.top` created earlier, we add `split_ssh_client.sls` for some AppVMs that will use the split SSH setup. Note that you should not deploy this state to your Vault, it would self reference for SSH and would prevent the agent to start (been there :P):
base:
'dom0':
- dom0
'fedora-*':
- globbing: true
- fedora
'MyDevAppVm or MyWebBrowserAppVM':
- split_ssh_client
Create `/srv/salt/split_ssh_client.sls`: this will add two files to load the environment variables from `/rw/config/rc.local` and `~/.bashrc`. It's actually easier to separate the bash snippets in separate files and use `source`, rather than using salt to insert the snippets directly in place where needed.
/rw/config/bashrc_ssh_agent:
file.managed:
- user: root
- group: wheel
- mode: 444
- contents: |
SSH_VAULT_VM="vault"
if [ "$SSH_VAULT_VM" != "" ]; then
export SSH_AUTH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
fi
/rw/config/rclocal_ssh_agent:
file.managed:
- user: root
- group: wheel
- mode: 444
- contents: |
SSH_VAULT_VM="vault"
if [ "$SSH_VAULT_VM" != "" ]; then
export SSH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
rm -f "$SSH_SOCK"
sudo -u user /bin/sh -c "umask 177 && exec socat 'UNIX-LISTEN:$SSH_SOCK,fork' 'EXEC:qrexec-client-vm $SSH_VAULT_VM qubes.SshAgent'" &
fi
/rw/config/rc.local:
file.append:
- text: source /rw/config/rclocal_ssh_agent
/rw/home/user/.bashrc:
file.append:
- text: source /rw/config/bashrc_ssh_agent
Edit `/srv/salt/dom0.sls` to add the SshAgent RPC policy:
/etc/qubes-rpc/policy/qubes.SshAgent:
file.managed:
- user: root
- group: wheel
- mode: 444
- contents: |
MyClientSSH vault ask,default_target=vault
Now, run `qubesctl --all state.apply` to configure all your VMs, which are the template, dom0 and the matching AppVMs. If everything went well, you shouldn't have errors when running the command.
Another real world example, using Salt to configure your AppVMs to open links in a dedicated AppVM (named WWW for me):
Qubes OS Community Documentation: Opening URLs in VMs
In your custom top file `/srv/salt/custom.top`, you need something similar to this (please adapt if you already have top files or state files):
'dom0':
- dom0
'fedora-*':
- globbing: true
- fedora
'vault or qubes-communication or qubes-devel':
- default_www
Add the following text to `/srv/salt/dom0.sls`, this is used to configure the RPC:
/etc/qubes-rpc/policy/qubes.OpenURL:
file.managed:
- user: root
- group: wheel
- mode: 444
- contents: |
@anyvm @anyvm ask,default_target=WWW
Add this to `/srv/salt/fedora.sls` to create the desktop file in the template:
/usr/share/applications/browser_vm.desktop:
file.managed:
- user: root
- group: wheel
- mode: 444
- contents: |
[Desktop Entry]
Encoding=UTF-8
Name=BrowserVM
Exec=qvm-open-in-vm browser %u
Terminal=false
X-MultipleArgs=false
Type=Application
Categories=Network;WebBrowser;
MimeType=x-scheme-handler/unknown;x-scheme-handler/about;text/html;text/xml;application/xhtml+xml;application/xml;application/rss+xml;x-scheme-handler/http;x-scheme-handler/https;
Create `/srv/salt/default_www.sls` with the following content, this will run xdg-settings to set the default browser:
xdg-settings set default-web-browser browser_vm.desktop:
cmd.run:
- runas: user
Now, run `qubesctl --target fedora-38,dom0 state.apply`.
From there, you MUST reboot the VMs that will be configured to use the WWW AppVm as the default browser, they need to have the new file `browser_vm.desktop` available for `xdg-settings` to succeed. Run `qubesctl --target vault,qubes-communication,qubes-devel state.apply`.
Congratulations, now you will have a RPC prompt when an AppVM wants to open a file to ask you if you want to open it in your browsing AppVM.
This method is a powerful way to handle your hosts, and it's ready to use on Qubes OS. Unfortunately, I still need to figure a nicer way to export the custom files written in /srv/salt/ and track the changes properly in a version control system.
Erratum: I found a solution to manage the files :-) stay tuned for the next article.