💾 Archived View for perso.pw › blog › articles › managing-a-fleet-of-nixos-part1.gmi captured on 2023-09-28 at 16:08:13. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Managing a fleet of NixOS Part 1 - Design choices

Author: Solène
Date: 02 September 2022
Tags: bento nixos nix

Comment on Mastodon

Introduction

I have a grand project in my mind, and I need to think about it before starting any implementation. The blog is a right place for me to explain what I want to do and the different solutions.

It's related to NixOS. I would like to ease the management of a fleet of NixOS workstations that could be anywhere.

This could be useful for companies using NixOS for their employees, to manage all the workstations remotely, but also for people who may manage NixOS systems in various places (cloud, datacenter, house, family computers).

In this central management, it makes sense to not have your users with root access, they would have to call their technical support to ask for a change, and their system could be updated quickly to reflect the request. This can be super useful for remote family computers when they need an extra program not currently installed, and that you took the responsibility of handling your system...

With NixOS, this setup totally makes sense, you can potentially reproduce users bugs as you have their configuration, stage new changes for testing, and users can roll back to a previous working state in case of big regression.

Cachix company made it possible before I figure a solution. It's still not late to propose an open source alternative.

Cachix Deploy

Defining the project

The purpose of this project is to have a central management system on which you keep the configuration files for all the NixOS around, and allow the administrator to make the remote NixOS to pick up the new configuration as soon as possible when required.

We can imagine three different implementations at the highest level:

a scheduled job on each machine looking for changes in the source. The source could be a git repository, a tarball or anything that could be used to carry the configuration.
NixOS systems could connect to something like a pub/sub and wait for an event from the central management to trigger a rebuild, the event may or not contain information / sources.
the central management system could connect to the remote NixOS to trigger the build / push the build

These designs have all pros and cons. Let's see them more in details.

Solution 1 - Scheduled job

In this scenario, The NixOS system would use a cron or systemd timer to periodically check for changes and trigger the update.

Pros

low maintenance
could interactively ask the user when they want to upgrade if not now

Cons

may not run at all if the system is not up at the correct time, or could be run at a delayed time depending on situation
can't force an update as soon as possible
not really bandwidth effective if you often poll
no feedback from the central management about who made/receive the update (except by adding a call to the server?)

Solution 2 - Remote systems are listening for changes (publisher / subscriber)

In this scenario, the NixOS system would always be connected to the central management, using some kind of protocol like MQTT, BOCH or similar.

Pros

you know which systems are up
events from central management are instantaneous and should wait for an acknowledgment
updates should propagate very quickly
could interactively ask the user when they want to upgrade if not now

Cons

this can lead to privacy issue as you know when each host is connected
this adds complexity to the server
this adds complexity on each client
firewalls usually don't like long-lived connections, HTTPS based solution would help bypass firewalls

Solution 3 - The central management pushes the updates to the remote systems

In this scenario, the NixOS system would be reachable over a protocol allowing to run commands like SSH. The central management system would run a remote upgrade on it, or push the changes using tools like deploy-rs, colmena, morph or similar...

Awesome-nix list: deployment-tools

Pros

update is immediate
SSH could be exposed over TOR or I2P for maximum firewall bypassing capability

Cons

offline systems may be complicated to update, you would need to try to connect to them often until they are reachable
you can connect to the remote machine and potentially spy the user. In the alternatives above, you can potentially achieve the same by reconfiguring the computer to allow this, but it would have to be done on purpose

Making a choice

I tried to state the pros and cons of each setup, but I can't see a clear winner. However, I'm not convinced by the Solution 1 as you don't have any feedback or direct control on the systems, I prefer to abandon it.

The Solutions 2 and 3 are still in the competition, we basically ended with a choice between a PUSH and a PULL workflow.

Conclusion

In order to choose between 2 and 3, I will need to experiment with the Solution 2 technologies as I never used them (MQTT, RabbitMQ, BOCH etc…).