💾 Archived View for gemini.hitchhiker-linux.org › gemlog › runit_vs_s6.gmi captured on 2023-09-28 at 15:58:51. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-06-14)
-=-=-=-=-=-=-
Since the init subject seems to have gotten people talking, I thought it might be worth going a little more in depth about the lineage, similarities, and differences between Runit and S6. Or, at the very least, my experience in relation to the subject.
The original model for both software suites was DaemonTools by Daniel J Bernstein. Daemontools is primarily a process supervisor. That is to say, it's main function is to spawn processes and watch over them, restarting any that have failed. This is in contrast to how SysV has historically been used, where long running services are started and stopped via shell scripts, running in the background, and their process state can only be directly observed through pidfiles and process lists.
Interestingly, SysV Init has this same capability built in. A service can be specified in /etc/inittab with the respawn option and it will be automatically restarted if it dies. However this functionality was never adopted by any major operating system that I know of, being relegated to use only as a way of respawning tty logins through the getty and later agetty programs.
Bernstein had some interesting ideas on software development that he passed on to his disciples. One of his tenets was that if an interface sucked, then you write a better interface and use that instead. Laurent Bercot definitely took that to heart and replaces a lot of libc functionality and low level primitives with his skalibs distribution, which is required for most of the other software that he writes (including S6 and Execline).
Another interesting idea that Bernstein espoused was using the filesystem itself as a database. This can be seen in the unique configuration system of Qmail as well as in the service directories of Daemontools and it's descendents. A service directory (in it's original daemontools form) consists of at minimum a directory and an executable file named `run`. The `svscanboot` program starts the `svscan` process with a directory containing a collection of service directories as an argument. Svscan in turn launches a single `supervise` process for each directory. If one doesn't want the service in a particular directory to be started, then one simply creates a file named `down` in that service directory and svscan will skip it. Should it later be desired for that service to run, simply remove the `down` file and re-run svscan.
The canonical way to write a service `run` script was to redirect the service's stdout to a logging program defined in the subdirectory of it's service directory named `log`, in the script named (wait for it..) `run`. The `run` file could even just be a named pipe, directing it's output to, say, the syslog.
It's a very simple and easy to understand system. But it is by nature designed to be managed manually, and the original source distribution is getting rather long in the tooth. There have been a number of offshoots and inspired by projects over the years, starting with daemontools-encore, which was basically an update of the original source which only aimed to make it compile and run properly again, then leading to Runit, which is a fairly faithful reimplementation of the original, and finally S6, which can be used in exactly the same way as the original but has a lot of more modern features available. Most of the offshoots were undertaken with the idea of running svscan as PID1, aka Init.
Runit was written by Garrit Pape and is newer than daemontools-encore yet older than S6 and SystemD. Runit functions better as PID1 than DaemonTools ever did, which is not surprising because that was not the original purpose of DaemonTools. It was available as a choice alongside SysV Init where you were free to make the switch, and it was well enough received that the code was ported into BusyBox and is available there to this day.
Void is one of several Linux distributions that use Runit as Init. My experience with it is that it is extremely reliable once set up, extremely fast, but requires the admin to write a lot of the business logic themselves if you are going to do anything out of the ordinary with it.
One of my main criticisms of Runit is that there is no official way to order dependencies between services. An example would be an Apache server that was running an application that requires a PostgreSQL server to already be up and running. With SysV Init, we had runlevels, with a directory which had the run scripts for each daemon linked into it, with a number prefixed to the name of each one. The scripts were run in numerical order, sequentially, forming a crude (and very hacky and brittle) dependency ordering. SystemD allows one to define in a Unit file that service A depends on service B, and won't try to start B until A has reported that it is ready. With Runit, svscan attempts to start everything in parallel and just restarts any services that fail the first, second or hundredth time. As an admin, you can modify the `run` script for a particular service to poll another service that it depends on before attempting to start.
As an example, since obviously a web server requires a network interface to be up before it can bind to it, one could write a run script sort of like the following:
#!/bin/sh
set -e
if [ $(ping -c 1 eth0) != 0 ]
then sleep 2
exit
else
<server> start --foreground
fi
This sort of manual intervention does work, but it's error prone. Back in the day a lot of folks would give advice on the internet such as `just put it in rc.local` and this feels much the same. There were a lot of machines running with just awful looking rc.local scripts, and a lot of hard to debug problems occurred upon system upgrades if the behavior of one or more programs changed in some subtle way.
My other gripe with Runit is that there is no such thing as a `oneshot` service. If you want to, say, mount your disks using a service directory then you can do it, but it requires a hack.
#!/bin/sh
set -e
mount -a
while true
do sleep 10000
done
The idea is that such a script will never return, because if the script were to exit then the `supervise` process assigned to watch over it would restart it.
The flipside to these types of concerns is that Runit is one of the only Init systems where it is possible to completely understand what is happening, which gives some piece of mind. The value of systems which can be fully understood by a single person cannot be overstated.
S6 is a more recent spiritual descendant of DaemonTools written by Laurent Bercot. As I alluded to earlier, s6 has a fair bit more functionality than DaemonTools or Runit, while still quite honestly showing it's heritage and adhering to the Unix philosophy of small programs that do one thing well. Some of the important features that s6 adds over what is provided by Runit are:
In addition to the above, the s6-rc package includes a suite of tools to manage service directories and bundles as a separate addon, and the s6-linux-init addon package provides a way to handle early boot and shutdown gracefully while running s6-svscan as PID 1 for the rest of the lifetime the machine is up.
Of neccessity, if you take all of that as a whole it is more complex than Runit, but due to the small size of each program they are easy to understand individually. Between Runit and s6, I have a strong preference for s6 because it provides that one blessed way to handle so many scenarios that Runit will leave up to the administrator. Any time you leave it up to the admin, or up to the packager for your distro, to implement functionality, then you will inevitably wind up with multiple different ways of doing things, making it more difficult to understand where things have gone wrong. This was a longstanding issue with SysV Init, and any replacement should seek to avoid making the same mistakes.
Bercot also has written the execline command interpreter and recommends it's use with s6. It's actually a great little domain specific scripting language and a very small in size addition to a distro, and well worth checking out. Due to the completely modular design of s6 it's also not required, and you could just as easily write your service `run` programs in POSIX shell or really any other language. A simple daemon that requires no command line arguments to start could even be started by making `run` a simlink to the binary, for that matter. But execline has some real advantages if all you need to do is run a series of commands without requiring much in the way of control flow, which is exactly what most of the `run` scripts will be. It is incredibly frugal on memory usage, spawns fewer processes and has an easy to understand grammar.
I began looking at supervision suites several years ago and settled on s6 for my own use in HitchHiker after looking at any and all Init systems that I could for Linux. When I installed Artix, it was originally in order to try it out in a real world environment, and while I had some other issues with Artix the s6 suite was not one of them.
The service definitions in HitchHiker are actually mostly ported over from Artix with changes such as adjusting paths to point to the correct location (/bin vs /usr/bin) and a few that required a scratch implementation. I actually have it installed on the laptop I'm typing this post with. My impression on setting it all up customized for my own use case is that wow, that was a -lot- of individual small scripts to look at, compared with the smaller number of much longer scripts that you would likely have seen in the past with SysV Init. But it's been entirely stable and dependable as well as lightening fast. I firmly believe that technically speaking it's a better system than what comes standard with the vast majority of Linux systems.
One of the other things that I really love about supervision suites such as Runit or s6 is that the code is generally -not- Linux specific. That is to say, your could compile these programs on FreeBSD, OpenBSD, Solaris or just about any POSIXish system and replace Init with a supervision scheme. SystemD was definitely -not- designed that way. Neither was Upstart (Ubuntu) or LaunchD(Mac). The only exception in the software that I've talked about is s6-linux-init. If you wanted to use s6 on another Unix system you could, but it would be up to you to handle the early boot and shutdown of the system, and let s6 run as PID 1 to manage things from there. This is surprisingly easy to do when you dig into it a little, but it does require some specialized knowledge. I may be in a minority for even caring about other Unix systems in 2023, but I still think it benefits everyone to maintain portability wherever possible. Having been a FreeBSD user for a number of years I'd also add that a BSD system would benefit greatly from replacing their ancient and slow boot/shutdown process with something like s6.
All content for this site is licensed as CC BY-SA.
© 2023 by JeanG3nie