Saturday, 10. December 2022
[This article has been bi-posted to Gemini and the Web]
One interesting feature introduced in early Unix (V7) is the chroot(2) syscall (accompanied by the chroot(8) command added later). They allow for putting processes in what is called a _chroot jail_. What this means is that for the chrooted process (and its children) the filesystem root is not the one of the actual host system but some directory further down the filesystem hierarchy. The result is restricting access to that directory and all the dirs below it and hiding away everything above. This allows for some simple sandboxing and is often used e.g. with FTP where users are chrooted to their home directory.
It is noteworthy that the SysV line of Unix (plus SysV-inspired Linux) and the BSD line follow a different philosophy regarding chroot jails. According to the former it is _not_ a security device and breaking out of a chroot is a documented feature. BSD in contrast was determined to make chroot jails useful for increasing security and tried to prevent escaping from them.
Ever since Poul-Henning Kamp implemented FreeBSD jails in 1998 as a more solid and more flexible means, this legendary feature allows for what has later been characterized as _OS-level virtualization_ or _containerization_. Unlike simple chroot jails they are not limited to filesystem restrictions but also do process isolation, have their own IP address and so on. Since they were introduced, many new features have been added. Today jails support multiple modes of isolation of SysV IPC, can be delegated ZFS datasets and may even have their own network stack (thanks to VNET). It's pretty obvious that such a powerful tool is not exactly trivial to master. And while you can create and operate your jails with just the FreeBSD base system, people have written a whole bunch of utilities to assist in using jails successfully. Those applications are called _jail managers_. This article covers CBSD as of version _13.1.19_.
Disclosure: I have been in contact with the CBSD maintainer for quite a while now and we also both participate in the Advance!BSD project. So if you were hoping for a _completely_ unbiased look, I'm sorry (I'm trying my best, though). On the plus side this means the information you can find in here can be considered reliable.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts.
There are multiple jail managers to choose from (and there have been many more which have been abandoned after a while), however CBSD is rather unique. After originally being used internally at a small game development company, it was open-sourced and first released to the general public in 2013 by Oleg Ginzburg. This makes it the oldest jail manager project that is still active (and which has been in continuous development for almost 10 years now!). This certainly is quite an achievement, yet it's not even one of its main selling points. While jail managers usually are conceived to allow for more convenient jail operation on a single node, CBSD was designed with _clusters_ in mind! In fact, the "C" in the name refers to clustering. Which of course doesn't mean that it cannot be very useful on single nodes as well (which is what we will of course focus on in the first parts of this series).
CBSD is also _much_ more versatile compared to common jail managers. You have surely noticed that it is being called a "virtual environment management framework" in the title. No, this is not just a fancy name for a jail manager! CBSD is simply more than that. Sure, it does jails. However it manages Bhyve Virtual Machines, too. It has support for Xen as well. It can manage Qemu VMs for you and will even lend a hand with VirtualBox if you insist. It does all of that in a mostly consistent way (there are places where it shows that it originally started with just jails). Its flexibility does not end with all of those domains that it works with. CBSD also supports multiple ways of management: Command-line driven, via dialog(1) text UI menus, Vagrant-style (CBSDfile), using an API and more!
At this point you are probably not surprised to hear the it does not stop at creating, destroying, starting and halting your virtual environments but assists with clusters (failover setups, migration and so on), too. CBSD provides tons of options to setup your environments as you need them. It supports resource limitation and accounting. It provides powerful means for automated deployment within jails as well as many other things.
The project is native to FreeBSD and is able to use optional features if their requirements are met (e.g. it's running on ZFS-backed storage). It is not limited to that OS, though. CBSD feels at home on the HardenedBSD variant as well and for example supports the special update process of that platform. It will happily manage your jails and VMs on DragonFly BSD (and supports HAMMERFS features), too. Support for VMs on NetBSD and OpenBSD might be added in the future if there's interest shown in it.
So to put it short: It's your Swiss Army Knife when it comes to any virtualization needs on the BSDs. Fasten your seat-belts, we have _a lot_ to explore. And it will be a wild ride for sure.
CBSD is in ports on DragonFly BSD, FreeBSD and HardenedBSD. But unless you require special firmware to support GPU passthrough (which is currently the only port option), you may want to just install from packages:
# pkg install cbsd
CBSD can also be installed as a package from Ravenports on both DragonFly BSD and FreeBSD. Right now this is discouraged as the port was only added recently and there are still various problems (due to the non-standard prefix) that need to be ironed out. If you're interested, check the date - it's likely that the port will be in good shape before too long (probably in early 2023).
# ravensw install cbsd-single-standard
The ports have few dependencies and are very quickly updated when there is a new release. If you plan on contributing to the development, you may later want to clone the repository from GitHub and use that. The readme file includes the steps needed to do so.
After installing the package, CBSD needs to be set up. The installation message tells you how:
cbsd is now installed, but requires additional setup steps:
Upon initial installation, run:
env workdir="/path" /usr/local/cbsd/sudoexec/initenv
Where /path is the path to the cbsd workdir, e.g.: /usr/jails
For upgrading an existing installation, run:
cbsd initenv
We're going to do this next time. For now let's first understand how CBSD is structured on the filesystem.
As you might have guessed from its many features, CBSD isn't exactly a trivial tool. There's *two* places that you need to know about.
If you install CBSD, its files will spread across several places in your filesystem. The package will populate (note: if you installed CBSD from Ravenports, a custom prefix (i.e. _/raven_) is used instead of _/usr/local_ by default!):
This directory scheme was chosen by the developer and the maintainer of the port according to FreeBSD's standards. If for some reason you really cannot live with that this means that you will have to package the program yourself.
The other one is CBSD's working directory and this is where you get to choose the location (but of course not the structure).
So what is the workdir and what will it contain? The example in the install message is /usr/jails which is reasonable if you plan to use jails only. I frequently use both jails and Bhyve VMs, so I usually create a dataset directly below the filesystem root: /cbsd. Here's what it looks like:
% tree -adL 1 /cbsd
/cbsd
├── .rssh
├── .ssh
├── basejail
├── etc
├── export
├── formfile
├── ftmp
├── import
├── jails
├── jails-data
├── jails-fstab
├── jails-rcconf
├── jails-system
├── modules
├── nodes
├── share
├── src
├── tmp
├── var
└── vm
20 directories
That's quite a bit of stuff there. Let's discuss some of the important ones before we're even setting up CBSD.
% ls /cbsd/basejail
base_amd64_amd64_12.3 base_amd64_amd64_13.0 base_amd64_amd64_13.1
On this machine I can quickly create jails with a 12.3, 13.0 or 13.1 userland. If I wanted to create a 12.4 jail (as that's due to be released in a couple of days as of when I'm writing this), a basejail for it would need to be created first. A basejail is literally the extracted content of the base.txz distribution set of the respective version in most cases.
As you can see, there's plenty going on here. Also you should have gotten an idea that the workdir is the kind of directory / dataset that can take quite a bit of space (depending on what you plan to do). So especially if you're using UFS: Choose the location of your workdir carefully! That already mostly filled-up partition over there might not be the ideal candidate - or it might, if all you want to run is a few small jails. Make use of ZFS if you can (ok, that's more of a general rule with FreeBSD and most probably one that you already follow!). Do some more planning ahead otherwise.
Your workdir is not carved in stone, though. As hinted above, you can even have multiple. However working with multiple workdirs not because you _want_ them separated but due to space constrains is one perfectly avoidable source of frustration.
You should now have a pretty good idea of what CBSD can do for you and know a bit about its structure. But we haven't done anything with it, yet. Actually, at this point we couldn't even do so! Next article will describe the setup process and the choices you need to make there before we finally start creating jails.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup