Saturday, 30. March 2024
[This article has been bi-posted to Gemini and the Web]
Before I started diving into basic jail management with CBSD, I wrote two articles about the tool in general. This one is more in line with those as we are going to take a look at how to update CBSD (and doing a backup beforehand). We will also be exploring a bit of the pretty comprehensive help system.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts. Part 1 covered an introduction of CBSD as well as the installation. Part 2 detailed CBSD's initial setup process. In part 3, jail creation with a dialog-based TUI as well as starting and stopping jails was covered. Part 4 covered creating jails via an interactive script. Part 5 was about removing jails and creating them both from conf files and from the command line. It also covered setting up a network-facing example application. Part 6 discussed jail options and upgrading the base system inside jails.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup
(February 2023) Exploring the CBSD virtual environment management framework - part 3: Jails (I)
(April 2023) Exploring the CBSD virtual environment management framework - part 4: Jails (II)
(May 2023) Exploring the CBSD virtual environment management framework - part 5: Jails (III)
(February 2024) Exploring the CBSD virtual environment management framework - part 6: Jails (IV)
Getting a new version of CBSD is fairly easy. If you need the very latest release basically the moment it becomes available, watch it on GitHub and clone it from there (the README.md file covers how to do this). Unless you are planning to join CBSD development there are few reasons to do so, though.
Kirill Ponomarev does an outstanding job on keeping the FreeBSD port current (it usually gets updated on the next day after a new release or even on the very day!). So by using ports you can get the latest release in a very timely manner if you wish. Make sure to use a ports builder like Poudriere or Synth, though. The times when building software on production machines directly from ports was considered an ok thing are long gone. If you are not familiar with package building on FreeBSD but are interested in the topic, I've written an article series on Synth, too (and plan to eventually cover Poudriere as well).
FreeBSD package building pt. 1: Introduction and test system
FreeBSD package building pt. 2: Basic Synth
FreeBSD package building pt. 3: Intermediate Synth
FreeBSD package building pt. 4: (Slightly) Advanced Synth
FreeBSD package building pt. 5: Sophisticated Synth
Chances are however that you can wait about a week until the package build cluster finishes building the most recent package set. If you have pkg(8) configured to use the _latest_ rather than the _quarterly_ repository, you are set for timely updates.
The situation is different if you are on DragonFly BSD with the port often being out of date for months in dports. On both fbsd and dfly you could use Ravenports, but while the port is usually updated rather quickly, new releases happen infrequently and full package sets are not published too often currently, either.
As you know, CBSD is doing _a lot_ of things and therefore it is not a trivial piece of software. When dealing with such a thing, you have two choices:
I've been using CBSD for years now and while I'm a bit lazy with my own stuff, I'm trying to do _the right thing_(tm) when dealing with production systems from customers (yes, I run CBSD in production and consider it a competitive advantage!). IIRC, there has been exactly one case where an upgrade failed in a spectacular way - and while this was entirely my own fault (I updated from a _horribly outdated_ version), I contacted the author and he helped me update properly. Even in that case, no data was lost.
Of course I'd advise you to lean towards the second strategy, especially as it's not a whole lot of work actually. But what does it look like? I do updates like this:
And that's really it.
Let's assume I found a server that I somehow forgot about for a while, so it's still running an older version of CBSD. I've built a test system to simulate this. There are two jails running on it:
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
test1 1 192.168.13.127/24 test1.advancebsd.net /cbsd/jails/test1 On
test2 2 192.168.13.128/24 test2.advancebsd.net /cbsd/jails/test2 On
And CBSD is still at an outdated version:
# cbsd version
13.2.6
So first I create a backup package of the currently installed CBSD like this:
# mkdir /root/pkg_bak && cd /root/pkg_bak
# pkg create cbsd
Creating package for cbsd-13.2.6
If you don't have a lot of packages installed on the host (which might very well be the case considering it's a jail host or VM host, right?) or you want to be extra cautious, you can issue _pkg create -a_ to have pkg(8) create backup packages for all the currently installed software.
Next is upgrading the package version:
# pkg upgrade cbsd
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
cbsd: 13.2.6 -> 14.0.3
Number of packages to be upgraded: 1
Proceed with this action? [y/N]: y
Now is the time to backup the database. I just make a copy of the complete directory for that (my workdir is /cbsd, adapt if yours is in another place):
# cp -r /cbsd/var/db /cbsd/var/db.13_2_6
If you want to be on the safe side completely, also backup the jails-system directory as for every environment it contains data that could be subject to format changes during upgrades:
# cp -r /cbsd/jails-system /cbsd/jails-system.13_2_6
Both directories should be fairly small - the former probably only several kilobytes, the latter below hundred megs unless you have a lot of environments.
As the last step, let CBSD update its files and db by running _initenv_:
# env workdir=/cbsd cbsd initenv
Warning: CBSD is 14.0.3 while workdir initializated for 13.2.6. Please re-run: env workdir=/cbsd cbsd initenv
-------[CBSD v.14.0.3]-------
This is install/upgrade scripts for CBSD.
Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
y
>>> Installing or upgrading
[Stage 0: upgrading and migration data from 13.2.6 to 14.0.3]
[Stage 1: account & dir hier]
* Check hier and permission...
cmdboot: no such optional executable: xl, xen_feat disabled
[Stage 2: build tools]
[Stage 3: local settings]
[Stage 4: update default skel resolv.conf]
[Stage 5: refreshing inventory]
[Stage 6: authentication keys]
[Stage 7: nodes]
[Stage 8: modules]
[Stage 9: cleanup]
* Remove obsolete files...
Removing obsolete file: /cbsd/etc/defaults/jail-freebsd-centos_7.conf
[...]
Removing obsolete file: /cbsd/etc/defaults/vm-linux-cloud-Fedora-36-x86_64.conf
[Stage X: upgrading]
>>> Done
The warning in line 1 is a little redundant in this case as CBSD suggests exactly what we're trying to do right now. It's a good message to have, though, letting you know with each CBSD action when your workdir is out of sync with the installed CBSD version. This means that you are less likely to not notice accidentally updating CBSD with your regular package upgrades.
As you can see in the "Stage 0" part, the script found that it's not meant to create a fresh working directory. It detected CBSD 13.2.6 data and proceeds to upgrade it for continued use with the new version, 14.0.3. During the cleanup phase it removes various obsolete profile files, but the details are not too important here so I cut most of the lines.
At the end of the upgrade, I regularly do this:
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
test1 2 192.168.13.127/24 test1.advancebsd.net /cbsd/jails/test1 On
test2 1 192.168.13.128/24 test2.advancebsd.net /cbsd/jails/test2 On
This is the expected output: The warning is gone and CBSD still sees two running jails (if you are running Bhyve VMs or something else, check these).
The one time my update went wrong I got this:
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
Uh, oh ... I knew that the jails were still running, because the monitoring system said the services were up. But CBSD couldn't "see" them anymore! So I first I removed the new CBSD version:
# pkg delete cbsd
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
Installed packages to be REMOVED:
cbsd: 14.0.3
Number of packages to be removed: 1
The operation will free 16 MiB.
Proceed with deinstalling packages? [y/N]: y
Next was re-installing the old version from the backup package:
# pkg add /root/pkg_bak/cbsd-13.2.6.pkg
Installing cbsd-13.2.6...
===> Creating groups.
Using existing group 'cbsd'.
===> Creating users
Using existing user 'cbsd'.
Extracting cbsd-13.2.6: 100%
[...]
Of course the old db directory had to be restored as well:
# mv /cbsd/var/db /cbsd/var/db.new
# mv /cbsd/var/db.13_2_6 /cbsd/var/db
With the old state restored, I was able to interact with the jails again (which bought me some time to figure out what had gone wrong).
When you need more information about any aspect of CBSD, there are multiple sources available. A lot of things are covered in the official documentation that you can view on the Web here:
While it is pretty comprehensive, it's structure takes some getting used to. If you are interested in what CBSD can do, I would suggest you take a look at which topics are covered and read the ones that sound interesting. There's a lot of good information in there which you can pick up along the way. Since CBSD is such a versatile tool, the sheer number of features can be overwhelming, though (which is part of the reason I started this article series, trying to cover the basics first so you can start benefiting from using the manager without getting confused by all the advanced options). If you run into something that sounds pretty sophisticated, put it aside for later when you have more experience. For example: Definitely get confident managing single nodes before you explore clustering!
Finding specific things in the documentation can be a bit difficult, though (I had multiple "oh, that's covered THERE!" moments when reading the docs and I have heard that I haven't been the first to struggle at times). More often than not you might want to know more about some functionality you already know exists rather than look for completely new things. Not just in that case but especially then you should take a look at CBSD's built-in help system.
Get a list of help topics available by running:
# cbsd help
Top entries of the CBSD help list (PNG)
It's a long list - covering over 270 subcommands at this point (see the ones at the top on the screenshot)! There's one main thing to know which will make you cope more easily with this massive list. Take a look at the first three entries from that list starting with the letter 'b':
bases -- [jail,sys] Show registered basejail for jails
baseupdate -- [sys] Update base jail
bcheckpoint -- [bhyve] bhyve checkpoint
As you can see, there's always the subcommand name, a separator and a short description. In front of the latter, you have something in square brackets (they can be empty, though). This is what CBSD calls _modules_ and is basically the _domain_ (as in _topic_ not as in FQDN) that the subcommand relates to. The 'bcheckpoint' subcommand is used only in conjunction with Bhyve while 'baseupdate' is related to the host system and 'bases' is relevant both managing things on the host and for jails.
While a lot of the subcommands that make sense for more than one domain are prefixed by the respective letter like 'jls' to list jails, 'bls' to list Bhyve VMs and so on, this is only a first indicator at which module a subcommand belongs to. There will likely be other subcommands for the module, too. Therefore CBSD lets you list subcommands for a specific domain only if you wish. For example to get a list of all jail-related ones, you can do this:
# cbsd help module=jail
[...]
Available commands:
apply -- [bhyve,jail] apply/re-configure virtual environment via CBSDfile
bases -- [jail,sys] Show registered basejail for jails
cpuset -- [bhyve,jail] show cpuset info
destroy -- [bhyve,jail] destroy jail/bhyve env from CBSDfile, vagrant-like behavior
[...]
This narrows down the list to about 50 subcommands. While that's still a lot, it's much better, isn't it?
So which modules are there and what are those for? Here's a list of the 11 modules CBSD uses currently:
So far we've only explored CBSD's jails capabilities, so for an example let's ask for help on a command that we already know about. What could be simpler than _jls_ the list command for jails?
# cbsd help jls
This will print the following help text (the examples and everything is part of it):
Options:
alljails= - (0 or 1): force to display foreign/remote resources. When
sqlreplica=1 and node available, alljails sets to 1 automatically;
display= - comma separated list of columns to display.
Default value: jid,jname,ip4_addr,host_hostname,path,status
When sqlrepica and node available:
nodename,jname,jid,vm_ram,vm_cpus,vm_os_type,ip4_addr,status,vnc_port
Additional field: real_ip4,etcver
Default display= setting can be reassigned globally via ~cbsd/jls.conf;
header= - don't print header info;
jname= - limit list by 'jname=' env only, alias for: cbsd jls xx';
node= - list only jails of the specified node;
order= - sort older, 'asc' (default) or 'desc';
shownode=1 - show node name(s) for listed jails;
When RCTL enabled, additional field available: datasize stacksize coredumpsize memoryuse memorylocked maxproc openfiles vmemoryuse pseudoterminals swapuse nthr msgqqueued msgqsize nmsgq nsem nsemop nshm shmsize wallclock pcpu readbps writebps readiops writeiops cpu fsquota nice bw
Examples:
# cbsd jls
# cbsd jls myjail1
# cbsd jls header=0 display=jname,status,ip4_addr,astart
# cbsd jls display=jname,cpu,vmemoryuse,fsquota,ver,etcver
# cbsd jls ver=13.1 astart=1
See also:
cbsd jget --help
cbsd jrctl --help
cat ~cbsd/etc/defaults/jls.conf
Check out manual page: man cbsd-jls
External help: https://www.bsdstore.ru/en/13.x/wf_jls_ssi.html
Wow! That's probably more info than you would have expected for such a basic little command. At least the first time I saw this, I was surprised. It serves as a nice example that CBSDs help system like the tool in general had quite a bit of thought put behind it (and work put into, of course).
Whenever in doubt, use the available help! I know that with many programs the help often enough is not that helpful at all. With CBSD though, it's actually useful - and it's really good to know that.
Should you run into problems that you cannot solve despite reading the documentation, reach out to the author. The telegram channel mentioned in the help list is the preferred way, I think. If you feel like there's a gap in the docs or something is wrong, opening an issue on GitHub is what you should do to let the developers know.
And with that I leave you to play around with CBSD on your own some more. In case you find something in the documentation that you think I should have covered by now, please let me know. Also feel free to share anything else you think is interesting. This article series is far from over and while I have something of a general plan, I'm very much open to suggestions.
The next article will be about customizing CBSD to best fit your use-case and taste. We will take a look at using profiles as well as changing default settings and I plan on covering hooks as well.
(July 2024) Exploring the CBSD virtual environment management framework - part 8: Customizing CBSD
