💾 Archived View for gemini.circumlunar.space › users › kraileth › neunix › 2024 › cbsd_pt8.gmi captured on 2024-08-18 at 20:20:43. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
Saturday, 20. July 2024
[This article has been bi-posted to Gemini and the Web]
Like the previous article this one is not related to any specific virtualization domain but a more general one. We're taking a look at customizing CBSD's configuration, command output, at profiles and hooks.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts. Part 1 covered an introduction of CBSD as well as the installation. Part 2 detailed CBSD's initial setup process. In part 3, jail creation with a dialog-based TUI as well as starting and stopping jails was covered. Part 4 covered creating jails via an interactive script. Part 5 was about removing jails and creating them both from conf files and from the command line. It also covered setting up a network-facing example application. Part 6 discussed jail options and upgrading the base system inside jails. Part 7 was about upgrading CBSD (and backing up metadata) as well as using the help system.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup
(February 2023) Exploring the CBSD virtual environment management framework - part 3: Jails (I)
(April 2023) Exploring the CBSD virtual environment management framework - part 4: Jails (II)
(May 2023) Exploring the CBSD virtual environment management framework - part 5: Jails (III)
(February 2024) Exploring the CBSD virtual environment management framework - part 6: Jails (IV)
(March 2024) Exploring the CBSD virtual environment management framework - part 7: Updating and Help
One of the changes that were made in recent versions of CBSD is that it now requires its own dataset if it's on a ZFS system. If you try to initialize it in a directory where that is not the case, CBSD will now error out:
>>> Installing or upgrading
[Stage 1: account & dir hier]
Error: on ZFS-based systems, the CBSD requires a separate dataset different from the root one (zroot/ROOT/default).
Please create a separate dataset first, e.g.:
/sbin/zfs create -o mountpoint=/cbsd -o atime=off zroot/cbsd
Then re-run: env workdir=/cbsd /usr/local/cbsd/sudoexec/initenv
The reason for this is that there has been a problem with freebsd-update(8) failing in 14.x (unless the default configuration was changed). So if you have existing installations you may want to think about migrating all data so it's placed under its own dataset.
Like any other application, CBSD has to make choices for a default configuration designed to work reasonably well for most people. Of course this is a field where there can be no "one size fits all". So the tool gives you the option to change its behavior in case you have special needs or preferences.
One of the things that I immediately liked about CBSD is its color scheme which helps me to keep a good overview of what is happening. But while this works pretty well for me and probably for a lot of people, there are exceptions. For example a customer that I'm working with has issues related to color-blindness. While he sees that there's something on the screen, he simply cannot read the purple messages. Initially he copied the text from the terminal to an editor window on his local workstation - but that solution gets old really quickly. Fortunately CBSD's author anticipated that people would maybe want to customize things. Let's take a look at how to do that!
Two example commands, colorized output (PNG)
If you just want to run a specific command and suppress colors once, you can do so on the command line. CBSD evaluates the environment variable _NOCOLOR_, so by setting that, you will receive the boring all gray output (or whatever you've configured your terminal like).
The same commands, monochrome output (PNG)
At times this can be useful. But let's say you're stuck with a weird terminal that doesn't cope with colors well (or you just hate them). You can of course simply export the environment variable and add that to your shell's profile - but hold it for a second! There's a better way. CBSD comes with a configuration file that we can use to disable colorized output:
% head -n 5 /cbsd/etc/defaults/global.conf
# Global CBSD configuration
# uncomment to disable colorized output
#NOCOLOR=1
Here we are, it's right at the top of the global configuration file. Just uncomment the line and as far as CBSD is considered, colors are gone on this system.
In the above mentioned case this certainly helps, but it's not an ideal solution because colors are gone completely. The better one is to adopt the colors to the specific needs or preferences. And that's not too hard to do, either! Let's take a look at the configuration file for the color scheme:
% head -n 8 /usr/local/cbsd/etc/defaults/color.conf
# custom color sample.
# for white schema see color-white.conf sample
# Put this file into ~cbsd/etc/ directory.
# See original schema and ansii color defines in
# /usr/local/cbsd/subr/ansiicolor.subr
N0_COLOR="${NORMAL}" # disable color
N1_COLOR="${GRAY}" # normal default CBSD text color
N2_COLOR="${CYAN}" # normal second CBSD color
So what do we have here? This file is an example of an alternate color scheme. Please keep in mind that it's an example. Among other things it sets H1_COLOR to black which means that for example when you do a 'jls' the headers will be invisible if your terminal background is also black. To use it anyway, copy it to your CBSD installation's _etc_ directory. In my case like this:
# cp /usr/local/cbsd/etc/defaults/color.conf /cbsd/etc/
It's also a solid start if you want to customize the color scheme. The comments at the top of the file also point the reader to /usr/local/cbsd/subr/ansiicolor.subr which contains the macros you can use for the colors as well as the default color scheme. I recommend taking a look.
The same commands as above again, but with a custom color scheme (PNG)
While I'm generally happy with the default, I quickly built another example to look pretty different from the standard just to be able to show it off here.
You're definitely very familiar with the output of the 'jls' subcommand by now. Here's what it looks on my test system currently:
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
test1 0 192.168.13.127/24 test1.advancebsd.net /cbsd/jails/test1 Off
test2 0 192.168.13.128/24 test2.advancebsd.net /cbsd/jails-data/test2-data Off
test3 0 192.168.13.129/24 test3.advancebsd.net /cbsd/jails/test3 Off
The second jail stands out due to its path following a different scheme but you cannot see here that they are all pretty different! Remember the example of the help system from the previous article? We can have display 'jls' different pieces of info for example like this:
# cbsd jls display=jname,ver,baserw,astart,allow_zfs
JNAME VER BASERW ASTART ALLOW_ZFS
test1 14.1 0 1 0
test2 14.1 1 1 0
test3 14.0 0 0 1
You can use everything that is accessible via 'jget' and once you know a couple of the keywords, it makes a lot of sense to customize the output! However the long command line is only needed if you're after a one-shot action. If you find that you use a certain combination regularly, it might be a good idea to just make it the default! Let's imagine my use case would make me prefer the output of the above example. All I have to do is to write it to the config file for 'jls':
# echo 'display="jname,ver,baserw,astart,allow_zfs"' > /cbsd/etc/jls.conf
Not too hard, really. But does it work?
# cbsd jls
JNAME VER BASERW ASTART ALLOW_ZFS
test1 14.1 0 1 0
test2 14.1 1 1 0
test3 14.0 0 0 1
You bet it does. And while that's certainly useful, there's more to it: You can even add your own data!
# echo 'echo "Kyle.Katarn@FckTheCan.on"' > /cbsd/jails-system/test2/facts.d/customer
# echo 'echo "Drizzt.DoUrden@Forgotten.Realms"' > /cbsd/jails-system/test3/facts.d/customer
# chmod u+x /cbsd/jails-system/test2/facts.d/customer
# chmod u+x /cbsd/jails-system/test3/facts.d/customer
With just two commands you can add your own custom facts to your virtual environments and they can be displayed by 'jls':
# cbsd jls display=jname,ver,baserw,astart,allow_zfs,customer
JNAME VER BASERW ASTART ALLOW_ZFS CUSTOMER
test1 14.1 0 1 0 -
test2 14.1 1 1 0 Kyle.Katarn@FckTheCan.on
test3 14.0 0 0 1 Drizzt.DoUrden@Forgotten.Realms
We needed to make these files executable since they are treated as shell scripts (don't use a shebang, though!). While that may sound slightly weird at first, consider these examples:
# echo 'date' > /cbsd/jails-system/test1/facts.d/customer
# chmod u+x /cbsd/jails-system/test1/facts.d/customer
# echo 'head -n1 /etc/rc.conf' > /cbsd/jails-system/test2/facts.d/customer
# echo 'uname -s' > /cbsd/jails-system/test3/facts.d/customer
Yes, custom facts can be static data but they don't have to be! These examples work just fine, even though the first one probably doesn't do what you think it does:
# cbsd jls display=jname,customer
JNAME CUSTOMER
test1 Sat
test2 hostname="cbsddemo.advancebsd.net"
test3 FreeBSD
The output of the first command was truncated since the most important limitation here is: _Do not use whitespace_! This does not work. If you want the full 'date' output, pipe it through a command that transforms the space characters to underscores or something. The other examples work as expected, displaying the first line from this system's rc.conf as well as the name of the OS that the machine runs.
With little scripts you can do very powerful things here. Especially if you plan on managing systems that have dozens or even hundreds of jails, the ability to use additional data will likely become crucial in keeping your environments organized.
Another thing that you might want to do if you are becoming a CBSD power user is adding your own profiles. We've only covered jails so far, so I will stick to them in my example, too.
All the standard profiles live in 'defaults' under the 'etc' directory of your CBSD installation (workdir). I frequently create new jails and to save myself the hassle of making some standard changes over and over again, I can create my own profile(s). It's as simple as copying the one that best matches my tastes and then making my changes. In general the default profile is a pretty good candidate:
# cp /cbsd/etc/defaults/jail-freebsd-default.conf /cbsd/etc/defaults/jail-freebsd-kraileth.conf
Selecting the new profile using 'jconstruct-tui' (PNG)
The rule here is that CBSD considers any file matching *.conf in this directory a profile. Depending on the contents it will become available for the selected virtualization domain or not.
Now for some customization. I edited the file to match my preferences more closely. Here's the unified diff:
# diff -u /cbsd/etc/defaults/jail-freebsd-default.conf /cbsd/etc/defaults/jail-freebsd-kraileth.conf
--- /cbsd/etc/defaults/jail-freebsd-default.conf 2024-07-19 19:07:06.940538000 +0200
+++ /cbsd/etc/defaults/jail-freebsd-kraileth.conf 2024-07-19 20:22:44.719983000 +0200
@@ -1,7 +1,7 @@
# JCREATE part
# jail skeldir for overwriting default files use $workdir for relative path from
# workdir or full path to directory. default: $workdir/share/${platform}-jail-skel
-jail_profile="default"
+jail_profile="kraileth"
# default $jail_profile for jconstruct
default_profile="default"
@@ -11,7 +11,7 @@
jail_active="1"
# this is one-string additional info strings in dialogue menu
-long_description="New empty jail"
+long_description="New empty jail without pkg(8)"
# jail name suggestion settings.
# freejname_script - helper for next free jail name recomendatin
@@ -28,7 +28,7 @@
freejname_script="freejname"
# suggest for jail1, jail2, jail3 as new jail name.
default_jailname="jail"
-default_domain="my.domain"
+default_domain="advancebsd.net"
# User area
user_pw_root="cbsd"
@@ -70,7 +70,7 @@
baserw="0"
mdsize="0"
mount_src="0"
-mount_ports="1"
+mount_ports="0"
mount_obj="0"
astart="1"
interface="auto"
@@ -109,7 +109,7 @@
# default password (is empty - use skel files where password is 'cbsd')
user_pw_root=''
-pkg_bootstrap="1"
+pkg_bootstrap="0"
# default index order for this group
b_order="10"
As you can see, I changed the profile name which is the bare minimum that needs to be changed to create a new one. Then I changed the description which is optional but makes sense. A more interesting change is that I chose a different default domain so I don't have to type it in every time. I don't like the ports tree mounted in every jail, so that option goes off. And finally I disable the pkg(8) bootstrap.
To explain why one would want to do the latter: I participate in the Ravenports project which provides an alternative means of installing packages on FreeBSD and other operating systems. While that does not mean you cannot use both I try to use RP only whenever possible (and force myself to create new ports when I'm missing something).
Default options after selecting the custom profile (PNG)
If you want to add your own customized profiles, I highly recommend taking a good look around in the default profile. Oh, and always _copy_ a profile before making changes! While technically you can of course edit for example the default profile, this may get overwritten during an update which means you'll lose your customizations.
After taking a look at all the definitions brought over from copying the default profile, have a look at another one that CBSD comes with: /cbsd/etc/defaults/jail-freebsd-vnet.conf. You will find that it's much, much shorter. The reason for this is that it's fine if a profile contains only the definitions that are different from the default! So while the approach I used above works, it's very much possible to reduce your profiles to just a couple of lines instead of >200.
One of the more interesting definitions in profiles that I want to point you to is _jailskeldir_. You've seen the following line before when creating jails:
Applying skel dir template from: /cbsd/share/FreeBSD-jail-skel
This is actually the directory which CBSD uses to populate the additional data for your jail. It comes with several skeleton directories that can be used. Let's have a look:
% ls -d /cbsd/share/FreeBSD-jail-*
/cbsd/share/FreeBSD-jail-puppet-skel
/cbsd/share/FreeBSD-jail-puppet-system-skel
/cbsd/share/FreeBSD-jail-skel
/cbsd/share/FreeBSD-jail-vnet-skel
Alright. What do these directories contain? Here's the default one:
% ls /cbsd/share/FreeBSD-jail-skel
etc root
% ls /cbsd/share/FreeBSD-jail-skel/root/
.bashrc .cshrc
% grep "prompt =" /cbsd/share/FreeBSD-jail-skel/root/.cshrc
set prompt = "%{^[[40;35;1m%}`/bin/hostname -s`:%{^[[40;31;1m%}%/@%{^[[40;33;1m%}%B[%T]%b # "
Aha! This is where the special prompt that CBSD jails come with originates from! CBSD's skel directories work just like the system skel directories that come into play when creating new users. They are overlays that can be used to add or change files in addition to the respective bases.
There's a lot more to see here. To just give you one example of how fancy you can get, take a look at the following comment lines in the default profile:
# jail name suggestion settings.
# freejname_script - helper for next free jail name recomendatin
# To use internal 'cbsd freejname' script, leave it as
#
# freejname_script="freejname"
#
# If you want to use your own implementation (e.g. for
# multinode setup to avoid name collisions, please use
# the full path to the executable script.
# external script will be executed with default_jailname args:
# ${freejname_script} default_jailname="${default_jailname}"
# see below.
This option lets you use a custom script for determining the next free jail name according to a defined scheme. The default one is fine but as the comments point out, especially in multi-node setups you may want to avoid being able to create jails on any node which get a name that may be available locally but can be taken on another node! Not only is such a situation going to foster confusion but it might lead to collisions when trying to do jail migrations!
If you plan on writing a clever script for your environment, CBSD is happy to make use of it if you point it at the right file (if you want to take a look at how the default one works, you can find it here: /usr/local/cbsd/tools/freejname).
Let's say that I find myself mostly using the new profile. In that case it may be a good idea to make it the default profile like this:
# echo "default_profile='kraileth'" >> /cbsd/etc/jail-freebsd-default.conf
Take a closer look at this: We did _not_ change /cbsd/etc/defaults/jail-freebsd-default.conf but created a new file one level higher. This works much the same way as settings made in /etc/rc.conf override those defined in /etc/defaults/rc.conf. Which in turn means that if I don't plan on using the original default, I could simply have defined the few changes I made to have another default domain, no ports mounted and pkg(8) not bootstrapped in /cbsd/etc/defaults/jail-freebsd-default.conf and be done!
I would recommend the latter for another reason as well; think of the 'defaults' directory as owned by CBSD. Yes, you're root, you can mess with it, but maybe you shouldn't. If you use overrides, version upgrades will not mess you up. And by not doubling all the information as I did with the new profile, you also don't cut yourself off from changes to the profiles as they are updated. You could deliberately do it, of course, and there are valid reasons to do so. But consider your options.
Speaking of options: There's three levels in the configuration hierarchy! In the command output section we used /cbsd/etc/jls.conf to override the /cbsd/etc/defaults/jls.conf. That's a general command, but of course we can also modify commands that affect a single jail. And let's assume we want it behave differently for only a _specific_ jail but not all the others. CBSD lets you do this as well.
Take a look at /cbsd/etc/defaults/jlogin.conf for example. It contains several other possible ways to do the login into jails. Maybe I always want to enter jail3 via SSH so my colleagues can easily see when I'm logged into the machine. Since I manage the jail host, I keep forgetting to use SSH and log in via CBSD. Here's what I could do:
# mkdir /cbsd/jails-system/test3/etc
# echo 'login_cmd="echo S S H, man!! You forgot again..."' > /cbsd/jails-system/test3/etc/jlogin.conf
So while 'jlogin' works normally for all the other jails, whenever I mess up and try to log into jail3, this is what happens:
# cbsd jlogin test3
Custom login command: echo S S H, man!! You forgot again...
S S H, man!! You forgot again...
As you can see, CBSD also tells the user that this is a custom login command which can be useful in case you configured something and a year later it completely slipped your mind.
This means within CBSD's workdir you have three places where you can customize configuration:
As usual, CBSD proves to be extremely flexible. At least I'm amazed time after time how many possibilities arise and how hard it is to come up with a sensible use case that simply cannot be done with the tool.
The last topic that I wanted to cover in this article are hooks. While CBSD has had this functionality for a while, another recently added feature is a backup hook that both is pretty useful to the admin and makes a nice example.
On my test machine I've deliberately installed an outdated version of CBSD (14.0.8), so that I can show what happens during updates now. So let's upgrade to 14.1.0!
# env workdir=/cbsd cbsd initenv
Warning: CBSD is 14.1.0 while workdir initializated for 14.0.8. Please re-run: env workdir=/cbsd cbsd initenv
-------[CBSD v.14.1.0]-------
This is install/upgrade scripts for CBSD.
Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
y
>>> Installing or upgrading
[Stage 0: upgrading and migration data from 14.0.8 to 14.1.0]
Warning: CBSD is 14.1.0 while workdir initializated for 14.0.8. Please re-run: env workdir=/cbsd cbsd initenv
Warning: CBSD is 14.1.0 while workdir initializated for 14.0.8. Please re-run: env workdir=/cbsd cbsd initenv
Warning: CBSD is 14.1.0 while workdir initializated for 14.0.8. Please re-run: env workdir=/cbsd cbsd initenv
Warning: CBSD is 14.1.0 while workdir initializated for 14.0.8. Please re-run: env workdir=/cbsd cbsd initenv
pre-initenv-backup in progress ( can be disabled via ~cbsd/etc/initenv.conf ): initenv_backup_bases=3
* pre-initenv-backup: [1/3] exprort/backup settings for: test2
* pre-initenv-backup: [2/3] exprort/backup settings for: test1
* pre-initenv-backup: [3/3] exprort/backup settings for: test3
* Update expose tables: fromips
* Update expose tables: fromips
[Stage 1: account & dir hier]
* Check hier and permission...
cmdboot: no such optional executable: xl, xen_feat disabled
[...]
The line "pre-initenv-backup in progress" and the next few are new. What is happening here is that the 'pre-initenv-backup' script is executed which tells us that it can be disabled via initenv.conf. Let's take a look at the default file first:
% cat /cbsd/etc/defaults/initenv.conf
# pre-initenv auto DB backups
# How many SQLite3 DB backups to keep, '0' - turn off.
initenv_backup_bases=3
Remember that in a previous article on updating I recommended making a backup copy of the 'db' directory? That is CBSD's new default action before doing any upgrade! You can take a look at the script (/usr/local/cbsd/upgrade/backup_db/pre-initenv-backup) for the nitty-gritty but here's the most important thing for the admin, the result:
% ls -d /cbsd/var/db*
/cbsd/var/db /cbsd/var/db-20240719220008
As you can see, the script created a backup folder so you and me don't have to. By default it will keep up to three backups and when it would create a fourth one delete the oldest backup. This is pretty convenient and by creating your own 'initenv.conf' you can tell it to keep more or less backups or none at all if you set it to 0.
This is only one example of the hooks system which can do much more for you if you want. The initenv script (/usr/local/cbsd/sudoexec/initenv) executes the special db backup script but is ready to use any custom hooks if you place them in the right place. It's a comprehensive script of almost 1,5k lines at this time. Among them are the following:
[ -d "${workdir}/upgrade" ] && env workdir="${workdir}" /usr/bin/find ${workdir}/upgrade/ -mindepth 1 -maxdepth 1 -type f -name pre-initenv-\* -exec {} \;
[ -d "${distdir}/upgrade" ] && env workdir="${workdir}" /usr/bin/find ${distdir}/upgrade/ -mindepth 1 -maxdepth 1 -type f -name pre-patch-\* -exec {} \;
as well as:
[ -d "${distdir}/upgrade" ] && env workdir="${workdir}" /usr/bin/find ${distdir}/upgrade/ -mindepth 1 -maxdepth 1 -type f -name post-patch-\* -exec {} \;
[ -d "${workdir}/upgrade" ] && env workdir="${workdir}" /usr/bin/find ${workdir}/upgrade/ -mindepth 1 -maxdepth 1 -type f -name post-initenv-\* -exec {} \;
So if you create a directory named 'upgrade' inside your CBSD workdir and put any files in there whose names begin with 'pre-initenv-' or 'pre-patch-' they will be run before the respective upgrade stages. And if you put files there whose names begin with 'post-patch-' or 'post-initenv-' you can have hooks which are executed after the respective stages are completed.
This is obviously very flexible and allows for a wealth of scenarios. Maybe you want to log some additional data during upgrades. Chances are you'd like your jails to be stopped and snapshotted before a CBSD upgrade and then started again. Or you have a multi-node setup and before an upgrade want CBSD to migrate all jails over to another node, do the upgrade and then migrate them back? This and more can be done if you utilize the hooks system. Basically your imagination (together with your scripting foo) is the limit here!
As usual I owe a debt of gratitude to Oleg not just for his work on CBSD but also for proofreading the articles of this series and giving very valuable input. A lot of the more interesting things discussed here would be missing if he hadn't given me pointers (either because I failed to see how well they fit in this topic or because I was ignorant about that feature).
We've once again covered quite a bit of ground. After reading this part of the series you should not only be quite capable of using CBSD to manage simple jails but you can truly make this great tool _your tool_.
The next article will be about a different topic. I definitely plan on coming back to CBSD as we have not even covered all the basics, yet. While I haven't decided on which topic to write about next there's several on my list, so I got more than enough to pick something from.
