Thursday, 29. February 2024
[This article has been bi-posted to Gemini and the Web]
After a _long_ forced break (sorry, life – and unfortunately the opposite, too – got in the way) I finally managed to get back to what I think is one of the most important topics I wanted to cover further: CBSD. This article will conclude the basic jails management parts, picking up several loose ends the previous ones left.
As quite some time passed, my test system is now on FreeBSD _14.0_, and CBSD advanced to version _1.14.03_.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts. Part 1 covered an introduction of CBSD as well as the installation. Part 2 detailed CBSD's initial setup process. In part 3, jail creation with a dialog-based TUI as well as starting and stopping jails was covered. Part 4 covered creating jails via an interactive script. It used an unusual example (a 32-bit jail of an outdated version of FreeBSD) to show some important concepts like how filesystem mounts work. Part 5 was about removing jails and creating them both from conf files and from the command line. It also covered setting up a network-facing example application.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup
(February 2023) Exploring the CBSD virtual environment management framework - part 3: Jails (I)
(April 2023) Exploring the CBSD virtual environment management framework - part 4: Jails (II)
(May 2023) Exploring the CBSD virtual environment management framework - part 5: Jails (III)
Whenever you want to do more complex things inside a jail, you log in and can then use the command line to the heart's content (within the limits that jails impose on you). But sometimes you just want to run a single command, so logging in first, issuing the command and logging out again is a little tedious. CBSD's _jexec_ subcommand to the rescue!
For example I cannot remember whether I created my user in the test jail or not and want to check real quick. I can do that like this:
# cbsd jexec jname=demojail getent passwd kraileth
No output? The user is probably not there. But to be real sure that this method of checking is actually valid, let's try to look up a user we know will exist:
# cbsd jexec jname=demojail getent passwd root
root:$6$2ReTIOBFYBnnfMlo$INeuu.qChDM/w/POLUSs7NaYSgEs46nSkOLKLhUDlsygtQIcoZd0bJPbKSM4hW/J9PvfXU17A9TMqwxYmwz2H/:0:0:Charlie &:/root:/bin/csh
Yes, there we are, it works. CBSD ran the given command inside the selected jail and reported the output back.
In a previous part it was already mentioned what CBSD's warning, not to edit the conf file, means after it was used to create a jail. But how do you actually change the configuration of existing jails? As you might have guessed by now, there's more than one option. But one thing at a time. Let's say we have a jail named "demojail" on our host. It's running but its service has what looks like a network problem. One of the most simple tests to see whether communication with another host is even possible, is to use the ping(8) utility. Let's give it a try by issuing this from the host:
# cbsd jexec jname=demojail ping -c1 bsdnow.tv
ping: ssend socket: Operation not permitted
If you are just getting started with jails, the result might not exactly be what you expected. So what's happening here? Ping works by sending ICMP packets which requires access to raw ports. However those are not allowed inside jails by default!
So if it's the problem with a default setting, chances are that we can change it, right? Indeed we can, for example using CBSD's _jconfig_ subcommand:
# cbsd jconfig jname=demojail
This opens up a menu with options that should look rather familiar (as it is similar to the one used to create the jail in part 3 of this article series):
CBSD jconfig dialog menu (PNG)
There's the _jail options_ submenu where we need to go. Also don't miss the warning printed at the top: Since we are editing the settings for a jail that's currently running, CBSD provides a reduced list of options only. If you are looking for something that you think should be there but it isn't, it's probably something that cannot be changed on-the-fly for jails that are up. In that case stop the jail and try _jconfig_ again for an even more comprehensive list.
CBSD jconfig jail options submenu (PNG)
In the submenu you will find the "allow_raw_sockets" option. Enable it and save the settings, then we can try again:
# cbsd jexec jname=demojail ping -c1 bsdnow.tv
PING bsdnow.tv (192.73.246.162): 56 data bytes
64 bytes from 192.73.246.162: icmp_seq=0 ttl=48 time=132.948 ms
--- bsdnow.tv ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 132.948/132.948/132.948/0.000 ms
Alright, now the ping command works!
Probably the ping was helpful and the problem the jail had could be solved. Time to reflect on what we just did. Yes, we obviously made the ping command work. Great! In fact that is a tool convenient enough to consider just letting that setting enabled (or even enabling it for all your future jails because being able to use it could be useful, right?).
Of course you can do that. It's your jails and your responsibility. There's a reason why the option defaults to off, though: A jail is meant to be a severely constrained environment and one of its benefits is that it's restricted to only its IP address. By allowing raw sockets you have basically lifted that limitation.
I'm not saying you should or should not set that option. Thinking about your use-case and doing the thread modelling will remain your task whether you follow nice tutorials on the net or not. So always consider the consequences of your actions! When you put someone in jail, you probably expect him or her to stay in the cell. Providing a lock-pick that would allow a skillful inmate to escape rather easily might not be such a great idea.
There's voices making the statement that jails are not really a security tool and they are able to make some valid points. Without claiming to be much of a security expert, I disagree. Jails _can_ be a useful component in your security architecture. They are not a cure-all, though, that magically makes things "secure". It's still up to you to carefully consider the furniture of your cells – and whether you want to leave the door open on purpose or not.
Yes, doing your research before setting options recommended somewhere on the Internet to solve specific problems is rather inconvenient. I encourage you however to think about how much more inconvenient you will make your jails for potential attackers of your valuable systems. Is that task beginning to look more inviting to you now? If not, I would suggest to leave advanced options alone. People who did think about this a lot have chosen them to be a good fit for the majority of cases. Could yours be the exception to the rule? Certainly. If you still need more motivation to do your homework, consider paying an expert to do it for you. The figure on the expected quote might help _a lot_.
Once more the dialog menu is really nice, especially when you're just getting started with CBSD. Of course you cannot change the settings at all from within the jail (that's the point, right?), but we're didn't need to log in for this simple test, anyway.
As useful as the menu system is, it's a bit of a hassle to navigate through it to change settings, isn't it? If you agree and can memorize the names of the options important for you, there's a much easier way to change settings on the command line using _jset_:
# cbsd jset jname=demojail allow_raw_sockets=0
allow_raw_sockets: 0
You might think that where there's a setter there would also be a getter – and you would be right of course. We can use the _jget_ subcommand to get the current value of the setting and confirm that the previous command line did just what we expected it to do:
# cbsd jget jname=demojail allow_raw_sockets
allow_raw_sockets: 0
Alright! No more raw sockets for this jail. Previously opened loophole closed again.
While updating third party packages within jails can be done using pkg(8) as you normally would, you might wonder how to upgrade the OS part of the jail (i. e. the base system). Let's see what version the example jail uses:
# cbsd jget jname=demojail ver
ver: 13.1
Whoops, that's not supported anymore and should really be upgraded to 13.2 (since 13.3 is just around the corner I would have loved to take that opportunity but at the same time didn't want to delay this article any longer). As you might have suspected, CBSD is going to assist you with the upgrade. How to do it depends on the type of jail you are looking at, though:
# cbsd jget jname=demojail baserw
baserw: 1
Ok, this is a jail with a writable base system. The first step is to set the desired new version:
# cbsd jset jname=demojail ver=13.2
ver: 13.2
Of course the jail should not be running at this time. If it is, stop it. Otherwise you will get this message when trying to change the version:
ver: on-the-fly currently unimplemented
If you start the jail now, CBSD is smart enough to figure that the defined version does not match the actual version and will let you know:
Notice: You have a more recent version of the base in /cbsd/basejail/base_amd64_amd64_13.2 (1302001/1301000).
Notice: Please consider upgrading jail base via cbsd jupgrade
But let's use the _jupgrade_ subcommand while the jail is off:
# cbsd jupgrade jname=demojail
If the target base is not already available for CBSD to use, it will present you with the familiar options to get it, with the easiest one being to fetch it from the repo. Recent versions of CBSD have added another option here, _pkg_ which means to use the pkgbase feature – but this is considered experimental and we will not look into it today.
When it finished downloading the new base (or if it was already available), you will see something like this:
Reduce jail data by switching from baserw -> basero: /cbsd/jails-data/demojail-data
Populate jail data from: /cbsd/basejail/base_amd64_amd64_13.2
2231945 blocks
CBSD temporarily switches the base system to read-only, does the upgrade and switches it back.
For jails that have a read-only base, the process is even simpler. Trying to use _jupgrade_ will result in this message:
Jail is basero mode. Therefore if you want to update the base version to another one, just change ver params in cbsd jconfig
If you want to update base files (for all jails in basero mode and current base version), please use: cbsd repo action=get sources=base mode=upgrade
And indeed: Just change the version for any jail that is not currently running and the next time it is started, the correct version of the base system will be ro monunted into the jail!
This took care of the base system's binaries, libraries and so on. But the configuration changes still need to be applied. Should you forget that step, CBSD once again is nice enough to let you know and even gives you a hint about what you should do:
Warning: jail version (13.2) and distribution (e.g: /etc dir content) (13.1) is differ
Warning: you can use etcupdate to sync content: cbsd etcupdate jname=demojail mode=update from=13.1 to=13.2 mode=diff
So let's do just that:
# cbsd etcupdate jname=demojail mode=diff from=13.1 to=13.2
etcupdate source hier: /cbsd/src/src_13.2/etcupdate/current
Index: /etc/group
===================================================================
--- /etc/group (stock)
+++ /etc/group (local)
@@ -18,8 +18,6 @@
mailnull:*:26:
guest:*:31:
video:*:44:
-realtime:*:47:
-idletime:*:48:
bind:*:53:
unbound:*:59:
proxy:*:62:
Index: /etc/master.passwd
===================================================================
[...]
There's quite a bit more output, but this should suffice for you to recognize what is happening here (if not ... Well, I would suggest reading the manpage for etcupdate(8) and / or taking a look at the FreeBSD handbook again!).
The diffs look good to me, so let's do the actual update:
# cbsd etcupdate jname=demojail mode=update from=13.1 to=13.2
etcupdate source hier: /cbsd/src/src_13.1/etcupdate/current
etcupdate destination hier: /cbsd/src/src_13.2/etcupdate/etcupdate.tgz
etcupdate: created backup: /cbsd/jails-system/demojail/etcupdate/backup/20240229185448.tgz
[debug]: /usr/sbin/etcupdate -d /cbsd/jails-system/demojail/etcupdate -t /cbsd/src/src_13.2/etcupdate/etcupdate.tgz -D /cbsd/jails-data/demojail-data -I /etc/\*.db -I /root/\* -I /etc/hosts -F
U /etc/defaults/devfs.rules
U /etc/defaults/rc.conf
[...]
U /etc/ssh/sshd_config
Warnings:
Needs update: /etc/localtime (required manual update via tzsetup(8))
etcupdate: update successfull, bootstrap for: 13.2
And that's it! If you start the jail again now, CBSD will no longer complain as everything is in sync and matches the defined jail version. Also you probably saw the warning issued by etcupdate and may want to take care of it.
If you paid close attention when CBSD first populated any base system, you might have caught this:
repo: extracting base...
Bases registered: /cbsd/basejail/base_amd64_amd64_13.2
register_base: auto_baseupdate=0 via FreeBSD-bases.conf, updates disabled
register_base: you might want to do cbsd baseupdate by hand to fetch latest patches
What this means is that while we just updated our test jail above from 13.1 to 13.2, this results in the userland being the unpatched release version! Let's check that real quick. Of course we first have to start the jail:
# cbsd jstart demojail
# cbsd jexec jname=demojail freebsd-version
13.2-RELEASE
Indeed, this is 13.2 without any errata fixes applied. Let's stop the jail again:
# cbsd jstop demojail
As suggested by CBSD, we can use the _baseupdate_ subcommand as a remedy (specifying the version will update only that one base while leaving it out will have CBSD try to update every local base available on the host):
# cbsd baseupdate ver=13.2
Updating /cbsd/basejail/base_amd64_amd64_13.2: FreeBSD 13.2-RELEASE
/usr/sbin/freebsd-update -f /tmp/tmp.11NgS8fgpu -b /cbsd/basejail/base_amd64_amd64_13.2 --not-running-from-cron
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 13.2-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata patches.. done.
Applying metadata patches... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 23 patches.....10....20. done.
Applying patches... done.
The following files will be added as part of updating to
13.2-RELEASE-p10:
/etc/ssl/certs/0179095f.0
/etc/ssl/certs/08063a00.0
[...]
/var/db/etcupdate/current/etc/ssh/sshd_config
/var/db/mergemaster.mtree
Installing updates...
Restarting sshd after upgrade
Cannot 'restart' sshd. Set sshd_enable to YES in /etc/rc.conf or use 'onerestart' instead of 'restart'.
Scanning /cbsd/basejail/base_amd64_amd64_13.2/usr/share/certs/blacklisted for certificates...
Scanning /cbsd/basejail/base_amd64_amd64_13.2/usr/share/certs/trusted for certificates...
done.
baseupdate done in 52 seconds
Once again the output will be familiar to FreeBSD users: Yes, CBSD makes use of the _freebsd-update_ tool which is just the right one for the job. But let's start the jail once more so we can check real quick if the update was actually done:
# cbsd jstart demojail
# cbsd jexec jname=demojail freebsd-version
13.2-RELEASE-p10
Yes, that's more like it! It did just what it was supposed to do.
If you are using CBSD on a system using ZFS, the tool will happily manage snapshots for you as well. So before updating a production jail like we just did before, it probably makes sense to create a snapshot. Or maybe you plan on messing with software configuration in the jail and don't know if things will work out. Create a snapshot and you'll be safe.
Make CBSD take a snapshot like this:
# cbsd jsnapshot jname=demojail snapname=13_2p10 mode=create
jsnapshot: created snapshot #1 for demojail in 0 seconds
Doing snapshots on live jails is not much of a problem in general, but various daemons may not be exactly happy if you need to roll back later. Especially when your jail runs a database I would strongly suggest to either stop the db service or better just stop the jail completely. Otherwise consistency of the db is not guaranteed even though the rest of the system may be fine.
If you want to see if there are any snapshots for this particular jail, you can do this:
# cbsd jsnapshot jname=demojail mode=list
JNAME SNAPNAME CREATION REFER
demojail 13_2p10 2024-02-29__19:26 70.4M
To see all snapshots that are managed by CBSD (which uses a special property so it can tell its snapshots apart from others that may be used on the system), simply don't specify a jname:
# cbsd jsnapshot mode=list
Since there's only one jail on this test system, the output would of course be identical.
In case something actually broke and you would like to return the jail to the previous state, use _rollback_ mode:
# cbsd jsnapshot jname=demojail snapname=13_2p10 mode=rollback
Restored state to 13_2p10 snapshot created in Thu Feb 29 19:26 2024
If however things went well, you moved on and want to clean up, you can remove ("destroy" in ZFS terminology) snapshots like this:
# cbsd jsnapshot jname=demojail snapname=13_2p10 mode=destroy
jsnapshot: destroyed 1 snapshots in 0 seconds
Should you have accumulated several snapshots and don't feel like destroying them one by one, you can blow all of them away for one jail at a time instead by not specifying a snapname and using _destroyall_ mode:
# cbsd jsnapshot jname=demojail mode=destroyall
Of course you could do all of this manually using the _zfs_ command. But CBSD not only makes it pretty convenient. It also makes it less error-prone since you don't have to make sure you snapshot the right dataset.
One last thing that may or may not appeal to you is the dialog-based control menu. Bring it up either by selecting from the list or by specifying jname like this:
# cbsd jcontrol-tui jname=demojail
This will bring up the control menu for the jail allowing you to start or stop the jail, to go to the configuration menu or to destroy the jail.
CBSD jcontrol-tui dialog menu (PNG)
And that's it for this article – we're done with all the basic jail operations. If you followed the series this far you should be able to manage simple jails with CBSD in a variety of ways (and find your own style that fits your workflow).
My thanks once again to Oleg, CBSD's maintainer! Not just for the tool but also for proof-reading and giving valuable input that helped make these articles significantly better and probably more useful.
There's more to be covered in terms of jails, but I'd like to shift gears a little. The next article will be about CBSD's help system, backups and version upgrades. Then we will look at using CBSD to manage Bhyve for full-blown VMs.
(March 2024) Exploring the CBSD virtual environment management framework - part 7: Updating and help
