Sunday, 30. April 2023
[This article has been bi-posted to Gemini and the Web]
This article is about using the interactive jail creation script and building something from ports. It also explores some important concepts along the way.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts. Part 1 covered an introduction of CBSD as well as the installation. Part 2 detailed CBSD's initial setup process. In part 3, jail creation with a dialog-based TUI as well as starting and stopping jails was covered.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup
(February 2023) Exploring the CBSD virtual environment management framework - part 3: Jails (I)
Due to various problems in real life this article has been delayed. I redid and finished it on a test system that now runs _FreeBSD 13.2_ and CBSD _13.2.0_. As usual, the workdir is _/cbsd_.
We've created a pretty standard jail last time. So what about doing something less common for a change? Let's say we want to know if a specific port still builds in an older FreeBSD release. This is a pretty special use-case but it makes a nice example that allows me to deliberately hit some problems and explain what is happening there.
We are going to find out whether the current version of the package manager still builds on FreeBSD 11.4. This version of the operating system is no longer supported and has been removed from the main download mirrors. It's still available from the _ftp archive_ but CBSD doesn't use that. So in order to create an 11.4 jail, we need to manually fetch the base system tarball first:
# mkdir -p /tmp/11.4i386
# fetch -o /tmp/11.4i386/base.txz http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/amd64/11.4-RELEASE/base.txz
Now that we have the old base distribution set we can continue. There's more preparation work to do, but we won't do it up-front. Even though it means flipping back and forth between the host and the jail, it makes more sense to do things linearly.
CBSD offers multiple ways of creating jails. We explored one (dialog menus) in the previous article. Lets take a look at using the interactive creation script next:
# cbsd jconstruct
---------[CBSD v.13.2.0]---------
Welcome to jcreate config constructor script.
For DIALOG-based menu please use jconstruct-tui utility
-----------------------
Proceed to construct? [yes or no]
The script hints the user that 'jconstruct-tui' is available which is very useful for people new to CBSD. We know however and want to use the script instead.
y
Jail name. Name must begin with a letter / a-z / and not have any special symbols: -,.=% e.g: jail1
This is exactly what it says: Specify the internal name for the jail here, i.e. what you like to call it. This is not the hostname. Definitely stick with something simple and avoid special characters as told to:
fbsd114
Jail Fully Qualified Domain Name e.g: fbsd114.my.domain
Next is the actual host name that should be used for the jail:
114jail.advancebsd.net
Jail IPv4 and/or IPv6 address e.g: 10.0.0.1
While it is possible to have jails without networking capabilities this is pretty unusual. The script assumes that you want to give the jail an IP address. Let's do that:
10.0.0.1
Jail base source version e.g: native
Now CBSD wants to know what version of the FreeBSD userland should run inside the jail. Use _native_ (the default) to make CBSD prepare a jail with the same version as the OS on the host system. You can use older versions than what the host system currently runs but of course not newer ones. As per the example use case we will create an 11.4 jail instead of a 13.2 one:
11.4-RELEASE
Target architecture, i386/amd64 or qemu-users arch e.g: native
On an amd64 system you can create jails with a 32-bit version of FreeBSD if you wish. CBSD will also support running jails of foreign platforms by using Qemu's emulation. This needs to be set up beforehand and is an advanced topic which we will disregard for now. Let's go with an i386 jail just to show that this actually works:
i386
1,yes - Jail have personal copy of base system with write access, no NULLFS mount. 0,no - read-only and NULLFS ( 0 - nullfs base mount in RO, 1 - no nullfs, RW ) e.g: 0
You can have jails with a writable base system just like the OS on your host machine. If you want to tighten security further (and this setting does not defeat your particular use-case) you can have CBSD make the base system read-only. Let's do this in our example:
0
1,yes - Jail have shared /usr/src tree in read-only (0 - no, 1 - yes): e.g: 0
Do you need the the system source code to be writable? If you don't, you can make it read-only, too. We'll do that here:
1
1,yes - Jail have /usr/ports /usr/ports tree in read-only ( 0 - no, 1 - yes) e.g: 1
Normally for ports development it is not such a good idea to have an immutable ports tree. But let's do that anyway for this example:
1
1,yes - Automatically start Jail when system boot ( 0 - no, 1 -yes) e.g: 1
You can set the jail to automatically start after the system booted up. For our purposes a manually maintained jail will do:
0
1.yes - Enable VIMAGE/VNet feature ( 0 - no, 1 -yes) e.g: 0
The script wants to know whether you'd like to create a VNET jail or a traditional one. VNET jails have their own virtual network stack which lets you do interesting things like running a firewall inside jails. This will be a topic to cover in depth in a future article.
0
Auto create and auto remove IP on selected NICs. 0 for disable, auto - for auto detect e.g: auto
CBSD can automatically assign the jail IP when the jail starts and take it away when it shuts down. Unless you have special needs, this is usually a good idea:
auto
1,yes - Apply CBSD templates for Jail environment ( 0 - no, 1 - yes) e.g: 1
Templates are a kind of default pieces of configuration for various system components. This involves dotfiles for the root user which configure the c-shell and bash (the latter of course only has any effect if you install it later) as well as several files under /etc. One example for the latter is an rc.conf file that by default disables sendmail. This mechanism is similar to the system's _skel_ directory that is typically used to initially populate a new user's home directory. The difference is that it's a system-level thing. We're going to make use of it (and will pick up that topic again later):
1
1,yes - Floating /etc/resolv.conf content ( 0 - no, 1 - yes) e.g: 1
Do you want CBSD to manage the jail's resolver configuration for you? Likely yes, but in case you have special needs, you can disable this. We're going with the default here:
1
And that's it. Now CBSD asks you whether you want to create the jail immediately which we do. It doesn't have a base system for 32-bit FreeBSD 11.4 in place, yet, so it needs to acquire one first:
no base dir in: /cbsd/basejail/base_i386_i386_11.4-RELEASE
Select base sources:
0 .. CANCEL
a .. build
b .. extract
c .. repo
Last time we used option 'c' which is generally the preferred one. It won't work for obsolete releases, though. So we will have to use one of the other two. Option 'a' is not terribly complicated but does take a long time (the OS is built from source!) and uses a lot of system resources. We've already fetched the required base distribution set and thus can use 'b':
Scan for config-based path to base archive: /mnt/usr/freebsd-dist/base.txz
info: no such archive file: /mnt/usr/freebsd-dist/base.txz
Please provide full path to base archive, (e.g. default: /usr/freebsd-dist/base.txz):
The default path (e.g. mounted FreeBSD ISO) is not available on this system, but we want to use our custom distset file, anyway:
/tmp/11.4i386/base.txz
[... lots of output from verbose archive extraction ...]
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: Not Found
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: Not Found
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
CBSD tries to bootstrap the pkg(8) package manager - but it's no longer available for our old release, so this is an expected failure.
Bases registered: /cbsd/basejail/base_i386_i386_11.4-RELEASE
register_base: auto_baseupdate=0 via FreeBSD-bases.conf, updates disabled
register_base: you might want to do cbsd baseupdate by hand to fetch latest patches
Alright, CBSD has successfully created and registered a base system for FreeBSD 11.4-RELEASE/i386. It gives you a hint about how to update the base system, but since we're working with a throw-away jail using a no longer supported release, anyway, we do not need to care at this point.
Please wait: this will take a while...
Applying skel dir template from: /cbsd/share/FreeBSD-jail-skel
To edit VM properties use: cbsd jconfig jname=fbsd114
To start VM use: cbsd jstart fbsd114
To stop VM use: cbsd jstop fbsd114
To remove VM use: cbsd jremove fbsd114
For attach VM console use: cbsd jlogin fbsd114
Creating fbsd114 complete: Enjoy!
etcupdate error: no such /cbsd/src/src_11.4-RELEASE/etcupdate/current
etcupdate: please run for init: cbsd etcupdate mode=extract ver=11.4-RELEASE
jcreate done in 2 minutes and 7 seconds
Now the template has been applied, too and the jail is ready. Again CBSD gives a hint about updating which we will ignore once more.
As you can see, the interactive script is less powerful than the much more comprehensive dialog menu. It only provides the basic options for creating jails. Whenever this is sufficient for your use-case, it allows for a quicker way to create jails, though.
As usual, let's first see the jails list:
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
fbsd114 0 10.0.0.1 114jail.advancebsd.net /cbsd/jails/fbsd114 Off
There it is. Since it's not running, yet, we need to turn it on. In the previous article we used the list option to select which jail to start. That is a nice way of doing things in some cases (for example due to the extra info it provides e.g. about how long a jail has been stopped). But especially when you have a lot of jails running, navigating through the list can be cumbersome. This is why CBSD lets you pick a jail without using a menu by just passing an additional argument:
# cbsd jstart jname=fbsd114
Default NIC automatically selected: ue0
set resource limit: [ ]
jail renice: 1
Starting jail: fbsd114, parallel timeout=5
fbsd114: created
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
Generating host.conf.
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Updating motd:.
Starting cron.
Sat Apr 22 18:35:18 CEST 2023
jstart done in 3 seconds
Setting the _jname_ option will let CBSD know which jail(s) to act upon. Yes, plural form: You can actually use the asterisk (*) wildcard to match multiple jails. To do this you must place the name in single single quotes to prevent the shell from expanding it. We'll do this in a minute.
OK, the jail has been started. For some CBSD subcommands, the _jname_ option is implicit if you just provide a positional first argument. So if we want to log into the new jail, you can leave out the optional 'jname=' to do less typing and achieve the same goal (for starting and stopping it's optional, too):
# cbsd jlogin fbsd114
FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC
There we are! Again, like last time, the last line printed is misleading. This message is from the host system's kernel. We can check just for good measure:
# freebsd-version
11.4-RELEASE
Alright. Now let's build some port, shall we? First we need to pick one of course:
# ls /usr/ports
distfiles packages
Whoops! This is not going to work, the ports tree is missing! How is that? The explanation is pretty simple: CBSD assumes that most users who use ports will want to use a shared tree as it is both wasteful to have it in every jail and cumbersome to keep up to date in multiple installations. Therefore it defaults to mounting the _host system's_ ports tree into the jail! Let's leave the jail now and get the ports tree in place on the outside system:
# logout
# portsnap auto
Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.
Fetching public key from dualstack.aws.portsnap.freebsd.org... done.
Fetching snapshot tag from dualstack.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Sat Apr 22 02:23:08 CEST 2023:
bceb16b9caa698bf0cbc89be7cd78a63c0d474ce28d538 102 MB 1315 kBps 01m20s
Extracting snapshot... done.
Verifying snapshot integrity...
Fetching snapshot tag from dualstack.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Sat Apr 22 02:23:08 CEST 2023 to Sat Apr 22 18:05:00 CEST 2023.
Fetching 5 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
[ ... ]
Building new INDEX files... done.
With the ports tree extracted into /usr/ports on the host system we are now ready to log into the jail again and can have a look in ports again:
# cbsd jlogin fbsd114
# ls /usr/ports/ports-mgmt/pkg
Makefile distinfo files pkg-descr pkg-plist
There we are, much better! This works because CBSD does a _nullmount_ (i.e. mounting a different part of the filesystem on a directory) of _/usr/ports_ on the host to _/usr/ports_ in the jail. Now we should be good to build something from ports, right? Let's check the network connectivity real quick to rule out another possible source of trouble:
# host bsdnow.tv
;; connection timed out; no servers could be reached
This jail has an address from the private IPv4 range and the host system has not been configured to do NAT! Of course there's no network access. We could certainly solve this, but for the example use-case we can also work around it.
The best candidate to start with is the package manager of course. Let's exit the jail again, fetch the distfile on the host machine and log back in:
# logout
# make -C /usr/ports/ports-mgmt/pkg fetch
[ ... ]
# cbsd jlogin fbsd114
Now let's try building pkg(8):
# make -C /usr/ports/ports-mgmt/pkg
===> Building/installing dialog4ports as it is required for the config dialog
===> Cleaning for dialog4ports-0.1.6_1
/!\ ERROR: /!\
Ports Collection support for your FreeBSD version has ended, and no ports are
guaranteed to build on this system. Please upgrade to a supported release.
No support will be provided if you silence this message by defining
ALLOW_UNSUPPORTED_SYSTEM.
*** Error code 1
Right, we're trying to build from an up-to-date ports tree on a no longer supported release. So the ports tree is telling us how we can proceed anyway (of course accepting that there's no guarantees whatsoever):
# echo ALLOW_UNSUPPORTED_SYSTEM=1 >> /etc/make.conf
# make -C /usr/ports/ports-mgmt/pkg
[ ... ]
=> freebsd-pkg-1.19.1_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://codeload.github.com/freebsd/pkg/tar.gz/1.19.1?dummy=/freebsd-pkg-1.19.1_GH0.tar.gz
fetch: https://codeload.github.com/freebsd/pkg/tar.gz/1.19.1?dummy=/freebsd-pkg-1.19.1_GH0.tar.gz: No address record
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/freebsd-pkg-1.19.1_GH0.tar.gz
fetch: http://distcache.FreeBSD.org/ports-distfiles/freebsd-pkg-1.19.1_GH0.tar.gz: No address record
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles/ and try again.
*** Error code 1
[ ... ]
# ls /usr/ports/distfiles
But what's this? It's not much of a surprise that the distfile cannot be fetched in a jail without network connectivity. But since the host's ports tree is there and we previously fetched it, why is it missing? Time to investigate:
# mount
/cbsd/basejail/base_i386_i386_11.4-RELEASE on / (nullfs, local, noatime, read-only, nfsv4acls)
/usr/ports on /usr/ports (nullfs, local, noatime, nosuid, read-only, nfsv4acls)
/cbsd/jails-data/fbsd114-data/var/cache/distfiles on /usr/ports/distfiles (nullfs, local, noatime, nfsv4acls)
[ ... ]
Aha! The /usr/ports/distfiles directory gets shadowed (inaccessible and invisible) due to another mount on top of it. Let's check that theory by escaping to the host, removing the mount and looking again from inside the jail:
# logout
# umount /cbsd/jails/fbsd114/usr/ports/distfiles
# cbsd jlogin fbsd114
# ls /usr/ports/distfiles/
freebsd-pkg-1.19.1_GH0.tar.gz
Just as expected, now the distfile is there. We'll get back to why CBSD does this in the first place in a minute. But before doing so we can now try building that port once again:
# make -C /usr/ports/ports-mgmt/pkg
[ ... ]
make[3]: "/tmp/usr/ports/ports-mgmt/pkg/work/pkg-1.19.1/mk/common.mk" line 16: Need an operator
make[3]: Fatal errors encountered -- cannot continue
make[3]: stopped in /tmp/usr/ports/ports-mgmt/pkg/work/pkg-1.19.1
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1
[ ... ]
Seems like pkg version 1.19.1 can in fact not be built on FreeBSD 11.4. The ports framework (which consists of the *.mk make fragments) is incompatible. So if we hadn't known already now we do. Objective completed.
As we have seen above, the various mounts that CBSD applied are a little... atypical. The reason for it is pretty simple but may not be completely obvious: The ports tree was mounted _read-only_! Building any port like normal would not work because even if the distfile could have been fetched, it could not have been written to disk. This is why CBSD mounts another location (which is writable) over the distfiles directory.
But would that suffice? Any ports are built inside the tree after all, right? Yes - by default. And this is where CBSD changes the default to work around this problem. Here's the file that we modified to allow the ports tree to build anything on an unsupported OS release:
# cat /etc/make.conf
WRKDIRPREFIX=/tmp
ALLOW_UNSUPPORTED_SYSTEM=1
As you can see, there's another definition in there, relocating the prefix for the ports working directory to a writable location:
# ls -1 /tmp/usr/ports/ports-mgmt/pkg/work/pkg-1.19.1/
.cirrus.yml
.gitignore
.gitlab-ci.yml
AUTHORS
CONTRIBUTING.md
[ ... ]
And there it is. But where did that other option come from? We owe it to using CBSD's templates. Let's take a look on the host system again:
# logout
# cat /cbsd/share/FreeBSD-jail-skel/etc/make.conf
WRKDIRPREFIX=/tmp
What else can the templates do? Quite a bit. It's worth taking a look around, especially at the important files like /etc/rc.conf. You may find that this is the place to put your preferred defaults so that they can conveniently get added to any new jail you create. They are not limited to the _etc_ directory, though:
# ls /cbsd/share/FreeBSD-jail-skel/root/
.bashrc .cshrc
You probably wondered where the colorful extended prompt that CBSD uses by default comes from. This is where. But wait a moment - wasn't the base system meant to be read-only as well? Since we interfered with the default mounts by doing our manual umount, let's quickly stop and start the jail again. As there's only one jail on the test system we can save some more characters by using the wildcard for it:
# logout
# cbsd jstop '*'
Stoping jail: fbsd114, parallel timeout=5
jstop done in 1 seconds
# cbsd jstart '*'
[ ... ]
# cbsd jlogin fbsd114
[ ... ]
Now let's take a look again:
# mount
/cbsd/basejail/base_i386_i386_11.4-RELEASE on / (nullfs, local, noatime, read-only, nfsv4acls)
/usr/ports on /usr/ports (nullfs, local, noatime, nosuid, read-only, nfsv4acls)
/cbsd/jails-data/fbsd114-data/var/cache/distfiles on /usr/ports/distfiles (nullfs, local, noatime, nfsv4acls)
/cbsd/tmp/usr/ports/packages on /usr/ports/packages (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/etc on /etc (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/root on /root (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/tmp on /tmp (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/usr/home on /usr/home (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/usr/local on /usr/local (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/compat on /compat (nullfs, local, noatime, nfsv4acls)
/cbsd/jails-data/fbsd114-data/var on /var (nullfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)
fdescfs on /dev/fd (fdescfs)
Here in the full output you can see that there are 8 directories (including the ports distfiles one) which get mounted on top of the basejail contents and are thus writable. Which means that for example the home directory is writable and /usr/local is writable, too, allowing for the installation of packages. The actual binaries, libraries and such however are not.
And that's already it for today. I hope that it has been interesting follow and helped you get more familiar with the CBSD framework.
In the next article we will take a look at even more ways to create jails and manage them.
(May 2023) Exploring the CBSD virtual environment management framework - part 5: Jails (III)