Wednesday, 08. February 2023
[This article has been bi-posted to Gemini and the Web]
This article is about using the dialog menus to create a jail, start it and enable SSH in there so the jail does something.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts. Part 1 covered an introduction of CBSD as well as the installation. Part 2 detailed CBSD's initial setup process.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup
Alright, for part 3 we are finally able to put CBSD to good use. Again, a FreeBSD 13.1 test system is used, CBSD is version _1.13.21_ and _/cbsd_ was chosen as the workdir.
After two entire articles without managing anything with a management framework... *drum roll* ...we're going to create our first jail using CBSD. Yes, finally! There's at least two more topics that I'd like to discuss before getting into jail management, but I also don't want to delay things further. Just so that you've heard it already, CBSD ties into bsdconfig(8) as a submodule. We're going to explore this a little later. It's good to know, though, in case you are in need for an easy way to change the configuration you made during the initialization.
In CBSD there are often multiple ways to do something. There certainly are several when it comes to creating jails. We'll start with what is the easiest method and the one that I would recommend to newcomers: Using a dialog-style menu. CBSD provides a nice, curses-based Text UI, building a menu for you to navigate. Since this article series assumes that you're just getting started with the tool, it is the perfect choice for now. It allows you to get an idea of what the options are that you can or should tweak for your jails. Later you may find that other means of achieving the same goal are more useful as they are way quicker to operate than navigating a menu.
CBSD jconstruct TUI version (PNG)
But let's get to it already! Run this command:
# cbsd jconstruct-tui
Nice menu, eh? You can absolutely give this to people who are not FreeBSD admins (I've done that and it worked perfectly well!). Most of the options here are self-explanatory, but we'll just go over everything quickly (at least in this main layer).
_Profiles_ are in fact exactly what you probably thought: Presets of options for various types of jails. You can completely customize any of them, but presets allow you to quickly select what is closest to what you want to do. For the beginning you may want to stick with the plain and boring _default_ jails. The _vnet_ profile is for you if you want to create jails that have their own network stack and e.g. allow for running their own firewall instance. If you want to try out something that you know needs options set that (for a good reason!) are off by default, give _trusted_ a spin. For example if you need to allow for raw sockets because you want to use ping(8) in a jail. I hope you know what you are doing, though. And in case you really need a _Linux jail_ (you have my sympathy!), those profiles may be a useful shortcut.
Choose _pkglist_ if you want CBSD to install packages inside your new jail for you. If you do, it will start indexing and preparing the available packages of every category for displaying via a menu.
Selecting packages from the 'archiver' category (PNG)
I do not really use this feature because it's usually easier to just use pkg(8) inside the jail (or _pkg -j_ on the host). I would definitely consider the menu if I were creating a template for various jails, though.
Next choose a name for your jail. This is a reference for internal use only, so unless you like typing long names, select something short (but ideally meaningful enough to you to know what that jail does). CBSD suggests 'jail1' which is fair enough as an example but a pretty bad name otherwise. Find a scheme that works for you and use that.
Whatever you go with in the end, __do__ mind what CBSD tells you here! Stick with letters and numbers. __Do not__ try anything fancy. You may find that you seem to get away with disregarding CBSD's warning. And indeed: You can do that. Things may even actually work for a while and you start laughing at stupid, conservative CBSD. But then _things break_. I've had to deal with the extremely weird effects of this after a customer forgot about the restrictions one time and put another character in there. It wasn't pretty. Nothing valuable was damaged, but we lucked out there and coming clean again was some work, too. So do your future self a favor: Stick with letters and numbers, period. And if you insist on living dangerously: At the very least avoid any character that may cause trouble in regard to shell variables.
Setting the jail host name (PNG)
Now it's time to choose the host name for the jail. CBSD will propose something based on the jail name, but with an invalid TLD. So you're clearly meant to fill in something sensible. Free to enter whatever you want to use as the hostname (as long as you own the domain, of course!).
Defining the IP address to use (PNG)
You need to set up your jail for networking. While the variable is called 'ip4_addr', this is for historic reasons only. Entering a v6 IP address here is perfectly fine and supported.
In rare cases when the jail does not need any network connection whatsoever, set this to '0'. Mind the difference between 'DHCP' and 'REALDHCP' here! The latter is what you probably suspected the former to be. What 'DHCP' means in this context is this: Let CBSD pick a free IP address from the pool that you assigned it to use (during the initial setup, you remember?).
Set this as your use case requires. E.g. "DHCP,DHCPv6" would assign an IP from the IPv4 pool of addresses as well as from nodeip6pool if it was configured. Manually defining it like "10.0.0.2/24,2a05:3580:d811:800::2/64" works, too. We'll come back to this when covering VNET jails.
Selecting the jail userland version (PNG)
Most of the time you probably want the userland in your new jail to match that of the host machine. In this case just leave it at CBSD's suggestion of 'native'. You can use older versions if you have a specific use case. I've been using this to quickly test a port that I maintain on all supported FreeBSD releases. You cannot run a jail with a newer userland than the kernel running on the host machine.
What you can do is run ancient FreeBSD versions in jails - but this is not a good idea in all but very special cases. If you think about doing that for anything but fun, do your fair share of research beforehand. Not only will the standard means of deploying a jail with CBSD not work with FreeBSD older than 9.0, but you will be facing very interesting other changes... (Believe me, I've had to save a one man business by taking a crucial internal application - for which the author has long since passed - off the dying hard disk of the FreeBSD 4.11 machine and keep it running on modern hardware...).
Then you need to decide whether you want to have your jail base system writable or read-only. The default setting for _baserw_ is not allowing for writing. Also you should decide on autostarting the jail when the host system booted up or not. The default for _astart_ is yes.
Network interface selection (PNG)
Select the network interface to use next. For a standard jail you will probably want to simply leave this on 'auto', but it doesn't hurt to define it explicitly, either.
Please note that CBSD also supports VALE(4), netgraph(3) and CBSD VPC. You don't see these options here because they require additional configuration to be done beforehand.
In almost all cases you will want to leave the architecture as 'native' which means: Same as the host machine. On amd64 you _can_ create an i386 jail if you really want to. Why are there other options listed there as 'unsupported' (see screenshot above)? This does not mean that CBSD won't support such a choice as one could think. If you try to pick one, CBSD will tell you why: Those architectures require emulation on the current platform and a new enough installation of Qemu was not detected!
If you satisfy the requirements and come back, CBSD will allow for creating e.g. a riscv jail on amd64. While this is pretty cool, it's also clearly advanced usage. I intend to revisit that later in this series but won't promise anything, yet.
You can enable or disable the VNET feature for your jail next. We've touched what this is in the profiles section, and indeed it makes sense to decide upfront and tune the other options accordingly.
Defining the devfs ruleset to use (PNG)
CBSD wants to know which devfs ruleset to use for the jail. If you don't know what that means, go with the default for now, but make a note to read more about it. As long as you don't understand what's happening there, this will make jails less useful for you. You will hit limitations and probably think something simply cannot be done while it clearly could by using different rules here.
Here's a very short summary: You know that Unix idea of "everything is a file" and that even devices are represented as files under /dev in the filesystem hierarchy. Well, devfs(5) is what is used to populate /dev with the _device nodes_ for devices that would otherwise only be known to the kernel. There's interesting stuff in there. Stuff that you might want to keep inaccessible in your jails, especially when you give somebody else root access to those. Be sure to read devfs.rules(5) and peek into _/etc/defaults/devfs.rules_ to see what that _ruleset 4_ that CBSD suggests, does. For VNET jails you may likely want to use _ruleset 5_. Define additional rulesets in _/etc/devfs.rules_ if you have special needs.
Create jail from snapshot (PNG)
On systems that run on ZFS you can select a snapshot as the source for jail data. This is a quick and convenient way to jail something that you have in a dataset on the host machine. If you want the jail data do be encrypted using ZFS encryption, you can opt to do that.
Next is a list of currently 33 jail options. We're not going to discuss those here. For almost all of them there's a description at the bottom line as you move through the menu. Those that don't have one bear names which make quite clear what they do, with one exception: 'allow_sysvipc'. If you enable this option, the jail will allow for SysV-style inter-process communication (per jail). You need this for some applications to work (the most popular example may be PostgreSQL).
What follows are some convenience options: You can set the root password in the jail, add a user, configure services and decide whether to do the pkg(8) bootstrap or not.
When you're happy with your choices, select _PROCEED!_.
Create jail immediately? (PNG)
Now CBSD is asking if it should actually create the jail now. You may be seriously tempted to answer: "Yes, please! I haven't navigated through all of those menus and selected to proceed instead of cancelling for nothing, you know!". But there's a good reason that CBSD makes this last prompt. If you choose NO here, it will just write a jail config file for you which you can edit and e.g. create multiple jails from. So you can use _jconstruct-tui_ both for creating a jail and for composing a template. But of course we're going to let it create the jail now.
How to acquire the base system? (PNG)
CBSD would really like to create the jail as asked for, but it can't. Why? I've requested a FreeBSD 13.1 jail (native version on a 13.1 host), yet it has no clean 13.1 userland available to use. And since it cannot create a jail out of the thin air (well, it _could_ but that jail would not be very useful...), it needs to get the base system from somewhere. This is where there are several options and CBSD would like to know which one you prefer:
If you're creating a jail for a version of FreeBSD that is currently supported, the last option is the easiest. If for some reason you need 10.4 for example, your best option is to manually obtain the base.txz archive, copy it to /tmp and use the second one instead. I would only use the first option if I need to customize the base system (playing with src.conf(5) and its WITHOUT_$SOMETHING directives is time-consuming and can lead to dysfunctional builds - but it's really cool nevertheless).
I'm going with the _repo_ option here which is also the default. CBSD does a quick speed test of the known mirrors, then picks the fastest one and downloads the compressed tarball. When that's done, it extracts it and takes various steps to prepare the base system. This only needs to be done once for every release version, of course. So the next 13.1 jail can use the already existing base, but for a 12.4 jail a new base would have to be downloaded and prepared.
All done, jail has been created! But did it really take 5.5 minutes (see screenshot)? Yes, it did, but that's mainly due to my bad internet connection. Setting up jails using already fetched and prepared bases takes far less time (about 30 to 60 seconds or so, depending on your hardware).
Ok, so we've created a jail. What now? CBSD was nice enough to print out a couple of commands we might want to run next. Note here that it says "VM" a couple of times while it actually means _jail_. There are places where the opposite happens and it refers to Virtual Machines as "jails". The reason for this is that CBSD started as just a jails manager and then got more and more support for other virtualization domains. Cleaning up all the cases where those references are incorrect (strictly speaking) and making the output dynamic is a lot of work. It'll probably happen eventually, but for now get used to it and don't let this confuse you.
Listing jails and starting the new one (PNG)
We're going to run some of the suggested commands in a minute. But before we do, let's see what services are listening on the host machine:
# sockstat -4l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root sshd 1578 4 tcp4 *:22 *:*
root syslogd 1477 7 udp4 *:514 *:*
Alright, SSHd and syslogd. Fair enough. Let's list all the jails on this system next. There should be one now after all, right?
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
demo 0 10.0.0.1 demojail.advancebsd.net /cbsd/jails/demo Off
There it is! It's off, though. Time to start it. We could use the command suggested by CBSD, but we've been using the menu system so far. Why not keep doing it for now?
# cbsd jstart
List of offline jail
0 .. CANCEL
a .. demo on local
Okay, should be running now. But it doesn't hurt to check by listing the jails again:
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
demo 1 10.0.0.1 demojail.advancebsd.net /cbsd/jails/demo On
Yes, it's on. Now do this to get into the jail:
# cbsd jlogin
List of online jail
0 .. CANCEL
a .. demo on local
After selecting the 'demo' jail, the prompt changes. We're in the correct jail:
FreeBSD 13.1-RELEASE-p3 GENERIC
demojail:/root@[23:04] #
Looks good, but there's one caveat: FreeBSD prints the _kernel_ version here (you've noticed GENERIC, right?). So this info is about the _host_ system! The jail userland is fresh, non-updated FreeBSD 13.1:
# freebsd-version
13.1-RELEASE
But the prompt is correct, we're in the jail:
# hostname
demojail.advancebsd.net
Enabling SSHd in the jail and checking on the host (PNG)
Let's enable SSHd now:
# sysrc sshd_enable=YES
sshd_enable: NO -> YES
And of course start it:
# service sshd start
Generating RSA host key.
3072 SHA256:nFhZbm6vvaddB0HGLKAUA36LZiYNCtJT38Kn+OfA/18 root@demojail.advancebsd.net (RSA)
Generating ECDSA host key.
256 SHA256:51gPonAdTKmmrC0z1/fGAYdZi/YpkoxVIr6+FnQe1tM root@demojail.advancebsd.net (ECDSA)
Generating ED25519 host key.
256 SHA256:PGoUUapX8HX1bHUzehVA13BOhEpgYmfLP9m3vf2JHbc root@demojail.advancebsd.net (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.
How do you leave a jail? Simply by ending your shell session:
# exit
logout
So we're back on the host system. If we look at the services listening, we should see a difference:
# sockstat -4l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root sshd 10978 3 tcp4 10.0.0.1:22 *:*
root sshd 1563 4 tcp4 *:22 *:*
root syslogd 1474 7 udp4 *:514 *:*
And there we are: There's a second SSHd running and bound only to the jail IP. We could now create a user in the jail and try connecting from outside instead of going via _jlogin_ from the host machine, but the NATting is not in place yet!
This is a good point to end this article, though. We've covered a lot of ground again. It wouldn't be quite complete without at least stopping the jail again first, though. So let's do this final step and then call it a day:
# cbsd jstop
Stoping jail: demo, parallel timeout=5
jstop done in 2 seconds
Checking the jails list again real quick:
cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
demo 0 10.0.0.1 demojail.advancebsd.net /cbsd/jails/demo Off
Yes, jail is down. That means there should again only one SSH daemon be running on the system now, shouldn't it?
# sockstat -4l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root sshd 1563 4 tcp4 *:22 *:*
root syslogd 1474 7 udp4 *:514 *:*
Indeed: The second one that was running inside the jail is gone.
After covering the bare basics, next time we'll have to look a bit more into managing jails.
(April 2023) Exploring the CBSD virtual environment management framework - part 4: Jails (II)
