Here I'm republishing an old blog post of mine originally from April 2017. The article has been slightly improved.
Ok, after introducing FreeBSD's jails and jail frameworks in the previous post, it's time to actually put the 4.11 system that was built in February behind bars. Before turning towards that special case, though, we'll have a look at how to use _iocage_ (one of the jail management frameworks).
FreeBSD jails (1/2): Introduction and frameworks
We've built iocage from ports last time, so let's just jump right into it and try out one of the most important commands:
# iocage list
Not too surprisingly, the jail list is empty. But the important thing here is that iocage automatically created its infrastructure when run for the first time. This is generally fine; if you happen to have more than one ZFS pool, though, you may want to manually _activate_ one for iocage usage.
The next thing to do is to fetch the distribution tarballs for the FreeBSD version that shall populate the jails you want to create:
# iocage fetch -r 11.0-RELEASE
Fetching the 11.0 release (PNG)
This process will also update the kernel and userland to the latest available version of the selected release. I chose 11.0 because that's the latest right now and obviously 4.11 is not available!
You can actually skip fetching the release as iocage will do it automatically if you create the jail, but doing it beforehand doesn't hurt. Now let's create our first jail:
# iocage create tag=fbsd11 ip4_addr="em0|192.168.2.10/24" -r 11.0-RELEASE
Each jail has a UUID. This one's UUID begins with _99a9941a_. You can interact with jails by using that UUID. But if you've got a lot of jails it would be pretty hard to keep track of which one is which. For that reason you can assign a _tag_ to a jail that you can use to identify it. We've effectively given our jail a name: _fbsd11_. In general it's a good idea to name it in a way that helps you (or your co-workers) to understand what the jail is used for. You may choose something like "postgres-jail" that includes the technology used, "loadbalancer" to describe its function or whatever fits your needs.
We've also assigned the jail an IPv4 address, 192.168.2.10 in this case, noted the subnet in CIDR notation and specified which NIC to use it on. Listing jails shows all that information at a glance.
The jail is ready to be started:
# iocage start fbsd11
As you can see here from the _ifconfig_ output, the ip address was not yet assigned to the NIC. According to iocage the jail has been started successfully. Let's take a look at it:
# iocage list
The state for our jail has changed to "up". Also the ip has been automatically assigned to our network interface when iocage started the jail. When the jail is stopped, it is also automatically removed again.
Iocage offers some means to interact jails without actually "entering" them, like e.g. installing packages, etc. The easiest way to use a jail however, is to get a shell prompt inside the jail (think entering a chroot environment). That's what we're going to do next:
# iocage console fbsd11
Entering the jail console (PNG)
Looks pretty much like we just logged into a FreeBSD system, huh? Well, we basically did just that. With the big difference of not being a full OS (which would be the case with a VM for example) but a jail with just a FreeBSD 11.0 userland (that is running on the host machine's kernel).
Ok, next I bootstrap pkg and then install nginx:
Installing the nginx webserver (PNG)
This also proves that the network is working since we're receiving the packages to install from. There's also nothing special in doing this, just use pkg the way you're used to. For the most part it's behaving just like a regular FreeBSD system. There are a few exceptions. One of the most notable ones is the fact that by default you won't be able to use the _ping_ command. You can enable jails to use ping using a sysctl but this affects the jails security and therefore is disabled by default.
Let's start nginx next:
Again it's nothing special. Nginx is running and bound to port 80. Does it do what you'd expect it to?
Yes, the webserver is happily serving the default page when accessed from other computers on the LAN! It doesn't make much of a difference from running nginx on the host system - unless somebody manages to use an exploit. Nginx is privilege separated and thus probably not a good example. But think about a case where an intruder manages to get into your system and get root privileges. That's pretty bad, yes. But if the application was running in a jail, the damage is at least contained and the other jails as well as the host system are unaffected. (That and it's much easier to tear that jail down and create a new one compared to having to reinstall the whole host system and setup several applications again!)
Alright. Let's take a quick look at where the jail data is kept:
Taking a look at iocage's directory structure (PNG)
Iocage created the _/iocage_ dataset. The most important path is _/iocage/jail_. All the jails are in there (named after their UUID). To make accessing them easier, there's _/iocage/tags_: Here you'll find symlinks named after the jail's tags which are pointing to the actual directories with the data.
Each jail consists of a configuration file, an fstab and a "root" directory (which holds all the files).
This jail has fulfilled its purpose and is no longer needed. So let's get rid of it! First exit the jail like you would exit a chroot (e.g. _CTRL-D_ out of it) before you proceed:
# iocage stop fbsd11
# iocage destroy fbsd11
Stopping and destroying the jail (PNG)
You cannot destroy a running jail. Therefore it must be stopped first and can then be destroyed.
That has been pretty easy and straight forward, right? And here's the good news: Jailing the 4.11 system isn't that difficult, either. I created a tarball of the complete filesystem on the FreeBSD 4.11 server, xz compressed it and scp'd that onto the jailhost. Now we'll create another 11.0 jail, delete the filesystem and extract the old OS data into the same place:
# iocage create tag=fbsd4.11 ip4_addr="em0|192.168.2.10/24" -r 11.0-RELEASE
# cd /iocage/tags/fbsd4.11/root
# rm -rf * .*
# tar xvJf /path/to/tarball.xz
Creating a new jail and putting the 4.11 OS into it
The old device nodes won't work on a current kernel and have to be removed. The same goes for the old _proc_ filesystem. We also don't need any kernels or kernel modules. And _/boot_ is obsolete in a jail, too. So it's a good idea to get rid of all that:
# rm dev/*
# rm kernel*
# rm -rf boot
# rm -rf modules*
# rm -rf proc/*
Deleting unneeded parts of FreeBSD 4.11 (PNG)
The old userland and the additional data remains. Let's try to start the jail now:
# iocage start fbsd4.11
Uhh... This doesn't look good. Iocage tries to do things that won't work with a 4.11 userland (anything older than 9.3 is unsupported with iocage, anyways!). But fortunately that looks worse than it actually is:
Iocage spits out errors... but the jail is running! (PNG)
Despite those errors the jail is up and running! Can we also get console access?
# iocage console fbsd4.11
Entering the console works, too! (PNG)
Yes, we can! (Let's make 4.11 great again!) Time to install a webserver in this jail, too. Since we still have Pkgsrc in place, that's an easy thing to do:
# cd /usr/pkgsrc/16/www/nginx
# bmake install clean clean-depends
Building nginx on FreeBSD 4.11 via Pkgsrc (PNG)
I decided to modify nginx's default starting page so it looks a little different from the stock one. Nginx can be started simply by executing the daemon. However all the system utilities regarding process management don't work! And sockstat is not working either.
Starting nginx manually, but tools like ps don't work! (PNG)
Does nginx actually run? There's an easy way to test that. Let's just open up a browser on another PC and see if we can get the modified webpage.
Nginx is able to serve the webpage from the 4.11 jail! (PNG)
It works! Yes, it was actually that easy. Jails are not necessarily a complicated thing at all (they can be more complicated than that of course, if you're doing more advanced stuff). But... No _ps_, no _top_, no _kill_... How are you supposed to work with a 4.11 jail?
Peter Wemm suggested using statically compiled binaries from the host system like those in /rescue. And that's actually a great idea! If you copy _/rescue_ over into the jail, it at least provides some tools to work with.
Thanks to /rescue process management is possible (PNG)
Things like top would be nice, too, of course. But to be honest, I don't know how to statically build them. If anybody knows, please comment and share your wisdom! [Somebody did! I'll put the comment at the end of the article.]
This article merely gave a very basic introduction to the topic (plus a little "stunt"). You can do a lot more with jails. I might write about it again in the future but I haven't decided, yet (feel free to comment and tell me what you're interested in!). But I wanted to at least give you an idea what other things there are to explore.
If you have a server with only one ip you can still use jails. The solution to this is to give the jails internal network addresses (like 192.168.x.x) and then use e.g. the _pf_ firewll to do network address translation (NAT). It took me a bit to figure out how this works (there are things on the net that are better documented) but in the end it's not that complicated.
With VIMAGE/VNET you can give your jails their own virtual network stack. That way you can even use a firewall inside the jail and assign multiple ip addresses!
Jails support quotas of various kinds. You can limit the amount of CPU that one jail may occupy, limit file system IO, etc. Lots of possibilities with this.
And of course iocage has many nice features. You can e.g. create jail templates with some software already installed and quickly create jails from those. There are multiple jail types like e.g. thin-provisioned jails. You can clone and snapshot jail. And you can have jails started automatically when your server boots.
In the previous article I said "think containers" when you're completely new to jails. Now you've seen a bit of what they can do and have a better idea of what they are. So stop "thinking containers" now. These are two things that are somewhat related but they are not the same thing. Linux's containers can do things that jails were never designed for (e.g. sharing namespaces). But jails have one huge advantage over containers: While containers are only a means for logical separation, jails were designed for security right from the start.
Sure, people are working on making Linux's containers more secure. But if there's one thing that we should have learned by now, it is this: Retrofitting security into something is not really going to work... And jails are pretty flexible, too. You might not have missed them when you didn't know that they exist. But get used to using them once and you'll think differently about a lot of things and make use of them for more and more things. Jails are a great feature of FreeBSD.
On host system in /usr/src/rescue/rescue/Makefile search line 'CRUNCH_PROGS_usr.bin+=' (196 in FreeBSD 11.1) and add top:
CRUNCH_PROGS_usr.bin= head mt sed tail tee
CRUNCH_PROGS_usr.bin+= gzip
CRUNCH_PROGS_usr.bin+= top systat
Read /usr/src/rescue/README:
cd /usr/src/rescue && make obj && make && make install
Copy /rescue/top and /rescue/systat to /usr/bin/ into jail.
It works for me.
