Here I'm republishing an old blog post of mine originally from April 2016. The article has been slightly improved.
In the previous post I wrote about how to get a computer up and running with a dual-boot of FreeBSD and OpenBSD while using full disk encryption. This worked quite well but a bit later I decided that it would be a good time to do the FreeBSD installation again - this time going the modern way and using ZFS on root. This has led to a surprisingly large amount of problems. In the end I got a working system that uses ZFS, so that I've actually got an alternative howto for FreeBSD (the OpenBSD part remains the same as before and can be looked up in the previous post). But it's all a bit different from what I first thought it would be.
Setting up a FreeBSD/OpenBSD dual-boot with full disk encryption
GPT (GUID Partition Table) is a more modern partitioning scheme meant to replace the old MBR style partitioning, redeeming us from limitations we had to live with in the past. It can deal with drives up to 8 Zettabyte (10²¹ Bytes) instead of being limited to 2 Terabyte (10¹² Bytes). That limitation used to be no problem and it still isn't too bad for most home users since drives bigger than 2 TB are not that cheap, yet. But it obviously won't be too long before this will be a common issue.
OpenBSD bootloader chainloaded by boot0 (PNG)
A more serious limitation is that MBR only supports 4 partitions. Yes, we all used to call these "primary partitions" and made use of "extended partitions" nested inside one of them to get around that limit. BSD users created disklabels instead to embed more partitions inside disk slices ("MBR partitions"). With GPT this is trouble of the past and you are free to create just about as many partitions as you think you need. There's no need for embedding BSD disklabels or doing any MBR trickery - which is nice.
Also GPT supports naming partitions. So by using them we can do away with the old _glabel_ mechanism that FreeBSD offers and use native GPT labeling instead. That advantage comes with a little disadvantage, though. By default FreeBSD's gpart shows such partitions multiple times: Once by their label and once by GPT-id. This can be a bit confusing as you first come to the GPT world. Fortunately there's a simple solution: Setting a sysctl you can simply disable GTP-ids if you opt for labels (which you should since they are much more meaningful).
GPT is a requirement for machines that use EFI booting. The good news is that all recent x86 hardware comes with EFI (and thus GPT) support. The bad news is that while some machines do support GPT when booting from EFI, they don't if you choose to boot from legacy BIOS emulation. So if you want to stick with BIOS you may want to check if it is capable of booting a system on a GPT partition. My EliteBook 8470p is fine with GPT partitions in BIOS mode. So I was good to give GPT a try.
FreeBSD works great with GPT (and has done so for quite some time now) so there's nothing wrong installing it using GPT partitions. The first obstacle that I encountered was the boot manager. FreeBSD comes with the nice and simple _boot0_ tool that I used in my previous howto. Too bad that it's MBR only! So to continue down the path with GPT, it's using another boot manager. Usually GRUB is used for that purpose. And while I would certainly prefer to go without a boot manager that has "grand" in its name, I think that using it is acceptable.
A bigger problem awaits, however. OpenBSD didn't support GPT prior to the current version 5.9. Since it does now, everything should be fine, right? Wrong. If I didn't miss anything, OpenBSD supports GPT only in conjunction with EFI booting! I did not find a way to use GPT partitions with OpenBSD in BIOS mode. Should anybody have more information on this, I'd very much like to know if it either is possible or if support for this is planned for the future.
This means that not too far down the road there's already a solid blocker. I had next to no knowledge about the EFI complex and I successfully avoided that topic in the past. So my two options to go on were to either give up on GPT and simply stick with MBR or to make the bold move and go for EFI. I would prefer to try out things one after another but meh... I decided to read a bit about the basics of EFI and then give it a try.
To be honest, I didn't like what I read too much. Sure, there are quite a few interesting things that EFI can do. But then again I do not believe in unneccessary complexity. And above all: I don't like the security implications of it. Not a bit. Trust is not something that I give away for free. Trust has to be earned. Unfortunately closed source vendors have done very little in the past that makes me think I want to trust them. Anyway... EFI is the (near?) future and it will not be easy to avoid it altogether in the comming years.
Alright. So I turned off legacy BIOS emulation on my machine and booted FreeBSD (there are separate EFI images for FreeBSD 10.3 - make sure you pick one of those if you want to go with EFI!) into the installer. Everything worked smoothly and after a couple of minutes my new FreeBSD system had booted. It was so simple and dull that there's not too much I could write about (which is a good thing since everything really just worked).
Next step: Installing OpenBSD. I read that the memstick image supports EFI booting and I can confirm that installing works just as well as with FreeBSD. Again the new system works like a charm - the OpenBSD people have obviously done a good job for 5.9. OpenBSD was able to cope with GPT just like you would expect.
FreeBSD desktop: EDE, pcmanfm, terminator, smplayer (PNG)
So far everything looked good. Final step: Make the computer offer a means of booting either system! Honestly I did not expect that step to be a show-stopper. It turned out that it is. Most people seem to use the EFI boot manager _rEFInd_. You can install it from Windows, OS X and Linux. Yes, that's it. Now, I was not surprised that it was not ported to OpenBSD. But it did surprise me that it's not available for FreeBSD!
It may be a quite simple thing to toss in a Linux CD and install an EFI bootloader - or maybe not. I have no clue if there are any other obstacles waiting. But this was the point where I stopped. I wanted a dual-boot BSD system and I don't want to have to use Linux to do that. I've got my pride, too, you know. :p No seriously, at that point I decided to give up EFI for now and continue another time. Maybe that will be worth its own blog post. Who knows.
Here we are, back to using BIOS / MBR. I wanted to use a purely ZFS setup for FreeBSD but another problem showed up: The bootcode to load a kernel from ZFS is quite a bit more complicated than its UFS equivalent. For that reason it doesn't fit into the boot sector of a partition but needs its own small partition instead. However with gpart the type _freebsd-boot_ seems to be only supported for GPT...
To get anything up and running again (for the time being my on-call laptop was broken after all and my next duty was around the corner!) I settled with a UFS boot partition for the unencrypted /boot partition.
The _auto_ashift_ adjustment is needed for ZFS to adhere the 4k alignment. And to be able to set that sysctl, the ZFS kernel module has to be loaded. That's a little detail but it took me a moment to figure out what's going on. Everybody advices to set the sysctl but the installer kept on telling me that there was no such sysctl... Which makes sense once you know that ZFS is not loaded into the kernel by default.
For the layout of the datasets I followed the defaults as used by the automated root-on-ZFS installation (_zpool history_ is a very interesting command! If you don't know it - try it!). I just changed the order so that similar datasets follow each other which avoids a bit of typing by bringing back the previous command and editing it.
It's more likely than not that this is not the best way to do things. But it does work. Any suggestions or other comments are welcome of course!
In the partitioning shell:
tcsh
dd if=/dev/zero of=/dev/ada0 bs=1m
gpart create -s mbr ada0
gpart add -a 4k -t freebsd -s 98G ada0
gpart add -a 4k -t freebsd ada0
gpart create -s bsd ada0s1
gpart bootcode -b /boot/boot0 ada0
gpart bootcode -b /boot/boot ada0s1
gpart set -a active -i 1 ada0
gpart add -t freebsd-ufs -s 2G ada0s1
gpart add -t freebsd-swap -s 4G ada0s1
gpart add -t freebsd-zfs ada0s1
glabel label clear /dev/ada0s1a
glabel label swap /dev/ada0s1b
glabel label system /dev/ada0s1d
newfs /dev/label/clear
dd if=/dev/random of=/dev/label/system bs=1m
geli init -b -s 4096 -l 256 /dev/label/system
geli attach /dev/label/system
kldload zfs
sysctl vfs.zfs.min_auto_ashift=12
zpool create -o altroot=/mnt -O compress=lz4 -O \
atime=off -m none -f zroot /dev/label/system.eli
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off \
zroot/tmp
zfs create zroot/usr/home
zfs create zroot/usr/src
zfs create -o setuid=off zroot/usr/ports
zfs create -o setuid=off zroot/var/tmp
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs set mountpoint=/zroot zroot
exit
exit
In the "final modifications" chroot:
mkdir /realboot
mount /dev/label/clear /realboot
mv /boot /realboot
ln -s /realboot/boot /boot
echo 'geom_eli_load="YES"' >> /boot/loader.conf
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot/ROOT/default"' > /boot/loader.conf
echo '/dev/label/swap.eli none swap sw 0 0' >> /etc/fstab
echo '/dev/label/clear /realboot ufs rw 1 1' >> /etc/fstab
sysrc zfs_enable="YES"
Now have fun with ZFS!
