💾 Archived View for gemini.circumlunar.space › users › kraileth › neunix › eerie › 2017 › building_a… captured on 2022-03-01 at 15:41:44. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-05)
-=-=-=-=-=-=-
Here I'm republishing an old blog post of mine originally from July 2017. The article has been slightly improved.
Most of the advanced installation covered in part 7 and in this post is obsolete as I revise the article in 2021. While it would still work, there's no need for such a strange setup anymore - OPNsense has been supporting booting off of ZFS for a while now!
Previous parts of this series:
Building a BSD home router (pt. 1): Hardware (PC Engines APU2)
Part 1 discusses why you want to build your own router and how to assemble the APU2
Building a BSD home router (pt. 2): The serial console (excursion)
Part 2 provides some Unix history explanation of what a serial console is
Building a BSD home router (pt. 3): Serial access and flashing the firmware
Part 3 demonstrates serial access to the APU and covers firmware update
Building a BSD home router (pt. 4): Installing pfSense
Part 4 details installing pfSense
Building a BSD home router (pt. 5): Installing OPNsense
Part 5 shows installing OPNsense instead
Building a BSD home router (pt. 6): pfSense vs. OPNsense
Part 6: Comparison of pfSense and OPNsense
Building a BSD home router (pt. 7): Advanced OPNsense installation
Part 7 covers the advanced installation of OPNsense
This is the last part of this series of building a BSD home router. In the previous article we did an advanced setup of OPNsense that works but is currently wasting valuable disk space. We also configured OPNsense for SSH access. Now let's SSH in, _su -l_ to root and continue on! Choose shell (menu point 8) so that we can have a look around.
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/OPNsense 1.9G 909M 916M 50% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ada0s1b 991M 8.0K 912M 0% /none
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev
Uhm... ada0s1b is mounted on _/none_? Seriously? Let's get rid of that real quick:
# umount /none
How did _that_ happen? This leads to the question: What does our disklabel on slice 1 look like?
# gpart show ada0s1
=> 0 6290865 ada0s1 BSD (3.0G)
0 16 - free - (8.0K)
16 4194288 1 freebsd-ufs (2.0G)
4194304 2096561 2 freebsd-ufs (1.0G)
There you have it. The second one is all wrong, it's not meant to be UFS! We have to correct it to have proper swap space configured:
# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart add -t freebsd-swap ada0s1
ada0s1b added
# swapon /dev/ada0s1b
# swapinfo
Device 1K-blocks Used Avail Capacity
/dev/ada0s1b 1048280 0 1048280 0%
That's better. Now we need to adjust fstab to make this change persistent:
# vi /etc/fstab
Change the ada0s1b line like this:
/dev/ada0s1b none swap sw 0 0
Ok, we have some swap now, but we're wasting most of the disk space of our drive. Let's address that one next!
In the installer we created a second slice (MBR partition) as a placeholder:
# gpart show ada0
=> 63 31277169 ada0 MBR (15G)
63 6290865 1 freebsd [active] (3.0G)
6290928 24986304 2 !57 (12G)
Let's delete it and create a second FreeBSD slice instead:
# gpart delete -i 2 ada0
ada0s2 deleted
# gpart add -t freebsd ada0
ada0s2 added
Now we need to create a disklabel inside and create a partition for ZFS:
# gpart create -s bsd ada0s2
ada0s2 created
# gpart add -t freebsd-zfs ada0s2
ada0s2a added
OPNsense does not load the ZFS kernel module by default. So let's do that now and also notify the loader to always insert that ko during startup (we're using _loader.conf.local_ because OPNsense overwrites loader.conf during startup):
# kldload zfs
# echo zfs_load=\"YES\" >> /boot/loader.conf.local
Then we set the ashift. This tells ZFS to adjust to a 4k blocksize which is better for most of today's drives to use instead of 512 byte ones, even though a lot of them will lie to you and claim to have a 512 byte sector size. But even on a drive that really has 512 byte sectors, using 4k is better than using 512 bytes on a 4k sector drive. You will only lose some space if you have a lot of very small files in this case. In the other case however, you will hurt performance badly. If you know your drive and you want to use another blocksize, look up how to do it. Otherwise just set the ashift like this:
# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 9 -> 12
With that we're good to go and can create a pool and some datasets.
I'm calling my pool _zdata_ but feel free to name yours whatever you like better. I also enable compression on the pool level and turn off atime:
zpool create -O compression=lz4 -O atime=off -O mountpoint=none zdata /dev/ada0s2a
Next is creating some basic datasets that won't be used directly (hence forbidden to mount) but only serve as parents for other datasets:
# zfs create -o canmount=off -o mountpoint=none zdata/var
# zfs create -o canmount=off -o mountpoint=none zdata/usr
Let's move the old log dir and create some new directories:
# mv /var/log /var/log.old
# mkdir /var/log
# mkdir /usr/ports
On with creating some more datasets:
# zfs create -o mountpoint=legacy zdata/var/log
# zfs create -o mountpoint=legacy zdata/usr/ports
# zfs create -o mountpoint=legacy zdata/usr/obj
To make the system use those we need to add them to the fstab:
# vi /etc/fstab
Add these lines to the file:
zdata/var/log /var/log zfs rw 0 0
zdata/usr/ports /usr/ports zfs rw 0 0
zdata/usr/obj /usr/obj zfs rw 0 0
Once these additional lines are in place, the datasets can be mounted and the old logs transferred to their new place:
# mount -a
# mv /var/log.old/* /var/log/
The directory _/var/log.old_ is no longer needed, but the system currently has some file descriptors open that prevent deleting it. Just _rmdir_ it after the next reboot. Speaking of which: Now is a good time to do updates (and change the firmware to the libressl-based one if you haven't switched already).
BTW: **Don't try to put everything on ZFS**! I made some experiments booting into single user mode and moving over /usr and /var. The results were... not pleasing. After doing some reading I found that while OPNsense works well with ZFS datasets, it's startup process doesn't cope with ZFS very well. Place its configuration on ZFS and you're left with a partially defunct system (that doesn't know its hostname and won't start a lot of things that are needed). [Update: Never versions of OPNsense cope with ZFS just fine.]
Full ZFS support is already on the wish list for OPNsense. It looks like that won’t make it into 17.7, but I'm pretty sure that it will eventually be available, making root-on-ZFS installations possible. Yes, pfSense already has that feature in their betas for the upcoming version 2.4. And they even ditched the DragonFly installer and use the familiar BSDinstall which is really cool (dear OPNsense devs, please also take this step in the future, it would be greatly appreciated!).
Is this a good reason to switch to pfSense? It might be, if for you this is the one killer feature and you are willing to let go of OPNsense's many improvements. But there's one big blocker: If you make the switch you don't really need to read on. You won't be able to create jails easily. Why? Because pfSense heavily customizes FreeBSD. So heavily in fact that you cannot even use the ports tree by default! And that is truly a rather sad state of affairs. Sure, a lot of pfSense users actually use MacOS or even Windows and only want to ever interact with the GUI. BSD means nothing to them at all. But if you're a FreeBSD user it's pretty annoying if things simply don't work (and OPNsense shows that there's no real need to screw things up as much as pfSense does it).
The OPNsense team provides packages for OPNsense that you can simply install via _pkg_. However they currently offer only 368 packages, so chances are that you want something that is not there. The FreeBSD ports tree on the other hand means that over 27,000 programs are easily available for you! So since OPNsense is based on FreeBSD (and tries to remain close to it) this is really an option.
On FreeBSD you'd probably use _portsnap_ to get a snapshot of the current ports tree. This won't work in our case since OPNsense doesn't come with that tool. The other common way on FreeBSD is to use _svnlite_ and checkout the ports tree from the Subversion repo. Again OPNsense doesn't provide that tool. And it also doesn't package the full SVN.
So what can we do to acquire the ports tree? OPNsense does provide a _git_ package and the FreeBSD project offers a git mirror of the SVN repositories. But wait a second! OPNsense works together with the HardenedBSD team and they have their own ports tree (based on the vanilla FreeBSD one with some additions). The whole ports tree is pretty big, but we don't really want (or need) the whole history. Just what various version control systems call "head", "tip", "leaf", ... will be enough. For git we can achieve this setting the clone "depth" to 1:
# pkg install git
# git clone --depth=1 https://github.com/HardenedBSD/hardenedbsd-ports.git /usr/ports
FreeBSD ships with OpenSSL in base and a lot of ports expect to link against that. We're using LibreSSL however and so we have to tell the build system to use that by making an entry in _make.conf_:
# echo DEFAULT_VERSIONS+=ssl=libressl >> /etc/make.conf
If - for whatever reason - you decided to stick to the OpenSSL firmware, you still need to edit make.conf. This is because OPNsense uses OpenSSL from ports which is usually newer than the version in base (that cannot be upgraded between releases for ABI stability reasons). Use _ssl=openssl_ in that case.
The next step is optional, but I recommend installing a tool for dealing with ports. My example is a pretty light-weight port but maybe you want to build something more demanding. Especially in those cases a ports management tool comes in very handy. I suggest _portmaster_ which is extremely light-weight itself:
# make -C /usr/ports/ports-mgmt/portmaster install clean
Once you have it installed, you can install the jail management tool. Yes, I know that I've written about _py3-iocage_ a while ago, but that comes with a lot of dependencies and doesn't provide enough of an advantage over the purely shell based _iocell_ fork. For that reason I would simply go with that one in this case:
# portmaster sysutils/iocell
Alright! Now you have iocell installed and can start creating jails. What services would you want to jail on a small router box that is always on? Think about it for a moment. There are many great possibilities (I'll likely write another article soon about what I have in mind right now).
What have we accomplished in this series? I now have a frugal little router on my desk that is quietly doing its work. So far it's just an additional machine between my network and the modem/router box from my ISP. Taking a break from topics directly related to the actual router, I'll setup some jails (and NAT) next. But then there is a lot more to look into: How to do proper firewalling? What about traffic shaping? How to configure logging? Also VPN and VoIP come to mind as well as NTP, a DNS cache or even vLANs or intrusion detection.
OPNsense places so many tools within reach of your hands. You only have to grab one of them at a time and learn to use 'em. That's what I intend to do. And then, at some point in the future, equipped with much more solid networking knowledge, I'll try to replace that box I got from my ISP with my own modem, too. But excuse me now, I have some reading to do and configurations to break and fix again.
Reader Julian Pawlowski commented on the article and provided useful additions that I'll include here:
Thank you so much! This was really helpful for a BSD beginner like me.
Some refinements I did to the configurations:
1. ZFS kernel configs added to OPNsense tunables using the web interface instead of changing the system configurations so it can be included to the OPNsense XML config file (zfs_load=YES and vfs.zfs.min_auto_ashift=12)
2. Running “sysrc zfs_enable=YES” in command line to ensure ZFS automount after reboot so that jails can be started after boot
3. Running “sysrc jail_enable=YES jail_reverse_stop=YES” in command line to allow jails to start automatically after boot