💾 Archived View for gemini.circumlunar.space › users › kraileth › neunix › 2021 › bsd_router_take_2_… captured on 2022-03-01 at 15:28:38. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-05)
-=-=-=-=-=-=-
Saturday, 24. April 2021
[This article has been bi-posted to Gemini and the Web]
In 2017 I wrote my longest (by far) series of posts on a single topic: 8 posts on hardware and the FreeBSD-based firewall solutions _pfSense_ and _OPNsense_. Even after almost four years, some of these posts are still very high on the list of frequently visited pages (on the weblog). A lot has happened since then, though. About time that I get back to the topic! Here's the list of the old posts:
Building a BSD home router (pt. 1): Hardware (PC Engines APU2)
Part 1 discusses why you want to build your own router and how to assemble the APU2
Building a BSD home router (pt. 2): The serial console (excursion)
Part 2 provides some Unix history explanation of what a serial console is
Building a BSD home router (pt. 3): Serial access and flashing the firmware
Part 3 demonstrates serial access to the APU and covers firmware update
Building a BSD home router (pt. 4): Installing pfSense
Part 4 details installing pfSense
Building a BSD home router (pt. 5): Installing OPNsense
Part 5 shows installing OPNsense instead
Building a BSD home router (pt. 6): pfSense vs. OPNsense
Part 6: Comparison of pfSense and OPNsense
Building a BSD home router (pt. 7): Advanced OPNsense installation
Part 7 covers the advanced installation of OPNsense
Building a BSD home router (pt. 8): ZFS and jails
Part 8 covers creating a ZFS pool for data and preparing the system for using jails
I meant to revisit this topic again much sooner, but it didn't work out. In 2018 I started to write a post that (among other text) contained the following paragraphs:
A lot has happened in the meantime. The new pfSense version that I mentioned before has left beta and is available regularly now. And with it comes one real boon: Complete ZFS support! OPNsense had two releases since 17.1 (the version that I discussed): 17.7 and 18.1. I had updated my box to 17.7 without any problems IIRC. After moving houses (and finally being online again after months!) I brushed the dust off my router and connected it. I decided to do a re-install this time and see if anything noteworthy had changed.
First I was a little disappointed to see that the USB3 issue has not yet been taken care off and still makes installing on the APU2 harder than it needs to be. But well, I had figured out how to do it previously, and still had my gear at hand to install using the internal USB2 connection. Second letdown: Still no trace of ZFS to be found. Looks like my priorities do not completely match the ones of the OPNsense team! 😉
But then the pleasant surprise: The 18.1.8 update brought experimental ZFS support! I had done a simple install this time as I had been a bit in a hurry, so I could not test it myself so far. However if I got things right, this means that the boot scripts are finally ZFS-capable and this will likely be supported in the 18.7 release. Chances are that there will not be installer support for ZFS, though. But we’ll see! Good things are going on and eventually we'll get there.
I didn't finish the article for reasons that escape me today, but when I wanted to return to the topic another year later, my hardware broke and so I had to postpone that again. My APU has since been fixed and even the old 16GB mSATA drive it originally came with replaced with a nice new one that has much more useful 512 GB of space - but I never found the time to even boot it up once since early 2020.
In late February 2021 I finally got around to install OPNsense on it again and thus get my own router back in business. I took notes but publishing this article was delayed for various reasons. Anyway, here we go! Let's do a fully ZFS-based installation (which was not possible, yet in 2017).
When I first built my router, I had to be a bit creative (see parts 7 and 8 linked above) to end up with an OPNsense installation that offered ZFS so that I could use a jails manager on it that depends on that filesystem. However it was only an additional pool for data. The actual system ran on UFS.
For a while now OPNsense _does_ support booting from ZFS! I definitely want that, so let's give it a try. I download a DVD image for OPNsense 21.1 and put that on my CD emulator device. Then I wire my APU up to my workstation via a serial cable and connect:
# cu -l /dev/cuaU0 -s 115200
Then I boot the machine up. As I'm using the DVD image, it's configured for VGA mode. That means in my case I have to manually configure it for serial usage. This can be done conveniently in the loader in FreeBSD 12.2 and newer. However... OPNsense 21.1 is still based on version 12.1 where that option is not available, yet. Even though this is an older and in fact unsupported version, I cannot blame the project. They don't use vanilla FreeBSD directly but are based on HardenedBSD instead, a close fork that is about hardening the code base.
While HardenedBSD has attracted enough attention and people to successfully form their own foundation, it's still a small project with limited resources. And 12.1 is the only release from FreeBSD major version 12 that they support. If you like to support Open Source projects and have some spare money, consider donating. They are doing important work to move FreeBSD in the right direction!
So we have to do this the old way. Pressing ESC drops me at the loader prompt. Time to set some values and then boot:
set boot_serial=YES
set comconsole_speed=115200
set console=comconsole
boot
Alright! Kernel is booting up and printing all messages to the serial console. It shouldn't take too long before... Whoo! What's this?
Mounting from cd9660:/dev/iso9660/12_1_RELEASE_AMD64_CD failed with error 19.
Good lord, I remember this! OPNsense 17.1 (or was it 16.7?) had a problem booting up on some USB3 ports. You had to use a workaround to be able to boot up to the installer... I totally expected that this would have been fixed by now. It's been four years! *sigh*
I quickly tried out installing from a usb memory stick - but ran into the same problem. So what now? Back in the day I opened up the case and connected an adapter to be able to use an internal USB2 connector. Of course I could do that again. Except - I have no idea whatsoever where I put that before moving houses... A little research doesn't exactly leave me in a much better mood: As of OPNsense 21.1 they've apparently not added an easy way to install to ZFS, yet. At this point I'm ready to question if this project was a good idea or if I should simply do something else.
But this is not as bad as you might think. Fortunately there's a very simple way of achieving it anyway: Just install vanilla FreeBSD and then use a _conversion script_. As I had just completed a series on PXE-booting (and the PXE server still at hand), I chose to go down that route. Take a look at that article (and the previous two) if you're interested in the PXE booting details or don't understand something that I do in the next section.
Multi-OS PXE-booting from FreeBSD 12: PXE menu and *BSD (pt. 3)
On the very day I'm finally writing this article though, the OPNsense team has announced switching the installer for the upcoming version 21.7, making installation to ZFS a regular installation option. So that's one thing to look forward to if you like ZFS but don't want to use the bootstrap script. 🙂
So I boot up my PXE server. It was prepared to serve FreeBSD 12.2. It is a very bad idea to try to bootstrap OPNsense on a version of FreeBSD that's newer than the one the firewall OS is based on. So for this special case I add FreeBSD 12.1 support on my PXE server:
# mkdir /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/MANIFEST -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/MANIFEST
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/kernel.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/kernel.txz
By default, PXE-booting is disabled on the APU. So when turning on the device without any USB device attached to it, this is what I get on the serial console:
Press F10 key now for boot menu
Select boot device:
1. Payload [setup]
2. Payload [memtest]
So let's see what options the setup has to offer:
Boot order - type letter to move device to top.
d SATA
c mSATA
b SDCARD
a USB
e mPCIe1 SATA1 and SATA2
f iPXE (disabled)
r Restore boot order defaults
n Network/PXE boot - Currently Disabled
u USB boot - Currently Enabled
t Serial console - Currently Enabled
o UART C - Currently Enabled
p UART D - Currently Enabled
m Force mPCIe2 slot CLK (GPP3 PCIe) - Currently Disabled
h EHCI0 controller - Currently Disabled
l Core Performance Boost - Currently Enabled
i Watchdog - Currently Disabled
j SD 3.0 mode - Currently Disabled
v IOMMU - Currently Disabled
y PCIe power management features - Currently Disabled
w Enable BIOS write protect - Currently Disabled
x Exit setup without save
s Save configuration and exit
Ok, using "n" I can enable PXE booting. Next time I get this message:
Press F10 key now for boot menu, N for PXE boot
Pressing _n_ takes me to the iPXE boot menu. I quickly interfere to manually specify what to do instead of letting it automatically guess (and guess wrong). At the iPXE prompt I ask it to get an ip address and then chainload pxelinux like this:
iPXE> dhcp
Configuring (net0 00:0d:b9:42:67:64)...... ok
iPXE> chain pxelinux.0
There we are. As the mfsBSD image that I use is FreeBSD 12.2-based, I can set the console to serial in the loader (by pressing "5") and then boot. Once the system is up, I login as _root_ with the password _mfsroot_. Next is preparing the manifest and then starting the familiar FreeBSD installer:
# mkdir -p /usr/freebsd-dist
# fetch http://10.11.12.1/bsd/fbsd/amd64/12.1-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
# bsdinstall
I want a minimal, bare-bones FreeBSD system and thus choose to not install any optional distsets. When asked about the installation source, I pick "other" and enter _http://10.11.12.1/bsd/fbsd/amd64/12.1-RELEASE_ as that's where my PXE server offers the required files. Now I can install FreeBSD using root-on-ZFS and everything as I like it.
When the installation is done, I reboot.
I enter setup again and disable network booting. As this is 12.1 again, I need to manually enable the system for use with the serial console. So when the loader menu comes up, I escape to the loader prompt and again use the same commands as above to set the variables to the right values and then boot.
After logging in as root, I download the conversion script:
# fetch --no-verify-peer https://raw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh
Again, we're using 12.1 which is the last version without without a certificate trust store in the base system. I could install the ca_root_nss package, but I'm choosing to just not validate the TLS cert this one time (the bootstrap process will do it correctly then).
All that's left to do now is running the script:
# sh opnsense-bootstrap.sh
This utility will attempt to turn this installation into the latest
OPNsense 21.1 release. All packages will be deleted, the base
system and kernel will be replaced, and if all went well the system
will automatically reboot.
Proceed with this action? [y/N]: y
It will then install packages and eventually the base system:
Fetching base-21.1.1-amd64.txz: ........................... done
Fetching kernel-21.1.1-amd64.txz: ....... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-21.1.1-amd64.txz... done
Installing base-21.1.1-amd64.txz... done
Please reboot.
Then it automatically reboots.
Settings for the serial console need to be entered again at the loader once more. Conversion worked like a charm: OPNsense boots up just fine. I'm starting over fresh here, so no configuration importer for me. But yes, I do want manual interface asignment.
For my box I do the following settings: No VLANs, manually configure: igb0 -> WAN, igb1 -> LAN, igb2 -> Opt
Once it's done, I can connect from my workstation on the LAN to it by opening _192.168.1.1_ in a browser. Login credentials are user _root_ & password _opnsense_ (yes, the conversion script wiped whatever root password you chose before).
Then I complete the wizard which mostly means changing the login password. Afterwards I go to _System -> Firmware -> Settings_ and change the Firmware flavor from the default OpenSSL to _LibreSSL_ (don't forget to save). Next is checking for updates and updating the system.
The Web UI is nice and all, but I'm a keyboard and terminal person. So for convenience I add a new user for me. To do so, I go to _System -> Access -> Users_. Obviously a username is required. I don't want a password for that user, so I leave that blank and check the box "Generate a scrambled password". My login shell of choice is /bin/tcsh (as zsh is unfortunately not in FreeBSD's base system). Since I need a privileged user, I add "admins" to the group memberships. Then I paste in my public SSH key (from ~/.ssh/id_ed25519.pub on my workstation - if you're using a different algorithm use the correct file name for that) and save.
The last thing that I do for settings is allowing my user to actually SSH into the box. So I'm going to _System -> Settings -> Administration_. There I check the box "Enable Secure Shell" and set "Listen Interfaces" to "LAN". I'm also allowing passwordless sudo for "wheel,admins". After I saved the settings, let's see if I can connect to the box via SSH and become root:
% ssh 192.168.1.1
Enter passphrase for key '/home/kraileth/.ssh/id_ed25519':
----------------------------------------------
| Hello, this is OPNsense 21.1 | @@@@@@@@@@@@@@@
| | @@@@ @@@@
| Website: https://opnsense.org/ | @@@\\\ ///@@@
| Handbook: https://docs.opnsense.org/ | )))))))) ((((((((
| Forums: https://forum.opnsense.org/ | @@@/// \\\@@@
| Code: https://github.com/opnsense | @@@@ @@@@
| Twitter: https://twitter.com/opnsense | @@@@@@@@@@@@@@@
----------------------------------------------
% sudo -i
*** OPNsense.localdomain: OPNsense 21.1.1 (amd64/LibreSSL) ***
[...]
0) Logout 7) Ping host
1) Assign interfaces 8) Shell
2) Set interface IP address 9) pfTop
3) Reset the root password 10) Firewall log
4) Reset to factory defaults 11) Reload all services
5) Power off system 12) Update from console
6) Reboot system 13) Restore a backup
Enter an option:
There we go. Now it's time to configure my firewall settings. As this is a topic of it's own, I'm going to skip this here.
Just like 4 years ago, there are still some hurdles to overcome to install OPNsense on my router (especially the USB3 issue). There's also the fact that it's based on FreeBSD 12.1 instead of 12.2 which is not ideal. But again: I'm not really complaining about that; HardenedBSD chose to concentrate on 13.0 and taking the resources of their small team into account, this was a perfectly reasonable decision. But regarding things like base system certificates and especially console selection in the loader... Well, you adapt to nice new features incredibly fast, making things you had to do before that (and did for years!) a bit of a nuisance. 😉
I started with OPNsense 17.1, I think and did a lot of point release updates as well as a couple of major release upgrades before my hardware went bad during the 18.7 or early 19.1 life cycle, I think. It had never failed me. Being able to use root-on-ZFS today is definitely nice progress. Since I installed the system in early March, a couple of updates came out. It's great to see how reliably everything still works. So I'd say that I'm back on track - and I'll definitely have a couple of things that I want to achieve (and write about) this time. Stay tuned!