Sunday, 24. January 2021
[This article has been bi-posted to Gemini and the Web]
This is an introductory article; if you're familiar with PXE you will want to skip the excursion but may be interested in the "Why". The article ends with the post-installation setup of my test machine, turning it into a simple router so that the next article can start with the actual PXE-related setup.
These days I decided to reactivate an old laptop of mine. It has processor of an older generation (Ivy Bridge) but it's an i7 - and it has 24 GB of RAM, making it a somewhat nice machine for certain things. Problem with it is: The USB does not work (I think that the controller was fried)!
So how do I get a current OS installed on there as I cannot use the CD emulator I'd normally use? Sure, I could always burn a CD, but I don't feel like it (unnecessary waste). I could open it up, remove the drive, place it in another machine, install the OS and then put it back. Nah, not too much fun, either. How about doing a network install then?
When I thought of that I realized that I had some old notes from about a year ago somewhere on my workstation. I originally meant to blog about that topic but never actually wrote the post. So here's how to do what the post's title says - updated to FreeBSD 12.2 and Ubuntu 20.04!
While I have a clear favorite OS and very much appreciate what FreeBSD has to offer for me as the administrator of a fleet of servers, there really is no reason to turn a blind eye to other Unix-like systems. For one thing it totally makes sense to always pick the best candidate for a job (which might not in all cases be one OS). That and the fact that you cannot judge which one is best suited if you don't have at least some level of familiarity with various options. But in addition to this my employer runs a heterogeneous environment, anyway - and while I'm mostly occupied with the BSD side of things, there's simply way, way too many Linux machines around to even think about avoiding them altogether all the time.
After an update, the old means of installing Linux servers as we do it at work had stopped working reliably. I looked at it briefly but soon found that too many things weren't set up like you'd do it today. Therefore I decided that it might make more sense to start fresh. And while at it - wouldn't it make sense to try and combine the Linux and FreeBSD PXE servers on one machine? Then one more old server could be retired.
The next installation due was for a customer who requested an Ubuntu server. As a matter of fact we are transitioning to use that distribution more often for our internal infrastructure, too (decision by management, certainly not that of the team responsible for the actual servers!). For that reason one weekend I ended up something that I hadn't done in a while: installing a fresh Ubuntu 16.04 system on a test machine I. After doing this I could write a whole post about how bad the installer actually is (static IP addressing, anyone??), but I don't want this to turn into a rant.
So let's just mention one single complaint: Installing takes quite a long time (I never understood why Debian and derivatives install stuff that they remove again later during the "cleaning up" phase...) Before the installation was even finished, I admittedly already had mixed feelings about this new system. But then this was what happened on the first boot:
[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-0ef05387\x2d50d9\x2d4cac\x2db96\x2d9808331328af.device.
[DEPEND] Dependency failed for /dev/disk/by-uuid/0ef05387-50d9-4cac-b796-9808331328af.
[DEPEND] Dependency failed for Swap.
[ *** ] A start job is running for udev Coldplug all Devices (3min 34s / no limit)
You've got to be kidding! A freshly installed system going sideways like that? Sorry Ubuntu that's not the kind of stuff that I like wasting my time with! I don't even care if it's systemd's fault or something else. The boss "preferably" wanted an Ubuntu system - I gave it a try and it failed me. Ah, screw it, let's deploy something I know of that it actually works (same hardware BTW, before anybody wants to blame that for a problem that's definitely Ubuntu's fault)!
I had a freshly installed FreeBSD with static IP configuration (which you probably want to use for a DHCP / PXE boot server after all) in a fraction of the time that the Ubuntu installation took. And it goes without saying: System starts as one would expect.
While there have been earlier methods for making a machine boot over the network, __PXE__ (_Preboot eXecution Environment_) is what you want to rely on if you need to do that today. It is a specification widely implemented (in fact just about everywhere) and chances are very low that you will run into problems with it. Have a look around in your computer's EFI or BIOS, often PXE-booting is disabled (and if you want to use it just once to install an OS on the machine, I recommend that you disable it again afterwards). How to make a machine boot over the network depends on its EFI / BIOS. Figure out how to do it four your metal and give it a try.
On your average home environment not too much should happen. The PXE environment will probe the network for the information that it needs, receive none and eventually give up. But what information does it need and which components are involved?
Well, it needs to load an operating system "from the net" - which means _from another machine_. To be able to talk to other hosts on the network, it needs to become part of said net. For this it needs to know the network parameters and requires a usable, unique IP address for itself. It can get all of that from a _DHCP (Dynamic Host Configuration Protocol) server_. If configured correctly, that daemon will accept DHCP requests and provide the client with a suggested IP as well as information about the network (like the netmask, default router, nameservers on the net, etc). It can also tell the client from where it can load the _NBP (Network Bootstrap Program)_.
The latter is a piece of software, usually a special bootloader. It will be downloaded via _TFTP (Trivial File Transfer Protocol)_. Think of it as very similar to FTP but much, much more primitive. The NBP is executed when the transfer is completed. It will then download its configuration file and then act accordingly, most likely fetching an operating system kernel and additional files, then booting the kernel.
TFTP is what the PXE bootcode speaks and uses. But due to its very simplistic nature, TFTP is not well fit for larger file transfers. Therefore such files are usually fetched via other means once the kernel has booted and more options are available. FreeBSD likes to use NFS, while on Linux HTTP is often used to receive a filesystem or installation medium image.
So the following services are involved when setting up a PXE boot server:
Today I'm redoing my previous work in my home lab with FreeBSD 12.2. The machine that I'm using is pretty old (32-bit only Atom CPU!) but it has _6 Ethernet ports_ and still works well for network tinkering. I got it for free some years ago, so there's really no reason to complain.
When it comes to the installation, in this case I'm going with MBR + UFS which I normally wouldn't do for a amd64 machine. It's a standard 12.2 installation otherwise with my "kraileth" user (member of the _wheel_ group) added during the installation.
First thing to do is copying my public SSH key onto the server and then SSHing into the machine:
% ssh-copy-id -i ~/.ssh/id_ed25519 kraileth@hel.local
% ssh kraileth@hel.local
Now I switch to the root user, disallow SSH password-based login and restart the SSH daemon:
% su -l
# sed -i "" 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
# service sshd restart
I don't need sendmail on the machine, so off it goes:
# sysrc sendmail_enable="NONE"
Next is bootstrapping the package manager, installing the _unbound_ DNS server and - for convenience - _doas_ (a simpler sudo replacement). Writing a one-line config for doas is fine for my use case:
# env ASSUME_ALWAYS_YES=TRUE pkg bootstrap
# pkg install -y doas unbound
# echo "permit :wheel" > /usr/local/etc/doas.conf
Then unbound needs to be configured. To do so, edit the config file like this:
# vi /usr/local/etc/unbound/unbound.conf
Delete all the content, put the following in there (adjust to your network, of course) and save:
server:
verbosity: 1
interface: 10.11.12.1
access-control: 10.11.12.0/24 allow
Keep in mind that this is __not__ a production configuration! You will have to do some research on the program if you want to put it into production. Do at the very least read the more than thousand lines of comments that you just deleted (you can find it in /usr/local/etc/unbound/unbound.conf.sample). The above config will make the daemon bind on the IP configured via the "interface" statement and allow DNS recursion for any host in the subnet defined via "access-control". Let's enable the daemon now (so it will start after rebooting the machine):
# sysrc unbound_enable="YES"
During installation I chose the first NIC (em0) to be configured via DHCP as offered by my home router. This machine will act as a DHCP server for another network, so I assign a static address to the respective NIC (em5) that I have connected to a spare switch I had around. It will also act as a gateway between the two networks it is part of:
# sysrc ifconfig_em5="inet 10.11.12.1 netmask 255.255.255.0"
# sysrc gateway_enable="YES"
To actually provide Internet connectivity to hosts in the 10.11.12.0/24 subnet, we have to enable NAT (Network Address Translation). Otherwise any host that is contacted via an IP in that range will have no idea how to answer, even though our gateway forwarded the message to it. NATing is done via the firewall. FreeBSD offers three of them. _Pf_ is my absolute favorite, but _ipfw_ is ok, too. I wouldn't recommend _ipf_ - it's old and there is really no reason to use it if you're not already doing so and don't want to switch.
First we'll create a configuration file:
# vi /etc/pf.conf
Now put the following in there (again: Adjust to your network) and save:
ext_if="em0"
int_if="em5"
localnet = $int_if:network
scrub in all
nat on $ext_if from $localnet to any -> ($ext_if)
This defines commonly used three macros: One for the external and internal NIC and one for the local network. It then turns scrubbing on (you almost always want to do this and I suggest that you look up what it does). The final line does the magic of the actual NAT.
You will want to define a more complex firewall configuration if you plan to use this gateway in production. But this is enough for my (demonstration) purposes. I suggest that you follow along and when you made sure that PXE booting works, tighten up security - then test if everything still works. Firewalling is a topic of its own, and in fact not a small one, though. If you're new to it it makes perfect sense to do some reading. There's lots of free material on the net as well as a great "Book of Pf" by Peter Hansteen if you prefer.
So let's enable Pf on startup:
# sysrc pf_enable="YES"
# sysrc pf_rules="/etc/pf.conf"
Reminder: If you want to do things right, you will probably want to enable pflog, too, for example. This post is meant to point you in the right direction and to get stuff to work quickly. Familiarizing yourself with the various components used is your own homework.
It's always a good idea to update the system to the latest patchlevel:
# freebsd-update fetch install
# freebsd-version -kr
12.2-RELEASE-p1
12.2-RELEASE
Ok, looks like kernel-related updates were installed, so it's time to reboot:
# shutdown -r now
That's it for the preparation work. Once the server comes back up again it should function as a router. If you configure a host with a static address in the 10.11.12.0/24 network, it should be able to have DNS queries answered as well as reach the Internet.
In part 2 we're going to complete the setup of the PXE boot server so that a client can boot into Ubuntu Linux. The rest will be in part 3.