Sunday, 30. January 2021

Multi-OS PXE-booting from FreeBSD 12: Required services (pt. 2)

[This article has been bi-posted to Gemini and the Web]

The previous post was about what lead me to do this in the first place, featured a little excursion for people new to PXE and most importantly detailed the setup of a FreeBSD router that will be turned into a PXE server in this article.

Multi-OS PXE-booting from FreeBSD 12: Introduction (pt. 1)

While I originally intended to show how to boot Ubuntu in this part, things have changed a little. I realized that the software choices I made might not be what a lot of people would have chosen. Therefore I did a little extra work and present my readers with multiple options. This made the article long enough even without the Ubuntu bits which will be in part 3 instead.

Current state and scope

Today we will make the machine offer all the services needed for network installations of many Open Source operating systems. My examples are all IPv4, but it should be possible to adapt this to IPv6 fairly easily. As this is *nix, there's always multiple ways to do things. For software that does not come included in FreeBSD's base installation, I'll be giving two options. One will be less commonly deployed but have some advantages in my book. The other will achieve the same goal with a more popular solution.

The machine that I use is an old piece of metal that has 6 NICs - which is why I like to still use it for tinkering with network-related stuff. After part 1, we left off with a gateway that has an Internet-facing adapter _em0_ which gets its IP via DHCP from my actual router. We're also using _em5_ which is statically configured to have the IP 10.11.12.1 and is connected to a separate switch. There's an _unbound nameserver_ running and serving that interface and a _pf firewall_ is active doing _NAT_.

This means that everything is in place to serve any host connected to said switch, if it's manually configured to use a static address in the 10.11.12.0/24 IP range and with default router and nameserver set to 10.11.12.1. Let's start by getting rid of that "manually configured" requirement!

Excursion: DHCP basics

We do so by configuring the machine as a _DHCP server_ handing out IP addresses for the 10.11.12.0/24 subnet on the 6th NIC. DHCP servers work by listening for _DHCP requests_ which are broadcasted on the network (as the client does not have it's own IP, yet). When receiving one, it will have a look at its configuration: Is there anything special for the host asking? Is the MAC address included with the request maybe blacklisted? Is there a reserved IP to be handed to this specific machine? Or any particular option to send to the "class" of device asking?

In our simple case there aren't really any bells and whistles involved. So it will have a look at the IP pool that it manages and if it can find an unused one, it will answer with a _DHCP offer_. The latter is a proposal for an IP to be leased by the client and also includes various network-related information: Usually at least the netmask, router and nameserver. A lot of additional information can be provided in form of _options_; you can point the client joining the network to a time server if you have one, inform about the domain being used and much, much more (even custom options are possible if you need them).

For PXE booting to work we need to make use of two particular options: We need the PXE code in the firmware to know which server to turn to if it wants to load the NBP (Network Bootstrap Program) from the net. It also needs to know what the file to ask for is called.

DHCP servers

There are multiple DHCP servers out there. If I were doing this on Linux, I'd probably just pick __Dnsmasq__ and be done: As the name implies, it does DNS. But it also does DHCP and TFTP (which we are in need of as well) and supports PXE. But FreeBSD comes with its own TFTP daemon that I'm going to use and I actually prefer the Unix way over all-rounder software: _Do one thing and do it well_!

http://www.thekelleys.org.uk/dnsmasq/doc.html

The first thing that comes to mind in terms of DHCP servers is ISC's _DHCPd_. It's small, simple to use (at least for our use case), battle-tested and in use pretty much everywhere. It's old, though, not extremely fast and certainly not as flexible as you might wish for today. This (among other things) lead the ISC to start a new project meant as a replacement: _Kea_.

The latter is a modern DHCP server with a lot of nice new features: It is a high-performance solution that's extensible, supports databases as backends, has a web GUI (Stork) available and more. But since DHCPd works well enough, adoption of Kea has been slow. There are a couple of downsides to it, too: First and foremost - its configuration is written in JSON. Yes, JSON! While there are legitimate use cases for that format, configuration is *not* one of them if you ask me. That was a terrible choice. Kea also pulls in big dependencies like the _boost_ C++ libraries not everybody is fond of.

https://github.com/isc-projects/stork

IMO the benefits of Kea outweight the drawbacks (if it wasn't for the JSON configuration, I'd even state: clearly). But it's your choice of course.

DHCP server option 1: Modern-day Kea

Alright, let's give Kea a try, shall we? First we need to install it and then edit the configuration file:

# pkg install -y kea
# vi /usr/local/etc/kea/kea-dhcp4.conf

The easiest thing to do is to delete the contents and paste the following. Then adapt it to your network and save:

{
"Dhcp4": {
"interfaces-config": {
"interfaces": [ "em5/10.11.12.1" ]
},
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea4-ctrl-socket"
},
"lease-database": {
"type": "memfile",
"lfc-interval": 3600
},
"expired-leases-processing": {
"reclaim-timer-wait-time": 10,
"flush-reclaimed-timer-wait-time": 25,
"hold-reclaimed-time": 3600,
"max-reclaim-leases": 100,
"max-reclaim-time": 250,
"unwarned-reclaim-cycles": 5
},
"renew-timer": 900,
"rebind-timer": 1800,
"valid-lifetime": 3600,
"next-server": "10.11.12.1",
"boot-file-name": "pxelinux.0",
"option-data": [
{
"name": "subnet-mask",
"data": "255.255.255.0"
},
{
"name": "domain-name-servers",
"data": "10.11.12.1"
},
{
"name": "domain-name",
"data": "example.org"
},
{
"name": "domain-search",
"data": "example.org"
}
],
"subnet4": [
{
"subnet": "10.11.12.0/24",
"pools": [ { "pool": "10.11.12.20 - 10.11.12.30" } ],
"option-data": [
{
"name": "routers",
"data": "10.11.12.1"
}
]
}
],
"loggers": [
{
"name": "kea-dhcp4",
"output_options": [
{
"output": "/var/log/kea-dhcp4.log"
}
],
"severity": "INFO",
"debuglevel": 0
}
]
}
}

Yes, looks pretty bad, I know. But that's only the representation; if something better had been used (say YAML), it'd be about 50 lines instead of 75, be much more readable and above all: less error-prone to edit. Oh well. If you can ignore the terrible representation, the actual data is not so bad and pretty much self-explaining.

I'd like to point you at the "next-server" and "boot-file-name" global options that I set here. These are required for PXE booting by pointing to the server hosting the NBP and telling its file name. Leave them out and you will still have a working DHCP server if you don't need to do PXE. While this configuration works, you will likely want to extend it for production use.

With the config in place, let's enable and start the daemon:

# sysrc kea_enable="YES"
# service kea start

A quick look if a daemon successfully bound to port 67 and is listening doesn't hurt:

# sockstat -4l | grep 67
root kea-dhcp4 1480 14 udp4 10.11.12.1:67 *:*

Ok, there we are. We now have a DHCP service on our internal network!

DHCP server option 2: Venerable ISC DHCPd

So you'd like to play simple and safe? No problem, going with DHCPd is not a bad choice. But first we need to install it and edit the configuration file:

# pkg install -y isc-dhcp44-server
# vi /usr/local/etc/dhcpd.conf

Delete everything. Then add the following (adjust to your network structure, of course!) and save:

option domain-name "example.org";
option domain-name-servers 10.11.12.1;
option subnet-mask 255.255.255.0;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
log-facility local7;
next-server 10.11.12.1;
filename "pxelinux.0";
subnet 10.11.12.0 netmask 255.255.255.0 {
range 10.11.12.10 10.11.12.20;
option routers 10.11.12.1;
}

Mind the "next-server" and "filename" options which define the server to get the NBP from as well as the file name of that. You can leave out that block and will still have a working DHCP server - but it won't allow for PXE booting in that case. I'd also advice you to do a bit of reading and probably do a more comprehensive configuration of DHCPd.

Next thing to do is to enable DHCPd, confine it to serve requests coming in from one particular NIC only and start the service:

# sysrc dhcpd_enable="YES"
# sysrc dhcpd_ifaces="em5"
# service isc-dhcpd start

Quick check to see if the service is running and binding on port 67:

# sockstat -4l | grep 67
dhcpd dhcpd 1396 8 udp4 *:67 *:*

Looking good so far, DHCP should be available on our internal net.

Optional: Checking DHCP

If you want to make sure that your DHCP server is not only running but that it can also be reached and actually does what it's meant to, you can either just try to power up a host in the 10.11.12.0/24 network and configure it to get its IP via DHCP. Or you can for example use the versatile _nmap_ tool to test DHCP discovery from any host on that network:

# pkg install -y nmap
# nmap --script broadcast-dhcp-discover
Starting Nmap 7.91 ( https://nmap.org ) at 2021-01-24 17:56 CET
Pre-scan script results:
| broadcast-dhcp-discover:
| Response 1 of 1:
| Interface: em0
| IP Offered: 10.11.12.21
| DHCP Message Type: DHCPOFFER
| Subnet Mask: 255.255.255.0
| Router: 10.11.12.1
| Domain Name Server: 10.11.12.1
| Domain Name: example.org
| IP Address Lease Time: 1h00m00s
| Server Identifier: 10.11.12.1
| Renewal Time Value: 15m00s
|_ Rebinding Time Value: 30m00s
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 10.56 seconds
# pkg delete -y nmap

All right! DHCP server is working and happily handing out leases.

TFTP

The Trivial File Transfer Protocol daemon is up next. FreeBSD ships with a TFTP daemon in the base system, so we're going to use that. It will not be used by itself but instead from the _inetd_ super daemon. To enable TFTP, we just need to put one line in the inetd configuration file:

# echo "tftp dgram udp wait root /usr/libexec/tftpd tftpd -l -s /usr/local/tftpboot" >> /etc/inetd.conf

Now we need to create the directory that we just referenced, as well as a subdirectory which we're going to use and create a file there:

# mkdir -p /usr/local/tftpboot/pxelinux.cfg
# vi /usr/local/tftpboot/pxelinux.cfg/default

Put the following in there (for now) and save:

DEFAULT vesamenu.c32
PROMPT 0

All is set, let's enable and start the service now:

# sysrc inetd_enable="YES"
# service inetd start

Again we can check real quick if the service is running:

# sockstat -4l | grep 69
root inetd 1709 6 udp4 *:69 *:*

Good, so now we can test to fetch the configuration file from either our server or from any FreeBSD machine in the 10.11.12.0/24 network:

# tftp 10.11.12.1
tftp> get pxelinux.cfg/default
Received 292 bytes during 0.0 seconds in 1 blocks
tftp> quit
# rm default

Everything's fine just as expected.

File Server

The remaining piece we need to set up is a means to efficiently transfer larger files over the wire - i.e. not TFTP! You can do it via FTP and use FreeBSD's built-in FTP daemon. While this works well, it is not the option that I'd recommend. Why? Because FTP is an old protocol that does not play nicely with firewalls. Sure, it's possible to do FTP properly, but that's more complex to do than using something else like a webserver that speaks HTTP.

If you want to go down that path, there are *a lot* of options. There's the very popular and feature-rich _Apache HTTPd_ and various more light-weight solutions like _LighTTPd_ and many more. I generally prefer _OpenBSD's HTTPd_ because it is so easy to work with - and when it comes to security or resisting feature creep its developers really mean it. If I need to do something that it cannot do, I usually resort to the way more advanced (and much more popular) _Nginx_.

https://httpd.apache.org

https://www.lighttpd.net

https://github.com/koue/httpd

https://nginx.org

Pick any of the two described here, go with FTPd instead or just ignore the following three sections and set up the webserver that you prefer to deploy.

If you didn't opt for FTP, as a first step create a directory for the webserver and change the ownership:

# mkdir -p /usr/local/www/pxe
# chown -R www:www /usr/local/www/pxe

File Server option 1: OpenBSD's HTTPd

Next is installing the program and providing the desired configuration. Edit the file:

# pkg install -y obhttpd
# vi /usr/local/etc/obhttpd.conf

Delete the contents and replace it with the following, then save:

chroot "/usr/local/www"
logdir "/var/log/obhttpd"
server "pixie.example.org" {
listen on 10.11.12.1 port 80
root "/pxe"
log style combined
}

Super simple, eh? That's part of the beauty of obhttpd. OpenBSD follows the "sane defaults" paradigm. That way you only have to configure stuff that is specific to your task as well as things where you want to change the defaults. Surprisingly, this configuration does it - and there's really not much I'd change for a production setup if it is the only site on this server.

It's always a good idea to check if the configuration is valid, so let's do that:

# obhttpd -nf /usr/local/etc/obhttpd.conf
configuration OK

If you ever need to debug something, you can start the daemon in foreground and more verbosely by running _obhttpd -dvv_. Right now the server would not start because the configured log directory does not exist. So this would be a chance to give debugging a try.

Let's create the missing directory and then enable and start the service:

# mkdir /var/log/obhttpd
# sysrc obhttpd_enable="YES"
# service obhttpd start

As always I prefer to take a quick look if the daemon did bind the way I wanted it to:

# sockstat -4l | grep httpd
www obhttpd 1933 7 tcp4 10.11.12.1:80 *:*
www obhttpd 1932 7 tcp4 10.11.12.1:80 *:*
www obhttpd 1931 7 tcp4 10.11.12.1:80 *:*

Looks good.

File Server option 2: Nginx

Next thing is installing Nginx and providing a working configuration:

# pkg install -y nginx
# vi /usr/local/etc/nginx/nginx.conf

Erase the example content and paste in the following:

user www;
error_log /var/log/nginx/error.log;
worker_processes auto;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
keepalive_timeout 65;
server {
listen 80;
location / {
root /usr/local/www/pxe;
}
}
}

This is by no means a perfect configuration but only an example. If you want to deploy Nginx in production, you'll have to further tune it towards what you want to achieve. But now let's enable and start the daemon:

# sysrc nginx_enable="YES"
# service nginx start

Time for the usual quick check:

# sockstat -4l | grep nginx
www nginx 1733 6 tcp4 *:80 *:*
www nginx 1732 6 tcp4 *:80 *:*
root nginx 1731 6 tcp4 *:80 *:*

Nginx is running and listening on port 80 as it should be.

File Server option 3: FTPd

FTP for you, eh? Here we go. FreeBSD comes with an _ftp_ group but not such a user by default. Let's create it:

# pw useradd -n ftp -u 14 -c "FTP user" -d /var/ftp -g ftp -s /usr/sbin/nologin

It's a convention that public data offered via anonymous FTP is placed in a "pub" directory. We're going to honor that tradition and create the directory now:

# mkdir -p /var/ftp/pub
# chown ftp:ftp /var/ftp/pub

If you intend to use logging, create an empty initial log file:

# touch /var/log/ftpd

Now we need to enable the FTP service for inetd (the "l" flag enables a transfer log, "r" is for operation in read-only mode, "A" allows for anonymous access and "S" is for enabling the download log):

# echo "ftp stream tcp nowait root /usr/libexec/ftpd ftpd -l -r -A -S" >> /etc/inetd.conf

As we intend to run a service that allows local users to log in via FTP, we need to consider the security implications of this. In my case I have created the "kraileth" user and it has the power to become root via doas. While OpenSSH is configured to only accept key-based logins, FTP is not. The user also has a password set - which means that an attacker who suspects that the user might exist, can try brute-forcing my password.

If you're the type of person who is re-using passwords for multiple things, take this scenario into consideration. Sure, this is an internal server and all. But I recommend to get into a "security first" mindset and just block the user from FTP access, anyway. To do so, we just need to add it to the _/etc/ftpusers_ file:

# echo "kraileth" >> /etc/ftpusers

Now let's restart the inetd service (as it's already running for TFTP) and check it:

# service inetd restart
# sockstat -4l | grep 21
root inetd 1464 7 tcp4 *:21 *:*

Ready and serving!

Optional: Checking File Server service

Time for a final test. If you're using a webserver, do this:

# echo "TestTest" > /usr/local/www/pxe/test
# fetch http://10.11.12.1/test
test 9 B 8450 Bps 00s
# cat test
TestTest
# rm test /usr/local/www/pxe/test

If you're using FTP instead:

# echo "TestTest" > /var/ftp/pub/test
# fetch ftp://10.11.12.1/pub/test
test 9 B 9792 Bps 00s
# cat test
TestTest
# rm test /var/ftp/pub/test

Everything's fine here, so we can move on.

What's next?

In part 3, we're finally going to add data and configuration to boot multiple operating systems via PXE.

BACK TO NEUNIX INDEX