💾 Archived View for perso.pw › blog › articles › nixos-with-live-usb-router.gmi captured on 2023-05-24 at 18:01:09. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-01-29)
-=-=-=-=-=-=-
At home, I'm running my own router to manage Internet, run DHCP, do filter and caching etc... I'm using an APU2 running OpenBSD, it works great so far, but I was curious to know if I could manage to run NixOS on it without having to deal with serial console and installation.
It turned out it's possible! By configuring and creating a live NixOS USB image, one can plug the USB memory stick into the router and have an immutable NixOS.
NixOS wiki about creating a NixOS live CD/USB
Here is a diagram of my network. It's really simple except the bridge part that require an explanation. The APU router has 3 network interfaces and I only need 2 of them (one for WAN and one for LAN), but my switch doesn't have enough ports for all the devices, just missing one, so I use the extra port of the APU to connect that device to the whole LAN by bridging the two network interfaces.
+----------------+
| INTERNET |
+----------------+
|
|
|
+----------------+
| ISP ROUTER |
+----------------+
| 192.168.1.254
|
|
| 192.168.1.111
+----------------+
| APU ROUTER |
+----------------+
|bridge #2 and #3|
| 10.42.42.42 |
+----------------+
|port #3 |
| | port #2
+----------+ |
| |
| +--------+ +----------+
| 10.42.42.150 | switch |-----| Devices |
+--------+ +--------+ +----------+
| NAS |
+--------+
Here is a list of services I need on my router, this doesn't include all my filtering rules and specific tweaks.
- DHCP server
- DNS resolving caching using unbound
- NAT
- SSH
- UPnP
- Munin
- Bridge ethernets ports #2 and #3 to use #3 as an extra port like a switch
For the curious, here is the whole configuration of the setup. In the sections after, I'll explain each parts of the code.
{ config, pkgs, ... }:
{
isoImage.squashfsCompression = "zstd -Xcompression-level 5";
powerManagement.cpuFreqGovernor = "ondemand";
boot.kernelPackages = pkgs.linuxPackages_xanmod_latest;
boot.kernelParams = [ "copytoram" ];
boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "cifs" ];
services.irqbalance.enable = true;
networking.hostName = "kikimora";
networking.dhcpcd.enable = false;
networking.usePredictableInterfaceNames = true;
networking.firewall.interfaces.eth0.allowedTCPPorts = [ 4949 ];
networking.firewall.interfaces.br0.allowedTCPPorts = [ 53 ];
networking.firewall.interfaces.br0.allowedUDPPorts = [ 53 ];
security.sudo.wheelNeedsPassword = false;
services.acpid.enable = true;
services.openssh.enable = true;
services.unbound = {
enable = true;
settings = {
server = {
interface = [ "127.0.0.1" "10.42.42.42" ];
access-control = [
"0.0.0.0/0 refuse"
"127.0.0.0/8 allow"
"10.42.42.0/24 allow"
];
};
};
};
services.miniupnpd = {
enable = true;
externalInterface = "eth0";
internalIPs = [ "br0" ];
};
services.munin-node = {
enable = true;
extraConfig = ''
allow ^63\.12\.23\.38$
'';
};
networking = {
defaultGateway = { address = "192.168.1.254"; interface = "eth0"; };
interfaces.eth0 = {
ipv4.addresses = [
{ address = "192.168.1.111"; prefixLength = 24; }
];
};
interfaces.br0 = {
ipv4.addresses = [
{ address = "10.42.42.42"; prefixLength = 24; }
];
};
bridges.br0 = {
interfaces = [ "eth1" "eth2" ];
};
nat.enable = true;
nat.externalInterface = "eth0";
nat.internalInterfaces = [ "br0" ];
};
services.dhcpd4 = {
enable = true;
extraConfig = ''
option subnet-mask 255.255.255.0;
option routers 10.42.42.42;
option domain-name-servers 10.42.42.42, 9.9.9.9;
subnet 10.42.42.0 netmask 255.255.255.0 {
range 10.42.42.100 10.42.42.199;
}
'';
interfaces = [ "br0" ];
};
time.timeZone = "Europe/Paris";
users.mutableUsers = false;
users.users.solene.initialHashedPassword = "$6$ffffffffffffffff$TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
users.users.solene = {
isNormalUser = true;
extraGroups = [ "sudo" "wheel" ];
};
}
This setup deserves some explanations with regard to each part of it.
I prefer to use zstd instead of xz for compressing the liveUSB image, it's way faster and the compression ratio is nearly identical as xz.
isoImage.squashfsCompression = "zstd -Xcompression-level 5";
There is currently an issue when trying to use a non default kernel, ZFS support is pulled in and create errors. By redefining the list of supported file systems you can exclude ZFS from the list.
boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "cifs" ];
The CPU frequency should stay at the minimum until the router has some load to compute.
powerManagement.cpuFreqGovernor = "ondemand"; services.acpid.enable = true;
This makes the system to use the XanMod Linux kernel, it's a set of patches reducing latency and improving performance.
boot.kernelPackages = pkgs.linuxPackages_xanmod_latest;
In order to reduce usage of the USB memory stick, upon boot all the content of the liveUSB will be loaded in memory, the USB memory stick can be removed because it's not useful anymore.
boot.kernelParams = [ "copytoram" ];
The service irqbalance is useful as it assigns certain IRQ calls to specific CPUs instead of letting the first CPU core to handle everything. This is supposed to increase performance by hitting CPU cache more often.
services.irqbalance.enable = true;
As my APU wasn't running Linux, I couldn't know the name if the interfaces without booting some Linux on it, attach to the serial console and check their names. By using this setting, Ethernet interfaces are named "eth0", "eth1" and "eth2".
networking.usePredictableInterfaceNames = true;
Now, the most important part of the router setup, doing all the following operations:
- assign an IP for eth0 and a default gateway
- create a bridge br0 with eth1 and eth2 and assign an IP to br0
- enable NAT for br0 interface to reach the Internet through eth0
networking = {
defaultGateway = { address = "192.168.1.254"; interface = "eth0"; };
interfaces.eth0 = {
ipv4.addresses = [
{ address = "192.168.1.111"; prefixLength = 24; }
];
};
interfaces.br0 = {
ipv4.addresses = [
{ address = "10.42.42.42"; prefixLength = 24; }
];
};
bridges.br0 = {
interfaces = [ "eth1" "eth2" ];
};
nat.enable = true;
nat.externalInterface = "eth0";
nat.internalInterfaces = [ "br0" ];
};
This creates a user solene with a predefined password, add it to the wheel and sudo groups in order to use sudo. Another setting allows wheel members to run sudo without password, this is useful for testing purpose but should be avoided on production systems. You could add your SSH public key to ease and secure SSH access.
users.mutableUsers = false;
security.sudo.wheelNeedsPassword = false;
users.users.solene.initialHashedPassword = "$6$bVPyGA3aTEMTIGaX$FYkFnOqwk8GNfeLEfppgGjZ867XxirQ19v1337.GSRdzxw7JrRi6IcpaEdeSuNTHSxIIhunter2Iy6clqB14b0";
users.users.solene = {
isNormalUser = true;
extraGroups = [ "sudo" "wheel" ];
};
This will run a DHCP server advertising the local DNS server and the default gateway, as it defines ranges for DHCP clients in our local network.
services.dhcpd4 = {
enable = true;
extraConfig = ''
option subnet-mask 255.255.255.0;
option routers 10.42.42.42;
option domain-name-servers 10.42.42.42, 9.9.9.9;
subnet 10.42.42.0 netmask 255.255.255.0 {
range 10.42.42.100 10.42.42.199;
}
'';
interfaces = [ "br0" ];
};
All systems require a name in order to work, and we don't want to use DHCP to get the IPs addresses. We also have to define a time zone.
networking.hostName = "kikimora"; networking.dhcpcd.enable = false; time.timeZone = "Europe/Paris";
This enables OpenSSH daemon listening on port 22.
services.openssh.enable = true;
This enables the service unbound, a DNS resolver that is able to do some caching as well. We need to allow our network 10.42.42.0/24 and listen on the LAN facing interface to make it work, and not forget to open the ports TCP/53 and UDP/53 in the firewall. This caching is very effective on a LAN server.
services.unbound = {
enable = true;
settings = {
server = {
interface = [ "127.0.0.1" "10.42.42.42" ];
access-control = [
"0.0.0.0/0 refuse"
"127.0.0.0/8 allow"
"10.42.42.0/24 allow"
];
};
};
};
networking.firewall.interfaces.br0.allowedTCPPorts = [ 53 ];
networking.firewall.interfaces.br0.allowedUDPPorts = [ 53 ];
This enables the service miniupnpd, this can be quite dangerous because its purpose is to allow computer on the network to create NAT forwarding rules on demand. Unfortunately, this is required to play some video games and I don't really enjoy creating all the rules for all the video games requiring it.
services.miniupnpd = {
enable = true;
externalInterface = "eth0";
internalIPs = [ "br0" ];
};
This enables the service munin-node and allow a remote server to connect to it. This service is used to gather metrics of various data and make graphs from them. I like it because the agent running on the systems is very simple and easy to extend with plugins, and on the server side, it doesn't need a lot of resources. As munin-node listens on the port TCP/4949 we need to open it.
services.munin-node = {
enable = true;
extraConfig = ''
allow ^13\.17\.23\.28$
'';
};
networking.firewall.interfaces.eth0.allowedTCPPorts = [ 4949 ];
By building a NixOS live image using Nix, I can easily try a new configuration without modifying my router storage, but I could also use it to ssh into the live system to install NixOS without having to deal with the serial console.