As I have more than one Raspberry Pi 4 here, I was in parallel re-installing one of them, which usually serves as my git repository and file server, running Debian Linux.
I though that I might be able to figure out a working kernel configuration better using a minimal Debian image.
So that's what I did, I downloaded the image [1] and wrote it to the SD-Card, then I've purged every dispensable Debian package I could find, and I ended up with an ultra minimal system running an `6.1.0-17-arm64` kernel with just about 250 installed packages.
I also replaced the default `systemd` init system with `sysv`, by first adding the `sysv` packages, rebooting, and then removing `systemd`.
I then proceeded to install the minimum amount of packages needed to compile the kernel:
apt-get install --no-install-recommends libncurses-dev linux-headers-arm64 libc-dev-bin linux-libc-dev cpp git bc bison flex libssl-dev make
And maybe a few more packages, where I can't read from the apt logs right now if I installed them on purpose, or if they were dependencies for one of the above packages.
In any case, it should be easy to figure out which package is missing when building the kernel and analysing the error messages. I.e. if it complains about a header file which couldn't be found, you can search for it using `apt-file`:
apt-get install --no-install-recommends apt-file apt-file update apt-file search sys/types.h [...] libc6-dev: /usr/include/aarch64-linux-gnu/sys/types.h [...]
In this particular case there are way more results coming back, but we know we're using `libc6` currently, not `dietlibc` or `musl` or anything else, although I might come back to them later on for static compilation.
So, for above example we would need to install the `libc6-dev` package.
We then go ahead and follow the official Raspberry guide for kernel compilation first [2], and modify things afterwards.
We clone the kernel repository:
git clone --depth=1 --branch <branch> https://github.com/raspberrypi/linux
And apply the default configuration for Raspberry Pi 4B, which uses the Broadcom BCM2711 System on Chip (SoC):
cd linux KERNEL=kernel8 make bcm2711_defconfig
The configuration for the build is stored in the `.config` file, I usually take a copy once in a while to be able to go back to a previous state if needed.
We customise the version string of the kernel by editing the following line in `.config`:
CONFIG_LOCALVERSION="-v8-mycustombuild"
We can then customise the kernel using `make menuconfig`.
First thing I am trying is to disable the modules entirely by unchecking `Enable loadable module support` in the main menu.
Also, we want to get rid of the initrd, which is only needed if modules need to be loaded before boot, or if filesystem encryption is used, amongst a few other things. For this we uncheck `Initial RAM filesystem and RAM disk (initramfs/initrd) support` in the `General setup` page.
I then disabled a whole plethora of configuration options which I think are not needed. If we disabled to much we will check the error messages later and fix it.
Check the footnote [3] for my `.config` at this point.
For this to work we apparently need to avoid using UUIDs, so I changed my `/etc/fstab` file to this:
/dev/mmcblk0p2 / ext4 rw,noatime,nodiratime 0 1 /dev/mmcblk0p1 /boot/firmware vfat rw,noatime 0 2
To the next step we need to understand that Debian mounts the boot partition of the SD card image, which is in `vfat` format, to `/boot/firmware` at boot time. You also see that in the above `fstab`. This means that we can easily edit all files from the boot partition using that mountpoint.
We adjust the `root` parameter in the `/boot/firmware/cmdline.txt` file, which contains the kernel parameters used at boot time, mine looks like this now:
console=tty0 console=ttyS1,115200 rw fsck.repair=yes net.ifnames=0 rootwait root=/dev/mmcblk0p2
Let's see if that actually works!
We go back to the git repository holding the Linux kernel source and compile:
time make -j4 Image.gz dtbs
The `time` command is not needed but I usually want to know how compile times perform after modifying configuration parameters.
Here a comparison of compile times:
You can force a fresh compile by removing all object files with `make clean`, if you measure time without the `clean` it may be quicker, as most modules are compiled already.
I then proceed to copy all files to where they belong, taking a backup of the existing files first so I can easily go back if something goes wrong, which it certainly will!
I do not install all of the firmware files, as they are for different models than ours. And even the final list I came up with contains a few files which are not needed. I'll remove them later.
Currently my installation script looks like this, which you obviously need to adjust according to your naming scheme and Source location:
#!/bin/bash KERNEL=v8-callistix cd ~/Sources/github.com/raspberrypi/linux/ # boot partition sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb /boot/firmware sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-400.dtb /boot/firmware sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-cm4-io.dtb /boot/firmware sudo cp arch/arm64/boot/dts/broadcom/bcm2837*.dtb /boot/firmware sudo cp arch/arm64/boot/Image.gz /boot/firmware/vmlinuz-$KERNEL # system partition sudo cp arch/arm64/boot/Image.gz /boot/vmlinuz-$KERNEL sudo cp System.map /boot/System.map-$KERNEL sudo rm -f /vmlinuz sudo ln -s boot/vmlinuz-$KERNEL /vmlinuz
The Raspberry firmware needs to be told where to find the kernel by editing the `/boot/firmware/config.txt` file, mine looks like this now:
arm_64bit=1 enable_uart=1 upstream_kernel=1 kernel=vmlinuz-v8-callistix
Some of the options can be removed, but we'll look into that later.
We `reboot` the Pi and see what happens.
It seems to boot much faster, but we run into an `UNEXPECTED INCONSISTENCY` reported by `fsck` for my external USB harddisk, while dropping me into maintenance mode. I'm sure that's nothing to be worried about. :)
Now, the `ext4` filesystem actually mounts fine when doing a `mount -a`, but it reports warnings: `mounting fs with errors`, so we unmount it again and run a manual `fsck`. Btw. it's absolutely crucial to unmount the filesytem first before doing an `fsck`! I can't stress this enough!
In my case, `fsck` marked the filesytem as clean without finding anything, so we're good to reboot.
And luckily this time it works! And we get awesome boot time improvements! The Raspberry Pi "rainbow" shows only for a fraction of a second before already showing the boot logs.
Boot times reported by boot log timestamps before and after:
That's a great win and I'll call it for today, next time we'll do more cleanup and I'll throw everything out of the kernel which is not indispensable.
We can then add features back in at a later point if we need them.
[2] Raspberry Pi Documentation: The Linux kernel
⬅ 4. EEPROM update and configuration
➡ 6. Reducing the kernel further
⬆ The minimal Raspberry 4 project
Created: 15/Jan/2024
Modified: 5/Feb/2024