I'm continuing to remove functionality from the kernel, these are the sections in `make menuconfig` that I have cleaned up:
I've enabled `CMDLINE_FORCE`, which always uses the kernel command string defined with `CMDLINE` at compilation time. That makes the `/boot/firmware/cmdline.txt` file obsolete, which I subsequently deleted.
I've disabled a few settings here that I likely want to enable again later on, i.e. branch optimisation and stack protection. Again, the goal in the first run here is to compile a kernel with the least amount of enabled options possible, then re-add important features from there.
All the schedulers are disabled now, will need to check if it makes sense to enable it back later on.
I've measured the total boot time now from powering the RPi on until the login prompt shows, which is now about 14 seconds, taking into account that some of the time is spend on activating USB periphery (currently USB harddisk, mouse, keyboard, webcam, headset). With everything disconnected except the keyboard and harddisk, I measured roughly 13 seconds.
A few settings here have no help text, I may need to go over them again at some point. I've also chosen the simplest SLAB allocator available, which I perhaps want to change later on, although I need to do some reading about it first.
After these changes the boot time is terribly slow now, about 20 seconds. I've set the Slab allocator parameter back to `SLUB`, and the boot time is now back to normal.
Just leaving the basic IPv4 and IPv6 networking capabilities in place and disabling everything else, including wireless. I still end up with enabled `bond0` and `dummy0` devices, which are probably disabled elsewhere.
I completely disabled PCI and MTD support, which is probably not what I want, but let's test it anyways.
I'm getting warnings and a generic `make` error this time:
make[1]: *** [scripts/Makefile.build:500: drivers] Error 2 make: *** [Makefile:2014: .] Error 2
Running `make` verbosely using `V=1` reveals the actual error:
time make V=1 -j4 Image.gz dtbs
[...]
drivers/irqchip/irq-bcm2712-mip.c: In function 'mip_init_domains':
drivers/irqchip/irq-bcm2712-mip.c:222:22: error: implicit declaration of function 'pci_msi_create_irq_domain'; did you mean 'msi_create_irq_domain'? [-Werror=implicit-function-declaration]
222 | msi_domain = pci_msi_create_irq_domain(of_node_to_fwnode(node),
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| msi_create_irq_domain
drivers/irqchip/irq-bcm2712-mip.c:222:20: warning: assignment to 'struct irq_domain *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
222 | msi_domain = pci_msi_create_irq_domain(of_node_to_fwnode(node),
| ^
CC drivers/dma/virt-dma.o
cc1: some warnings being treated as errors
This looks like a missing dependency of a bcm2712 driver on PCI functionality at compile time. Let's try to nail that down.
When searching for `mip` in `make menuconfig` (using the / (slash) key), the first result seems to be the one throwing the error:
Symbol: BCM2712_MIP [=y]
Type : bool
Defined at drivers/irqchip/Kconfig:112
Prompt: Broadcom 2712 MSI-X Interrupt Peripheral support
Depends on: ARM_GIC [=y]
Location:
-> Device Drivers
-> IRQ chip support
(1) -> Broadcom 2712 MSI-X Interrupt Peripheral support (BCM2712_MIP [=y])
Selects: GENERIC_IRQ_CHIP [=y] && IRQ_DOMAIN [=y]
It tells us that this option is enabled (`[=y]`) and lists the dependencies in the `Selects:` line, but no PCI related dependency though.
The error message tells us that it expects the `pci_msi_create_irq_domain` function to be declared, but it isn't, let's `grep` through the kernel sources for where it is declared:
grep -IR pci_msi_create_irq_domain . | grep -v "= pci_msi_create_irq_domain" ./drivers/pci/controller/pci-aardvark.c: pci_msi_create_irq_domain(dev_fwnode(dev), ./drivers/pci/msi/irqdomain.c: * pci_msi_create_irq_domain - Create a MSI interrupt domain ./drivers/pci/msi/irqdomain.c:struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode, ./drivers/pci/msi/irqdomain.c:EXPORT_SYMBOL_GPL(pci_msi_create_irq_domain); ./drivers/irqchip/irq-gic-v3-its-pci-msi.c: if (!pci_msi_create_irq_domain(handle, &its_pci_msi_domain_info, ./drivers/irqchip/irq-armada-370-xp.c: pci_msi_create_irq_domain(of_node_to_fwnode(node), ./.tmp_vmlinux.kallsyms1.syms:ffffffc0084537b0 T pci_msi_create_irq_domain ./include/linux/msi.h:struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode, ./System.map:ffffffc0084537b0 T pci_msi_create_irq_domain ./.tmp_vmlinux.kallsyms2.syms:ffffffc0084537b0 T pci_msi_create_irq_domain
It seems that it is declared in `./drivers/pci/msi/irqdomain.c`, so in order for the `BCM2712_MIP` setting to work we need to enable `PCI support` again in the `Device drivers` page of `make menuconfig`, which automatically enables `CONFIG_PCI_MSI` as well.
The kernel now builds correctly.
I've opened a bug report [1] for this missing dependency declaration.
The kernel now boots normally, but it doesn't detect any USB devices anymore, including the external USB harddisk which contains the kernel sources. I have no choice than to roll back to a previous backup.
I'm starting over again with disabling all settings in Device Drivers except what's needed for booting and network. This was a lot of try and error, and I had to restore from a previous copy once in a while when I ended up with an unbootable system.
To have only the necessary amount of devices shown in `/dev` I directly edited the file `include/uapi/linux/vt.h`, changing `MAX_NR_CONSOLES` from 64 to 8. This option is unfortunately not available via `make menuconfig`.
Additionally I commented out all but the first two `getty` lines in `/etc/inittab`. We don't need more as we're connecting through ssh anyways.
Currently my external harddisk is `ext4` formatted, same as the data partition on the SD card, and although I want to try `f2fs` for the card we still need `ext4` to be compiled in.
After disabling support for `Inotify`, a `systemd` related `udevd` warning popped up in the boot logs, triggering a boot delay of 15s. I found no nice way of getting rid of `/lib/systemd`, so I just renamed it to `/lib/systemd_bak` and that actually works! The error is gone and the console, keyboard input, and network still work, which is all I want at this point, with the side effect of a substantially improved boot time. Now it takes 8s from power on until I see the `getty` prompt! This also means that later on I want to investigate if there are any `udev` alternatives out there, or maybe try patching the package to get rid of everything `systemd`.
I was reading an article [2] about how to speed up USB enumeration and I've added `usb-storage.delay_use=0` to not wait for USB storage to settle at all. I didn't see any difference in boot times though, although the setting is definitely active:
cat /sys/module/usb_storage/parameters/delay_use 0
Will leave it at zero for the time being.
Another thing I wanted to try is to disable IPv6. I've disabled the settings in the kernel and removed all `ipv6` entries from `/etc/hosts`. There's probably much more I can remove later on from `init` scripts etc. This did gain me some simplicity, but the boot time is not affected.
I've disabled everything here, will revisit this later if needed.
Another round of try and error here.
I ended up disabling everything, I have seen no negative impact whatsoever. There's a submenu about arm64 accelerated crypto algorithms, which may be interesting to revisit.
Disabled everything I can.
Disabled everything I can.
[1] GitHub: BCM2712_MIP missing dependency #5890
[2] Using ‘delay_use’ to speed up USB Enumeration
⬅ 6. Reducing the kernel further
➡ 8. Kernel reduction second pass
⬆ The minimal Raspberry 4 project
Created: 24/Jan/2024
Modified: 5/Feb/2024