💾 Archived View for shit.cx › tech › devenv › 2022-04-01-a-serial-console captured on 2023-04-19 at 22:29:01. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

        .     *  ⠈       +                ⠈
                   +  ──╕   ○    .   ⠈           ⠐
   ●     .           ╒═╕╞═╕ ╕ ╪═        *               .
                     ╘═╕│ │ │ │  .cx            +
           .     ....╘═╛╘ ╘ ╘ ╘═ ....:      ⠐        .
                 .               *                ⠐        .

A Serial Console

2022-04-01T21:48

I try to log into my devbox. SSH is taking a while. Hmm. That's weird. Is my network up? Yes. Can I ping the WireGuard address? No. Can I ping the host directly? Nope. Hmm. Maybe it crashed. I walk to the machine and press the reset button. I wait for ping to stop the sequence of "connection timed out" messages. This is taking a while.

My devbox is headless. I power it off again to install a graphics card and connect a monitor and keyboard to see what is happening.

It turns out that removing NetworkManager was a bad idea. It isn't some random GUI thing, it actually manages the network. I deleted it the previous day because it was using 4% of my systems memory. It coasted through the remainder of the day just fine giving me a false sense of security that it wasn't needed. But then sometime during the night the network stopped. Fortunately, I remembered removing the package and its name. With the hassle of downloading a few debs and shuttling them across on a USB stick, I restored the network. But that isn't the point of this post. The point is that I need to make it easier to recover my machine when I lose the network.

This was a frequent issue with my previous machine. It would sometimes crash and I had no idea what had happened or why. It eventually got so bad that I replaced it. My current machine has been perfectly stable, but still, I sometimes need console access. I've known this for months. There's been Serial to USB cable sitting on my shelf waiting for me to make it work. Well, the time has come.

My machine is running Debian Bullseye, but these instructions should be generic enough to work on any distro using systemd. You can put a console on ttyS0 with this:

# ln -s /lib/systemd/system/serial-getty@.service /etc/systemd/system/getty.target.wants/serial-getty@ttyS0.service
# systemctl enable serial-getty@ttyS0.service
# systemctl start serial-getty@ttyS0.service

I checked `ps` and found a new command:

/sbin/agetty -o -p -- \u --keep-baud 115200,57600,38400,9600 ttyS0 vt220

I first tried to log in with screen. I could, but only at 9600 baud. That's so slow that it takes seconds to print a full screen of text.

I then checked that colours worked. They didn't. For context, the TERM environment variable is used by applications to know the capabilities of the connected terminal or terminal emulator. In this case, it's set to support the escape sequences of the DEC VT220.¹ That's a bit unusual. I expected it to be vt100. I checked the value of ${TERM} in the console and found it set to vt220 as was configured. If I manually change it to vt100, `ls` becomes colourful again.

I had a wild goose chase tracking down what was setting TERM to vt220. Nothing in `/etc` mentioned it. I eventually find a GitHub Issue on the systemd project that goes into the details.² From that, I worked out that my system doesn't know what to do when TERM is vt220 because I hadn't installed terminfo. After installing terminfo, the colour came back.

The issue explains that xterm is a superset of vt220 which is a superset of vt100. The systemd project chose vt220 as the default because it's nearly as good as a xterm but is also supported by some real serial terminals.

In case you're wondering, I didn't want to just hack the systemd command to force it to do what I wanted. Sprinkling magic config around makes the system really hard to maintain and upgrades upgrades become risky.

The other problem was the speed. I messed around with various incantations of the agetty command but nothing I did would cause it to run faster than 9600 baud.

I RTFM and found this:

The default for serial terminals is keep the current baud rate (see --keep-baud) and if unsuccessful then default to '9600'.

and:

-s, --keep-baud: Try to keep the existing baud rate. The baud rates from the command line are used when agetty receives a BREAK character. If another baud rates specified then the original baud rate is also saved to the end of the wanted baud rates list. This can be used to return to the original baud rate after unexpected BREAKs.

I still don't know why I couldn't force it to go faster, but it does say that agetty will preserve the baud set by something else. I was planing to configure Grub to work over serial next. I guessed maybe if I did that first, those settings would carry over to agetty.

So I added this config to `/etc/default/grub`:

GRUB_CMDLINE_LINUX='console=tty0 console=ttyS0,115200n8'
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

Then I ran an `update-grub` and rebooted.

It worked! My serial port kicked in at 115200 baud as soon as Grub started and it persisted all the way through to my login prompt.

So there we have it. A bit of an adventure, but we have a fully-functioning serial console for a headless Debian Bullseye machine.

1. VT220

2. systemd sets incorrect terminal type for serial terminals

---