2024-01-09 | #servers #python | @Acidus
I'm not a server guy. For me, servers are just a means to an end: building and running something cool. For that matter, I'm really not an engineer or developer. Writing software is just a mechanism for me to create cool things. While I appreciate aspects of the craft, it's largely not important to me. This approach usually doesn't cause problems and enables me to mostly ignore the grinding treadmill that is modern software development.
Until it doesn't.
Beside a fury of activity to get it setup, I had let my little VPS linux server that runs this capsule mostly go fallow. Updating CGI's or content on it is handled via a Makefile on local machines. I only ssh into it to look at server logs now and then. While I would run `apt update` and `apt upgrade` when I was in there, it certainly wasn't often enough to be a regular thing. I probably should have done something about this with a cron job shouldn't I? 🫤
Well, turns out I've been running a non-LTS version of Ubuntu, which eventually fell into end-of-life. So upgrades failed. And running `sudo do-release-upgrade` also failed because of how out-of-date my version was. While it isn't yet an issue, it could easily become one. Out-of-date software is a footgun that is the backbone of most modern computer security practices.
Now, if I were a server/software guy, I'm sure I'd have some infra-as-code system setup that would make deploying a new instance with an updated base image a snap. But I'm not, so I don't. And so I needed to take an Ubuntu 22.10 server and bring it up-to-date.
Luckily Ubuntu has some pretty great docs, and eventually I found this:
This guide explains how to upgrade an End of Life (EOL) Ubuntu release to a supported system.
Ubuntu Documentation: EOLUpgrades
Perfect! I was running 22.10, which was EOL-ed in July 2023. I really needed to get to a Long Term Support (LTS) release, since those last 5 years, which would allow me to go back to neglecting server stuff for a nice long time. Ubuntu 22.04 LTS was the most recent LTS. While this version had been updated recently, I wasn't sure about moving from 22.10 to 22.04 LTS, since that's technically a lower version number. Instead I decided to go to the latest supported version 23.10 (Mantic Minotaur).
I checked my /etc/apt/sources.list and then I followed the "Run the upgrade" instructions
Ubuntu Documentation: Run the Upgrade
Upgrading directly from 22.10 (Kinetic Kudu) to 23.10 is not supported. So instead I needed to upgrade 22.10 to 23.04 and then upgrade 23.04 to 23.10. The instructions to do that are here:
Ubuntu Documentation: Unsupported upgrades
Basically, this involved directly downloading the upgrade script for a specific version of Ubuntu and running it. I did this to get upgrade to 23.04, then again to upgrade to 23.10.
Success!
The upgrade installer alerted me to a few warnings along the way. First was to ensure that changes I had made to the ssh config weren't clobber by new default configs. When setting up the capsule I made a few changes like disallowing root and requiring certificates. The installer showed me a diff so I could ensure that my custom config would continue to work. I got the same warning about some small crontab changes I had made that run clean-up jobs. This was a nice touch and gave me more confidence that I wasn't going to spend the rest of the day fixing a terrible terrible mistake.
My knowledge of Python and it's ecosystem is kind of like my knowledge of ffmpeg command line arguments. Every few years, for a period of a few days, I'm doing something complex and I end up I self-teaching myself enough to be a Jedi, only to then immediately forget what I've learned, until the ignorance cycle repeats again in a year or two. For Python specifically:
While I'm sure these answers are interesting to someone, perhaps even to me, I largely don't care. It's all debris between me and what I'm trying to accomplish: execute a python program reliably, so I can benefit from the program.
(Yes, I recognize the irony that all of that is true and yet I have also submitted at PR to the md2gemini python program and run my variant often. Life is weird)
Anyway, while upgrading to Ubuntu 23.10, I notice that the Jetforce service is no longer starting successfully. Using `journalctl` to show the logs for the jetforce service shows a "can't find module jetforce"-type error. So somehow the location of the jetforce module is missing / a link got broken.
Following @mozz's instructions on the Jetforce GitHub page involved running `pip` (`pip`? `pip3`? What's my alias again? 🤷), and it yielded an error I hadn't seen before:
$ pip install
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
Oh fun! A new pip, `pipx`. And I guess I can add a Python "virtual environment" to my list of things I don't know. But funny enough Jetforce's documentation does talk about how to "install into a python virtual environment":
# Create a project directory somewhere $ mkdir /opt/jetforce # Activate a virtual environment and install jetforce $ python3 -m virtualenv /opt/jetforce/venv $ source /opt/jetforce/venv/bin/activate $ pip install jetforce # The launch script will be installed here $ /opt/jetforce/venv/bin/jetforce
This... seems to work. For both my current shell, and logging back in. So I update my jetforce.service definition to point to the new `/opt/jetforce/venv/bin/jetforce` executable and my capsule is back. The debris is clear, and since I wasn't piping a wget-ed shell script directly to sudo, I probably haven't reduced my security posture too badly, so I can now promptly forget this crazy Python world yet again.
Note to future Acidus:
Hi there handsome! How are you doing? Let me guess: You messed something up and now you're having Python problems again aren't you? Did you ever learn what `pipx` was? No? oh well, cool cool. I don't really know how much python you actually know now, but hopefully this ☝️ helps you. Stay classy.
Where was I? oh yeah, Ubuntu. Non-LTS versions of Ubuntu get updates for less than a year, which seems wild to me:
So I really want to get on an LTS version and go back to neglecting things again. April 2024 is when the next new Ubuntu LTS release comes out. Now that I'm on the latest version, it will have support through July 2024. So I set a reminder for May to update to LTS. With any luck I can then forget all this for another 5 years.