Here I'm republishing an old blog post of mine originally from February 2020. The article has been slightly improved.
:: This was originally my January article that I decided to postpone for another one. So it's really the February one - except for myself not getting around to publishing it before August due to... well, the world going crazy. So this is still the January state of affairs for the most part.
:: On the plus side of all the havoc: I went over the article again and improved it. I intend to publish the other articles that I missed out one by one.
After getting my hands on an old SPARC64-based SUN machine, I've written a couple of posts about the actual hardware, installing and running OpenBSD as well as an illumos distro on it. I've also hinted that I wanted to try out my favorite OS on it, too: FreeBSD.
A SPARC in the night - SunFire v100 exploration (September 2019)
OpenBSD on SPARC64 (6.0 to 6.5) (October 2019)
Running OpenBSD on SPARC64 (HTTPd, packages, patching, X11, ...) (October 2019)
Illumos (v9os) on SPARC64 SunFire v100 (October 2019)
I'm coming pretty late to the party, because SPARC64 support in FreeBSD is apparently doomed: After the POWER platform made the switch to a LLVM/Clang-based toolchain, SPARC64 is one of the last ones that still uses the ancient GCC 4.2-based toolchain that the project wants to finally get rid off (it has already happened as I was writing this - looks like the firm plan was not so firm after all, since they killed it off early). And compared to the other platforms it has seen not too much love in recent times... SPARC64 being a great platform, I'd be quite sad to see it go. But before that happens let's see what the current status is and what would need to be done if it were to survive, shall we?
The Ultra SPARC platform is being dropped
The FreeBSD project provides ISO images for various platforms. At the time of my first attempt I downloaded a 12.0-RELEASE SPARC64 iso. Since my machine cannot boot from USB I can't use the virtual CD drive that I typically use. So I went old-school and burned a CD. Then I prepared everything and turned the machine on. It started booting but before even reaching the installer or anything, I witness something completely unexpected - here's the last few lines from the serial console:
nexus0: type unknown (no driver attached)
rtc0: at port 0x70-0x71 pnpid PNP0b00 on isa0
rtc0: registered as a time-of-day clock, resolution 1.000000s
uart0: console (9600,n,8,1)> at port 0x3f8-0x3ff irq 43 pnpid PNP0501 on isa0
uart1: at port 0x2e8-0x2ef irq 43 pnpid PNP0501 on isa0
Timecounter "tick" frequency 548000000 Hz quality 1000
Event timer "tick" frequency 548000000 Hz quality 1000
Timecounters tick every 1.000 msec
Power Failure Detected: Shutting down NOW.
Wow, what's that? FreeBSD thinks that the machine has a damaged PSU and immediately shuts down, giving me no chance to actually do anything. Now that was less than impressive... This was when I decided to try out OpenBSD instead. That worked well and the machine ran for a couple of days, compiling OpenBSD. So the power supply is definitely fine.
I did a little research and it seems that some models fire an interrupt that means "PSU is damaged" with other models. OpenBSD seems to know this for my machine and ignores it, but FreeBSD doesn't.
Ok, so what to do now? We have a defunct OS but we also have a string to look for. It's not hard to find it in _/sys/sparc64/pci/psycho.c_. I don't know C and thus cannot implement new features or do real debugging of any sort. But I can comment stuff out. And since this is a false alarm that should be sufficient. So I get rid of the check that ruins the boot for me and save.
Now, one of the things that I love about FreeBSD is its build system. It's so easy to build the OS - or to even cross-build it! It's as simple as doing this:
cd /usr/src
make TARGET=sparc64 TARGET_ARCH=sparc64 buildworld buildkernel
This cross-builds FreeBSD for SPARC64. But how to get it onto my SUN machine? Fortunately there is a pretty interesting directory in _/usr/src/release_. The Makefile and scripts there allow for extremely easy creation of installation media. So it should just be two more commands and I'd end up with an ISO image that contains my modified FreeBSD. Let's give it a try:
cd /usr/src/release
make cdrom NOPKG=1 NOPORTS=1 NOSRC=1 NODOC=1 TARGET=sparc64 TARGET_ARCH=sparc64
This means: Build a minimal install CDROM image, leaving out any packages, the ports tree as well as the source and documentation distribution sets. But what is this?
gpart: scheme 'VTOC8': Invalid argument
*** Error code 1
The manpage for gpart reveals the cause of the problem: By default VTOC8 is not supported on amd64 kernels (which is a perfectly reasonable choice). However solving the problem is as easy as adding _GEOM_PART_VTOC8_ to the kernel options and building a custom kernel on my amd64 build machine.
If you've never worked with a custom kernel (as it's far less often required than it was in the past), have a look in _/usr/src/sys/amd64/conf_. You'll find a file called _GENERIC_ there. It holds all the definitions for the standard kernel. While you could just edit this file it's bad practice to do so. At least copy GENERIC to a new file:
cp /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/GENERIC-VTOC
Next let's change the _ident_ line to GENERIC-VTOC, too and add the option GEOM_PART_VTOC8 as the gpart manpage suggests. That's all, now I can rebuild the kernel with
make -C /usr/src kernel KERNCONF=GENERIC-VTOC
This command line builds and installs the new kernel for me, I just have to reboot afterwards. And not too much later I have an ISO to burn that would finally let me install FreeBSD on the machine!
The first surprise for me: The installer is missing the option to install on ZFS as I have become used to. Looks like ZFS can be used, but the booting facility for sparc64 does not support booting from ZFS.
There are no pre-built packages available for SPARC64. There used to be up until about a year ago, but now it's building everything from ports.
No more FreeBSD packages for SPARC64
After building _tmux_ I decided to upgrade the system. Of course there's no way to use _freebsd-update_, so it's also compiling from source. I gave _-CURRENT_ a try, but the old GCC 4.2 in base cannot build that. Eventually I settled on _12-STABLE_. Compiling it took many hours, but it worked without problems.
After rebooting, however, my system doesn't come back up. Uh-oh. What's happening here? I connect via the serial console - and see that I forgot to patch psycho.c... Fortunately I can just boot _kernel.old_ and be good again. While editing the file again, I notice that I did unnecessary work before: There's a sysctl that can turn the automatic shutdown off! So, yes, it's just a simple _hw.psycho.powerfail=0_ in the loader.conf and the unmodified stable kernel boots. If I had entered _set hw.psycho.powerfail=0_ at the OK prompt in the loader, I wouldn't even had to roll my own ISO. But I like exploring how FreeBSD works, so that extra step was not so bad after all.
Now having a pretty fresh system, I go back to exploring ports again - and run into problems. The default version of GCC on FreeBSD is currently 9.2, but that version is broken on sparc64. GCC8, GCC7 and GCC48 work and can be used to compile software (the latter needs the port edited, though, as sparc64 is not listed as a supported platform even though the port is fine). GCC6 is also broken after the latest minor release.
This is pretty unfortunate since it means that most software that requires a newer GCC to compile cannot be built easily. Sure, I could mess with the ports infrastructure to make it use GCC8 instead, but I'd rather investigate something else.
While the PPC platform recently moved to the LLVM/Clang-based toolchain, SPARC64 is still stuck with the ancient GCC in base as mentioned before. For such projects, FreeBSD has created a path to move forward: Using an external toolchain as the base compiler.
Have a look in _/usr/ports/base_. You'll find the two ports _binutils_ and _gcc6_ there. This wiki page gives an overview of how many steps the various platforms have already made:
So I built the binutils and gcc packages, installed them - and broke my system. As I found out, the base ports actually _replace_ said component in base. So after installing, it's no longer binutils 2.17.50, but the much more recent 2.33.1 and gcc 6.5.0 instead of 4.2.1. The latter however is broken... So time to make installworld from my -STABLE source again to fix my system.
What to do next? Well, -CURRENT is where the exciting stuff happens, so I've got to get that onto my machine somehow! The native way is blocked, as I have no idea now to fix the compiler. But what about cross-compiling for now?
There's a package with the name _sparc64-xtoolchain-gcc_ available for my amd64 machine, so installing the cross-toolchain is about as easy as it gets. Let's try to build FreeBSD with it!
It takes a while but of course finishes much more quickly compared to a native build on the old SUN hardware. Ok, next step is preparing a release for it, so that I can try it out. Unfortunately trying to do so leads to this:
--- libc.a ---
/usr/local/sparc64-unknown-freebsd13.0/bin/ranlib -D libc.a
--- libc.so.7.full ---
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.
/usr/local/bin/sparc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.31.1 assertion fail elfxx-sparc.c:765
*** [libc.so.7.full] Error code 1
Ouch! What is this? It looks like the linker is broken! Time to fix it, eh? Except when you're not a C programmer, what do you do now? After a moment of consideration I decided that this might be well beyond my skills - but sometimes you're lucky. What if the change that broke it was rather simple to understand? You don't know until you try, so here we go...
There must have been a time where things worked, so I start by testing a build with an older version of Binutils: Using 2.28 yields a successful build with GCC 9 (gtest disabled). Same thing for 2.30. Using 2.31 however leads to the known linker problem. Aha, so the error was introduced during the development of Binutils 2.31. That's good to know.
So now it was time to clone the git repo (I don't quite understand why they put binutils and gdb into the same one but never mind) and dig into it. It took me a while to figure out exactly which commit broke things but was eventually able to find the culprit. It was a not a simple commit as I had hoped. Cleanup of old cruft had been done (like phasing out of a.out and COFF support) und that somehow also broke ELF. I couldn't see why and so I started taking the commit apart.
Finally I traced the problem to a change in gas (the GNU assembler) where the cleanup lead to an oversimplification with targets that made gas on FreeBSD behave as if it was NetBSD! I provided the patch on the FreeBSD bugtracker and it was accepted. I posted my patch to the upstream mailing list for Binutils, too and a couple of days later Alan Modra who did the change back then got back to me after having merged a fix.
FreeBSD bug tracker page for linker issue
GNU Binutils mailing list entry
Once the patch hit the ports tree I was able to cross-compile a -CURRENT SPARC64 kernel on AMD64 out of the box without any need for patching. Yay!
Trying to boot up the kernel resulted in a panic quite early in the boot, but after getting this far I didn't want to just give up now. So I filed another bug report and the problem was in fact solved really quickly. With that fix the kernel was able to boot further - only to hit another panic. Fortunately the second one was sorted out almost instantly as well on the same bug report:
FreeBSD bug tracker page for kernel issue
Having a working -CURRENT kernel booted up felt just great. But what about the userland? It was also broken in multiple ways and did not compile. While it took me days to figure out all the breaking commits, reverting or otherwise fixing them for testing purposes, I ended up with a patched userland that did successfully cross-compile. The downside of things was that every single resulting binary that I tried out would just crash... At that point I hit a dead end. I submitted one final bug report but as nobody had any suggestions for me this was the game over point.
FreeBSD bug tracker page for userland issue
When I had learned that FreeBSD was about to drop SPARC64 support I hoped that I could step in and help in keeping it alive. This proved to be an illusion as obviously nobody with a deeper technical understanding than me had any real interest in that platform. I admit being deeply disappointed in January when SPARC64 was dropped. Not because it was dropped, but mainly because it was dropped early. Maybe a bit more time would have helped to keep it in, maybe it wouldn't. We'll never know.
But it's probably a good thing that I didn't get around to publish this article until months after the events. I'd like to especially thank Ed Maste, Mark Johnston and Baptiste Daroussin who merged fixes for a most obviously doomed platform as well as other people from the team who were helpful about it until the end. FreeBSD on SPARC64 failed to survive until 13.0-RELEASE, but the efforts that were made show how much of a professionally developed OS FreeBSD is.
So let's put SPARC64 to rest in dignity. The team has stated that support over the lifespan of 11-STABLE and 12-STABLE will be done as "best effort" work since it's gone in HEAD. Especially 12 will live for some time and I'll try to continue doing things with my SunFire until the curtain falls some time after 12.x.