Thursday, 25. November 2021
[This article has been bi-posted to Gemini and the Web]
The previous article (part 1) on cross-platform package management / package building covered the basics by taking a look at what some of the many difficulties are. It briefly described some common strategies, too.
Cross-platform package building: Pkgsrc vs. Ravenports (1/2)
Part 2 discusses two package building frameworks that were developed to allow for a cross-platform solution. It contains a short introduction of both, a somewhat detailed elaboration of how they compare as well as information on the test scenario and of course the results. It ends with a conclusion.
Pkgsrc, pronounced _package source_, is NetBSD's ports system. Since one of NetBSD's primary goals is portability, it's not much of a surprise that Pkgsrc is also portable across several platforms and architectures. It was originally derived from FreeBSD's ports system but has later been redesigned in a complete rewrite. In a nutshell, you can think of a "port" as a buildsheet for some kind of software. You tell the system that you want it to build PostgreSQL, Firefox or whatever and if there is a port for that, the system knows what to do to accomplish that task.
The individual ports can be used directly by building the software on the host machine or to build the binary packages that can then be installed using a package manager. To build a package like Firefox, a lot of dependencies are required. If they are not already present on the system, they will be automatically built before it (there are ports for those programs or libraries as well).
If you want to know more about this, here are two articles that I wrote a couple of years ago. They provide an introduction to the FreeBSD ports system. If you are unfamiliar with the concept, you probably want to skim over the first one and read it and the second one completely if you find it interesting:
FreeBSD: Building software from ports (1/2) [2017 article]
FreeBSD: Building software from ports (2/2) [2017 article]
Once you understand what FreeBSD's ports tree is, you've got an idea of what Pkgsrc is, too: From the user's perspective it has the same basic purpose.
Ravenports is a much more recent take on the topic of package building. It was conceived by John Marino who had before contributed to the maintenance of thousands of ports in Pkgsrc and FreeBSD ports and who also was the inventor of dports (DragonFly BSD's adaption of FreeBSD's ports). After his attempts to modernize the old ports frameworks had failed, he decided to design a new one based on the lessons learned.
RP is meant to overcome limitations due to design decisions that looked reasonable more than two decades ago but have since proven to be obstacles. Modern-day tooling, high concurrency and a very high level of automation as well as convenient helpers for port maintenance is what it offers over others. It's these features that allow it to reduce the army of porters usually required to maintain several thousand packages to a couple of capable people at most.
It is different from a classical ports system in it not being capable of building and installing software directly on the host system. It _always_ uses a pristine "build jail" for each package in which only the relevant dependencies are installed before the actual build process begins. When the build is completed and the software packaged up, the build jail is torn down. FreeBSD ports and Pkgsrc allow for doing the same by using additional tools. With Ravenports this is the only supported (and built-in) mode of operation: Building packages properly, not installing software directly.
I've used both Pkgsrc and Ravenports for private projects as well as for customers and thus benefited from them. While I like both, I won't hide that I was so impressed by Ravenports that after trying it out I soon became involved in the project. So one could say that I'm biased and you cannot expect a fair comparison. I tried hard to come up with sensible criteria to put both to the test and I've asked other people (like the other two Advance!BSD members who _are_ neutral) for their judgement.
I also deliberately treated Ravenports harshly during the comparison; there's no bonus points from me. On the contrary: I proposed to evaluate Pkgsrc first, because I was under the impression that it would probably fit our project better (being already available for so many platforms and being very mature). It was only when we hit unexpected problems that I decided to go for a somewhat comprehensive evaluation and comparison.
A direct comparison of both candidates is actually pretty difficult to do. You could compare the platforms that both support. Pkgsrc wins this easily. However when it comes to support for _bulk builds_ (building packages from all ports or a considerable subset), that's broken on almost all of those platforms that Pkgsrc supports! And so when it comes to this kind of "fully supported" platforms, Ravenports takes it home.
One of the most obvious things you could compare is package count. Again this is not that simple because you're comparing apples to oranges. Pkgsrc offers so many more packages than Ravenports; but among those are e.g. over 1,400 packages related to TeX - which Ravenports packages differently, ending up with only 15 packages for basically the same thing! Pkgsrc keeps various older compilers and versions of popular interpreters (Python, Ruby) in the tree even when they are no longer supported. Ravenports has a much stricter policy of trying hard to purge ports of software that has officially reached the status of EOL (End Of Life). Neither approach is inherently better than the other - it's a question of what your preferences or requirements are.
On the other hand the number of Pkgsrc packages buildable for the various platforms (as listed below) is the _minimum_ count (hence >=). The reason being that when not bulk-building, all dependencies for packages must be installed on the live system (and are not automatically deinstalled). Some dependencies will block others from being installed. And thus a considerable number of packages will fail to build as a result of it. So while I included the figures here to give an impression on how well each platform is supported, they have no _absolute_ meaning and thus score neutral (total count is among the criteria in the "general" section, though).
So keep in mind that there are multiple problems that do not allow for simply comparing numbers. Think about what all the results _mean_ and what is better _for your use case_. For example in our case a platform that does not support bulk builds is close to being worthless. However even partial support means that fixing bulk builds would be much less effort than having to bring it up on an entirely new platform.
To make the already difficult tests at least compare as best as possible, a fresh VM was created for each system to test. Resources for those VMs were identical of course: 48 GB of RAM, 16 CPU cores (of a 3.0 GHz E5-2623 server) and 1 TB of disk space.
The tests started immediately after _pkgsrc-2021-Q3_ was published in late September and the Ravenports comparison is against the at that time latest released package set. So in this comparison newer additions (namely NetBSD support on Ravenports) are mentioned but treated absent for the score.
We are mostly interested in the *BSD operating systems and thus our focus is on those. As it might be of interest to a lot of people, we've included results for Linux and illumos, too, but gave the respective results a neutral score. The operating systems used in the tests were:
For all criteria tested, a score is awarded. It can be very positive (++), positive (+), neutral (0), negative (-) or very negative (--). All of those added up lead to the final score.
For all tests freshly installed operating systems were used. The installations were done as close to a standard / minimal installation as possible. This is true for DragonFly BSD and FreeBSD.
On NetBSD and OpenBSD all of the package sets were installed, since various X11 ports in Pkgsrc depend on those being present and would fail otherwise. Compared to many Linux distributions this is still a fairly slim installation, but it's not _minimal_ in the true sense of the word and thus should be noted here. On OpenBSD the standard partitioning scheme was also replaced with a simplistic one.
Since Linux does not know the concept of a base system, there's also no base compiler. As one is needed to bootstrap Pkgsrc (Ravenports is self-contained in this case), 'yum groupinstall "Development Tools"' was used to make the system capable of building software. On OmniOSce there is also no base compiler; 'pkg install gcc10' took care of making the system fit to continue on.
With all systems the first step after preparing the environment - i.e. making a compiler available if it's missing as well as configuring SSH - was to fetch pkgsrc-2021Q3.tar.xz and decompress it to /usr. Then the typical bootstrap was run (except on NetBSD where this is not required) and the PATH environment variable modified to include /usr/pkg/sbin and /usr/pkg/bin.
The next step was always to build _pkgtools/mksandbox_ as well as _pkgtools/pbulk_ and to attempt a bulk build. Since this failed for one reason or another in almost all cases, plan b was to at least run (b)make in /usr/pkgsrc, letting the system build as many packages as possible. A list of packages was created and the build started over after declaring all the licenses acceptable (otherwise quite a number of packages that _can_ be build are skipped).
With Ravenports it's doing the bootstrap as explained in the documentation for each supported OS and eventually running 'ravenadm build-everything'. The results for Solaris / illumos were taken from a list of the packages in the official repository at that time. The reason is that I did not have enough spare time to try and install the specific old version of Solaris (from 2009!) required in the VM.
Here are the results of what is likely one of the more comprehensive evaluations of Pkgsrc ever done. Package building on all the platforms took almost two months' time. The various criteria have been grouped together into the "OS support", "General" and "Technical evaluation" groups. Unfortunately the tables did not fit well either in Gemtext or in blog-style HTML and are therefore available as images.
Pkgsrc vs. Ravenports evaluation results (PNG)
As mentioned above, keep in mind that the amount of packages that built with Pkgsrc is the _minimum_ count of buildable ones. Where the numbers look particularly low, that's usually due to a very basic dependency package being broken, cutting off a substantial part of the tree (e.g. Python on DragonFly BSD and pkgconf on illumos).
Disregarding Linux and illumos support (as those are out of scope for our use case), both score almost evenly. Pkgsrc comes out very slightly ahead [1:0] of Ravenports due to it's excellent integration into one of our main platforms (NetBSD).
If both Linux and illumos are to be taken into account, too, Pkgsrc's score remains the same as two additional supported platforms with two failed bulk builds neutralize each other. Ravenports however gains two points for Linux support including bulk builds and one more point for illumos support (since it can bulk-build _for_ but not _on_ illumos, I'd give a 0 instead of a + rating in this case). This would place Ravenports ahead [1:3].
Pkgsrc vs. Ravenports evaluation results (PNG)
While the former section ended up very surprising for me (having taken for granted that full bulk build support was the rule rather than the exception!), this one is very much what you'd expect.
Pkgsrc, being an established project with a long history and a lot of contributors, can show its strengths to the fullest here. The newcomer does not perform too badly actually, but despite getting ahead in one test (package freshness) it doesn't stand the slightest chance against the veteran. Pkgsrc wins with a huge lead [12:-1] and Ravenports leaves the field beaten and bruised in crushing defeat.
Pkgsrc vs. Ravenports evaluation results (PNG)
While it looked like the match was already over after the previous round, this one is where being slow and old-fashioned despite being mature puts you at risk. So here's where the youngster can shine with its modern-day approaches and new tricks! I expected Ravenports to clearly beat Pkgsrc this time. It did - but it even managed to make this round look like roles being inverted compared to the previous one...
This time it's Pkgsrc that can win only one test (popularity of languages used). Ravenports however manages to bring out the big guns and eventually leaves Pkgsrc utterly destroyed by an even higher margin [-3:11] than it lost the previous round.
Pkgsrc vs. Ravenports evaluation results (PNG)
For me this duel between Pkgsrc and Ravenports has been more interesting than I would have thought. After Pkgsrc messed up the jump start that I was expecting, things began to look like a more evenly matched comparison. I still would have lost my money as I'd have bet Pkgsrc to win at least by a tiny margin.
It's a bit weird but at the same time probably fitting to see that both contenders have such a diverse set of strengths and weaknesses - and that their comparison eventually even ended [10:10] in a draw!
But that's kind of delivering old news, since I'm writing about the comparison as of late September. In October, Ravenports gained official support for NetBSD, which would shift the results in Ravenports favor [10:14]. It could provide more insights to give Pkgsrc a try again when the Q4 release happens. I haven't decided, yet. Maybe I'll do that.
As stated above, Pkgsrc is NetBSD's ports system. Our tests show that you need to stress this: It's not simply a statement about its origin. It's a statement about its main _purpose_.
NetBSD is where Pkgsrc development takes place. Other operating systems are invited to participate, but maintaining excellent support for any other OS is a _huge_ task. Before I took a closer look at this, I didn't know that even Joyent (the company behind SmartOS which uses Pkgsrc as its official means of package management) do maintain a downstream fork of Pkgsrc:
Pkgsrc downstream fork (GitHub)
They have employees who are doing paid work on Pkgsrc - and obviously they still do not manage to keep upstream Pkgsrc in good enough shape for themselves (leading to the less than impressive number of packages that currently work on illumos).
Ravenports is more like the new kid on the block. It has proven to be a serious effort (being in active development for over 5 years now) but has yet to find its niche. Despite it being technically superior in many ways, it's held back by the very low number of contributors.
Both solutions can be used for cross-platform package management. So which one should you choose? While they can be used for the same scenario, their goals are very different. Pkgsrc is meant to run even on the most exotic, limited and obsolete platforms and is willing to pay a price for that. Ravenports in contrast focuses entirely on modern hardware and can take advantage of their superior capabilities. Therefore there's no "one size fits all" in this case, it really depends on what you want to do.
Need to build ports directly? Pkgsrc's the answer here. Have to install software on AIX? Use Pkgsrc. Running a SPARC64 machine? Pkgsrc again. If however you need the latest software versions, go for Ravenports. Are reliable package builds across multiple platforms what you're looking for? Give Ravenports a try. Don't want to invest into build clusters for acceptable performance? Go with Ravenports if you can.
Consider your use case, compile a table of requirements. Then see which solution fits you best. But whichever you pick, always consider that you might have to do some work of your own (and please contribute it back!).
