Here I'm republishing an old blog post of mine originally from May 2018. The article has been slightly improved.
A little rant first: Many thanks to the EU (and all the people who decide on topics related to tech without having any idea on how tech stuff actually works). Their GDPR is the reason for me having been really occupied with work this month! Email being a topic that I'm teaching myself while writing the series of posts about it, I have to get back to it as time permits. This means that for May I'm going to write about a topic that I'm more familiar with.
I've written about package management before, telling a bit about the history of it and then focusing on how package management is done on FreeBSD:
The history of *nix package management (2017)
FreeBSD package management with Pkg (1/2) (2017)
FreeBSD package management with Pkg (2/2) (2017)
The benefits of package management are so obvious that I don't see any reason not to content myself with just touching them:
Package management makes work put into building software _re-usable_. It helps you to _install_ software and to keep it _up to date_. It makes it very easy to _remove_ things in a _clean_ manner. And package management provides a _trusted source_ for your software needs. Think about it for just a moment and you'll come up with more benefits.
But let's take a look at the same topic from a different angle. What do we actually _require_ our package systems to do? What features are necessary? While this may sound like a rather similar question, I assure you that it's much less boring. Why? Because we're looking at what we _need_ - and it's very much possible that the outcome actually will be: No, we're not using the right tool!
Yes, we need package management, obviously. While there's this strange, overly colorful OS that cannot even get the slashes in directories right, we can easily dismiss that. We're talking *nix here, anyway!
Ok, ok, there's OmniOS with its _KYSTY_ policy. That stands for "keep your software to yourself" and is how the users of said OS describe the fact that there's _no_ official packages available for it. While it's probably safe to assume that the common kiddies on the web don't know their way around on Solaris, I'm still not entirely convinced that this is an approach to recommend.
Going down that road is a pretty bold move, though. Of course it's possible to manage your software stack properly. With a lot of machines and a lot of needed programs this will however turn into an abundance of work (maybe there are companies out there who enjoy paying highly qualified staff to carefully maintain software while others rarely spend more than a couple of minutes per day to keep their stuff up-to-date).
Also if you're a genius who uses the method that's called "It's all in my head!" in the _Linux from Scratch_ book, I'm not going to argue against it (except that this is eventually going to fail when you have to hand things over to a mere mortal when you're leaving).
But enough of those really special corner cases. Let's discuss what we actually require our package systems to provide! And let's do so from the perspective not of a hobby admin but from a business-orientated one. There are three things that are essential and covered by just about any package system.
One of the major requirements we have today is that package management needs to be _easy to use_. Yes, building and installing software from source is usually easy enough on *nix today. However figuring out which configure options to use isn't. Build one package without some feature and you might notice much later that it's actually needed after all. Or even find that you compiled something in that's getting in the way of something else later! Avoiding this means having to do some planning.
Reading (and understanding!) the output of ./configure --help probably isn't something you're going to entrust the newly employed junior admin with. Asking that person to just install mysql on the new server will probably be ok, though. Especially since package managers will usually handle dependencies, too.
Making use of package management means that somebody else (the package maintainer) has already thought about how the software will be used in most of the cases. For you this means that not having to hire and pay senior admins for work that can be done by a junior in your organization, too.
Time is money and while "compiling!" is a perfectly acceptable excuse for a developer, it shouldn't be for an admin who is asked why the web server still wasn't deployed on the new system.
Compiling takes time and uses resources. Even if your staff uses terminal multiplexers (which they should), thus being able to compile stuff on various systems at the same time, customers usually want software available when they call - and not two hours later (because the admin was a bit confused with the twenty-something tmux sessions and got stuck with one task while a lot of the other compile jobs have been finished ages ago).
Don't make your customers wait longer than necessary. Most requests can be satisfied with a standard package. No need to delay things where it doesn't make any sense.
It's 2018 and you probably want that new browser version that mitigates some of the Spectre vulnerabilities on your staff's workstations ASAP. And maybe you even have customers that are using Drupal, in which case... Well, you get the point.
While it does make sense to subscribe to security newsletters and keep an eye on new CVEs, it takes a specialist to maintain your own software stack. When you got word of a new CVE for a program that you're using that doesn't mean the way you built the software makes it vulnerable. And perhaps you have a special use-case where it _is_ but the vulnerability is not exploitable.
Again this important task is one that others have already done for you if you use packaged software from a popular repository. Of course those people are not perfect either and you may very well decide that you do not trust them. Doing everything yourself because you think you can do better is a perfectly legitimate way of handling things. Chances are however that your company cannot afford a specialist for this task. And in that case you're definitely better off trusting the package maintainers than carelessly doing things yourself that you don't have the knowledge for.
Some package managers offer special features not found in other ones. If your organization needs such a feature this can even mean that a new OS or distribution is chosen for some job because of that. Also repositories vary greatly in the number of software they offer, in the software versions that they hold and in the frequency of updates taking place.
A lot of organizations prefer "stable", well-tested software versions. In many cases I think of "stable" as a marketing word for _really old_. For certain use-cases I agree that it makes sense to choose a system where not much will change within the next decade. But IMO this is far less often the case than some decision makers may think.
The other extreme is _rolling-release_ systems which generally adaot the newest software versions after minimal testing. And yes, at one point there was even the "Arch server project" (if I remember the name correctly), which was all about running Arch Linux on a server. In fact this is not as bad an idea as it may seem. There are people who really live Arch and they'll be able to maintain an Arch server for you. But I think this makes most sense as a box for your developers who want to play with new versions of the software that you're using way before it hits your actual dev or even prod servers.
Where possible I definitely prefer the "deliver current versions" model. Not even due to the security aspect (patches are being backported in case of the "stable" repositories) but because of the newer features. It's rather annoying if you want to make use of the jumphost ability of OpenSSH (for which a nice new way of doing it was introduced not too long ago) and then notice you can't use it because there's that stupid CentOS box with its old version of OpenSSH involved!
If you need one or a couple of packages that are not available (or too old) in the package repository of your OS or distribution, chances are that external repos exist or that the upstream project provides packages. That may be ok. However if you find that a lot of the software that you require is not available this may very well be a good reason to think about using a different OS or distribution.
A large number of packages in the repository increases the chance that you may get what you need. Still it can very well be the case that certain packages which you require (and which are rather costly to maintain yourself) are available on another repo.
Some package systems allow you to audit the installed packages. If security is very important for your organization, you'll be happy to have your package tool recommend to "upgrade or deinstall" the installed version of some application because it's known to be vulnerable.
What if you have special needs on some servers and require support for rarely needed functionality to be compiled into some software? With most package systems you're out of luck. The best thing that you can do is roll your own customized package using a different name.
The ports tree on *BSD or portage on Gentoo Linux really show their power in this case, allowing you to just build the software easily and with the options that you choose.
So most of the time it makes perfect sense to stick to the standard repository for your OS or distribution. If you have special needs you'd probably consider another one and use the standard repo for that one. But what about heterogeneous environments?
Perhaps your database product only runs on, say, CentOS. You don't have much choice here. However a lot of customers want their stuff hosted on Linux but they demand newer program versions. So a colleague installed several Ubuntu boxes. And another co-worker, a really strange guy, slipped in some FreeBSD storage servers! When the others found out that this was not even Linux and started protesting (because "BSD is dying"), they were already running too damn well to replaced with something that does not have as good ZFS support.
A scenario like that is not too uncommon. If you don't do anything about it, this might lead to "camps" among the employees; some of them are sure that CentOS is so truly enterprise that it's the way to go. And of course _yum_ is better than _apt-get_ (and whatever that BSD thing offers - if anything!). Some others laugh at that because Ubuntu is clearly superior and using apt-get feels a lot more natural than having to use yum (which is still better than that BSD thing which they refuse to even touch). And then there's the BSD guy who is happy to have a _real OS_ at his hand rather than "kernel + distro-chosen packages".
In general if you are working for a small organization, every admin will have to be able to work with each system that is being used. Proper training for all package systems is probably expansive and thus managers will quite possible be reluctant to accept more than two package systems.
There's a little known (in the Linux community) solution to this: Pkgsrc ("package source"). It's NetBSD's package management system. But with probably the most important goal of the NetBSD project being _portability_, it's portable, too!
Pkgsrc is available for many different platforms. It runs on NetBSD, of course. But it runs on Linux as well as on the other BSDs and on Solaris. It's even available for commercial UNIX platforms and various exotic platforms.
For this very nature of it, Pkgsrc may be one answer for your packaging needs in heterogeneous environments. It can provide a unified means of package management across multiple platforms. It rids you of the headache of version jungle if you use different repositories for different platforms. And it's free and open source, too!
Is it the only solution out there? No. Is it the best one? That certainly depends on what you are looking for specifically. But it's definitely something that you should be aware of.
The next post will be about a relatively new alternative to traditional package management systems that tries to deliver all the strong points in one system while avoiding their weaknesses!