💾 Archived View for gemini.circumlunar.space › users › kraileth › neunix › 2021 › freebsd_package_bu… captured on 2022-06-11 at 21:32:49. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-05)
-=-=-=-=-=-=-
Tuesday, 30. May 2021
[This article has been bi-posted to Gemini and the Web]
The previous article was an introduction into package building on FreeBSD in general. It also included references to various other articles about package management and working with ports. Familiarity with those topics is assumed here. If you don't feel confident, have a look there and do a bit of reading first. Or just go ahead and do some research if you hit something that you have trouble understanding - package building is not a beginner's topic but it ain't rocket science, either.
FreeBSD package building pt. 1: Introduction and test system
In the following section I'm following a "naive approach" of just installing and exploring Synth on a typical desktop system. Afterwards I'm switching to the test system that was described in the previous post so that I can show updating, too.
So let's get started with Synth now, shall we? The first thing to do is of course installing it:
# pkg install synth
If you want to or need to build it from source instead, the origin is _ports-mgmt/synth_. It has one compile-time option: Enabling or disabling the _watchdog_ feature. By default it is enabled which means that Synth is supervising the build processes and if one looks like it has stalled, the watchdog will kill it. When you are building on very slow hardware (why are you doing this in the first place?) it may be an anti-feature that causes very complex ports (e.g. Firefox) to be killed even though it would otherwise eventually finish. In that case you can disable the watchdog option and rebuild Synth. Or you go to the junkyard and get some more capable hardware!
As with every new piece of software, it's probably a good idea to get an overview of what it can do:
% synth help
Summary of command line options - see synth.1 man page for more details
===============================================================================
synth status Dry-run: Shows what 'upgrade-system' would build
synth configure Brings up interactive configuration menu
synth upgrade-system Incremental rebuild of installed packages on system.
Afterwards, the local repository is rebuilt and the
system packages are automatically upgraded.
synth prepare-system Like 'upgrade-system' but ends when repo is rebuilt
synth rebuild-repository Rebuilds local Synth repository on command
synth purge-distfiles Deletes obsolete source distribution files
synth status-everything Dry-run: Shows what 'everything' would build
synth everything Builds entire ports tree and rebuilds repository
synth version Displays version, description and usage summary
synth help Displays this screen
synth status [ports] Dry-run: Shows what will be rebuilt with given list
synth build [ports] Incrementally build ports based on given list, but
asks before updating repository and system
synth just-build [ports] Like 'build', but skips post-build questions
synth install [ports] Like 'build', but upgrades system without asking
synth force [ports] Like 'build', but deletes existing packages first
synth test [ports] Just builds with DEVELOPER=yes; pre-deletes pkgs
[ports] is a space-delimited list of origins, e.g. editors/joe editors/emacs.
It may also be a path to a file containing one origin per line.
That's a nice list and even better: The commands are explained in a way that mere mortals can understand. Basically it can compare installed program versions to those in ports for you with _synth status_. You can configure synth with _synth configure_ (we'll look at this later, the default config is fine for now). And it can bulk-build packages and use those to upgrade the system for you with _synth upgrade-system_. And those are the important ones that you'll need to begin with.
There's also _synth prepare-system_ in case you want Synth to only build the packages (maybe you want to do the upgrade manually later). With _synth rebuild-repository_ you can make the tool, well, rebuild the repository from the currently built packages. Since it does this after finishing package builds, you will need it only if you cancelled building somewhere in the middle and want it to rebuild the repo anyway. The _synth purge-distfiles_ command is useful if you use Synth for a while. It will scan for no longer needed distfiles (for obsolete program versions) and potentially free quite a bit of space for you.
And that's it. The _synth status-everything_ and _synth everything_ commands are only useful if you want to build packages for the *entire* FreeBSD ports collection. But that's certainly not basic usage! The various options that act on a single port or a list of ports is also advanced usage. They'll come in handy if you plan on developing ports, build packages for other machines to use or have special needs. If you plan to keep your machine's installed packages consistent for production use, know that these commands exist but stay away.
So much for some theory and on to actually doing something. Let's ask Synth about the status of our packages:
# synth status
It seems that a blank PORTSDIR is defined in /etc/make.conf
Configuration failed to load.
Alright, this message can be a little confusing as by default there is no such file in FreeBSD! What Synth actually wants to tell you is that it cannot find a ports tree on the system and thus cannot do anything. So let's get the ports in place right now:
# portsnap auto
With the ports tree available on our system we can try again:
# synth status
Configuration invalid: [C] Distfiles directory: /usr/ports/distfiles
Rather than manually creating a directory at this location, consider
using a location outside of the ports tree. Don't forget to set
'DISTDIR' to this new location in /etc/make.conf though.
This time the problem is pretty straight-forward: The distfiles directory does not exist. Synth gives us some advice in case we want to use a custom location, however I prefer the standard one.
# mkdir /usr/ports/distfiles
# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
progress: 3.32%
Depending on your build machine this can take anything from a couple of minutes to a quarter of an hour or so. Before FreeBSD introduced port flavors, it was a matter of seconds, but for a few years now Synth has to do some additional work up-front. When it's done it will print something like this:
Querying system about current package installations.
Installed package ignored, devel/py-evdev package unmatched
Installed package ignored, devel/py-pyudev package unmatched
Installed package ignored, devel/py-six package unmatched
Stand by, comparing installed packages against the ports tree.
Stand by, building pkg(8) first ... done!
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
N => print/indexinfo
N => devel/gettext-runtime
[...]
For the first run, Synth has to build all of the packages. Therefore all are marked _new_. But what's that "Installed package ignored" thing? That usually happens if a package is installed on the system that was built from a port that does no longer exist in the new tree. Sometimes it's also weirdness that can happen with flavored ports. Disregard this if it's only affecting packages that you don't want to use directly. The correct ones will be pulled in as dependencies anyway. Now let's build our first package set:
# synth prepare-system
After gathering some information, Synth will start building the packages that are currently installed on this machine (which for simplicity's sake resemble those of the test system). Let's have a look:
Synth started building packages (PNG)
By default, Synth shows a nice curses-based text UI that displays a lot of information (see screenshot above). It shows the total count of packages to build, the number of packages that remain to be built and the count of already built packages. In case a port does not successfully build on this system (update ports and try again - if the problem persists either file a bug report or contact the maintainer), Synth displays that, too. Ignored packages are those that don't work on your particular system; maybe the application is known to not build on the version of FreeBSD you are using or not on your architecture (if for example you're building on ARM). Skipped count goes up if a failed port was a dependency for others. Obviously Synth cannot build them.
Synth also displays the current system load: A load of e.g. 2.0 means that two cores of your CPU are completely busy. If your CPU supports Hyper-threading that basically doubles the available core count. A load higher than 8.0 on a 4 core system with HT means that the system has currently more work than it's able to fulfill concurrently. It's not ideal but it's not something to worry too much about, either. Watch swap usage. A little bit of swapping for large ports is not much of a problem. But if your system swaps a lot that will slow down the package building a lot. Should you manage to run out of swap, package builds will fail. You can adjust Synth's configuration if you're unhappy with either load or swap usage. But we'll get to that.
Synth after about 20 minutes (PNG)
There's also the number of packages built per hour and the impulse as well as the elapsed time. Initially packages per hour and impulse are identical because the two are similar. However the former is the average number of packages built over the whole build time while the latter is the number of packages built within the last 500 seconds.
But that's only the top bar. The next part of the UI is for the _builders_. Builders are clean build environments ("chroot jails") that are created before building a package and then teared down afterwards. The next package gets a new fresh environment. The number of builders means how many packages can be building concurrently. On the screenshot you can see 6 builders which are color-coded to be easier to distinguish. This may look different on your machine and here's why: Synth tries to guess reasonable defaults _for your machine_. If you have an old dual-core PC, it will use less than the six builders it deemed right for the quad-core i7 that I used as my test machine here. Expect to see a much higher number of builders on modern servers with a higher core count.
Idle builders after 5 hours (PNG)
For each builder you see how long it has been working already, which phase it is currently in (remember the various build targets of the ports infrastructure?) which port the builder is occupied with and how many lines of log output the builder has has produced so far. Especially the last information is not going to help you a lot when you begin building packages. After a while you'll know roughly how many lines of output some of the big ports like LLVM produce and can judge if it's going to take another two hours to finish or more like half an hour. (And then a new version of LLVM comes out which takes even longer to build so that your previous idea of "how many lines" is no longer valid. That's how things go.)
And finally there's the rest of the screen made up of a list of recently finished packages. If you take a look at the second screenshot, you'll see some ports where the origin ends with _@py37_. Here the builder is busy building a flavored port - and it's building for Python 3.7. What's _@native_ you ask? Well, Python ports are a typical example of flavored ports but they are not the only ones. The binutils port for example is able to be built as part of a _native toolchain_ or a _cross toolchain_ in case you want to e.g. cross compile packages for riscv on your much more powerful amd64 machine.
First builder has shut down (PNG)
What's the deal with idle builders like those on screenshot 4? Idle builders are the ones that Synth has already prepared for another package to build in but has not been able to use so far. Take a look at screenshot 4: There's 53 more packages to build but only two builders are occupied with work while four are idle. Can you guess why? The two ports currently building are LLVM10 and Rust. And the "problem" is that all other 51 packages that are still on the list depend on either LLVM or Rust directly or indirectly! So while there's more work for the builders, Synth has to wait for the dependencies to finish building before it can start building the remaining packages.
At some point no further builders will be required. In that case Synth shuts them down (see screenshot 5).
Last package build starting after about 6 hours (PNG)
When Synth has finished building all the packages requested, it will present you with the final statistics. Then it cleans up the package directory, removing obsolete old packages. Eventually it will build a valid pkg(8) repository for you - and then it's finally done.
Synth has completed its task (PNG)
Should you ever want to quit a package run early, try to avoid _CTRL-C_. If you press _CTRL-Q_ instead, Synth will shutdown gracefully. This means that now new builders will be started and the tool exits properly once those that are already running complete.
I'm covering the case where you want to use your own package repository _instead of_ the official FreeBSD one. If you want to use _both_, make sure you read what I pointed my readers to in the previous article. Then configure your new repository as I do here but simply don't disable the official repo.
The standard FreeBSD repository is configured in _/etc/pkg/FreeBSD.conf_. Do not make changes there, though! This file belongs to the base system and upgrades may overwrite it. If you want to change the settings, create _/usr/local/etc/pkg/repos/FreeBSD.conf_ (and the missing directories). The latter file will override the former. In case you just want to disable the official package repository, simply put this single line into the file:
FreeBSD: { enabled: no }
Synth should automatically generate a config file there if you use it to upgrade the system. The file is called _/usr/local/etc/pkg/repos/00_synth.conf_ and has the following content:
# Automatically generated.
Synth: {
url : file:///var/synth/live_packages,
priority : 0,
enabled : yes,
}
Now you only need to update the repository information:
# pkg update
And that's it. Pkg(8) will now happily use your local package repository.
The next article will feature Synth configuration, upgrading and advanced usage.
