💾 Archived View for gemini.circumlunar.space › users › kraileth › neunix › 2021 › freebsd_package_bu… captured on 2022-07-16 at 14:12:02. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-05)
-=-=-=-=-=-=-
Thursday, 10. April 2021
[This article has been bi-posted to Gemini and the Web]
In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program's configuration, working with compiler cache and using the tool to update the installed applications was covered as well.
FreeBSD package building pt. 1: Introduction and test system
FreeBSD package building pt. 2: Basic Synth
FreeBSD package building pt. 3: Intermediate Synth
This article is about some advanced functionality of Synth like using _web reports_, building package sets and serving repositories as well as taking a look at logs.
Synth comes with very useful reporting reporting functionality. For the default _LiveSystem_ profile, Synth writes its logs to _/var/log/synth_. There it also creates a subdirectory called _Reports_ and puts an advanced web report in there. It looks like this:
% ls -1 /var/log/synth/Report
01_history.json
favicon.png
index.html
progress.css
progress.js
summary.json
synth.png
We're going to build and setup a web server to make it accessible. I will use OpenBSD HTTPd for a reason that we're going to talk about in a minute (besides me liking it quite a bit). Let's use Synth to install it first. Then we're going to enable it and create a log directory for it to use:
# synth install www/obhttpd
# sysrc obhttpd_enable=YES
# mkdir /var/log/obhttpd
OpenBSD HTTPd successfully installed by Synth (PNG)
Alright. Remember how I told you not to change Synth's directory configuration unless you have a reason for it? Now we have: We're going to serve both the web report and the packages over http and we're using OpenBSD HTTPd for that. That browser is _chrooted_: Not just by default, BTW but _by design_! You _cannot_ turn it off. Unless told otherwise, Synth has a directory structure that doesn't fit this scenario well. So we're going to change it.
First thing is creating a new directory for the logs and then changing the configuration so Synth uses that:
# mkdir -p /var/synth/www/log
# synth configure
Change setting _E_ to _/var/synth/www/log_ and save. The new directory will of course be empty. We could copy the old files over, but let's just rebuild HTTPd instead. Using _synth build_ or _synth just-build_ doesn't work, though; the tool will detect that an up to date package has already been built and that nothing needs to be done. That's what the _force_ command is handy for:
# synth force www/obhttpd
Now that we have something to serve, we can edit the webserver's configuration file. Simple delete everything and put something like this into _/usr/local/etc/obhttpd.conf_:
chroot "/var/synth/www"
logdir "/var/log/obhttpd"
server synth.local {
listen on * port 80
root "/log"
log style combined
location "/" {
block return 302 "http://$HTTP_HOST/Report/index.html"
}
}
This defines where the chroot begins and thus where the unprivileged webserver processes are contained. It also defines the log directory as well as a virtual host (simply called "server" in OpenBSD HTTPd) with the name "synth.local". Either replace this with a proper domain name that fits your scheme and configure your DNS accordingly or use _/etc/hosts_ on all machines that need to access it to define the name there.
The server part is pretty simple, too. It makes HTTPd bind on port 80 on every usable network interface. The web root is defined relative to the chroot, so in this case it points to /var/synth/www/log. I've grown a habit of using the more detailed _combined_ log style; if you're fine with the default _common_ format, you can leave the respective line out. Finally the configuration block defines a special rule for _location /_ which means somebody accesses the virtual host directly (i.e. http://synth.local in this case). It will make the browser be redirected to the report index instead. Getting a special file (like e.g. lang___python37.log in the log directory) will not trigger the rule and thus still work. This is just a convenience thing and if you don't like it leave it out.
All that's missing now is starting up the webserver:
# service obhttpd start
You should now be able to point your browser at the the vhost's name (if you made it resolve). Just using the machine's IP address is also going to work in this case since it's the default vhost. But better make it reachable using the configured name as we're adding another vhost in a moment.
Synth web report for the latest package run (PNG)
But for now what about security? Let's say you don't want to share your report with the whole world. One easy means of protecting it is by using HTTP _basic auth_. OpenBSD HTTPd uses standard .htpasswd files. These can however use various cryptographic hashes for the passwords - whereas HTTPd only supports one: _bcrypt_.
The first time I tried to do authentication with OpenBSD HTTPd, it drove me completely nuts as I couldn't get it working. Fortunately I own Michael W. Lucas' excellent book "Httpd and Relayd Mastery". After digging it out the index indicated that I might want to read page 28. I did, banged my head against the table and immediately got it working. Don't be like me, skip trying to use foreign tools and just do it right in the first place. HTTPd comes with its own _htpasswd_ binary. Use that.
# obhtpasswd /var/synth/www/.htpasswd synth
In this example I'm adding a user called "synth". Use whatever you prefer. Then give the password two times. This leaves you with a valid .htpasswd file that HTTPd _could_ use - if it was allowed to access it! Let's fix that problem:
# chown root:wheel /var/synth/www/.htpasswd
# chmod 0640 /var/synth/www/.htpasswd
Having the authentication information in place, we only need to add another _location_ block to the webserver's vhost configuration. Put the following in there after the line that closes the previous location block:
location "/Report/* {
authenticate with "/.htpasswd"
}
Note the htpasswd file's location! It's within the chroot (or it couldn't be accessed by the webserver), but _outside_ the webroot directory. So HTTPd could never accidentally serve it to somebody who knew that it was there and requested the file.
The only thing that remains is restarting the webserver. Next time you visit the report page, you'll be asked to authenticate first.
# service obhttpd restart
So far all of our packages have been created in a directory outside of the webserver's chroot. If we want to make them available via HTTP, we need to use another path for them. Therefore we're going to create a directory and reconfigure Synth again:
# mkdir -p /var/synth/www/packages
# synth configure
This time it's setting _B_. Change it to _/var/synth/www/packages_ and save. Now let's build a package that draws in a couple of dependencies:
# synth just-build chocolate-doom
We can watch it now via the web reports while it's building. Since it's a new directory where no packages exist, yet, Synth is first going to build the package manager again. During this early stage no report is available, but once that's finished the reports work.
While we're rebuilding all packages due to the new package directory, Synth can take advantage of ccache as we haven't modified it's path. Wonder how much of a difference that actually makes? Building llvm10 on its own, one time using the cache and one time (for testing purposes) without it will show you the difference:
Duration: 00:13:32 (with ccache)
Duration: 02:09:37 (uncached)
Synth web report while it's building (PNG)
The report gives us all the information that the curses UI holds - and more. The number of entries for completed packages can be changed. You can browse those page-wise back to the very first packages. It's possible to use filters to e.g. just list skipped or failed packages. You can view / download (whatever your browser does) the log files for all those packages. And there's even a search (which can be very handy if you're building a large package set).
Report with only 10 entries per page (PNG)
As long as packages are being built, the report page also shows the builder information and automatically refreshes the page every couple of seconds. Once it completes, it removes builder info (which would only waste space) and stops the polling. You can always come back later and inspect everything about the latest package run. The next one will overwrite the previous information, though.
Synth report search function (PNG)
Now that we have a bunch of newly built packages, let's see what that looks like:
# ls -1 /var/synth/packages/All
autoconf-2.69_3.txz
autoconf-wrapper-20131203.txz
automake-1.16.3.txz
binutils-2.33.1_4,1.txz
bison-3.7.5,1.txz
ca_root_nss-3.63.txz
ccache-3.7.1_1.txz
celt-0.11.3_3.txz
chocolate-doom-3.0.1.txz
cmake-3.19.6.txz
curl-7.76.0.txz
db5-5.3.28_7.txz
docbook-1.5.txz
docbook-sgml-4.5_1.txz
docbook-xml-5.0_3.txz
docbook-xsl-1.79.1_1,1.txz
doom-data-1.0_1.txz
evdev-proto-5.8.txz
expat-2.2.10.txz
flac-1.3.3_1.txz
[...]
Showing only ignored packages in the report (none in this case) (PNG)
The packages are there. But what's in the main directory?
# ls -l /var/synth/www/packages
total 18
drwxr-xr-x 2 root wheel 150 Jun 7 23:57 All
drwxr-xr-x 2 root wheel 3 Jun 7 23:21 Latest
This is not a valid pkg(8) repository. Which is no wonder since we used _just-build_. So we're going to have Synth create an actual repository from these packages next:
Searching in the report after the build was completed (PNG)
# synth rebuild-repository
# ls -l /var/synth/www/packages
total 117
drwxr-xr-x 2 root wheel 150 Jun 7 23:57 All
drwxr-xr-x 2 root wheel 3 Jun 7 23:21 Latest
-rw-r--r-- 1 root wheel 163 Jun 8 00:02 meta.conf
-rw-r--r-- 1 root wheel 236 Jun 8 00:02 meta.txz
-rw-r--r-- 1 root wheel 40824 Jun 8 00:02 packagesite.txz
Here we go, that's all that pkg(8) needs. Synth should have automatically updated your repository configuration to use the new location. Have a look at _/usr/local/etc/pkg/repos/00_synth.conf_ - the URL should point to the new directory.
The next step is to make the repository available in the network, too. So edit _/usr/local/etc/obhttpd.conf_ once more and add another "server" (i.e. vhost):
server pkg.local {
listen on * port 80
root "/packages"
log style combined
location * {
directory auto index
}
}
One service restart later you should be able to access the repository via a web browser from any machine in the same subnet (if you got your DNS right):
# service obhttpd restart
Looking at the package repository with a browser (PNG)
This is already it, but let's prove that it works, too. I'm adding the "pkg.local" name to the machine's 127.0.0.1 definition in _/etc/hosts_, then change the URL in the Synth repo to fetch packages via HTTP.
url : http://pkg.local,
I've also created a FreeBSD.conf to disable the official repository. Let's stop the webserver for a second and then try to update the package DB:
# service obhttpd stop
# pkg update
Updating Synth repository catalogue...
pkg: Repository Synth has a wrong packagesite, need to re-create database
pkg: http://pkg.local/meta.txz: Connection refused
Unable to update repository Synth
Error updating repositories!
Ok, so there's no other repository configured anymore and this one is not accessed via the filesystem. So we're going to start the webserver once more (give it a sec) and then try again:
# service obhttpd start
# pkg update
Updating Synth repository catalogue...
Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
Fetching packagesite.txz: 100% 40 KiB 40.8kB/s 00:01
Processing entries: 100%
Synth repository update completed. 148 packages processed.
All repositories are up to date.
Great! So now we can install DooM on this box or on any other machine running FreeBSD 13.0 which can reach it over the network.
So far we've only either built all packages for what was already installed on the machine or for single ports that we selected at the command line. But now that we can serve packages over the network, it's rather tempting to use a powerful build machine to build packages for various other FreeBSD machines, isn't it? Let's assume that you're going to share packages with a KDE lover.
First we should prepare a list of packages that we need, starting with what is installed on our machine.
# pkg info | wc -l
345
Wow, that's already quite some packages for such a pretty naked system! But we don't need to consider them all as most of them are just dependencies. Let's ask pkg(8) for the origin of all packages we _explicitly_ installed (i.e. which were not recorded as _automatically_ installed):
# pkg query -e %a=0 %o > /var/synth/pkglist
# cat /var/synth/pkglist
x11-wm/awesome
devel/ccache
graphics/drm-kmod
devel/git
www/obhttpd
ports-mgmt/pkg
ports-mgmt/portmaster
x11/sakura
x11/setxkbmap
security/sudo
ports-mgmt/synth
sysutils/tmux
x11/xfce4-screenshooter-plugin
x11/xorg-minimal
That's better! But we certainly _don't_ need portmaster anymore, so we can take it off the list (and deinstall it actually). Let's add _www/firefox_ and _x11/kde5_ for our pal (and sort the list since it's a bit ugly right now).
Once that's done, we should be able to do a simple:
# synth build /var/synth/pkglist
Error: port origin 'devel/git' not recognized.
Perhaps you intended one of the following flavors?
- devel/git@default
- devel/git@gui
- devel/git@lite
- devel/git@svn
- devel/git@tiny
Oh yes, right! We need to edit our list and specify the flavor to build! I'm going with the _lite_ variant here, so the git line needs to be changed to this:
devel/git@lite
Then we can try again - and yes, it starts building after calculating the required dependencies.
Whoopsie! After about an hour stupid me removed the network cable for a moment. This has caused a couple of build failures (see screenshot). The report will display the phase that the build failed. In this case it's the _fetch phase_ (and we don't have to look for the reason as we already know it). Sometimes a distfile mirror is temporary down or the distfile has been removed. In that case you will have to manually get the files and put them into the distfiles directory. Skipped ports also display the reason, i.e. which dependency failed previously.
Failed and skipped ports due to a connection problem (PNG)
I better re-attach that cable right away and start the building over... Many hours later it has finished. But what's this? Rust has failed again (and this time it wasn't me)! And it failed at the _stage phase_. When this happens it's usually because of a broken port got committed. Update your ports tree and hope that it has been fixed in the meantime. This is not the reason in our case, however.
Another phase, another failure! (PNG)
But how do we find out what actually happened? Well, by looking at the logs, of course. Here's the last 15 lines of the relevant log:
finished in 183.141 seconds
< Docs { host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-freebsd", file: None }))
running: "sh" "/construction/xports/lang/rust/work/rustc-1.51.0-src/build/tmp/tarball/rust-docs/x86_64-unknown-freebsd/rust-docs-1.51.0-x86_64-unknown-freebsd/install.sh" "--prefix=/construction/xports/lang/rust/work/stage/usr/local" "--sysconfdir=/construction/xports/lang/rust/work/stage/usr/local/etc" "--datadir=/construction/xports/lang/rust/work/stage/usr/local/share" "--docdir=/construction/xports/lang/rust/work/stage/usr/local/share/doc/rust" "--bindir=/construction/xports/lang/rust/work/stage/usr/local/bin" "--libdir=/construction/xports/lang/rust/work/stage/usr/local/lib" "--mandir=/construction/xports/lang/rust/work/stage/usr/local/share/man" "--disable-ldconfig"
install: creating uninstall script at /construction/xports/lang/rust/work/stage/usr/local/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'
### Watchdog killed runaway process! (no activity for 78 minutes) ###
--------------------------------------------------
-- Termination
--------------------------------------------------
Finished: Wednesday, 9 JUN 2021 at 00:04:47 UTC
Duration: 04:27:03
Ha! The build process was killed by the watchdog! Bad doggy? It does happen that the process would eventually have finished. Not this time. We have to dig a little deeper. In _/var/log/messages_ of the build machine I can find the messages _kernel: swap_pager: out of swap space_ and _kernel: swp_pager_getswapspace(4): failed_. This machine has 24 GB of RAM and 8 GB of swap space configured. And by building 6 huge ports concurrently, it exceeded these resources! Keep in mind that package building can be quite demanding, especially if you use tmpfs (which you should if you can).
So, there we are. We've configured our build server for web reports and serving the repository. We've looked at building package sets and covered a few examples of what can go wrong. And that's it for today.
The last article about Synth will cover _make.conf_, signing repositories and using cron for automated builds. We'll also take a brief look at profiles.
