💾 Archived View for perso.pw › blog › articles › openbsd-dpb.gmi captured on 2023-04-26 at 13:37:15. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-01-29)
-=-=-=-=-=-=-
NIL=> https://bsd.network/@solenepercent/106324291780467822 Comment on Mastodon
Today I will explain how to easily setup your own OpenBSD dpb infra. dpb is a tool to manage port building and can use chroot to use a sane environment for building packages.
This is particularly useful when you want to test packages or build your own, it can parallelize package compilation in two way: multiples packages at once and multiples processes for one package.
The dpb and proot executable files are available under the bin directory of the ports tree.
Building your packages provide absolutely NOTHING compared to using binary packages except wasting CPU time, disk space and bandwidth.
You need a ports tree and a partition that you accept to mount with wxallowed,nosuid,dev options. I use /home/ for that. To simplify the setup, we will create a chroot in /home/build/ and put our ports tree in /home/build/usr/ports (then your /usr/ports can be a symlink).
Create a text file that will be used as a configuration file for proot
chroot=/home/build WRKOBJDIR=/tmp/pobj LOCKDIR=/tmp/locks PLIST_REPOSITORY=/data/plist DISTDIR=/data/distfiles PACKAGE_REPOSITORY=/data/packages actions=unpopulate sets=base comp etc xbase xfont xshare xetc xserver
This will tell proot to create a chroot in /home/build and preconfigure some variables for /etc/mk.conf, use all sets listed in "sets" and clean everything when run (this is what actions=unpopulate is doing). Running proot is as easy as "proot -c proot_config".
Then, you should be able to run "dpb -B /home/build/ some/port" and it will work.
I wrote a script to clean locks from dpb, locks from ports system and pobj directories but also taking care of adding the mount options.
Options -p and -j will tell dpb how many cores can be used for parallel compilation, note that dpb is smart and if you tell it 3 ports in parallel and 3 threads in parallel, it won't use 3x3, it will compile three ports at a time and once it's stuck with only one port, it will add cores to its build to make it faster.
#!/bin/sh
CHROOT=/home/build/
CORES=3
rm -fr ${CHROOT}/usr/ports/logs/amd64/locks/*
rm -fr ${CHROOT}/tmp/locks/*
rm -fr ${CHROOT}/tmp/pobj/*
mount -o dev -u /home
mount -o nosuid -u /home
mount -o wxallowed -u /home
/usr/ports/infrastructure/bin/dpb -B $CHROOT -c -p $CORES -j $CORES $*
Then I use "doas ./my_dpb.sh sysutils/p5-Rex lang/guile" to run the build process.
It's important to use -c in dpb command line which will clear compilation logs of the packages but retains the log size, this will be used to estimate further builds progress by comparing current log size with previous logs sizes.
You can harvest your packages from /home/build/data/packages/ , I even use a symlink from /usr/ports/packages/ to the dpb packages directory because sometimes I use make in ports and sometimes I use dpb, this allow recompiling packages in both areas. I do the same for distfiles.
dpb can spread the compilation load over remote hosts (or even manage compilation for a different architecture), it's not complicated to setup but it's out of scope for the current guide. This requires setting up ssh keys and NFS shares, the difficulty is to think with the correct paths depending on chroot/not chroot and local / nfs.
I extremely recommend reading dpb man pages, it supports many options such as providing it a list of pkgpaths (package address such as editor/vim or www/nginx) or building ports in random order.
Here is a simply command to generate a list of pkgpaths of outdated packages on your system compared to the ports tree, the -q parameter is to make it a lot quicker but less accurate for shared libraries.
/usr/ports/infrastructure/bin/pkg_outdated -q | awk '/\// { print $1 }'
I use dpb when I want to update my packages from sources because the binary packages are not yet available or if I want to build a new package in a clean environment to check for missing dependencies, however I use a simple "make" when I work on a port.