pkgsrc has an advanced system to build each package in isolation of the world except for its stated dependencies. This is a good thing and helps package maintenance tremendously. It makes bulk building a little harder. In keeping with its philosophy, the new style (pbulk style building) requires bootstrapping a new pkgsrc instance (i.e. new prefix that is normally /usr/pkg), e.g. /usr/pbulk. The bulk build process needs to work in /usr/pkg, but that's where your currently installed packages live. So we backup the old tools from /usr/pkg and then restore them after. There are a few other things that need to happen, so we created a script to automate the process.

For more info on pbulk style bulk builds, see the following resources:

• pbulk section in the pkgsrc guide
• DragonFlyBSD pBulk Building Guide
• pbulk HOWTO

As for the minix specifics, there are a few steps that involve some variables that shouldn't be made up on the spot all the time, so there is a script in pkgsrc that takes care of this for Minix. The script is /usr/pkgsrc/minix/pbulk.sh It executes the build in several phases detailed below. Through the use of command line options, you can run any phase individually with the –phase-name argument or run them all with the –all argument.

• fetch - runs bmake fetch for every package in limited_list.pbulk.
• bootstrap - a new pkgsrc instance is bootstraped into /usr/pbulk using /usr/pkgsrc/bootstrap/bootstrap
• backup - as mentioned above, the bulk build process will clobber your current package source installation if you aren't careful. This phase backs up your /usr/pkg directory as well as your package database.
• bin-kit - bootstraps a binary kit used by pbulk.
• build - executes the bulk build.
• restore - restores system's original /usr/pkg and the package database.
• all - executes all of the above steps in order (except fetch).

We are going to run the pbulk process in a minix chroot jail. This has a few advantages:

• Sterile, reproducible environment of building
• Easy to automatically make an up-to-date userland (except for the image of course) to test compiling in
• The pbulk process wipes out /usr/pkg, which is tricky to keep a live machine working on (sshd etc.), but in a jail that's no problem.

This needs quite a lot of disk space though. At least 3GB extra for the jail currently.

Assuming pkgsrc is created in /usr/pkgsrc:

# cd /usr/pkgsrc/minix/
# sh pbulk-jail.sh -A

This will invoke /usr/src/tools/release.sh, use git to checkout the latest version of the minix tree, build it in a jail hierarchy at /usr/pbulk-jail, copy the host pkgsrc tree there to re-use its objects, and retrieve the latest pkgsrc tree from the pkgsrc repository, and invoke pbulk.sh –all in the chrooted jail.

When the script finishes, you'll have some packages in /usr/pkgsrc/packages/ in the jail. Note, the bootstrapping and binary kit creation takes a while. To do a bulk build of just the misc/figlet package from scratch takes over an hour.

The pbulk-jail script will mkfs and mount a new filesystem for the jail if you want; use the -d option to specify a device, and the jail script will umount it, mkfs it, and mount it, and build a jail on it etc.

# sh pbulk-jail.sh -d/dev/c0d0p2s3 -A

The script will automatically customize the jail directory name based on the devname, so starting multiple bulk builds at on different filesystems is ok.

In the jail, there are several opportunities to customize the build. You can decide which packages to attempt building by listing them in /usr/pkgsrc/minix/limited_list.pbulk. You can set various pbulk parameters (package upload location, reports, etc.) in /usr/pkgsrc/minix/pbulk.conf. pbulk.conf is installed to /usr/pbulk/etc/pbulk.conf during the bootstrap step, so if you want to do further modifications later, be sure to edit /usr/pbulk/etc/pbulk.conf. The various /usr/pkgsrc/minix/mk.conf* files can be edited as well.

There isn't good support for saving your customisations though, so you can't update the pkgsrc tree and keep your edits easily currently.

• An automated system that, from any OS, runs a qemu, using the qemu builds a new minix from current git, then runs that .iso to install a Minix, then boots into that minix, then fetches the latest pkgsrc tree, and does the above to bulk-build all known-good packages, and produces pretty html status overviews of it so we can make sure this process, and known-good packages, don't break by base system or pkgsrc changes. And if they do, see in which minix git / pkgsrc commit things were still OK.

• Have multiple such instances be able to work together to do parallel pbulk builds. There are several approaches that could be used:
  • discc - distributes the compilation tasks across several systems. Another tool that works well in combination with distcc is ccache. Note, some packages might not support parallel make (i.e. MAKEOPTS=“-j2” or higher).
  • pbulk slaves - within pbulk itself there is support for multiple slave machines. The upstream sources rely on NFS for this, but we have some local changes which try to replicate the files with rsync. Last time this was tried, it didn't work but maybe it can be made to work.