User Tools

Site Tools


releases:3.2.0:developersguide:bulkpkgsrc

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

releases:3.2.0:developersguide:bulkpkgsrc [2014/11/11 08:52] (current)
Line 1: Line 1:
 +
 +====== Minix and Pkgsrc ======
 +
 +pkgsrc has an advanced system to build each package in isolation of the world except for its stated dependencies. This is a good thing and helps package maintenance tremendously. It makes bulk building a little harder. In keeping with its philosophy, the new style (pbulk style building) requires bootstrapping a new pkgsrc instance (i.e. new prefix that is normally /usr/pkg), e.g. /usr/pbulk. The bulk build process needs to work in /usr/pkg, but that's where your currently installed packages live. So we backup the old tools from /usr/pkg and then restore them after. There are a few other things that need to happen, so we created a script to automate the process.
 +
 +For more info on pbulk style bulk builds, see the following resources:
 +  * [[http://​www.netbsd.org/​docs/​pkgsrc/​bulk.html#​bulk.pbulk|pbulk section in the pkgsrc guide]]
 +  * [[http://​www.dragonflybsd.org/​docs/​developer/​PbulkBuilding/​|DragonFlyBSD pBulk Building Guide]]
 +  * [[http://​cvsweb.netbsd.org/​bsdweb.cgi/​pkgsrc/​doc/​HOWTO-pbulk?​rev=1.7&​content-type=text/​x-cvsweb-markup&​only_with_tag=MAIN|pbulk HOWTO]]
 +
 +===== pbulk.sh =====
 +
 +As for the minix specifics, there are a few steps that involve some variables that shouldn'​t be made up on the spot all the time, so there is a script in pkgsrc that takes care of this for Minix. The script is /​usr/​pkgsrc/​minix/​pbulk.sh It executes the build in several phases detailed below. Through the use of command line options, you can run any phase individually with the //​--phase-name//​ argument or run them all with the //--all// argument.
 +  * //​bootstrap//​ - a new pkgsrc instance is bootstraped into /usr/pbulk using /​usr/​pkgsrc/​bootstrap/​bootstrap
 +  * //backup// - as mentioned above, the bulk build process will clobber your current package source installation if you aren't careful. This phase backs up your /usr/pkg directory as well as your package database.
 +  * //bin-kit// - bootstraps a binary kit used by pbulk.
 +  * //build// - executes the bulk build.
 +  * //restore// - restores system'​s original /usr/pkg and the package database.
 +  * //all// - executes all of the above steps in order.
 +
 +===== Configuration =====
 +
 +We are going to run the pbulk process //in a minix chroot jail.// This has a few advantages:
 +  * Sterile, reproducible environment of building
 +  * Easy to automatically make an up-to-date userland (except for the image of course) to test compiling in
 +  * The pbulk process wipes out /usr/pkg, which is tricky to keep a live machine working on (sshd etc.), but in a jail that's no problem.
 +
 +This needs quite a lot of disk space though. At least 3GB extra for the jail currently.
 +
 +===== Bulk Build In Jail =====
 +
 +Assuming pkgsrc is created in /​usr/​pkgsrc:​
 +<​code>​
 +# cd /​usr/​pkgsrc/​minix/​
 +# sh pbulk-jail.sh -A
 +</​code>​
 +
 +This will invoke /​usr/​src/​tools/​release.sh,​ use git to checkout the latest version of the minix tree, build it in a jail
 +hierarchy at /​usr/​pbulk-jail,​ copy the host pkgsrc tree there to re-use its objects, and retrieve the latest pkgsrc tree from the pkgsrc repository, and invoke pbulk.sh --all in the chrooted jail.
 +
 +When the script finishes, you'll have some packages in /​usr/​pkgsrc/​packages/​ in the jail. Note, the bootstrapping and binary kit creation takes a while. To do a bulk build of just the misc/figlet package from scratch takes over an hour.
 +
 +===== Running the above on a separate filesystem =====
 +
 +The pbulk-jail script will mkfs and mount a new filesystem for the jail if you want; use the -d option to specify a device, and the jail script will umount it, **mkfs it**, and mount it, and build a jail on it etc.
 +<​code>​
 +# sh pbulk-jail.sh -d/​dev/​c0d0p2s3 -A
 +</​code>​
 +
 +The script will automatically customize the jail directory name based on the devname, so starting multiple bulk builds at on different filesystems is ok.
 +
 +
 +===== Bulk build customisation =====
 +
 +In the jail, there are several opportunities to customize the build. You can decide which packages to attempt building by listing them in /​usr/​pkgsrc/​minix/​limited_list.pbulk. You can set various pbulk parameters (package upload location, reports, etc.) in /​usr/​pkgsrc/​minix/​pbulk.conf. pbulk.conf is installed to /​usr/​pbulk/​etc/​pbulk.conf during the bootstrap step, so if you want to do further modifications later, be sure to edit /​usr/​pbulk/​etc/​pbulk.conf. The various /​usr/​pkgsrc/​minix/​mk.conf* files can be edited as well.
 +
 +There isn't good support for saving your customisations though, so you can't update the pkgsrc tree and keep your edits easily currently.
 +
 +
 +====== TODO ======
 +  * An automated system that, from any OS, runs a qemu, using the qemu builds a new minix from current svn, then runs that .iso to install a Minix, then boots into that minix, then fetches the latest pkgsrc tree, and does the above to bulk-build all known-good packages, and produces pretty html status overviews of it so we can make sure this process, and known-good packages, don't break by base system or pkgsrc changes. And if they do, see in which minix svn / pkgsrc commit things were still OK.
 +
 +  * Have multiple such instances be able to work together to do parallel pbulk builds.
  
releases/3.2.0/developersguide/bulkpkgsrc.txt · Last modified: 2014/11/11 08:52 (external edit)