This shows you the differences between two versions of the page.
— |
releases:3.2.0:developersguide:bulkpkgsrc [2014/11/11 14:52] (current) |
||
---|---|---|---|
Line 1: | Line 1: | ||
+ | |||
+ | ====== Minix and Pkgsrc ====== | ||
+ | |||
+ | pkgsrc has an advanced system to build each package in isolation of the world except for its stated dependencies. This is a good thing and helps package maintenance tremendously. It makes bulk building a little harder. In keeping with its philosophy, the new style (pbulk style building) requires bootstrapping a new pkgsrc instance (i.e. new prefix that is normally /usr/pkg), e.g. /usr/pbulk. The bulk build process needs to work in /usr/pkg, but that's where your currently installed packages live. So we backup the old tools from /usr/pkg and then restore them after. There are a few other things that need to happen, so we created a script to automate the process. | ||
+ | |||
+ | For more info on pbulk style bulk builds, see the following resources: | ||
+ | * [[http://www.netbsd.org/docs/pkgsrc/bulk.html#bulk.pbulk|pbulk section in the pkgsrc guide]] | ||
+ | * [[http://www.dragonflybsd.org/docs/developer/PbulkBuilding/|DragonFlyBSD pBulk Building Guide]] | ||
+ | * [[http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup&only_with_tag=MAIN|pbulk HOWTO]] | ||
+ | |||
+ | ===== pbulk.sh ===== | ||
+ | |||
+ | As for the minix specifics, there are a few steps that involve some variables that shouldn't be made up on the spot all the time, so there is a script in pkgsrc that takes care of this for Minix. The script is /usr/pkgsrc/minix/pbulk.sh It executes the build in several phases detailed below. Through the use of command line options, you can run any phase individually with the //--phase-name// argument or run them all with the //--all// argument. | ||
+ | * //bootstrap// - a new pkgsrc instance is bootstraped into /usr/pbulk using /usr/pkgsrc/bootstrap/bootstrap | ||
+ | * //backup// - as mentioned above, the bulk build process will clobber your current package source installation if you aren't careful. This phase backs up your /usr/pkg directory as well as your package database. | ||
+ | * //bin-kit// - bootstraps a binary kit used by pbulk. | ||
+ | * //build// - executes the bulk build. | ||
+ | * //restore// - restores system's original /usr/pkg and the package database. | ||
+ | * //all// - executes all of the above steps in order. | ||
+ | |||
+ | ===== Configuration ===== | ||
+ | |||
+ | We are going to run the pbulk process //in a minix chroot jail.// This has a few advantages: | ||
+ | * Sterile, reproducible environment of building | ||
+ | * Easy to automatically make an up-to-date userland (except for the image of course) to test compiling in | ||
+ | * The pbulk process wipes out /usr/pkg, which is tricky to keep a live machine working on (sshd etc.), but in a jail that's no problem. | ||
+ | |||
+ | This needs quite a lot of disk space though. At least 3GB extra for the jail currently. | ||
+ | |||
+ | ===== Bulk Build In Jail ===== | ||
+ | |||
+ | Assuming pkgsrc is created in /usr/pkgsrc: | ||
+ | <code> | ||
+ | # cd /usr/pkgsrc/minix/ | ||
+ | # sh pbulk-jail.sh -A | ||
+ | </code> | ||
+ | |||
+ | This will invoke /usr/src/tools/release.sh, use git to checkout the latest version of the minix tree, build it in a jail | ||
+ | hierarchy at /usr/pbulk-jail, copy the host pkgsrc tree there to re-use its objects, and retrieve the latest pkgsrc tree from the pkgsrc repository, and invoke pbulk.sh --all in the chrooted jail. | ||
+ | |||
+ | When the script finishes, you'll have some packages in /usr/pkgsrc/packages/ in the jail. Note, the bootstrapping and binary kit creation takes a while. To do a bulk build of just the misc/figlet package from scratch takes over an hour. | ||
+ | |||
+ | ===== Running the above on a separate filesystem ===== | ||
+ | |||
+ | The pbulk-jail script will mkfs and mount a new filesystem for the jail if you want; use the -d option to specify a device, and the jail script will umount it, **mkfs it**, and mount it, and build a jail on it etc. | ||
+ | <code> | ||
+ | # sh pbulk-jail.sh -d/dev/c0d0p2s3 -A | ||
+ | </code> | ||
+ | |||
+ | The script will automatically customize the jail directory name based on the devname, so starting multiple bulk builds at on different filesystems is ok. | ||
+ | |||
+ | |||
+ | ===== Bulk build customisation ===== | ||
+ | |||
+ | In the jail, there are several opportunities to customize the build. You can decide which packages to attempt building by listing them in /usr/pkgsrc/minix/limited_list.pbulk. You can set various pbulk parameters (package upload location, reports, etc.) in /usr/pkgsrc/minix/pbulk.conf. pbulk.conf is installed to /usr/pbulk/etc/pbulk.conf during the bootstrap step, so if you want to do further modifications later, be sure to edit /usr/pbulk/etc/pbulk.conf. The various /usr/pkgsrc/minix/mk.conf* files can be edited as well. | ||
+ | |||
+ | There isn't good support for saving your customisations though, so you can't update the pkgsrc tree and keep your edits easily currently. | ||
+ | |||
+ | |||
+ | ====== TODO ====== | ||
+ | * An automated system that, from any OS, runs a qemu, using the qemu builds a new minix from current svn, then runs that .iso to install a Minix, then boots into that minix, then fetches the latest pkgsrc tree, and does the above to bulk-build all known-good packages, and produces pretty html status overviews of it so we can make sure this process, and known-good packages, don't break by base system or pkgsrc changes. And if they do, see in which minix svn / pkgsrc commit things were still OK. | ||
+ | |||
+ | * Have multiple such instances be able to work together to do parallel pbulk builds. | ||