Student: Thomas Cort
Mentor: Ben Gras and Gautam Tirumala
git repository: minix-pkgsrc and pkgsrc

The primary goals of this project are to complete any remaining tasks for the minix pkgsrc port, send the changes upstream, and port more software applications to minix. Along the way, the minix pkgsrc tree will be resynced with the upstream tree, and changes will be made to ease the transition from the classic Minix libc to the Minix port of NetBSD's libc.

The table below lists packages with changes to be upstreamed. The following packages have been upstreamed or the patches have been eliminated (for example, by switching to NetBSD libc): archivers/bsdtar, archivers/gtar-base, devel/cscope, devel/gmp, devel/gtexinfo, devel/ncurses, devel/patch, devel/pcre, devel/scmgit, devel/scmgit-base, editors/zile, games/cmatrix, games/ninvaders, games/sex, graphics/tiff, ham/locator, lang/gawk, misc/figlet, net/rsync, pkgtools/pkgin, textproc/gsed, textproc/wdiff, wip/ppl.

This week I spent most of my time focused on pbulk. I completed the pbulk.sh script in order to automate the bulk building process and to make it easier to get consistent results. I was able to successfully complete several bulk builds. Along the way I did several minor fixes: update LD and AR in bootstrap-mk-files, fix the cmake dependency in compiler-rt, and set the proper permissions on /usr.

I created a limited_list file (/usr/pkgsrc/minix/limited_list.pbulk) to tell pbulk which packages to build. The list contains every package that is expected to work on minix. I ran a full build of all of those packages. There were some packages that failed to build. Most of the errors seemed to be related to minor changes in the base system. I will investigate the problems more deeply in the next week or two after I update the minix pkgsrc repository with the upstream changes. I want to avoid fixing the same package twice.

I also worked on parallel bulk building. The documentation says to use NFS when running pbulk across several machines. Since minix doesn't have NFS, I added rysnc commands to push packages and logs between the clients and the master. I can start the build, but when it builds the initial dependency tree I get messages about select() failing. Then when the build starts, it halts with an error about circular dependencies. The upstream pkgsrc repository has a version of pbulk that is several revisions newer than what I've been working with. When I update the minix pkgsrc repository with the upstream changes, I'll come back to this.

Hours worked: 44

This week I merged the upstream pkgsrc changes into the minix pkgsrc repository. I fixed a few breakages, did some light porting, adjusted a couple of patches, did some bulk building, and some testing. The updates/changes have been pushed out to the minix pkgsrc repository.

There were a few bumps along the way. I ran into a problem building perl module packages. This is caused by files in binary packages having bad timestamps (year 2038). I'm still looking into this, but it was a problem that was happening before I started the merge. security/sudo needs some more porting work. It too was broken before the merge. I wasn't able to test everything because some packages have been ported to elf and I was using an a.out system. I did test some packages where it was easy to temporarily back out the ELF changes, like openssl.

In the coming week I will work on fixing the file timestamp problem mentioned above, and I will work on upstreaming minix changes.

Hours worked: 48

This week I tracked down and fixed the libarchive timestamp problem mentioned in my previous report. I also installed Minix 3.2.0 (elf), and I did a bulk build of pkgsrc packages with decent results. My main focus this week was upstreaming minix changes.

The Upstreaming Status table above was generated by git diffstat master minix-master and some command line magic. It lists all of the packages with minix changes and where I'm at in terms of getting those changes upstream. As of June 12, I've submitted 10 patches and 2 have been accepted.

In the coming week I will continue preparing and submitting patches as well as responding to feedback from upstream.

Hours worked: 42

Over the past week I've been working on upstreaming Minix changes to pkgsrc. That involved responding to feedback, asking questions, re-writing patches, submitting problem reports, investigating issues, and doing testing and building. I also began adding a feature to mark a package as must compile with minix libc.

In the coming week I'll be responding to feedback on the Minix pkgsrc patches, I'll finish off the feature to mark a package as must compile with minix libc, I'll do some builds against the new NetBSD-based libc, and I'll submit more patches upstream.

Hours worked: 40

Over the past week I finished implementing the feature to mark a package as must compile with minix libc. I'll be writing up a guide for users interested in helping build/test software with the new C library. Also during the week, I continued my effort to prepare and submit patches upstream. See the status table above for details. Additionally, I updated the minix pkgsrc tree with changes from upstream, and I tested the new version of bmake.

In the coming week I'll be responding to feedback on the Minix pkgsrc patches, documenting how to help with moving packages to NetBSD libc, and I'll submit more patches upstream.

Hours worked: 40

Over the past week I worked on converting packages to work with NetBSD libc. I wrote a guide on the wiki (here) and asked for people to help. I tracked down a bug in Minix libc's opendir() that caused the new version of git to fail. I worked on upstreaming some patches as well.

In the coming week I'll be converting more packages to use NetBSD libc, and I will continue my upstreaming work.

Hours worked: 40

Over the past week I worked on converting packages to work with NetBSD libc. I've got bootstrap working, and next week I'll be working on pbulk and it's dependencies. While doing the conversion I discovered a few small glitches. I patched namespace.h (for shutdown()), I developed a workaround for a problem I ran into with uid_from_user() / user_from_uid(), and I discovered a problem with the way hostnames are resolved when using dhcp.

In the coming week I'll be getting pbulk working, start some bulk builds, and fix what's broken. I'll also continue upstreaming patches.

Hours worked: 42

Over the past week I worked on converting packages to work with NetBSD libc. I got pbulk working, and I did some bulk builds. With the help of Ben, pikpik, Antoine, Ivan, and others, most of the major packages are compiling now. Some packages that were broken before, work now (example: zsh). Though, some are broken in new ways; sudo, for example, doesn't work because of the password file format differences between NetBSD and Minix. It looks like Vivek will be working on that issue. Also during the week I submitted changes to gnulib.

In the coming week I'll be upstreaming new patch sets (they changed due to the switch to NetBSD libc) and finishing up the NetBSD libc conversion.

Hours worked: 43

During the beginning of last week I worked on converting packages to work with NetBSD libc. Pretty much all of the really important stuff is compiling with NetBSD libc. There are just a few minor packages left.

I spent most of the week doing upstreaming work. Many Minix specific patches were no longer needed because of NetBSD libc and the new struct stat. All of the core pkgsrc patches that needed to be redone after the switch to NetBSD libc were re-created and re-submitted. Several other patches that are still needed were submitted upstream, and some were accepted (rsync in only 1 day!), some were queued (the git patch has made it into the “pu” branch), and other are awaiting upstream developer action (acceptance, rejection, questions, feedback, etc).

Towards the end of the week I began working on gcc and binutils upstreaming. gcc is only accepting new code in the 4.7.x branch (we're currently on the 4.4.x branch), so some forward porting is needed as well as porting new dependencies (cloog, ppl, libelf, …). I'm still working on the gcc dependencies. The binutils patches are mostly ported to the newest version 2.21 from 2.17. There are still a couple of issues to resolve for binutils to build.

Last but not least, over the weekend I looked into creating a way to install all of the packages with one command. I came up with a target in /usr/Makefile, but the old way was with a package and users seem to like the old way. I changed course and worked on a meta package. The meta package idea is nice in theory, but to make the actual package (bmake package) you need to install all of the software – this takes a long time and a lot of disk space. I'm going to try another approach to creating a meta package this week.

In the coming week I'll be focusing on gcc and binutils upstreaming. During my downtime (i.e. when my VM is compiling gcc or binutils) I'll look into the minix-all package more.

Hours worked: 45

During the past week I've been working on a forward porting binutils-2.17 and gcc-4.4.5 patches to binutils-2.21 and gcc-4.7 snapshot. In order to share my work without messing up the current binutils and gcc packages, I created a wip (work-in-progress) category. The binutils package came from devel/binutils and the gcc47 package came from the pkgsrc-wip project as did wip/cloog and wip/ppl.

During the week I discovered that the pkgsrc-wip project is pretty active and manually merging the changes was a pain, so I created a repository on github (https://github.com/tcort/pkgsrc-wip) that has the pkgsrc-wip repository (master branch) and the pkgsrc-wip repository with minix changes (minix-master branch). It's not a full cvs2git conversion; I just have a script to import the daily snapshots, but it makes staying current much easier.

During the week I forward ported the patches for binutils. I ended up getting stuck – binutils compiled but didn't work. As I was debugging the issue, I got a much appreciated e-mail from Antoine with a patch that fixed the issue. During the week I tested/ported some dependencies for gcc47. There were two packages requiring patches… ppl and cloog. The ppl patches have been accepted upstream. I have submitted a patch to cloog, and I am working on another to shorten file names. Minix's NAME_MAX is 60 and some are file names are longer than 60 characters.

Also during the week a pkg_install bug was discovered by Jan Wieck. My re-write of the circular dependency checking didn't work as it needed to work. Jan Wieck tracked down the bug and sent me a patch which I've applied. So far I haven't had any issues with pkg_install. I'm going to submit the patch upstream soon.

On Friday afternoon and Saturday I worked on bulk building. The current pbulk.sh script has the system /usr/pkg in the PATH as /usr/pkg.sav. At least one package (gzip) is picking up things from this path (/usr/pkg.sav/bin/bash). I'm trying to come up with a way to bulk build without /usr/pkg.sav (which currently has gcc) and without installing gcc in the base system. I tried adding the gcc package to bootstrap, but it refused to build in bootstrap mode. I tried adding the gcc package to the binary kit, but when running bulkbuild I get a circular dependency error (even though pkg_info reports that gcc and gmp are installed). There is one more thing I want to try that I saw on the mailing list. If all else fails, I write a script to grep each package for the offending PATHs and fix the packages (mostly autoconf configure scripts).

In between everything else this week I added a “minix-all” package. The package will install every package that is known to work on Minix.

The plan for next week and beyond:

• refresh pkg_install patches on the upstream tracker
• fix cloog filenames
• forward port gcc47 patches
• pbulk without packages picking up dependencies that they shouldn't (gzip picks up /usr/pkg.sav/bin/bash but bash isn't a dependency)
• resync pkgsrc repository with upstream
• get parallel bulk build working
• port GNU emacs

Hours worked: 48

This week past week I created a new patch set for pkg_install with the updated circular dependency checking, and I submitted it to the upstream problem report system. I also fixed an issue with wip/cloog's use of filenames that are longer than 60 characters. Additionally, I did some package maintenance (reviewed and applied tk patches, patched links, etc).

I did get further with gcc47 porting. It gets to stage 2 of the build. While working on gcc I ran into a problem with binutils-2.21.1 where a specific autoconf test results in an object file that causes VM to panic. That will have to be fixed at some point, but the good news is that gcc47 doesn't seem to need such a new version of binutils (i.e. the bug won't stop me from working on gcc47).

In between my other tasks I'm still working on upstreaming changes. Recently, the git patches were merged into git's master branch. Also, some of the pkgsrc patches got more feedback. Specifically, the way we handle X11_TYPE (using “native” for a package installed with pkgsrc with some OPSYS=Minix checkes) isn't as elegant as adding a new X11_TYPE, so I'll be working on that. Thankfully, Jan Wieck got X11 going with NetBSD libc, so I can focus on just the pkgsrc changes.

The plan for next week and beyond:

• forward port gcc47 patches
• create an X11_TYPE
• pbulk without packages picking up dependencies that they shouldn't (gzip picks up /usr/pkg.sav/bin/bash but bash isn't a dependency)
• resync pkgsrc repository with upstream
• get parallel bulk build working (try distbb?)
• port GNU emacs

Hours worked: 40

This week I did another pull from dragonflybsd's git mirror of pkgsrc, and I brought our pkgsrc tree up to date with upstream. We're current as of 10 August 2011. After some minor changes to get things working again, I did a bulk build to ensure that I hadn't broken anything important. That turned out well.

Also during the week, I fixed an issue with gzip where it was picking up bash from /usr/pkg.sav/bin/bash during bulk builds and inserting “#!/usr/pkg.sav/bin/bash” at the top of the scripts. I grepped the packages after a bulk build to ensure the /usr/pkg.sav wasn't getting put in any other scripts or configuration files.

The big accomplishment this week was getting gcc47 working (C, C++, fortran). I pulled in some changes from pkgsrc-wip to use a newer snapshot, added some patches to deal with big_off_t in struct stat, and I dealt with a few other issues. I found a bug in mkdir and wrote a patch. I also tested Vivek's NetBSD mkdir port.

The coming week is the last week of Google's Summer of Code. I'll be trying to finish up some remaining tasks.

The plan for next week:

• create X11_TYPE
• fix out of memory bug when doing a git clone of pkgsrc.git
• get parallel bulk build working (try distbb?)
• try to get a few more patches upstream
• documentation clean-up
• port GNU emacs

Hours worked: 52

This week I worked on the x11 build infrastructure in pkgsrc. I added an X11_TYPE called “monolithic” for our minix/x11 package. I incorporated Jan Weick's pkgsrc changes and made additional changes to a few packages. Now building X applications seems to be working. I was able to build and run games/xbill. That package wasn't working before. I also did some light porting work to get nethack and the Atmel AVR cross compiling toolchain working on Minix. Additionally, I did some documentation clean-up on the wiki. During the week, I did a lot of debugging of the git out of memory issue. Ultimately Ben tracked it down to an issue with malloc() itself. Lastly, I did some testing of NetBSD userland utilities that I suggested be ported to Minix.

This will be my final report here as Summer of Code is coming to a close on Monday August 22, 2011. I very much enjoyed my time working for the Minix project this summer. I thank Ben Gras and Gautam Tirumala for their great support and mentorship throughout the term. I will be continuing my pkgsrc and porting work as a volunteer.

Hours worked: 45

This project doesn't have one all encompassing design. Rather, it has several small designs which are enumerated below.

For building packages, we will be using pbulk (additional documentation here). This feature is partially complete, but it still needs a little more work. A script, pbulk.sh, has some of the preliminaries.

Tasks left to complete:

• Finish the pbulk.sh script
  • pbulk tools installation (line 58 may need the correct PATH – see HOWTO-pbulk)
  • binary kit bootstrap
  • kick off the build
• Attempt building pbulk (my first attempt at building failed with Cannot find gld)
• Create a limited_list file to tell pbulk which packages to build. Initially this list will contain a list of the packages currently on the FTP server.
• Testing
  • Run pbulk on a single host with limited_list containing just figlet
  • Run pbulk on multiple hosts
  • Try a full build
• Documentation
  • Write a user guide on the wiki

The Minix project maintains it's own copy of the pkgsrc repository with minix specific changes in a git repo named pkgsrc.git. Currently, there doesn't exist an automated way of synchronizing the minix pkgsrc repo with the NetBSD pkgsrc repo, so that task is done manually.

DragonFlyBSD does a CVS→git conversion of the official NetBSD pkgsrc repository. They maintain a git branch with a pristine copy of the NetBSD pkgsrc repo. I will be merging the the DragonFlyBSD pkgsrc repo into the 'master' branch of the minix repository. That will bring the master branch up to date. From there, I will merge the changes into the minix-master branch and fix what breaks.

Tasks:

• Merge the DragonFlyBSD pkgsrc repository in the minix pkgsrc repository (master branch).
• Merge the changes from the master branch into the minix-master branch.
• Deal with the merge conflicts.
• Regenerate the digests
• Run a bulk build
• Fix build failures
• Test the software

The Minix port of pkgsrc currently lives in the minix pkgsrc repository. We want to get the minix port integrated into the mainline (NetBSD) pkgsrc. Another goal is to send patches to package source code upstream to the software maintainers.

Tasks (pkgsrc changes):

• diff master and minix-master branches to find the changes
• review the changes and look at feedback from last year.
• clean up where needed
• create patches
• send patches to tech-pkg for feedback
• respond to feedback
• submit a “Problem Report”

Tasks (other changes):

• diff master and minix-master branches to find the changes
• clean up where needed
• create patches
• send patches to the software author

To ease the transition from the classic minix libc to the minix port of NetBSD's libc, we will add a feature in pkgsrc that will mark a package as “must compile with minix libc instead of netbsd libc”. The flag will cause the compiler invocation to compile the package against minix libc instead of netbsd libc. Initially, all packages will be marked as 'must compile with minix libc'. Around week 10 of GSoC, I will begin making changes to the packages to compile with the netbsd libc and removing individual packages from the list of “must compile with minix libc” packages.

I will be adding a list of packages to a file like mk.conf, and in there I will add the code to set the proper compiler invocation for those packages. Since the compile command will change when netbsd libc becomes the default, an environment variable with the proper flags (supplied by the base system) will be helpful to ease the transition.

Tasks:

• Add environment variables MINIX_LIBC_LINK=“” and NETBSD_LIBC_LINK=“-nostdlib -lnbsd_libc” to a file in the base system that will get sourced by pkgsrc.
• determine the correct compiler invocations for minix libc and netbsd libc
• add the compiler invocation code and package list of mk.conf
• test

Packman is the former minix package manager. We want to make sure that all of the packages that packman enabled a user to install can be installed via pkgsrc. I made a list of remaining packages by diff'ing the packman package list and the binary package list from pkgsrc. These are the packman packages I found without corresponding pkgsrc packages: apache, bc, cvs, dungeon, ede, efltk, emacs, exim, fb, flawfinder, fltk, ghostscript, glib, gnu-coreutils, gv, !ImageMagick, irssi, jpeg, kermit, less, libpng, lynx, ncftp, nethack, nvi, pine, python, rcs, screen, simh, slrn, tiff, unrtf, vile, wget, Xaw3d, xv. Note, the list may get smaller. There are a few packages already created that are waiting for testing/review ( see https://github.com/pikpik/minix-pkgsrc/issues ), some packages may not be worth spending summer of code time porting (dungeon, nethack, etc), and some packages may not be worth porting at all (glib-2.10 is so old).

Tasks:

• go through the list above and determine which packman packages should be ported to pkgsrc.
• attempt to compile them
  • if success, then add to pbulk's limited_list
  • if failure
    • extract the minix fixes from the packman package
    • apply the changes to the pkgsrc package
    • do anything else required to get the package working on minix
    • send the patches upstream

As described above, we want to transition to the point where all pkgsrc packages compile against the netbsd libc. I will go through all of the packages, remove the 'must compile with minix libc' flag, and fix what breaks.

Tasks:

• Try building every package that compiles on minix with the 'must compile with minix libc' flag turned off. The list of packages can be taken from the pbulk limited_list file.
  • If a package works with netbsd libc, the option will be removed.
  • If a package doesn't work with netbsd libc, the option will stay.
• Fix any packages that don't compile with netbsd libc, removing the option as I go.

The final part of the project is to port new software applications to minix. The list of applications has yet to be determined, but it will probably include applications listed on the Wishlist and applications suggested by the community.

Tasks:

• Determine the software to be ported
• Port the software
• Submit patches upstream
• Add the package name to pbulk's limited_list file.
• Repeat

This schedule provides a general outline for how I will proceed this summer.

Community Bonding

• I'll join the pkgsrc mailing list and send an e-mail about the GSoC project and its goals. I'll ask about how to proceed with getting the minix changes upstream.
• I'm already on the minix3 google group and I'll be on IRC in #minix-pkgsrc, #minix-gsoc, #minix, and #pkgsrc

Learning

• I'll finish reading the pkgsrc documentation and experiment in my own little sandbox.
• I'll read through Gautam's changes from last summer.

Other Stuff

• I'll create a wiki page to report my progress and write up a design.
• In the days leading up to the start of Summer of Code I'll do a fresh install of Minix and update it to trunk.

Finish bulk building.

Sync the Minix pkgsrc repository with the upstream repository. Use some git magic to merge the changes, deal with the merge conflicts, regenerate the digests, run a bulk build, fix build failures, test the software, etc.

Continue fixing things that were broken as part of the sync with the upstream repository.

Prepare patches for pkgsrc upstream, submit them for review, and respond to feedback. Along with submitting patches to pkgsrc I will also submit minix-specific patches to the upstream software projects.

Continue working with upstream to get our changes accepted.

Add a feature to mark a package as “must compile with minix libc”. Then I'll run a bulk build to find out which packages compile with netbsd libc and which packages only compile with minix libc.

Port remaining packman packages.

Port remaining packman packages.

Port remaining packman packages.

Fix packages that don't compile against netbsd libc.

Fix packages that don't compile against netbsd libc.

Port new software.

Port new software.

pkgsrc docs

• The pkgsrc guide

Bulk Building

• Minix Bulk Building Guide
• pbulk section in the pkgsrc guide
• DragonFlyBSD pBulk Building Guide
• pbulk HOWTO

Minix pkgsrc docs

• Testing pkgsrc
• Volunteering to help with pkgsrc
• Guide to Converting Packages to use NetBSD libc
• Installing Binary Packages
• Porting Guide
• Syncing the Minix pkgsrc Repository with Upstream
• Google Summer of Code Project 2010 - Porting Pkgsrc to Minix

Mailing Lists

• tech-pkg mailing list
• Message to tech-pkg: Porting Pkgsrc to Minix
• Message to tech-pkg: Progress on Minix port of pkgsrc
• Message to tech-pkg: pkgsrc on Minix - Google Summer of Code Project

pkgsrc repos

• Official pkgsrc Repository (CVS)
• DragonFlyBSD pkgsrc Repository (git)
• Minix3 pkgsrc Repository (git)

Other

• pkgsrc.se Web-based Package Viewer