Student: Thomas Cort
Mentor: Ben Gras and Gautam Tirumala
git repository: minix-pkgsrc and pkgsrc
The primary goals of this project are to complete any remaining tasks for the minix pkgsrc port, send the changes upstream, and port more software applications to minix. Along the way, the minix pkgsrc tree will be resynced with the upstream tree, and changes will be made to ease the transition from the classic Minix libc to the Minix port of NetBSD's libc.
|☐||pbulk style bulk builds||80%||Bulk building on a single machine works well. Parallel bulk builds still needs some work. Documented in the Minix Bulk Building Guide|
|☑ ☑ ☑||resync the minix pkgsrc tree with upstream||100%||Completed 3 times and documented in Syncing the Minix pkgsrc Repository with Upstream. Latest pull from upstream was August 10, 2011.|
|☐||prepare and submit changes upstream||50%||In progress. See upstreaming status table below for details.|
|☑||feature to mark a pkg as must compile with minix libc||100%||This feature is gone now. Everything uses NetBSD libc.|
|☑||fix packages that don't compile against NetBSD libc||100%||All of the packages that compiled and worked with Minix libc now compile with NetBSD libc.|
|☐||port remaining packman packages to minix||10%||Some just worked with NetBSD libc, I fixed a few, and users submitted patches. In each case, the package was added to the bulk build system's build list.|
|☐||port new software applications to minix||10%||In my free time I tried building a lot of packages. There were a lot that built with little or no changes after the NetBSD libc switch. I also ported a few. Our bulk build list more than doubled, from 101 packages in May 2011.|
The table below lists packages with changes to be upstreamed. The following packages have been upstreamed or the patches have been eliminated (for example, by switching to NetBSD libc): archivers/bsdtar, archivers/gtar-base, devel/cscope, devel/gmp, devel/gtexinfo, devel/ncurses, devel/patch, devel/pcre, devel/scmgit, devel/scmgit-base, editors/zile, games/cmatrix, games/ninvaders, games/sex, graphics/tiff, ham/locator, lang/gawk, misc/figlet, net/rsync, pkgtools/pkgin, textproc/gsed, textproc/wdiff, wip/ppl.
|archivers/libarchive||☑||☑||☐||patch||Rewrote the patch to just display a warning since I'm not able to reproduce the segfaults. Problem Report: pkg/45044 Issue: 158|
|archivers/pax||☑||☑||☐||patch||Submitted again after NetBSD libc Changes. Problem Report: pkg/45065|
|archivers/unzip||☑||☑||☐||patch||Submitted. I'll have a link once it gets through the moderator approval queue. Need to redo patch after NetBSD libc changes.|
|bootstrap||☑||☑||☐||patch||Submitted again after NetBSD libc Changes. Problem Report: pkg/45041|
|databases/db4||☑||☑||☐||patch||Submitted. Problem Report: pkg/45087|
|databases/sqlite3||☐||☐||☐||Has just a couple of small patches.|
|devel/apr||☐||☐||☐||A few obvious changes.|
|devel/autoconf||☑||☐||☐||patch||The only patch left changes the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting this patch.|
|devel/binutils||☐||☐||☐||In progress. I'm Currently forward porting the patches to the latest release.|
|devel/bison||☑||☑||☐||Submitted spawni.c change request to gnulib.|
|devel/bmake||☑||☑||☐||patch||Received some questions and submitted a response. Problem Report: pkg/45042 No changes since the switch to NetBSD libc.|
|devel/flex||☐||☐||☐||We disable the PIC library by removing the code. This will need to be reworked before upstreaming.|
|devel/gettext||☑ ☐||☑ ☐||☐ ☐||Submitted spawni.c change request to gnulib. IPv6 change can go upstream to gettext.|
|devel/m4||☑ ☐||☑ ☐||☐ ☐||Submitted spawni.c change request to gnulib. Need to prepare stdio-impl.h change.|
|devel/pth||☐||☐||☐||Most changes are due to a lack of wait4(). It looks like the new libc has that function. I'll do some testing. The other change is MSG_PEEK related. After testing with the new libc, I'll prepare and upstream the MSG_PEEK change.|
|devel/subversion-base||☐||☐||☐||Patch only changes the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting this patch.|
|editors/mg||☐||☐||☐||Some changes might be taken care of by NetBSD libc, others will definitely be taken care of by NetBSD libc. I'll hold off on upstreaming these changes until I can test with NetBSD libc.|
|editors/vim-share||☐||☐||☐||Only patch is for statfs(). Should work out of the box with NetBSD libc.|
|games/gnugo||☐||☐||☐||The changes only include a simple header fix which shouldn't be needed once we switch to NetBSD libc.|
|graphics/netpbm||☐||☐||☐||nameEntry.sin_zero #ifdef should use autoconf. Other changes look good for upstreaming. I'll fix the sin_zero thing and prepare patches.|
|lang/gcc44||☐||☐||☐||In progress. I'm Currently forward porting the patches to the latest release (gcc.7.x).|
|lang/perl5||☑ ☐||☐ ☐||☐ ☐||patch||Thanks to the new C library, we're down to 1 minix specific code patch. Will submit soon. / The other changes are in the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting the other changes.|
|misc/getopt||☑||☐||☐||patch||Simple fix to add “-liconv”. Will send upstream soon.|
|mk||☑||☑||☐ ☐||patch patch||Submitted again after NetBSD libc Changes. Problem Reports: pkg/45046 pkg/45079|
|net/libfetch||☑||☑||☐||patch||Submitted. Problem Report: pkg/45039 No changes since the switch to NetBSD libc.|
|pkgtools/bootstrap-mk-files||☑||☑||☐ ☑||patch patch||Received some questions and submitted a response. bootstrap.sh patch accepted. Problem Reports: pkg/45045, pkg/45061. Commits: commit No changes since the switch to NetBSD libc.|
|pkgtools/libnbcompat||☑||☑||☐||patch||Submitted again after NetBSD libc Changes. Problem Report: pkg/45043|
|pkgtools/pbulk||☐||☐||☐||Changes are parallel bulk build related. Holding off on upstreaming the changes until it's working.|
|pkgtools/pbulk-base||☐||☐||☐||Changes are parallel bulk build related. Holding off on upstreaming the changes until it's working.|
|pkgtools/pkg_install||☑||☑||☐ ☐ ☐||patch patch patch||Submitted again after NetBSD libc Changes. Problem Reports: pkg/45047 pkg/45077 pkg/45078|
|print/ghostscript||☑||☐||☐||patch||Patch only changes the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting this patch.|
|security/openssh||☐||☐||☐||Lots of header related changes. Hopefully NetBSD libc will eliminate some of those.|
|security/openssl||☐||☐||☐||Only a few source code and configuration changes. Patch coming soon.|
|security/sudo||☐||☐||☐||Got this one ported after the version bump broken the package on Minix. It currently doesn't work right due to the conflict between the password format used by NetBSD libc and the Minix password format. It should fix itself with when the NetBSD password format is in use. Waiting to submit patches until it works.|
|shells/bash||☑ ☑||☑ ☐||☑ ☐||patch||Submitted job_control patch to bash mailing list, and I got a thank you so I assume it's in (I can't find a public source repository for bash-current). The only patch left changes configure args in the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting this patch.|
|shells/bash2||☑||☐||☐||patch||The only patch left changes configure args in the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting this patch.|
|shells/zsh||☑||☐||☐||patch||Cleaned up the patch a bit (use HAVE_SETRLIMIT, HAVE_SETPGRP, etc). It doesn't apply cleanly to the latest trunk. I'll try patching zsh-current.|
|wip/cloog||☑||☑||☐||patch||Patch Submitted. A second patch has been created to limit the file names to 60 characters (NAME_MAX). That will need to be submitted too.|
|www/neon||☑||☐||☐||patch||Patch only changes configure args in the package's Makefile. Waiting until core pkgsrc changes are upstream before submitting this patch.|
This week I spent most of my time focused on pbulk. I completed the pbulk.sh script in order to automate the bulk building process and to make it easier to get consistent results. I was able to successfully complete several bulk builds. Along the way I did several minor fixes: update LD and AR in bootstrap-mk-files, fix the cmake dependency in compiler-rt, and set the proper permissions on /usr.
I created a limited_list file (/usr/pkgsrc/minix/limited_list.pbulk) to tell pbulk which packages to build. The list contains every package that is expected to work on minix. I ran a full build of all of those packages. There were some packages that failed to build. Most of the errors seemed to be related to minor changes in the base system. I will investigate the problems more deeply in the next week or two after I update the minix pkgsrc repository with the upstream changes. I want to avoid fixing the same package twice.
I also worked on parallel bulk building. The documentation says to use NFS when running pbulk across several machines. Since minix doesn't have NFS, I added rysnc commands to push packages and logs between the clients and the master. I can start the build, but when it builds the initial dependency tree I get messages about select() failing. Then when the build starts, it halts with an error about circular dependencies. The upstream pkgsrc repository has a version of pbulk that is several revisions newer than what I've been working with. When I update the minix pkgsrc repository with the upstream changes, I'll come back to this.
Hours worked: 44
This week I merged the upstream pkgsrc changes into the minix pkgsrc repository. I fixed a few breakages, did some light porting, adjusted a couple of patches, did some bulk building, and some testing. The updates/changes have been pushed out to the minix pkgsrc repository.
There were a few bumps along the way. I ran into a problem building perl module packages. This is caused by files in binary packages having bad timestamps (year 2038). I'm still looking into this, but it was a problem that was happening before I started the merge. security/sudo needs some more porting work. It too was broken before the merge. I wasn't able to test everything because some packages have been ported to elf and I was using an a.out system. I did test some packages where it was easy to temporarily back out the ELF changes, like openssl.
In the coming week I will work on fixing the file timestamp problem mentioned above, and I will work on upstreaming minix changes.
Hours worked: 48
This week I tracked down and fixed the libarchive timestamp problem mentioned in my previous report. I also installed Minix 3.2.0 (elf), and I did a bulk build of pkgsrc packages with decent results. My main focus this week was upstreaming minix changes.
The Upstreaming Status table above was generated by git diffstat master minix-master and some command line magic. It lists all of the packages with minix changes and where I'm at in terms of getting those changes upstream. As of June 12, I've submitted 10 patches and 2 have been accepted.
In the coming week I will continue preparing and submitting patches as well as responding to feedback from upstream.
Hours worked: 42
Over the past week I've been working on upstreaming Minix changes to pkgsrc. That involved responding to feedback, asking questions, re-writing patches, submitting problem reports, investigating issues, and doing testing and building. I also began adding a feature to mark a package as must compile with minix libc.
In the coming week I'll be responding to feedback on the Minix pkgsrc patches, I'll finish off the feature to mark a package as must compile with minix libc, I'll do some builds against the new NetBSD-based libc, and I'll submit more patches upstream.
Hours worked: 40
Over the past week I finished implementing the feature to mark a package as must compile with minix libc. I'll be writing up a guide for users interested in helping build/test software with the new C library. Also during the week, I continued my effort to prepare and submit patches upstream. See the status table above for details. Additionally, I updated the minix pkgsrc tree with changes from upstream, and I tested the new version of bmake.
In the coming week I'll be responding to feedback on the Minix pkgsrc patches, documenting how to help with moving packages to NetBSD libc, and I'll submit more patches upstream.
Hours worked: 40
Over the past week I worked on converting packages to work with NetBSD libc. I wrote a guide on the wiki (here) and asked for people to help. I tracked down a bug in Minix libc's opendir() that caused the new version of git to fail. I worked on upstreaming some patches as well.
In the coming week I'll be converting more packages to use NetBSD libc, and I will continue my upstreaming work.
Hours worked: 40
Over the past week I worked on converting packages to work with NetBSD libc. I've got bootstrap working, and next week I'll be working on pbulk and it's dependencies. While doing the conversion I discovered a few small glitches. I patched namespace.h (for shutdown()), I developed a workaround for a problem I ran into with uid_from_user() / user_from_uid(), and I discovered a problem with the way hostnames are resolved when using dhcp.
In the coming week I'll be getting pbulk working, start some bulk builds, and fix what's broken. I'll also continue upstreaming patches.
Hours worked: 42
Over the past week I worked on converting packages to work with NetBSD libc. I got pbulk working, and I did some bulk builds. With the help of Ben, pikpik, Antoine, Ivan, and others, most of the major packages are compiling now. Some packages that were broken before, work now (example: zsh). Though, some are broken in new ways; sudo, for example, doesn't work because of the password file format differences between NetBSD and Minix. It looks like Vivek will be working on that issue. Also during the week I submitted changes to gnulib.
In the coming week I'll be upstreaming new patch sets (they changed due to the switch to NetBSD libc) and finishing up the NetBSD libc conversion.
Hours worked: 43
During the beginning of last week I worked on converting packages to work with NetBSD libc. Pretty much all of the really important stuff is compiling with NetBSD libc. There are just a few minor packages left.
I spent most of the week doing upstreaming work. Many Minix specific patches were no longer needed because of NetBSD libc and the new struct stat. All of the core pkgsrc patches that needed to be redone after the switch to NetBSD libc were re-created and re-submitted. Several other patches that are still needed were submitted upstream, and some were accepted (rsync in only 1 day!), some were queued (the git patch has made it into the “pu” branch), and other are awaiting upstream developer action (acceptance, rejection, questions, feedback, etc).
Towards the end of the week I began working on gcc and binutils upstreaming. gcc is only accepting new code in the 4.7.x branch (we're currently on the 4.4.x branch), so some forward porting is needed as well as porting new dependencies (cloog, ppl, libelf, …). I'm still working on the gcc dependencies. The binutils patches are mostly ported to the newest version 2.21 from 2.17. There are still a couple of issues to resolve for binutils to build.
Last but not least, over the weekend I looked into creating a way to install all of the packages with one command. I came up with a target in /usr/Makefile, but the old way was with a package and users seem to like the old way. I changed course and worked on a meta package. The meta package idea is nice in theory, but to make the actual package (bmake package) you need to install all of the software – this takes a long time and a lot of disk space. I'm going to try another approach to creating a meta package this week.
In the coming week I'll be focusing on gcc and binutils upstreaming. During my downtime (i.e. when my VM is compiling gcc or binutils) I'll look into the minix-all package more.
Hours worked: 45
During the past week I've been working on a forward porting binutils-2.17 and gcc-4.4.5 patches to binutils-2.21 and gcc-4.7 snapshot. In order to share my work without messing up the current binutils and gcc packages, I created a wip (work-in-progress) category. The binutils package came from devel/binutils and the gcc47 package came from the pkgsrc-wip project as did wip/cloog and wip/ppl.
During the week I discovered that the pkgsrc-wip project is pretty active and manually merging the changes was a pain, so I created a repository on github (https://github.com/tcort/pkgsrc-wip) that has the pkgsrc-wip repository (master branch) and the pkgsrc-wip repository with minix changes (minix-master branch). It's not a full cvs2git conversion; I just have a script to import the daily snapshots, but it makes staying current much easier.
During the week I forward ported the patches for binutils. I ended up getting stuck – binutils compiled but didn't work. As I was debugging the issue, I got a much appreciated e-mail from Antoine with a patch that fixed the issue. During the week I tested/ported some dependencies for gcc47. There were two packages requiring patches… ppl and cloog. The ppl patches have been accepted upstream. I have submitted a patch to cloog, and I am working on another to shorten file names. Minix's NAME_MAX is 60 and some are file names are longer than 60 characters.
Also during the week a pkg_install bug was discovered by Jan Wieck. My re-write of the circular dependency checking didn't work as it needed to work. Jan Wieck tracked down the bug and sent me a patch which I've applied. So far I haven't had any issues with pkg_install. I'm going to submit the patch upstream soon.
On Friday afternoon and Saturday I worked on bulk building. The current pbulk.sh script has the system /usr/pkg in the PATH as /usr/pkg.sav. At least one package (gzip) is picking up things from this path (/usr/pkg.sav/bin/bash). I'm trying to come up with a way to bulk build without /usr/pkg.sav (which currently has gcc) and without installing gcc in the base system. I tried adding the gcc package to bootstrap, but it refused to build in bootstrap mode. I tried adding the gcc package to the binary kit, but when running bulkbuild I get a circular dependency error (even though pkg_info reports that gcc and gmp are installed). There is one more thing I want to try that I saw on the mailing list. If all else fails, I write a script to grep each package for the offending PATHs and fix the packages (mostly autoconf configure scripts).
In between everything else this week I added a “minix-all” package. The package will install every package that is known to work on Minix.
The plan for next week and beyond:
Hours worked: 48
This week past week I created a new patch set for pkg_install with the updated circular dependency checking, and I submitted it to the upstream problem report system. I also fixed an issue with wip/cloog's use of filenames that are longer than 60 characters. Additionally, I did some package maintenance (reviewed and applied tk patches, patched links, etc).
I did get further with gcc47 porting. It gets to stage 2 of the build. While working on gcc I ran into a problem with binutils-2.21.1 where a specific autoconf test results in an object file that causes VM to panic. That will have to be fixed at some point, but the good news is that gcc47 doesn't seem to need such a new version of binutils (i.e. the bug won't stop me from working on gcc47).
In between my other tasks I'm still working on upstreaming changes. Recently, the git patches were merged into git's master branch. Also, some of the pkgsrc patches got more feedback. Specifically, the way we handle X11_TYPE (using “native” for a package installed with pkgsrc with some OPSYS=Minix checkes) isn't as elegant as adding a new X11_TYPE, so I'll be working on that. Thankfully, Jan Wieck got X11 going with NetBSD libc, so I can focus on just the pkgsrc changes.
The plan for next week and beyond:
Hours worked: 40
This week I did another pull from dragonflybsd's git mirror of pkgsrc, and I brought our pkgsrc tree up to date with upstream. We're current as of 10 August 2011. After some minor changes to get things working again, I did a bulk build to ensure that I hadn't broken anything important. That turned out well.
Also during the week, I fixed an issue with gzip where it was picking up bash from /usr/pkg.sav/bin/bash during bulk builds and inserting “#!/usr/pkg.sav/bin/bash” at the top of the scripts. I grepped the packages after a bulk build to ensure the /usr/pkg.sav wasn't getting put in any other scripts or configuration files.
The big accomplishment this week was getting gcc47 working (C, C++, fortran). I pulled in some changes from pkgsrc-wip to use a newer snapshot, added some patches to deal with big_off_t in struct stat, and I dealt with a few other issues. I found a bug in mkdir and wrote a patch. I also tested Vivek's NetBSD mkdir port.
The coming week is the last week of Google's Summer of Code. I'll be trying to finish up some remaining tasks.
The plan for next week:
Hours worked: 52
This week I worked on the x11 build infrastructure in pkgsrc. I added an X11_TYPE called “monolithic” for our minix/x11 package. I incorporated Jan Weick's pkgsrc changes and made additional changes to a few packages. Now building X applications seems to be working. I was able to build and run games/xbill. That package wasn't working before. I also did some light porting work to get nethack and the Atmel AVR cross compiling toolchain working on Minix. Additionally, I did some documentation clean-up on the wiki. During the week, I did a lot of debugging of the git out of memory issue. Ultimately Ben tracked it down to an issue with malloc() itself. Lastly, I did some testing of NetBSD userland utilities that I suggested be ported to Minix.
This will be my final report here as Summer of Code is coming to a close on Monday August 22, 2011. I very much enjoyed my time working for the Minix project this summer. I thank Ben Gras and Gautam Tirumala for their great support and mentorship throughout the term. I will be continuing my pkgsrc and porting work as a volunteer.
Hours worked: 45
This project doesn't have one all encompassing design. Rather, it has several small designs which are enumerated below.
For building packages, we will be using pbulk (additional documentation here). This feature is partially complete, but it still needs a little more work. A script, pbulk.sh, has some of the preliminaries.
Tasks left to complete:
The Minix project maintains it's own copy of the pkgsrc repository with minix specific changes in a git repo named pkgsrc.git. Currently, there doesn't exist an automated way of synchronizing the minix pkgsrc repo with the NetBSD pkgsrc repo, so that task is done manually.
DragonFlyBSD does a CVS→git conversion of the official NetBSD pkgsrc repository. They maintain a git branch with a pristine copy of the NetBSD pkgsrc repo. I will be merging the the DragonFlyBSD pkgsrc repo into the 'master' branch of the minix repository. That will bring the master branch up to date. From there, I will merge the changes into the minix-master branch and fix what breaks.
The Minix port of pkgsrc currently lives in the minix pkgsrc repository. We want to get the minix port integrated into the mainline (NetBSD) pkgsrc. Another goal is to send patches to package source code upstream to the software maintainers.
Tasks (pkgsrc changes):
Tasks (other changes):
To ease the transition from the classic minix libc to the minix port of NetBSD's libc, we will add a feature in pkgsrc that will mark a package as “must compile with minix libc instead of netbsd libc”. The flag will cause the compiler invocation to compile the package against minix libc instead of netbsd libc. Initially, all packages will be marked as 'must compile with minix libc'. Around week 10 of GSoC, I will begin making changes to the packages to compile with the netbsd libc and removing individual packages from the list of “must compile with minix libc” packages.
I will be adding a list of packages to a file like mk.conf, and in there I will add the code to set the proper compiler invocation for those packages. Since the compile command will change when netbsd libc becomes the default, an environment variable with the proper flags (supplied by the base system) will be helpful to ease the transition.
Packman is the former minix package manager. We want to make sure that all of the packages that packman enabled a user to install can be installed via pkgsrc. I made a list of remaining packages by diff'ing the packman package list and the binary package list from pkgsrc. These are the packman packages I found without corresponding pkgsrc packages: apache, bc, cvs, dungeon, ede, efltk, emacs, exim, fb, flawfinder, fltk, ghostscript, glib, gnu-coreutils, gv, !ImageMagick, irssi, jpeg, kermit, less, libpng, lynx, ncftp, nethack, nvi, pine, python, rcs, screen, simh, slrn, tiff, unrtf, vile, wget, Xaw3d, xv. Note, the list may get smaller. There are a few packages already created that are waiting for testing/review ( see https://github.com/pikpik/minix-pkgsrc/issues ), some packages may not be worth spending summer of code time porting (dungeon, nethack, etc), and some packages may not be worth porting at all (glib-2.10 is so old).
As described above, we want to transition to the point where all pkgsrc packages compile against the netbsd libc. I will go through all of the packages, remove the 'must compile with minix libc' flag, and fix what breaks.
The final part of the project is to port new software applications to minix. The list of applications has yet to be determined, but it will probably include applications listed on the Wishlist and applications suggested by the community.
This schedule provides a general outline for how I will proceed this summer.
Finish bulk building.
Sync the Minix pkgsrc repository with the upstream repository. Use some git magic to merge the changes, deal with the merge conflicts, regenerate the digests, run a bulk build, fix build failures, test the software, etc.
Continue fixing things that were broken as part of the sync with the upstream repository.
Prepare patches for pkgsrc upstream, submit them for review, and respond to feedback. Along with submitting patches to pkgsrc I will also submit minix-specific patches to the upstream software projects.
Continue working with upstream to get our changes accepted.
Add a feature to mark a package as “must compile with minix libc”. Then I'll run a bulk build to find out which packages compile with netbsd libc and which packages only compile with minix libc.
Port remaining packman packages.
Port remaining packman packages.
Port remaining packman packages.
Fix packages that don't compile against netbsd libc.
Fix packages that don't compile against netbsd libc.
Port new software.
Port new software.
Minix pkgsrc docs