Student: Vivek Prakash
Mentor: Gianluca Guida
git repository: minix-userland

The goal of the project is to expand the MINIX userland by porting the userland utilities from NetBSD and replace the old and obsolete userland part of MINIX with that of NetBSD’s, while conforming to newer POSIX revisions. Minimizing the changes in the NetBSD sources themselves will also be a high priority in this project.

• I have got familiar with the git repository structure of Minix. Now, i also have a good idea of the workflows and various commands mentioned on the wiki page UsingGit.

(Please add here any NetBSD utility that you would love to see in Minix.)

• file
• sh - Depends on libterminfo, libedit, libl, test, printf, kill, sysctl
• ping
• netstat - In progress
• traceroute - In progress
• sort (the current /usr/bin/sort utility doesn't work with a 20MB test file I got from the gnulib team ~ tcort)

• Ported netinet/in_system.h . Tried to port netinet/ip.h as it is required for telnet, and studied it's dependencies and consequences. NetBSD ip.h depends on it's own fully-fledged TCP/IP stack, and hence it would require changes in the source of telnet to make it compatible with the TCP/IP stack of Minix.
• Examined request types supported in ioctl() system call in Minix in more detail.
• Fixed _MINIX conditional in termios.h.

• Worked on porting df. It is heavily dependent on NetBSD statvfs and corresponding syscalls and functions e.g. getmntinfo(). Mount related headers and control codes e.g. MNT_WAIT, MNT_NOWAIT, etc. are not supported in Minix, and the mount structure is different too, which makes it even harder to port. I am giving a good and thorough look at all the dependencies and trying to determine minimum changes with many functionality reduced in the process.

• In the first half of week, i mainly worked on updating the hier(7) man page which is now available at the tip of the master branch in my github repository : git@github.com:vivekp/minix-nbsd . Along with, i examined df , tracerout, sh and their dependencies in more detail and will be working ahead on porting them. I lost some time in the second half of the week due to illness and will try hard to compensate for the lost time in this week itself.

This week i mostly worked on submitting the changes upstream. I cleaned up the code for all the previous utilities ported, refined their commits and rebased them on master. I tested for 'make world' and native tests after installing each of the NetBSD utility. I also compared the NetBSD utilities with the corresponding Minix utility and looked for missing flags which are required.

I had ported mkdep earlier, but i didn't test it thoroughly that time. After switching to ELF, i tested it again, but it is giving errors while generating the .depend file. I am looking into it. I also tested xinstall, but it is giving errors for root user. I need to look into it too. I have ported cksum, and it is giving consistent results with all the hashes. I cleaned up the NetBSD libutil code, to submit it upstream next week.

I have prioritized the submission of these utilities to upstream next week: printf, chmod, mkdir, pwd, banner, basename, ctags, xstr, yes, ln, ls, cp, dd, column, kill, cksum, libutil

I have prioritized the porting/testing of these utilities next week: mkdep, xinstall, ping, telnet

Over the past week, i worked on cleaning up more code and submitting more utilities for integrating in upstream. I submitted the patches of ls, stat, uniq, sleep and cat for review. I completed the port of xinstall, and worked on porting ping and telnet. ping works fine, except for the statistics printed. I also worked on mkdep, but during dependency file generation it is picking up not-supposed-to arguments causing it to fail.

I ported mtree, mknod and tested them successfully. mtree can now replace the mtree.sh script.

I was able to build the whole system using NetBSD xinstall and create a bootable image without any error. For that i needed to replace all the INSTALLFLAGS with -S by chmem and equivalent stack size. I also enabled stripping of symbol table from the ack compiled executable, as the /usr/pkg/bin/strip only works with elf and other formats. One problem that emerged during the port was that NetBSD functions uid_from_user(), user_from_uid(), etc. in nbsd_libc don't work in Minix mainly due to non netbsd style passwd file. I got them working in xinstall using local tweaks, but they aren't nice.

So, in the next week, my goal is to submit the remaining ported utilities for review. And, i will work next on migration to native NetBSD passwd format.(xinstall also relies on it.) For that, i will first port the corresponding utilities e.g. passwd, useradd, userdel, usermod, chpass, vipw, pwd_mkdb . Then, i will migrate the /etc/passwd file to NetBSD format, while ensuring backward compatibility.

This week i completed the port of mkdep, and refined the patches of ls and stat after new changes to struct stat. I also ported libterminfo and terminfo database this week.

Then, i worked on migration to the NetBSD style passwd format. I ported passwd, pwd_mkdb, and nbsd_libutil for this purpose. Then, i switched to NetBSD format passwd and master.passwd file. master.passwd file makes the /etc/shadow file obsolete. Now that, i had master.passwd file, i created pwd.db and spwd.db (password databases) using pwd_mkdb. Then, i removed the related functions in nbsd_libcompat_minix and used nbsd_libc functions . Now, the uid/guid⇔ name functions are working properly as password databases are present.

The work left ahead is to get Minix' login to work with new format, or port NetBSD login. I will also work ahead on ensuring compatibility with old binaries and testing that the system doesn't break after the complete migration.

This week i ported the NetBSD login. I got stalled a few days in it- trying to make sockaddr_snprintf() in libutil work, and partially in examining the problem in Minix login. I also lost some time in moving to college.

login works with the new NetBSD passwd format at the end of the week, but old utilities in base system are broken.

This week i was able to produce an initial version of steps for migrating to NetBSD passwd format. In addition, i added fallback functions to nbsd_libc getpw*() functions to ensure that somebody doesn't get locked out of the system if he doesn't switch to new passwd format after make world; reboot. I also identified all users of pwdauth and crypt() from old libc, and removed pwdauth and crypt(). After much testing, i found out that keeping NetBSD login, new libc, fallback functions in new libc, and retaining getpw*() functions in old libc keeps everything intact.

This week i mainly worked over fixing problems in the initial version of commit for switching to NetBSD passwd format. I ported NetBSD su and newgrp. I fixed user* and group* utilities for flock() error. I came to know that a process can only hold exclusive lock over a file only if it's open in write mode. I also wrote a shell script that fixes the clobbering of existing accounts.

This week i came to know of more problems in the passwd branch, and spent most of the time fixing them. I got Minix su working with new libc. Also, i ported ctags, spell, spellprog and deroff. They are available in the ctags and spell branch in the repository.

Over past week, i imported most of the utilities suggested by others. I imported bzip2, libbz2, hexdump, od, tput, tset, col, hostname, domainname, infocmp, col, du, and cksum. I also imported NetBSD sort, but it is failing the pipeline test with the error ENOTTY. Gdb helped in tracking the problem to tcsetpgrp, and i hope to fix it soon.

Userland refers to all application software that runs in user space. Userland usually refers to the various programs and libraries that the operating system uses to interact with the kernel: software that performs input/output, manipulates file system objects, etc. (Adapted from Wikipedia).

I have planned to complete the project in four phases :

• Identifying the trivial and MINIX independent/supported userland utilities, and porting them first.
• Then, grouping the utilities according to libraries needed, which are either not present in MINIX or not supported by MINIX currently.
• Then, porting the needed library. This is an iterative process, since one library may depend on other in turn, which needs to be ported first. Porting the library may not be the end, as many syscalls and functions are not supported in MINIX which may need to be implemented as well.
• Then, porting all the userland utilities dependent on the libraries and missing functions in the previous step.

All the userland utilities are currently present in /usr/src/commands in MINIX. Following the current directory structure and the modular approach used to design the whole MINIX operating system, the directory structure of the ported userland component of the NetBSD source tree should be imitated from the NetBSD source tree itself with a few modifications.

The userland part of the MINIX source tree should be organized as following:

• /usr/src/commands: This directory has currently all the sources for the utilities present in MINIX, without any groupings. The utilities in it would be removed whenever the corresponding utilities from NetBSD are ported.
• Now, there would be four more directories in /usr/src , namely bin, sbin, usr.bin, usr.sbin taken directly from the NetBSD userland source tree.
• /usr/src/bin: Similar to NetBSD’s /usr/src/bin .
• /usr/src/sbin: Similar to NetBSD’s /usr/src/sbin .
• /usr/src/usr.bin: Similar to NetBSD’s /usr/src/usr.bin .
• /usr/src/usr.sbin: Similar to NetBSD’s /usr/src/usr.sbin .
• GNU-licensed utilities from NetBSD’s /usr/src/gnu can be replaced with utilities from other userland which are licensed under BSD. If this is not possible at all, making packages out of them is an option.

Advantage : This directory structure for MINIX offers the advantage that it separates the MINIX userland utilities from those ported from NetBSD, and thus offers clarity in the structure. Moreover, the MINIX utilities may coexist with NetBSD utilities, while the compilation for former may be disabled if the NetBSD’s are newer and superior.

The plan is to checkout the userland source from a stable release of NetBSD and port utilities contained in that source. A minix-port.patch file would be present in the top level directory of a NetBSD component that contains the changes done to that component when porting to MINIX.

A system has been set up by Gianluca Guida so that one can easily track the version of NetBSD's libc currently being used. It would be convenient for the developers to modify the nbsd sources in the minix tree and update the minix-port patches accordingly. He has also automated the minix-port.patch creation. On the same lines, the userland source version will be tracked and minix-port.patch file will be generated in each of the directories: bin, sbin, usr.bin, usr.sbin .

Initially, empty directories /usr/src/bin, /usr/src/sbin, /usr/src/usr.bin, /usr/src/usr.sbin will be created. As soon as a utility from NetBSD userland is ported to MINIX , it would be added to the corresponding directory. The changes would be pushed upstream after a few utilities have been ported and minix-port.patch has been modified accordingly in each of the four directories.

Also, I would maintain a list of all the NetBSD userland utilities that have been ported in the /usr/src/commands/ directory, so that one knows where to look for the source of missing utilities in commands.

• The separate system for tracking the the userland source version will be same as that for tracking the nbsd_libc version. A separate reference source tree system need to be set up because the userland may need to be updated to newer release, which may have fixed bugs in few utilities while the older nbsd_libc is absolutely compatible with the newer release, or the nbsd_libc hasn’t changed at all in the newer release.
• This implies that whenever a newer release of NetBSD comes, we would have to deal with merge conflicts, fix them, and check all the utilities affected by merge whether they compile now or not. If they don’t compile, fix them and send all the changes upstream.
• Related Tasks :
  1. Switch to ELF based system
  2. Set up the system for tracking the userland source version analogous to that for nbsd_libc

Here is an approximate schedule of how i will work on the project:

• I will take a deep and thorough look at minix-nbsd git repository.
• I will be active on MINIX mailing list and IRC channels - #minix, #minix-gsoc and discuss any doubts that i may have.
• I will set up a new machine, install MINIX on it and update it to the trunk before the start of the Coding Period.
• I will discuss with my mentor about the modifications needed in existing design and planning for the project.
• In between, i will identify easy to port utilities, and port them if i get enough time.

Complete the dependency table of all the important userland utilities, which will list missing references to functions, missing control codes and NetBSD libraries to port.

Port all the userland utilities, whose dependencies are trivial or minimal.

Port as many NetBSD libraries as possible which are listed in the dependency table.

Learn about implementing the missing functions and control codes and implement them. Or, find the workarounds of doing the same if implementing the function is not feasible/supported in MINIX.

Continue porting the required NetBSD libraries, if there are any.
Port the userland utilities whose dependencies have been resolved.

Port more userland utilities.

Test all the userland utilities ported till now, and fixing any bugs encountered in the process.

Complete the mid-term evaluation.
Check if the MINIX base utilities compile with the new ported libraries. Discuss with the mentor if there are issues in it.

Release the ported utilities to the MINIX community for testing.
Complete the implementation of missing functions and control codes, or find their alternatives and code them.

Port more userland utilities in which references to missing functions and Macros have been fixed.

Documenting all the work done till now.
Port more utilities.

Port more utilities and test them.

Test all the ported userland utilities and releasing them to the community to get feedbacks.
Clean the code, and improve the documentation.

• Sources of NetBSD release versions
• NetBSD's src/share/mk/bsd.README
• Source of NetBSD-current development branch
• http://wiki.minix3.org/en/[[:DevelopersGuide:PortingGuide]]
• NetBSD userland tour
• http://www.opengroup.org/onlinepubs/000095399
• http://www.gnu.org/prep/standards/standards.html
• Makefile Conventions