soc:2011:netbsduserland [Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== NetBSD ======
Student: Vivek Prakash\\  
Mentor: Gianluca Guida\\ 
git repository: [[http://git.minix3.org/?p=minix-userland.git;a=log;h=refs/heads/userland|minix-userland]]\\

===== Abstract =====
The goal of the project is to expand the MINIX userland by porting the userland utilities from NetBSD and replace the old and obsolete userland part of MINIX with that of NetBSD’s, while conforming to newer POSIX revisions. Minimizing the changes in the NetBSD sources themselves will also be a high priority in this project.

===== Current Status =====
|**Done** |**Item** |**Percentage Complete** |**Comments** |**Status** |
| {o} |Switch to ELF based system |100% | | |
| {o} |Setup userland reference source system |100% | | |
| {o} |Port initially suggested utilities |80% | |In progress |
| {o} |List utilities whose dependencies are minimal |80% | |In progress |
| {o} |Port utilities listed above |70% | |In progress |
| {o} |List utilities whose dependencies are high |50% | | |
| {o} |Port required libraries and implement missing functions |60% | |In progress |
| {o} |Migrate to NetBSD passwd format |95% | | |


===== Status Reports =====
==== Pre-Coding Period ====
  * I have got familiar with the git repository structure of Minix. Now, i also have a good idea of the workflows and various commands mentioned on the wiki page [[http://wiki.minix3.org/en/DevelopersGuide/UsingGit|UsingGit]].

==== Coding Period ====
=== Possible userland tasks ===
(//Please add here any NetBSD utility that you would love to see in Minix.//)
  * file
  * sh - //Depends on libterminfo, libedit, libl, test, printf, kill, sysctl//
  * ping
  * netstat - //In progress//
  * traceroute - //In progress//
  * sort (the current /usr/bin/sort utility doesn't work with a 20MB test file I got from the gnulib team ~ tcort)

=== Week 1 ===
|**Utilites** |**Dependencies*** |**Remarks** |**Porting status** |**Testing status** |
|uniq |none |Required trivial changes to Makefile |Done |Done |
|m4 |libutil, libl |Overrode compilation with Minix libutil and libl libraries |Done |- |
|mkdep |vfork() |Used fork() instead of vfork(), as it was appropriate in context |Done |- |
|indent |none |Changes to Makefile |Done |Done |
|stat |struct stat |Enabled Minix Emulated NetBSD stat |Done |- |
|libterminfo |stat utility |Removed shlib dependency; forced static linking |Done |- |
|libedit |libterminfo |Changes to Makefile |Done |- |
|printf |none |,, |Done |Done |
|kill |none |,, |Done |Done |
|test |csh |Modified testing script for 'test' to also satisfy 'sh' |Done |Done |


=== Week 2 ===
|**Utilites** |**Dependencies*** |**Remarks** |**Porting status** |**Testing status** |
|du |struct stat, st_flags |Functionality given by st_flags disabled. Enabled Minix Emulated NetBSD stat. Testing failed with major switches : debugging the failures. |Done |Failed |
|netstat |netiso, pf, libkvm |Integrating pf port from last year GSoC project. Tried to port libkvm : failed due to absence of many headers in sys/, hence researching workaround if possible |Pending |- |
|traceroute |headers in netinet |Many headers in netinet missing, and hence the corresponding control codes and functions used in traceroute. Their port is yet to be analysed, if not possible to port those headers i can always port traceroute with very minimal functionality that suit Minix. |Pending |- |
|cat |st_blksize |Enabled Minix Emulated NetBSD stat. |Done |Done |
|chmod |lchmod system call |hflag disabled to avoid any call of lchmod() |Done |Done |
|echo |none |Changes In Makefile. |Done |Done |
|mkdir |none |Changes In Makefile. |Done |Done |
|pwd |none |Changes In Makefile. |Done |Done |
|sleep |SIGINFO |Related variables and code altered in sleep.c to avoid use of SIGINFO, as it is not supported in Minix |Done |Done |
|banner |none |Changes In Makefile. |Done |Done |
|basename |none |Changes In Makefile. |Done |Done |
|bdes |libcrypt |Ported libcrypt and disabled USE_FORT |Done |- |
|libcrypt|libc |Created static library |Done |- |
|column |libutil |Linked with Minix libutil. Implemented estrndup() in column.c |Done |- |
|csplit |libutil |Linked with Minix libutil. raise_default_signal() not present in Minix libutil, hence that function call has been disabled in csplit.c without affecting the behaviour of the program. |Done |- |
|libutil | | To many compilation errors and dependencies. I will give a thorough look again.|In progress | |


=== Week 3 ===
|**Utilites** |**Dependencies*** |**Remarks** |**Porting status** |**Testing status** |
|ctags |none |Makefile modified |Done |Done |
|vis |none |Makefile modified |Done |Done |
|what |none |Makefile modified |Done |Done |
|whois |none |Changes in Makefile. Disabled USE_FORT to turn off substitute functions. |Done | - |
|xstr |none |Changes in Makefile |Done |Done |
|yes |none |Changes in Makefile |Done |Done |
|libtelnet |none |Disabled USE_KERBEROS, USE_PAM, MKCRYPTO and USE_FORT. Didn't rename it to nbsd_libtelnet to prevent breaking of telnet utility. |Done | - |

  * Ported netinet/in_system.h . Tried to port netinet/ip.h as it is required for telnet, and studied it's dependencies and consequences. NetBSD ip.h depends on it's own fully-fledged TCP/IP stack, and hence it would require changes in the source of telnet to make it compatible with the TCP/IP stack of Minix.
  * Examined request types supported in ioctl() system call in Minix in more detail.  
  * Fixed _MINIX conditional in termios.h.


=== Week 4 ===
|**Utilites** |**Dependencies*** |**Remarks** |**Porting status** |**Testing status** |
|libutil | - |Disabled the compilation of unsupported calls. Linked utilities, earlier linked with Minix libutil, to NetBSD libutil to prevent cross-compilation. To be tested more thoroughly. |Done | - |
|dd |libutil |Disabled SIGINFO inovacation. Compiled with flag SMALLPROG, reducing a few switches in the way.|Done |Done |
|telnet |libtelnet |Disabled compilation with USE_INET6, USE_KERBEROS, USE_PAM, USE_FORT. Removed sa_len, a few message pointers, vfork(), SIGINGO, and hostnames beginning in ['@', '!'] . Used sizeof(struct sockaddr_in) in place of sa_len. There is no way to send urgent data in Minix TCP/IP stack and perform non-blocking I/O, hence that had to be disabled too. |Done |Bugs |
|cp | - |Defined MAXPHYS to (64*1024). MAXBSIZE is defined to be MAXPHYS, but latter's value was missing in netbsd headers. Used utime() in place of unsupported lutimes(), and changed struct timeval to struct utimbuf for doing that. Used MAP_SHARED in place of MAP_FILE. Disabled unsupported madvise() which may affect the performance. Used chown() and chmod() in place of lchown() & lchmod() but handling the symbolic links will be tricky. Disabled st_flags and all operation concerning it too. |Done |Done |
|xinstall | - |Disabled futimes() and everything related to st_flags. Used utime() in place of utimes() by converting struct timeval to struct utimbuf. Used MAP_SHARED and fork() inplace of MAP_FILE and vfork().|Done |Done |

  * Worked on porting df. It is heavily dependent on NetBSD statvfs and corresponding syscalls and functions e.g. getmntinfo(). Mount related headers and control codes e.g. MNT_WAIT, MNT_NOWAIT, etc. are not supported in Minix, and the mount structure is different too, which makes it even harder to port. I am giving a good and thorough look at all the dependencies and trying to determine minimum changes with many functionality reduced in the process. 

=== Week 5 ===
|**Utilites** |**Dependencies*** |**Remarks** |**Porting status** |**Testing status** |
|date |adjtime() |Disabled '-a' flag to disable adjtime() call, which correct the time to synchronize the system clock. I have been reading about system time, clock and how the latter is synchronized in BSD adjtime() implementation. I will find a workaround for Minix too.|Done |Done |
|ls | struct stat |Defined Macro to use netbsd stat. Disabled '-o' flag and ST_FLAGS. Removed S_IFWHT case in printtype() call. |Done |Done |
|ln | - |Trivial changes to Makefile |Done |Done |
 
  * In the first half of week, i mainly worked on updating the hier(7) man page which is now available at the tip of the master branch in my github repository : git@github.com:vivekp/minix-nbsd . Along with, i examined //df// , //tracerout//, //sh// and their dependencies in more detail and will be working ahead on porting them. I lost some time in the second half of the week due to illness and will try hard to compensate for the lost time in this week itself.

====== NetBSD ======

=== Week 6 ===
This week i mostly worked on submitting the changes upstream. I cleaned up the code for all the previous utilities ported, refined their commits and rebased them on master. I tested for 'make world' and native tests after installing each of the NetBSD utility. I also compared the NetBSD utilities with the corresponding Minix utility and looked for missing flags which are required.

I had ported mkdep earlier, but i didn't test it thoroughly that time. After switching to ELF, i tested it again, but it is giving errors while generating the .depend file. I am looking into it.
I also tested xinstall, but it is giving errors for root user. I need to look into it too. I have ported cksum, and it is giving consistent results with all the hashes. I cleaned up the NetBSD libutil code, to submit it upstream next week. 

I have prioritized the submission of these utilities to upstream next week: printf, chmod, mkdir, pwd, banner, basename, ctags, xstr, yes, ln, ls, cp, dd, column, kill, cksum, libutil

I have prioritized the porting/testing of these utilities next week: mkdep, xinstall, ping, telnet

=== Week 7 ===
Over the past week, i worked on cleaning up more code and submitting more utilities for integrating in upstream. I submitted the patches of ls, stat, uniq, sleep and cat for review. I completed the port of xinstall, and worked on porting ping and telnet. ping works fine, except for the statistics printed. I also worked on mkdep, but during dependency file generation it is picking up not-supposed-to arguments causing it to fail.

I ported mtree, mknod and tested them successfully. mtree can now replace the mtree.sh script. 

I was able to build the whole system using NetBSD xinstall and create a bootable image without any error. For that i needed to replace all the INSTALLFLAGS with -S by chmem and equivalent stack size. I also enabled stripping of symbol table from the ack compiled executable, as the /usr/pkg/bin/strip only works with elf and other formats. One problem that emerged during the port was that NetBSD functions uid_from_user(), user_from_uid(), etc. in nbsd_libc don't work in Minix mainly due to non netbsd style passwd file. I got them working in xinstall using local tweaks, but they aren't nice.

So, in the next week, my goal is to submit the remaining ported utilities for review. And, i will work next on migration to native NetBSD passwd format.(xinstall also relies on it.) For that, i will first port the corresponding utilities e.g. passwd, useradd, userdel, usermod, chpass, vipw, pwd_mkdb . Then, i will migrate the /etc/passwd file to NetBSD format, while ensuring backward compatibility.

=== Week 8 ===
This week i completed the port of mkdep, and refined the patches of ls and stat after new changes to struct stat. I also ported libterminfo and terminfo database this week.

Then, i worked on migration to the NetBSD style passwd format. I ported //passwd//, //pwd_mkdb//, and nbsd_libutil for this purpose. Then, i switched to NetBSD format //passwd// and //master.passwd file//. //master.passwd// file makes the ///etc/shadow// file obsolete. Now that, i had //master.passwd// file, i created //pwd.db// and //spwd.db// (password databases) using //pwd_mkdb//. Then, i removed the related functions in nbsd_libcompat_minix and used nbsd_libc functions . Now, the uid/guid<=> name functions are working properly as password databases are present. 

The work left ahead is to get Minix' login to work with new format, or port NetBSD login. I will also work ahead on ensuring compatibility with old binaries and testing that the system doesn't break after the complete migration.

=== Week 9 ===
This week i ported the NetBSD login. I got stalled a few days in it- trying to make sockaddr_snprintf() in libutil work, and partially in examining the problem in Minix login. I also lost some time in moving to college.

//login// works with the new NetBSD passwd format at the end of the week, but old utilities in base system are broken.

=== Week 10 ===
This week i was able to produce an initial version of steps for migrating to NetBSD passwd format. In addition, i added fallback functions to nbsd_libc getpw*() functions to ensure that  somebody doesn't get locked out of the system if he doesn't switch to new passwd format after ''make world; reboot''. I also identified all users of pwdauth and crypt() from old libc, and removed pwdauth and crypt(). After much testing, i found out that keeping NetBSD login, new libc, fallback functions in new libc, and retaining getpw*() functions in old libc keeps everything intact.

=== Week 11 ===
This week i mainly worked over fixing problems in the initial version of commit for switching to NetBSD passwd format. I ported NetBSD su and newgrp. I fixed user* and group* utilities for flock() error. I came to know that a process can only hold exclusive lock over a file only if it's open in write mode. I also wrote a shell script that fixes the clobbering of existing accounts. 

=== Week 12 ===
This week i came to know of more problems in the passwd branch, and spent most of the time fixing them. I got Minix su working with new libc. Also, i ported ctags, spell, spellprog and deroff. They are available in the ctags and spell branch in the repository.

=== Week 13 ===
Over past week, i imported most of the utilities suggested by others. I imported bzip2, libbz2, hexdump, od, tput, tset, col, hostname, domainname, infocmp, col, du, and cksum. I also imported NetBSD sort, but it is failing the pipeline test with the error ENOTTY. Gdb helped in tracking the problem to tcsetpgrp, and i hope to fix it soon.


===== Design =====
Userland refers to all application software that runs in user space. Userland usually refers to the various programs and libraries that the operating system uses to interact with the kernel: software that performs input/output, manipulates file system objects, etc.  (Adapted from Wikipedia).

==== Implementation Plan ====
I have planned to complete the project in four phases :

  * Identifying the trivial and MINIX independent/supported userland utilities, and porting them first.
  * Then, grouping the utilities according to libraries needed, which are either not present in MINIX or not supported by MINIX currently.
  * Then, porting the needed library. This is an iterative process, since one library may depend on other in turn, which needs to be ported first.             Porting the library may not be the end, as many syscalls and functions are not supported in MINIX which may need to be implemented as well.
  * Then, porting all the userland utilities dependent on the libraries and missing functions in the previous step.

=== Directory structure of the userland source in MINIX ===
All the userland utilities are currently present in /usr/src/commands in MINIX. Following the current directory structure and the modular approach used to design the whole MINIX operating system, the directory structure of the ported userland component of the NetBSD source tree should be imitated from the NetBSD source tree itself with a few modifications.

The userland part of the MINIX source tree should be organized as following:

  * /usr/src/commands: This directory has currently all the sources for the utilities present in MINIX, without any groupings. The utilities in it would be removed whenever the corresponding utilities from NetBSD are ported.
  * Now, there would be four more directories in /usr/src , namely bin, sbin, usr.bin, usr.sbin taken directly from the NetBSD userland source tree.
  * /usr/src/bin: Similar to NetBSD’s /usr/src/bin .
  * /usr/src/sbin: Similar to NetBSD’s /usr/src/sbin .
  * /usr/src/usr.bin: Similar to NetBSD’s /usr/src/usr.bin .
  * /usr/src/usr.sbin: Similar to NetBSD’s /usr/src/usr.sbin .
  * GNU-licensed utilities from NetBSD’s /usr/src/gnu can be replaced with utilities from other userland which are licensed under BSD. If this is not possible at all, making packages out of them is an option.

//Advantage// : This directory structure for MINIX offers the advantage that it separates the MINIX userland utilities from those ported from NetBSD, and thus offers clarity in the structure. Moreover, the MINIX utilities may coexist with NetBSD utilities, while the compilation for former may be disabled if the NetBSD’s are newer and superior.

====== NetBSD ======
The plan is to checkout the userland source from a stable release of NetBSD and port utilities contained in that source. A minix-port.patch file would be present in the top level directory of a NetBSD component that contains the changes done to that component when porting to MINIX.

A system has been set up by Gianluca Guida so that one can easily track the version of NetBSD's libc currently being used. It would be convenient for the developers to modify the nbsd sources in the minix tree and update the minix-port patches accordingly. He has also automated the minix-port.patch creation. On the same lines, the userland source version will be tracked and minix-port.patch file will be generated in each of the directories: bin, sbin, usr.bin, usr.sbin .

Initially, empty directories /usr/src/bin, /usr/src/sbin, /usr/src/usr.bin, /usr/src/usr.sbin will be created. As soon as a utility from NetBSD userland is ported to MINIX , it would be added to the corresponding directory. The changes would be pushed upstream after a few utilities have been ported and minix-port.patch has been modified accordingly in each of the four directories.

Also, I would maintain a list of all the NetBSD userland utilities that have been ported in the /usr/src/commands/ directory, so that one knows where to look for the source of missing utilities in commands.

  * The separate system for tracking the the userland source version will be same as that for tracking the nbsd_libc version. A separate reference source tree system need to be set up because the userland may need to be updated to newer release, which may have fixed bugs in few utilities while the older nbsd_libc is absolutely compatible with the newer release, or the nbsd_libc hasn’t changed at all in the newer release.
  * This implies that whenever a newer release of NetBSD comes, we would have to deal with merge conflicts, fix them, and check all the utilities affected by merge whether they compile now or not. If they don’t compile, fix them and send all the changes upstream.
  * Related Tasks :
    - Switch to ELF based system
    - Set up the system for tracking the userland source version analogous to that for nbsd_libc

===== Schedule =====
Here is an approximate schedule of how i will work on the project:

==== Pre-Coding Period (Apr 25 - May 22) ====
  * I will take a deep and thorough look at minix-nbsd git repository.
  * I will be active on MINIX mailing list and IRC channels - #minix, #minix-gsoc and discuss any doubts that i may have.
  * I will set up a new machine, install MINIX on it and update it to the trunk before the start of the Coding Period.
  * I will discuss with my mentor about the modifications needed in existing design and planning for the project.
  * In between, i will identify easy to port utilities, and port them if i get enough time.

==== Week 1 (May 23 - May 29) ====
Complete the dependency table of all the important userland utilities, which will list missing references to functions, missing control codes and NetBSD libraries to port.

==== Week 2 (May 30 - Jun 5) ====
Port all the userland utilities, whose dependencies are trivial or minimal.

==== Week 3 (Jun 6 - Jun 12) ====
Port as many NetBSD libraries as possible which are listed in the dependency table.

==== Week 4 (Jun 13 - Jun 19) ====
Learn about implementing the missing functions and control codes and implement them. Or, find the workarounds of doing the same if implementing the function is not feasible/supported in MINIX.

==== Week 5 (Jun 20 - Jun 26) ====
Continue porting the required NetBSD libraries, if there are any.\\  Port the userland utilities whose dependencies have been resolved.

==== Week 6 (Jun 27 - Jul 3) ====
Port more userland utilities.

==== Week 7 (Jul 4 - Jul 10) - MIDTERM ====
Test all the userland utilities ported till now, and fixing any bugs encountered in the process.

==== Week 8 (Jul 11 - Jul 17) ====
Complete the mid-term evaluation.\\  Check if the MINIX base utilities compile with the new ported libraries. Discuss with the mentor if there are issues in it.

==== Week 9 (Jul 18 - Jul 24) ====
Release the ported utilities to the MINIX community for testing.\\  Complete the implementation of missing functions and control codes, or find their alternatives and code them.

==== Week 10 (Jul 25 - Jul 31) ====
Port more userland utilities in which references to missing functions and Macros have been fixed.

==== Week 11 (Aug 1 - Aug 7) ====
Documenting all the work done till now.\\  Port more utilities.

==== Week 12 (Aug 8 - Aug 14) ====
Port more utilities and test them.

==== Week 13 (Aug 15 - Aug 21) - FINAL ====
Test all the ported userland utilities and releasing them to the community to get feedbacks.\\  Clean the code, and improve the documentation.

===== Resources =====
  * [[ftp://ftp.netbsd.org/pub/NetBSD/|Sources of NetBSD release versions]]
  * [[http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/src/share/mk/bsd.README?rev=HEAD|NetBSD's src/share/mk/bsd.README]]
  * [[ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-current/|Source of NetBSD-current development branch]]
  * http://wiki.minix3.org/en/[[:DevelopersGuide:PortingGuide]]
  * [[http://www.feyrer.de/NetBSD/tour-de-source-1userland.html|NetBSD userland tour]]
  * http://www.opengroup.org/onlinepubs/000095399
  * http://www.gnu.org/prep/standards/standards.html
  * [[http://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html|Makefile Conventions]]
User Tools

Site Tools

Page Tools
