Shared Memory Support in Minix

Student: Guanqun Lu
Mentor: BenGras
SVN branch name: src.20090422.r4228.guanqun (Original) and ipc.rebase.beng (Newer: Rebase to Ben's branch)

Minix 3 is on its way to be a fully POSIX-compliant system. The lack of the features of IPC hinders developers from porting useful programs which take advantage of IPC from other platforms.

Shared memory is a method of inter-process communication, it is a way of exchanging data with different processes. Unlike the widely message passing in Minix3, each process is able to access the block of shared memory simultaneously. The obvious advantage is that it is fast without sacrificing any inter-process overhead. Posix shared memory is defined through a series of interface APIs, such as shm_open, shmget etc, although the internal implementation varies on different operating system.

Note that the design is subject to change.

Shared memory support is only a small part of POSIX inter-process communication methods. This design aims at not only the shared memory support, but also trying to provide a generic IPC framework. A new server will be introduced to accomplish this goal. Based on this framework, other IPC mechanisms are created, such as shared memory, semaphore and message queues. The figure below depicts such relation.

 +---+ +---+ +---+
 |SHM| |SEM| |MSG| ...
 +---+ +---+ +---+
  ^     ^     ^
  |     |     |
+----------------------+
|      IPC Server      |
+----------------------+

As a server, it is useless if it can not communicate with others. And to work properly, IPC server has to send and receive messages to different other servers or managers. The below picture shows that it interacts with

• virtual memory server
  especially useful for the shared memory support.
• system library
  be able to communicate directly with libraries.
• process manager
  notify IPC about the fork()/exit() events, so that IPC server can handle it accordingly.

+---+             +--+
|IPC| <---------> |VM|
+---+      ^      +--+
           |
           +----> +---+
           |      |SYS|
           |      +---+
           |
           +----> +--+
                  |PM|
                  +--+

In order to make sure that the code functions well and it is POSIX-compliant, Linux Test Project will be partially ported (only the shared memory portion), through which I can test easily and find the potential regression at some bug-fixing patches. Linux test project is more like a platform. When it has been successfully ported, I myself will have to write more test cases to verify my own code.

Besides that, the code will be published and announced on mailinglist at some specific milestone. Therefore the code is exposed under more testers, the chances of living bugs is minor.

Last but not least, my code has to be reviewed by my mentor.

Deliverables before the mid term will be: shmat() and shmdt() functions well. Deliverables before the final will be: make it function well and get merged into trunk.

I'm recording the whole precious experience involved in this GSoC event.

That's when the whole story begain…

I learn from BBS in my school that this years' Google Summer of Code is open. (Technically speaking I heard of GSoC last year, unfortunately I missed it.) I can't wait to search the potential project which suits my ability and interests, soon after, Minix catches my eye. It's a micro-kernel system which I learnt on my first operating system course. I joined the mailing list without hesitation and found that the shared memory support is a good idea to explore and it is also in my interests.

I spend some time on writing the proposal, trying to be nice and well written. This project is the only one I'm applying for. In order to make my application convincing, I also made out a simple prototype on how to communicate between different processes easily. As the code now upgrades to 3.1.4, the basic virtual memory support is there. Such a prototype is simple with the help of map_region function.

Wait anxiously for the result and any upcoming events. Well, there is one piece of homework left for me, which is about implementating a semaphore server. The idea is simple, when DOWN operation is carried out, the counter is decremented and if it's negative, put the calling process into an queue; When UP operation is seen, the counter is incremented and wake up one process in case. It's interesting to write the stuff in Minix, because there's no big mass of code. Every part is a stand-alone server, which is simple and elegant and of course easy to understand. With the help of Ben, I worked out the prototype really quickly, in less than about three days.

Wait anxiously and start reading the code of VM server.

The final result is released and I'm so happy that I'm becoming one member of this year's Google Summer of Code, involving in Minix project.

The real work doesn't begin until 1st May. Before that, I'm busy with the work at school, and I only have some leisure time to scan through the code I'll be involved with.

The work in this week also includes the environment setup. At first, I'm going to use my own repository so that the access will be trivial. But Ben reminds me that the repository is better to be hosted at Vrije University. And I spend some time setting all this up. The network is not so good at my school…

Settle down the detailed design issues, as you can see from the above, the outline is listed. According to Ben's reply, there are lots of things that need to sort out. For example:

• how to handle the fork()/exit() situation.
• what will happen when the IPC server crashes.
• …

Anyway, the header files are ported so that Minix now can recognize these macroes.

Busy with reading the code and writing the code… Work done is:

• shmat() works fine.
• shmdt() and shmctl() partially done.
• several bug fixes.

This is the week of the honor! I'm working like an engine full of fresh gas(poor metaphor…), checking in at most about 30 commits one day. Wow! The frame work of ipc server is now finalized. There is one thing that needs mentioning, the big change in the week is to remove all the stuff in IPC server trying to catch every processes' status, for example, whether it's alive or dead, and how many segments they have etc. As the code goes on, it's found that such a design is not so good. Since the idea of trying to store per-process information in IPC server is and duplicate effort, as the VM server already handles the job well. According to the KISS principle, let the IPC server be a consumer of the information stored in VM server, that's why we come up with the lots of helper functions in VM server.

Work done in this week:

• Linux test project partially ported.
• Framework of IPC server is done.
• It can pass all the test cases in Linux test project now.
• Some bug fixes and enhancement.
• Find out something needs porting or implementing (unmap, lrand48…)
• And more to find in the code …

I discussed with Ben that we should restrict the operation exposed by VM server to those who really need and who explicitly specify in the drivers.conf file.

Why would we do such a change? Because currently, some additional VM operations such as getphys, getrefcount, are exposed by c library interface, which means that any legal user applications can invoke such operation, which implies a potential security leak.

To accomodate this problem, we decide to incorporate another token named VM in drivers.conf file.

During the process of implementation, several problems occurs:

• add new fields in the middle of a structure crashes the system on boot, the workaround is to add the new fields to the end of the structure. It at first haunted me for a while, later Ben reminds me that it may be caused by the stale compilation, make clean should work.
• printf() in service program fails. It's kind of black magic, you know what, the stdout is used by the later file system. Hence you have to use stderr to see what you're interested in.
• a weird initialization problem… still struggling to find the bug…

(This project is still going on. I just forgot to update this wiki in this period.) During this period, a new token named VM in drivers.conf is implemented. And more parts of linux test project is ported to Minix to help my debugging. Through this debugging, some of bugs relates to that I'm not coding per POSIX standard. After consulting the POSIX manual, such bugs are fixed.

As Ben and I think that SYSV semapohre operations are important and helpful for Minix platform, I start to take a look at the semaphore server and after sometime the basic functionality is OK. A simple program that implements dining philosophy is done.

• Out for a week for vocation.
• rebase my current branch on top of Ben's working tree.
• fix a bug of passing different sizes to pt_writemap() function. It will result in a serious shmdt() failure.
• discussed with Ben how to implement a mechanism to inform IPC server of the exits of processes. The way is to inform VM of the signals by PM, and then VM passes these signal information to IPC server. There's a subtle problem that the direction of passing message is not right. However, that's only easy and possible way to accomplish this goal.

This summer of code project is reaching an end, and happily to say that I've finished what I have promised. Postgresql is ported to Minix and it can be used basically, as the shared library isn't supported in Minix yet. Some functionalities are missing due to this problem. And Ben and I are rebasing the code to his latest branch. We hope that my code will be in trunk tree soon.