Differences

This shows you the differences between two versions of the page.

--- users:pikpik:now [2014/11/17 11:56]
127.0.0.1 external edit
+++ users:pikpik:now [2014/11/17 14:14] (current)
lionelsambuc [Looking for ways to lower memory usage]
@@ Line 1: / Line 1: @@
+====== What I'm working on now ======
+===== Active =====
+==== Removing Custom 64-bit Functions ====
+I'm currently removing MINIX's custom 64-bit functions in favor of compiler-supported implementations. The current idea is to find the places where they're used, and replace those uses with the bodies of the functions. I'm not sure how correct this approach is in some cases though. Here's my branch for these changes: [[https://github.com/pikpik/minix/compare/master...less64|less64]].
+I've finished the initial conversion of the 64-bit functions and am now testing my changes.
+I expect a fair amount of fallout. If you'd like to track current and then switch to building release ISO's with "less64" (switch to branch, and then in src/releasetools: sh release.sh -cp) to find bugs, that would be most awesome!
+Once it builds successfully, the full test suite will need to be run on a VM with the built ISO to find more problems.
+Once all bugs are fixed, the changes could definitely use some review.
+Additionally: so, we're switching away from composite types (for emulating 64-bit numbers with two 32-bit ones). The high and low end code still exists. It might have better performance when fully converted to single 64-bit numbers.
+It might not be possible in all places though, since so system-level messages are using two fields for one 32-bit integer each (to transfer a single 64-bit integer).
+I'm not sure single fields could contain full 64-bit integers though. I haven't checked.
+I talked to David, and he said that the team hopes to improve the message situation in the future.
+Everything now builds successfully. But there are problems with at least two tests. One might have a problem and test53.c produces errors on lines 186 - 190. I have no idea why though. I haven't tried rebooting the system yet, and Top only reports the load average, process count, and main memory usage lines. Something's definitely wrong.
+==== Porting x11/libxcb ====
+<sys/ucred.h>
+  * pid_t is missing.
+  * NGROUPS_MAX needs to be included, from sys/syslimits.h maybe?
+This is no longer a problem when I import [[http://nxr.netbsd.org/xref/src/sys/sys/ucred.h|src/sys/sys/ucred.h]] from NetBSD.
+LibXCB has more problems though.
+I need to get my changes online for //ucred.h//.
+==== Simplifying RC scripts ====
+I'd like to help improve the RC scripts by making the code more concise, able to handle parallelism, able to control parts individually, and able to handle service dependencies correctly.
+I have two ideas. One for just [[http://pastebin.com/raw.php?i=UZU0qgXk|simplifying]] the scripts and another for [[http://pastebin.com/raw.php?i=7ikykaka|beginning to handle dependencies]].
+BenGras also suggested [[http://www.freebsd.org/cgi/man.cgi?query=rc&sektion=8|FreeBSD's RC system]] as it already supports these ideas.
+I had [[http://pastebin.com/raw.php?i=VgKdZppv|another idea]]. It follows BenGras' idea of using //tsort// for managing service dependencies. It's a work-in-progress and is currently just [[http://simple.wikipedia.org/wiki/Pseudocode|pseudocode]].
+I wrote a draft for another version, but I couldn't keep it. I'm hoping to try out what I remember from it.
+==== x11/libXau ====
+The library package libXau seems to have a problem with its "xthreads" (or something) not finding proper definitions. If I remember correctly, the problem was that "xthreads" were attempting to use Pthreads, which are not available on MINIX currently. I had fixed this previously by using GNU Pth as a drop-in replacement for Pthreads. This commit on !GitHub shows what I'd done: [[https://github.com/pikpik/minix-pkgsrc/commit/3d8487e306068afd4a1fc019e3297ae9cb97c0c2|"It works using GNU Pth."]]
+However, pkgsrc comes with the Official Pthread Replacement (OPR) mechanism, which is a system for "automatically" using Pthread replacements. It would be better, though, if MINIX had a compliant Pthread implementation.
+With the libpthread effort on hold, I'll be adding the use of OPR to x11/libXau. From what I understand, it should have been using it all along anyway.
+I'll continue the port of NetBSD's pthread system later. For now, I'm using OPR. Here's a commit for pkgsrc that [[https://github.com/pikpik/minix-pkgsrc/commit/817b0fd1f960993b4a135f46fb20e8efd18c9f17|fixes libXau]].
+It's important to mention that r0ller doesn't currently think that this helps. Here's [[http://groups.google.com/group/minix3/browse_thread/thread/9748ccf10117f719|our discussion]] on the Google Group about this.
+I've had more problems where I've needed OPR, so I'm going ahead and just using it.
+==== x11/libX11 ====
+This needs to be done:
+ xproto
+  Xos_r.h
+   MINIX needs <sys/params.h> enabled.
+==== Porting a new X.org ====
+I'm working on this now.
+I'm currently porting dependencies.
+| **Thing** | **About** | **Ported** | **On GitHub** | **Officially available** |
+| Pkgsrc | Needs to be X11_TYPE=modular | {*} | {o} | {o} |
+| <sys/ucred.h> | Mixed NetBSD, MINIX, and old MINIX | {*} | {o} | {o} |
+| libXau | Not accepted yet | {*} | {*} | {o} |
+| libxml2 | Convince it not to use threads | {*} | {o} | {o} |
+| xproto | Pretend we're BSD | {*} | {o} | {o} |
+| libX11 | Use libpthread and OPR | {*} | {o} | {o} |
+| /dev/pci | I'm importing Jorn van Engelen's [[http://ftp.quzart.com/minix/modular-xorg-server/|/dev/pci efforts]] | {o} | {o} | {o} |
+| Pkgsrc fetch sites | MINIX's mk.conf breaks the backup distfile fetch site system | {o} | {o} | {o} |
+| libpciaccess | Waiting on /dev/pci | {o} | {o} | {o} |
+| More | Waiting | {o} | {o} | {o} |
+| modular-xorg-server | Avoid DRI and DRM for now. | {o} | {o} | {o} |
+Note: Many necessary packages depend on others that must be rebuilt to propagate changes. Pbulking with the additional ported packages does this easily.
+==== Renaming Boot Directories ====
+I'd like to make things make more sense and be easier and shorter to type at the boot prompt. I started a post about this: [[http://groups.google.com/group/minix3/browse_thread/thread/d50e1d5f6db6c10b|Renaming Boot Directories]].
+I've put my changes online in the [[https://github.com/pikpik/minix/commit/99f47b6ace3080b76ca1bf9c228977915223f5e6|Renamed boot directories]] commit of my [[https://github.com/pikpik/minix/commits/rename_boot_directories|rename_boot_directories]] branch.
+I've tested the following:
+  * Source update from the 3.2.1 dev. snapshot
+    * No problems noticed.
+  * Install from a CD
+    * Delete existing system, then Full install
+      * From the 3.2.1 dev. snapshot
+        * No problems noticed.
+      * From the same version
+        * No problems noticed.
+    * Full install
+      * From the 3.2.1 dev. snapshot
+        * No problems noticed.
+      * From the same version
+        * No problems noticed.
+    * Reinstall (preserves only /home)
+      * From the 3.2.1 dev. snapshot
+        * No problems noticed.
+      * From the same version
+        * No problems noticed.
+    * Update bootloader only
+      * From the 3.2.1 dev. snapshot
+        * It's bootable, but...
+          * there are warnings during the update about existing kernel entries,
+            * A potential solution is to copy /etc/boot.cfg.default to /mnt/etc before update_bootcfg is run.
+          * the boot menu options are different,
+          * and the new boot directories aren't all created.
+      * From the same version
+        * No problems noticed.
+I did not test source or binary updating from 3.2.0. Please let me know if that would be a good idea.
+===== Waiting =====
+==== The bootloader can crash MINIX ====
+This problem occurs in VirtualBox, but not QEMU.
+Do this:
+  * Boot
+  * Boot prompt
+  * quit
+  * (Garbled text at the top of the screen)
+  * Press enter
+...and you'll see this:
+  * System panic
+ {{http://i.imgur.com/Juyn2.png}}
+I'll test this with NetBSD to see if MINIX just inherited it.
+==== Keeping documentation revisions ====
+The purpose of keeping revisions of documentation is to help users who do not or cannot run the latest version of MINIX.
+Keeping versions of the wiki's documentation per release is potentially messy when done on the wiki itself. One idea is to export the wiki's content as static HTML Web pages, and to serve it from the MINIX 3 website. Additionally, the documentation could be copied into each release of MINIX.
+There are [[http://moinmo.in/FormatterMarket/MoinMoin2HTML|examples for exporting]] content from the MoinMoin wiki system.
+If the release script is adapted to export the wiki's live content into itself, it might need Python to do it. On the other hand, the website could be updated by someone with access, and then the release script could just download pages or a bundled set of documentation from it.
+==== Packaged drivers ====
+I'm wondering if drivers, such as the floppy one, should be packaged.
+==== Wondering about packaged updates ====
+I heard from some people in the NetBSD community that it could be possible to put packaged system updates into pkgsrc for distribution as binary packages. Naturally, I thought this could be helpful for MINIX. I talked to [[:users:BenGras]] about it, and he suggested the possibility of FreeBSD's update system. While it creates a second chronological "stream" of updates, I think it makes sense.
+I wonder what the next step is. If it's ever in place, it seems like it would be perfect using live updates, once that's supported of course.
+I found another set of tools. But this time, they're from NetBSD! See [[http://blog.netbsd.org/tnf/entry/introducing_sysbuild_and_sysupgrade|Introducing sysbuild and sysupgrade]] on the NetBSD Foundation's blog.
+Could the kernel and servers be distributed as binary packages? Essentially yes.
+My current idea involves modifying the "setup" and "release" scripts to use compiled sets that are generated from builds.
+These could be /boot/release and /boot.cfg as boot.cfg, the rest of / as root.txz, /usr as usr.txz, /home as home.sh (it primarily needs to be scripted using "user add"), /usr/src as src.txz, /usr/pkgsrc as pkgsrc.txz. /usr/packages (for the CD) would be minimal, and all packages would be installed from that for installation; not from /usr/pkg on the CD.
+These sets could then be distributed as one or more pkgsrc packages, or they could be in a separate repository that is used via the pkgsrc tools. (Pkgsrc supports arbitrary pkg areas; I think each with a single repository.)
+==== Porting Archivemount for FUSE ====
+I'm currently attempting to port Archivemount so that it can be used during MINIX's build process. This software uses the LGPL license, which I think is close to being permissive.
+My current problem is that it builds dynamically by default. I'm trying to figure out how to tell it to build statically.
+I'm trying to build it with -static.
+That didn't really help. It turns out that when linking dynamically, I'm able to compile and run (with errors from RS) //archivemount//. However, RS needs executables to be statically-linked.
+Here's my progress:
+<code>
+diff --git a/filesystems/fuse-archivemount/Makefile b/filesystems/fuse-archivemount/Makefile
+index 677b74f..86de7cd 100644
+--- a/filesystems/fuse-archivemount/Makefile
++++ b/filesystems/fuse-archivemount/Makefile
+@@ -22,6 +22,15 @@ NO_CONFIGURE=	yes
+ CPPFLAGS+=	-DHAVE_STATVFS
+  *endif
++.if ${OPSYS} == "Minix"
++#LDFLAGS+=	-Wl,-wrap,main
++LDFLAGS+=	-lpuffs -lminixfs -lsys -lc
++LDFLAGS+=	-larchive -lbz2 -llzma -lz
++LDFLAGS+=	-lminlib -static
++.endif
++
++MAKE_ARGS+=	DEBUG=1
++
+ INSTALLATION_DIRS=	bin
+ do-install:
+</code>
+Build errors: [[http://pastebin.com/raw.php?i=qvNYA09q|log of the build]].
+==== Fixing NTFS-3G for FUSE ====
+When I tried building this, the resulting program didn't work with RS (the Reincarnation Server). The problem was that it was being built dynamically. I assume that this wasn't a problem when MINIX didn't support dynamically-linked libraries. But now that MINIX has support for that, it's a problem for any packages that are used as server: RS will only work with statically-linked executables.
+I found the following option which allows the filesystems/fuse-ntfs-3g package to be built and linked statically.
+<code>
+CONFIGURE_ARGS+= --enable-really-static
+</code>
+Unfortunately, I don't have any NTFS filesystems available for MINIX to read. When I run //mount -t ntfs-3g /dev/c0d0p0s1 /mnt//, I get errors about the partition not being mountable. So, I'll put my changes online once I've tested //ntfs-3g// successfully.
+//ntfs-3g// builds and installs. I also don't see errors from RS about its executable format. It has problems when I try to mount ///dev/c0d0p0s1// with it though.
+Here's a screenshot of the problem: [[http://i.imgur.com/PLKgD.png|errors when mounting MFSv3 partitions]].
+Here are my changes so far:
+<code>
+diff --git a/filesystems/fuse-ntfs-3g/Makefile b/filesystems/fuse-ntfs-3g/Makefile
+index 2ddfbfa..9db9225 100644
+--- a/filesystems/fuse-ntfs-3g/Makefile
++++ b/filesystems/fuse-ntfs-3g/Makefile
+@@ -20,6 +20,8 @@ USE_TOOLS+=	gmake pkg-config
+ CONFIGURE_ARGS+=	--exec-prefix=${PREFIX}
+ CONFIGURE_ARGS+=	--disable-ldconfig
++CONFIGURE_ARGS+=	--enable-really-static
++
+  *include "../../mk/fuse.buildlink3.mk"
+  *include "../../mk/pthread.buildlink3.mk"
+  *include "../../mk/bsd.pkg.mk"
+</code>
+==== Testing the USB Subsystem ====
+I've been testing the new USB subsystem, worked on by Dirk and Kees, in VirtualBox and on an old Gateway Solo (it has a 434 MiB of memory, and ~6 GB of storage).
+I'm writing a [[.:now:testingusb|short guide]] for using the USB system.
+==== Looking for virtual machine-based debugging methods ====
+I'm interested in finding a way to monitor the memory usage of all processes running inside MINIX, including the kernel. It should be possible to do this with specialized statistical functions inside components, but that would be invasive, take a long time, require a lot of effort, and could prevent the ease of making MINIX smaller and simpler (unless a pluggable instrumentation system were involved, such as the existing one).
+I found some guides that involve using Bochs' special instruction and memory-based tracing features as well as guides for using GDB and Valgrind with QEMU. I really don't want to use Bochs, it's very slow and its networking system is awkward to set up. But it might be the easiest solution. QEMU has tools I like better, but I get the feeling that they involve filtering out QEMU's information. That could be easy, very messy, or not helpful at all.
+I found [[http://code.google.com/p/insight-vmi/|insight-vmi]], which appears to allow you to see what's happening inside an operating system that it knows about. I plan on following the [[http://code.google.com/p/insight-vmi/wiki/LinuxDebugSymbols|guide for Linux]], before attempting to use it with MINIX.
+==== Looking for ways to lower memory usage ====
+Memory requirement timeline:
+  * 64 MiB - This was when Intel segmentation was relied on
+  * 32 MiB - Same changes, but with the BOOTPROG_LOAD_START variable modified
+  * 67+ MiB - Intel segments were relied on less, and RS ballooned
+  * 30 MiB - RS' self-reliance feature was turned off
+  * 20 MiB - Same changes, but in single-user mode
+I had been able to lower MINIX's memory usage down to about 32 MiB. That involved changing some settings inside src/commands/mkimage. I didn't look for ways of lowering MINIX's memory requirements further.
+More recently, MiB. However, when RS is starting it seems to use about 67 MiB of memory. Instead of lowering MINIX's memory requirements, it increases them.
+I'd like to find a way of seeing what's happening and why this much memory is used. Then, I'd like to help fix it.
+I talked to [[:users:BenGras]], and he explained a way to watch RS' dynamic memory use: add //printf// statements of relevant information to where RS ''malloc'''s and ''free'''s memory for system services. I did this, and then I gave [[:users:BenGras]] and David M. a (slightly cleaned-up) log of the resulting messages from RS.
+They analyzed the results and explained that, in some cases, copies of critical service executables are kept in RS' memory. However, the combined memory used by these parts didn't add up to RS' 67 MiB of allocated memory. BenGras and David then realized that RS was keeping memory for itself to be able to recover in the event that VM isn't available. BenGras [[http://git.minix3.org/?p=minix.git;a=commit;h=b3f47f5835c2689fede93eb18ca365cc0783140a|made changes]], and they are now in MINIX.
+Here are the next largest memory users:
+  * Kernel, 8.5 MiB
+  * Memory, 4888 KiB
+  * RS, 1392 KiB
+  * PFS, 1176 KiB
+  * Inet, 1164 KiB
+  * IS, 1036 KiB
+  * MFS, variable and adaptive
+  * VM, unknown
+==== Porting fluxbox ====
+It currently is blocked by failing dependencies. With the modular X.org server this should be much easier.
+==== Figuring out POSIX threads ====
+I'm attempting to import NetBSD's [[http://nxr.netbsd.org/xref/src/lib/libpthread/|libpthread]]. If it goes nowhere, then I'll just give up and use pkgsrc's OPR system.
+I'm currently trying to see how Thread Local Storage (TLS) can be supported by MINIX. One idea is to change MINIX's use of registers to fit NetBSD's behavior. Another is to find a minimal way to insert the TLS state information into MINIX's register layout as an additional component. I'm not sure which makes the most sense, but I'm currently trying the first approach.
+I'm currently [[http://groups.google.com/group/minix3/browse_thread/thread/20c7d3f19902c881|awaiting responses]] on how to proceed.
+To me, a goal of compliance with [[http://www.unix.org/what_is_unix/single_unix_specification.html|S.U.S.]] is __critical__ for real systems. Because of that, I think supporting POSIX threads is a very important first step toward standardization. I'm not adverse to putting in a little time and effort.
+I've received advice on two possible approaches. One involves porting NetBSD-current's POSIX thread library and finding or implementing LWP parking (sleep/resume) and possibly "futex-like" functionality in MINIX. Another approach is to import NetBSD 4's POSIX thread implementation, because it is less dependent on system functionality.
+I will try porting NetBSD 4's threads. If I'm successful, then I'll passively look for ways to port a newer implementation.
+==== Out-of-the-box USB support ====
+I recently added USB support to the installation CD. This is only in [[https://github.com/pikpik/minix/commit/d13817ca61fab1e1dede26582b797428a7757f16|my repository]].
+In my changes, there are problems with /etc/fstab and /etc/mtab. These would need to be understood and fixed before the changes can be useable.
+The problem with this is that it puts GPL-licensed (perpetual copyleft) drivers and code from Linux in the installation system. If a third-party decided to distribute MINIX or a derivative, they would have to make their changes to the USB drivers public or drop USB support.
+For this reason, I'd like to port NetBSD or FreeBSD drivers to have a license-compatible replacement. There's a [[http://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf|thesis paper]], by Thomas Friebel, on a related effort that I have only skimmed through so far.
+====== NetBSD ======
+There are quite a few cases where MINIX's imported version of NetBSD's C library essentially cuts out important or useful things. I recently had a problem with //ucred.h// for this very reason.
+I'd really prefer it if MINIX took NetBSD's C library verbatim, ''#ifdef'''d-out unsupportable sections, and then linked-in its own "glue code." That way it would be as close to its source as possible, we'd have almost all functionality, problems would be easily traceable, and added full support would be easier.
+I'd like to progressively update the C library and header files to make this be the case.
+==== Source code layout ====
+I'm completely confused about what's going on with MINIX's NetBSD-like directory layout for source code. It has vague similarities, but when you look deeper it seems very messy. In some cases, there are two //sys// directories for header files.
+I started a discussion about this on the Google Group. Antoine LECA responded. So far there've been no other responses.
+==== Being generally concerned about legal stuff ====
+I'm slightly concerned about MINIX's un-permissively-licensed software, although that shouldn't pose a problem for normal users. For commercial users, it could be awkward. However, if such users don't distribute executable forms of the copyleft software, then it shouldn't be a problem for them. At least, that's my understanding.
+==== Cleaning up the wiki ====
+I'm trying to steadily improve MINIX's documentation. The areas I'm working on include:
+  * The user's guide
+  * The developer's guide
+  * Per-release copies of both guides

User Tools

Site Tools

Differences

Page Tools