Apple has always marketed the power of Unix, and the number of Open Source projects as strengths of running a Mac OS X system. At the last WWDC conference, new tools were introduced which allow system administrators to easily update and patch open source components, i.e. Darwin, of the operating system.

Darwin is the mach-based kernel architecture that serves as the foundation for Mac OS X. Apple, with the Internet Consortium, founded the OpenDarwin project in 2002 to provide a way for direct access to the CVS repository of Darwin-based projects. The project website describes it best:

OpenDarwin.org, jointly founded in April 2002 by Internet Systems Consortium and Apple Computer, takes cooperative development of Darwin, the core of Mac OS X, to the next level.

Since Apple's Mac OS X releases are based directly on their CVS repository, it has a fairly comprehensive level of quality control in place. It is desirable to increase the collaboration between Apple and the open source community beyond this model.

The mission of the OpenDarwin project is to increase the practical use of Apple's open source Darwin releases, to coordinate and increase participation of the community in Darwin development, to innovate and explore new technologies while still remaining relevant to the mainstream computing environments that Apple provides.

Many OpenDarwin members are either Apple employees or committers to Apple's CVS repository, who have an active interest in merging technologies from OpenDarwin.org into Darwin and Mac OS X releases. With OpenDarwin, project members have greater latitude in producing incremental updates or interim releases of Darwin. It complements Apple's infrastructure.

Membership in the OpenDarwin project and access to its works are open to everyone. The project is also fully independent, with control over its own web site, project news, bug tracking information and CVS repository, as well as any other services that the community owners may wish to provide. Neither Apple nor ISC take any responsibility for, or exercise any editorial control over, the OpenDarwin project.

One project in particular, darwinbuild, is of great use to both system administrators and programmers who would like to add functionality to open source components of the Macintosh OS X operating system. System administrators can easily use a security patch to update a project before a certified patch was released by Apple. As an example, if the ssh binary was found to be vulnerable, and a patch existed on the openssh.org website, a patch could quickly be applied, built and distributed to computers. A programmer could choose to add functionality to an open source project. An example of this model might be the addition of Rendezvous functionality to talk.

darwinbuild!

The first step is to begin building the binaries for darwinbuild. The source can be directly downloaded from opendarwin.org, but it is preferable to obtain the source from the OpenDarwin Concurrent Versioning System repository (CVS).

Getting the source code

The code below expects use of the bash shell (default in 10.3 and higher)

Note, when checking out the source code (the last command), it will create the directory in the current working directory. If the terminal is newly opened, this will be the home directory.

Next, build and install darwinbuild. Xcode must be installed in order to build the binary. If a message about not having cc in the path pops up, XCode is not installed. Once Xcode is installed, the procedure is straightforward:

make

sudo make install

After issuing these commands, darwinbuild, and associated binaries are installed in /usr/local/bin. Set the PATH variable to include this directory, or type the full path to the command, i.e. /usr/local/bin/darwinbuild to use them.

Creating a build directory

By default, darwinbuild functions in a chroot environment. All items needed to build any project will be placed in this build directory. For convenience, it is best to work directly on a disk image. Some sources also require a case-sensitive file system. Most machine will not have a dedicated partition for this purpose, therefore, it is easiest to create a disk image. Use the following command:

hdiutil create -size 4g -type SPARSE -fs HFSX -volname Builds -uid 0 -gid 0 -attach Builds.dmg

Examining the command, it creates a sparse disk image of 4 gigabytes in size, using the case-sensitive file system. It is also owned by root, and in the wheel group (uid/gid 0). Lastly, it mounts the image with the attach verb. Again, this image will be created in the current working directory. Why not just use Disk Utility? Disk Utility does not have the ability to directly create either HFSX images or ownership.

Next, make the disk image writable:

sudo vsdbutil -a /Volumes/Builds

It is important to follow this step. If this step is ignored, items within the Builds directory may not be built correctly. This has the same effect as unchecking the Ignore ownership on this volume in the finder.

Initializing the build tree

Next, initialize the build directory. It is best to build with the current version of Mac OS X. Use System Profiler to find the build number. It is at the top level of Software. Currently, the build number is 8G32. The following sequence of commands, issued in order, creates the build tree.

sudo -s

cd /Volumes/Builds

mkdir Build8G32

cd Build8G32

darwinbuild -init 8G32

Attempting to download http://darwinsource.opendarwin.org/plists//8G32.plist ...
Download complete
Attempting to download http://darwinsource.opendarwin.org/plists//8F46.plist ...
Download complete
Attempting to download http://darwinsource.opendarwin.org/plists//8C46.plist ...
Download complete
Attempting to download http://darwinsource.opendarwin.org/plists//8B15.plist ...
Download complete
Attempting to download http://darwinsource.opendarwin.org/plists//8A428.plist ...
Download complete
Initialization Complete

ls

.build  Headers Logs    Roots   Sources Symbols

The last line shows the directory structure when finished. Each directory contains information for building, or is used during the build process.

• .build contains the plist files just downloaded, the darwinbuild database, as well as a file with a single line, the build number.
• Headers contains files that are created by using the installhdrs option. More information about this option can be found at the opendarwin site
• Logs is the directory that darwinbuild will use to place a log containing the build commands and results.
• Roots contains the built binaries.
• Sources contains the sources needed for building, as obtained from the opendarwin cvs respository
• Symbols contains the debug-symboled version of any binary built

In this article, the directories that are most important are in the Roots and Logs directory. The system is now ready to build and modify any open source project as described by the downloaded plist files.

Using darwinxref

darwinxref can be used to do many things before the build process begins. A few of the commands are listed below

1. Listing all project versions. All the current versions of Open Source projects can be listed with:

   darwinxref version '*'

2. Finding files. Files can be searched with the findFile option, once a project has been built with XDarwin. As an example, to find the binary top, enter:

   darwinxref findFile top

For more options, issue darwinxref without any command.

Patching an open source project

Begin by downloading the project to be patched. For this article, rsync will be patched.

Step 1, download the project source:

darwinbuild -fetch rsync

*** Fetching Sources ...
Attempting to download http://darwinsource.opendarwin.org/tarballs/other//rsync-20.tar.gz ...
Download complete

Once complete, the rsync archive, rsync-20.tar.gz is placed in the Sources directory.

Step 2, unpack the rsync directory for patching:

tar xvzf rsync-20.tar.gz

Step 3, change to the rsync-20 directory.

In this example, a patch for rsync from www.lartmaker.nl will be used. This file fixes some of the known issues with Tiger's rsync implementation. Get the diff file with curl:

curl -O http://www.lartmaker.nl/rsync/rsync-tiger-fixes.diff

Apply the patch to the source files:

patch -p0 < rsync-tiger-fixes.diff

The following should appear in the Terminal window:

patching file flist.c
patching file receiver.c
patching file rsync.c

Once complete, the rsync source files are patched.

Step 4, use darwinbuild to build the binary.

Change back to the root directory of the build tree, Build8G32 in our example. darwinbuild will not work if building from the root level directory, as it will not be able to find commands needed to successfully build any project. Issue the following command:

darwinbuild rsync

The build process will be presented on screen. The entire log of the success or failure of the build is kept in the Logs directory. When finished, the complete build time and exit status are shown:

BUILD TIME: 0h 2m 8s
EXIT STATUS: 0
rsync - 5 files registered.
                                         40755 0 0 0 ./usr
                                         40755 0 0 0 ./usr/bin
563fff635a3a8791c9ce33f6f5c1baf2e122c467 100755 0 0 448144 ./usr/bin/rsync
                                         40755 0 0 0 ./usr/local
                                         40755 0 0 0 ./usr/local/OpenSourceLicenses
075d599585584bb0e4b526f5c40cb6b17e0da35a 100444 0 0 17982 ./usr/local/OpenSourceLicenses/rsync.txt
                                         40755 0 0 0 ./usr/local/OpenSourceVersions
269a5a99d87ca48ae93cd63b4611c438252d2977 100444 0 0 595 ./usr/local/OpenSourceVersions/rsync.plist
                                         40755 0 0 0 ./usr/share
                                         40755 0 0 0 ./usr/share/man
                                         40755 0 0 0 ./usr/share/man/man1
db4acab463272aebdf6168687563b9e6dc5c57fa 100644 0 0 69473 ./usr/share/man/man1/rsync.1
                                         40755 0 0 0 ./usr/share/man/man5
508f51cd1bcf39fdd6b5909d70a86b7b2876ddeb 100644 0 0 24073 ./usr/share/man/man5/rsyncd.conf.5

If it does not exit with a status of 0, the build has failed.

The log file

After the project has been built, the complete log file of the build process is located with the project name, in the Logs directory. In this example, the log file is named rsync-20.log~1. If the project is built again, it will increment the serial number by 1, with the next log being rsync-20.log~2.

Deploying the patched binary

The darwinbuild project recently added a new command for patching and rollback of a binary on a working system. It is called darwinup. Though in its early stages, it provides the ability to install and rollback any newly built binary. The process is shown below:

1. Change to the Roots directory of the build tree, /Volumes/Builds/Build8G32/Roots/
2. Change to the build directory. It uses a similar format as the Logs directory does, but instead uses root in the projects name. In this example, it could be rsync-20.root~1.

3. Install the new binaries, and any other associated files with darwinup:

   darwinup install rsync-20.root~1

4. Verify that the binary is installed:

   darwinup list

The output should be similar to the following:

UUID                                  Date Installed           Name
====================================  =======================  =================
9C4023F1-BFFA-4F61-AE30-4364FB2C84E0  2006-01-05 13:07:48 EST  rsync-20.root~1

The UUID is the Universally Unique Identifier, and is how any further operations are performed.

Uninstallation

Uninstallation requires using the UUID, not the name of the archive. This operation is accomplished in the following manner:

`darwinup uninstall 9C4023F1-BFFA-4F61-AE30-4364FB2C84E0`

Now, type darwinup list, the binary should no longer appear.

Architectures

By default, darwinbuild will build any project as a universal binary. The size of any project is larger, as it will have both the ppc binary, and the i386 binary.

To change the build style, an environment variable can be used to build for either PPC or i386. However, a more effective method of doing so is to use the little known command line utility lipo. lipo can strip either architecture from the universal binary. In this example with rsync, a universal binary can be quickly converted to a ppc or i386 binary with the following command:

`lipo rsync -thin ppc -output rsync.ppc`

This command will create a file which is no longer a universal file, but specifically for the ppc architecture.

Final Notes

darwinbuild is a very powerful tool, and allows for much greater flexibility in patching and/or modifying the open source components included in Mac OS X. Using and providing feedback to the developers will strengthen the core foundation of Macintosh OS X.