Jonathan Perkin

Reducing RAM usage in pkgin

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 16 Jul 2015 00:00:00 +0100

Recently I’ve had a number of users complain about pkgin running out of memory when installing packages. This turned into a nice example of how to use DTrace to show memory allocations and help track down excessive use.

My test case was pkgin -y install gcc47. This is usually one of the first commands I run in a new SmartOS zone anyway, and as gcc47 happens to be one of the largest packages we ship it will help to exaggerate any memory allocation.

Trace heap allocations

As a first step I wanted to answer the question of how much memory was being allocated for pkgin. A simple and naive way to do this would be to run tools such as ps(1) or prstat(1) (the SmartOS equivalent to top(1)) whilst pkgin is running, and monitor the memory columns. This may give you a very rough idea of how much memory is being used, but it’s not very accurate and you may miss a large allocation just before the process exits.

Instead we can use DTrace to trace the brk() system calls and calculate the exact amount of memory that has been allocated. brk() is where libc memory allocation functions such as malloc() end up on SmartOS, so by tracing that single system call we can see exactly what has been allocated by the process.

Tracing brk() has the additional advantage of only showing heap growth. If we traced all the libc *alloc() calls, we would have to perform additional analysis to determine whether we actually allocated more memory or whether an existing allocation was reused. For more information about the different ways to trace memory allocations, see Brendan Gregg’s excellent Memory Flame Graphs page, which is where many of the DTrace scripts in this post are based on.

I used the following DTrace script to output 3 pieces of information over the lifetime of the target process:

A quantized set of brk() allocation sizes.
The total heap allocation.
The number of brk() calls.

Comments are inline. pid == $target ensures we only log brk() calls made by the process we specify as opposed to all brk() calls across the entire system, and arg0 is the argument to the brk() system call.

#!/usr/sbin/dtrace -qs

self int heap_ptr;

/*
 * The first brk() call by this application.  As we do not know the initial
 * value of the heap pointer, and thus be able to calculate the increase made
 * by this call, all we can do is save it.
 */
syscall::brk:entry
/pid == $target && !self->heap_ptr/
{
	self->heap_ptr = arg0;
}

/*
 * A subsequent brk() call.  Calculate the size of the allocation and
 * update our running totals.
 */
syscall::brk:entry
/pid == $target && self->heap_ptr != arg0/
{
	/* The heap grows up, so the size is simply (new addr - old addr) */
	this->size = arg0 - self->heap_ptr;

	/* A quantized distribution of allocation sizes. */
	@sizes["brk() allocation sizes"] = quantize(this->size);

	/* A running total of allocated bytes. */
	@bytes["Total bytes allocated"] = sum(this->size);

	/* The total number of calls to brk(). */
	@brks["Total number of brk() calls"] = count();

	/* Update our heap pointer for the next call. */
	self->heap_ptr = arg0;
}

Saving the script as brkquantize.d and running it gives us the following output:

: Start with a clean pkgin cache.
$ rm -rf /var/db/pkgin

: Ensure gcc47 is not installed, plus its dependency (binutils).
$ pkg_delete binutils gcc47

$ ./brkquantize.d -c "./pkgin-orig -py install gcc47"

  brk() allocation sizes
           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@       573
          131072 |@@                                       32
          262144 |@                                        23
          524288 |@@                                       39
         1048576 |                                         1
         2097152 |                                         2
         4194304 |                                         1
         8388608 |                                         2
        16777216 |                                         1
        33554432 |                                         0
        67108864 |                                         0
       134217728 |                                         1
       268435456 |                                         0

  Total bytes allocated                                     401752064
  Total number of brk() calls                                     675

Wow, that’s a lot of memory. 383MB has been allocated on the heap, with one of those allocations alone being between 128MB and 256MB. No wonder users are running out of memory!

This answers the questions regarding how much memory is being allocated, but doesn’t answer the question of what is causing it. I have my suspicions at this point (the gcc47 package tarball is 250MB, is pkgin caching the entire thing?), but in order to prove my suspicion I want to produce a flame graph.

Memory flame graph

If you didn’t read the earlier link to Brendan Gregg’s “Memory Flame Graphs” page, go and do that now. The reason for creating one is to see visually and easily which code paths are responsible for the allocations.

To create the memory flame graph I used a slightly modified version of Brendan’s brkbytes.d with additional comments:

#!/usr/sbin/dtrace -qs

self int prev;

syscall::brk:entry
/pid == $target/
{
	/* On entry, record the current heap pointer. */
	self->cur = arg0;
}

syscall::brk:return
/pid == $target && arg0 == 0 && self->prev/
{
	/* On return log the stack ordered by allocation size. */
	@[ustack()] = sum(self->cur - self->prev);
}

syscall::brk:return
/pid == $target && arg0 == 0/
{
	/* Save the previous heap pointer. */
	self->prev = self->cur;
}

Again we execute the script with pkgin as our target, after ensuring a clean environment:

$ rm -rf /var/db/pkgin/cache; pkg_delete binutils gcc47
$ ./brkbytes.d -c "./pkgin-orig -py install gcc47" >pkgin-orig.brkbytes

: Remove any pkgin output, we just want the stack traces.
$ vi pkgin-orig.brkbytes

Now we can use a couple of tools from Brendan’s FlameGraph repository to convert the stack traces into a flame graph:

: Download Brendan Gregg's "FlameGraph" repository
$ git clone https://github.com/brendangregg/FlameGraph.git
$ cd FlameGraph

$ ./stackcollapse.pl ~/pkgin-orig.brkbytes \
    | ./flamegraph.pl \
        --countname=bytes \
        --title='"pkgin install" heap expansion - original' \
        --colors=mem --width=696 > ~/pkgin-orig-brkbytes.svg

The resulting SVG is below, you should be able to mouse-over the individual elements for further details.

From the flame graph it’s clear that the majority of allocations are coming from download_file(), and we now have an accurate count of how much memory is being allocated by that function.

We can further drill down on our hypothesis by comparing sizes. The command we are running is downloading and installing these two files:

-rw-r--r--   1 root     root     9774130 Jul 10 10:47 binutils-2.24nb3.tgz
-rw-r--r--   1 root     root     261890422 Jul 10 10:47 gcc47-4.7.4.tgz

That’s a total of 271,664,552 bytes. According to the flame graph, download_file() allocated 271,671,296 bytes. So it seems highly likely it is caching those files, the 6,744 byte descrepancy likely due to rounding to the nearest page size (4K on SmartOS) and an additional page for something else.

Let’s go to the source to confirm.

Optimising `download_file()`

The download_file() function is reasonably straight-foward, and it’s quite clear that we are indeed reading the entire file into RAM before writing it out to disk. Source edited for clarity and added comments (full version here):

Dlfile *
download_file(char *str_url, ...
{
...
	/* Get information about the remote file. */
	f = fetchXGet(url, &st, "");
...
	/*
	 * Allocate our Dlfile structure, as well as a buffer equal to the size
	 * of the remote file.
	 */
	buf_len = st.size;
	XMALLOC(file, sizeof(Dlfile));
        XMALLOC(file->buf, buf_len + 1);
...
	/* Download the file 1024 bytes at a time into our allocated buffer */
	while (buf_fetched < buf_len) {
		cur_fetched = fetchIO_read(f, file->buf + buf_fetched, 1024);
		buf_fetched += cur_fetched;
	}
...
	/* NUL-terminate the buffer and return the buffer and size. */
	file->buf[buf_len] = '\0';
	file->size = buf_len;
	return file;
}

On return to the caller it writes the returned buffer to a file descriptor and then frees the buffer.

Optimising this is pretty straight-foward. We will instead pass an open file descriptor to a new download_pkg() function, which will stream to it directly from each successful fetchIO_read() via a static 4K buffer. The commit to implement this is here.

Running brkquantize.d on the new implementation we see significantly reduced memory usage:

$ ./brkquantize.d -c "./pkgin-dlpkg -py install gcc47"

  brk() allocation sizes
           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        511
          131072 |@@                                       32
          262144 |@                                        23
          524288 |@@@                                      43
         1048576 |                                         1
         2097152 |                                         2
         4194304 |                                         2
         8388608 |                                         1
        16777216 |                                         1
        33554432 |                                         0

  Total bytes allocated                                     134299648
  Total number of brk() calls                                     616

We’ve reduced our initial 383MB usage down to 128MB, and saved around 60 calls to brk() in the process - a good start.

Optimising `pkg_summary` handling

However 128MB still seems a lot for what the software is doing, can we do even better?

Let’s start with an updated memory flame graph to see where we stand with the new version:

It’s clear that our download_*() functions are no longer on the scene, and now the majority of the memory usage is caused by update_db(), accounting for 97MB. This function handles fetching the remote pkg_summary.bz2 file and transferring its contents into pkgin’s local sqlite3 database, which is then used for local queries.

Analysing update_db() is a little more involved than download_file(), but we can use flame graphs to help us identify which functions to look at. In this case we want to take a closer look at decompress_buffer() and insert_summary().

`decompress_buffer()`

After calling download_file() to fetch the the pkg_summary.bz2 file, the decompress_buffer() function is called to decompress it into memory and then free the download_file() allocation.

However, why uncompress the entire file before parsing it? Instead we can use libarchive to stream the decompression and process chunks at a time. As it turns out pkgin already links against libarchive but doesn’t actually use it, so this is easy enough to add.

`insert_summary()`

While parsing the pkg_summary buffer, a set of INSERT statements are constructed by this function. However, again we are buffering the whole lot, when instead we could just stream them one by one.

Testing streaming updates

I made some changes to implement streaming updates at each end, reading chunks of our compressed pkg_summary file and, once we’d read a complete record, stream an update to the database. Here’s how the allocations look afterwards:

$ ./brkquantize.d -c "./pkgin-streamsum -y up"
  brk() allocation sizes
           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 375
          131072 |                                         2
          262144 |                                         0
          524288 |                                         0
         1048576 |                                         1
         2097152 |                                         1
         4194304 |                                         0

  Total bytes allocated                                      30502912
  Total number of brk() calls                                     379

That’s better, now just 29MB to perform an update. However, that still seems quite a lot, so let’s generate an updated flame graph to see where the rest of the memory is being used.

Ok, so it’s clear the rest of the memory is being used by sqlite. Anything we can optimise there?

Turns out there is. I looked through pkgin to see if it was setting any non-default sqlite parameters, and the very first one immediately caught my eye:

static const char *pragmaopts[] = {
        "cache_size = 1000000",

The manual says that this value is in pages, with a default of 2000, and that the page size defaults to 1024 bytes, so we’re setting up a 976MB cache instead of the default 2MB. This seems to be rather larger than we need, so let’s try just removing that PRAGMA and using the default.

$ ./brkquantize.d -c "./pkgin-nocache -y up"
  brk() allocation sizes
           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   55
          131072 |@                                        1
          262144 |                                         0
          524288 |                                         0
         1048576 |@                                        1
         2097152 |@                                        1
         4194304 |                                         0

  Total bytes allocated                                       9388032
  Total number of brk() calls                                      58

That’s worked out very well, and we’re now down to just 9MB, which seems entirely reasonable to me. One final flame graph:

The majority of our usage is now handling the compressed pkg_summary.bz2 file. For now we will stop there as it’s only 2MB, but future work could include looking at streaming it directly from libfetch to libarchive rather than having to load it all into memory first.

Final thoughts

Given we’ve changed a lot of code, and especially options around cache sizes, how have they affected performance? We can’t be as accurate as with our DTrace measurements here, but we can perform a real-world benchmark of timing a pkgin update run against a localhost repository. I ran each multiple times and took the fastest result:

: Original
$ time ./pkgin-orig up
real	0m42.401s
user	0m40.835s
sys	0m0.829s

: Modified
$ time ./pkgin-nocache up
real	0m9.912s
user	0m9.005s
sys	0m0.652s

Less RAM and significantly faster? I’ll take that!

Summary

By using DTrace and Flame Graphs we are able to quickly identify code paths using large resources. By streaming data instead of caching we are able to significantly reduce the amount of RAM required and simultaneously boost performance.

With these commits in place:

the amount of RAM required to run pkgin install gcc47 on a clean SmartOS install reduces from 383MB to just 16MB.

I am hoping to get these changes in to the version of pkgin we ship for our 2015Q2 package sets, and will work to get these changes into upstream pkgin.

August 2015 update (streaming `pkg_summary`)

Since writing this post I revisited the improvement I mentioned where we can use libarchive to stream directly from libfetch rather than downloading the entire pkg_summary file first.

Let’s see where things stand with the 2015Q2 pkgin which includes all of the fixes described above:

$ rm -rf /var/db/pkgin; ./brkquantize.d -c "pkgin up"
  brk() allocation sizes
           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   55
          131072 |@                                        1
          262144 |                                         0
          524288 |                                         0
         1048576 |@                                        1
         2097152 |@                                        1
         4194304 |                                         0

  Total bytes allocated                                       9388032
  Total number of brk() calls                                      58

To integrate libfetch directly into libarchive, we split our download_summary() function into separate archive_read_open() callbacks. These callbacks are called when the archive is opened, read, and closed. Not only does this reduce our memory requirements, it also simplifies the code a little as libarchive can handle EOF and detect download failures.

The commit to implement this is here. One side-effect of this change is that now the remote INSERTions are performed inline, we need to remove the separate progress meter as it conflicts with the libfetch one.

With that change applied we can see the RSS has decreased by a further 2MB which corresponds to the size of the pkg_summary.bz2 file we were previously caching in RAM first:

$ rm -rf /var/db/pkgin; ./brkquantize.d -c "./pkgin up"
  brk() allocation sizes
           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  57
          131072 |@                                        1
          262144 |                                         0
          524288 |                                         0
         1048576 |                                         0
         2097152 |@                                        1
         4194304 |                                         0

  Total bytes allocated                                       7495680
  Total number of brk() calls                                      59

And for completeness sake, a final flame graph:

How does it affect runtime? Again I ran each multiple times against a localhost repository and took the fastest time:

: Original
$ ptime pkgin up
real       10.912125551
user        9.672048265
sys         0.778736990

: Modified
$ ptime ./pkgin up
real        9.487046885
user        9.069830722
sys         0.419527343

That’s another clear win, with a 2MB reduction in RSS usage, and 1.5 seconds shaved off the runtime.

pkgsrc-2014Q4: LTS, signed packages, and more

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 03 Mar 2015 00:00:00 +0000

The latest quarterly release of our binary package sets for SmartOS and illumos introduces a number of new features that I’m excited to announce.

Long Term Support

We have produced quarterly releases of pkgsrc for a number of years, and since the pkgsrc-2013Q2 release have built every package available (10,000+), but until now have not formalised our support for them.

This has meant that when serious security issues such as Heartbleed are disclosed, we are obliged to backport these fixes to every branch we have ever produced. Despite all our efforts on performance improvements this is still a large effort and takes a long time on older branches where we do not have huge resources available and backports can be tricker due to the differences involved.

We’ve tried as best we can to keep these older branches updated, but as we’ve added new branches each quarter the load increases further, and we cannot keep doing this forever.

So, from pkgsrc-2014Q4 (SmartOS 14.4.x images) we are introducing a new yearly Long Term Support (LTS) model, which can be summarised as:

Each Q4 release (pkgsrc-2014Q4, pkgsrc-2015Q4, …) will be an LTS release, and will receive suitable backports for 3 years from the time it is made available.
We will continue to produce the other quarterly releases (SmartOS 15.1.x images and onwards), so that users can get the latest packages available, but each of those releases will be closed for updates as soon as the next one is available.

What is a “suitable” backport? Anything which is a security or build fix, and which does not affect API or ABI compatibility. For example, we would not introduce a new major version of OpenSSL or PHP into an LTS release, but we would update OpenSSL from 1.0.1j to 1.0.1k or PHP from 5.4.37 to 5.4.38 as they are minor releases which only introduce fixes. We may also introduce new leaf packages (i.e. those with no dependencies), for example new releases of nodejs.

Who is the target market for each type of release?

LTS is primarily useful for people who have a very static set of requirements, do not like changes, and are primarily interested in ensuring that the software they run does not have active vulnerabilities.
Latest quarterly releases are for everyone else, users who want the latest stuff (and the latest security fixes), and are happy to reprovision their applications onto the newest images at regular intervals.

We hope the introduction of LTS releases satisfies both types of users, and that by freeing up our resources spent on maintaining our legacy branches, we can invest more time into ensuring the stability and security of the LTS releases.

For SmartOS users, the LTS releases will receive an additional -lts suffix on the image name to make it even easier to identify which are LTS.

Image Name Changes

Related to LTS, we are also slightly changing our naming scheme for the base images. This is to accommodate the new -lts suffix, and also to allow us to introduce a new “minimal” image, and make it clear which is which.

The current base image names are:

arch	name
32-bit	`base`
64-bit	`base64`
32-bit multiarch	`multiarch`

With the introduction of the “minimal” image, the new names will be:

arch	base name	minimal name
32-bit	`base-32`	`minimal-32`
64-bit	`base-64`	`minimal-64`
32-bit multiarch	`base-multiarch`	`minimal-multiarch`

And for LTS releases:

arch	base name	minimal name
32-bit	`base-32-lts`	`minimal-32-lts`
64-bit	`base-64-lts`	`minimal-64-lts`
32-bit multiarch	`base-multiarch-lts`	`minimal-multiarch-lts`

What’s the new “minimal” image? It’s effectively a stripped down “base”, with only the pkgsrc bootstrap and a couple of packages installed which are required for the zone to boot correctly. This will be of primary interest to users who have custom requirements for their zones and/or produce their own images, and want to ensure they are building on the smallest possible foundation.

As a quick comparison:

image	packages installed	image size (compressed)
base-multiarch-lts 14.4.0	58	161M
minimal-multiarch-lts 14.4.0	25	31M

The “minimal” images are fully functional and use the same package set, the only difference is fewer packages are installed by default.

Updated SmartOS Build Hosts

Up until now we have built all our package sets on an old SDC 6.5 install, to ensure that the packages we built can run across all hosts in the Joyent Public Cloud. Building on the lowest common denominator is great for compatibility, but has meant we are running on a limited number of older machines, and each quarterly release added yet more strain to already overloadeded systems.

Starting with 2014Q4 LTS we have moved to newer build hosts, running joyent_20141030T081701Z. This will soon be the most common platform available in the Joyent cloud, and ensures we aren’t tied to a legacy release for another 3 years. The next 3 quarterly releases (2015Q1-3) will also be produced on this platform, and we will then evaluate which platform to choose for the next LTS in 2015Q4.

This may mean incompatibilities if you are either running an older SmartOS release, or if you are running a different illumos distribution which does not have some of the newer SmartOS features. You are most likely to see issues where packages have picked up support for newer interfaces such as epoll or inotify, which have been introduced as part of the LX brand work.

Please feel free to raise a GitHub issue if this is causing problems for you. We are happy to turn off support for newer features if it improves compatibility, as often these features are picked up by autoconf checks but either aren’t used correctly or should be using different interfaces on illumos platforms anyway.

Signed Packages

One of the primary concerns in recent times is provenance, and ensuring that what you are receiving hasn’t been tampered with in any way. Until now our packages have been protected by checksums, so that it is difficult for an attacker to modify packages in-flight and deliver something we did not provide.

However, it isn’t impossible, and to further ensure that what you are installing came from Joyent we have implemented signed packages for 2014Q4 onwards. Here’s how it works:

During the package build process, a detached package hash file is created which contains a SHA512 checksum of the package.
This hash file is then signed with our PGP key.
All three files are bundled into an ar(1) archive and delivered as the package.tgz file.
At package install time, the archive is unpacked, the hash file is verified against our public PGP key, and then the package is checksummed against the recorded checksum in the hash file. If all these checks pass, the package is installed, otherwise the installation is aborted.

Let’s take a look at a package file (digest) to see in more detail:

: We need to use GNU ar(1) from binutils as the Sun format is too limited.
$ gar xv /path/to/digest-20121220.tgz
x - +PKG_HASH
x - +PKG_GPG_SIGNATURE
x - digest-20121220.tgz

The +PKG_HASH file contains all the details about the actual digest package which is stored in the archive.

$ cat +PKG_HASH
pkgsrc signature

version: 1
pkgname: digest-20121220
algorithm: SHA512
block size: 65536
file size: 49083

b011cb5e9cdea303f3958a7338b37fd85252313da354ff86a82170974f384700634c5fbe9d5f7035f67ff8a4eecacc6cfbff43ba4d62b4e4743837d72612feef
end pkgsrc signature

We can verify that the checksum is correct.

$ /usr/bin/digest -a sha512 digest-20121220.tgz
b011cb5e9cdea303f3958a7338b37fd85252313da354ff86a82170974f384700634c5fbe9d5f7035f67ff8a4eecacc6cfbff43ba4d62b4e4743837d72612feef

So we know that the +PKG_HASH file matches the digest-20121220.tgz package file. However, how do we know that both haven’t been tampered with? That’s where the +PKG_GPG_SIGNATURE file comes in. It is a detached signature of the +PKG_HASH file, signed with the Joyent key, so that if a malicious user has tampered with the package file and generated a new checksum, the +PKG_HASH file will no longer be verified and we know that it isn’t what was originally built.

We can verify that on the command line with GPG, as long as you have imported the public key for that package set:

$ gpg --verify +PKG_GPG_SIGNATURE +PKG_HASH
gpg: Signature made Sat Feb 21 02:10:43 2015 UTC using RSA key ID DE817B8E
gpg: Good signature from "Joyent Package Signing <pkgsrc@joyent.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 74C4 F303 BB45 7421 E42C  4DC4 FAE5 0048 FAA6 6EE0
     Subkey fingerprint: 2163 0D8B 4486 4587 9655  3748 76FA BBBB DE81 7B8E

A quick note about the warnings shown above. This is where the PGP web of trust comes in. We know that the files were signed with the DE817B8E key, but how do we know that the key belongs to pkgsrc@joyent.com? It essentially comes down to trust, and whether you believe this is really our key, or whether someone has tricked you to believe that when it’s not. We can help persuade you in a few ways:

The bootstrap packages and our SmartOS images will come by default with that key installed in /opt/local/etc/gnupg/pkgsrc.gpg. This is required so that you can start installing signed packages out of the box with no setup necessary.
The public keys will be published to PGP key servers, and I will sign them with my key (D532A578). My key in turn is signed by a number of other people, so that you can verify whether you believe I am who I say I am.
We will publish the keys in a couple of other places, certainly on the main http://pkgsrc.joyent.com/ site.

So, if you are a diligent user who checks all of these sources, an attacker would need to infiltrate every single one of them simultaneously to have a chance of delivering you a malicious packages which bypasses all of the checks. Hopefully you are convinced that this would be extremely difficult.

Finally, how is all of this used in practise? We’ve worked hard to make this as transparent as possible, including integration of Alistair Crooks’ excellent libnetpgpverify library into pkg_install, so as a user you should never be aware of any of it unless there is a problem (a core Unix philosophy):

As mentioned above, the PGP key is distributed by default, so you don’t need to import keys or anything to get started. We have added the following to /opt/local/etc/pkg_install.conf:

GPG_KEYRING_VERIFY=/opt/local/etc/gnupg/pkgsrc.gpg
VERIFIED_INSTALLATION=trusted

GPG_KEYRING_VERIFY is set to our public key, and VERIFIED_INSTALLATION=trusted means that a signature is required, and if one isn’t available then you are prompted for how to proceed. Trying to install the package file from our ar(1) archive example above shows what happens:

$ pkg_add ./digest-20121220.tgz
No valid signature found for digest-20121220.
Do you want to proceed with the installation [y/n]?
n
Cancelling installation
pkg_add: 1 package addition failed

And if we try to install a package with an incorrect signature/hash:

$ ed +PKG_HASH >/dev/null 2>&1 <<EOF
/^b011/s/b011/b010/
w
q
EOF
$ gar r test.tgz +PKG_HASH +PKG_GPG_SIGNATURE digest-20121220.tgz
gar: creating test.tgz
$ pkg_add ./test.tgz
pkg_add: unable to verify signature: Signature on data did not match

If you build your own packages then you’re going to want to handle this properly. The simplest option is to use a custom pkg_install.conf when installing your own packages, for example:

$ echo "VERIFIED_INSTALLATION=never" >pkg_install_noverify.conf
$ pkg_add -C ./pkg_install_noverify.conf ./digest-20121220.tgz

The alternative is to sign your own packages. This is reasonably straight-forward:

Enable SIGN_PACKAGES in /opt/local/etc/mk.conf:

SIGN_PACKAGES=	gpg

Install GPG, create a signing key, and then configure /opt/local/etc/pkg_install.conf with:

GPG=/path/to/gpg
GPG_SIGN_AS=your_pgp_key_id

With those additions, pkgsrc will prompt you for your PGP passphrase at package time, and then sign the package with the key you have configured. You can use gpg-agent to automate this in a controlled environment.

Bundled `pkg-vulnerabilities` Verification

Closely related to package signing, now that we have infrastructure support for verification in our bootstrap packages, we’ve also enabled easy verification of the pkg-vulnerabilities file. For those who aren’t aware, there is a team of volunteers for pkgsrc who maintain a list of security vulnerabilities, which can be checked against the list of installed packages and show you which ones are currently vulnerable.

: Fetch the latest pkg-vulnerabilities file.  SmartOS images have a crontab
: entry which does this nightly by default.
$ pkg_admin fetch-pkg-vulnerabilities

: The file is a compressed signed message containing a list of all known
: vulnerabilities.
$ gzip -dc /opt/local/pkg/pkg-vulnerabilities | nl -ba | sed -ne '1,4p' -e '27,28p'
     1  -----BEGIN PGP SIGNED MESSAGE-----
     2  Hash: SHA1
     3
     4  # $NetBSD: pkg-vulnerabilities,v 1.6187 2015/03/02 14:22:28 ryoon Exp $
    27  # package               type of exploit         URL
    28  cfengine<1.5.3nb3       remote-root-shell       ftp://ftp.NetBSD.org/pub/NetBSD/security/advisories/NetBSD-SA2000-013.txt.asc

: Show list of current known vulnerabilities
$ pkg_admin audit
Package gcc47-4.7.3nb6 has a denial-of-service vulnerability, see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61601
Package gcc47-4.7.3nb6 has a memory-corruption vulnerability, see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61582
Package mit-krb5-1.10.7nb4 has a denial-of-service vulnerability, see http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-5353
[...]

This provides you as the administrator with the information necessary to decide whether the current vulnerabilities are acceptable in your environment.

However, there is a missing piece. As you can see above, the vulnerabilities file is signed. This is important as an attacker with access to modify this file could hide vulnerabilities from you and leave your system exposed. With the verification infrastructure now in place, we can now provide the pkgsrc-security PGP key for you to easily verify that the pkg-vulnerabilities file is as expected.

First we need to install the pkgsrc-security@pkgsrc.org PGP key on the system. As this key changes quite frequently, we cannot include it directly in the bootstrap tarball as we have done with the package signing key, as it will eventually be out of date. So we instead provide a new pkgsrc-gnupg-keys package which includes it, bundle that in the bootstrap, and we can then distribute updates to this package as normal via pkgin upgrade.

: The package contains a PGP keyring with the current pkgsrc-security key.
$ pkg_info -qL pkgsrc-gnupg-keys
/opt/local/share/gnupg/pkgsrc-security.gpg

We then add that file to our /opt/local/etc/pkg_install.conf file with:

GPG_KEYRING_PKGVULN=/opt/local/share/gnupg/pkgsrc-security.gpg

To verify the pkg-vulnerabilities file, use pkg_admin again:

: Verify the basic checksum, looks good.
$ pkg_admin check-pkg-vulnerabilities /opt/local/pkg/pkg-vulnerabilities

: Verify the PGP signature, looks good.
$ pkg_admin check-pkg-vulnerabilities -s /opt/local/pkg/pkg-vulnerabilities

: Modify the file and try again, checks fail.
$ gzip -dc /opt/local/pkg/pkg-vulnerabilities \
  | grep -v 'mutt.*denial-of-service' \
  | gzip -9 >/var/tmp/pkg-vulnerabilities-test

$ pkg_admin check-pkg-vulnerabilities /var/tmp/pkg-vulnerabilities-test
pkg_admin: SHA1 hash doesn't match

$ pkg_admin check-pkg-vulnerabilities -s /var/tmp/pkg-vulnerabilities-test
pkg_admin: unable to verify signature: Signature on data did not match

Add these checks to your automated reports to ensure you aren’t being lied to about possible vulnerabilities.

Reduced Package REQUIRES

Each package lists the libraries that it requires, and those are checked prior to installation to ensure the package will work correctly on the target host. Recently we’ve seen a few issues where some illumos distributions have moved platform libraries to a different location (but still in the default search path), which means the REQUIRES no longer match and the package won’t install.

From 2014Q4 we have reduced the way that REQUIRES are computed. Previously every library that was pulled in was recorded, essentially using the output of ldd, so for example with the libpcap package you end up with:

$ pkg_info -Q REQUIRES libpcap
/lib/libavl.so.1
/lib/libc.so.1
/lib/libcurses.so.1
/lib/libdevinfo.so.1
/lib/libdladm.so.1
/lib/libdlpi.so.1
/lib/libgen.so.1
/lib/libinetutil.so.1
/lib/libkstat.so.1
/lib/libm.so.2
/lib/libmd.so.1
/lib/libmp.so.2
/lib/libnsl.so.1
/lib/libnvpair.so.1
/lib/libpthread.so.1
/lib/librcm.so.1
/lib/libscf.so.1
/lib/libsec.so.1
/lib/libsocket.so.1
/lib/libumem.so.1
/lib/libuutil.so.1
/lib/libxml2.so.2
/lib/libz.so.1
/opt/local/gcc47/i386-sun-solaris2.11/lib/./libgcc_s.so.1
/usr/lib/libexacct.so.1
/usr/lib/libidmap.so.1
/usr/lib/libpool.so.1
/usr/lib/libsmbios.so.1

In 2014Q4 we have stopped using ldd to resolve the library dependencies, and instead use elfdump to only look at the NEEDED entries that are recorded in the SHT_DYNAMIC section for each executable. This results in a much simpler and direct list:

$ pkg_info -Q REQUIRES libpcap
/lib/libc.so.1
/lib/libdlpi.so.1
/lib/libnsl.so.1
/lib/libsocket.so.1
/lib/libumem.so.1
/opt/local/gcc47/i486-sun-solaris2.11/lib/./libgcc_s.so.1

due to excluding all of /lib/libdlpi.so.1’s dependencies, and increases the portability of our packages across illumos distributions.

We also get a side benefit of being able to easily identify packages which are incorrectly linking against system versions of e.g. libxml2.so.2 when they should instead be using the pkgsrc version.

Miscellaneous Improvements

There is the usual grab bag of updates in 2014Q4/14.4.x:

go-1.4.2 now includes Keith Wesolowski’s patches to add support for cgo. This brings Go for illumos up to feature parity with other operating systems and increases the amount of Go software that will build and run.
SmartOS 14.4.x images now deliver an SSL bundle in /etc which makes Go work correctly, and we also ensure that /usr/bin/curl has access to certificates.
libgo has been removed from the gcc47-libs package. It is unused, and doing this saves 40MB from the bootstrap kits and images.
We now build with cwrappers, as detailed in my performance post. This speeds up the builds a lot, so in the event of another Heartbleed we should be able to deliver updated packages a lot faster.
pkgin is now at version 0.8.0 including support for the new preferred.conf, plus a number of important bug fixes.
A number of small internal improvements to the build infrastructure. As a user you shouldn’t notice any changes, if you do please let us know!

Plus all the usual upstream pkgsrc changes as announced here.

As always, please raise a GitHub issue if you run into any problems or have any suggestions on ways we can improve any of this stuff.

Enjoy!

Building packages at scale

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 06 Oct 2014 00:00:00 +0100

tl;dr We are able to build 14,000 packages across 6 zones in 4.5 hours

At Joyent we have long had a focus on high performance, whether it’s through innovations in SmartOS, carefully selecting our hardware, or providing customers with tools such as DTrace to identify bottlenecks in their application stacks.

When it comes to building packages for SmartOS it is no different. We want to build them as quickly as possible, using the fewest resources, but without sacrificing quality or consistency.

To give you an idea of how many packages we build, here are the numbers:

Branch	Arch	Success	Fail	Total
2012Q4	i386	2,245	20	2,265
2012Q4	x86_64	2,244	18	2,262
2013Q1	i386	2,303	40	2,343
2013Q1	x86_64	2,302	39	2,341
2013Q2	i386	10,479	1,277	11,756
2013Q2	x86_64	10,290	1,272	11,562
2013Q3	i386	11,286	1,317	12,603
2013Q3	x86_64	11,203	1,308	12,511
2013Q4	i386	11,572	1,277	12,849
2013Q4	x86_64	11,498	1,270	12,786
2014Q1	i386	12,450	1,171	13,621
2014Q1	x86_64	12,356	1,150	13,506
2014Q2	i386	13,132	1,252	14,384
2014Q2	x86_64	13,102	1,231	14,333
Total				139,122

Now of course we don’t continuously attempt to build 139,122 packages. However, when something like Heartbleed happens, we backport the fix to all of these branches, and a rebuild of something as heavily depended upon as OpenSSL can cause around 100,000 packages to be rebuilt.

Each quarter we add another release branch to our builds, and as you can see from the numbers above (2013Q1 and earlier were limited builds) the total number of packages in pkgsrc grows with each release.

Recently I’ve been focussing on improving the bulk build performance, both to ensure that fixes such as Heartbleed are delivered as quickly as possible, and also to ensure we aren’t wasteful in our resource usage as our package count grows. All of our builds happen in the Joyent public cloud, so any resources we are using are taking away from the available pool to sell to customers.

Let’s first take a walk through pkgsrc bulk build history, and then look at some of the performance wins I’ve been working on.

pkgsrc bulk builds, 2004

The oldest bulk build I performed that I can find is this one. My memory is a little fuzzy on what hardware I was using at the time, but I believe it was a SunFire v120 (1 x UltraSPARCIIi CPU @ 650MHz) with 2GB RAM. This particular build was on Solaris 8.

As you can see from the results page, it took 13.5 days to build 1,810 (and attempt but fail to build 1,128) packages!

Back then the build would have been single threaded with only one package being built at a time. There was no support for concurrent builds, make -j wouldn’t have helped much, and essentially you just needed to be very patient.

May 2004: 2,938 packages in 13.5 days

pkgsrc bulk builds, 2010

Fast forward 6 years. At this point I’m building on much faster x86-based hardware (a Q9550 Core2Quad @ 2.83GHz and 16G RAM) running Solaris 10, however the builds are still single threaded and take 4 days to build 5,524 (and attempt but fail to build 1,325) packages.

All of the speed increase is coming directly from faster hardware.

May 2010: 6,849 packages in 4 days

pkgsrc @ Joyent, 2012 onwards

Shortly after joining Joyent, I started setting up our bulk build infrastructure. The first official build from this was for general illumos use. We were able to provide over 9,000 binary packages which took around 7 days to build.

At this point we’re starting to see the introduction of very large packages such as qt4, kde4, webkit, etc. These packages take a significant amount of time to build, so even though we are building on faster hardware than previously, the combination of an increased package count as well as individual package build times increasing mean we’re not seeing a reduction in total build time.

July 2012: 10,554 packages in 7 days

Performance improvements

At this point we start to look at ways of speeding up the builds themselves. As we have the ability to create build zones as required, the first step was to introduce distributed builds.

pbulk distributed builds

For pkgsrc in the 2007 Google Summer of Code Jörg Sonnenberger wrote pbulk, a replacement for the older bulk build infrastructure that had been serving us well since 2004 but had started to show its age. One of the primary benefits of pbulk was that it supported a client/server setup to distribute builds, and so I worked on building across 6 separate zones. From my work log:

2012-09-25 (Tuesday)

 - New pbulk setup managed a full bulk build (9,414 packages) in 54 hours,
   since then I've added another 2 clients which should get us well under 2
   days.

September 2012: 10,634 packages in 2 days

Distributed chrooted builds

By far the biggest win so far was in June 2013, however I’m somewhat ashamed that it took me so long to think of it. By this time we were already using chroots for builds, as it ensures a clean and consistent build environment, keeps the host zone clean, and also allowed us to perform concurrent branch builds (e.g. building i386 and x86_64 packages simultaneously on the same host but in separate chroots).

What it took me 9 months to realise, however, was that we could simply use multiple chroots for each branch build! This snippet from my log is enlightening:

2013-06-06 (Thursday)

 - Apply "duh, why didn't I think of that earlier" patch to the pbulk cluster
   which will give us massively improved concurrency and much faster builds.

2013-06-07 (Friday)

 - Initial results from the re-configured pbulk cluster show it can chew
   through 10,000 packages in about 6 hours, producing 9,000 binary pkgs.
   Not bad.  Continue tweaking to avoid build stalls with large dependent
   packages (e.g. gcc/webkit/qt4).

Not bad indeed. The comment is somewhat misleading, though, as this comment I made on IRC on June 15th alludes to:

22:27 < jperkin> jeez lang/mercury is a monster
22:28 < jperkin> I can build over 10,000 packages in 18 hours, but that
                 one package alone takes 7.5 hours before failing.

Multiple distributed chroots get us an awfully long way, but now we’re stuck with big packages which ruin our total build times, and no amount of additional zones or chroots will help.

However, we are now under 24 hours for a full build for the first time. This is of massive benefit, as we can now do regular daily builds.

June 2013: 11,372 packages in 18 hours

make -j vs number of chroots

An ongoing effort has been to optimise the MAKE_JOBS setting used for each package build, balanced against the number of concurrent chroots. There are a number of factors to consider:

The vast majority of ./configure scripts are single threaded, so generally you should trade extra chroots for less MAKE_JOBS.
The same goes for other phases of the package build (fetch, checksum, extract, patch, install, package).
Packages which are highly depended upon (e.g. GCC, Perl, OpenSSL) should have a high MAKE_JOBS as even with a large number of chroots enabled, most of them will be idle waiting for those builds to complete.
Larger packages are built towards the end of a bulk build run (e.g. KDE, Firefox) and these tend to be large builds. Similar to above, as they are later in the build there will be fewer chroots active, so a higher MAKE_JOBS can be afforded.

Large packages like webkit will happily burn as many cores as you give them and return you with faster build times, however giving them 24 dedicated cores isn’t cost-effective. Our 6 build zones are sized at 16 cores / 16GB DRAM, and so far the sweet spot seems to be:

8 chroots per build (bump to 16 if the build is performed whilst no other builds are happening).
Default MAKE_JOBS=2.
MAKE_JOBS=4 for packages which don’t have many dependents but are generally large builds which benefit from additional parallelism.
MAKE_JOBS=6 for webkit.
MAKE_JOBS=8 for highly-dependent packages which stall the build, and/or are built right at the end.

The MAKE_JOBS value is determined based on the current PKGPATH and is dynamically generated so we can easily test new hypotheses.

With various tweaks in place, fixes to packages etc., we were running steady at around 12 hrs for a full build.

August 2014: 14,017 packages in 12 hours

cwrappers

There are a number of unique technologies in pkgsrc that have been incredibly useful over the years. Probably the most useful has been the wrappers in our buildlink framework, which allows compiler and linker commands to be analysed and modified before being passed to the real tool. For example:

# Remove any hardcoded GNU ld arguments unsupported by the SunOS linker
.if ${OPSYS} == "SunOS"
BUILDLINK_TRANSFORM+=  rm:-Wl,--as-needed
.endif

# Stop using -fomit-frame-pointer and producing useless binaries!  Transform
# it to "-g" instead, just in case they forgot to add that too.
BUILDLINK_TRANSFORM+=  opt:-fomit-frame-pointer:-g

There are a number of other features of the wrapper framework, however it doesn’t come without cost. The wrappers are written in shell, and fork a large number of sed and other commands to perform replacements. On platforms with an expensive fork() implementation this can have quite a detrimental effect on performance.

Jörg again was heavily involved in a fix for this, with his work on cwrappers, which replaced the shell scripts with C implementations. Despite being 99% complete, the final effort to get it over the line and integrated into pkgsrc hadn’t been finished, so in September 2014 I took on the task and the sample package results speak for themselves:

Package	Legacy wrappers	C wrappers	Speedup
wireshark	3,376 seconds	1,098 seconds	3.07x
webkit1-gtk	11,684 seconds	4,622 seconds	2.52x
qt4-libs	11,866 seconds	5,134 seconds	2.31x
xulrunner24	10,574 seconds	5,058 seconds	2.09x
ghc6	2,026 seconds	1,328 seconds	1.52x

As well as reducing the overall build time, the significant reduction in number of forks meant the system time was a lot lower, allowing us to increase the number of build chroots. The end result was a reduction of over 50% in overall build time!

The work is still ongoing to integrate this into pkgsrc, and we hope to have it done for pkgsrc-2014Q4.

September 2014: 14,011 packages in 5 hours 20 minutes

Miscellaneous fork improvements

Prior to working on cwrappers I was looking at other ways to reduce the number of forks, using DTrace to monitor each internal pkgsrc phase. For example the bmake wrapper phase generates a shadow tree of symlinks, and in packages with a large number of dependencies this was taking a long time.

Running DTrace to count totals of execnames showed:

$ dtrace -n 'syscall::exece:return { @num[execname] = count(); }'
  [...]
  grep                                                             94
  sort                                                            164
  nbsed                                                           241
  mkdir                                                           399
  bash                                                            912
  cat                                                            3893
  ln                                                             7631
  rm                                                             7766
  dirname                                                        7769

Looking through the code showed a number of ways to reduce the large number of forks happening here

cat -> echo

cat was being used to generate a sed script, which sections such as:

cat <<EOF
s|^$1\(/[^$_sep]*\.la[$_sep]\)|$2\1|g
s|^$1\(/[^$_sep]*\.la\)$|$2\1|g
EOF

There’s no need to fork here, we can just use the builtin echo command instead:

echo "s|^$1\(/[^$_sep]*\.la[$_sep]\)|$2\1|g"
echo "s|^$1\(/[^$_sep]*\.la\)$|$2\1|g"

Use shell substitution where possible

The dirname commands were being operated on full paths to files, and in this case we can simply use POSIX shell substitution instead, i.e.:

dir=`dirname $file`

becomes:

dir="${file%/*}"

Again, this saves a fork each time. This substitution isn’t always possible, for example if you have trailing slashes, but in our case we were sure that $file was correctly formed.

Test before exec

The rm commands were being unconditionally executed in a loop:

for file; do
	rm -f $file
	..create file..
done

This is an expensive operation when you are running it on thousands of files each time, so simply test for the file first and use a cheap (and builtin) stat(2) call instead of forking an expensive unlink(2) for the majority of cases.

for file; do
	if [ -e $file ]; then
		rm -f $file
	fi
	..create file..
done

At this point we had removed most of the forks, with DTrace confirming:

$ dtrace -n 'syscall::exece:return { @num[execname] = count(); }'
  [...]
  [ full output snipped for brevity ]
  grep                                                             94
  cat                                                             106
  sort                                                            164
  nbsed                                                           241
  mkdir                                                           399
  bash                                                            912
  ln                                                             7631

The result was a big improvement, going from this:

$ ptime bmake wrapper
  real     2:26.094442113
  user       32.463077360
  sys      1:48.647178135

to this:

$ ptime bmake wrapper
  real       49.648642097
  user       14.952946135
  sys        33.989975053

Again note, not only are we reducing the overall runtime, but the system time is significantly less, improving overall throughput and reducing contention on the build zones.

Batch up commands

The most recent changes I’ve been working on have been to further reduce forks both by caching results and batching up commands where possible. Taking the previous example again, the ln commands are a result of a loop similar to:

while read src dst; do
	src=..modify src..
	dst=..modify dst..
	ln -s $src $dst
done

Initially I didn’t see any way to optimise this, but upon reading the ln manpage I observed the second form of the command which allows you to symlink multiple files into a directory at once, for example:

$ ln -s /one /two /three dir
$ ls -l dir
lrwxr-xr-x 1 jperkin staff 4 Oct  3 15:40 one -> /one
lrwxr-xr-x 1 jperkin staff 6 Oct  3 15:40 three -> /three
lrwxr-xr-x 1 jperkin staff 4 Oct  3 15:40 two -> /two

As it happens, this is ideally suited to our task as $src and $dst will for the most part have the same basename.

Writing some awk allows us to batch up the commands and do something like this:

while read src dst; do
	src=..modify src..
	dst=..modify dst..
	echo "$src:$dst"
done | awk -F: '

{
	src = srcfile = $1;
	dest = destfile = destdir = $2;
	sub(/.*\//, "", srcfile);
	sub(/.*\//, "", destfile);
	sub(/\/[^\/]*$/, "", destdir);
	# 
	# If the files have the same name, add them to the per-directory list
	# and use the 'ln file1 file2 file3 dir/' style, otherwise perform a
	# standard 'ln file1 dir/link1' operation.
	#
	if (srcfile == destfile) {
		if (destdir in links)
			links[destdir] = links[destdir] " " src
		else
			links[destdir] = src;
	} else {
		renames[dest] = src;
	}
	#
	# Keep a list of directories we've seen, so that we can batch them up
	# into a single 'mkdir -p' command.
	#
	if (!(destdir in seendirs)) {
		seendirs[destdir] = 1;
		if (dirs)
			dirs = dirs " " destdir;
		else
			dirs = destdir;
	}
}
END {
	#
	# Print output suitable for piping to sh.
	#
	if (dirs)
		print "mkdir -p " dirs;
	for (dir in links)
		print "ln -fs " links[dir] " " dir;
	for (dest in renames)
		print "ln -fs " renames[dest] " " dest;
}

' | sh

There’s an additional optimisation here too - we keep track of all the directories we need to create, and then batch them up into a single mkdir -p command.

Whilst this adds a considerable amount of code to what was originally a simple loop, the results are certainly worth it. The time for bmake wrapper in kde-workspace4 which has a large number of dependencies (and therefore symlinks required) reduces from 2m11s to just 19 seconds.

Batching wrapper creation: 7x speedup

Cache results

One of the biggest recent wins was in a piece of code which checks each ELF binary’s DT_NEEDED and DT_RPATH to ensure they are correct and that we have recorded the correct dependencies. Written in awk there were a couple of locations where it forked a shell to run commands:

cmd = "pkg_info -Fe " file
if (cmd | getline pkg) {

...

if (!system("test -f " libfile))) {

These were in functions that were called repeatedly for each file we were checking, and in a large package there may be lots of binaries and libraries which need checking. By caching the results like this:

if (file in pkgcache)
	pkg = pkgcache[file]
else
	cmd = "pkg_info -Fe " file
	if (cmd | getline pkg) {
		 pkgcache[file] = pkg

...

if (!(libfile in libcache))
	libcache[libfile] = system("test -f " libfile)
if (!libcache[libfile]) {

This simple change made a massive difference! The kde-workspace4 package includes a large number of files to be checked, and the results went from this:

$ ptime bmake _check-shlibs
=> Checking for missing run-time search paths in kde-workspace4-4.11.5nb5

real     7:55.251878017
user     2:08.013799404
sys      5:14.145580838

$ dtrace -n 'syscall::exece:return { @num[execname] = count(); }'
dtrace: description 'syscall::exece:return ' matched 1 probe
  [...]
  greadelf                                                        298
  pkg_info                                                       5809
  ksh93                                                         95612

to this:

$ ptime bmake _check-shlibs
=> Checking for missing run-time search paths in kde-workspace4-4.11.5nb5

real       18.503489661
user        6.115494568
sys        11.551809938

$ dtrace -n 'syscall::exece:return { @num[execname] = count(); }'
dtrace: description 'syscall::exece:return ' matched 1 probe
  [...]
  pkg_info                                                        114
  greadelf                                                        298
  ksh93                                                          3028

Cache awk system() results: 25x speedup

Avoid unnecessary tests

The biggest win so far though was also the simplest. One of the pkgsrc tests checks all files in a newly-created package for any #! paths which point to non-existent interpreters. However, do we really need to test all files? Some packages have thousands of files, and in my opinion, there’s no need to check files which are not executable.

We went from this:

if [ -f $file ]; then
	..test $file..

  real     1:36.154904091
  user       17.554778405
  sys      1:10.566866515

to this:

if [ -x $file ]; then
	..test $file..

  real        2.658741177
  user        1.339411743
  sys         1.236949825

Again DTrace helped in identifying the hot path (30,000+ sed calls in this case) and narrowing down where to concentrate efforts.

Only test shebang in executable files: ~50x speedup

Miscellaneous system improvements

Finally, there have been some other general improvements I’ve implemented over the past few months.

bash -> dash

dash is renowned as being a leaner, faster shell than bash, and I’ve certainly observed this when switching to it as the default $SHELL in builds. The normal concern is that there may be non-POSIX shell constructs in use, e.g. brace expansion, but I’ve observed relatively few of these, with the results being (prior to some of the other performance changes going in):

Shell	Successful packages	Average total build time
bash	13,050	5hr 25m
dash	13,020	5hr 10m

It’s likely with a small bit of work fixing non-portable constructs we can bring the package count for dash up to the same level. Note that the slightly reduced package count does not explain the reduced build time, as those failed packages have enough time to complete successfully before other larger builds we’re waiting on are completed anyway.

Fix libtool to use printf builtin

libtool has a build-time test to see which command it should call for advanced printing:

# Test print first, because it will be a builtin if present.
if test "X`( print -r -- -n ) 2>/dev/null`" = X-n && \
   test "X`print -r -- $ECHO 2>/dev/null`" = "X$ECHO"; then
  ECHO='print -r --'
elif test "X`printf %s $ECHO 2>/dev/null`" = "X$ECHO"; then
  ECHO='printf %s\n'
else
  # Use this function as a fallback that always works.
  func_fallback_echo ()
  {
    eval 'cat <<_LTECHO_EOF
$[]1
_LTECHO_EOF'
  }
  ECHO='func_fallback_echo'
fi

Unfortunately on SunOS, there is an actual /usr/bin/print command, thanks to ksh93 polluting the namespace. libtool finds it and so prefers it over printf, which is a problem as there is no print in the POSIX spec, so neither dash nor bash implement it as a builtin.

Again, this is unnecessary forking that we want to fix (libtool is called a lot during a full bulk build!) Thankfully pkgsrc makes this easy - we can just create a broken print command which will be found before /usr/bin/print:

.PHONY: create-print-wrapper
post-wrapper: create-print-wrapper
create-print-wrapper:
	${PRINTF} '#!/bin/sh\nfalse\n' > ${WRAPPER_DIR}/bin/print
	${CHMOD} +x ${WRAPPER_DIR}/bin/print

saving us millions of needless execs.

Parallelise where possible

There are a couple of areas where the pkgsrc bulk build was single threaded:

Initial package tools bootstrap

It was possible to speed up the bootstrap phase by adding custom make -j support, reducing the time by a few minutes.

Package checksum generation

Checksum generation was initially performed at the end of the build running across all of the generated packages, so an obvious fix for this was to perform individual package checksum generation in each build chroot after the package build had finished and then simply gather up the results at the end.

`pkg_summary.gz` generation

Similarly for pkg_summary.gz we can generate individual per-package pkg_info -X output and then collate it at the end.

Optimising these single-threaded sections of the build resulted in around 20 minutes being taken off the total runtime.

Summary

The most recent build with all these improvements integrated together is here, showing a full from-scratch bulk build taking under 5 hours to build over 14,000 packages. We’ve come a long way since 2004:

Date	Package Builds	Total Build Time (hours)
2004/05	2,938	322
2010/05	6,849	100.5
2012/07	10,554	166.5
2012/10	10,634	48
2013/06	11,372	18
2014/08	14,017	12
2014/10	14,162	4.5

We’ve achieved this through a number of efforts:

Distributed builds to scale across multiple hosts.
Chrooted builds to scale on individual hosts.
Tweaking make -j according to per-package effectiveness.
Replacing scripts with C implementations in critical paths.
Reducing forks by caching, batching commands, and using shell builtins where possible.
Using faster shells.
Parallelising single-threaded sections where possible.

What’s next? There are plenty of areas for further improvements:

Improved scheduling to avoid builds with high MAKE_JOBS from sharing the same build zone.
make(1) variable caching between sub-makes.
Replace /bin/sh on illumos (ksh93) with dash (even if there is no appetite for this upstream, thanks to chroots we can just mount it as /bin/sh inside each chroot!)
Dependency graph analysis to focus on packages with the most dependencies.
Avoid the “long tail” by getting the final few large packages building as early as possible.
Building in memory file systems if build size permits.
Avoid building multiple copies of libnbcompat during bootstrap.

Many thanks to Jörg for writing pbulk and cwrappers, Google for sponsoring the pbulk GSoC , the pkgsrc developers for all their hard work in adding and updating packages, and of course Joyent for employing me to work on this stuff.

A node.js-powered 8-bit CPU - part four

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 04 Dec 2013 00:00:00 +0000

This post is part four of a series, the other posts available are

In part two we constructed a circuit using shift registers whereby we could write a byte of data and then read it back in. In part three we booted a 6309 CPU and ran it with a hard-coded operation on the data bus. In this final post, we will put the two together so that we can write arbitrary data onto the data bus and then read back any results.

First, we need to introduce one more 7400-series IC, the 74HC14 hex inverter.

Inverters

These are really simple chips, all they do is take input on a pin, and then output the opposite onto another pin.

The “A”s are input and “Y”s are output. So, inputting a 0 into 1A/Pin1 results in a 1 being output from 1Y/Pin2. No need for any clocks, just power, and we have 6 converters available.

Hooking It All Up

Ok, let’s connect everything up. Here’s what we need to do:

Connect the two bread boards together, with the shift register board above the CPU board. The bread boards have tongue and groove joints for a snug fit.
Wire up +5V and ground rails between the two so that they both have power from a single source.
Wire up the D0-D7 data bus from the 6309 to the data bus on the shift register board - it is easiest to hook them in just below the green LEDs.

The GPIO pins from the Raspberry Pi are connected as follows:

GPIO17/Pin11 goes to Q on the 6309.
GPIO18/Pin12 goes to E on the 6309.
GPIO21/Pin13 goes to STCP on the 74HC595.
GPIO22/Pin15 goes to DS on the 74HC595.
GPIO23/Pin16 goes to Q7 on the 74HC165.
GPIO24/Pin18 goes to RESET on the 6309.

There are 4 more connections we need to make, which is where the 74HC14 comes in.

RW On The 6309

The RW pin on the 6309 denotes whether from the 6309’s point of view the bus is being written to or read from. When high, the 6309 is reading, and when low, it is writing.

We can use this feature to automatically switch the OE (“output enable”) pin on the 74HC595 on or off, so that when the 6309 is reading OE is enabled, and when it is writing OE is disabled, leaving the data bus clear for the 6309s data. In this way the bus can be shared between devices.

However, we cannot connect them directly, as the logic is the wrong way around

we want OE to be low when RW is high and vice versa. So, we use the 74HC14, and by connecting them through pins 1A and 1Y on the 74HC14, we get the logic we want.

PL On The 74HC165

The final connection we need to make is a way to trigger the 74HC165 to read from the data bus, so that we can get data back into the Raspberry Pi.

The solution for this is to wire the 74HC165’s PL pin to clock E on the 6309. This isn’t immediately obvious, but is determined by careful reading of the 6309’s datasheet. The quality of the image below isn’t great, but it shows when data from the 6309 is valid.

The part we are interested in is the “Data” line, and the timing when the data is readable is the time between the Q clock going low and the E clock going low.

So, for my design, I have hooked up the PL pin to the inverse (again, going through the 74HC14 to invert the logic) of the Q clock, using pins 2A and 2Y. This way, whenever Q goes low, the 74HC165 reads whatever is on the data bus at that time, and if this happens to be at a point when we want to grab data from the 6309, it should be valid at that point in time and we then just need to shift the data out of the 74HC165 before the next Q clock when the data will be replaced.

The finished boards should look something like this:

The 74HC14 is at the top left, with the orange and white jumper wires used for the inverted connections.

Software

At this point we are done with the hardware, and can move to software. First we need to come up with a program, and for this I have chosen a very simple one

add two numbers together and return the result. The code is below.

;
; Add two 16-bit numbers (doubles) together.  The final version will accept
; arbitrary numbers on the command line, but for now we hard code two numbers
; which helps us to understand the opcodes.
;
	LDD	#$04	; Load 4 into the D register (D=4)
	ADDD	#$05	; Add 5 to D and save the result (D=D+5)
	STD	$B6B7	; Store the value of D to address 0xB6B7

By using asm6809.pl and od we can look at what the instruction opcodes need to be.

$ asm6809.pl -o 6309-adder 6309-adder.s

$ od -t x1 6309-adder
0000000    cc  00  04  c3  00  05  fd  b6  b7
0000011

From this we can determine:

0xcc is the opcode for LDD with an immediate value.
0xc3 is the opcode for ADDD with an immediate value.
0xfd is the opcode for STD to store at an extended address.

An immediate value is one that is used as-is for the data, rather than being an address containing the data. An extended address is where a full address is specified, rather than an offset. Different addressing modes result in different opcodes, as the instruction needs to do different things.

Now that we have our opcodes, we need to find out how many cycles each of them take to complete. As we are hard coding every clock cycle in this set up, it is critical that we read and write data at exactly the right time. The best reference I’ve found for the 6809/6309 is here and it gives the number of bytes and clock cycles for every instruction.

At this point we have all the information required, and it is a Simple Matter Of Programming. My script to add two arbitrary numbers using the 6309 going via the shift registers for input and output is below. The comments inline hopefully explain exactly what is happening.

var rpio = require('rpio');

/*
 * The two numbers to add together from command line arguments.
 */
var num1 = parseInt(process.argv[2], 10);
var num2 = parseInt(process.argv[3], 10);

/*
 * Set up our pin mappings and configure them.
 */
var pin = {
	'clockQ': 11,
	'clockE': 12,
	'clockS': 13,
	'write': 15,
	'read': 16,
	'reset': 18,
}

rpio.setOutput(pin.clockQ);
rpio.setOutput(pin.clockE);
rpio.setOutput(pin.clockS);
rpio.setOutput(pin.write);
rpio.setInput(pin.read);
rpio.setOutput(pin.reset);

// Start everything low.
rpio.write(pin.clockQ, 0);
rpio.write(pin.clockE, 0);
rpio.write(pin.clockS, 0);
rpio.write(pin.write, 0);
rpio.write(pin.reset, 0);

/*
 * Clocks - 'sixclock' for the 6309, 'shiftclock' for the shift registers.
 */
var sixclock = function ()
{
	rpio.write(pin.clockQ, 1);
	rpio.write(pin.clockE, 1);
	rpio.write(pin.clockQ, 0);
	rpio.write(pin.clockE, 0);
}
var shiftclock = function ()
{
	rpio.write(pin.clockS, 1);
	rpio.write(pin.clockS, 0);
}

/*
 * Shift a byte of data out to the 74HC595.
 */
var shiftout = function (data)
{
	// Convert the hex to a binary string
	var bits = String("00000000" + data.toString(2)).slice(-8);

	bits.split('').forEach(function (bit) {
		rpio.write(pin.write, bit);
		shiftclock();
	});
	shiftclock();
}

/*
 * Shift a byte of data in from the 74HC165, returning as a hex value.
 */
var shiftin = function ()
{
	var inbyte = "";
	for (var i = 0; i < 8; i++) {
		inbyte += rpio.read(pin.read);
		shiftclock();
	}
	return String("00" + parseInt(inbyte, 2).toString(16)).slice(-2);
}

/*
 * Convert a 16-bit number to an array of two 8-bit hexadecimals.
 */
var dec2hex16 = function (number)
{
	var num = String("0000" + number.toString(16)).slice(-4);

	return [parseInt(num.slice(0,2), 16), parseInt(num.slice(2,4), 16)];
}

/*
 * Initialise the 6309, starting with 10 cycles with the RESET line held down
 * to clear all state.  We then run for 3 cycles after RESET - one to load in
 * the RESET, and two to get the chip ready.
 *
 * The number of cycles required here was determined by experimentation.
 */
for (var i = 0; i < 10; i++) {
	sixclock();
}
rpio.write(pin.reset, 1);
for (var i = 0; i < 3; i++) {
	sixclock();
}

/*
 * At this point we are ready to read in the first address.  The 6309's RESET
 * vector is at $FFFE-$FFFF, which is the hardcoded location where it reads
 * the first address from.
 *
 * We provide the starting address of $1020, though it is somewhat arbitrary,
 * and we could use any address except for $FFF0-$FFFF which is reserved.
 */
shiftout(0x10); sixclock();	// Load address from $FFFE
shiftout(0x20); sixclock();	// Load address from $FFFF

/*
 * Jump to the first address (0x1020) that we input.  After that we can
 * start to input our first instruction.
 */
sixclock();

/*
 * LDD our first number.
 */
var lddbytes = dec2hex16(num1);
shiftout(0xcc); sixclock();		// LDD immediate
shiftout(lddbytes[0]); sixclock();	// Read in num1's high byte
shiftout(lddbytes[1]); sixclock();	// Read in num1's low byte

/*
 * ADDD our second number.  ADDD with an immediate is a 4 cycle
 * instruction so we have an additional clock at the end.
 */
var adddbytes = dec2hex16(num2);
shiftout(0xc3); sixclock();		// ADDD immediate
shiftout(adddbytes[0]); sixclock();	// Read in num2's high byte
shiftout(adddbytes[1]); sixclock();	// Read in num2's low byte
sixclock();				// Clock to perform ADDD operation

/*
 * STD the result to an arbitrary memory location, in this case we have
 * chosen address $C880 but it is irrelevant as we ignore it.
 *
 * STD extended is a 6 clock instruction, with the two byte result available
 * on the data bus during cycles 5 and 6.
 */
shiftout(0xfd); sixclock();		// STD extended
shiftout(0xc8); sixclock();		// Read in $C8 high byte
shiftout(0x80); sixclock();		// Read in $80 low byte
sixclock();				// Clock operation
sixclock(); byte1 = shiftin();		// Write high byte to memory
sixclock(); byte2 = shiftin();		// Write low byte to memory

/*
 * We are done, output the calculation.
 */
console.log("%d + %d = %d", num1, num2, parseInt("0x" + byte1 + byte2, 16));

/*
 * Reset the CPU to be nice, and to clear the LEDs.  Four clocks appear to be
 * enough to complete the reset.
 */
rpio.write(pin.reset, 0);
for (var i = 0; i < 4; i++) {
	sixclock();
}

We can run the program as follows.

# node 6309-adder.js 1 2
1 + 2 = 3

The output appears to be the correct answer. We now have a fully functioning co-processor attached to our Raspberry Pi ;)

Of course, there are some limitations. As we are dealing with 16-bit numbers, we can overflow:

# node 6309-adder.js 32768 32767
32768 + 32767 = 65535
# node 6309-adder.js 32768 32768
32768 + 32768 = 0

However, adding support for that and executing arbitrary instructions is left as an exercise for the reader!

Hopefully this was interesting. It has certainly been very useful for me to learn exactly what is going on at the hardware level, and I’d strongly encourage anyone involved in software development to do likewise - it gives you a new appreciation for the operations involved in running your code.

A node.js-powered 8-bit CPU - part three

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 03 Dec 2013 00:00:00 +0000

This post is part three of a series, the other posts available are

In part two we constructed a circuit using shift registers whereby we could write a byte of data and then read it back in. In this post we are going to look at a more complicated integrated circuit - a CPU - however the principles are the same.

In addition to the pieces required for the first two posts, you will need:

An additional bread board.
Enough LEDs and resistors for 16 more LEDs.
A 0.1µF “smoothing” capacitor.
An 8-bit CPU.

With that said, let’s meet my chosen CPU.

6809/6309

One of the advantages of retro computing is that we can easily (and reasonably cheaply) pick and choose anything we fancy, and so I’ve chosen the Hitachi 6309, specifically the HD63C09EP. This is Hitachi’s compatible version of the venerable 6809, which is considered by many to be the best 8-bit CPU ever made. Hitachi made it even faster and added some new instructions and registers, however it was developed right at the end of the 8-bit era, and with the introduction of 16-bit CPUs soon afterwards it didn’t have time to became as popular as its 8-bit counterparts such as the Z80, 6502 and 8008.

Thankfully, it’s still possible to buy them (second hand of course), and I managed to pick one up from littlediode.com for around £20.

To get started, let’s look at the pinout and describe what each pin does. Below is an ASCII diagram, with thanks to Jonathan Bowen’s 6809E manual (there are scanned PDFs of the original datasheets but the quality is not great).

                       _________    _________
                     _|         \__/         |_  ____
                Vss |_| 1                 40 |_| Halt  <--
                ___  _|                      |_
           -->  NMI |_| 2                 39 |_| TSC   <--
                ___  _|                      |_
           -->  IRQ |_| 3                 38 |_| LIC   -->
               ____  _|                      |_  _____
           --> FIRQ |_| 4                 37 |_| Reset <--
                     _|                      |_
           <--   BS |_| 5                 36 |_| AVMA  -->
                     _|                      |_
           <--   BA |_| 6                 35 |_| Q     <--
                     _|                      |_
                Vcc |_| 7                 34 |_| E     <--
                     _|                      |_
           <--   A0 |_| 8                 33 |_| Busy  -->
                     _|                      |_    _
           <--   A1 |_| 9                 32 |_| R/W   -->
                     _|                      |_
           <--   A2 |_| 10      6809E     31 |_| D0   <-->
                     _|                      |_
           <--   A3 |_| 11                30 |_| D1   <-->
                     _|                      |_
           <--   A4 |_| 12                29 |_| D2   <-->
                     _|                      |_
           <--   A5 |_| 13                28 |_| D3   <-->
                     _|                      |_
           <--   A6 |_| 14                27 |_| D4   <-->
                     _|                      |_
           <--   A7 |_| 15                26 |_| D5   <-->
                     _|                      |_
           <--   A8 |_| 16                25 |_| D6   <-->
                     _|                      |_
           <--   A9 |_| 17                24 |_| D7   <-->
                     _|                      |_
           <--  A10 |_| 18                23 |_| A15   -->
                     _|                      |_
           <--  A11 |_| 19                22 |_| A14   -->
                     _|                      |_
           <--  A12 |_| 20                21 |_| A13   -->
                      |______________________|

The arrows denote whether each pin is input, output, or both. Again, if a pin name has a line above it, for example Halt, it means it is “active low” and the function is activated when the pin is off/low, as opposed to being active when the pin is on/high.

The CPU is placed onto the new bread board as shown below, so that the CPU straddles the middle of the breadboard, and the bottom side has more holes available - this helps with wiring as we will need more connections on that side.

We can sort the pins into logical groups, for now only concentrating on the bare minimum required to get things running.

Power (Vss/Pin1, Vcc/Pin7)

As with the shift registers, the CPU requires 5V to operate, and so we hook them up directly to the power rails. “Vss” is another term for ground.

One recommended addition is an 0.1µF capacitor between Vss and Vcc. This acts like a small power buffer, and helps to smooth out any power fluctuations, ensuring that the CPU runs more reliably. Without this, you may notice odd behaviour, especially if you plug/unplug nearby electric devices.

To achieve maximum effectiveness, smoothing capacitors should be placed as close as possible to the input pins, as I’ve done in the picture below (again using brown/blue wires to denote power):

Clocks (E/Pin34, Q/Pin35)

Again, same deal as with the shift registers - the CPU needs a ticking clock in order to drive the internal circuitry. The 6309 has two clocks which must be driven in a specific order.

According to the datasheet “Q must lead E; that is, a transition on Q must be followed by a similar transition on E”. All that means is that repeating the following sequence is enough to drive the 6309 clock:

Set Q high
Set E high
Set Q low
Set E low

The speed at which we repeat this sequence determines the speed of the CPU. In the past with NMOS designs we would have had to keep running at a certain frequency for the chip to function correctly, but with newer CMOS designs we can run as slowly as we want, which is very helpful for seeing exactly what is happening.

By wiring E/Pin34 to GPIO17/Pin11 and Q/Pin35 to GPIO18/Pin12, we can run the CPU at 2Hz (two clock cycles per second) using a slightly extended version of our previous clock code:

var rpio = require('rpio');

rpio.setOutput(11);
rpio.setOutput(12);

/* 2Hz is slow enough to see what's happening without being overly cautious */
var speed = 2;

/*
 * Our repeating clock.  I've used the hour/quarter/half nomenclature as
 * a helpful visualisation of when each pin fires.  setInterval() and
 * setTimeout() take a millisecond argument, so we use parseInt() to ensure
 * no floating point.  This actually means we are limited to a clock speed
 * of 250Hz (one pin changing every millisecond) if we want a regular cycle.
 */
setInterval(function clock() {

	/* Set Q high immediately (on the hour) */
	rpio.write(12, 1);

	/* Set E high at quarter past the hour */
	setTimeout(function ehigh() {
		rpio.write(11, 1);
	}, parseInt(1000 / speed / 4 * 1));

	/* Set Q low at half past the hour */
	setTimeout(function qlow() {
		rpio.write(12, 0);
	}, parseInt(1000 / speed / 4 * 2));

	/* Set E low at quarter to the hour */
	setTimeout(function elow() {
		rpio.write(11, 0);
	}, parseInt(1000 / speed / 4 * 3));

}, parseInt(1000 / speed));

Data Bus (D0-7/Pin31-24)

With the 6309 being an 8-bit CPU, these are the 8 data pins on which a byte of data is read (“load”) or written (“store”) at a time. These are the only pins which are used for both input and output, and would normally be routed to a static RAM chip.

To get a CPU to do some work, we provide instructions (“opcodes”) and operands (an address or a literal value) on the data bus. Let’s start with the simplest program possible, a call to the nop (“No Operation”) instruction:

	nop	; Do nothing

We can assemble this program using Ciaran Anscomb’s asm6809.pl assembler and look at the machine code it generated using od. This tells us the corresponding opcode byte for the nop instruction:

: Assemble the source code in 6309-nop.s into an object file containing raw
: binary data.  No headers or object format data, just pure instructions.
$ asm6809.pl -o 6309-nop 6309-nop.s

: Display the contents of the binary in single-byte hexadecimal (-t x1) format.
$ od -t x1 6309-nop
0000000    12
0000001

So, to execute a nop instruction on the 6309, we write 0x12 onto the data bus.

In the final post I will show how we can use our shift register setup to handle the data bus, however for now we will simply hard-code a nop instruction on the data bus, by wiring each bit either to the +5V or ground planes.

To calculate the pin settings for writing a particular byte, I wrote a small shell function to convert from hexadecimal to binary:

hex2bin8()
{
	printf "%08d\n" $(echo "16 i 2 o $(echo $1 | tr a-z A-Z) p" | dc);
}

which we can use like so:

$ hex2bin8 12
00010010

To execute a nop then, we set D1 and D4 to high and the rest to low (D0 is the right-most bit, D7 is the left-most bit). Below is a picture of mine, where I am using the green/white strands of the CAT5 cable for the data bus connections.

Due to the gaps on the power rails, we need to divert the D2 wire up a little to reach the ground rail, however it should hopefully be clear which rail each pin is connected to.

Address Bus (A0-A15/Pin8-23)

Whilst it is an 8-bit CPU, it has a 16-bit address bus, and so is able to directly address up to 65536 bytes of memory. The current address being read from or written to is output on these pins.

Again, these pins would normally be routed to some RAM chip, and by using the address in conjunction with the data bus, we can either load or store one byte of RAM at a time, with the address containing the particular area of memory to load or store, and the data containing the byte to be read or written.

For now, the most useful thing we can do is hook each of these pins up to an LED. By doing this we can simply show what address the CPU is currently at, even if it does mean converting from binary to hex.

Here is mine, using orange wire strips to denote the address bus:

This can be a bit fiddly, due to the number of wires involved, but it should be possible to wire them all up as shown.

Note that the LEDs are back to front in terms of reading binary addresses. We could put them the other way around, but that would require a lot more cabling and it might get messy. This way we also match up the orientation of the data bus.

If you wanted to get fancy you could instead attach LED segment displays (or nixie tubes!), however I do not have any of those .. yet :)

Interrupts (NMI/Pin2, IRQ/Pin3, FIRQ/Pin4)

These pins allow you to send hardware interrupts which interrupt the CPU’s normal operation and execute specific functions. These would normally be used by something like an IO controller to indicate that, for example, some data has been read from disk and is now available.

For now we do not need any of these, so we simply disable these by hard wiring them to +5V (they are all active-low).

TSC/Pin39

This is the Tri-State Control pin, and can be used to synchronize data with other processors or controllers. As we do not have any of those yet, we simply hard wire it to GND and ensure this function is disabled.

Halt/Reset (Reset/Pin37, Halt/Pin40)

These pins are mostly self-explanatory, and are both active-low. This means that applying a negative voltage to Reset/Pin37 makes the CPU reset, and to Halt/Pin40 makes it pause indefinitely until Halt is released (set back to high).

For now we will keep things as simple as possible, and wire halt to +5V, effectively disabling it. We do however need to handle reset, as when the CPU is first powered on it needs a reset cycle to initialise correctly. This is done by holding reset down for at least one clock cycle, and so we will simply hook the reset pin to GPIO21/Pin13 and control this from software.

Alternatively, you could wire up a simple binary switch and have a hard reset button on your board.

Pin summary

So, at this stage we should have the CPU wired up as following:

Power (Vss/Pin1 and Vcc/Pin7) wired to +5V and GND.
Clock E/Pin34 wired to GPIO17 (pin 11).
Clock Q/Pin35 wired to GPIO18 (pin 12).
Data bus (D0-D7/Pin31-24) wired to +5V/GND for 0x12 (“nop”).
Address bus (A0-A15/Pin8-23) wired to 16 LEDs.
Interrupts (NMI/Pin2, IRQ/Pin3, FIRQ/Pin4) wired to GND.
TSC/Pin39 wired to GND.
Reset/Pin37 wired to GPIO21 (pin 13).
Halt/Pin40 wired to +5V.

and the board should look something like this:

Software

We can extend our clock example above and add handling of the reset pin:

var rpio = require('rpio');

rpio.setOutput(11);
rpio.setOutput(12);
rpio.setOutput(13);

// Bump this to 2Hz from the clock example.
var speed = 2;

// Start with the reset pin held low while we start up the clock.
rpio.write(13, 0);

// The clock function is the same as before.
setInterval(function clock() {

	rpio.write(12, 1);

	setTimeout(function ehigh() {
		rpio.write(11, 1);
	}, parseInt(1000 / speed / 4 * 1));

	setTimeout(function qlow() {
		rpio.write(12, 0);
	}, parseInt(1000 / speed / 4 * 2));

	setTimeout(function elow() {
		rpio.write(11, 0);
	}, parseInt(1000 / speed / 4 * 3));

}, parseInt(1000 / speed));

/*
 * Ensure we run the clock at least once, then set the reset pin back high,
 * releasing the CPU to execute normally.
 */
setTimeout(function resethigh() {
	rpio.write(13, 1);
}, parseInt(2 * 1000 / speed));

Start It Up!

We now have everything in place to get the CPU running. Below is a video of mine, running at 2Hz.

At this point we can see and explain what happens right at the start of a CPU’s execution. I’ve trimmed the first second or so from the video where the CPU is running through its pre-initialisation routine. The video starts at the point where the CPU is reading the very first instructions from memory.

The 6309 starts by reading a byte each from 0xFFFE and 0xFFFF, and this gives it the first 16-bit address to read from. You can see these first two addresses shown by the LEDs at the beginning of the video. Remember that the address LEDs are back-to-front, so the least-significant bit is on the left, not the right.

Ordinarily there would be a ROM handling this part, and it would direct the CPU to start reading an area it has set aside to store a basic operating system. Once the address has been loaded, the CPU jumps to it and starts reading instructions.

As we are hard-coding the data bus with 0x12, the first address that the CPU jumps to is 0x1212, shown by the 00010010 00010010 on the LEDs. At this “address” the CPU again reads data from the data bus ready for its first instruction, and of course gets another 0x12. This time it is executed as a nop instruction, where the CPU does nothing for an instruction cycle.

This pattern then continues indefinitely:

The CPU increments the memory location where it should fetch the current instruction from. This is known as the “program counter” or “instruction pointer”.
The CPU reads an instruction from that location, in our case always reading 0x12 which is its opcode for the nop instruction.
The nop is executed, i.e. nothing happens.

Sure, it’s not practically useful. However, we are powering this entirely from JavaScript, we can see exactly what is happening, and for me at least it is immensely helpful having a visual overview of exactly how a CPU works.

The next and final step is to make the data bus controllable, so that we can issue arbitrary instructions and read back results. If only we had some way of reading and writing 8 bits of data using only a couple of GPIO pins… ;)

A node.js-powered 8-bit CPU - part two

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 02 Dec 2013 00:00:00 +0000

This post is part two of a series, the other posts available are:

In part one we reached the stage where we were able to control an LED from node.js. We’re now going to move forward and control some simple Integrated Circuits (ICs). The principles are the same - using GPIO to turn pins on and off - but we use more pins to control something a bit more complicated.

The ICs we will look at in this post come from the 7400 series family of chips, which provide a whole variety of logic functions.

In addition to the pieces we needed for the introduction, you will need:

Enough LEDs and resistors for 8 lights.
Some wire for creating custom jumper lengths (I use CAT5 cable).
A 74HC595N and 74HC165N shift registers (the ‘N’ denotes that these are DIP and suitable for our bread board)

The shift registers can be bought very cheaply, most chips in the 7400 series cost less than a pound each.

Before we start, I prepare the breadboard like the picture below, where we configure both power rails with +5V (red jumper wire going to 5V/Pin2) and ground (black jumper wire going to GND/Pin6) to the Pi. The brown and blue wires come from a CAT5 cable, trimmed with wire cutters to fit, and are colour-coded to conform to IEC 60446.

This allows us to easily bridge across to individual pins lower down the board.

I use CAT5 cable as it is a good size for the bread board, is strong enough to insert into the board yet flexible enough to be shaped to particular angles. It’s also really helpful that you get 8 different colour combinations to choose from, so that you can choose particular colours for specific functions, e.g. power, data, address, etc. However you could just use a lot of jumper wires if you prefer, they would certainly be quicker to set up.

Shift Registers

A shift register is an integrated circuit which is designed for serial-to-parallel or parallel-to-serial communication. Let’s dive straight in and take a look at an 8-bit serial-to-parallel shift register - the trusty 74HC595.

We start by placing the chip (be careful to avoid static electricity) onto the middle of the bread board. This allows us to connect to each pin, with the gap down the middle of the bread board used to separate each side.

Once we have done that, we need to look at the datasheet which explains the function of each of the pins. The full version is here but for now the only part we are concerned with is the pinout description:

By connecting to these pins and setting them high or low, we can control the function of this chip. Let’s go through them and describe what they do.

Power (Vcc/Pin16, GND/Pin8)

These pins provide the power that the chip requires to operate. Vcc is +5V and Vss is ground, so we can connect these directly to the corresponding power rails. I use the same colours (brown and blue) as above to make the functions of these pins clear.

For the connections to the Raspberry Pi, we use the ground pin as before, and now move our red wire over to Pin 2 which is +5V power.

So, to summarise:

Pin 2 on the RPi (+5V power) to one of the pins on either “+” (red line) power rail.
Pin 6 on the RPi (Ground) to one of the pins on either “-“ (blue line) power rail.
A small piece of brown wire connecting the nearest +5V rail to the line of Pin 16 (Vcc).
A small piece of blue wire connecting the nearest ground rail to the line of Pin 8 (GND).

This provides the power that the IC needs to function, now we need to start controlling its behaviour.

Serial Input/Output (DS/Pin14, Q7S/Pin9)

DS is the pin where we load our input, one bit at a time, by setting it either high or low. Q7S is the serial output pin, and is useful if you want to tie two 74HC595s together to form one 16-bit register, by taking the output of Q7S and feeding it to the DS input of the second 74HC595.

As we are only using one IC for now we can ignore Q7S. We want to input data though, so hook the DS pin to GPIO17, that is:

Pin 11 on the RPi (GPIO17) to Pin 14 on the 74HC595 (DS).

We can leave the Q7S pin unconnected as it is used solely for output, though for your own experimentation you may want to hook it up and read the values while following the rest of this post.

Parallel Out (Q0-Q7/Pin15,1-7)

Once 8 bits of input have been loaded in, those 8 bits can then be output simultaneously on the parallel out pins. This gives us our serial-to-parallel conversion.

In order to show the output of these pins, we will hook them up to 8 LEDs. Start by laying out the LEDs on the board and insert a resistor for each, connecting between the ground rail and the cathode line, like so:

Next, cut and strip 8 lengths of wire and hook them up to their corresponding parallel output pin. In the picture below I have wired them up so that the LED furthest away is Q7 and the LED closest to the IC is Q0 (which is located on the other side, so the wire needs to go around the top of the IC).

The reasons for doing it this way around will become clear later on, but for now note that in this orientation it means that if we shift in e.g. 00001111, the LEDs will show 11110000.

Clocks (SHCP/Pin11, STCP/Pin12)

Clocks are what get any integrated circuit running, with each tick/tock of a clock pin driving the internal circuitry. All that is required to generate a clock cycle is to switch a pin on and then off again. The speed at which a pin is cycled determines the speed at which the chip functions.

The 74HC595 contains two 8-bit storage registers, and each has its own independent clock for loading data. The diagram below is helpful for having a visual overview of how the chip is connected.

Data is read in from DS, using the SHCP clock to load one bit per clock cycle. After doing this 8 times, the data we want to load is now ready in the “8-stage shift register” shown above.

To load the 8-bits from the shift register into the storage register, a single clock cycle is required on the STCP clock. As soon as that happens, the data is latched into the “8-bit storage register”.

For simplicity, we will connect the two clock pins together, using a small piece of wire running vertically and joining the two lines. By doing this we can drive both clocks simultaneously from a single GPIO pin on the Raspberry Pi, which simplifies our software.

This means that on every clock cycle the data is copied from the shift register into the storage register, but this is helpful as it shows the data being loaded in.

To control the clocks from software, we wire either SHCP or STCP to pin 12 (GPIO18) on the Raspberry Pi.

Master Reset (MR/Pin10)

MR is useful if you just want to clear (set all to 0) the shift register, by setting it low for one clock cycle, rather than shifting in 0 on the DS pin for 8 clock cycles.

You’ll note the line above MR on the diagrams above. This denotes that this pin is “active low”, which means that its function is enabled when the pin is off, whereas all the other pins we have seen so far are “active high”, and they are active or enabled when on.

We don’t need to use this pin, so for now it is simply wired high (i.e. connected to the +5V power rail) and thus disabled.

Output Enable (OE/Pin13)

OE controls whether the data in the storage register is displayed on the parallel output. Again this is an “active low” pin, and for now we simply wire this low, to always enable output.

Wiring it all up.

At this point we have something which looks like this:

To summarise the various connections:

Power is provided to the bread board power rails from pin 2 (+5V) and pin 6 (GND) on the Pi.
Power is provided to the 74HC595 to pin 16 (Vcc, +5V) and pin 8 (GND) to the appropriate power rails using small strips of brown and blue wire.
DS is serial input and is wired using the green jumper wire to pin 11 (GPIO17) on the Pi.
STCP and SHCP are the clocks, and are joined together with a small piece of orange wire. One of them is then chosen (in my picture it is STCP) and connected using a yellow wire to GPIO18 (pin 12).
The parallel output pins (Q0-Q7) are hooked up to green LEDs using lengths of green wire.
The MR “master reset” pin 10 is hard wired to the +5V power rail to disable it (active low).
The OE “output enable” pin 13 is hard wired to the ground rail to enable it (active high).

The only pin which should be left unconnected is Q7S, unless you wish to hook it up to a spare GPIO pin on the Pi and verify what it is being set to at various points.

Software

We now write a small script to write some bits into the shift register. We start by setting up the two pins we want to control, and then we define an array which contains the bits we want to send into the chip.

One part which may be confusing is that we have one extra bit at the end of the input array. The reason for this is due to the clocks being linked together. According to the datasheet “If both clocks are connected together, the shift register will always be one clock pulse ahead of the storage register.” This means we need an extra clock cycle to update the storage register.

var rpio = require('rpio');

var pinDS = 11;
var pinClk = 12;

rpio.setOutput(pinDS);
rpio.setOutput(pinClk);

// Start with the clock pin low
rpio.write(pinClk, 0);

/*
 * This is what we want to send through the shift register - start with
 * all zeros to clear the buffer, then insert 10110111, with a final bit
 * to clock in the storage register.
 */
var input = [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0];

For our clock function we will use something similar to the blinking LED example I showed in the first post, except this time on each “blink” we pop the first bit of input onto the DS pin. This loads the bit in, and we repeat until the input array is empty.

We can configure the clock speed with a variable - see the comment below.

/*
 * Configure the speed in Hz.  setInterval() and setTimeout have a maximum
 * resolution of 1ms, and we need a regular clock, so with each 'tick' and
 * 'tock' at a max of 1ms intervals our theoretical top speed is 500Hz.
 *
 * The 74HC595 is rated to 52MHz so we are in no danger of overclocking ;)
 */
var speed = 1;

setInterval(function clock() {

	/*
	 * Pop the first bit of input into DS.  If there is no more
	 * input then exit.
	 */
	if (input.length)
		rpio.write(pinDS, input.shift());
	else
		process.exit(0);

	/*
	 * Then cycle the clock at regular intervals, starting by setting it
	 * high then setting a timeout for half way until the next interval
	 * when we set it low.
	 */
        rpio.write(pinClk, 1);

        setTimeout(function clocklow() {
                rpio.write(pinClk, 0);
        }, parseInt(1000 / speed / 2));

}, parseInt(1000 / speed));

Ship It!

Running the program at 1Hz allows us to clearly see what happens, as shown in the video below. Prior to running the program and taking the video I set all the bits high, so that it is clear what happens when the first 8 zeros are shifted in.

That’s about it for the 74HC595, they are simple but incredibly useful devices.

And, as you might expect, there is a similar chip for doing parallel-to-serial conversion…

74HC165

The 74HC165 is an 8-bit parallel-to-serial IC, and pretty similar to the 74HC595. There are just a few differences we can note from the diagrams below:

Unlike the 74HC595 there is no separate storage register, the data is loaded straight into the shift register and can then be read out.

We will hook it up to our bread board and use it to read the data from the 74HC595. Here is the pinout:

And the pins we are concerned with are:

Vcc/Pin16 and GND/Pin8 are our power pins, hook up the same as before.
D0-3/Pins11-14 and D4-7/Pins3-6 are the 8 parallel inputs. We connect D0 to the same LED that Q0 from the 74HC595 is output to, D1 to Q1 etc., all the way up to D7 to Q7.
PL/Pin1 is the active low “parallel load” pin, and when active reads 8-bits of data from the parallel input lines into the shift register. Afterwards, set it high and read out the data. We want to hook this up to pin 13 (GPIO21/27) on the Pi.
Q7/Pin9 is the serial output pin where we read data from, and we hook this up to pin 15 (GPIO22) on the Pi.
CP/Pin2 is the clock for reading data out. CE/Pin15 is an active low clock enable. As we can use the PL pin to control when to load the parallel input, we do not need a separate clock for this chip and can simply link it to the 74HC595 clock with a jumper wire.

DS/Pin10 and Q7/Pin7 are serial input/output pins, and can be useful for daisy-chaining, however we have no need for them, and can leave them unconnected.

After performing all of the connections above, it should look something like this:

As for software, the program below takes a single argument of a byte of data, loads it into the 74HC595, then reads it out from our data bus using the 74HC165 before displaying the two for comparison. If everything went well, the two bytes should be identical.

var rpio = require('rpio');

/*
 * The byte we will write and hopefully read back.
 */
var outbyte = parseInt(process.argv[2], 16);

/*
 * Set up our pin mappings and configure them.
 */
var pin = {
	'output': 11,
	'clock': 12,
	'pload': 13,
	'input': 15,
}

rpio.setOutput(pin.output);
rpio.setOutput(pin.clock);
rpio.setOutput(pin.pload);
rpio.setInput(pin.input);

/*
 * The clock and parallel read pins are triggered on a transition
 * to high, so we start with them low.
 */
rpio.write(pin.clock, 0);
rpio.write(pin.pload, 0);

/*
 * Function to cycle (high then low) a specified pin.
 */
var cycle = function (p)
{
	rpio.write(p, 1);
	rpio.write(p, 0);
}

/*
 * Ensure that our output is converted to binary and is padded.
 */
outbyte = String("00000000" + outbyte.toString(2)).slice(-8);

/*
 * Shift the output onto the 'DS' pin one bit at a time followed by
 * a clock to load it into the shift register.
 */
outbyte.split('').forEach(function (bit) {
	rpio.write(pin.output, bit);
	cycle(pin.clock);
});

/*
 * If the shift and storage clocks are connected, as they are in our setup,
 * then the storage clock will always trail the shift clock by one pulse, so
 * we need to do one final cycle to syncronise it with our input.
 *
 * We can then cycle the parallel load pin to clock the data into the 74HC165.
 */
cycle(pin.clock);
cycle(pin.pload);

/*
 * Finally, read in the data from the 74HC165, one bit at a time.  As we have
 * read the data from the 74HC595 we can ignore the fact that these clock
 * cycles will now "corrupt" the 74HC595's storage with whatever the last bit
 * we set pin.output to.
 */
var inbyte = "";
for (var i = 0; i < 8; i++) {
	inbyte += rpio.read(pin.input);
	cycle(pin.clock);
}

/*
 * Convert both bytes to hexadecimal and print our final sanity check.
 */
outbyte = parseInt(outbyte, 2);
console.log("send: 0x%s", String("00" + outbyte.toString(16)).slice(-2));

inbyte = parseInt(inbyte, 2);
console.log("recv: 0x%s", String("00" + inbyte.toString(16)).slice(-2));

The output should look like this:

# node shifty.js 0xb7
send: 0xb7
recv: 0xb7

Congratulations! You have added 8 bits of storage to your computer! ;)

As you may appreciate, the 7400 series of chips can be incredibly useful, somewhat like lego where you can use them for building blocks into larger things. One engineer has even used them exclusively to build an entire computer - including creating a custom CPU!

I’m not quite that adventurous, so in the next post I will use an existing CPU and show how we can drive it in a similar way to the shift registers above.

A node.js-powered 8-bit CPU - part one

jonathan@perkin.org.uk (Jonathan Perkin) — Sun, 01 Dec 2013 00:00:00 +0000

This post is part one of a series, the other posts available are

Introduction

As a child growing up in the 1980s, I was naturally drawn towards the 8-bit computers of the day. I spent most of my early childhood on my Atari 400 and Amstrad CPC 6128, as well as being familiar with the Spectrum, Acorn Electron, and BBC Model B computers owned by my friends.

Other than the occasional bit of BASIC and CP/M, however, I was not at the time that interested in how they worked (I was too busy playing games), and so never took the chance to learn assembly and electronics at that crucial early age.

This is something I’ve always regretted, and so recently I’ve found myself more and more interested in revisiting those older systems. While computers today are far more capable and many orders of magnitude faster than those early systems, they have also become significantly more complicated. Nowadays, even if you are curious about exactly how they work, they just are not as accessible to answer those questions in the same way that computers from my childhood are.

As a result, there is today quite a large homebrew community, with people building their own 8-bit systems. One of my favourites is Matthew Sarnoff’s Ultim809 computer, and his work inspired me to have a go myself.

Realistically, with a busy job and a family, I’m never going to be able to get to the level of Matthew’s work, however I’ve had a lot of fun working on the bits I’ve done so far, and wanted to share it so hopefully others can learn too.

So, in these posts we’re going to get to the stage where we can drive an 8-bit CPU from a Raspberry Pi, using node.js (but any language will suffice). To start with, we need to introduce GPIO.

GPIO

Quoting Wikipedia:

“General-purpose input/output (GPIO) is a generic pin on an integrated circuit (commonly called a chip) whose behaviour (including whether it is an input or output pin) can be controlled (programmed) by the user at run time.”

Essentially you can think of GPIO pins as small power switches which you can turn on or off. On the Raspberry Pi they provide 3.3V, and in our first simple example we are going to control an LED from one.

Here’s what you will need to follow along:

A Raspberry Pi (or similar system with user-programmable GPIO) running Linux.
A BreadBoard.
Some LEDs, preferably green, yellow, and red.
Some jumper wire.
Some 270 Ohm (Ω) resistors

Let’s take a quick look at the bread board. These allow fast and re-usable construction of electronic circuits, and are laid out in rows - the verticals down the side are for power (positive and negative), and each horizontal row is for individual components. In the diagram below, the power lines are indicated by the red and blue boxes, and the component lines by the green boxes.

To construct the simplest possible electronic circuit, wire up the following:

An LED vertically, with the anode (positive) above and the cathode (negative) below. To determine the correct orientation the bottom side is usually flat, and/or the anode is longer.
A jumper wire going from pin 1 on the Raspberry Pi (this is the +3.3V line) to a socket on the positive line.
A jumper wire going from pin 6 on the Raspberry Pi (ground) to the blue (ground) power rail.
A 270Ω resistor connecting the ground rail to the cathode line. The resistor is required to reduce the voltage from 3.3V down to the 2.0V or so that the LED needs - without it the LED will likely burn brightly for a short time before blowing.

Doing this should give you a lit LED, and look something like this:

I’ve added annotating arrows, with the positive arrows in red and negative arrows in blue, showing the direction of current.

This is a good check that everything is working correctly, now let’s move on to make it software controlled. To do that, move the red jumper wire from pin 1 to go to pin 11 instead. This is known as GPIO17 (there are different numbering schemes for the pins depending on whether you use the physical layout or the chipset’s view - see http://elinux.org/images/2/2a/GPIOs.png for the full layout).

Software

To turn it on or off, let’s get node up and running. If you don’t already have it installed, grab the latest stable version for linux-arm-pi from http://nodejs.org/dist (latest as of writing is v0.10.21):

: You will need to be root to install and use the rpio module.
# curl -O http://nodejs.org/dist/v0.10.21/node-v0.10.21-linux-arm-pi.tar.gz
# tar zxf node-v0.10.21-linux-arm-pi.tar.gz -C /usr/local
# PATH=/usr/local/node-v0.10.21-linux-arm-pi/bin:$PATH

Next, install my rpio module. There are a number of GPIO modules available, however mine appears to be the only one which links against the bcm2835 library rather than going via the much slower /sys file system interface.

# npm install -g rpio

Finally, write this JavaScript to a file named led-on.js…

// Load the rpio module
var rpio = require('rpio');

// Configure pin 11 (GPIO17) for output (i.e. read/write).
rpio.setOutput(11);

// Turn GPIO17 on, also known as 'high'.
rpio.write(11, 1);

…and run it…

# node led-on.js

…which should result in the LED being lit. You could then create an led-off.js which is a copy of led-on.js except changing this:

// Turn GPIO17 on (1), also known as 'high'.
rpio.write(11, 1);

to this:

// Turn GPIO17 off (0), also known as 'low'.
rpio.write(11, 0);

and then we have a script which will turn the LED off.

For our final example, we can use setInterval() and setTimeout() to implement a blinking LED:

var rpio = require('rpio');

rpio.setOutput(11);

/*
 * Blink the LED quickly (10 times per second).  It is switched on every
 * 100ms, and a timeout is set for 50ms later to switch it off, giving us
 * the regular blink.
 */
setInterval(function blink() {
	rpio.write(11, 1);
	setTimeout(function ledoff() {
		rpio.write(11, 0);
	}, 50);
}, 100);

Here’s a video of my setup running this script.

This covers the introduction to GPIO and getting started with using node to control pins.

If you wanted to stay at this level and experiment further, you could use a few more of the GPIO pins to control additional LEDs, perhaps adding a yellow and a green for some traffic lights. I’ve done this with my kids and it’s a great way for them to play with electronics. My pilights repository on GitHub gives them an easy to use shell script interface, with some example programs to get started.

In the next post we move on to control something a little more complicated.

MDB support for Go

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 21 Nov 2013 00:00:00 +0000

Last week was Joyent Engineering’s inaugural hackathon, where we spent a couple of days together in San Francisco working in small teams on projects which interested us and were at least in some way relevant to our business.

Whilst the obvious choice for me would have been to work on something which used Manta for pkgsrc analysis (no shortage of ideas or potential there), I wanted to take advantage of being in the same room as my illustrious co-workers and work on something a bit more low-level and personally challenging.

For the past few months Aram Hăvărneanu has been doing some amazing work porting Go to SunOS, and we wanted to help out. So, while Bryan and Josh got stuck in with Go core and Brendan started cooking up some Go-related DTrace, I ventured into MDB land to add support for Go stack traces.

At this point you should go and read Dave’s excellent post on stack traces, as it provides a lot of background material and was super helpful for me whilst working on this. Done? Ok.

C Stack Traces

Let’s start by looking at a contrived example in C. Here is the code:

#include <stdio.h>

int printout(int num1, int num2)
{
	printf("%d -> %d\n", num1, num2);
}

int addone(int num)
{
	printout(num, num + 1);
}

int main(int argc, char *argv[])
{
	addone(atoi(argv[1]));
}

Compile it with no optimisations to ensure we do not lose any functions to inlining. The -g isn’t strictly necessary for these examples, but is good practise nonetheless.

$ gcc -O0 -g test.c -o test-c

Now we can show a stack trace using MDB. Comments inline:

$ mdb ./test-c

#
# Print a disassembly of the printout() function.  We want to set a break
# point where the stack pointer (`%rsp`) and frame pointer (`%rbp`) have been
# callee saved, so that the stack trace is accurate.  If we simply set the
# break point to the beginning of printout() the stack trace would not show
# the addone() call.
#
# I include the '-b' argument to ::dis to show real address in addition to
# the symbolic ones.
#
> printout::dis -b
401058    printout:             pushq  %rbp
401059    printout+1:           movq   %rsp,%rbp
40105c    printout+4:           subq   $0x10,%rsp
401060    printout+8:           movl   %edi,-0x4(%rbp)
401063    printout+0xb:         movl   %esi,-0x8(%rbp)
401066    printout+0xe:         movl   -0x8(%rbp),%edx
401069    printout+0x11:        movl   -0x4(%rbp),%eax
40106c    printout+0x14:        movl   %eax,%esi
40106e    printout+0x16:        movl   $0x401130,%edi
401073    printout+0x1b:        movl   $0x0,%eax
401078    printout+0x20:        call   -0x21d   <PLT:printf>
40107d    printout+0x25:        leave
40107e    printout+0x26:        ret

#
# Set a break point of the call to printf() within printout()
#
> printout+20::bp

#
# Run the program with an argument of '6'.
#
> ::run 6
mdb: stop at printout+0x20
mdb: target stopped at:
printout+0x20:  call   -0x21d   <PLT:printf>

#
# Print out a stack trace, including stack addresses ('$C').  Using '$c' would
# show just the function calls.
#
> $C
fffffd7fffdff310 printout+0x20()
fffffd7fffdff330 addone+0x1d()
fffffd7fffdff350 main+0x2e()
fffffd7fffdff360 _start+0x6c()

#
# Print out the status of all registers.
#
> ::regs
%rax = 0x0000000000000000       %r8  = 0xfffffd7fffdff596
%rbx = 0xfffffd7fff3fafd8       %r9  = 0xfffffd7fff331ee3
%rcx = 0x0000000000000000       %r10 = 0x0000000000000000
%rdx = 0x0000000000000007       %r11 = 0x00000000000000c0
%rsi = 0x0000000000000006       %r12 = 0x0000000000000000
%rdi = 0x0000000000401130       %r13 = 0x0000000000000000
                                %r14 = 0x0000000000000000
                                %r15 = 0x0000000000000000

%cs = 0x0053    %fs = 0x0000    %gs = 0x0000
%ds = 0x0000    %es = 0x0000    %ss = 0x004b

%rip = 0x0000000000401078 printout+0x20
%rbp = 0xfffffd7fffdff310
%rsp = 0xfffffd7fffdff300

%rflags = 0x00000286
  id=0 vip=0 vif=0 ac=0 vm=0 rf=0 nt=0 iopl=0x0
  status=<of,df,IF,tf,SF,zf,af,PF,cf>

%gsbase = 0x0000000000000000
%fsbase = 0xfffffd7fff172a40
%trapno = 0x3
   %err = 0x0

#
# Print out the contents of the stack in memory.
#
> 0xfffffd7fffdff300,70::dump
                   \/ 1 2 3  4 5 6 7  8 9 a b  c d e f  v123456789abcdef
fffffd7fffdff300:  00000000 00000000 07000000 06000000  ................
fffffd7fffdff310:  30f3dfff 7ffdffff 9c104000 00000000  0.........@.....
fffffd7fffdff320:  50f3dfff 7ffdffff 280936ff 06000000  P.......(.6.....
fffffd7fffdff330:  50f3dfff 7ffdffff cc104000 00000000  P.........@.....
fffffd7fffdff340:  78f3dfff 7ffdffff 78f3dfff 02000000  x.......x.......
fffffd7fffdff350:  60f3dfff 7ffdffff ec0e4000 00000000  `.........@.....
fffffd7fffdff360:  00000000 00000000 00000000 00000000  ................

#
# Print the contents of %rdi.
#
> 401130::dump
         \/ 1 2 3  4 5 6 7  8 9 a b  c d e f  v123456789abcdef
401130:  2564202d 3e202564 0a000000 00000000  %d -> %d........

#
# Print the instruction pointers saved on the stack, with one instruction
# either side for context.
#
> 40109c::dis -b -n 1
401097    addone+0x18:                          call   -0x44    <printout>
40109c    addone+0x1d:                          leave
40109d    addone+0x1e:                          ret
> 4010cc::dis -b -n 1
4010c7    main+0x29:                            call   -0x4d    <addone>
4010cc    main+0x2e:                            leave
4010cd    main+0x2f:                            ret

From the output we can match up a few things:

The instruction pointer %rip is set to our break point, and you can see both the real (0x401078) and the symbolic (printout+0x20) addresses.
The arguments to printf() are passed in the registers, with %rdx and %rsi containing the integers we are printing, and %rdi containing a pointer to the string “%d -> %d” which we can see using ::dump.
The stack contains arguments (7, 6), instruction pointers (40109c, 4010cc) and frame pointers (fffffd7fffdff330, fffffd7fffdff330).

If you are wondering where I am getting those addresses from, note that this is on x86 which is little-endian, and so you need to read each byte backwards - that is, if you have a dump line which contains a 64-bit value, a 32-bit value, a 16-bit value and two 8-bit values, then:

xxxxxxxxxxxxxxxx:  78f3dfff 7ffdffff 78f3dfff 0207410d  ................

corresponds to:

0xfffffd7fffdff378 (64-bit)
0xffdff378 (32-bit)
0x0702 (16-bit)
0x41 (8-bit)
0x0d (8-bit)

Playing around with MDB like this was super helpful for me to get a visual representation of memory, and how functions and arguments are passed around.

Thanks to the frame pointers being saved on the stack, we are able to easily get useful stack traces by simply following them and printing each frame in turn.

At this point it’s worth mentioning the infamous GCC argument -fomit-frame-pointer. If you are of a certain age, you may remember some Linux distributions using this flag in an attempt to make programs faster. What happens is that instead of using the frame pointer to record valuable information about where we came from, it is instead used as a general purpose register, and the stack information is lost.

This can be verified pretty easily:

$ gcc -O0 -g -fomit-frame-pointer test.c -o test-nofp

$ mdb ./test-nofp

> printout+0x20::bp

> ::run 6
mdb: stop at printout+0x20
mdb: target stopped at:
printout+0x20:  call   -0x225   <PLT:printf>

> $C
fffffd7fffdff350 printout+0x20()

> ::regs ! grep 'r[ibs]p'
%rip = 0x0000000000401068 printout+0x20
%rbp = 0xfffffd7fffdff350
%rsp = 0xfffffd7fffdff2f0

That’s all we have. %rbp is no longer used to record the previous %rsp, and so after printing the current instruction, we have nowhere else to go, and the stack is useless.

Friends don’t let friends go without their frame pointers. There is very little to suggest that the extra register provides any performance benefits, and the cost is way too high. Just Say No.

Go Stack Traces

So, let’s look at Go. One of the nice things about Go is that it compiles programs down to a self-contained binary, which can then be copied around and executed. This may lead you to think that stack traces will simply work, as there is no dynamic runtime stuff going on. Well, let’s see.

Let’s start with a comparative program to the one above, written in Go.

package main

import (
	"fmt"
	"os"
	"strconv"
)

func printout(num1 int, num2 int) {
	fmt.Printf("%d -> %d\n", num1, num2)
}

func addone(num int) {
	printout(num, num + 1)
}

func main() {
	num, _ := strconv.Atoi(os.Args[1])
	addone(num)
}

Compile it and start MDB.

$ go build -o test-go test.go
$ mdb ./test-go

The first thing we notice is that we do have symbols, which is good! We can also show the first few instructions of our printout() function:

> ::nm ! egrep 'addone|printout'
0x0000000000400c00|0x00000000000000cb|FUNC |GLOB |0x0  |1       |main.printout
0x0000000000400cd0|0x0000000000000034|FUNC |GLOB |0x0  |1       |main.addone

> main.printout::dis -b -n 8
400c00    main.printout:                        movq   %fs:-0x10,%rcx
400c09    main.printout+9:                      cmpq   (%rcx),%rsp
400c0c    main.printout+0xc:                    ja     +0x7     <main.printout+0x15>
400c0e    main.printout+0xe:                    call   +0x2078d <runtime.morestack16>
400c13    main.printout+0x13:                   jmp    -0x15    <main.printout>
400c15    main.printout+0x15:                   subq   $0x80,%rsp
400c1c    main.printout+0x1c:                   leaq   0x60(%rsp),%rdi
400c21    main.printout+0x21:                   xorq   %rax,%rax
400c24    main.printout+0x24:                   movq   $0x4,%rcx

However, note that the frame pointer is never saved - the function just goes right ahead and tries to allocate more stack. And, indeed, if we set a breakpoint and run the program, we do not get a useful stack trace.

> main.printout::bp

> ::run 6
mdb: stop at main.printout
mdb: target stopped at:
main.printout:  movq   %fs:-0x10,%rcx

> $C
8000000000000000 main.printout()

The problem is that Go uses a completely different calling convention, so the usual method of following frame pointers does not work. At this point we would normally be stuck, however Go does provide enough information in the binary for us to calculate the stack in a different way. We just need to dig it out.

First, let’s take a look to see what the stack does contain. I deliberately set a breakpoint to the start of the printout() function so that the stack did not contain any local variables for that function.

> ::regs ! grep rsp
%rsp = 0xfffffd7ffef8ff00

> 0xfffffd7ffef8ff00,b0::dump
                   \/ 1 2 3  4 5 6 7  8 9 a b  c d e f  v123456789abcdef
fffffd7ffef8ff00:  ff0c4000 00000000 06000000 00000000  ..@.............
fffffd7ffef8ff10:  07000000 00000000 610d4000 00000000  ........a.@.....
fffffd7ffef8ff20:  06000000 00000000 01000000 00000000  ................
fffffd7ffef8ff30:  06000000 00000000 00000000 00000000  ................
fffffd7ffef8ff40:  00000000 00000000 ee364100 00000000  .........6A.....
fffffd7ffef8ff50:  c0764100 00000000 00000000 00000000  .vA.............
fffffd7ffef8ff60:  00000000 00000000 00000000 01000000  ................
fffffd7ffef8ff70:  ffffffff ffffffff 00000000 00000000  ................
fffffd7ffef8ff80:  084e5600 00000000 00000000 00000000  .NV.............
fffffd7ffef8ff90:  00000000 00000000 f05b4100 00000000  .........[A.....
fffffd7ffef8ffa0:  00000000 00000000 00000000 00000000  ................

A few things stood out here. Knowing the function arguments in advance allows us to spot the various 7, 6 and 1 integers being passed, so we can take a guess that the other addresses refer to functions, and we can confirm this with MDB:

> 400cff::dis -b -n 1
400cfa    main.addone+0x2a:                     call   -0xff    <main.printout>
400cff    main.addone+0x2f:                     addq   $0x10,%rsp
400d03    main.addone+0x33:                     ret

> 400d61::dis -b -n 1
400d5c    main.main+0x4c:                       call   -0x91    <main.addone>
400d61    main.main+0x51:                       addq   $0x28,%rsp
400d65    main.main+0x55:                       ret

> 4136ee::dis -b -n 1
4136e9    runtime.main+0xe9:                    call   -0x129de <main.main>
4136ee    runtime.main+0xee:                    cmpl   $0x0,0x573868
4136f6    runtime.main+0xf6:                    je     +0x20    <runtime.main+0x118>

So, we can dig through the stack and find functions and arguments. However, because functions can have differing argument lengths, we can’t step over a fixed size - we need to know how many arguments a function takes, so that we know how much of the stack to skip in order to get to the next function.

Thankfully, Go provides us with the answer, in the form of the pclntab.

The pclntab is written into the binary by the Go linker, and contains useful information about each function. The format is described in this document for Go 1.2, which is the version we are targetting.

Here is the beginning of the pclntab:

> pclntab,50::dump
         \/ 1 2 3  4 5 6 7  8 9 a b  c d e f  v123456789abcdef
524840:  fbffffff 00000108 4d060000 00000000  ........M.......
524850:  000c4000 00000000 f0640000 00000000  ..@......d......
524860:  d00c4000 00000000 58650000 00000000  ..@.....Xe......
524870:  100d4000 00000000 b0650000 00000000  ..@......e......
524880:  700d4000 00000000 10660000 00000000  p.@......f......

If you read the document above you can see and compare the pclntab header format - we start with the magic number 0xfffffffb, followed by 0x0000. Then 0x01 for the instruction size quantum and 0x08 for the size of a pointer (both accurate for 64-bit x86). After that we have 0x064d as the 64-bit size of the function symbol table.

After the header we have pairs of function addresses to function offsets within the pclntab. So, for example, we start with 0x400c00, for which the information is stored at offset 0x64f0 within the pclntab. Decoding the data at this offset gives us the details we need.

At this point we need to dig through the Go source code to find how this information is stored, and thankfully we can mostly lift it directly into our MDB module.

MDB Module

The current version of the MDB module is here, you may want to follow along there.

We can start by transplanting the layouts for the pclntab and Go function information.

/*
 * The pclntab header.
 */
struct pctabhdr {
	uint32_t magic;		/* 0xfffffffb */
	uint16_t zeros;		/* 0x0000 */
	uint8_t quantum;	/* 1 on x86, 4 on ARM */
	uint8_t ptrsize;	/* sizeof(uintptr_t) */
	uintptr_t tabsize;	/* size of function symbol table */
};

/*
 * The function -> offset entries.
 */
typedef struct go_func_table {
	uintptr_t entry;
	uintptr_t offset;
} go_functbl_t;

/*
 * The information about each function which is stored in the pclntab.
 */
typedef struct go_func {
	uintptr_t entry;
	uint32_t nameoff;
	uint32_t args;
	uint32_t frame;
	uint32_t pcsp;
	uint32_t pcfile;
	uint32_t pcln;
	uint32_t npcdata;
	uint32_t nfuncdata;
} go_func_t;

Once we have that, we can do the following:

Start by loading the pclntab and checking it is valid in configure(). If it is good, we store its location and size.
Get the previous instruction from the top of the stack. In our example this is 0x400cff, and is done using load_current_context() which reads the current value of %rsp.
Perform a binary search through the go_functbl_t entries for that address. If our address is larger than the current lookup but less than the next one, we have a match. This is implemented in findfunc().
Read in the function information into our go_func_t using pcvalue() and the information stored at our function offset.

This is implemented with the ::goframe dcmd, which we can use after loading the new go.so module.

> ::load /root/mdb_go/go.so
Configured Go support

> ::goframe
fffffd7fffdfdf30 = {
	entry = 400cd0,
	nameoff = 6590 (main.addone),
	args = 8 (len=1),
	frame = 18,
	pcsp = 659d (delta=16),
	pcfile = 65a4 (/root/test.go),
	pcln = 65a7 (15),
	npcdata = 0,
	nfuncdata = 2,
}

Note that we have some additional useful information here, including the filename and the line number. I implemented a -p name argument which pretty prints this information:

> ::goframe -p name
main.addone(0x6)
	(in /root/test.go line 15)

The final part of the puzzle is to dig out the argument length and skip over that size, so that we can get to the next function. This is pretty straight-forward, we just skip over the size of the frame entry.

With this in place we can implement an MDB “walker”, which walks our stack and prints each address. MDB makes this really easy, and is implemented in walk_goframes_init and walk_goframes_step, which simply describe how to find the first frame and then how to find each subsequent frame.

> ::walk goframe
fffffd7ffef8ff00
fffffd7ffef8ff18
fffffd7ffef8ff48
fffffd7ffef8ff98
fffffd7ffef8ffa8

And, for the finale, we show the beautiful modularity of MDB by simply piping those addresses to the ::goframe dcmd:

> ::walk goframe | ::goframe -p name
main.addone(0x6)
	(in /root/test.go line 15)
main.main()
	(in /root/test.go line 20)
runtime.main()
	(in /root/go-sunos/src/pkg/runtime/proc.c line 199)
runtime.goexit()
	(in /root/go-sunos/src/pkg/runtime/proc.c line 1395)

And hey presto, we have a stack trace.

Future Work

So, as you may have noticed, I cheated a little by setting a break point to the start of the printout() function. This allows us to print a full stack trace, including the current instruction, using the ::gostack dcmd:

> ::gostack -p name
main.printout(0x6, 0x7)
	(in /root/test.go line 9)
main.addone(0x6)
	(in /root/test.go line 15)
main.main()
	(in /root/test.go line 20)
runtime.main()
	(in /root/go-sunos/src/pkg/runtime/proc.c line 199)
runtime.goexit()
	(in /root/go-sunos/src/pkg/runtime/proc.c line 1395)

However, if we set the break point to the point where we call fmt.Printf():

> main.printout+0xb5::bp

> ::run 6
mdb: stop at main.printout+0xb5
mdb: target stopped at:
main.printout+0xb5:     call   +0x24a26 <fmt.Printf>

> ::gostack -p name
main.printout(0x9, 0xfef8fee0)
	(in /root/test.go line 10)

We are unable to walk the stack, and the arguments to main.printout() are incorrect, due to main.printout() allocating a bunch of the stack for its local variables which messes up our offsets.

I need to figure out how to calculate these reliably, plus there are a bunch of cleanups to do in the module. It’s also likely that there are lots of edge cases where things will break.

It’s also likely that the format of the symbol tables will change, so we will need to track changes upstream.

Finally, the most useful work we can do based on this initial implementation would be to translate the same information into a DTrace ustack helper. This would allow us to dynamically instrument running Go programs, and do all sorts of useful performance and debugging analysis. The information is all there, and we can in theory get to it using uregs[], but there are a number of challenges to overcome first, most notably trying to hook into the Plan9 linker which currently rejects DTrace ELF sections. Help in this area would be appreciated :)

Hopefully this was a useful introduction to MDB and how to implement support for esoteric languages. I welcome any feedback and improvements.

What's new in pkgsrc-2013Q2

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 30 Jul 2013 00:00:00 +0100

Note

This page is obsolete. Please see http://pkgsrc.joyent.com/ where you will find the latest package sets and improved instructions.

Original Article

The latest branch of pkgsrc was released at the beginning of July, and binary packages for SmartOS/illumos and OSX are now available.

On OSX there are almost 9,000 binary packages available, whilst on illumos we finally breached the 10,000 package mark! Congratulations to everyone who has worked on SunOS pkgsrc support over the past 15 years, this is a great milestone.

Installing

Please see the individual instruction pages for your platform:

OSX
illumos

What’s New

As usual there were many hundreds of changes which went into this quarterly release of pkgsrc. Here are some of the more interesting and useful changes.

OpenJDK7 is now default

Thanks to the great work by SmartOS user ‘jesse’, we now have a working OpenJDK7 on illumos, built with GCC. This is now the default JRE/JDK, as we are unable to provide updated sun-{jre,jdk} packages due to Oracle’s more restrictive redistribution policies.

The only user-visible change from this is that the Java binaries are prefixed with openjdk7-, so call e.g. openjdk7-java instead of java, or alternatively put /opt/local/java/openjdk7/bin at the front of your $PATH.

This allows co-existance with the legacy sun-{jre,jdk} packages.

Desktop support

Thanks to many Xorg updates from Richard Palo, Xorg is now functional on illumos, enabling many common desktop environments to now be used.

: Install the meta-package containing Xorg
$ pkgin in modular-xorg

: On OmniOS these are required on top of the basic install.
$ pkg install driver/x11/xsvc developer/macro/cpp

: Also on OmniOS 'od' is located in a different location
$ sed -i -e 's,/usr/bin/od,/usr/gnu/bin/od,' /opt/local/bin/startx

Here are some examples and how to install them:

GNOME 2.32 with Evolution and Firefox 22.

Screenshot:

Install:

$ pkgin in gnome-session gnome-themes gnome-themes-extras \
           gnome-terminal gnome-backgrounds evolution

$ vi .xinitrc
#!/bin/sh
PATH=/opt/local/sbin:/opt/local/bin:$PATH
/opt/local/bin/gnome-session

$ startx

: Currently the pkgsrc firefox22 fails on startup, so for now use the
: pre-built binaries from Mozilla (with some library hacks).
$ curl -s http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/contrib/solaris_tarball/firefox-22.0.en-US.opensolaris-i386.tar.bz2 \
    | bzcat | tar -xf -
$ ln -s /opt/local/lib/libX11.so firefox/libX11.so.4
$ ln -s /opt/local/lib/libXt.so firefox/libXt.so.4
$ env LD_LIBRARY_PATH=/opt/local/lib ./firefox/firefox

KDE 4.10.3

Screenshot:

Install:

$ pkgin in kde-runtime4 kde-workspace4 kde-baseapps4 \
           kde-wallpapers4 kde-base-artwork konsole

$ vi .xinitrc
/opt/local/bin/startkde

XFCE 4.6 with Gnumeric and Abiword

Screenshot:

Install:

$ pkgin in xfce4 gnumeric abiword

$ vi .xinitrc
/opt/local/bin/xfce4-session

Enlightenment 0.17 with GIMP

Screenshot:

Install:

$ pkgin in enlightenment-0.17 gimp

$ vi .xinitrc
/opt/local/bin/enlightenment_start

Awesome 3.4.13

And finally, for you terminal fans ;)

Install:

$ pkgin in awesome

$ vi .xinitrc
/opt/local/bin/awesome

Finally

Enjoy!

Distributed chrooted pkgsrc bulk builds

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 24 Jul 2013 00:00:00 +0100

Once you are up and running with pkgsrc, one of the most common requests is a way to automatically build a number of packages, either a specific list plus their dependencies, or everything currently available.

There are a number of ways to accomplish this, but for this tutorial I will concentrate on pbulk, as it is used by a number of pkgsrc developers, and has support for distributed and chrooted builds.

In this example I am building a set of pkgsrc-2013Q2 packages, and I have tested it on:

Linux
OSX
SmartOS

Please let me know if it doesn’t work correctly on your platform.

Layout

First, have a think about where you will store pkgsrc, source tarballs, packages, etc. I put everything under /content which makes it easy to then mount just that directory and have everything below it available:

/content/bulklog is where pbulk saves the per-package build logs
/content/distfiles is where source tarballs are kept
/content/mk contains some make fragment files for configuration
/content/packages is the top-level directory of binary packages
/content/pkgsrc is where pkgsrc is located
/content/scripts to hold some miscellaneous scripts

Pre-create some required directories:

$ mkdir -p /content/{distfiles,mk,packages/bootstrap,scripts}

Then write a couple of mk.conf files which will be used by the packaging tools.

/content/mk/mk-generic.conf:

ALLOW_VULNERABLE_PACKAGES=	yes
SKIP_LICENSE_CHECK=		yes
DISTDIR=			/content/distfiles

# If your system has a native curl, this avoids building nbftp
FAILOVER_FETCH=		yes
FETCH_USING=		curl

# Change this to a closer mirror
MASTER_SITE_OVERRIDE=	ftp://ftp2.fr.NetBSD.org/pub/NetBSD/packages/distfiles/

# Tweak this for your system, though take into account how many concurrent
# chroots you may want to run too.
MAKE_JOBS=		4

/content/mk/mk-pbulk.conf:

.include "/content/mk/mk-generic.conf"

PACKAGES=	/content/packages/2013Q2/pbulk
WRKOBJDIR=	/var/tmp/pkgbuild

/content/mk/mk-pkg.conf:

.include "/content/mk/mk-generic.conf"

PACKAGES=	/content/packages/2013Q2/x86_64
WRKOBJDIR=	/home/pbulk/build

Get pkgsrc

$ cd /content

Either use git..

$ git clone -b pkgsrc_2013Q2 https://github.com/joyent/pkgsrc.git

..or CVS

$ cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2013Q2 -P pkgsrc

At the present time (2013Q2) there are a couple of patches you need to apply, one for mksandbox to support some additional features, and one for pbulk to support chroots and a couple of other bits we’ve developed at Joyent.

$ cd pkgsrc
$ curl -s http://www.netbsd.org/~jperkin/mksandbox-1.3.diff | patch -p0
$ curl -s http://www.netbsd.org/~jperkin/pbulk-joyent.diff | patch -p0

Build pbulk

pbulk needs to be installed to its own prefix, from where it will manage the main build.

$ cd bootstrap
$ ./bootstrap --abi=64 --prefix=/usr/pbulk --mk-fragment=/content/mk/mk-pbulk.conf
$ ./cleanup; cd ..

Then build the necessary pacakges

$ PATH=/usr/pbulk/sbin:/usr/pbulk/bin:$PATH
$ cd pkgtools/pbulk
$ bmake package-install
$ cd ../mksandbox
$ bmake package-install

It is recommended that builds are done as an unprivileged user, which is normally named pbulk, so now would be a good time to create that user, usually with something like this:

$ groupadd -g 500 pbulk
$ useradd -u 500 -g 500 -c 'pbulk user' -s /bin/bash -m pbulk

Check the useradd/groupadd syntax for your system. The user can be set to no-password, it will only be used via su from the root user.

Set up chroot

Next, check that the mksandbox script works on your system. It is designed to be cross-platform, but on certain systems (e.g. OSX) there is no native support for loopback mounts, and so you will first need to configure NFS in order to share system directories to the chroot, usually with / -alldirs -maproot=root in /etc/exports then nfsd enable.

$ mkdir /chroot

: This command should create the chroot under /chroot/test
$ mksandbox --rodirs=/usr/pbulk --rwdirs=/content --without-pkgsrc /chroot/test

: This should execute a shell inside the chroot.  Check that directories are
: mounted as expected, and that you can't e.g. write to a read-only file system.
$ /chroot/test/sandbox 

: This should unmount the chroot mounts
$ /chroot/test/sandbox umount

: Test that there are no left-over mounts before removing, else you may delete
: files on a read-write mount!  This should return no results.
$ mount -v | grep /chroot/test/

$ rm -rf /chroot/test

Once you are happy the chroot is working as expected, write a couple of wrapper scripts to create and delete them with an optional argument with the name of the chroot, which will be used by pbulk. Below are the scripts I use.

/content/scripts/mksandbox:

#!/bin/sh

chrootdir=$1; shift

while true
do
	# XXX: limited_list builds can recreate chroots too fast.
	if [ -d ${chrootdir} ]; then
		echo "Chroot ${chrootdir} exists, retrying in 10 seconds or ^C to quit"
		sleep 10
	else
		break
	fi
done

/usr/pbulk/sbin/mksandbox --without-pkgsrc \
    --rodirs=/usr/pbulk --rwdirs=/content ${chrootdir} >/dev/null 2>&1
mkdir -p ${chrootdir}/home/pbulk
chown pbulk:pbulk ${chrootdir}/home/pbulk

/content/scripts/rmsandbox:

#!/bin/sh

chrootdir=`echo $1 | sed -e 's,/$,,'`; shift

if [ -d ${chrootdir} ]; then
	#
	# Try a few times to unmount the sandbox, just in case there are any
	# lingering processes holding mounts open.
	#
	for retry in 1 2 3
	do
		${chrootdir}/sandbox umount >/dev/null 2>&1
		mounts=`mount -v | grep "${chrootdir}/"`
		if [ -z "${mounts}" ]; then
			rm -rf ${chrootdir}
			break
		else
			sleep 5
		fi
	done
fi

Build pkg bootstrap

Next step is to build the bootstrap for the target packages, i.e. the main prefix you will be using. Again we use the bootstrap script, but here you may want to tweak the settings - check the pkgsrc guide or the --help output for more information.

If the prefix you want to build for (i.e. /usr/pkg) is already in use on the system, simply do the bootstrap inside a chroot.

I use something like this:

$ /content/scripts/mksandbox /chroot/build-bootstrap
$ /chroot/build-bootstrap/sandbox
$ cd /content/pkgsrc/bootstrap

: Use the defaults of /usr/pkg.  --gzip-binary-kit is important, it is the
: tarball that pbulk will use for builds.
$ ./bootstrap \
    --abi=64 \
    --gzip-binary-kit=/content/packages/bootstrap/bootstrap-2013Q2-pbulk.tar.gz \
    --mk-fragment=/content/mk/mk-pkg.conf
$ ./cleanup
$ exit

$ /content/scripts/rmsandbox /chroot/build-bootstrap

Configure pbulk

Now we’re finally ready to configure pbulk. There is a single configuration file you need to edit, and I will show the changes I have made to it.

$ diff /usr/pbulk/share/examples/pbulk/pbulk.conf /usr/pbulk/etc/pbulk.conf

This section adds a ulimit to stop runaway processes from hanging the build.

2a3,6
> # Limit processes to an hour of CPU time.  Anything which takes longer than
> # this is most probably broken.
> ulimit -t 3600
>

This section configures the location of the bulk build report. I upload my results to Joyent’s Manta object store as it allows arbitrary storage plus distributed Unix queries on the data at a later time.

11c15
< base_url=http://www.pkgsrc-box.org/reports/current/DragonFly-1.8
---
> base_url=http://us-east.manta.joyent.com/pkgsrc/public/reports/Darwin/2013Q2/x86_64

Turn on reuse_scan_results, it makes subsequent runs faster.

14c18
< reuse_scan_results=no
---
> reuse_scan_results=yes

In this example I am using a single host which will perform concurrent builds inside chroots, and so I need to unset scan_clients and build_clients and set master_ip to localhost.

If you have multiple hosts, simple set master_ip to a public address, and add the list of slave IP addresses to *_clients. They will need to be accessible via SSH as root from the master, and will need to have their own installs of /usr/pbulk as well as sharing the same /content mount as the master, most likely over NFS.

If you wish to completely disable any concurrency or distributed builds, set master_mode=no, though note that the build with then run completely single-threaded and will be much slower.

22,24c26,28
< master_ip=192.168.75.10
< scan_clients="192.168.75.21 192.168.75.22 192.168.75.23 192.168.75.24"
< build_clients="192.168.75.21 192.168.75.22 192.168.75.23 192.168.75.24"
---
> master_ip=127.0.0.1
> scan_clients=""
> build_clients=""

If you wish to publish to Manta, here are the settings you will need. I have installed a local copy of the Manta tools to /content/manta, as the upload script will need them.

28a33,39
> # Manta upload settings
> MANTA_USER="pkgsrc"
> MANTA_KEY_ID="40:b7:2e:b5:de:04:17:78:35:0b:d8:72:b9:da:8d:0e"
> MANTA_URL="https://us-east.manta.joyent.com"
> MANTA_PATH="/usr/pbulk/bin:/content/manta/node_modules/.bin"
> report_manta_target="/pkgsrc/public/reports/Darwin/2013Q2/x86_64"
>

Configure the location where to rsync packages to and where to send the report. If you are not using Manta, then you will want to set report_rsync_target to an appropriate location.

33c44
< pkg_rsync_target="pkgsrc@192.168.75.1:/public/packages/current/DragonFly-1.8"
---
> pkg_rsync_target="pkgsrc.joyent.com:/packages/Darwin/2013Q2/x86_64"
36,37c47,48
< report_subject_prefix="pkgsrc"
< report_recipients="pkgsrc-bulk@netbsd.org"
---
> report_subject_prefix="pkgsrc-2013Q2"
> report_recipients="jperkin@joyent.com"

Where to find the /usr/pkg bootstrap tarball:

41c52
< bootstrapkit=/usr/pkgsrc/bootstrap/bootstrap.tar.gz
---
> bootstrapkit=/content/packages/bootstrap/bootstrap-2013Q2-pbulk.tar.gz

Configure build chroots. Here we set the paths to the mksandbox and rmsandbox scripts we created earlier, and provide a basename of the chroot directory. By setting chroot_dir=/chroot/pkgsrc-2013Q2, pbulk will actually create /chroot/pkgsrc-2013Q1-build-{1,2,3,4} and /chroot/pkgsrc-2013Q1-scan-{1,2,3,4}.

You will want to experiment with the tradoffs between MAKE_JOBS and the number of chroots. Generally it will be better to have more chroots compared to an increase in MAKE_JOBS, as certain parts of the build will be single threaded anyway (e.g. large configure scripts). However, you also need to be aware of the increased disk I/O caused by too many chroots.

As long as you have everything correctly shared, there is nothing stopping you using distributed hosts and chroots, and it is highly recommended if you can as clearly it provides the best performance. With such a setup, at Joyent we are able to do full bulk builds of all 12,000 packages in pkgsrc in under 12 hours.

46a58,64
> # Chroot scripts.
> chroot_create=/content/scripts/mksandbox
> chroot_delete=/content/scripts/rmsandbox
> chroot_dir=/chroot/pkgsrc-2013Q2
> build_chroots=4
> scan_chroots=4
>

Finally, configure paths to the ones we have chosen.

74,75c92,93
< bulklog=/bulklog
< packages=/packages
---
> bulklog=/content/bulklog
> packages=/content/packages/2013Q2/x86_64
77c95
< pkgsrc=/usr/pkgsrc
---
> pkgsrc=/content/pkgsrc

One option not mentioned above is limited_list. If you only want to build a subset of packages rather than run a full bulk build, simply set limited_list to a file containing paths to packages you want. It is worth doing this initially anyway, just to check that everything is working fine, e.g.:

$ cat >/content/mk/pkglist <<EOF
sysutils/coreutils
EOF

45c45
< #limited_list=/limited_list
---
> limited_list=/content/mk/pkglist

Run the bulk build

Assuming everything was done correctly, it should now just be a matter of running the bulkbuild. If you have set the chroot_* variables then this will run chrooted at the appropriate places, so that your host system’s /usr/pkg is not affected.

$ bulkbuild

One of the benefits of the Joyent patch is that it adds support for different configuration files, so if you really want to you can run concurrent instances of pbulk. Just write separate pbulk.conf files and then pass them as arguments to bulkbuild. Again, we use this to run multiple builds across the same hosts, all thanks to the chroot support.

$ bulkbuild pbulk-32bit.conf
$ bulkbuild pbulk-64bit.conf

Caveats

There are some known issues, I will document them here as they are found.

OSX chroot DNS resolution

On OSX, name resolution is broken inside a chroot. This is due to mDNSResponder being used for DNS lookups, which relies on the /var/run/mDNSResponder UNIX socket. Unfortunately, making that socket available in the chroot (either by mounting or creating a proxy with socat) does not fix the issue, so I would welcome input on this.

For now you need to set MASTER_SITE_OVERRIDE and then ensure that the IP address for that mirror is set in /etc/hosts.

Chroot creation race conditions

As you can see in my example mksandbox script, I have to work around a race condition where a previous scan run may still be cleaning up whilst a new one is starting. For now I am simply sleeping until the chroot is free, but this should be fixed properly, probably with process groups and waiting for them to complete.

pkgsrc on SmartOS - creating new packages

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 07 Jun 2013 00:00:00 +0100

Having set up a build zone and started fixing packages, the next area we will look at is creating new packages. For this tutorial I will use augeas as an example, as it happens to be a package I recently created.

First, install the pkgtools/url2pkg package, as this massively simplifies the task in hand.

: If you have pkgin and binary packages available..
$ sudo pkgin in url2pkg

: ..else build from source
$ (cd pkgtools/url2pkg; bmake install)

Then create a new package directory, choosing the most appropriate category (in this case devel), then run url2pkg giving it an argument of the source tarball.

$ mkdir devel/augeas
$ cd devel/augeas
$ url2pkg http://download.augeas.net/augeas-1.0.0.tar.gz

First it will open an editor session on Makefile. You should customise a few variables:

Set MAINTAINER either to your email address, or to pkgsrc-users@NetBSD.org if you do not want to be the primary maintainer of the package.
Write a brief one-line COMMENT describing what the package is.
Set LICENSE to a list of the licenses the package is made available under (see mk/license.mk for an available list).

Here is what I wrote for augeas:

# $NetBSD$

DISTNAME=	augeas-1.0.0
CATEGORIES=	devel
MASTER_SITES=	http://download.augeas.net/

MAINTAINER=	pkgsrc-users@NetBSD.org
HOMEPAGE=	http://download.augeas.net/
COMMENT=	Configuration file editing tool and library
LICENSE=	gnu-lgpl-v2

# url2pkg-marker (please do not remove this line.)
.include "../../mk/bsd.pkg.mk"

After writing the file and exiting the editor session, url2pkg will continue and download the source tarball, create the DESCR, PLIST and distinfo files, then unpack the source ready for you to start applying patches.

At this point you should edit DESCR and put in a few lines which describe the package.

Build

The general cycle will then be:

Try building the package with bmake.
If something goes wrong, modify the Makefile or patch the source with pkgdiff.

until the package builds. Before running your first bmake, I would strongly recommend adding PKG_DEVELOPER=yes to your mk.conf to turn on a lot of additional checks.

For augeas, I needed a couple of things.

Patches

The first problem I hit was:

$ bmake
[...]
=> Checking for portability problems in extracted files
ERROR: [check-portability.awk] => Found test ... == ...:
ERROR: [check-portability.awk] configure: test "`uname`" == "SunOS" && \

Explanation:
===========================================================================
The "test" command, as well as the "[" command, are not required to know
the "==" operator. Only a few implementations like bash and some
versions of ksh support it.

When you run "test foo == foo" on a platform that does not support the
"==" operator, the result will be "false" instead of "true". This can
lead to unexpected behavior.

There are two ways to fix this error message. If the file that contains
the "test ==" is needed for building the package, you should create a
patch for it, replacing the "==" operator with "=". If the file is not
needed, add its name to the CHECK_PORTABILITY_SKIP variable in the
package Makefile.
===========================================================================

In this case pkgsrc gives us a very helpful message telling us about the problem and what needs to be done to fix it (unfortunately not all problems are handled this well!). As configure is pretty important, we’ll need to patch it.

This is my usual way of handling patches (ensuring pkgtools/pkgdiff is installed first):

: Set up patches directory and some useful variables.
$ mkdir patches
$ patchdir=$(pwd -P)/patches
$ wrkdir=$(bmake show-var VARNAME=WRKSRC)

: Edit file and generate patch
$ cd $wrkdir
$ pkgvi configure
$ pkgdiff configure >$patchdir/patch-configure

: Return to pkgsrc directory, re-generate distinfo and try again.
$ cd -
$ bmake clean; bmake mps; bmake

This was enough to fix the == problem. I then ran into other issues.

Dependencies

Dependencies in pkgsrc are primarily handled in two ways, either with a buildlink file when depending upon shared libraries or particular infrastructure, or a simple DEPENDS line to pull in a required package.

The first dependency problem I hit was:

[...]
checking readline/readline.h usability... no
checking readline/readline.h presence... no
checking for readline/readline.h... no
checking for readline in -lreadline... no
checking for library containing tgetent... -lcurses
checking for rl_initialize in -lreadline... no
configure: WARNING: readline library not found
configure: error: Could not find a working readline library (see config.log for details).
*** Error code 1

Ok, so this package depends upon readline. As that is a library, we are looking for a suitable buildlink3.mk file, and one of the easiest ways to do this is:

$ ls -1d ../../*/*readline*/buildlink3.mk
../../devel/readline/buildlink3.mk

To use that file, all we need to do is add an .include line at the bottom of Makefile, just above the bsd.pkg.mk include.

.include "../../devel/readline/buildlink3.mk"
.include "../../mk/bsd.pkg.mk"

After that, bmake clean; bmake to try again. I then hit:

[...]
checking for LIBXML... no
configure: error: Package requirements (libxml-2.0) were not met:

No package 'libxml-2.0' found

Ok, so similar method:

$ ls -1d ../../*/*libxml*/buildlink3.mk
../../textproc/libxml++/buildlink3.mk
../../textproc/libxml++1/buildlink3.mk
../../textproc/libxml/buildlink3.mk
../../textproc/libxml2/buildlink3.mk
../../textproc/py-libxml2/buildlink3.mk

The configure message said libxml-2.0, so we’ll pick textproc/libxml2. If you want to be more thorough you can look at the PLIST files for each of the possible candidate packages - I just happen to know from experience that textproc/libxml2 is the one I need.

The bottom of Makefile now looks like this:

.include "../../devel/readline/buildlink3.mk"
.include "../../textproc/libxml2/buildlink3.mk"
.include "../../mk/bsd.pkg.mk"

You’ll note that we try to keep all includes other than bsd.pkg.mk (which is special) sorted alphabetically.

After this, the package finally completes a bmake with no problems. The final step is to get the install phase working.

Install

First we need to run a stage-install which will execute make install into a temporary DESTDIR:

$ bmake stage-install

This will almost certainly fail as we haven’t configured the PLIST yet, so pkgsrc has no idea what will be installed from this package.

[...]
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for augeas-1.0.0
ERROR: ************************************************************
ERROR: The following files are in /var/tmp/pkgsrc-build/devel/augeas/work/.destdir/opt/pkg but not in the PLIST:
ERROR:         /var/tmp/pkgsrc-build/devel/augeas/work/.destdir/opt/pkg/bin/augparse
ERROR:         /var/tmp/pkgsrc-build/devel/augeas/work/.destdir/opt/pkg/bin/augtool
[...]

However, now that we have a populated DESTDIR, we can use the print-PLIST target to generate it for us:

$ bmake print-PLIST >PLIST

Finally, it’s worth doing a full clean and install to ensure everything works as expected.

$ bmake clean
$ bmake install
[...]
=> Creating binary package /var/tmp/pkgsrc-build/devel/augeas/work/.packages/augeas-1.0.0.tgz
===> Install binary package of augeas-1.0.0

buildlink

As we are installing a library package, we should provide a buildlink3.mk file of our own so that other packages can depend on us correctly. Again, there is a package that can help with this - pkgtools/createbuildlink.

$ createbuildlink >buildlink3.mk

This will create a template file but you should read and edit, removing the lines marked with XXX and making any changes they recommend.

Verify

Once we have a working package it’s worth doing a couple of checks to make sure everything looks ok:

: Check the package listing looks ok
$ pkg_info -L augeas

: View the included DESCR and homepage
$ pkg_info -d augeas

: Check what the package lists as its dependencies
$ pkg_info -N augeas

We also should run pkglint (pkgtools/pkglint) which will perform some sanity checks on our infrastructure files prior to us importing the package into pkgsrc itself. In my case it pointed out some possible issues:

$ pkglint
WARN: PLIST:8: Packages that install libtool libraries should define USE_LIBTOOL.
WARN: PLIST:9: Packages that install libtool libraries should define USE_LIBTOOL.
ERROR: patches/patch-configure:3: Comment expected.
1 errors and 2 warnings found. (Use -e for more details.)

For the first issue, add USE_LIBTOOL=yes to Makefile which will ensure that the pkgsrc libtool is used (has better support for cross-platform issues), and for the second, add a comment at the top of the file explaining what it is for, then re-generate distinfo with bmake mps.

After making those changes, everything looks great:

$ pkglint
looks fine.

Upstream

The final - and sometimes most challenging - part is to get your shiny new package integrated into pkgsrc. There are a few options:

Find a NetBSD developer (like myself) who can review your package and put it straight into pkgsrc.
Get a pkgsrc-wip account and work on your packages there, before getting it reviewed and integrated.
If it’s somewhat niche just publish it up on GitHub or similar for people to use, in a similar way to what we do with pkgsrc-joyent

Further information

This post only scratches the surface of adding a new package. pkgsrc provides a huge amount of infrastructure to help get packages working on multiple platforms, and there are lots of options available.

Certain parts of the infrastructure, like buildlink, are very complicated, and for more in-depth information you should refer to the pkgsrc guide.

The best way is often just to look at other packages to see what they do and re-use useful bits you find. With 12,000 packages there is almost certainly another package which does something similar to what you need!

Enjoy!

What's new in pkgsrc-2013Q1

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 15 Apr 2013 00:00:00 +0100

Note

This page is obsolete. Please see http://pkgsrc.joyent.com/ where you will find the latest package sets and improved instructions.

Original Article

The latest branch of pkgsrc was released at the beginning of April, and binary packages for SmartOS/illumos and OSX are now available.

Instructions for installing, as well as a list of the major new features in pkgsrc-2013Q1 are below.

Installing

The instructions are similar to previous branches.

SmartOS/illumos

SmartOS users are encouraged to use our pre-built machine images, and installing your choice of base/standard image with version 13.1.x (available very soon) will get you a pkgsrc-2013Q1 based image.

For general illumos users or SmartOS users who want access to a full package set, the instructions are below:

# curl http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2013Q1-illumos.tar.gz | gtar -zxpf - -C /
# PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
# pkgin -y update
# pkgin avail | wc -l
    9842
# pkgin search <package>
# pkgin -y install <package> <package...>

OSX

Beginning with pkgsrc-2013Q1 I will now be providing regular builds for OSX. Again, the instructions are similar to those previously provided.

These packages are built on OSX Leopard (10.5) but use the PREFER_PKGSRC mechanism to ensure that they are portable across OSX releases, and have been successfully tested on OSX Lion (10.7).

$ curl http://pkgsrc.smartos.org/packages/Darwin/bootstrap/bootstrap-2013Q1-Darwin.tar.gz | sudo gnutar -zxpf - -C /
$ PATH=/usr/pkg/sbin:/usr/pkg/bin:$PATH
$ sudo pkgin -y update
$ pkgin avail | wc -l
    8108
$ pkgin search <package>
$ sudo pkgin -y install <package> <package...>

What’s New

As usual there were many hundreds of changes which went into this quarterly release of pkgsrc. Here are some of the more interesting and useful changes.

OpenSSL 1.0.1 with AES-NI support

OpenSSL has been upgraded from the 0.9.8 series to the 1.0.1 series. The driving reason to pursue this upgrade was to take advantage of AES-NI support which significantly improves crypto performance on Intel CPUs which provide that feature.

On my OSX 10.7 Core i7 laptop the numbers below speak for themselves:

: /usr/bin/openssl 'OpenSSL 0.9.8r 8 Feb 2011'
$ openssl speed -evp aes-128-cbc
  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
  aes-128-cbc     157297.05k   173874.73k   176805.45k   177719.17k   179441.78k

: pkgsrc openssl 'OpenSSL 1.0.1e 11 Feb 2013'
$ openssl speed -evp aes-128-cbc
  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
  aes-128-cbc     643315.29k   685811.37k   696899.67k   699977.39k   693968.90k

A pretty significant 4x improvement for many hundreds of applications which use OpenSSL for crypto.

GCC Go support for SmartOS/illumos

Go is a reasonably new programming language from Google that a number of our users have asked us to support, so we are pleased to announce that beginning with pkgsrc-2013Q1 you will be able to use the gccgo front-end to compile and run Go applications on SmartOS.

You simply compile the go source code as you would for any other language that GCC supports, for example:

# pkgin -y install gcc47

: /opt/pkg for the illumos package set, /opt/local for SmartOS datasets..
# PATH=/opt/pkg/gcc47/bin:$PATH

# gccgo app.go -o app
# ./app

Networking utilities on SmartOS

Thanks to initial work by @postwait there is now proper Zone support in libpcap, which has opened up the possibility to run a number of networking utilities in Joyent SmartMachines.

Yes, this means you can finally run tcpdump instead of snoop.

One of my favourites is trafshow which is a top-like interface for network, and looks like this:

Alternatively you can try nicstat for a more {io,mp,vm}stat style display.

Major package versions

As usual there was also a slew of version updates, and the most notable package versions are listed below. These of course are not exhaustive lists.

Development:

Clang 3.2
GCC 4.7.2
Git 1.8.1.5
Mercurial 2.5.2
Subversion 1.6.20, 1.7.8

Languages:

Lua 5.1.15
Node.js 0.8.23, 0.10.2
Ocaml 4.00.1
Oracle JRE/JDK 6.0.37, 7.0.15
Perl 5.16.2
PHP 5.3.23, 5.4.13
Python 2.6.8, 2.7.3, 3.1.5, 3.2.3, 3.3.0
R 2.15.1
Ruby 1.8.7.371, 1.9.3p392

Web Stack:

Apache 1.3.42, 2.0.64, 2.2.24, 2.4.4
CouchDB 1.2.1
MongoDB 2.2.2
MySQL 5.0.96, 5.1.67, 5.5.30, 5.6.10
Nginx 1.2.7, 1.3.14
PostgreSQL 8.3.23, 8.4.17, 9.0.13, 9.1.9, 9.2.4
Riak 1.2.1

Desktop:

evilwm 1.1.0
GNOME 2.32.1, 3.6.2
KDE 3.5.10, 4.8.4
XFCE 4.6.1

Enjoy!

Installing SVR4 packages on SmartOS

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 19 Mar 2013 00:00:00 +0000

Up until and including Solaris 10 the default packaging tools on Solaris were the historical SVR4 pkg* commands. First written in the early 1980s they were standard across commercial Unix systems and provided a simplistic interface to installing and removing binary packages.

With the introduction of IPS in OpenSolaris and beyond they have been mostly consigned to history, however there is still software provided for Solaris which is only available in the .pkg format, and thus it is useful to still be able to handle them.

Whilst the pkg* tools continue to be maintained in illumos and are provided by various distributions, they are not all provided in SmartOS. There are a few reasons for this:

SmartOS has a different design to other illumos distributions, and some key differences such as a read-only /usr mean that some packages will simply break in unexpected ways.
SmartOS is designed to be a slimmed-down distribution providing only that which is necessary for the majority of our users and use cases. Including the SVR4 tools and metadata would bloat the system.
SVR4 packages are often available only for older versions of Solaris, and whilst the excellent ABI compatability in Solaris means that the binaries themselves will often function correctly, the package may not support newer features such as SMF, or again make assumptions about the system which could result in irrevocable damage.
SmartOS uses pkgsrc to manage third-party software, and we believe it is better to convert SVR4 packages to pkgsrc format so that all packages on the system can be managed with a single toolset.

However, we do continue to ship the pkgtrans utility with SmartOS, and this is our gateway into converting SVR4 packages into more useful formats. The rest of this post will explore how we can do that.

Unpacking SVR4 packages

Let’s start with an example SVR4 package and unpack it to see what it contains. I’m going to use Riak, a popular open source database as the example package.

: Download the Solaris 10 SVR4 package from http://docs.basho.com/riak/latest/downloads/
$ curl -Os http://s3.amazonaws.com/downloads.basho.com/riak/1.3/1.3.0/solaris/10/BASHOriak-1.3.0-1-Solaris10-i386.pkg.gz

: Decompress it
$ gzip -d BASHOriak-1.3.0-1-Solaris10-i386.pkg.gz

: Use pkgtrans to unpack it into /var/tmp/BASHOriak
$ pkgtrans BASHOriak-1.3.0-1-Solaris10-i386.pkg /var/tmp all

SVR4 packages can contain multiple sub-packages, and so the ‘all’ is necessary to unpack everything in the archive. If we didn’t specify ‘all’, we would have seen:

$ pkgtrans BASHOriak-1.3.0-1-Solaris10-i386.pkg /var/tmp

The following packages are available:
  1  BASHOriak     riak
                   (i386) 1.3.0-1

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:

So whilst we could have specified ‘riak’, we can always use ‘all’ to avoid having to first look at the package to see what sub-packages it contains.

We now have an unpacked package, let’s go through what it contains.

`install` sub-directory

The install/ directory contains some files and scripts:

$ ls -l install/
total 29
-rw-------   1 admin    deniedssh   10175 Feb 19 15:17 copyright
-rw-------   1 admin    deniedssh     214 Feb 19 15:17 depend
-rwx------   1 admin    deniedssh     438 Feb 19 15:17 i.preserve
-rwx------   1 admin    deniedssh     339 Feb 19 15:17 preinstall
-rwx------   1 admin    deniedssh     469 Feb 19 15:17 r.preserve

copyright is self-explanatory, and is normally displayed when using the pkgadd command to let the admin know what they are agreeing to.
depend is a list of other SVR4 packages that this one depends upon. In this case they are:

$ cat install/depend
# Same dependencies as Erlang
P SUNWlibmsr    Math & Microtasking Libraries (Root)
P SUNWlibms     Math & Microtasking Libraries (Usr)
P SUNWopensslr  OpenSSL (Root)
P SUNWopenssl-libraries OpenSSL Libraries (Usr)

As this package is originally from Solaris 10 there is a chance that dependencies could cause issues. For example, in SmartOS we have updated OpenSSL to 1.0.x. Additionally, if a third-party dependency was required (i.e. one not beginning with SUNW) then naturally you would need to recursively apply this entire procedure to each dependency.

i.preserve and r.preserve are scripts executed during install (i.) and removal (r.). The ones for Riak simply try to retain modified files from an existing install, so we will ignore these as pkgsrc handles that by default.
preinstall is, as the name suggests, a script which is executed prior to installing the package. In Riak’s case it is used to create the ‘riak’ user and group if they do not already exist.

`pkginfo`

This provides some basic metadata about the package. The main bits we care about are:

ARCH=i386. As long as the package only depends upon libraries provided by the base OS (SUNW*) then it shouldn’t matter whether ARCH is 32-bit or 64-bit. However, if it requires third-party dependencies then you need to ensure that the correct ABI is provided.
BASEDIR=/opt. This is where the package would be installed by the pkgadd tool.
DESC=.... This would be output by the legacy pkginfo command, and we will re-use this text for our pkg_info description.
VERSION=1.3.0-1. Self-explanatory.

`pkgmap`

This is somewhat equivalent to the pkgsrc PLIST file and is a record of all the files the package provides, however it also includes file permissions and a basic checksum:

$ less pkgmap
: 1 112659
1 i copyright 10175 24223 1361287043
1 i depend 214 18268 1361287043
...
1 d none riak 0700 riak riak
1 d none riak/bin 0700 riak riak
1 f none riak/bin/riak 0755 riak riak 9041 51698 1361286795
...
1 e preserve riak/etc/app.config 0600 riak riak 14214 8625 1361286647

The last two lines, the important fields are:

i is an SVR4 metadata file, f or d denote whether it is a file or a directory, e are configuration files.
none means no special handling, preserve does just that, and the next field is the full path relative to reloc/
0700 and 0755 are the file/directory permissions
riak riak are the user and group ownership

We will need to ensure at least the file entries are handled correctly.

`reloc/` sub-directory

This directory contains the binaries etc. which make up the actual package. The contents of this directory would normally be installed under BASEDIR from the pkginfo file, so in Riak’s case:

: This..
$ ls reloc
riak
$ ls reloc/riak
bin         erts-5.9.1  etc         lib         releases

: ..would result in this
$ ls /opt/riak
bin         erts-5.9.1  etc         lib         releases

This concludes the examination of the SVR4 package. Let’s turn it into a useful pkgsrc package.

Creating pkgsrc binary package

For more information on creating binary pkgsrc packages from scratch, see this post.

pkgsrc metadata

Create the necessary pkgsrc metadata files.

$ mkdir /var/tmp/pkgsrc-riak
$ cd /var/tmp/pkgsrc-riak

: Standard build-info section.  Change MACHINE_ARCH to x86_64 if you are
: using a base64 image.
$ cat >build-info <<EOF
MACHINE_ARCH=i386
OPSYS=SunOS
OS_VERSION=5.11
PKGTOOLS_VERSION=20091115
EOF

: Generate comment file directly from the DESC field in pkginfo
$ awk -F= '/DESC/ {print $2}' < /var/tmp/BASHOriak/pkginfo >comment

: Generate PLIST directly from pkgmap
$ awk '$2 ~ /[ef]/ {print $4}' < /var/tmp/BASHOriak/pkgmap >plist

: For now just re-use DESC for the description file, however it would normally
: be longer
$ cp comment descr

pkgsrc INSTALL script

To handle the Riak preinstall script, we will create a pkgsrc INSTALL script.

The existing script can be mostly used as-is, we just need to put the entire contents of preinstall inside a PRE-INSTALL case statement so that it is executed prior to installing the package:

: Start with the existing preinstall script
$ cp /var/tmp/BASHOriak/install/preinstall inst

: Alter the script to create the 'riak' user/group during PRE-INSTALL, and
: after install to chown everything to 'riak' (which 
$ vi inst

PKGNAME="$1"
STAGE="$2"

case ${STAGE} in
PRE-INSTALL)
	# Existing preinstall script goes here, changing /opt references
	# to $PKG_PREFIX
	;;

If we recall from the pkgmap file, the entries there contained a user/group that each file should be owned by, and we can handle that in the INSTALL script too with a POST-INSTALL action:

POST-INSTALL)
	chown -R riak:riak ${PKG_PREFIX}/riak
	;;
esac

pkgsrc files

First we simply copy everything from the reloc/ directory to a files/ directory we will use for pkgsrc:

$ mkdir files
$ rsync -a /var/tmp/BASHOriak/reloc/ files/
$ chown -R root:root files

Next we can use the pkgmap file to ensure that the file modes are set correctly with a quick and dirty script:

while read line
do
    set -- $line
    case "$3" in
    [def])
        chmod $5 files/$4
        ;;
    esac
done < /var/tmp/BASHOriak/pkgmap

Create the package

We should now have everything necessary to create a binary package, taking the version from the pkgmap file.

$ pkg_create -B build-info -c comment -d descr -f plist -I /opt/local -i inst -p files -U riak-1.3.0.tgz

Testing

If all went well then we should be able to install the package:

$ pkg_add riak-1.3.0.tgz

and we will find it under /opt/local/riak as expected. If we try to run the binary, we get:

$ /opt/local/riak/bin/riak
/opt/local/riak/bin/riak: line 30: whoami: not found
sudo doesn't appear to be installed and your EUID isn't riak

This nicely proves my earlier point about packages often not working unmodified on SmartOS, in this case because whoami is no longer provided. Thankfully this is an easy fix, and we can simply change whoami to id -un.

Making that change and trying again, but this time as the riak user:

$ su - riak
$ /opt/local/riak/bin/riak
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
Usage: riak {start|stop|restart|reboot|ping|console|attach|chkconfig|escript|version|getpid}
$ /opt/local/riak/bin/riak start
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
$ pgrep -fl riak
20628 /opt/local/riak/erts-5.9.1/bin/epmd -daemon
20650 /opt/local/riak/erts-5.9.1/bin/beam.smp -K true -A 64 -W w -- -root /opt/local/
20648 /opt/local/riak/erts-5.9.1/bin/run_erl -daemon /tmp//opt/local/riak/ /opt/local
20718 /opt/local/riak/lib/os_mon-2.2.9/priv/bin/cpu_sup
20716 /opt/local/riak/lib/os_mon-2.2.9/priv/bin/memsup

This seems to work about as well as one can hope, and concludes my basic example.

## Further work

I’ve covered the basics here, but there are additional things you could do to tidy up the conversion:

Fold the whoami fix back into the source file and re-generate the package.
Turn this into a real pkgsrc package, which would simplify some areas such as metadata and user creation.
Come up with a script to automate a lot of this work.
Turn the riak script into an SMF service.

Also note that Basho very helpfully already provide a native SmartOS package on their download page, so this example is somewhat pointless, however I hope it has still proven useful ;)

SmartOS is Not GNU/Linux

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 27 Feb 2013 00:00:00 +0000

One of the requests we get from time to time is for SmartOS to look more like GNU/Linux in layout and behaviour. For example, config files in /etc instead of /opt/local/etc, binaries under /usr instead of /opt/local/{s,}bin, GNU userland by default, etc.

Whilst we believe in the technical merits of our current implementation and the clean separation and upgrade possibilities it provides, we do recognise that some users just don’t care about those things and would prefer a system which looks as close to the GNU/Linux environments they are used to.

Ordinarily this simply wouldn’t be possible given that /usr is a read-only mount from the global zone, however with the highly flexible SmartOS Zones architecture, coupled with Joyent employing Zones guru Jerry Jelinek, we are able to provide you with an option to do exactly this. As Jerry says, at Sun there was even a native Linux brand, so pretty much anything is possible!

We call it ‘SNGL’ (pronounced ‘snuggle’), which is an acronym for ‘SmartOS is Not GNU/Linux’. Currently it is somewhat experimental, but we’d love for people to try it out and provide feedback.

Here’s how you can get it running.

Install the latest platform

You need to be running SmartOS 20130222 or later. Older platforms can be coerced into working, you will just need to work around the lack of this commit by creating an empty sngl_base.tar.gz or so.

As usual, follow the instructions here to upgrade an existing install.

Get the SNGL dataset

: Fetch the dataset image and manifest files.  The image is 107MB.
$ mkdir -p /usbkey/images
$ cd /usbkey/images
$ curl -O http://pkgsrc.smartos.org/datasets/sngl-0.99.0.dsmanifest
$ curl -O http://pkgsrc.smartos.org/datasets/sngl-0.99.0.zfs.bz2

: Import it
$ imgadm install -m sngl-0.99.0.dsmanifest -f sngl-0.99.0.zfs.bz2

Create a new dataset

The important point to note here is that brand is set to sngl.

: Create a new zone using the dataset (change your json to suit).
$ vmadm create <<EOF
{
  "brand": "sngl",
  "image_uuid": "4bf9530a-7ae5-11e2-bb4e-3bad5fbc3de9",
  "ram": 256,
  "quota": 10,
  "alias": "sngl-0.99.0",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}
EOF

At this point you should be able to log in and start using pkgin etc to install new software (there are over 2,000 packages available) as normal, but notice that:

binaries are running from /usr/bin
configuration files are in /etc
the default userland tools are GNU variants (ls, sed, awk, grep, etc.)

For those that are interested, here is some further detail on how this is all implemented.

Brand configuration

The main setup is in /usr/lib/brand/sngl. Firstly, platform.xml defines the mount points to be used inside the zone, and here you can see how we are able to use /usr:

        <global_mount special="/lib" directory="/system/lib"
            opt="ro,nodevices" type="lofs" />
        <global_mount special="/sbin" directory="/system/sbin"
            opt="ro,nodevices" type="lofs" />
        <global_mount special="/usr" directory="/system/usr"
            opt="ro,nodevices" type="lofs" />

We are transplanting the main system directories and mounting them under /system. This leaves /usr free for us to write packages to.

In order to support having the OS under /system there is some additional configuration in config.xml.

        <initname>/system/sbin/init</initname>
        <login_cmd>/system/usr/bin/login -z %Z %u</login_cmd>
        <forcedlogin_cmd>/system/usr/bin/login -z %Z -f %u</forcedlogin_cmd>
        <user_cmd>/system/usr/bin/getent passwd %u</user_cmd>

This is where the flexibility of Zones really shines. We are able to redefine the path to init(1M) and others so that the zone can boot correctly.

In addition, we copy in the crle configuration files ld.sys.config and ld.sys64.config so that binaries will look in /system/usr/lib for their runtime libraries.

Runtime and packages

The brand configuration is enough to set the zone up, but in order to make it boot we need additional files available under /usr, there are simply too many hardcoded paths. For this we just symlink back to /system/usr from /usr any files required.

Finally, we are able to perform a full pkgsrc bulk build with LOCALBASE set to /usr within a chroot which emulates this layout, and when those packages are installed they overwrite the compatability symlinks we have configured and replace them with files from the packages.

Not all symlinks will be overwritten, though, which is why standard SmartOS utilities such as prstat(1M) are still available, as the symlink for it still exists.

Reporting issues

As I mentioned, this is currently experimental, and there will be plenty of problems. However, at least from some initial testing, a reasonable amount of things appear to work fine, and for users who want this particular layout it may be good enough.

Please feel free to give it a try and report issues against our GitHub project. Once we have it working with a reasonable amount of stability we may be able to offer it as an option in the Joyent Public Cloud.

SmartOS development preview dataset

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 18 Feb 2013 00:00:00 +0000

The datasets we produce for SmartOS are usually released on a quarterly cadence, matching the upstream pkgsrc release branches. This is often enough to ensure that people can get up-to-date software with all the usual improvements and bug fixes.

Occasionally though, users want the very latest and don’t really want to wait for 3 months to get it, and so to satisfy those users who crave the bleeding edge I have produced a new dataset which is based upon pkgsrc trunk.

The package repository for this dataset will be constantly updated with the very latest that pkgsrc has to offer, and so you will occasionally see breakage as we integrate updates for core libraries and add new features. Think of it as being similar to Debian ‘unstable’, you get the very latest stuff but you may need to do some maintenance every so often.

Here are some reasons why you may want to use this dataset:

It is based upon the multiarch code, so only one image is necessary. No more needing to decide between the 32-bit or 64-bit options.
It is a full bulk build, so there are in the region of 9,000 packages to choose from (depending on the current state of pkgsrc trunk).
It has all the Joyent specific changes integrated, so that means SMF support for many packages which we have written manifests for, static UID/GID allocation, various improvements, and uses the standard /opt/local prefix.
There are some nice improvements only available in pkgsrc trunk, for example OpenSSL 1.0.1e with proper AES-NI support, and libpcap fixes from @postwait with a slew of networking utilities now available and working (these will be documented in a future post).

How to install

Here is a quick start guide to getting the preview dataset up and running:

: Fetch the dataset image and manifest files.  The image is 82MB.
$ mkdir -p /usbkey/images
$ cd /usbkey/images
$ curl -O http://pkgsrc.smartos.org/datasets/trunk-0.99.0.dsmanifest
$ curl -O http://pkgsrc.smartos.org/datasets/trunk-0.99.0.zfs.bz2

: Import it
$ imgadm install -m trunk-0.99.0.dsmanifest -f trunk-0.99.0.zfs.bz2

: Create a new zone using the dataset (change your json to suit).
$ vmadm create <<EOF
{
  "brand": "joyent",
  "image_uuid": "c91b3752-79c5-11e2-ad33-67667b9ee2c2",
  "max_physical_memory": 256,
  "alias": "trunk-0.99.0",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}
EOF

then login and start using it as you would with any other dataset.

Known issues

With this being a bleeding-edge distribution, there will undoubtedly be problems.The main one I am currently aware of is dependencies upon the GCC runtime from the /opt/pbulk prefix used to build the packages, which looks like this:

# pkgin in samba
calculating dependencies... done.
/opt/pbulk/gcc47/lib/./libgcc_s.so.1, needed by samba-3.6.12nb1 is not present in this system.
/opt/pbulk/gcc47/lib/./libgcc_s.so.1, needed by tdb-1.2.11 is not present in this system.
[...]

We are actively working on fixing these dependencies, but in the meantime if you require a package which is broken in this way you can work around it by installing the /opt/pbulk bootstrap like this:

# curl http://pkgsrc.smartos.org/packages/SmartOS/bootstrap/bootstrap-pbulk.tar.gz \
    | gtar -zxf - -C /
# /opt/pbulk/bin/pkgin -y install gcc47

The other issue to note is that not all packages have been converted to multiarch. Again we are actively working on this, but it will take some time to go through all of the 12,000 or so packages in pkgsrc.

If you find any other problems, please feel free to raise them against our GitHub project here. Or, even better, follow my guides on building pkgsrc (you will want the joyent/release/trunk branch) and have a go at fixing things yourself, we very much welcome patches and pull requests!

pkgsrc on SmartOS - fixing broken builds

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 17 Jan 2013 00:00:00 +0000

This is the second in a series of posts looking at pkgsrc on SmartOS. In the previous post I got us up and running with building packages. This post will focus on what to do when the build fails.

Currently pkgsrc is able to build around 9,500 packages on SmartOS, however pkgsrc carries over 12,000 packages in total, so there is a reasonable chance that you will come across a package which will not produce a binary package.

Let’s have a look at some of the most common failure modes and the facilities pkgsrc provides for them to be fixed.

Adjusting the environment

Probably the most common failures on Solaris are those caused by incorrect compiler or linker flags, where the author of the software has not taken into account differences across Unix platforms. Examples include:

the usage of u_int* types (e.g. u_int32_t) instead of the portable uint* C99 types (e.g. uint32_t).
missing -lsocket -lnsl when using the socket interface.

pkgsrc provides an easy way to pass CFLAGS, LDFLAGS and other common environment variables to the build, and these are often enough to resolve such issues.

Taking net/nsd as an example:

# cd net/nsd
# bmake package
...
gcc -I/opt/local/include -I/opt/local/include -I. -I. -O -I/opt/local/include -c ./util.c
./util.c:745:1: error: unknown type name 'u_int32_t'
*** [util.o] Error code 1

If we edit the Makefile and add the following line below CONFIGURE_ARGS:

CFLAGS.SunOS+=	-Du_int32_t=uint32_t

then a rebuild resolves the problem and results in a binary package:

# bmake clean
# bmake package
...
=> Creating binary package /content/packages/All/nsd-3.2.14.tgz

By using CFLAGS.SunOS rather than the global CFLAGS this is only performed on systems where uname is SunOS.

We can resolve missing libraries in a similar way, taking net/rootprobe as an example:

# cd net/rootprobe
# bmake package
...
gcc -Wl,-R/opt/local/lib  rootprobe.o -o rootprobe
Undefined                       first referenced
 symbol                             in file
recv                                rootprobe.o
send                                rootprobe.o
getsockname                         rootprobe.o
socket                              rootprobe.o
getdomainname                       rootprobe.o
connect                             rootprobe.o
recvfrom                            rootprobe.o
inet_aton                           rootprobe.o
inet_ntoa                           rootprobe.o
shutdown                            rootprobe.o
ld: fatal: symbol referencing errors. No output written to rootprobe
collect2: error: ld returned 1 exit status
*** [rootprobe] Error code 1

Add the following to the Makefile:

LDFLAGS.SunOS+=	-lsocket -lnsl

and hey presto, out comes a binary package:

# bmake clean
# bmake package
...
gcc -lsocket -lnsl -Wl,-R/opt/local/lib  rootprobe.o -o rootprobe
/bin/rm -f cctldprobe;  ln rootprobe cctldprobe
...
=> Creating binary package /content/packages/All/rootprobe-200301.tgz

Not all packages can be fixed directly like this, only those which obey the normal ${CC} and ${LD} environment variables, as pkgsrc creates wrappers for those commands where it can insert these alterations. For packages which directly call e.g. gcc, some additional digging will be required to see how the arguments can be passed.

Taking net/3proxy as an example, the build fails with missing socket libraries:

# cd net/3proxy
# bmake package
...
gcc -opop3p -Wall -O2 -pthread  sockmap.o pop3p.o sockgetchar.o myalloc.o common.o  
Undefined                       first referenced
 symbol                             in file
bind                                pop3p.o
send                                sockgetchar.o
getsockname                         common.o
accept                              pop3p.o
listen                              pop3p.o
gethostbyname                       common.o
sendto                              sockmap.o
socket                              pop3p.o
setsockopt                          pop3p.o
connect                             common.o
recvfrom                            sockmap.o
shutdown                            sockmap.o
ld: fatal: symbol referencing errors. No output written to pop3p
collect2: error: ld returned 1 exit status
*** [pop3p] Error code 1

but adding LDFLAGS.SunOS+= -lsocket -lnsl to the Makefile as before does not resolve the problem. Delving further into the Makefile we can see that the build is driven from a custom Makefile rather than through autoconf/automake:

MAKE_FILE=      Makefile.unix

and it is there we will need to perform the changes. First, let’s find the work area, which we can get from the WRKSRC variable:

# bmake show-var VARNAME=WRKSRC
/var/tmp/pkgsrc-build/net/3proxy/work
# cd /var/tmp/pkgsrc-build/net/3proxy/work
# ls -l Makefile.unix 
-rw-r--r-- 1 11001 10512 675 Apr 30  2005 Makefile.unix

Picking out the relevant bits from Makefile.unix:

CC = gcc
...
CFLAGS = -Wall -g -O2 -c -pthread -D_THREAD_SAFE -D_REENTRANT -DNOODBC -DWITH_STD_MALLOC -DFD_SETSIZE=4096 -DWITH_POLL
...
LDFLAGS = -Wall -O2 -pthread
...

Here, the author has provided no functionality for adding to the environment, and has instead chosen to hardcode a specific compiler and build flags, significantly reducing the portability of their software - good luck to Clang/LLVM or SunStudio users!

If the author had provided a way in to add to these flags, for example:

LDFLAGS = -Wall -O2 -pthread ${USER_LDFLAGS}

then we could have fixed that in the main pkgsrc Makefile with

MAKE_ENV+=	USER_LDFLAGS="-lsocket -lnsl"

or similar, however we now have no choice but to directly edit Makefile.unix. Thankfully, pkgsrc provides a couple of ways to easily do this, which we will look at over the next few sections.

The substitution framework

The subst framework allows basic editing of files within the work area. I will show the solution for the above problem, and then discuss it:

.include "../../mk/bsd.prefs.mk"
.if ${OPSYS} == "SunOS"
SUBST_CLASSES+=		libs
SUBST_STAGE.libs=	pre-build
SUBST_MESSAGE.libs=	Adding SunOS socket libraries
SUBST_FILES.libs=	Makefile.unix
SUBST_SED.libs=		-e '/^LDFLAGS/s/$$/ -lsocket -lnsl/'
.endif

Firstly, we need to limit this to SunOS systems, else we would break platforms which do not have libsocket or libnsl. Including ../../mk/bsd.prefs.mk gives us access to the OPSYS variable so we can test the platform we are running on.

Next we set up the substitution framework:

SUBST_CLASSES creates a new class, which I have named libs. This class name is then appended to the remaining SUBST_* variables to assign them to that class.
SUBST_STAGE defines the make stage when the substitution will be run, and should almost always be pre-build, to ensure it is done after any configuration stage which could rewrite files itself.
SUBST_MESSAGE is optional, and is simply a line which will be printed to the user when the substitution takes places.
SUBST_FILES is a list of files the substitution operates on.
SUBST_SED is the actual substitution, in the form of a sed(1) operation. Here we append -lsocket -lnsl to any line beginning with LDFLAGS. Note $$ is required to get make to escape a $.

# bmake clean
# bmake package
...
===> Building for 3proxy-0.5.3.11nb1
=> Adding SunOS socket libraries
...
=> Creating binary package /content/packages/All/3proxy-0.5.3.11nb1.tgz

Two other subst features you may want are:

SUBST_VARS.foo= VARNAME is a shortcut for the -e 's,@VARNAME@,${VARNAME},g' operation, common with autoconf-based packages.
SUBST_FILTER_CMD.foo= <cmd> allows you to run an arbitrary command, rather than the default of sed.

For more information, see the implementation in pkgsrc/mk/subst.mk.

## Patches

While CFLAGS, LDFLAGS and the substitution framework allow for simple one-liner fixes, often more significant changes are required, and in those cases the only sensible option is to create patches. Thankfully, there are some tools provided in pkgsrc to make this a relatively easy process.

First, let’s take a broken package, devel/bglibs:

# cd devel/bglibs
# bmake package
...
./ltcompile net/cork.c
net/cork.c: In function 'socket_cork':
net/cork.c:39:27: error: 'SOL_TCP' undeclared (first use in this function)
net/cork.c:39:27: note: each undeclared identifier is reported only once for each function it appears in
*** [net/cork.lo] Error code 1

which comes from the following function:

/*
 * ..It is known to work on Linux (with the TCP_CORK option) and to at least
 * compile on BSD (with the TCP_NOPUSH option).  On OS's which lack either of
 * these two options, this function is essentially a no-op.
 */
int socket_cork(int sock)
{
#if defined(TCP_CORK)
  int flag = 1;
  return setsockopt(sock, SOL_TCP, TCP_CORK, &flag, sizeof flag) == 0;
#elif defined(TCP_NOPUSH)
  int flag = 1;
  return setsockopt(sock, SOL_SOCKET, TCP_NOPUSH, &flag, sizeof flag) == 0;
#else
  return 1;
#endif
}

Unfortunately, while SmartOS provides the TCP_CORK flag, it does not understand SOL_TCP, and so we need to modify the test and add an additional one.

Before doing this, we should install the pkgtools/pkgdiff package, which contains a pkgvi wrapper. This allows you to edit a file, and if you make changes, it will save the resulting diff in the pkgsrc patches directory ready for use. Much easier than delving into the work area and creating patches yourself.

: Install via pkgin..
# pkgin in pkgdiff

: ..or through pkgsrc
# (cd ../../pkgtools/pkgdiff && bmake install)

Now edit the offending file:

# pkgvi $(bmake show-var VARNAME=WRKSRC)/net/cork.c

This is what I changed the function to:

int socket_cork(int sock)
{
#if defined(TCP_CORK) && defined(SOL_TCP)
  int flag = 1;
  return setsockopt(sock, SOL_TCP, TCP_CORK, &flag, sizeof flag) == 0;
#elif defined(TCP_CORK) && defined(SOL_SOCKET)
  int flag = 1;
  return setsockopt(sock, SOL_SOCKET, TCP_CORK, &flag, sizeof flag) == 0;
#elif defined(TCP_NOPUSH)
  int flag = 1;
  return setsockopt(sock, SOL_SOCKET, TCP_NOPUSH, &flag, sizeof flag) == 0;
#else
  return 1;
#endif
}

This still isn’t ideal, and would be better converted to a proper autoconf test where we can test functionality instead of definitions, but it will do for example purposes.

After writing, you should get output such as:

pkgvi: File was modified. For a diff, type:
pkgdiff "/var/tmp/pkgsrc-build/devel/bglibs/work/bglibs-1.106/net/cork.c"

and running the pkgdiff command will show you the diff.

The final step is storing the diff into a patch file, and that is accomplished with:

# mkpatches
# bmake mps

The mkpatches command creates a new file in the patches directory called patch-net_cork.c, and the bmake mps (short for the makepatchsum target) regenerates the distinfo file with the correct checksum for that patch.

Finally, you can rebuild

# bmake clean
# bmake package

and, hey presto .. another failure!

--- net/uncork.lo ---
net/uncork.c: In function 'socket_uncork':
net/uncork.c:30:27: error: 'SOL_TCP' undeclared (first use in this function)
net/uncork.c:30:27: note: each undeclared identifier is reported only once for each function it appears in

Now that you know how to fix this, I’ll leave it to you to come up with a patch ;)

One final word on this, you will perhaps notice during the build some warnings:

=> Applying pkgsrc patches for bglibs-1.106nb1
=> Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-ab.orig
=> Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-ac.orig
=> Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-net_cork.c.orig

These files are left by mkpatches, and to remove them you can run

# mkpatches -c

Also note that both patch-ab and patch-ac changed, even though we only modified the net/cork.c file. This is due to differences in diff(1) output, and is ultimately harmless.

Summary

pkgsrc provides a number of ways to fix up broken software:

CFLAGS, LDFLAGS, and other environment variables.
The subst.mk framework for simple file changes.
Patch files for more substantial changes.

The important thing, after using any of these features, is to feed back your changes so that we can integrate them into pkgsrc and everyone can benefit. Probably the easiest way to do that is simply raise an issue against our GitHub pkgsrc fork, and either myself or Filip can commit the patch upstream.

pkgsrc on SmartOS - zone creation and basic builds

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 15 Jan 2013 00:00:00 +0000

Our goal at Joyent is that our binary packages for SmartOS fulfill all of our users’ needs. This isn’t always possible, however - users may want packages we do not yet provide, or build with different options.

To satisfy those demands, it should instead be reasonably straight-forward for users to build their own packages, and this guide hopefully provides all the information for them to do just that.

This is the first in a series of posts, and will focus on getting pkgsrc set up in a clean SmartOS zone and then building some packages. Other posts currently available in the series are:

How to fix build failures

Create a build machine

For this guide I use a base:1.8.1 SmartMachine, but any recent dataset should be fine, and it shouldn’t matter whether it’s 32-bit or 64-bit.

If you are a Joyent customer:

: Fill in your SDC details.  You can use the web interface too, if you prefer.
$ export SDC_CLI_ACCOUNT=account name
$ export SDC_CLI_IDENTITY=/path/to/ssh/key
$ export SDC_CLI_KEY_ID=key identifier
: .. or us-west-1 or us-sw-1 or eu-ams-1
$ export SDC_CLI_URL=https://us-east-1.api.joyentcloud.com

: You really really want json(1), it's awesome
$ sudo npm -g install jsontool

: Get list of 'base' datasets, we'll use the latest
$ sdc-listdatasets | json -a urn id | grep :base: | head -1
sdc:sdc:base:1.8.1 55330ab4-066f-11e2-bd0f-434f2462fada

: You can use sdc-listpackages to show available machine types.  I want one
: with good CPU performance.
$ sdc-createmachine --dataset 55330ab4-066f-11e2-bd0f-434f2462fada \
>   --name "pkgsrc-build-machine" --package "XL  8GB High CPU"

If you are using SmartOS:

{
  "brand": "joyent",
  "image_uuid": "55330ab4-066f-11e2-bd0f-434f2462fada",
  "max_physical_memory": 1024,
  "quota": 24,
  "alias": "pkgsrc-build-machine",
  "hostname": "pkgsrc-build-machine",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}

Put the above json into a file, you may want to tweak some values, then

# imgadm update
# imgadm import 55330ab4-066f-11e2-bd0f-434f2462fada
# vmadm create -f /path/to/json/file

Finally, log into the freshly-created zone.

Install dependencies

First thing you’ll need in the zone are build tools, so:

# pkgin -y up
# pkgin -y in gcc47 scmgit-base

Fetch pkgsrc

We maintain a fork of pkgsrc on github, which includes some additional SmartOS fixes and improvements, so we recommend using that rather than upstream.

I put everything under /content, adjust to your own tastes.

# mkdir /content
# cd /content
# git clone git://github.com/joyent/pkgsrc.git

Next, check out the latest stable branch. Our naming scheme is joyent/release/YYYYQQ, and the latest branch at time of writing is pkgsrc_2012Q4, so:

# cd pkgsrc
# git checkout joyent/release/2012Q4

If you want to be particularly adventurous, you could checkout trunk and get the very latest pkgsrc code, however pkgsrc is a very fast-moving target and you may run into unexpected breakages, and you will certainly end up rebuilding packages very often.

Fetch pkgsrc-wip

pkgsrc-wip is a set of additional work-in-progress packages for pkgsrc, and can be useful for trying out the latest packages. It is designed to be extracted directly into pkgsrc, and we maintain a git submodule of it in our release trees, so all you need to do, assuming you are using a joyent/release/* tree, is:

# git submodule init
# git submodule update

This will also get you a joyent/ directory which is a module of our pkgsrc-joyent tree of additional packages.

If you are on trunk or another tree which does not have the wip submodule, you can fetch it manually with:

# git clone git://github.com/joyent/pkgsrc-wip.git wip

Configure mk.conf

The primary method of configuring pkgsrc is through the /opt/local/etc/mk.conf file. The base image comes with one populated to handle the basic layout, but there are some additional basic variables you may want to set prior to building your first package:

DISTDIR determines where source tarballs are stored, default pkgsrc/distfiles.
PACKAGES is where pkgsrc will store binary packages it has built, default pkgsrc/packages.
WRKOBJDIR is where pkgsrc performs the builds, default pkgsrc/<category>/<package>/work.
ALLOW_VULNERABLE_PACKAGES and SKIP_LICENSE_CHECK should both be set to yes if you just want the package to be built, and don’t care whether it is currently vulnerable to security issues or is released under a specific license.
MAKE_JOBS determines the argument to make -j, i.e. the number of concurrent make threads, default 1.
FETCH_USING sets the default program to use for fetching source code, defaulting to an in-tree version of NetBSD’s ftp program. Setting this to curl on SmartOS, which is included in the base platform, will avoid any dependencies.
BINPKG_SITES is a URL pointing to binary packages which can be used by this tree rather than building everything from source, and settings DEPENDS_TARGET to bin-install enables using these binary packages as dependencies.

Thus, this is how I would configure mk.conf:

# mkdir -p /content/{distfiles,packages}

: Do not change the existing entries in this file, they tell pkgsrc
: where to find important files.
# vi /opt/local/etc/mk.conf

# ...
# Keep the existing mk.conf entries, they are important.
# ...

DISTDIR=	/content/distfiles
PACKAGES=	/content/packages
WRKOBJDIR=	/var/tmp/pkgsrc-build
#
ALLOW_VULNERABLE_PACKAGES=	yes
SKIP_LICENSE_CHECK=		yes
#
MAKE_JOBS=	8
#
FETCH_USING=	curl
#
# Adjust BINPKG_SITES, depending upon the dataset chosen, it should be the URL
# from /opt/local/etc/pkgin/repositories.conf without the trailing 'All'.
BINPKG_SITES=	http://pkgsrc.joyent.com/sdc6/2012Q2/i386/
DEPENDS_TARGET=	bin-install

Building a package

Ok, let’s get started and build a package. pkgsrc is organised into categories, with all packages following the pkgsrc/<category>/<package> layout, and everything is driven with bmake, the BSD implementation of make(1). I’m going to choose nmap as an example, as it will show a couple of things I want to cover.

First, finding it. I tend to just do a simple:

# ls -d */*nmap*
net/nmap

If you want a more featured search, you can do:

# bmake search key=nmap

though the first time you run this it creates the INDEX file it requires, and that can take a long time. Another option is to use the pkgsrc.se web interface.

Once you have found the package, cd into the package directory.

# cd net/nmap

Next, let’s look at any options the package supports.

# bmake show-options

If the package supports build options, as net/nmap does, you’ll see something like:

Any of the following general options may be selected:
	inet6	 Enable support for IPv6.
	ndiff	 Enable tool to compare Nmap scans.
	zenmap	 Enable nmap GUI frontend.

These options are enabled by default:
	inet6

These options are currently enabled:
	inet6

You can select which build options to use by setting PKG_DEFAULT_OPTIONS
or PKG_OPTIONS.nmap.

The place to configure these options is again in /opt/local/etc/mk.conf, so if you want to enable ndiff support for example, then you’d add this to mk.conf:

PKG_OPTIONS.nmap=	ndiff

Re-running the show-options command should now print:

...
These options are currently enabled:
        inet6 ndiff
...

And finally, we can go ahead and build the package. The output from this will be long, so you may want to tee it to a file for reviewing:

# bmake install 2>&1 | tee /var/tmp/nmap.log

Assuming this completes ok, you should note the main stages that make up a package build:

bootstrap-depends comes first, and installs all the dependencies required for pkgsrc to get started. pkgtools/digest is required to calculate the SHA1 and RMD160 checksums of the source tarball. If we hadn’t specified FETCH_USING=curl in our mk.conf then it’s likely that net/tnftp would have been pulled in during this stage too and used to fetch the source.

=> Bootstrap dependency digest>=20010302: NOT found
=> Verifying bin-install for ../../pkgtools/digest
===> Binary install for digest>=20010302
=> Installing digest>=20010302 from /content/packages/All;http://pkgsrc.joyent.com/sdc6/2012Q2/i386//All
pkg_add: Can't process file:///content/packages/All/digest*: No such file or directory
digest-20111104 successfully installed.
...

fetch and checksum then run to download the source tarball for this particular package, and then verify the checksum matches that stored by pkgsrc, to ensure it was downloaded from a good source:

=> Fetching nmap-6.01.tar.bz2
=> Total size: 21640157 bytes
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.6M  100 20.6M    0     0   699k      0  0:00:30  0:00:30 --:--:--  721k
=> Checksum SHA1 OK for nmap-6.01.tar.bz2
=> Checksum RMD160 OK for nmap-6.01.tar.bz2

depends then installs all packages required for both build and runtime for the package in question:

=> Build dependency libtool-base>=2.2.6bnb3: NOT found
=> Verifying bin-install for ../../devel/libtool-base
===> Binary install for libtool-base>=2.2.6bnb3
=> Installing libtool-base>=2.2.6bnb3 from /content/packages/All;http://pkgsrc.joyent.com/sdc6/2012Q2/i386//All
libtool-base-2.2.6bnb6 successfully installed.
...

extract and patch then unpack the source and apply any pkgsrc patches to the package. The patches are located in the patches/ sub-directory for each package:

===> Extracting for nmap-6.01nb5
===> Patching for nmap-6.01nb5

The bulk of the build is performed by configure and build which for most software will consist of ./configure && make.

===> Configuring for nmap-6.01nb5
...
checking whether NLS is requested... yes
...
Configuration complete.  Type make (or gmake on some *BSD machines) to compile.
...
===> Building for nmap-6.01nb5
...
gmake[1]: Leaving directory `/var/tmp/pkgsrc-build/net/nmap/work/nmap-6.01'
=> Unwrapping files-to-be-installed.

The final steps are package and install, firstly package does a fake install of the package to a temporary install directory and creates a binary package from that, and then install installs that binary package into place:

===> Building binary package for nmap-6.01nb5
=> Creating binary package /content/packages/All/nmap-6.01nb5.tgz
===> Install binary package of nmap-6.01nb5

You can now verify it is installed, and start using your new software:

# type nmap
nmap is /opt/local/bin/nmap

# nmap -p 22 localhost

Starting Nmap 6.01 ( http://nmap.org ) at 2013-01-15 15:36 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00012s latency).
PORT   STATE SERVICE
22/tcp open  ssh

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

Cleanup

Once everything is working, you may want to clean up the build areas. The pkgsrc way of doing this is:

# bmake clean clean-depends

However, if you just want to blow everything away without prejudice, you can simply remove everything under WRKOBJDIR, so in my case:

# rm -rf /var/tmp/pkgsrc-build/*

This should hopefully be enough to get you started. Over the next few posts we’ll investigate some more advanced topics. Stay tuned!

Multi-architecture package support in SmartOS

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 10 Jan 2013 00:00:00 +0000

[Updated on 2013-01-16 to use the 12.4.1 dataset which fixes some issues.]

Ever since the release of Solaris 7 back in 1998, Solaris has had the ability to run both 32-bit and 64-bit binaries on the same machine. Even now, 15 years later, with much of the world 64-bit only, there are still reasons to retain 32-bit support:

32-bit binaries can be faster in many cases, and if you do not need the additional address space afforded by the 64-bit version then there may be no advantage to running it.
Some software you depend upon may only provide a 32-bit version, or may have better compatability in that mode.
It’s sometimes nice to have a hard 4GB memory limit on a runaway process instead of it completely trashing your machine ;)

Unfortunately, while the base SmartOS platform is set up to provide both 32-bit and 64-bit binaries, the packaging infrastructure we use, pkgsrc, has not traditionally supported building multi-architecture packages. This has meant we have needed to provide both 32-bit and 64-bit versions of each dataset, which is not ideal:

It’s confusing to customers and users, who may think it applies to the kernel and platform version.
Users want the choice to be able to run 32-bit for some applications and 64-bit for others, all on the same machine.
It is additional work and maintenance for us.

In order to resolve this, I have been working on providing multi-architecture support to pkgsrc, and this work is now available for preview testing.

Getting started

Here is a quick start guide to getting the multi-architecture dataset up and running:

: Fetch the dataset image and manifest files.  The image is 85MB.
$ mkdir -p /usbkey/images
$ cd /usbkey/images
$ curl -O http://pkgsrc.smartos.org/datasets/multiarch-12.4.1.dsmanifest
$ curl -O http://pkgsrc.smartos.org/datasets/multiarch-12.4.1.zfs.bz2

: Import it
$ imgadm install -m multiarch-12.4.1.dsmanifest -f multiarch-12.4.1.zfs.bz2

: Create a new zone using the dataset (change your json to suit).
$ vmadm create <<EOF
{
  "brand": "joyent",
  "image_uuid": "ee1fb198-5fe1-11e2-9cce-e319fd47df7b",
  "max_physical_memory": 256,
  "alias": "multiarch-12.4.1",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}
EOF

then login and start using it as you would with any other dataset.

Multi-architecture libraries

Libraries are reasonably straight-forward. For most packages which provide shared libraries, you should find both 32-bit and 64-bit libraries are included, for example:

$ pkg_info -L sqlite3 | grep 'so$'
/opt/local/lib/libsqlite3.so
/opt/local/lib/amd64/libsqlite3.so

Other directories under lib/ should be handled correctly too, such as .pc files for pkg-config:

$ pkg_info -L sqlite3 | grep 'pc$'
/opt/local/lib/pkgconfig/sqlite3.pc
/opt/local/lib/amd64/pkgconfig/sqlite3.pc

Multi-architecture binaries

Binaries are similar, but follow a different layout scheme, and have additional controls to allow the user to select which architecture to use.

The basic layout is:

$ pkg_info -L sqlite3 | grep bin
/opt/local/bin/i86/sqlite3
/opt/local/bin/amd64/sqlite3
/opt/local/bin/sqlite3

In order to explain this, let us look at how the base platform supports multi-architecture binaries, using dtrace as an example:

$ ls -li /usr/sbin/*/dtrace /usr/sbin/dtrace /usr/lib/isaexec
4255 -r-xr-xr-x 72 root bin 12776 Dec 28 02:38 /usr/lib/isaexec
6330 -r-xr-xr-x  1 root bin 52728 Dec 28 02:38 /usr/sbin/amd64/dtrace
4255 -r-xr-xr-x 72 root bin 12776 Dec 28 02:38 /usr/sbin/dtrace
6640 -r-xr-xr-x  1 root bin 41544 Dec 28 02:38 /usr/sbin/i86/dtrace

$ file /usr/sbin/*/dtrace /usr/lib/isaexec
/usr/sbin/amd64/dtrace: ELF 64-bit LSB executable AMD64 Version 1, dynamically linked, not stripped, no debugging information available
/usr/sbin/i86/dtrace:   ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available
/usr/lib/isaexec:       ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available

The i86 and amd64 hold the per-architecture binaries, and the main /usr/sbin/dtrace command is a hardlink to the /usr/lib/isaexec wrapper (as shown by the inode being identical). This wrapper detects whether the running kernel is 32-bit or 64-bit, and calls the appropriate native binary, which nowadays will almost certainly be the 64-bit version.

For the pkgsrc implementation, I needed a way to override this behaviour so that users could select to run the 32-bit version if so desired, without having to munge their $PATH. To do this I took a copy of isaexec and added it to pkgsrc, with additional support for an ABI environment variable.

You can see the behaviour below with the calls to execve().

Default is to run 64-bit

$ truss -t execve sqlite3 -version
execve("/opt/local/bin/sqlite3", 0x08047DA8, 0x08047DB4)  argc = 2
execve("/opt/local/bin/amd64/sqlite3", 0x08047DA8, 0x08047DB4)  argc = 2
3.7.15 2012-12-12 13:36:53 cd0b37c52658bfdf992b1e3dc467bae1835a94ae

Set ABI=32 or ABI=i86 to run the 32-bit version

$ ABI=32 truss -t execve sqlite3 -version
execve("/opt/local/bin/sqlite3", 0x08047D9C, 0x08047DA8)  argc = 2
execve("/opt/local/bin/i86/sqlite3", 0x08047D9C, 0x08047DA8)  argc = 2
3.7.15 2012-12-12 13:36:53 cd0b37c52658bfdf992b1e3dc467bae1835a94ae

Note that not all binaries have been converted to multi-architecture. In fact, the majority have been left as plain 32-bit binaries. While all libraries ultimately have to be provided for both architectures so that users can choose to compile their own software against either, 64-bit binaries only make sense for certain classes of software:

Databases such as SQLite, and other servers which may require >4GB address space.
Language interpreters.
Software which provides a foo-config script with hardcoded references to libdir.

Compiler support

The GCC 4.7.2 package provided has been made aware of this layout, and will add the correct library paths depending upon the ABI you target. For example:

$ cat >test.c <<EOF
int main(){}
EOF

$ gcc -m32 test.c -o test32 -lsqlite3
$ ldd test32 | grep libsqlite3
        libsqlite3.so.0 =>       /opt/local/lib/libsqlite3.so.0

$ gcc -m64 test.c -o test64 -lsqlite3
$ ldd test64 | grep libsqlite3
        libsqlite3.so.0 =>       /opt/local/lib/amd64/libsqlite3.so.0

No additional flags or linker settings should be required (if they are, let me know!)

Interpreter support

For those interpreters which have been converted, their respective module systems should be multi-architecture aware:

Perl already had reasonable support for multi-architecture files, and all perl modules provided should be enabled, for example:

$ pkgin -y in p5-Digest-SHA1
$ pkg_info -L p5-Digest-SHA1 | grep 'so$'
/opt/local/lib/perl5/vendor_perl/5.16.0/i386-solaris-thread-multi/auto/Digest/SHA1/SHA1.so
/opt/local/lib/perl5/vendor_perl/5.16.0/x86_64-solaris-thread-multi-64/auto/Digest/SHA1/SHA1.so
$ ABI=32 perl -MDigest::SHA1 -e 'print'
$ ABI=64 perl -MDigest::SHA1 -e 'print'

Python is similar to Perl, but needed a lot more work to support multi-architecture modules. Modules provided by pkgin should work fine, but there may be issues with locally-built modules - let me know!
node is built as a multi-architecture binary, but npm has received no special handling. This means you need to be careful not to mix-and-match modules. I don’t perceive this to be too much of an issue, as the node community appears to have settled on having one node_modules per application, but again let me know if there are better ways to handle this.

Coverage

This work is incomplete, hence it not being available in the default datasets yet, but a large number of packages have been converted:

Pretty much every package which provides shared libraries.
Python, Perl, Lua and node.js interpreters (and a significant number of modules).
MySQL 5.5, PostgreSQL, SQLite and DB4 databases, Apache 2.2.

Notable exceptions currently are Ruby and PHP, and I will be working on those in due course. There may also be a number of packages which should include multi-architecture binaries, please let me know if I have missed any obvious candidates.

You can raise issues against our GitHub project here.

Enjoy!

Solaris portability - cfmakeraw()

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 09 Jan 2013 00:00:00 +0000

Converting from flock() to fcntl() unfortunately wasn’t enough to get tmux to build on Solaris, there was one additional failure:

client.c: In function 'client_main':
client.c:246:3: warning: implicit declaration of function 'cfmakeraw' [-Wimplicit-function-declaration]

...

Undefined                       first referenced
 symbol                             in file
cfmakeraw                           client.o
ld: fatal: symbol referencing errors. No output written to tmux
collect2: error: ld returned 1 exit status
*** [tmux] Error code 1

Let’s look at the code:

	struct termios tio;
...
	cfmakeraw(&tio);
	tio.c_iflag = ICRNL|IXANY;
	tio.c_oflag = OPOST|ONLCR;
...

Running man cfmakeraw on my OSX laptop I can read what this function does, and where it is implemented (edited for brevity):

SYNOPSIS
     void
     cfmakeraw(struct termios *termios_p);

DESCRIPTION
     The cfmakeraw() function sets the flags stored in the termios structure
     to a state disabling all input and output processing, giving a ``raw I/O
     path''.

STANDARDS
     The cfmakeraw() and cfsetspeed() functions, as well as the TCSASOFT option
     to the tcsetattr() function are extensions to the IEEE Std 1003.1-1988
     (``POSIX.1'') specification.

Ok, so all it’s doing is setting some additional flags in the tio structure, and a quick search in the NetBSD OpenGrok gives us their implementation (again edited for brevity):

void
cfmakeraw(struct termios *t)
{
...
	t->c_iflag &= ~(IMAXBEL|IGNBRK|BRKINT|PARMRK|ISTRIP|INLCR|IGNCR|ICRNL|IXON);
	t->c_oflag &= ~OPOST;
	t->c_lflag &= ~(ECHO|ECHONL|ICANON|ISIG|IEXTEN);
	t->c_cflag &= ~(CSIZE|PARENB);
	t->c_cflag |= CS8;
...
}

All that’s left is to patch the tmux code to set those flags. A good patch would add a feature test to the tmux autoconf setup and/or provide a compatability macro, but in this case I simply add a Solaris-specific section to the code:

	struct termios tio;
...
#ifdef __sun
	tio.c_iflag &= ~(IMAXBEL|IGNBRK|BRKINT|PARMRK|ISTRIP|INLCR|IGNCR|ICRNL|IXON);
	tio.c_oflag &= ~OPOST;
	tio.c_lflag &= ~(ECHO|ECHONL|ICANON|ISIG|IEXTEN);
	tio.c_cflag &= ~(CSIZE|PARENB);
	tio.c_cflag |= CS8;
#else
	cfmakeraw(&tio);
#endif
	tio.c_iflag = ICRNL|IXANY;
	tio.c_oflag = OPOST|ONLCR;
...

Summary

cfmakeraw() is an extension to POSIX and thus not fully portable. If you use it, please consider including a compatability version for systems which do not implement it.

Solaris portability - flock()

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 08 Jan 2013 00:00:00 +0000

This is the first of what I hope will be a regular series of posts looking at software portability when it comes to Solarish (Solaris, illumos, SmartOS etc.) systems. We will begin with flock(), as it’s what I happen to have been fixing today.

The most recent version of tmux fails with the following errors:

client.c: In function 'client_get_lock':
client.c:81:2: warning: implicit declaration of function 'flock' [-Wimplicit-function-declaration]
client.c:81:20: error: 'LOCK_EX' undeclared (first use in this function)
client.c:81:20: note: each undeclared identifier is reported only once for each function it appears in
client.c:81:28: error: 'LOCK_NB' undeclared (first use in this function)

from this section of code:

int
client_get_lock(char *lockfile)
{
        int lockfd;

        if ((lockfd = open(lockfile, O_WRONLY|O_CREAT, 0600)) == -1)
                fatal("open failed");

        if (flock(lockfd, LOCK_EX|LOCK_NB) == -1 && errno == EWOULDBLOCK) {
                while (flock(lockfd, LOCK_EX) == -1 && errno == EINTR)
                        /* nothing */;
                close(lockfd);
                return (-1);
        }

        return (lockfd);
}

This code is reasonably straight-forward, trying to create a lock on a file descriptor, and if it isn’t able to waits until the lock is released then fails, ready for the calling function to retry.

However, the flock() routine is of BSD heritage and does not exist on Solaris. There used to be a compatability version as part of the /usr/ucb environment, but that no longer exists in newer versions.

Thankfully, there is an alternative interface which has existed for as long as flock() but with the added benefit of being standardised by POSIX and thus much more portable - enter fcntl()!

Let’s look at how the two are similar-but-different, and then show how we can change this code to be more portable.

flock() implementation

As we can see from the NetBSD manual page below, flock() is very simple, and this is probably the main reason why people choose to use it over fcntl() for file locking.

NAME
     flock -- apply or remove an advisory lock on an open file

LIBRARY
     Standard C Library (libc, -lc)

SYNOPSIS
     #include <fcntl.h>

     #define   LOCK_SH   1    /* shared lock */
     #define   LOCK_EX   2    /* exclusive lock */
     #define   LOCK_NB   4    /* don't block when locking */
     #define   LOCK_UN   8    /* unlock */

     int
     flock(int fd, int operation);

There isn’t much to it, you call flock() specifying either a shared or exclusive lock, with an option to make the call non-blocking. Afterwards you unlock the previously held lock.

The tmux code in question uses LOCK_NB for the initial call, then if that fails reverts to the default blocking operation so that the client can retry as soon as the lock is released.

Use fcntl() instead!

In contrast to flock(), setting up fcntl() is a bit more involved as it allows finer-grained control over the locking.

Set up struct flock

There is a flock structure which controls the lock, defined as:

struct flock {
        off_t       l_start;    /* starting offset */
        off_t       l_len;      /* len = 0 means until end of file */
        pid_t       l_pid;      /* lock owner */
        short       l_type;     /* lock type: read/write, etc. */
        short       l_whence;   /* type of l_start */
};

So, let’s set that up:

int
client_get_lock(char *lockfile)
{
	int lockfd;
	struct flock lock;

	lock.l_start = 0;
	lock.l_len = 0;
	lock.l_type = F_WRLCK;
	lock.l_whence = SEEK_SET;
...

Setting l_start and l_len both to 0 means ‘lock the entire file’, l_whence of SEEK_SET means we are setting absolute values rather than relative, and we set l_type to be a write lock. If we wanted a read lock, we’d use F_RDLCK here instead.

Note that l_pid is only used for a F_GETLK operation - we are only interested in attempting to set a lock, so it is left unset.

Add fcntl() calls

The synopsis for fcntl() is:

SYNOPSIS
     #include <fcntl.h>

     int
     fcntl(int fd, int cmd, ...);

and the commands available for file locking are (from BSD):

#define F_GETLK         7               /* get record locking information */
#define F_SETLK         8               /* set record locking information */
#define F_SETLKW        9               /* F_SETLK; wait if blocked */

Note there is no ‘clear lock’ command. To clear a lock, you use the F_GETLK command with l_type set to F_UNLCK.

So, to show how we would rewrite the flock() instances to fcntl() instead, I’ve put them together below:

#ifdef __sun
	if (fcntl(lockfd, F_SETLK, &lock) == -1 && errno == EAGAIN) {
		while (fcntl(lockfd, F_SETLKW, &lock) == -1 && errno == EINTR)
			/* nothing */;
		close(lockfd);
		return(-1);
	}
#else
	if (flock(lockfd, LOCK_EX|LOCK_NB) == -1 && errno == EWOULDBLOCK) {
		while (flock(lockfd, LOCK_EX) == -1 && errno == EINTR)
			/* nothing */;
		close(lockfd);
		return (-1);
	}
#endif

As you can see, they are very similar, making it relatively straight-forward to rewrite code to use the more portable fcntl().

An important note on semantics

There are two important difference between flock() and fcntl() you need to be aware of which may affect a simple conversion:

fcntl() locks are not held across a fork(), so you cannot pass locks down to child processes.
The semantics of fcntl() are such that any closure of a file descriptor in your application will release the locks held against that file. This is best illustrated with a lock on /etc/passwd that gets released if you call getpwname() as that opens and closes the /etc/passwd file.

Generally such semantics do not apply, but you should be aware of them, and it always pays to carefully read the manual pages, preferably those from BSD.

flock() -> fcntl() cheat sheet

To aid your own conversions, here are some further examples (without error checking for clarity) of flock() and fcntl() equivalents.

#if defined(USING_FLOCK)

	/* Blocking */
	flock(lockfd, LOCK_SH);  // Shared (read) lock
	flock(lockfd, LOCK_EX);  // Exclusive (write) lock

	/* Non-blocking */
	flock(lockfd, LOCK_SH|LOCK_NB);  // Shared (read) lock
	flock(lockfd, LOCK_EX|LOCK_NB);  // Exclusive (write) lock

	/* Release */
	flock(lockfd, LOCK_UN):

#elif defined(USING_FCNTL)

	struct flock lock;

	lock.l_start = 0;
	lock.l_len = 0;
	lock.l_whence = SEEK_SET;

	/* Blocking */
	lock.l_type = F_RDLCK;
	fcntl(lockfd, F_SETLKW, &lock);  // Shared (read) lock
	lock.l_type = F_WRLCK;
	fcntl(lockfd, F_SETLKW, &lock);  // Exclusive (write) lock

	/* Non-blocking */
	lock.l_type = F_RDLCK;
	fcntl(lockfd, F_SETLK, &lock);  // Shared (read) lock
	lock.l_type = F_WRLCK;
	fcntl(lockfd, F_SETLK, &lock);  // Exclusive (write) lock

	/* Release */
	lock.l_type = F_UNLCK;
	fcntl(lockfd, F_GETLK, &lock);

#endif

flock() wrapper

Alternatively, there is a flock() wrapper which is used by the NetBSD toolchain, which you could include in a compatability library or so:

int flock(int fd, int op) {
	int rc = 0;

#if defined(F_SETLK) && defined(F_SETLKW)
	struct flock fl = {0};

	switch (op & (LOCK_EX|LOCK_SH|LOCK_UN)) {
	case LOCK_EX:
		fl.l_type = F_WRLCK;
		break;

	case LOCK_SH:
		fl.l_type = F_RDLCK;
		break;

	case LOCK_UN:
		fl.l_type = F_UNLCK;
		break;

	default:
		errno = EINVAL;
		return -1;
	}

	fl.l_whence = SEEK_SET;
	rc = fcntl(fd, op & LOCK_NB ? F_SETLK : F_SETLKW, &fl);

	if (rc && (errno == EAGAIN))
		errno = EWOULDBLOCK;
#endif

	return rc;
}

It might be handy if the illumos folks merged this, it would immediate fix at least 19 packages in the pkgsrc collection :)

Summary

flock() is simpler, and retains locks across fork() and concurrent access boundaries, but at the cost of portability. Please try to use fcntl() where possible, it isn’t much harder to use, and makes your software run on more platforms.

pkgsrc-2012Q4 illumos packages now available

jonathan@perkin.org.uk (Jonathan Perkin) — Sun, 06 Jan 2013 00:00:00 +0000

Note

This page is obsolete. Please see http://pkgsrc.joyent.com/ where you will find the latest package sets and improved instructions.

Original Article

In keeping with the regular quarterly pkgsrc releases, I’m pleased to announce that packages from the pkgsrc-2012Q4 branch are now available for general illumos platforms.

As usual, the quick start instructions are:

$ curl -s http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2012Q4-illumos.tar.gz \
    | gtar -zxf - -C /
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
$ pkgin -y up
$ pkgin avail | wc -l
    9518
$ pkgin search <package>
$ pkgin -y install <package> <package...>

For those interested in helping to increase the number of packages available, the bulk build report for this set is available here. It would be great to get past the 10,000 mark!

SmartOS and the global zone

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 23 Nov 2012 00:00:00 +0000

One of the most common issues new users of SmartOS face is understanding the role and design of the global zone. Often they will download SmartOS and try to start using it as they would any other Unix operating system, but quickly run into basic problems such as installing packages or adding users. However, SmartOS is not your usual operating system, and it is imperative that you understand two key principles:

SmartOS is specifically designed as an OS for running Virtual Machines, not as a general purpose OS.
The global zone is effectively a read-only hypervisor, and should only be used for creating and managing Virtual Machines. Everything else should be performed inside Virtual Machines.

I should be clear at this point that I am specifically addressing the issue of downloading and running your own SmartOS installation. If you provision a SmartMachine from Joyent, then you are running inside a Virtual Machine, and this post does not apply - you are free to start serving up awesome applications!

Let’s look at a few aspects of the global zone which make it a great fit for its intended purpose, and help explain why it doesn’t work as you might expect.

The global zone is a ramdisk

This is the key reason why things don’t work as you might expect. SmartOS does not install to disk like other operating systems, instead it boots directly from USB/CD/PXE into a mostly read-only environment.

Why is it done this way? There are a number of good reasons:

Upgrades are trivial. No more patching, just reboot into a new image!
Increased disk space. No wasted space on disk to hold the OS, all the space is dedicated to VMs and user data.
Increased disk performance. It’s common with other systems to have your OS installed to a pair of mirrored disks and then pool the remaining disks for data. With SmartOS you can have all your disks in the same RAIDZ pool, increasing performance.
Additional security. Most of the system files are read-only, and /etc is re-created on each boot, making it much harder to exploit.
Increased stability. Ever had your root disks start to fail and system commands no longer run? This doesn’t happen on SmartOS.
Much simpler to install and provision, especially when you have a large number of machines.

So, what does this look like, and what are the implications?

# zonename
global
# df -h / /usr
Filesystem             size   used  avail capacity  Mounted on
/devices/ramdisk:a     264M   219M    45M    83%    /
/devices/pseudo/lofi@0:1
                       376M   354M    22M    95%    /usr
# lofiadm
Block Device             File                           Options
/dev/lofi/1              /usr.lgz                       Compressed(gzip)

The root file system is a ramdisk, and /usr is a read-only loopback mount of a single compressed file held on the ramdisk.

Apart from a few specific files and directories listed below, this means that you cannot change the global zone. To emphasise this point further:

You cannot add users.
You cannot write anywhere under /usr.
You cannot permanently store or change files under /etc, /root, ..
Changes to SMF services will be reset each reboot.

There are plenty of other things you cannot do, but hopefully this gives you some idea of the restricted nature of the global zone environment.

If you are looking for a general purpose OS to serve as a NAS etc, then you can do worse than give OmniOS a spin - it has most of the same features as SmartOS, but in a normal install-to-disk configuration.

Having said all that, let’s look at some ways you can manage the global zone, if you have settled on running SmartOS.

So what can I do?

Firstly, let’s look at the key writeable areas:

Filesystem             size   used  avail capacity  Mounted on
zones                  2.0T   143M   1.8T     1%    /zones
zones/var              2.0T   4.0G   1.8T     1%    /var
zones/opt              2.0T    55K   1.8T     1%    /opt
zones/usbkey           2.0T   154K   1.8T     1%    /usbkey
/usbkey/shadow         1.8T   154K   1.8T     1%    /etc/shadow
/usbkey/ssh            1.8T   154K   1.8T     1%    /etc/ssh

zones is the big zpool which is spread across all your disks, and SmartOS will create a number of file systems on it for use in the global zone:

/zones

This is where your Virtual Machines are stored, the datasets used to create them, and some other key files related to zones. Don’t mess around in here unless you know what you are doing.

/var

We all like logs, so those are retained in /var as normal, as well as various state files such as the current list of imgadm datasets and the SSH host keys.

/opt

/opt is your main hook into more advanced setup of the global zone. SmartOS will import any SMF manifests it finds in /opt/custom/smf, which allows you to implement rc.local functionality as demonstrated in this gist.

As /opt is writeable you can also install packages as per this wiki page, however you need to bear in mind that, as explained earlier, things such as adding users are not possible, so you may see various errors.

/usbkey

When you first boot SmartOS and go through the rudimentary installer the details you provide are stored in /usbkey/config, which is used during boot to configure the machine. If you need to change any of those variables, this is the file you should edit.

There are also some variables which aren’t set up by default, so if you want to

Install an authorized_keys file for the root user.
Set a keyboard map.

then have a read of this post I wrote a while ago to find out how to configure those.

/etc/shadow and /etc/ssh

In order to support some very basic configuration, a few /etc files are their own mount points onto /usbkey so that changes to them are saved.

/etc/shadow so that you can change the root password.
/etc/ssh is initially where the SSH host keys were stored, however they are now stored under /var/ssh so don’t be surprised if this mount point disappears at some point.

Implementation

While we’re on the subject, let’s complete this post with a look at how the global zone is implemented, for interested readers.

GRUB

There is a good wiki guide which describes how to mount the USB key. Once you have done that you can take a look at the GRUB configuration:

: Assumes your current working directory is the USB key mountpoint.
# less boot/grub/menu.lst
...
title Live 64-bit (text)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='<crypt>',smartos=true
   module /platform/i86pc/amd64/boot_archive

title Live 64-bit (noinstall)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='<crypt>',standalone=true,noimport=true
   module /platform/i86pc/amd64/boot_archive
...

There are a few boot configuration variables which alter how the system is started, and you can see how they are used by grepping for them in the SMF init scripts under /lib/svc.

root_shadow=’crypt string’

This configures the default root password prior to /etc/shadow being mounted, and changes for each release. If you need to know what it is, then browse the download site and look at the SINGLE_USER_ROOT_PASSWORD.release.txt which correlates with your uname -v output, or if you wish you could generate your own password and paste the new crypted string in here.

Alternatively as a convenience you can find the default root password inside the platform directory:

# cat platform/root.password
seitee3oome4aiPh

smartos=true

This variable defines whether to perform the normal SmartOS global zone initialisation, such as mounting /usbkey and configuring the system from the /usbkey/config file (see /lib/svc/method/fs-joyent and /lib/svc/method/smartdc-config.

If left unset (like in the ‘Live 64-bit (noinstall)’ boot option), these will not be performed. standalone=true should be set in those cases to avoid trying to be a Joyent compute node.

noimport=true

Setting this will skip any configuration and mounting of zpools, and is useful for if you have issues during the first installation and need to check disks, etc.

It’s likely that you do not need to change any of these variables, and if you hit problems simply use the noinstall boot option and figure things out from there.

Platform image

The only other files on the USB key are:

# find platform -type f
platform/i86pc/kernel/amd64/unix
platform/i86pc/amd64/boot_archive
platform/i86pc/amd64/boot_archive.gitstatus
platform/i86pc/amd64/boot_archive.manifest

These are mostly self-explanatory:

unix is the SmartOS kernel.
boot_archive is the ramdisk image containing the entire OS.
boot_archive.gitstatus contains the tip revisions of the github repositories used to build that particular image.
boot_archive.manifest contains MD5 checksums of the OS files.

Upgrades

Given we have just explained about the platform directory, it is also worth pointing out that you do not have to reflash your USB key every time, which is great if your server is inaccessible. Instead, assuming your USB key is big enough, you can simply download the newest platform-latest.tgz file, move the existing platform directory out of the way, and unpack platform-latest.tgz there instead. Doing this as an atomic operation is preferred, for example:

: As of writing this is 20121115T191935Z
# gtar zxf ~/platform-latest.tgz
# mv platform platform-$(uname -v) && mv platform-20121115T191935Z platform

This avoids having no platform directory, just in case you have a power cut at that exact point in time!

Summary

If you want to set up a file server or similar, then SmartOS is probably not for you. However, if you are interested in running Virtual Machines and are able to do all of your work inside them, then SmartOS is perfect for that purpose, and has a number of advantages over other operating systems.

Hopefully this has been useful, even if it is to deter you from using SmartOS!

Setting up Samba on SmartOS

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 24 Oct 2012 00:00:00 +0100

A frequent question on #smartos IRC is how to set up SmartOS as a file server. Due to the architecture of SmartOS and its focus on virtualisation, this isn’t as easy as on other systems, and some parts are simply not supported at this time (e.g. NFS).

This guide, therefore, is a simple way to get Samba up and running so that you can at least use a SmartOS machine as a file server, even if it’s perhaps not the protocol you would initially choose.

Create new virtual machine

Let’s start by creating a VM, here is what I use locally, you may want to tweak the settings for your environment. If you already have a suitable VM configured you can just skip this step.

# Import the base1.8.1 image, though any base image will suffice.
$ imgadm update
$ imgadm import 55330ab4-066f-11e2-bd0f-434f2462fada

# I store all my VM configs in this directory
$ mkdir /usbkey/vmcfg

# vmadm configuration for 'store' VM
$ vi /usbkey/vmcfg/store.json

{
  "brand": "joyent",
  "zfs_io_priority": 30,
  "image_uuid": "55330ab4-066f-11e2-bd0f-434f2462fada",
  "max_physical_memory": 256,
  "quota": 750,
  "alias": "store",
  "hostname": "store",
  "resolvers": [
    "193.178.223.141",
    "208.72.84.24"
  ],
  "dns_domain": "adsl.perkin.org.uk",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.1.11",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.1"
    }
  ]
}

# Create the VM
$ vmadm create -f /usbkey/vmcfg/store.json

Install Samba package

Next, log in to the VM and install the required packages. These are not currently available from the default repository, so we will use the generic illumos package set:

$ zlogin <uuid>

# Download and unpack bootstrap kit
$ curl -s http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2012Q3-illumos.tar.gz \
    | gtar -zxf - -C /
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH

# Install latest Samba package (others are available if necessary)
$ pkgin -y up
$ pkgin -y install samba-3.6

Configure Samba

This part will differ based on your requirements, here are a couple of simple examples.

Shared guest mount with full read/write access

For if you just want somewhere to share stuff on a local network with no security.

# Create shared mount user and mountpoint.
$ groupadd -g 500 store
$ useradd -u 500 -g 500 -c "Store user" -s /usr/bin/false -d /store store
$ mkdir /store
$ chown store:store /store

# Configure Samba
$ vi /etc/opt/pkg/samba/smb.conf

[global]
  security = share
  load printers = no
  guest account = store

; Comment out [homes] section

[store]
  path = /store
  public = yes
  only guest = yes
  writable = yes
  printable = no

Shared mount with per-user access

For a bit more fine-grained control.

# Create per-user accounts and mountpoint.
$ groupadd -g 1000 alice
$ useradd -u 1000 -g 1000 -c "Alice" -s /usr/bin/false -d /store alice
$ groupadd -g 1001 bob
$ useradd -u 1001 -g 1001 -c "Bob" -s /usr/bin/false -d /store bob
$ mkdir /store
$ chmod 1777 /store
$ /opt/pkg/bin/smbpasswd -a alice
$ /opt/pkg/bin/smbpasswd -a bob

# Configure Samba
$ vi /etc/opt/pkg/samba/smb.conf

[global]
  security = user
  load printers = no

; Comment out [homes] section

[store]
  path = /store
  valid users = alice bob
  public = no
  writable = yes
  printable = no

Access will be via username and the password set with smbpasswd. Users will be able to create files and read other user files, but will only be able to modify files they created.

Startup scripts

Okay, so I’m lazy and whilst I should provide some SMF scripts and manifests for this, it’s simpler to just write an rc.d script! :-)

$ vi /etc/rc2.d/S99samba

#!/bin/sh

case "$1" in
start)
	# Start up Samba daemons
	/opt/pkg/sbin/nmbd -D
	/opt/pkg/sbin/smbd -D
	;;
stop)
	pkill -9 nmbd
	pkill -9 smbd
	;;
reload)
	pkill -HUP nmbd
	pkill -HUP smbd
	;;
esac

Finally, start it up!

$ chmod +x /etc/rc2.d/S99samba
$ /etc/rc2.d/S99samba start

Multicast DNS

In order for shares to automatically show up in e.g. the OSX Finder, you will need to be running some kind of mDNS service on the server.

The easiest solution is to simply enable dns/multicast in the zone, i.e.:

$ svcadm enable dns/multicast

This will then show up based on the hostname of the server, and clicking on it should show the store mount we created.

All done

I hope this proves useful!

pkgsrc-2012Q3 packages for illumos

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 10 Oct 2012 00:00:00 +0100

Note

This page is obsolete. Please see http://pkgsrc.joyent.com/ where you will find the latest package sets and improved instructions.

Original Article

Continuing from my previous bulk build for pkgsrc-2012Q2 I’m pleased to announce that packages from the pkgsrc-2012Q3 branch are now available.

Quick start:

$ curl -s http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2012Q3-illumos.tar.gz \
    | gtar -zxf - -C /
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
$ pkgin -y up
$ pkgin avail | wc -l
    9542
$ pkgin search <package>
$ pkgin -y install <package> <package...>

As usual there have been many hundreds of updates since the previous branch, and hopefully the software you need is included. If not, please get involved! We welcome new contributors.

The bootstrap kit and a number of packages have been tested successfully on OpenIndiana 151a and OmniOS.

Enjoy!

Creating local SmartOS packages

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 23 Aug 2012 00:00:00 +0100

Ok, so you have a SmartOS machine set up, and are using pkgin to install packages. Now you may want to also handle your local software using the same tools, so that you can take advantage of easily installing and upgrading packages. This post describes how you can do this.

We will be creating a hypothetical package meminfo, containing a script of the same name which prints basic memory information, and just enough metadata for it to be understood by the packaging tools. This guide assumes you are running a standard dataset which includes packaging tools in /opt/local/sbin, that is:

$ /opt/local/sbin/pkg_info

should give you a list of currently installed packages.

Files to install

First, let’s start in a clean directory, and then create a files sub-directory which will hold the files we want to package. The files in this directory will be installed relative to the packaging prefix, which for most SmartOS installs will be /opt/local.

$ mkdir package
$ cd package
$ mkdir -p files/bin

Now we create the meminfo script. You will probably just copy your binaries into place at this stage.

$ vi files/bin/meminfo

#!/usr/bin/bash
#
# This is just a subset of 'sm-summary' from the 'smtools' package.
#
SM_UUID=$(zonename);
SM_ID=$(zoneadm list -p | awk -F: '{ print $1 }');
SM_MEMCAP=$(kstat -p -c zone_caps -n lockedmem_zone_${SM_ID} -s value | awk '{print $2/1024/1024 }');
SM_MEMUSED=$(prstat -Z -s rss 1 1 | awk -v zone=${SM_UUID} '$8 ~ zone { printf("%d", $4 ) }');
SM_MEMFREE=$(echo "${SM_MEMCAP}-${SM_MEMUSED}" | bc);
cat <<EOF
Memory (RSS) Cap    ${SM_MEMCAP}M
Memory (RSS) Used   ${SM_MEMUSED}M
Memory (RSS) Free   ${SM_MEMFREE}M
EOF

Ensure the script is executable!

$ chmod +x files/bin/meminfo

Finally for this part, generate a packlist file which simply contains a list of files we want packaged, relative to the files directory:

$ (cd files; find * -type f -or -type l | sort) >packlist

The packlist file supports many directives which are lines beginning with @<cmd>. These let you do things such as change file permissions once the file has been installed. See the ‘PACKING LIST DETAILS’ section of pkg_create(1) for more information.

Package metadata

In order to successfully create a package, we need a few metadata files which describe the package.

build-info

This file contains basic information about the package, and is primarily used to ensure that the package can be installed on the target machine. The minimum information required is:

MACHINE_ARCH. On SmartOS this is either i386 or x86_64 depending upon whether you chose a base/smartos or base64/smartos64 dataset. Attempting to install a package intended for one architecture with package tools built for a different architecture will work due to SmartOS being able to run both 32-bit and 64-bit applications, but you will get warnings when installing.
OPSYS. On SmartOS this is SunOS, i.e. the output of uname -s.
OS_VERSION. On SmartOS this is 5.11, i.e. the output of uname -r.
PKGTOOLS_VERSION. An integer describing the version of the pkg_install tools required to understand this package. At the current time pkgsrc sets a base version of 20091115 so just use this.

The easiest way to generate this file is simply to get it from an existing package, for example:

$ pkg_info -X pkg_install \
  | egrep '^(MACHINE_ARCH|OPSYS|OS_VERSION|PKGTOOLS_VERSION)' >build-info
$ cat build-info
MACHINE_ARCH=i386
OPSYS=SunOS
OS_VERSION=5.11
PKGTOOLS_VERSION=20091115

comment

This is a short string describing the package which is shown in the default pkg_info or pkgin list output. Keep this short, preferably under 60 characters, so that it displays correctly in 80-column terminals.

$ echo "Show basic memory information on SmartOS" >comment

description

This is a longer multi-line description of the package which is output by pkg_info <package name> or pkgin pkg-descr <package name>. Format this to 80-columns.

$ cat >description <<EOF
meminfo prints basic information about available memory on a SmartOS machine,
listing the total memory size, used, and free.  The information provided
should be understood in light of how memory is used and reported on SmartOS,
for more information refer to:

http://wiki.smartos.org/display/DOC/About+Memory+Usage+and+Capping
EOF

So we should now have:

build-info          # basic package information
comment             # one-line comment
description         # multi-line comment
files/bin/meminfo   # the file we wish to package
packlist            # package listing

Create the package

We now have enough information to create a basic package. The magic invocation to perform the operation is:

$ pkg_create -B build-info -c comment -d description -f packlist \
  -I /opt/local -p files -U meminfo-1.0.tgz

The -B, -c, -d and -f arguments simply pull in the metadata files we’ve written.

The -I argument specifies the destination prefix. As we are creating a package outside of this prefix we need to tell pkg_create of the ultimate destination.

The -p argument is used in conjunction with -I to tell pkg_create where our files to be packaged can be found.

The -U argument means we should just create the package, and not register it.

And finally, we provide the package file. The packaging tools understand version numbers, so we can just provide the file name and it will determine the package name and version from that.

To verify the package, you can run:

$ pkg_info -X meminfo-1.0.tgz

PKGNAME=meminfo-1.0
COMMENT=Show basic memory information on SmartOS
MACHINE_ARCH=i386
OPSYS=SunOS
OS_VERSION=5.11
PKGTOOLS_VERSION=20091115
FILE_NAME=meminfo-1.0.tgz
FILE_SIZE=922
DESCRIPTION=meminfo prints basic information about available memory on a SmartOS machine,
DESCRIPTION=listing the total memory size, used, and free.  The information provided
DESCRIPTION=should be understood in light of how memory is used and reported on SmartOS,
DESCRIPTION=for more information refer to:
DESCRIPTION=
DESCRIPTION=http://wiki.smartos.org/display/DOC/About+Memory+Usage+and+Capping

Installing

For a single machine, you can now simply install the package with:

$ pkg_add meminfo-1.0.tgz

If you want to publish the package so that it is available to pkgin you need to create a pkg_summary file and put it alongside the package for download. This file is usually compressed to speed things up, and pkgin supports both .gz and .bz2.

$ pkg_info -X meminfo-1.0.tgz | gzip -9 >pkg_summary.gz

For example purposes I will simply use file://, but of course you can use http:// instead if you put the packages and pkg_summary file somewhere accessible:

$ mkdir /var/tmp/packages
$ cp meminfo-1.0.tgz pkg_summary.gz /var/tmp/packages

Add this repository to pkgin and reload:

$ echo "file:///var/tmp/packages" >>/opt/local/etc/pkgin/repositories.conf
$ sudo pkgin -fy up

Now the package should show up:

$ pkgin avail | grep meminfo
meminfo-1.0          Show basic memory information on SmartOS

and be installable:

$ sudo pkgin -y install meminfo
$ /opt/local/bin/meminfo
Memory (RSS) Cap    8192M
Memory (RSS) Used   92M
Memory (RSS) Free   8100M

This covers the basics and should be enough to get started. As mentioned, there are many other options available, and I suggest that if you need additional functionality you take a look at the pkg_create manual page, or simply use pkg_info -X on more complicated packages to see what they do.

7,000 binary packages for OSX Lion

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 10 Jul 2012 00:00:00 +0100

Note

This page is obsolete. Please see http://pkgsrc.joyent.com/ where you will find the latest package sets and improved instructions.

Original Article

For my day job I build packages for our SmartOS operating system using pkgsrc. pkgsrc is a cross-platform package manager which began life as the FreeBSD ports system and was ported to NetBSD back in 1997. Since then it has been ported to many other different systems, 19 at the current count, one of which is OSX/Darwin.

For the recent pkgsrc-2012Q2 release, I have performed a full bulk build on OSX Lion, and the result is 7,374 binary packages for you to use as an alternative to brew/macports/etc.

To use them, run the following:

$ curl http://pkgsrc.smartos.org/packages/Darwin/2012Q2/bootstrap.tar.gz \
    | (cd /; sudo gnutar -zxpf -)
$ PATH=/usr/pkg/sbin:/usr/pkg/bin:$PATH
$ sudo pkgin -y update
$ pkgin avail | wc -l
    7374
$ pkgin search ...
$ sudo pkgin -y install ...

pkgin is a tool similar to apt-get and allows you to easily search for and install/upgrade packages.

Please let me know if you find them useful, and if you have any feedback.

Enjoy!

9,000 packages for SmartOS and illumos

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 09 Jul 2012 00:00:00 +0100

Note

This page is obsolete. Please see http://pkgsrc.joyent.com/ where you will find the latest package sets and improved instructions.

Original Article

At Joyent, we provide SmartOS users with a set of regularly updated binary packages which are tuned to work well on our operating system. The package manager we use, pkgsrc, does not yet come with native support for e.g. SMF, and so we provide a limited set of packages, ensuring that those we do we provide include such features.

However, often users will want packages which are not yet included in our build set, and so to assist those users we have performed a full bulk build of the unmodified pkgsrc-2012Q2 release, which resulted in well over 9,000 binary packages to choose from.

In addition, thanks to binary compatibility, these packages can also be used on your illumos distribution of choice (please let me know if this is not the case!) if your native package manager does not provide the software you want.

Note that these packages do not come with SMF support, and may be different from our official packages in terms of build options, etc. Also, to avoid conflict with our official packages, these have been built under a different prefix:

/opt/pkg	# main prefix
/etc/opt/pkg	# configuration files
/var/opt/pkg	# transient files

compared to our official packages:

/opt/local	# main prefix
/opt/local/etc	# configuration files
/var		# transient files

To use them, run the following:

# For SmartOS use the original build.
$ curl http://pkgsrc.smartos.org/packages/SmartOS/2012Q2/bootstrap.tar.gz \
    | gzip -dc | (cd /; sudo tar -xpf -)

# For other illumos, use the updated build which should work, but has fewer
# packages, and you will need to ignore warnings about x86_64 != i386.
$ curl http://pkgsrc.smartos.org/packages/SmartOS/2012Q2j1/bootstrap.tar.gz \
    | gzip -dc | (cd /; sudo tar -xpf -)

$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
$ sudo pkgin -y update
$ pkgin avail | wc -l
    9401
$ pkgin search ...
$ sudo pkgin -y install ...

What’s included? Here are just a few highlights:

Developer tools

PHP: 5.4.4, 5.3.14 + ~100 modules for each
Perl: 5.14.2 + 1,780 modules
Python: 3.2.3, 3.1.5, 2.7.3, 2.6.8, 2.5.6 + ~260 modules for 2.x
Ruby: 1.9.3p194, 1.9.2pl320, 1.8.7.358 + ~450 modules for each
Editors: (vim 7.2, emacs 23.4 and 24.1, xemacs 21.4)
Revision control: (git 1.7.10.5, mercurial 2.2.2, bzr 2.5, svn 1.6.17)

Web stack

Apache: 2.4.2, 2.2.22, 2.0.64, 1.3.42 + modules
MySQL: 5.5.25, 5.1.63, 5.0.96
PostgreSQL: 9.1.3, 9.0.7, 8.3.18
Wordpress 3.4.1

Desktop

GNOME 2.32.1 (not complete)
KDE 4.8.4 (not complete)
XFCE 4.6.1
evilwm 1.1.0

plus thousands of miscellaneous utilities.

The long-term goal of course is to integrate all of our changes back into the upstream pkgsrc repository and eventually provide a single package repository, and we will continue to work on that endeavour.

Enjoy!

Goodbye Oracle, Hello Joyent!

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 07 May 2012 00:00:00 +0100

Back in 2001 when I started working for the BBC I was given a Sun Ultra 10 workstation running Solaris 8 and CDE to do my job. As a Linux/FreeBSD user at the time, and someone accustomed to things such as a working vi implementation, recursive grep and Window Maker, this came as something of a culture shock, and it wasn’t long before I was compiling my own software in a rapid attempt to build and configure a more usable desktop environment - or at least one I was more comfortable in.

Building all that software by hand quickly became tedious, and so I was delighted to find that Christos Zoulas had written a bunch of portability glue, known as Zoularis, which allowed pkgsrc (similar to the FreeBSD ports system I was used to) to be used on Solaris.

It was still early days, however, and so over the next few years I and others helped to improve Solaris support in pkgsrc. By 2004 we had over 1500 packages building on Solaris 9/SPARC using the Sun Studio compiler, and in 2010 there were 5000 packages for Solaris 10/x86, along with a native SFW package including pkgin (an “apt-get” clone), making it easy to get up and running.

It was always something of a disappointment to me though that pkgsrc on Solaris wasn’t more widely deployed as I felt it provided major benefits over Blastwave or IPS. So it was great to see Filip Hajný of Joyent join the pkgsrc developers in 2009 for his work on Solaris with Joyent using pkgsrc internally.

Since then they have:

hired a number of fantastic engineers from the ashes of Sun Microsystems, who have
ported KVM to their SmartOS descendant of Solaris, making it a clear differentiator in the cloud computing arena (DTrace+ZFS+Zones+KVM is the perfect combination for multi-tenancy environments), as well as
taken stewardship of node.js and again hired the most prominent engineers involved in that project

and thus over that time my desire to work there has only increased.

So, I am obviously delighted to say that today is my first day working for Joyent, focusing on making pkgsrc work even better on SmartOS.

It isn’t just the people and the technology which attracted me, though. A recent Twitter post by ex-Sun CEO and co-founder Scott McNealy says it well:

As a long time fan of Sun (I got over the initial shock of Solaris) who was fortunate enough to work there, and as someone who experienced the fantastic company culture of MySQL AB, it is this chance to again join an engineering-driven company that “kicks butt, has fun, doesn’t cheat, loves its customers, changes computing forever” which excites me the most.

Let’s just hope Joyent don’t turn out exactly the same as Sun Microsystems ;)

SmartOS global zone tweaks

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 13 Apr 2012 00:00:00 +0100

The basic premise behind SmartOS is to provide a netbooted global zone, and then do actual work inside KVM or zone-based virtual machines. This has a number of advantages, not least that it’s trivial to upgrade to a newer release - you simply boot a newer image.

However, the read-only nature of the global zone means that if you want to make changes to the global zone, then they need to be read from permanent storage, rather than making changes as you would normally.

SmartOS creates a /usbkey file system on stable storage which holds configuration for the global zone, and an init script /lib/svc/method/smartdc-config which sources the file /usbkey/config for configuration details. By default this will mostly include details you provided at install time, however there are additional values you can set to customise the global zone further.

Here are a couple that I have found, and need for my personal use.

Upload a root authorized_keys file

The root user’s home directory is provided from the ramdisk, so any changes you make will be wiped out on the next boot. This is presumably so that newer images can make changes to the shell rc files, for example to update $PATH without worrying about all users having to merge these changes.

However, I’m not one for typing in passwords all day, so in order to store a /root/.ssh/authorized_keys file you can do this:

mkdir -p /usbkey/config.inc
# paste your key into this file
vi /usbkey/config.inc/authorized_keys
echo "root_authorized_keys_file=authorized_keys" >>/usbkey/config

The root_authorized_keys_file variable points to a file in /usbkey/config.inc, you can of course change the name if you wish.

Set a keyboard map

By default a US keymap will be loaded for the console, if you want to use a different one then find a suitable layout in /usr/share/lib/keytables/type_6 (in my case uk) and then:

echo "default_keymap=uk" >>/usbkey/config

Look for more tweaks

As of time of writing (joyent_20120405T204624Z), these two appear to be the only tweaks available which aren’t already used by default, however this may change in the future. To see if there are any newer ones available, you can:

grep CONFIG_ /lib/svc/method/smartdc-config

and then have a deeper look into the script to see how they are used ;-)

Run ad-hoc scripts

Note that while you may be tempted to think “aha, it’s just a shell script, I can use it like rc.local”, you can’t - it’s explicitly parsed into variables, and trying to put commands in will just break the init script.

However, the /lib/svc/method/manifest-import script does import any SMF manifests it finds in /opt/custom/smf, so if you want to run arbitrary scripts, then have a look at @ryancnelson’s example and modify to suit your tastes.

Automated VirtualBox SmartOS installs

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 12 Apr 2012 00:00:00 +0100

This is a quick script I wrote to:

Download the latest live ISO image of SmartOS
Create a VirtualBox VM, or update an existing VM with the latest ISO
Configure the VM with a zones disk, and boot it!

For the first install you’ll need to:

Configure the network (likely just “dhcp”)
Add c0d0 as the zones disk
Set a root password

After that it will just update the live ISO and use existing settings. By default it will create a port forward so that you can ssh -p 8322 root@localhost into the VM.

Here’s the script in full (or you can download it here):

#!/bin/sh
#
# Configurables:
#
#  - Disk size is in GB
#  - Memory size is in MB
#  - SSH port is the local forwarded port to the VM:22
#
disksize="32"
memsize="1024"
sshport="8322"
vmname="SmartOS"
dlsite="https://download.joyent.com/pub/iso"

vboxdir=$(VBoxManage list systemproperties \
            | awk '/^Default.machine.folder/ { print $4 }')

#
# Find a suitable md5sum program.
#
if type md5 >/dev/null 2>&1; then
    md5sum='md5'
    column='NF'
elif type digest >/dev/null 2>&1 &&
     digest md5 /dev/null >/dev/null 2>&1; then
    md5sum='digest md5'
    column='NF'
elif type digest >/dev/null 2>&1 &&
     digest -a md5 /dev/null >/dev/null 2>&1; then
    md5sum='digest -a md5'
    column='1'
elif type md5sum >/dev/null 2>&1; then
    md5sum='md5sum'
    column='1'
elif type openssl >/dev/null 2>&1 &&
     openssl md5 -hex /dev/null >/dev/null 2>&1; then
    md5sum='openssl md5 -hex'
    column='NF'
else
    echo "ERROR: Sorry, could not find an md5 program" 1>&2
    exit 1
fi

#
# Download MD5 file and parse it for the latest ISO image and checksum
#
curl -o smartos-sums.txt ${dlsite}/md5sums.txt 2>/dev/null
latest_md5=$(awk '/latest.iso/ { print $1 }' smartos-sums.txt)
smartos_version=$(sed -ne "/^${latest_md5}/s/.*-\(.*\).iso/\1/p" \
                    smartos-sums.txt)
if [ -z "${smartos_version}" ]; then
  echo "ERROR: Couldn't determine latest version"
  exit 1
fi

#
# Download the latest ISO image and verify
#
mkdir -p "${vboxdir}/${vmname}"
if [ ! -f "${vboxdir}/${vmname}/smartos-${smartos_version}.iso" ]; then
  echo "Downloading ${dlsite}/smartos-${smartos_version}.iso"
  curl -o "${vboxdir}/${vmname}/smartos-${smartos_version}.iso" \
          ${dlsite}/smartos-${smartos_version}.iso
  dl_md5=$(${md5sum} "${vboxdir}/${vmname}/smartos-${smartos_version}.iso" \
             | awk '{ print $'${column}' }')
  if [ -z "${dl_md5}" ]; then
    echo "ERROR: Couldn't fetch ISO image"
    exit 1
  fi
  if [ "${latest_md5}" != "${dl_md5}" ]; then
    echo "ERROR: md5 checksums do not match"
    exit 1
  fi
fi

#
# Create VirtualBox VM
#
echo "Creating/Updating Virtual Machine"
VBoxManage showvminfo "${vmname}" >/dev/null 2>&1
if [ $? -eq 0 ]; then
    # VM already exists, just update the ISO image
    VBoxManage storageattach "${vmname}" --storagectl "IDE Controller" \
      --port 1 --device 0 --type dvddrive \
      --medium "${vboxdir}/${vmname}/smartos-${smartos_version}.iso"
else
    # Create the VM
    VBoxManage createvm --name "${vmname}" --ostype OpenSolaris_64 --register
    VBoxManage storagectl "${vmname}" --name "IDE Controller" --add ide

    # Attach the ISO image
    VBoxManage storageattach "${vmname}" --storagectl "IDE Controller" \
      --port 1 --device 0 --type dvddrive \
      --medium "${vboxdir}/${vmname}/smartos-${smartos_version}.iso"

    # Create and attach the zone disk
    VBoxManage createhd --filename "${vboxdir}/${vmname}/smartos-zones.vdi" \
      --size $(echo "${disksize}*1024" | bc)
    VBoxManage storageattach "${vmname}" --storagectl "IDE Controller" \
      --port 0 --device 0 --type hdd \
      --medium "${vboxdir}/${vmname}/smartos-zones.vdi"

    # Set misc settings
    VBoxManage modifyvm "${vmname}" --boot1 dvd --boot2 disk --boot3 none
    VBoxManage modifyvm "${vmname}" --memory ${memsize}
    VBoxManage modifyvm "${vmname}" --natpf1 "SSH,tcp,,${sshport},,22"
fi

#
# Start it up
#
echo "Starting Virtual Machine"
VirtualBox --startvm "${vmname}" &

Hopefully there will be a follow-up post which updates my pkgsrc on Solaris for SmartOS, including zone setup etc.

iptables script for Debian / Ubuntu

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 30 Mar 2012 00:00:00 +0100

Most Linux distributions seem to have their own way of handling iptables. Red Hat based distributions come with an init script /etc/init.d/iptables which saves/restores configuration and allows you to check the status.

Debian / Ubuntu come with .. nothing.

So, there is a plethora of advice and ways of setting iptables up them, and this is mine.

It’s a simple shell script which is installed to /etc/network/if-pre-up.d/iptables, meaning it is executed prior to an interface being brought up - better to do it then than afterwards :)

I provide a couple of shell functions to make it easy write rules which are to be applied to both IPv4 and IPv6.

Here it is in its entirety, feel free to use/copy/whatever, it’s public domain.

#!/bin/sh
#
# IPTables firewall script.  There are many.  This is mine.
#


#
# Ensure sane path
#
PATH=/sbin:/usr/sbin:/bin:/usr/bin

#
# When running from the command line, provide a -v option to print the
# installed rules at the end.
#
verbose=
if [ "$1" = "-v" ]; then
    shift
    verbose=on
fi

#
# Rather than duplicate entries for iptables and ip6tables, have some small
# wrapper functions do it for us.
#
# ip4tbl - apply ruleset for just iptables
# ip6tbl - apply ruleset for just ip6tables
# iptbl  - apply ruleset for both iptables and ip6tables
#
ip4tbl()
{
    iptables "$@"
}
ip6tbl()
{
    ip6tables "$@"
}
iptbl()
{
    ip4tbl "$@"
    ip6tbl "$@"
}

#
# Flush all rulesets
#
iptbl -F
iptbl -X

#
# Block by default except outgoing traffic
#
iptbl -P INPUT DROP
iptbl -P FORWARD DROP
iptbl -P OUTPUT ACCEPT

#
# Allow everything on loopback
#
iptbl -A INPUT -i lo -j ACCEPT

#
# Permit established connections
#
iptbl -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

#
# Permit allowed services on all interfaces.  DNS is restricted to my public
# DNS servers, this just runs a hidden master.
#
iptbl  -A INPUT -p tcp -m tcp --dport 22   -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 25   -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 85.158.46.77    -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 85.158.46.77    -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 193.108.199.128 -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 193.108.199.128 -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 193.108.199.130 -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 193.108.199.130 -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 213.5.89.46     -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 213.5.89.46     -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 80   -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 113  -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 443  -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 465  -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 993  -j ACCEPT

#
# Permit ICMP and traceroute
#
ip4tbl -A INPUT -p icmp -j ACCEPT
ip6tbl -A INPUT -p ipv6-icmp -j ACCEPT
iptbl  -A INPUT -p udp -m udp --dport 33434:33523 -j ACCEPT

#
# Log denied connections
#
LOGCOMMON="-m limit --limit 5/min -j LOG --log-prefix 'iptables: ' --log-level 7"
iptbl  -A INPUT -p tcp       ${LOGCOMMON}
iptbl  -A INPUT -p udp       ${LOGCOMMON}
ip4tbl -A INPUT -p icmp      ${LOGCOMMON}
ip6tbl -A INPUT -p ipv6-icmp ${LOGCOMMON}

#
# Finally, reject to keep open connections down
#
iptbl -A INPUT -j REJECT

#
# Display INPUT chain if verbose
#
if [ -n "${verbose}" ]; then
    iptables  -L INPUT -vn --line-numbers
    ip6tables -L INPUT -vn --line-numbers
fi

Just remember to make it executable ;-)

New site design

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 20 Feb 2012 00:00:00 +0000

I am a minimalist at heart, and I love command line interfaces and text files.

However, I’m also pragmatic, and so for a long time my blog has been powered by Wordpress, which made it relatively easy to manage content and design. I just didn’t want the hassle of maintaining something myself.

Somewhere along the way recently I stumbled across Jekyll, and the idea of generating a static site from templates.

I’d seen a few sites I liked the look of, and I loved the idea of simplifying my site (and editing posts in vim!), and so the end result is the site you see now.

I pondered implementing something like disqus for comments, but honestly I get so few proper responses, compared to the hundreds and hundreds of spam attempts, that it’s not worth it - if you want to comment on a post, just use twitter (I’ll probably implement a little button to do that at some point).

Also, I got rid of categories, simpler to just have tags.

It’s likely I’ve messed up conversion of my old blog posts, as I did them all manually. If you spot anything obviously broken, please let me know. One of the great things of the Jekyll approach is that you can simply manage your site via github, and so if you prefer you can comment on breakages here!

Set up anonymous FTP upload on Oracle Linux

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 11 Jan 2012 00:00:00 +0000

Just because this took a little longer than I expected, here’s a quick howto for setting up an anonymous FTP drop-off on Oracle Linux, which I use as a simple way to transfer files out of my Virtual Machines.

Install vsftpd

$ sudo yum -y install vsftpd

Configure iptables

As FTP is a more complicated protocol than most, there is a special netfilter module required in order to correctly keep track of connections.

# You will perhaps want to change the insert number here.
$ sudo iptables -I INPUT 4 -m state --state NEW -p tcp --dport 21 -j ACCEPT
$ sudo /etc/init.d/iptables save

Add nf_conntrack_ftp to IPTABLES_MODULES

$ sudo vi /etc/sysconfig/iptables-config

Then load the module rather than reboot

$ sudo modprobe nf_conntrack_ftp

Create /incoming

Create /incoming area and ensure it has the correct file permissions and SELinux context. This is the bit which had me stumped for a little while, as I didn’t know about allow_ftpd_anon_write, and while I normally just disable SELinux, I do also like to know how things should work (and be able to write about them!):

$ sudo mkdir /var/ftp/incoming
$ sudo chown ftp:ftp /var/ftp/incoming

# This allows anonymous users to upload, but not see what is in the directory
$ sudo chmod 750 /var/ftp/incoming

$ sudo chcon -u system_u -t public_content_rw_t /var/ftp/incoming
$ sudo setsebool allow_ftpd_anon_write=1

Configure vsftpd

$ sudo vi /etc/vsftpd/vsftpd.conf

anon_upload_enable=YES

Startup

Finally, enable and start vsftpd:

$ sudo chkconfig vsftpd on
$ sudo /etc/init.d/vsftpd start

And that’s it, you should now be able to FTP as anonymous and upload files into /incoming.

Kickstart Oracle Linux in VirtualBox

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 09 Jan 2012 00:00:00 +0000

In my previous post I configured an Ubuntu laptop as a Kickstart install server for a physical machine I wanted to build.

Now that everything is configured for automated installs, it makes sense to use the same infrastructure to install virtual machines too. Everything can be done from the command line, and new virtual machines can be provisioned very quickly.

Again, I will use Oracle Linux 6.2 as an example.

Create Virtual Machine

These commands create a new virtual machine, including disk, and configures it for network booting. If you don’t already have VirtualBox installed:

$ sudo apt-get install virtualbox

Set the name of the virtual machine as it appears in VirtualBox, this variable is then used throughout:

$ VM="Oracle Linux 6.2"

Create the VM:

$ VBoxManage createvm --name "${VM}" --ostype "Oracle_64" --register

Create the hard disk (32GB expanding) and attach it via SATA. Note that I store my VMs in ${HOME}/VirtualBox rather than the default ${HOME}/VirtualBox VMs.

$ VBoxManage createhd --filename "VirtualBox/${VM}/${VM}.vdi" --size 32768

$ VBoxManage storagectl "${VM}" --name "SATA Controller" --add sata \
>   --controller IntelAHCI

$ VBoxManage storageattach "${VM}" --storagectl "SATA Controller" --port 0 \
>   --device 0 --type hdd --medium "VirtualBox/${VM}/${VM}.vdi"

Default RAM is 128MB, Oracle Linux installer requires at least 512MB however. Once installed we can drop back down to 256MB or so:

$ VBoxManage modifyvm "${VM}" --memory 512

Configure boot order. Put disk first, as on the first boot there is nothing on it so it falls through to PXE for install, then after the install the disk is bootable.

$ VBoxManage modifyvm "${VM}" --boot1 disk --boot2 net --boot3 none --boot4 none

Change NIC type from the default e1000, as a vanilla VirtualBox install does not include the firmware necessary to network boot from that device – it is available in the “VirtualBox Extension Pack” add-on. Switch to a plain PCNet Fast III which does include PXE firmware.

$ VBoxManage modifyvm "${VM}" --nictype1 Am79C973

VirtualBox does have the ability to serve TFTP directly from the file system by placing files inside ${HOME}/.VirtualBox/TFTP/ but I prefer to just use the network as it’s already configured.

$ VBoxManage modifyvm "${VM}" --nattftpserver1 10.0.2.2
$ VBoxManage modifyvm "${VM}" --nattftpfile1 pxelinux.0

Configure pxelinux/kickstart

We just need a couple of tweaks to the configs from the last blog entry, as the network addresses are different inside VirtualBox, and we also may want a different kickstart configuration.

$ sudo vi /var/lib/tftpboot/pxelinux.cfg/default

I just amended the existing entry to point to 10.0.2.2 which is the address of the machine running VirtualBox, and a different ks-vm.cfg kickstart configuration file, but you could also create a new label if you wanted to regularly switch between different configurations:

LABEL ol6.2
    KERNEL /ol6.2/vmlinuz
    APPEND initrd=/ol6.2/initrd.img ks=http://10.0.2.2/ks-vm.cfg

For virtual machines I use a slightly different configuration compared to previously. I’ve only shown the changes below, not the full file:

$ sudo cp /usr/share/nginx/www/ks.cfg /usr/share/nginx/www/ks-vm.cfg

# Update network configuration for DHCP instead of static
network --bootproto=dhcp
url --url=http://10.0.2.2/ol6.2

# Don't specify disks, just use default layout
bootloader --location=mbr --driveorder=sda
clearpart --all --initlabel

# Just add 'screen' to the default set of @base and @core packages.
%packages
screen
%end

Start Virtual Machine

All that’s left to do is to boot up the VM, and everything else should run automatically.

$ VirtualBox --startvm "${VM}"

All done!

Kickstart Oracle Linux from Ubuntu

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 09 Jan 2012 00:00:00 +0000

As my new job involves working on Oracle Linux, I figured I should migrate my home server to it, which would also mean I could move it to a proper RAID10 configuration rather than relying on multiple RAID1s.

My laptop runs Ubuntu, and I wanted to install the server from it using PXE and Kickstart, so here’s how I did it.

Configure dhcpd

DHCP is required for two things, to give the server its network configuration, and to point it at the PXE boot loader we want to use.

$ sudo apt-get install isc-dhcp-server
$ sudo vi /etc/dhcp/dhcpd.conf

Here’s the configuration I used, which says to configure 192.168.2.0/24 with a dynamic DHCP range between 192.168.2.100 – 192.168.2.200, and to boot machines using pxelinux.0 which is relative to the TFTP root directory (configured in the next section):

subnet 192.168.2.0 netmask 255.255.255.0 {
    range 192.168.2.100 192.168.2.200;
    filename "pxelinux.0";
}

The isc-dhcp-server install automatically tries to start the server, but will fail as it isn’t configured, so we restart it now that there is a working configuration installed

$ sudo /etc/init.d/isc-dhcp-server restart

Configure tftpd

TFTP is a simple protocol used to transfer files over the network, and due to its simplicity it is the primary way to network boot, as it can be easily embedded into firmware.

All we need to do is install the TFTP daemon and syslinux which includes the PXE boot loader, then put the pxelinux file into the tftproot area:

$ sudo apt-get install syslinux tftpd-hpa
$ sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot

Configure Oracle Linux DVD

To save space on the laptop we can just mount the DVD read-only and install from that:

$ sudo mkdir /media/ol6.2
$ sudo mount -o loop,ro /path/to/OracleLinux-R6-U2-Server-x86_64-dvd.iso /media/ol6.2

However, we do need to copy the kernel and initrd image from the DVD into the tftproot as they are required for booting:

$ sudo mkdir /var/lib/tftpboot/ol6.2 /var/lib/tftpboot/pxelinux.cfg
$ sudo cp -a /mnt/images/pxeboot/{initrd.img,vmlinuz} /var/lib/tftpboot/ol6.2/

Configure pxelinux

All that’s left for the PXE stage is to configure the boot loader, and tell it what kernel and initrd we want to use:

$ sudo vi /var/lib/tftpboot/pxelinux.cfg/default

This configuration has just one entry which is booted after a short wait, but pxelinux has many more options, including the ability to boot from local disk.

DEFAULT	ol6.2
PROMPT 1
TIMEOUT 5

LABEL ol6.2
    KERNEL /ol6.2/vmlinuz
    APPEND initrd=/ol6.2/initrd.img ks=http://192.168.2.1/ks.cfg

Note the ks argument which specifies the kickstart file we will use, that and the web server required to serve it will be set up next.

Configure nginx

I chose nginx as it is small and simple to configure, but any web server will do.

$ sudo apt-get install nginx
$ sudo vi /etc/nginx/sites-available/default

The install will be performed over HTTP, so we need to make the DVD we mounted earlier available. This entry in the server { } section makes the DVD available via http://192.168.2.1/ol6.2/:

server {
[...]
    location /ol6.2 {
        root /media;
        autoindex on;
        allow all;
    }

Then start nginx (unlike isc-dhcp-server this isn’t done automatically):

$ sudo /etc/init.d/nginx start

Configure kickstart

Finally, we create a kickstart configuration which specifies exactly how our target machine is to be installed, and this allows a completely unattended installation.

Ideally I should create a specific area for holding files like this, but as a quick hack I simply put it into the default nginx web root (and thus available as http://192.168.2.1/ks.cfg as configured earlier in the pxelinux.cfg/default file:

sudo vi /usr/share/nginx/www/ks.cfg

Here is my ks.cfg file in full. The only thing missing is a rootpw line to automatically set a root password, however for maximum security I am happy to forego a completely unattended installation and instead have the installer prompt me to type it in during the install.

Some notes:

UK keyboard language and timezone selected.
Automatically reboot when the installer is finished.
Point to the Oracle Linux DVD using the url directive.
I have 4 disks configured with RAID1 for /boot, and RAID10 for swap and /.
Disks are referred to by path, to ensure correct ordering.
A small set of packages are installed, containing just the functionality I require.
A small %post section is used to perform any fixes I want for the first boot.

#
# Miscellaneous options.
#
install
keyboard uk
lang en_GB.UTF-8
reboot
selinux --enforcing
timezone Europe/London

#
# User setup.  I'd create my local user here and configure sudoers, but
# kickstart doesn't yet support creating a user with uid/gid of 1000 (the
# gid is always 500, even if you add the named group first).
#
authconfig --enableshadow --passalgo=sha512

#
# Networking.  The 'network' line needs to be on a single line
# for kickstart to work - it is only split here for the blog.
#
firewall --service=ssh
network --bootproto=static \
        --hostname=gromit.adsl.perkin.org.uk \
        --ip=192.168.2.10 \
        --netmask=255.255.255.0 \
        --gateway=192.168.2.1 \
        --nameserver=193.178.223.141,208.72.84.24 \
        --ipv6=auto
url --url=http://192.168.2.1/ol6.2

#
# Disk configuration.
#
bootloader --location=mbr --driveorder=sda,sdb,sdc,sdd
clearpart --all --initlabel
#
# /boot (RAID1 necessary as booting from RAID10 isn't supported)
#
part raid.00 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.01 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.02 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.03 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-3*0
raid /boot --level=1 --device=md0 --fstype=ext4 raid.00 raid.01 raid.02 raid.03
#
# swap, RAID10 of size RAM+2GB, give or take..
#
part raid.10 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.11 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.12 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.13 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-3*0
raid swap --level=10 --device=md1 --fstype=swap raid.10 raid.11 raid.12 raid.13
#
# /, RAID10 of remainder (have to specify an arbitrary --size even with --grow)
#
part raid.20 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.21 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.22 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.23 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-3*0
raid / --level=10 --device=md2 --fstype=ext4 raid.20 raid.21 raid.22 raid.23

#
# Packages.  @base and @core are pre-selected.
#
%packages
@cifs-file-server
@console-internet --optional
@development
@legacy-unix --optional
@mail-server
@network-server --optional
@network-tools
@nfs-file-server
@web-server
screen
%end

#
# Post-install fix-ups.
#
%post
#
# The 'network' directive doesn't support DNS search paths, so set those
# manually, and disable Network Manager.
#
printf "/^NM_CONTROLLED/s/yes/no/\nw\nq\n" \
  | ed /etc/sysconfig/network-scripts/ifcfg-eth0
printf "/^#/s/.*/search adsl.perkin.org.uk perkin.org.uk/\nw\nq\n" \
  | ed /etc/resolv.conf
#
# Disable unwanted services
#
chkconfig --del cups
%end

All done.

Last day at MySQL

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 22 Dec 2011 00:00:00 +0000

Today is my final day working on MySQL. It has been an amazing 4.5 years, and I’ve loved working on the technical challenges involved in producing a piece of software which runs on so many different platforms, as well as working with so many talented individuals. I have learned a lot, which is just how I like it.

Times have changed since I first joined MySQL AB, of course. But despite going through two acquisitions, the day to day work hasn’t changed much at all. I still get to work from home, which is something I am passionate about and feel many companies are missing out on massively by still being stuck in an outdated 20th century mindset. Oracle have continued to invest in MySQL, providing additional headcount and extra hardware. And the vast majority of people I started working with back in 2007 are still with the company (don’t believe the FUD, folks) and working harder than ever.

I’m most proud of the work we have done internally to ensure our MySQL releases are in excellent shape. When I first joined, our internal “PushBuild” system was very limited, only able to handle a small number of pushes with reduced testing across a few platforms, and the builds produced were completely different to those that were released.

Nowadays, thanks to the investment made by Sun and Oracle, we have a large server farm producing thousands of builds and hundreds of gigabytes of data every day across all our supported platforms. Those which are based on MySQL 5.5 or newer have additional package verification tests, using chroots and virtual machines, to ensure that the RPM/MSI/etc packages can be installed, run, and uninstalled, all automatically and on every push by developers. We run additional nightly and weekly tests which extend the default set of test suites. Our 5.5+ releases are produced directly in PushBuild. And we are looking to extend this to all MySQL products, not just the Server.

If the above sounds fun, keep an eye out for job posts, as the Release Engineering team is looking to expand, and they are an amazing group of people to work with.

I certainly found it fun, which is why I’m going to continue in that line of work, and stay within Oracle, but move to the Linux group. To be honest, databases have never been a passion of mine, however I am rather fond of Operating Systems, and so I am really looking forward to continuing to work on continuous integration and testing, but with Oracle Linux and Oracle VM instead of MySQL. Plus, it will be nice to go back to a purely technical role – I have learned over the last year or so that management isn’t really my thing :)

To all my colleagues, past and present, thank you for the wonderful ride. I have many, many good memories, and hopefully we will keep in touch.

Installing OpenBSD with softraid

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 15 Dec 2011 00:00:00 +0000

This is more of a log for me than anything else, but perhaps someone will find this useful.

OpenBSD includes a software RAID implementation which supports booting in newer snapshots, and I was itching to install the latest version and use it as my file server, which has 4 750GB disks. There is a small bit of preparation work to do prior to installing, which is the bulk of this entry, most of which is based on this undeadly.org article.

Grab latest amd64 snapshot, boot it, drop into (S)hell mode and set up the disks:

#!/bin/sh

cd /dev
sh MAKEDEV sd1 sd2 sd3 sd4 sd5
for disk in 0 1 2 3
do
  # Clear beginning of disks..
  dd if=/dev/zero of=/dev/rsd${disk}c bs=1m count=10

  # ..and initialise new partition table
  fdisk -iy sd${disk}

  #
  # Create BSD disklabel:
  #
  # - 128m partitions at start to hold kernels for booting
  # - 4g spare raid on each disk for testing
  # - rest raid for main OS and data
  #   - OS and /home on first two mirrored disks
  #   - /store on second two mirrored disks
  #
  print "a a\n\n128m\n\na d\n\n4g\nraid\na e\n\n\nraid\nw\nq\n" \
    | disklabel -E sd${disk}

  # Clear beginning of raid partitions
  dd if=/dev/zero of=/dev/rsd${disk}d bs=1m count=10
  dd if=/dev/zero of=/dev/rsd${disk}e bs=1m count=10
done

# Create RAID1 mirrors
bioctl -c 1 -l sd0e,sd1e softraid0
bioctl -c 1 -l sd2e,sd3e softraid0

# Exit shell and start the (I)nstall
exit

As for the install, go with the sensible defaults, other than:

change keyboard layout to “uk”
manually configure network, enable rtsol
start ntpd
do not expect to run X

When it comes to disk selection, choose sd4 as the root disk, and use the following layout:

# partition  size  mount
  sd4a       1G    /
  sd4b       8G    swap
  sd4d       1G    /tmp
  sd4e       8G    /var
  sd4f       16G   /usr
  sd4h       rest  /home

then initialise sd5 with:

# partition  size  mount
  sd5a       2G    /altroot
  sd5d       8G    /scratch
  sd5e       rest  /store

Install the full OS, set the correct timezone, then before rebooting initialize the boot partitions and copy the kernels to them. Doing this on all of them means we can boot from any disk.

#!/bin/sh

for disk in 0 1 2 3
do
  newfs sd${disk}a
  mount /dev/sd${disk}a /mnt2
  cp /mnt/bsd* /mnt2
  umount /mnt2
done
eject cd0a
reboot

Job done.

Create VirtualBox VM from the command line

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 21 Sep 2011 00:00:00 +0100

As something of a follow-up post to the previous entry, here’s a quick recipe for creating a Virtual Machine using the VirtualBox command line tools:

We’re using Windows Server 2008 64bit as an example, modify to taste.

$ VM='Windows-2008-64bit'

Create a 32GB “dynamic” disk.

$ VBoxManage createhd --filename $VM.vdi --size 32768

You can get a list of the OS types VirtualBox recognises using:

$ VBoxManage list ostypes

Then copy the most appropriate one into here.

$ VBoxManage createvm --name $VM --ostype "Windows2008_64" --register

Add a SATA controller with the dynamic disk attached.

$ VBoxManage storagectl $VM --name "SATA Controller" --add sata \
>  --controller IntelAHCI
$ VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 \
>  --device 0 --type hdd --medium $VM.vdi

Add an IDE controller with a DVD drive attached, and the install ISO inserted into the drive:

$ VBoxManage storagectl $VM --name "IDE Controller" --add ide
$ VBoxManage storageattach $VM --storagectl "IDE Controller" --port 0 \
>  --device 0 --type dvddrive --medium /path/to/windows_server_2008.iso

Misc system settings.

$ VBoxManage modifyvm $VM --ioapic on
$ VBoxManage modifyvm $VM --boot1 dvd --boot2 disk --boot3 none --boot4 none
$ VBoxManage modifyvm $VM --memory 1024 --vram 128
$ VBoxManage modifyvm $VM --nic1 bridged --bridgeadapter1 e1000g0

Configuration is all done, boot it up! If you’ve done this one a remote machine, you can RDP to the console via vboxhost:3389.

$ VBoxHeadless -s $VM

Once you have configured the operating system, you can shutdown and eject the DVD.

$ VBoxManage storageattach $VM --storagectl "IDE Controller" --port 0 \
>  --device 0 --type dvddrive --medium none

Finally, it’s a good idea to take regular snapshots so that you can always revert back to a known-good state rather than having to completely re-install.

$ VBoxManage snapshot $VM take <name of snapshot>

And, if you need to revert back to a particular snapshot:

$ VBoxManage snapshot $VM restore <name of snapshot>

Enjoy!

Creating chroots for fun and MySQL testing

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 14 Sep 2011 00:00:00 +0100

Virtualisation is all the rage today, however there are still a few cases where good old-fashioned Unix chroot is still applicable, and testing MySQL across multiple platforms and architectures is one of those cases.

At Oracle we do full automated package verification testing of our MySQL server binaries prior to release, which attempts to install the package, start the server, run some functionality testing, then uninstall. It is of course highly desirable that the testing environment this is performed in is as close to a clean install of the target operating system as possible, to avoid problems such as our packages depending upon some local changes or packages we may have installed which won’t be available on a customer system.

Given the large number of platforms and architectures that MySQL supports, going the virtualisation route would mean having to use many different products: VirtualBox for x86, zones for SPARC, qemu for ia64/PA-RISC/others (if it even supports them), and this gets complicated quickly and is not very maintainable. Thus I chose to use chroot as much as possible. In addition, it’s much faster and less intensive on resources to use a chroot than boot up an entire OS image each time.

I built all images directly from the original installation images (DVD, ISO, RPM, etc), to ensure that there was no contamination from our build environment or local install scripts in the image – they should be as close to what a normal user or customer will be running in their setup. From the install image, the packages are installed to a temporary directory, some final modifications are made, then the directory is tarred up ready to be extracted by the test framework and used.

Here are some operating system specific examples, which set up an extracted chroot image into ${CHROOTDIR}. There may be additional steps required to get a fully functioning chroot, such as copying device files (/dev/zero and /dev/null are usually the minimum requirements) and adding users.

FreeBSD

FreeBSD 7 and 8 come as a number of sets in tar format, and for our purposes we only need to extract the base set. You may wish to add more sets if you want to use your chroot for building packages:

#!/bin/sh
mdunit=$(mdconfig -a -t vnode -o readonly -f /path/to/dvd1/of/freebsd.iso)
mount_cd9660 /dev/${mdunit} /mnt
cat /mnt/*/base/base.?? | tar -xpzf - -C ${CHROOTDIR}
umount /mnt
mdconfig -d -u ${mdunit}

HP-UX

HP-UX has since been EOL’d for MySQL, however this information might still prove useful. The HP-UX installation media contains per-directory packages, with the contents representing how they are laid out on the destination file system with each file gzip compressed.

#!/bin/sh
cd /path/to/extracted/hpux-dvd1
for pkg in $LIST_OF_PKGS
do
  for subpkg in ${pkg}/*
  do
    if [ ! -d "${subpkg}" ]; then
      continue
    fi
    for d in $(find ${subpkg} -type d)
    do
      mkdir -p ${CHROOTDIR}/$(echo ${d} \
        | sed -e "s#${subpkg}/##g" \
              -e "s#usr/newconfig/##g")
    done
    for f in $(find ${subpkg} -type f)
    do
      gzip -dc ${f} >${CHROOTDIR}/$(echo ${f} \
        | sed -e "s#${subpkg}/##g" \
              -e "s#usr/newconfig/##g")
    done
  done
done

Once this is done you’ll need to fix up permissions in bin and lib directories (make files executable), as well as create a bunch of symlinks for e.g. /bin and /lib.

OSX

For OSX we don’t actually use a chroot tarball but instead create a sparse disk image. Currently the size of the “chroot” is very large as there’s no easy way to strip down an OSX install, so mounting a disk image is faster than unpacking a chroot, plus it preserves various HFS-specific attributes.

You will likely need at least the BSD, BaseSystem, and Essentials packages.

# Create a sparse image to hold the chroot (which isn't really a directory)
hdiutil create -fs HFS+ -size 8g -type SPARSE -volname osx-chroot ${CHROOTDIR}
hdiutil attach -mountpoint ${CHROOTDIR} ${CHROOTDIR}.sparseimage
# Either attach a DVD image or the real thing
hdiutil attach -mountpoint /Volumes/osx-install /path/to/dvd
# Install packages
for pkg in BSD BaseSystem Essentials
do
  installer -verbose \
   -pkg /Volumes/osx-install/System/Installation/Packages/${pkg}.pkg \
   -target ${CHROOTDIR}
done
# Unmount
hdiutil detach ${CHROOTDIR}
hdiutil detach /Volumes/osx-install

Red Hat / Oracle Linux / SuSE

For RPM-based distributions we use rpm to directly install packages into the chroot. The list of RPMs we install varies quite a lot from release to release, usually by having to increase the number:

RH3: 81
RH4: 85
RH5: 114
RH6: 203

for the same functionality.

# Kludge for 'setup' RPM to install
mkdir -p ${CHROOTDIR}/var/lock/rpm
# If installing from an ISO:
mount -o loop /path/to/iso /mnt
# Path varies from release to release
cd /mnt/path/to/RPMs
rpm --root=${CHROOTDIR} -Uvh ${LIST_OF_RPMS}

Solaris

Similar to RPM, we use the native package manager to install packages directly into the chroot directory:

# Avoid prompts
sed -e "s/ask$/nocheck/" /var/sadm/install/admin/default > /tmp/admin-$$
pkgadd -a /tmp/admin-$$ -R ${CHROOTDIR} -d . ${LIST_OF_PKGS}
rm /tmp/admin-$$

Windows

Ok, so of course we can’t use chroot images for Windows, as it doesn’t have chroot(2). So here we use VirtualBox and its snapshot ability to load a clean snapshot of a basic Windows install, do the tests, then shut down the virtual machine, restore the snapshot, and boot up again.

Graphing memory usage during an MTR run

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 30 Jun 2011 00:00:00 +0100

In order to optimally size the amount of RAM to allocate to a set of new machines for running MTR, I ran a few tests to check the memory usage of an MTR run for mysql-trunk and cluster-7.1. As using a RAM disk considerably speeds things up, I set the vardir to be on /ramdisk and logged the usage of that too.

The tests were performed on an 8-core E5450 @ 3.00GHz with 24GB RAM, with 8GB allocated to /ramdisk. Each branch ran the default.daily collection, which generally contains the most testing we do per-run. Between each run I rebooted the machine to clear the buffer cache and /ramdisk.

I used something like the script below, which saved the per-second usage of /ramdisk, the total RAM used, and the RAM used minus buffers.

#!/bin/bash

BRANCH="mysql-trunk"
BUILDDIR="mysql-5.6.3-m5-linux2.6-x86_64"
TESTDIR="${HOME}/mtr-test/${BRANCH}"

stats()
{
  i=1
  rm -f ${TESTDIR}/stats-${BRANCH}
  while [ -f ${TESTDIR}/running ]; do
    rd=$(df -k /ramdisk | awk '/^\// {print $3}')
    mem=$(free | awk '/^Mem/ {print $3}')
    mem1=$(free | awk '/cache:/ {print $3}')
    echo "${i} ${rd} ${mem} ${mem1}" >>${TESTDIR}/stats-${BRANCH}
    i=$((i+1))
    sleep 1
  done
}

export TMPDIR="${TESTDIR}/tmp"
rm -rf ${TMPDIR}
mkdir -p ${TMPDIR}

>${TESTDIR}/running
stats &

(
  cd ${TESTDIR}/${BUILDDIR}/mysql-test

  perl mysql-test-run.pl ... --parallel=8 --vardir=/ramdisk/mtr-${BRANCH}/...
  mv /ramdisk/mtr-${BRANCH}/* ${TMPDIR}/
  ...
)

sync
rm -f ${TESTDIR}/running
wait

First I graphed a straight run of the two branches, using the following gnuplot script:

set terminal png enhanced font "Times,11" size 640,768
set output "mtr-ram.png"
set title "MTR memory usage (8-core Xeon, 24GB, 8GB RAM disk)"
set xlabel "Time (minutes)"
set ylabel "Memory usage (GB)"
set yrange [0:16]
set xtics 10
set key top box
set grid
plot "stats-mysql-trunk" every 60 using (($1)/60):(($2)/1024/1024) \
        title 'mysql-trunk /ramdisk usage' with lines, \
     "stats-mysql-trunk" every 60 using (($1)/60):(($3)/1024/1024) \
        title 'mysql-trunk RAM (inc buf)' with lines, \
     "stats-mysql-trunk" every 60 using (($1)/60):(($4)/1024/1024) \
        title 'mysql-trunk RAM (exc buf)' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($2)/1024/1024) \
        title 'cluster-7.1 /ramdisk usage' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($3)/1024/1024) \
        title 'cluster-7.1 RAM (inc buf)' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($4)/1024/1024) \
        title 'cluster-7.1 RAM (exc buf)' with lines

I then performed a valgrind run on mysql-trunk using similar scripts. As valgrind takes considerably longer (and uses more RAM) I kept it separate as the combined graph isn’t very clear:

So, based on these results, the host machine (16GB RAM + 8GB RAM disk) is probably a sensible guide for now, and allows for some future growth.

Fix input box keybindings in Firefox

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 29 Jun 2011 00:00:00 +0100

Those of us used to command line editing will no doubt have been frustrated many times in Firefox when editing text in an input box and subconciously hitting ctrl-w to delete-word, only to have the tab close and your work deleted.

Thankfully there is a workaround to this. It used to be a case of adding the following to .gtkrc:

gtk-key-theme-name = "Emacs"

However these days it’s a gconf setting:

$ gconftool-2 --set /desktop/gnome/interface/gtk_key_theme Emacs --type string

This will bind ctrl-w to delete-word when in an input box, but retain the close tab binding elsewhere, a nice implementation of DWIM. See this page for more information.

How to lose weight

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 24 Jun 2011 00:00:00 +0100

This year I set myself a target to get back to my pre-marital weight. Back in 2001 I was 90kg, but over the past decade I’d steadily gone up and for quite a few years had hovered around 100kg.

I’m a bit of a perfectionist, and like shiny graphs, so from Jan 1st this year I weighed myself every day on the Wii Fit, and kept a record of my progress. This morning I was finally under 90kg. Here’s how things have gone thus far:

And for the geeks, my gnuplot script, assuming an input file named weight.txt containing lines of the format "%Y-%m-%d <weight>":

set terminal png size 640, 400
set output "weight-jun11.png"
set xdata time
set timefmt "%Y-%m-%d"
set format x "%b"
set xlabel 'Date'
set ytics 2
set ylabel 'Weight (kg)'
set multiplot
plot 'weight.txt' using 1:2 title 'Weight' with lines linecolor 2

My three step plan for losing weight

The interwebs are awash with a million different ways to lose weight, and there are thousands of companies who will gladly accept large chunks of your cash in order to provide you with their expert opinion on how to do it. This is my three-step plan, and you can have it for free:

Eat less.
Eat less!
Profit!!!

Really, that’s it. It’s not rocket science, after all. If you eat less stuff (especially saturated fat and sugar), there is less excess for your body to store as fat, and thus over time you weigh less as your body starts to use up the excess you have stored.

Here’s some things I did to achieve this.

Eat less

The basic aim. For me this was mainly about reducing my portion size.

Previously, I’d eat a massive bowl of cereal in the morning. which I thought was a good thing as everyone always says to ensure you have a good breakfast. However I was eating more than my body needed and as a consequence it was likely storing up the excess.

Also at meal times I’d usually serve myself a full plate, and always feel that I needed to finish it. I liked feeling full. However, again, this likely just meant I was eating more than I needed, and I found that just by serving less food I was still satisfying my hunger but with less excess.

After a relatively short period of time I found that by avoiding these large meals, I needed less to feel full – as if my stomach had shrunk and gotten used to the reduced size. This was a great positive feedback loop, as it massively helped avoid the temptation to snack between meals.

Cut down on sugar/fat

I realise this can be hard for some people, I found it relatively easy but as you can see on the graph above there are some upward trends, which were mainly when I went to visit my parents, who have a cupboard full of chocolates, crisps, cakes, sweets, etc, and I find it really hard not to have at least one or two fake Lidl snickers bars per day!

However, also note that after an upward trend, I lost the weight again pretty quickly, so don’t worry too much about a few days of eating junk, I actually noticed how bad I felt afterwards after getting used to a reduced sugar/fat intake, and that provided good incentive to cut down again.

Some practical things I did:

Stopped putting sugar in tea/coffee. I’ve since regressed, but it was helpful to do this for a while, and definitely helped wean my body off desiring sugar.
Avoided snacking on crisps, chocolate, biscuits, etc. When you work from home this can be difficult, but I found it helped to ensure they weren’t in the house to begin with, and that we were stocked up on bananas, apples, and other less sugary/fatty snacks.
Used a fine cheese grater when making sandwiches, beans on toast, etc. Previously I’d put a good few slabs of cheese in, when I didn’t really need that quantity. The fine grater ensured I still got the taste, but with less quantity.
Bought reduced fat mayonnaise, margarine, etc. These actually taste pretty good these days, and in a sandwich with lots of fresh cucumber, tomato, salad etc you don’t notice the difference

All these things helped to lose weight and, in a similar manner to the ‘eating less’ part, trained my body to not require them as much as it used to – and actually, to noticeably feel worse if ever I regressed, which made getting back on track very easy.

Weigh myself daily

It seems this is generally not recommended, and advice is that you should weigh yourself weekly (at the same time each week). Logically this doesn’t make sense to me, as a geek it’s obvious that the more data points you have, the better. Your body weight can fluctuate quite a lot from day to day especially with regards to how much liquid you have drunk, and if you are only weighing yourself once per week you could get caught out by a daily spike.

Aside from the geek factor of a better graph resolution, I also prefer daily weighing as it provides me, as a perfectionist, with incentives both ways. If I weigh less compared to the previous day, I feel good that I am achieving my aim, and am encouraged to continue. If I weigh more, it’s a warning that I may have eaten too much, and I am then motivated to be more careful that day.

Weighing daily also helps to just keep your mind reminded of the task, plus if you’re on the Wii Fit already you might be tempted to do some exercise :)

Next steps

I’m not content to stay here, the next plan is to get closer to my ideal BMI weight which is around 80kg. This is likely to be much harder, as there is less excess fat for me to get rid of now. However, one thing I haven’t done so far is increase the amount of exercise I do, and there is definitely room for improvement there!

Hopefully I can provide another update later this year and perhaps be around 85kg.

How to fix stdio buffering

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 23 Jun 2011 00:00:00 +0100

It’s a common problem. You write some shell command like:

$ tail -f /var/log/foo | egrep -v 'some|stuff' | sed | awk

and wonder why nothing is printed, even though you know some text has matched. The problem is that stdio is being buffered, and there’s a very good write-up of the problem here so I won’t repeat the technical background.

What I will provide though is how to fix it for common cases.

stdbuf

stdbuf is part of GNU coreutils, and is essentially an LD_PRELOAD hack which calls setvbuf() for an application. Thus it is a generic solution to the problem and can be used to fix most applications. Usage looks like this:

$ tail -f /var/log/foo | stdbuf -o0 app ...

which will disable output buffering for app, assuming it does not do something itself to reverse the setvbuf() call. An example of a misbehaving application is mawk, below.

awk

GNU awk needs no modifications, that is it does not buffer when there is no controlling tty.

mawk however (the default awk in Debian/Ubuntu and possibly others) buffers output, and also does not seem to work with stdbuf. It does however provide a -Winteractive option which will turn off buffering.

$ tail -f /var/log/foo | gawk

$ tail -f /var/log/foo | mawk -Winteractive

sed

GNU sed provides the -u option which calls fflush(), thereby providing unbuffered output. You can also use stdbuf as above.

$ tail -f /var/log/foo | sed -u

$ tail -f /var/log/foo | stdbuf -o0 sed

grep

Similar to sed, GNU grep provides a specific option, --line-buffered, to disable buffering, or again you can use stdbuf.

$ tail -f /var/log/foo | grep --line-buffered

$ tail -f /var/log/foo | stdbuf -o0 grep

Serving multiple DNS search domains in IOS DHCP

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 13 Jun 2011 00:00:00 +0100

I have a Cisco router at home which I also use as a DHCP server, and it works pretty well. Today I wanted to fix a long-standing issue on my network, in that I want multiple DNS search domains.

First off, the domain-name DHCP option doesn’t support multiple entries so we can’t use that. So, off to try raw DHCP option codes. You can find the list of options here, thus 119 is the one I want.

Trying a simple:

ip dhcp pool host.net.example.com
   option 119 ascii net.example.com,example.com

didn’t work at all. A quick prod of lazyweb (in this case Simon on IRC) suggested using hex input instead. In order to do that we need to convert the ASCII string into Cisco’s hex sequence, which is as follows:

Split domain name by dot
Prepend each string by its length (in hex)
NUL terminate each domain
Dot-seperate the final string in 16bit chunks

To do this I wrote a quick Python script:

#!/usr/bin/python

import sys

hexlist = []
for domain in sys.argv[1:]:
    for part in domain.split("."):
        hexlist.append("%02x" % len(part))
        for c in part:
            hexlist.append(c.encode("hex"))
    hexlist.append("00")

print "".join([(".%s" % (x) if i and not i % 2 else x) \
               for i, x in enumerate(hexlist)])

which can be used like this:

$ ./cisco.py net.example.com example.com
036e.6574.0765.7861.6d70.6c65.0363.6f6d.0007.6578.616d.706c.6503.636f.6d00

Then back to IOS and paste it in:

ip dhcp pool host.net.example.com
   option 119 hex 036e.6574.0765.7861.6d70.6c65.0363.6f6d.0007.6578.616d.706c.6503.636f.6d00

This seems to do what we want, though IOS appears to append a dot to each domain when serving via DHCP.

One last note, if you use this in addition to domain-name then the option 119 list will be appended to the domain-name name in the search list, so you’d actually want something like this:

ip dhcp pool host.net.example.com
   domain-name net.example.com
   option 119 hex 0765.7861.6d70.6c65.0363.6f6d.00

to generate a resolv.conf containing:

domain net.example.com
search net.example.com example.com.

Fix Firefox URL double click behaviour

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 13 Jun 2011 00:00:00 +0100

One of the best things I find with OS X is the consistent and sane handling of keyboard and mouse bindings. Since having to move back to Linux for work I am constantly frustrated that different applications all have their own idea of what shortcuts to use.

One of the biggest annoyances was the mouse click behaviour in the browser URL bar. OS X behaviour is one click to position cursor, two clicks to select a word, three clicks to select all. I rely on this behaviour a lot as I frequently copy/paste parts of URLs.

Thankfully that’s one that can be fixed easily. Simply navigate to

and double click on the `browser.urlbar.doubleClickSelectsAll` key to set it to false. One annoyance down, lots more to go..

SSH via HTTP proxy in OSX

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 20 Apr 2011 00:00:00 +0100

If you happen to be stuck behind a corporate firewall with only HTTP proxies for external access, you might still be able to SSH out through them using the built-in nc on OSX.

First, hope that the proxies haven’t disabled the CONNECT method, then simply add a section to your .ssh/config like this:

Host foobar.example.com
    ProxyCommand          nc -X connect -x proxyhost:proxyport %h %p
    ServerAliveInterval   10

This will tunnel the connection through the HTTP proxy to the remote server. The ServerAliveInterval setting is required as most proxies will drop the connection after a period of inactivity.

To avoid issues with trying to connect to the host when not behind the corporate firewall, replace the above with a fake entry for the proxy method like this:

Host foobar-proxy.example.com
    HostName              foobar.example.com
    ProxyCommand          nc -X connect -x proxyhost:proxyport %h %p
    ServerAliveInterval   10

Then use

$ ssh foobar-proxy.example.com

when inside the firewall, and

$ ssh foobar.example.com

when outside.

Simples, no external software required.

How to build MySQL releases

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 09 Nov 2010 00:00:00 +0000

One of the major benefits from the CMake work available in MySQL 5.5 is that in the MySQL Release Engineering team we have been able to make it easy for users and developers to build MySQL exactly as we do for the official releases. For too long there has been a disconnect between the binaries produced as part of a regular “./configure; make” build and what we ship to users and customers.

We’re still not exactly where we want to be, there are still some parts which need to be integrated into the server tree, but for now it’s relatively straightforward to build exactly as we do.

Here are the instructions, using the mysql-5.5.6-rc.tar.gz source tarball as an example.

tar.gz

These are the generic instructions to build a tarball release. Note that we make use of CMake’s out-of-srcdir support to

ensure the source directory is kept pristine
allow us to easily build both debug and release binaries, and package them together

#!/bin/sh

tar -zxf mysql-5.5.6-rc.tar.gz

# Build debug binaries first, they are picked up in the final 'make package'
mkdir debug
(
  cd debug
  cmake ../mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release -DCMAKE_BUILD_TYPE=Debug
  make VERBOSE=1
)

# Build release binaries and create final package
mkdir release
(
  cd release
  cmake ../mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release
  make VERBOSE=1 package
)

Assuming everything goes ok, you should end up with a tarball in the release/ directory.

Some platforms require extra flags to be specified on the CMake command line to ensure the correct compiler etc is used. Here are some that we use.

Avoid libstdc++ dependancy

In order to create “generic” binaries, on GCC platforms we compile using CXX=gcc. This avoids libstdc++ being pulled in, and means that the server will run across a larger range of releases as you do not rely on having the exact version of libstdc++ installed by your system package manager.

Paste this before running cmake.

# CXXFLAGS is required to trick CMake into believing it is a C++ compiler
$ CXX=gcc; CXXFLAGS="-fno-exceptions"; export CXX CXXFLAGS

Specify target architecture

On systems which support multiple targets, you can specify exactly which one you want, rather than relying on the OS default. These strings are meant to be added to the cmake command lines above.

$ -DCMAKE_OSX_ARCHITECTURES="i386"    # 32bit
$ -DCMAKE_OSX_ARCHITECTURES="x86_64"  # 64bit

GCC/Sun Studio

$ -DCMAKE_C_FLAGS="-m32" -DCMAKE_CXX_FLAGS="-m32"  # 32bit
$ -DCMAKE_C_FLAGS="-m64" -DCMAKE_CXX_FLAGS="-m64"  # 64bit

HP/UX

$ -DCMAKE_C_FLAGS="+DD64" -DCMAKE_CXX_FLAGS="+DD64"

RPM

We’ve spent a lot of time improving the RPM builds too. Previously some of the configuration was only available in the commercial RPM builds, but now we have merged them into the community version.

#!/bin/sh

mkdir -p rpm/{BUILD,RPMS,SOURCES,SPECS,SRPMS} tmp

# Create spec file.
# XXX: We should probably just include this in the source tarball.
tar -zxf mysql-5.5.6-rc.tar.gz
(
  mkdir bld; cd bld
  cmake ../mysql-5.5.6-rc
)

cp bld/support-files/*.spec rpm/SPECS
cp mysql-5.5.6-rc.tar.gz rpm/SOURCES

rpmbuild -v --define="_topdir $PWD/rpm" --define="_tmppath $PWD/tmp" \
 -ba rpm/SPECS/mysql.5.5.6-rc.spec

You should end up with nice shiny RPMs in rpm/RPMS. One thing to note is that the RPM spec no longer runs the test suite as part of the build, so you will need to run that separately. On the plus side, you get your RPMs much quicker.

Another nice thing about the improved RPM spec is that you can now build targetted distribution RPMs, as we do. These have extra dependancy information in them tailored to the target distribution. Currently the spec file supports the distributions we build on, but we will accept patches for others.

To enable this, use:

$ rpmbuild -v --define="distro_specific 1" ...

Windows

Our Windows builds have relied on CMake since MySQL 5.0, but the procedure has still changed to ensure that you can build as we do. These instructions use cmd.exe but you can use the Visual Studio front end if you prefer.

unzip mysql-5.5.6-rc.zip

rem There is no separate 'debug' directory on Windows, as the CMake
rem infrastructure doesn't yet know to pull files in from there on Windows.
md release
cd release

rem Choose your target architecture, 32bit
set VSTARGET=Visual Studio 9 2008
rem or 64bit
set VSTARGET=Visual Studio 9 2008 Win64

cmake ..\mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release \
 -DCMAKE_BUILD_TYPE=Debug -G "%VSTARGET%"

devenv MySql.sln /build Debug

cmake ..\mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release \
 -DCMAKE_BUILD_TYPE=RelWithDebInfo -G "%VSTARGET%"

You can now choose which type of package to create. 5.5 includes new code to create minimal MSI packages, these should work ok, and only differ from the official MySQL MSI packages in that they do not include the instance config wizard.

To create the MSI packages you will need to install WiX.

rem Standard zip package
devenv MySql.sln /build RelWithDebInfo /project package

rem Full MSI package
devenv MySql.sln /build RelWithDebInfo /project msi

rem 'Essentials' MSI package
devenv MySql.sln /build RelWithDebInfo /project msi_essentials

Assuming everything goes ok, you should have some packages in the release\ directory.

Work still to be done

We still have a number of scripts only available internally, for example those we use to create SVR4, DMG and DEPOT packages. However, we are looking to integrate these into the MySQL Server source tree so that all users can benefit from them.

Enjoy!

'apt-get' and 5,000 packages for Solaris10/x86

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 29 Apr 2010 00:00:00 +0100

Here’s how:

# Install pkg_* tools and the 'pkgin' package manager
$ pkgadd -d http://www.netbsd.org/~jperkin/TNFpkgsrc-x86.pkg all

# Add tools to PATH
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH

# Update package repository (akin to 'apt-get update')
$ pkgin up

# Search for a particular package (you can use regexp)
$ pkgin search ^ap.*python 

# Install it
$ pkgin install ap22-py25-python

# Update all packages (akin to 'apt-get dist-upgrade')
$ pkgin full-upgrade

# How many packages are available?
$ pkgin avail | wc -l
   4970

Ok, so the headline might be slightly mis-leading, this isn’t really apt-get but a tool which is very similar. This is work which builds upon my previous post using pkgsrc to build binary packages on Solaris.

See http://imil.net/pkgin/ for more information on pkgin.

Hopefully this will prove really useful to people still using Solaris 10 and unable to use the new pkg(5) stuff in OpenSolaris.

Please try it out and provide any feedback to pkgsrc-users@netbsd.org. I’m hoping to keep the packages updated for the 2009Q3 branch.

ZFS and NFS vs OSX

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 16 Sep 2009 00:00:00 +0100

gromit is my Solaris 10 server, chorlton is my OSX desktop. Explain this:

root@gromit# zfs set sharenfs='anon=shared,sec=none' gromit/store

root@gromit# zfs get sharenfs gromit/store
NAME          PROPERTY  VALUE                 SOURCE
gromit/store  sharenfs  anon=shared,sec=none  local

user@chorlton$ ls /net/gromit/store
ls: cannot open directory /net/gromit/store/: Operation not permitted

root@gromit# zfs set sharenfs='anon=shared' gromit/store

user@chorlton$ ls /net/gromit/store
file1  file2

user@chorlton$ touch /net/gromit/store/file3
touch: /net/gromit/store/file3: Permission denied

root@gromit# zfs set sharenfs='anon=shared,sec=none' gromit/store

user@chorlton$ touch /net/gromit/store/file3
user@chorlton$ echo $?
0

pkgsrc on Solaris

jonathan@perkin.org.uk (Jonathan Perkin) — Sat, 12 Sep 2009 00:00:00 +0100

For many years, building and installing third-party software on Solaris has been a huge pain. For people who do not use pkgsrc, that is.

Originating from the NetBSD project, pkgsrc is a source-based cross-platform package manager. If you’ve used FreeBSD ports, then it is very similar as it derives from the same codebase, so the basic premise is that you:

$ cd /usr/pkgsrc/www/apache22
$ make package

and pkgsrc will download the source for Apache 2.2, compile it and all dependancies, install it on the local system and create a package which can be installed on other similar systems.

However, we’ve taken ports further and applied the NetBSD philosophy of portability, meaning that it not only works on NetBSD, but across all *BSD as well as Linux, OSX, HP/UX, AIX, IRIX, QNX, Windows (via Interix), and of course Solaris.

So while apt-get might be awesome, it only really works on Linux. FreeBSD might have way more ports than us, but only runs on FreeBSD and OSX. pkgsrc provides a consistent interface across all the platforms listed above, and in some cases provides a superior package manager than the system provides.

Here’s how I use pkgsrc on Solaris, in this specific case Solaris 10/x86. Paths are specific to my setup, you can of course change them.

Create a chroot/zone environment

I use the zones feature of Solaris 10 to ensure that all packages are built in a sandbox. This has a number of benefits:

The running system is unaffected by the builds, in that they are not writing to the same file system. This is good when you have misbehaving packages.
You can separate the build and install phases, so that you can verify all the packages have been built and are correct before starting any install/upgrade procedure
It’s easier to catch package mistakes, e.g. unpackaged files.
It avoids pollution from the host environment which may produce bad packages

For creating a zone, I wrote the following create-zone script:

#!/bin/sh

DOMAIN="adsl.perkin.org.uk"
MASQDOMAIN="perkin.org.uk"
PHYSIF="e1000g0"
RPOOL="gromit"

if [ $# -eq 1 ]; then
    name=$1; shift
    ipaddr=`getent hosts ${name} | awk '{print $1}'`
    if [ -z "${ipaddr}" ]; then
        echo "ERROR: Could not determine IP address of $name"
        echo "Either add to hosts database or provide on command line"
        echo
        echo "usage: $0 <name> [ <ipaddr> ]"
        exit 2
    fi
elif [ $# -eq 2 ]; then
    name=$1; shift
    ipaddr=$1; shift
else
    echo "usage: $0 <name> [ <ipaddr> ]"
    exit 2
fi

zfs list ${RPOOL}/zones/$name >/dev/null 2>&1
if [ $? -eq 0 ]; then
    echo "${RPOOL}/zones/$name already exists, not continuing"
    exit 1
fi

# Inherited directories are read-only
cat >/tmp/zonecfg.$$ << EOF
    create
    set zonepath=/zones/$name
    set autoboot=true
    add net
        set address=$ipaddr/24
        set physical=${PHYSIF}
    end
    verify
    commit
    exit
EOF

zonecfg -z $name -f /tmp/zonecfg.$$
rm /tmp/zonecfg.$$

zoneadm -z $name install

#
# Automatically configure new zone with most of the settings from the current
# global, grabbing the encrypted root password directly.  Note that the
# service_profile keyword doesn't seem to be used here, so we configure the
# limited_net profile manually and optionally install a custom site.xml which
# is parsed and activated during first boot.
#
cat << EOF >/zones/$name/root/etc/sysidcfg
network_interface=PRIMARY
{
    hostname=$name.${DOMAIN}
}
name_service=DNS
{
    domain_name=${DOMAIN}
    name_server=`awk '/^nameserver/ { print $NF; exit }' /etc/resolv.conf`
}
nfs4_domain=dynamic
root_password=`awk -F: '/^root/ {print $2}' /etc/shadow`
security_policy=NONE
service_profile=limited_net
system_locale=C
terminal=xterm
timezone=Europe/London
EOF

#
# Use the limited_net profile and install custom site.xml if provided.
#
rm /zones/${name}/root/var/svc/profile/generic.xml
ln -s generic_limited_net.xml /zones/${name}/root/var/svc/profile/generic.xml
if [ -f /install/zones/${name}.xml ]; then
    cp /install/zones/${name}.xml /zones/${name}/root/var/svc/profile/site.xml
fi

#
# Disable nscd host cache
#
ex /zones/$name/root/etc/nscd.conf >/dev/null 2>&1 <<EOF
/enable-cache/s/^#//
wq
EOF

#
# Configure sendmail to masquerade and route via smarthost.
#
ex /zones/$name/root/etc/mail/cf/cf/submit.mc >/dev/null 2>&1 <<EOF
/^dnl/
a
FEATURE(\`masquerade_envelope')dnl
MASQUERADE_AS(\`${MASQDOMAIN}')dnl
dnl
.
/^FEATURE.*msp/s/.127.0.0.1./mail.perkin.org.uk/
wq!
EOF
(
  cd /zones/$name/root/etc/mail/cf/cf
  /usr/ccs/bin/make submit.cf >/dev/null 2>&1
)
cp /zones/$name/root/etc/mail/cf/cf/submit.cf \
   /zones/$name/root/etc/mail/submit.cf

#
# Use SHA512 crypted passwords, and change root's home directory.
#
ex /zones/$name/root/etc/security/policy.conf >/dev/null 2>&1 <<EOF
/^CRYPT_DEFAULT/s/__unix__/6/
wq
EOF
mkdir -m 0700 /zones/$name/root/root
ex /zones/$name/root/etc/passwd >/dev/null 2>&1 <<EOF
/^root/s,:/:,:/root:,
/^root/s,:/sbin/sh,:/bin/bash,
wq
EOF

cp /root/.bash_profile /zones/$name/root/root/.bash_profile

#
# Allow root login over SSH (rest of network takes care of external access
# so this isn't a problem) and enforce SSH key authentication.
#
ex /zones/$name/root/etc/ssh/sshd_config >/dev/null 2>&1 <<EOF
/^PermitRootLogin/s/no$/yes/
/^PasswordAuthentication/s/yes$/no/
wq
EOF
mkdir -m 0700 /zones/$name/root/root/.ssh
cp /home/jperkin/.ssh/id_rsa.pub /zones/$name/root/root/.ssh/authorized_keys

#
# Automount shared directories from global zone using direct mounts
#
ex /zones/$name/root/etc/auto_master >/dev/null 2>&1 <<EOF
a
/-        auto_direct
.
wq
EOF
ex /zones/$name/root/etc/auto_direct >/dev/null 2>&1 <<EOF
a
/content    gromit-lan:/content
/install    gromit-lan:/install
.
wq
EOF

#
# All done!  Fire it up...
#
zoneadm -z $name boot

And the corresponding delete-zone script is:

#!/bin/sh

RPOOL="gromit"

if [ $# -ne 1 ]; then
    echo "usage: $0 <zone>"
    exit 2
fi

zoneadm -z $1 halt
zonecfg -z $1 delete -F
zfs destroy ${RPOOL}/zones/$1

If you want to use them then there are some variables to set at the top, and you may want to scan through them for additional bits to change, for example create-zone copies my ssh public key which will most likely be wrong for your setup :-)

One additional piece of configuration for create-zone is an optional SMF xml file. I use this file to disable inetd inside the zone for additional security, like so:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='profile' name='extract'>
  <service name='network/inetd' type='service' version='0'>
    <instance name='default' enabled='false'/>
  </service>
</service_bundle>

The file should be named <yourzonename>.xml. Mine is named vm-generic.xml and I then create symlinks to it for each VM I want with that default configuration.

Fetch pkgsrc

pkgsrc is developed very rapidly. Tracking nearly 9,000 pieces of third-party software means there are always many updates. Thankfully, we provide quarterly branches for people who want more stability, and I recommend using the latest quarterly release. At time of writing, this is known as pkgsrc-2009Q2. Within the next month or so we will release pkgsrc-2009Q3, and you can figure out the names of future releases yourself.

The easiest way to get pkgsrc is using cvs. I keep stuff like this under /content as opposed to the default of /usr, you can use whatever you wish but will need to change all my example scripts to match where you put it.

$ cd /content
$ BRANCH="pkgsrc-2009Q2"
$ cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -r${BRANCH} -d${BRANCH} -P pkgsrc

Alternatively, you can fetch either bzip2 or gzip archives of the current branch. I recommend the cvs method as, with the branch being updated for security fixes and other important changes, you can easily track it using

$ cd /content/pkgsrc-2009Q2
$ cvs update

pkgsrc configuration

pkgsrc is configured using a mk.conf file, this is mine:

#
# Local settings
#
OS_HOST!= uname -n

#
# Per-host settings
#
#    vm0 builds local packages
#
.if !empty(OS_HOST:Mvm0.*) || !empty(OS_HOST:Mgromit.*)
PKGDIRSUFFIX=
PKG_DEFAULT_OPTIONS=    inet6 perl sasl ssl ncursesw
PKG_OPTIONS.mutt+=      mutt-hcache mutt-smtp
PKG_UID.cyrus=          2000
#PKG_GID.cyrus=         mail
PKG_SHELL.cyrus=        /usr/bin/false
PKG_UID.www=            2001
PKG_GID.www=            2001
PYTHON_VERSION_DEFAULT= 24
X11_TYPE=               modular
#
#    vm1 modular X11 bulk builds
#
.elif !empty(OS_HOST:Mvm1.*)
X11_TYPE=       modular
PKGDIRSUFFIX=   -${X11_TYPE}
#
#    vm2 native X11 bulk builds
#
.elif !empty(OS_HOST:Mvm2.*)
X11_TYPE=       native
PKGDIRSUFFIX=   -${X11_TYPE}
#
#    vm3 pkgsrc dev builds
#
.elif !empty(OS_HOST:Mvm3.*)
X11_TYPE=       modular
PKGDIRSUFFIX=   -${X11_TYPE}
.endif

.if exists(../../CVS/Tag)
PKG_BRANCH!=    ${TOOLS_PLATFORM.awk} -F- '{print $$2}' < ../../CVS/Tag
.else
PKG_BRANCH=     HEAD
.endif

ALLOW_VULNERABLE_PACKAGES=  YES

BULKFILESDIR=           ${WRKOBJDIR}
DISTDIR=                /install/pkgsrc/distfiles/
FAILOVER_FETCH=         YES
INSTALL_UNSTRIPPED=     YES
LINTPKGSRC_DB=          ${_PKGSRCDIR}/.lintpkgsrc.db
MASTER_SITE_OVERRIDE=   ftp://ftp.netbsd.org/pub/NetBSD/packages/distfiles/
PACKAGES=               /install/pkgsrc/packages/${PKG_BRANCH}${PKGDIRSUFFIX}
PKGCHK_CONF=            /install/pkgsrc/misc/pkgchk.conf
SKIP_LICENSE_CHECK=     YES
WRKOBJDIR=              /tmp/pkgsrc

#
# Parse pkgchk.conf and supply list of packages for the bulk build framework.
#
.if defined(SPECIFIC_PKGS)
PKGLIST!= ${TOOLS_PLATFORM.awk} '$$1 !~ /^(\#|$$)/ {print $$1}' ${PKGCHK_CONF}
.  for _pkg_ in ${PKGLIST}
HOST_SPECIFIC_PKGS+= ${_pkg_}
.  endfor
.endif

This is the primary configuration file for pkgsrc. Again, you may need to tailor this to your environment, and may find it useful to read the pkgsrc guide to understand what it all means.

As I do a lot of pkgsrc development I have a number of virtual machines up and running doing various bits and pieces. Obviously I don’t want to copy that mk.conf around, so I also have a small fragment file which is appended to each virtual machine’s mk.conf (using the --mk-fragment argument to bootstrap) and includes the global copy:

.include "/install/pkgsrc/misc/mk.conf"

The bulk build setup in pkgsrc requires its own configuration, and for this you will need to edit a file inside pkgsrc. There is an example file provided, so what I usually do is symlink this to the real copy then I can easily keep it up-to-date via cvs.

$ cd /content/pkgsrc-2009Q2/mk/bulk
$ ln -s build.conf-example build.conf
$ vi build.conf

Again, you can find my personal build.conf here.

Finally, there is a configuration file for pkg_chk which is a package inside pkgsrc which makes managing upgrades easier (ideally it should be a part of the main pkgsrc tools but that’s for another day). pkgchk.conf is a list of package directories, relative to the pkgsrc top level, which are to be built and installed for this setup. If you have a large installation then pkg_chk has extra features to make it possible to share pkgchk.conf across a number of machines and configure packages on a per-host, per-OS etc basis.

Thus, a sample pkgchk.conf:

chat/irssi
chat/irssi-icb
converters/xlhtml
databases/lbdb
databases/rrdtool
devel/autoconf
devel/automake
devel/bzr
devel/bzrtools
devel/diffutils
devel/libtool-base
devel/scmcvs
editors/vim
mail/mutt-devel
misc/screen
net/p5-Net-SNMP
net/p5-Net-SNMP-Interfaces
net/rsync
net/samba
net/tnftp
pkgtools/digest
pkgtools/pkg_chk
pkgtools/rc.subr
print/mp
security/cy2-login
security/cy2-plain
security/sudo
sysutils/coreutils
textproc/antiword
textproc/grep
textproc/urlview
www/apache22
www/w3m

It is highly likely you will want to change the pkgchk.conf file from what I use :-)

Build scripts

Once everything is set up, I have two scripts to build then update my packages, intuitively called build-packages:

#!/bin/sh

PATH="/opt/pkg/sbin:/opt/pkg/bin:/sbin:/usr/sbin:/usr/bin:/usr/ccs/bin"
export PATH

BRANCH="2009Q2"
export BRANCH

PDIR=/install/pkgsrc

if [ ! -f ${PDIR}/bootstrap-${BRANCH}.tar ]; then
    cd /content/pkgsrc-${BRANCH}/bootstrap
    ./bootstrap --workdir=/tmp/pkgsrc --prefix=/opt/pkg \
                --varbase=/var/opt/pkg --sysconfdir=/etc/opt/pkg \
                --mk-fragment=${PDIR}/misc/mk-include.conf \
                --binary-kit=${PDIR}/bootstrap-${BRANCH}.tar
    rm -rf /tmp/pkgsrc
fi

if [ ! -f /opt/pkg/bin/bmake ]; then
    (cd /; tar -xf ${PDIR}/bootstrap-${BRANCH}.tar)
fi

cd /content/pkgsrc-${BRANCH}

/bin/ksh mk/bulk/build -s  2>&1 | tee -a ${PDIR}/logs/build-${BRANCH}.log
/bin/ksh mk/bulk/upload -n 2>&1 | tee -a ${PDIR}/logs/upload-${BRANCH}.log

and update-packages:

#!/bin/sh

PATH="/opt/pkg/sbin:/opt/pkg/bin:/sbin:/usr/sbin:/usr/bin:/usr/ccs/bin"
export PATH

BRANCH=2009Q2
export BRANCH

PKGSRCDIR=/content/pkgsrc-${BRANCH}
export PKGSRCDIR

if [ ! -f /opt/pkg/bin/bmake ]; then
    (cd /; tar -xf /install/pkgsrc/bootstrap-${BRANCH}.tar)
fi

# Run this anyway in case there are updates to pkg_chk
env PKG_PATH=/install/pkgsrc/packages/${BRANCH}/All pkg_add -u pkg_chk

# Dry run first
pkg_chk -aurbn

printf "Do you wish to install? [y/N]: "
read ans
case "${ans}" in
[Yy])
    pkg_chk -aurb 2>&1 | tee -a /install/pkgsrc/logs/pkg_chk-${BRANCH}.log
    ;;
esac

These are pretty simple as all the hard work has all been done. build-packages is ran inside the zone, then update-packages on the main host. These scripts hardcode the name of the branch currently used, so you will need to update this when moving to newer releases.

Quick recap

Ok, so here is the stuff I have for my setup and where I keep them:

Path	Description
`/content/pkgsrc-2009Q2`	Checked out pkgsrc tree, "2009Q2" branch
`/content/scripts/create-zone`	Creates Solaris zone
`/content/scripts/delete-zone`	Uninstalls and deletes zone
`/content/scripts/build-packages`	Bulk build packages inside the zone
`/content/scripts/update-packages`	Updates installed packages
`/install/pkgsrc/misc/mk.conf`	Main pkgsrc configuration file
`/install/pkgsrc/misc/mk-include.conf`	Fragment file included in each zone's `mk.conf`, sources the global `mk.conf`
`/install/pkgsrc/misc/pkgchk.conf`	`pkg_chk` configuration file
`/install/zones/vm-generic.xml`	Shared SMF configuration file, symlinked to from e.g. "`vm0.xml`"

And these are the paths where stuff will be created:

Path	Description
`/install/pkgsrc/distfiles`	Source tarballs of packages
`/install/pkgsrc/packages/2009Q2`	Resulting binary packages
`/tmp/pkgsrc`	Temporary build area for packages
`/content/vwww/www.adsl.perkin.org.uk/pkgstat`	Bulk build results directory

It’s definitely harder than it should be to get this all setup, but the good news is that once it’s done there’s very little maintenance.

Kicking it all off

Once everything is setup:

$ /content/scripts/create-zone vm0
$ ssh vm0 /content/scripts/build-packages
$ /content/scripts/update-packages
$ /content/scripts/delete-zone vm0

This should do the lot. Once build-packages has finished you should, if you configured your email address in build.conf, get an email with the bulk build results which looks similar to this:

http://mail-index.netbsd.org/pkgsrc-bulk/2009/08/23/msg006883.html

A fuller report is available if you configure a web server to serve the pkgstat directory created by the bulk build, and this can help debug problems (again see the above URL for an example).

You will need to add /opt/pkg/sbin:/opt/pkg/bin to $PATH. Configuration files are in /etc/opt/pkg, and log files and metadata are kept in /var/opt/pkg.

This stuff should be obsolete

While this all works well for me, it’s pretty lame for users who just want to install packages and have stuff work. I’m working on providing regular updates of binary packages, including a SVR4 package of the bootstrap kit, so that in theory all a user needs to do is

$ pkgadd TNFpkgsrc.pkg
$ pkg_add apache22
# Upgrade all installed packages to latest releases
$ pkg_chk -aurb

I’m almost there, just needs some tidying up and regular builds. Please feel free to help out!

Jumpstart from OSX

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 09 Dec 2008 00:00:00 +0000

I recently built a new file server on which I planned to install Solaris 10 10/08. I’m not a fan of CD/DVD installs, so wanted to jumpstart via PXE, though I only had OSX handy.

Here’s how I installed my new machine (gromit.adsl.perkin.org.uk/192.168.1.10) from my iMac (192.168.1.30) over the local network.

Note that since writing this piece I’ve updated to Solaris 10 10/09, and have changed the examples to use that instead.

Step 1, Prepare File System

First off, create a dedicated file system which we can export our jumpstart configuration from. You can probably skip this and just use any existing file system but this way everything is self-contained and we avoid NFS exporting more than we need.

We use HFSX to ensure that the file system is case sensitive, HFS+ can cause problems with pkgadd(1M).

$ hdiutil create -size 1g -type SPARSE -fs HFSX -volname "install" install
$ hdiutil attach install.sparseimage -mountpoint /install

Next up, download and mount sol-10-u8-ga-x86-dvd.iso.

$ open sol-10-u8-ga-x86-dvd.iso

Step 2, NFS

Share /install and the DVD via NFS with the correct options. -alldirs allows clients to mount from any point within that file system (which jumpstart requires), and -maproot=root is also required by jumpstart. As this allows root-owned files to be created, make sure you understand the security risks.

$ sudo vi /etc/exports

/install                  -alldirs -maproot=root
/Volumes/SOL_10_1009_X86  -alldirs -maproot=root

$ sudo nfsd checkexports && sudo nfsd enable

Step 3, DHCP

For DHCP I happen to already use my Cisco router as a DHCP server on the local network, so added the following configuration:

ip dhcp pool gromit.adsl.perkin.org.uk
   host 192.168.1.10 255.255.255.0
   hardware-address xxxx.xxxx.xxxx
   bootfile /boot/grub/pxegrub
   next-server 192.168.1.30
   client-name gromit
   domain-name adsl.perkin.org.uk
   dns-server xxx.xxx.xxx.xxx
   default-router 192.168.1.1

however, given this is a guide for setting everything up under OSX I also tried using ISC DHCP on OSX to prove it can be done that way too.

I used pkgsrc to install it (I’ll add another blog some time showing how to set up pkgsrc)

$ cd /usr/pkgsrc/net/isc-dhcpd
$ sudo bmake package

And here is my DHCP configuration file in full:

option domain-name "adsl.perkin.org.uk";
option domain-name-servers xxx.xxx.xxx.xxx;
ddns-update-style none;
authoritative;
log-facility local7;

subnet 192.168.1.0 netmask 255.255.255.0 {
    option routers 192.168.1.1;
}

group {
    filename "/boot/grub/pxegrub";
    next-server 192.168.1.30;

    host gromit {
        hardware ethernet xx:xx:xx:xx:xx:xx;
        fixed-address 192.168.1.10;
        option host-name "gromit.adsl.perkin.org.uk";
    }
}

Finally, start DHCP with:

$ sudo /usr/pkg/sbin/dhcpd

Most parts of these configurations should be self-explanatory. The /boot/grub/pxegrub entry is important for our next step, and I’d recommend using that exact pathname for reasons explained later.

Step 4, TFTP

Now, enable the TFTP server which comes with OSX. I added the -s option so tftpd would chroot to the tftpboot directory, both for security reasons and also to ensure that paths specified as /path/to/file would work correctly (relative to /install/tftpboot).

I also changed the location of the tftpboot directory so that everything was self-contained within the UFS image. In previous attempts I didn’t do this and ran into problems with GRUB which I think are again caused by case-insensitive file systems.

$ sudo vi /System/Library/LaunchDaemons/tftp.plist

[...]
    <key>ProgramArguments</key>
    <array>
        <string>/usr/libexec/tftpd</string>
        <string>-i</string>
        <string>-s</string>
        <string>/install/tftpboot</string>
    </array>
[...]

$ mkdir /install/tftpboot
$ sudo launchctl load -w /System/Library/LaunchDaemons/tftp.plist

You can then create a test file and check that it’s working as you expect, using:

$ echo "testing" >/install/tftpboot/testfile
$ printf "verbose\ntrace\nget testfile\n" | tftp localhost
$ rm /install/tftpboot/testfile

Step 5, GRUB

Next up, configure PXE booting using GRUB. We need to copy the GRUB images and configuration from the Solaris install DVD then modify it for our environment:

$ rsync -av /Volumes/SOL_10_1009_X86/boot/grub /install/tftpboot/boot/
$ rsync -av /Volumes/SOL_10_1009_X86/boot/multiboot /install/tftpboot/sol10u8x/
$ rsync -av /Volumes/SOL_10_1009_X86/boot/x86.miniroot /install/tftpboot/sol10u8x/

As we are copying the boot files from the DVD, they come hardcoded with particular pathnames to e.g. the menu.lst file. While it may be possible to pass extra parameters to pxegrub and load this from a different path, I simply recommend doing as I do and replicating the /boot/grub/ path structure so that everything Just Works.

The menu.lst file includes kernel arguments and allows you to choose which type of install to perform at startup. My file listed below has 3 choices:

Unattended install using a graphical environment (if available). The “install” keyword after the kernel instructs it to perform an unattended install, so long as it can find the necessary settings from sysidcfg etc.
As above, but force the use of the console and do not start a graphical environment (using the “nowin” keyword)
A manual install, so you need to go through the steps of layout out disks, selecting packages, etc.

$ vi /install/tftpboot/boot/grub/menu.lst

default=0
timeout=60

title Solaris PXE Unattended Install
    kernel /sol10u8x/multiboot kernel/unix - install -B \
      install_media=192.168.1.30:/Volumes/SOL_10_1009_X86,\
      sysid_config=192.168.1.30:/install/jumpstart,\
      install_config=192.168.1.30:/install/jumpstart
    module /sol10u8x/x86.miniroot

title Solaris PXE Unattended Install (console)
    kernel /sol10u8x/multiboot kernel/unix - install nowin -B \
      install_media=192.168.1.30:/Volumes/SOL_10_1009_X86,\
      sysid_config=192.168.1.30:/install/jumpstart,\
      install_config=192.168.1.30:/install/jumpstart
    module /sol10u8x/x86.miniroot

title Solaris PXE Manual Install
    kernel /sol10u8x/multiboot kernel/unix -B \
      install_media=192.168.1.30:/Volumes/SOL_10_1009_X86
    module /sol10u8x/x86.miniroot

Anyone used to doing jumpstart but with RARP/bootparams will notice the symmetry between install_config etc in the GRUB configuration and similar options in /etc/bootparams. Make sure that the full kernel arguments are all on one line, and that there are no spaces in between the install_media=..,sysid_config=.. options.

Step 6, Jumpstart

Finally, set up your Jumpstart configuration. Here’s what I personally use, you may want something different:

$ mkdir /install/jumpstart
$ cd /install/jumpstart
$ vi sysidcfg

name_service=DNS
{
    domain_name=adsl.perkin.org.uk
    name_server=xxx.xxx.xxx.xxx
}
network_interface=PRIMARY
{
    default_route=192.168.1.1
    netmask=255.255.255.0
    protocol_ipv6=yes
}
nfs4_domain=dynamic
root_password=xxxxxxxx
terminal=xterm
timeserver=localhost
timezone=Europe/London
security_policy=NONE
service_profile=limited_net
system_locale=C

Ordinarily this file is processed using a check script available in the jumpstart_sample directory on the Solaris DVD, however this only works from a Solaris host. To create the rules.ok file, we need to strip out any comments and put entries on one line, then create the checksum (although this isn’t actually necessary).

$ vi rules # emacs sucks :)

hostname gromit.adsl.perkin.org.uk - profile -

$ cp rules rules.ok
$ echo "# version=2 checksum=$(cksum -o 2 rules | awk '{print $1}')" >> rules.ok

Machine profile. This gives me a full Solaris install (minus OEM stuff) on mirrored ZFS disks with additional dump/swap space (the defaults made dump a bit too small I found).

$ vi profile

install_type    initial_install
pool            store auto 4g 4g mirror c1t0d0s0 c1t1d0s0
bootenv         installbe bename sol10u8x
cluster         SUNWCall

Step 7, Make A Cup Of Tea

With everything set up you should be able to enable PXE booting in your BIOS and watch it automatically install. One small minor problem you may have if you don’t have a BIOS which allows you to hit F12 or similar and choose PXE booting for one boot only is that it will infinitely cycle through installing, rebooting, installing, rebooting.. until you change your boot options.

If this happens, I recommend making more cups of tea until you happen to return in time to change the BIOS settings. If you aren’t able to do this for a while, you may need to add the extra steps 8, 9 and 10 titled “Visit The Bathroom”.

Set up local caching DNS server on OSX 10.4

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 31 Dec 2007 00:00:00 +0000

OSX 10.4 ships with ISC named, but is not configured correctly out of the box. Follow the instructions below to run a local caching name server.

First, we need to generate an rndc key. The default named.conf comes configured to use one, but if it does not exist will bail with an error. Simply run rndc-confgen(1) to create one.

$ sudo rndc-confgen -a

Next, download the latest named.root file, as the supplied named.ca is rather outdated (as you can see from the diff output).

$ cd /var/named
$ sudo curl -O ftp://ftp.internic.net/domain/named.root
$ diff -u named.ca named.root

Now edit /etc/named.conf, we need to make three changes. Firstly, the rndc port is wrong in the default configuration, so make it:

        inet 127.0.0.1 port 953 allow {any;}

Next, add a listen-on directive in the options section to instruct named to only listen on certain interfaces, ensuring that we keep as secure as possible. I have added the internal 192.168/16 LAN to my configuration so that Parallels VMs and other machines can use the server, but if you do not need this then just keep the localhost entry:

        directory "/var/named";
        listen-on {
                127.0.0.1;
                192.168/16;
        };

Finally, change the root DNS server file from named.ca to named.root as downloaded above. You could simply overwrite the named.ca file instead with named.root, however I like to keep defaults around as much as possible - you never know if an update will blat your changes or not:

zone "." IN {
        type hint;
        file "named.root";
};

All done. All that’s left to do is (re)start the service:

$ sudo service org.isc.named stop
$ sudo service org.isc.named start

then run a quick test to ensure it is working correctly:

$ dig www.perkin.org.uk @localhost

If you get a reply, configure your DNS server to be 127.0.0.1 and you’re all set.

Jonathan Perkin

Reducing RAM usage in pkgin

Trace heap allocations

Memory flame graph

Optimising download_file()

Optimising pkg_summary handling

decompress_buffer()

insert_summary()

Testing streaming updates

Final thoughts

Summary

August 2015 update (streaming pkg_summary)

pkgsrc-2014Q4: LTS, signed packages, and more

Long Term Support

Image Name Changes

Updated SmartOS Build Hosts

Signed Packages

Bundled pkg-vulnerabilities Verification

Reduced Package REQUIRES

Miscellaneous Improvements

Building packages at scale

pkgsrc bulk builds, 2004

pkgsrc bulk builds, 2010

pkgsrc @ Joyent, 2012 onwards

Performance improvements

pbulk distributed builds

Distributed chrooted builds

make -j vs number of chroots

cwrappers

Miscellaneous fork improvements

cat -> echo

Use shell substitution where possible

Test before exec

Batch up commands

Cache results

Avoid unnecessary tests

Miscellaneous system improvements

bash -> dash

Fix libtool to use printf builtin

Parallelise where possible

Initial package tools bootstrap

Package checksum generation

pkg_summary.gz generation

Summary

A node.js-powered 8-bit CPU - part four

Inverters

Hooking It All Up

RW On The 6309

PL On The 74HC165

Software

A node.js-powered 8-bit CPU - part three

6809/6309

Power (Vss/Pin1, Vcc/Pin7)

Clocks (E/Pin34, Q/Pin35)

Data Bus (D0-7/Pin31-24)

Address Bus (A0-A15/Pin8-23)

Interrupts (NMI/Pin2, IRQ/Pin3, FIRQ/Pin4)

TSC/Pin39

Halt/Reset (Reset/Pin37, Halt/Pin40)

Pin summary

Software

Start It Up!

A node.js-powered 8-bit CPU - part two

Shift Registers

Power (Vcc/Pin16, GND/Pin8)

Serial Input/Output (DS/Pin14, Q7S/Pin9)

Parallel Out (Q0-Q7/Pin15,1-7)

Clocks (SHCP/Pin11, STCP/Pin12)

Master Reset (MR/Pin10)

Output Enable (OE/Pin13)

Wiring it all up.

Software

Ship It!

74HC165

A node.js-powered 8-bit CPU - part one

Introduction

GPIO

Software

MDB support for Go

C Stack Traces

Optimising `download_file()`

Optimising `pkg_summary` handling

`decompress_buffer()`

`insert_summary()`

August 2015 update (streaming `pkg_summary`)

Bundled `pkg-vulnerabilities` Verification

`pkg_summary.gz` generation

`install` sub-directory

`pkginfo`

`pkgmap`

`reloc/` sub-directory