Jonathan Perkin

What's new in pkgsrc-2013Q1

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 15 Apr 2013 00:00:00 +0100

The latest branch of pkgsrc was released at the beginning of April, and binary packages for SmartOS/illumos and OSX are now available.

Instructions for installing, as well as a list of the major new features in pkgsrc-2013Q1 are below.

Installing

The instructions are similar to previous branches.

SmartOS/illumos

SmartOS users are encouraged to use our pre-built machine images, and installing your choice of base/standard image with version 13.1.x (available very soon) will get you a pkgsrc-2013Q1 based image.

For general illumos users or SmartOS users who want access to a full package set, the instructions are below:

# curl http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2013Q1-illumos.tar.gz | gtar -zxpf - -C /
# PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
# pkgin -y update
# pkgin avail | wc -l
    9842
# pkgin search <package>
# pkgin -y install <package> <package...>

OSX

Beginning with pkgsrc-2013Q1 I will now be providing regular builds for OSX. Again, the instructions are similar to those previously provided.

These packages are built on OSX Leopard (10.5) but use the PREFER_PKGSRC mechanism to ensure that they are portable across OSX releases, and have been successfully tested on OSX Lion (10.7).

$ curl http://pkgsrc.smartos.org/packages/Darwin/bootstrap/bootstrap-2013Q1-Darwin.tar.gz | sudo gnutar -zxpf - -C /
$ PATH=/usr/pkg/sbin:/usr/pkg/bin:$PATH
$ sudo pkgin -y update
$ pkgin avail | wc -l
    8108
$ pkgin search <package>
$ sudo pkgin -y install <package> <package...>

What’s New

As usual there were many hundreds of changes which went into this quarterly release of pkgsrc. Here are some of the more interesting and useful changes.

OpenSSL 1.0.1 with AES-NI support

OpenSSL has been upgraded from the 0.9.8 series to the 1.0.1 series. The driving reason to pursue this upgrade was to take advantage of AES-NI support which significantly improves crypto performance on Intel CPUs which provide that feature.

On my OSX 10.7 Core i7 laptop the numbers below speak for themselves:

: /usr/bin/openssl 'OpenSSL 0.9.8r 8 Feb 2011'
$ openssl speed -evp aes-128-cbc
  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
  aes-128-cbc     157297.05k   173874.73k   176805.45k   177719.17k   179441.78k

: pkgsrc openssl 'OpenSSL 1.0.1e 11 Feb 2013'
$ openssl speed -evp aes-128-cbc
  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
  aes-128-cbc     643315.29k   685811.37k   696899.67k   699977.39k   693968.90k

A pretty significant 4x improvement for many hundreds of applications which use OpenSSL for crypto.

GCC Go support for SmartOS/illumos

Go is a reasonably new programming language from Google that a number of our users have asked us to support, so we are pleased to announce that beginning with pkgsrc-2013Q1 you will be able to use the gccgo front-end to compile and run Go applications on SmartOS.

You simply compile the go source code as you would for any other language that GCC supports, for example:

# pkgin -y install gcc47

: /opt/pkg for the illumos package set, /opt/local for SmartOS datasets..
# PATH=/opt/pkg/gcc47/bin:$PATH

# gccgo app.go -o app
# ./app

Networking utilities on SmartOS

Thanks to initial work by @postwait there is now proper Zone support in libpcap, which has opened up the possibility to run a number of networking utilities in Joyent SmartMachines.

Yes, this means you can finally run tcpdump instead of snoop.

One of my favourites is trafshow which is a top-like interface for network, and looks like this:

Alternatively you can try nicstat for a more {io,mp,vm}stat style display.

Major package versions

As usual there was also a slew of version updates, and the most notable package versions are listed below. These of course are not exhaustive lists.

Development:

Clang 3.2
GCC 4.7.2
Git 1.8.1.5
Mercurial 2.5.2
Subversion 1.6.20, 1.7.8

Languages:

Lua 5.1.15
Node.js 0.8.23, 0.10.2
Ocaml 4.00.1
Oracle JRE/JDK 6.0.37, 7.0.15
Perl 5.16.2
PHP 5.3.23, 5.4.13
Python 2.6.8, 2.7.3, 3.1.5, 3.2.3, 3.3.0
R 2.15.1
Ruby 1.8.7.371, 1.9.3p392

Web Stack:

Apache 1.3.42, 2.0.64, 2.2.24, 2.4.4
CouchDB 1.2.1
MongoDB 2.2.2
MySQL 5.0.96, 5.1.67, 5.5.30, 5.6.10
Nginx 1.2.7, 1.3.14
PostgreSQL 8.3.23, 8.4.17, 9.0.13, 9.1.9, 9.2.4
Riak 1.2.1

Desktop:

evilwm 1.1.0
GNOME 2.32.1, 3.6.2
KDE 3.5.10, 4.8.4
XFCE 4.6.1

Enjoy!

Installing SVR4 packages on SmartOS

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 19 Mar 2013 00:00:00 +0000

Up until and including Solaris 10 the default packaging tools on Solaris were the historical SVR4 pkg* commands. First written in the early 1980s they were standard across commercial Unix systems and provided a simplistic interface to installing and removing binary packages.

With the introduction of IPS in OpenSolaris and beyond they have been mostly consigned to history, however there is still software provided for Solaris which is only available in the .pkg format, and thus it is useful to still be able to handle them.

Whilst the pkg* tools continue to be maintained in illumos and are provided by various distributions, they are not all provided in SmartOS. There are a few reasons for this:

SmartOS has a different design to other illumos distributions, and some key differences such as a read-only /usr mean that some packages will simply break in unexpected ways.
SmartOS is designed to be a slimmed-down distribution providing only that which is necessary for the majority of our users and use cases. Including the SVR4 tools and metadata would bloat the system.
SVR4 packages are often available only for older versions of Solaris, and whilst the excellent ABI compatability in Solaris means that the binaries themselves will often function correctly, the package may not support newer features such as SMF, or again make assumptions about the system which could result in irrevocable damage.
SmartOS uses pkgsrc to manage third-party software, and we believe it is better to convert SVR4 packages to pkgsrc format so that all packages on the system can be managed with a single toolset.

However, we do continue to ship the pkgtrans utility with SmartOS, and this is our gateway into converting SVR4 packages into more useful formats. The rest of this post will explore how we can do that.

Unpacking SVR4 packages

Let’s start with an example SVR4 package and unpack it to see what it contains. I’m going to use Riak, a popular open source database as the example package.

: Download the Solaris 10 SVR4 package from http://docs.basho.com/riak/latest/downloads/
$ curl -Os http://s3.amazonaws.com/downloads.basho.com/riak/1.3/1.3.0/solaris/10/BASHOriak-1.3.0-1-Solaris10-i386.pkg.gz

: Decompress it
$ gzip -d BASHOriak-1.3.0-1-Solaris10-i386.pkg.gz

: Use pkgtrans to unpack it into /var/tmp/BASHOriak
$ pkgtrans BASHOriak-1.3.0-1-Solaris10-i386.pkg /var/tmp all

SVR4 packages can contain multiple sub-packages, and so the ‘all’ is necessary to unpack everything in the archive. If we didn’t specify ‘all’, we would have seen:

$ pkgtrans BASHOriak-1.3.0-1-Solaris10-i386.pkg /var/tmp

The following packages are available:
  1  BASHOriak     riak
                   (i386) 1.3.0-1

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:

So whilst we could have specified ‘riak’, we can always use ‘all’ to avoid having to first look at the package to see what sub-packages it contains.

We now have an unpacked package, let’s go through what it contains.

`install` sub-directory

The install/ directory contains some files and scripts:

$ ls -l install/
total 29
-rw-------   1 admin    deniedssh   10175 Feb 19 15:17 copyright
-rw-------   1 admin    deniedssh     214 Feb 19 15:17 depend
-rwx------   1 admin    deniedssh     438 Feb 19 15:17 i.preserve
-rwx------   1 admin    deniedssh     339 Feb 19 15:17 preinstall
-rwx------   1 admin    deniedssh     469 Feb 19 15:17 r.preserve

copyright is self-explanatory, and is normally displayed when using the pkgadd command to let the admin know what they are agreeing to.
depend is a list of other SVR4 packages that this one depends upon. In this case they are:

$ cat install/depend
# Same dependencies as Erlang
P SUNWlibmsr    Math & Microtasking Libraries (Root)
P SUNWlibms     Math & Microtasking Libraries (Usr)
P SUNWopensslr  OpenSSL (Root)
P SUNWopenssl-libraries OpenSSL Libraries (Usr)

As this package is originally from Solaris 10 there is a chance that dependencies could cause issues. For example, in SmartOS we have updated OpenSSL to 1.0.x. Additionally, if a third-party dependency was required (i.e. one not beginning with SUNW) then naturally you would need to recursively apply this entire procedure to each dependency.

i.preserve and r.preserve are scripts executed during install (i.) and removal (r.). The ones for Riak simply try to retain modified files from an existing install, so we will ignore these as pkgsrc handles that by default.
preinstall is, as the name suggests, a script which is executed prior to installing the package. In Riak’s case it is used to create the ‘riak’ user and group if they do not already exist.

`pkginfo`

This provides some basic metadata about the package. The main bits we care about are:

ARCH=i386. As long as the package only depends upon libraries provided by the base OS (SUNW*) then it shouldn’t matter whether ARCH is 32-bit or 64-bit. However, if it requires third-party dependencies then you need to ensure that the correct ABI is provided.
BASEDIR=/opt. This is where the package would be installed by the pkgadd tool.
DESC=.... This would be output by the legacy pkginfo command, and we will re-use this text for our pkg_info description.
VERSION=1.3.0-1. Self-explanatory.

`pkgmap`

This is somewhat equivalent to the pkgsrc PLIST file and is a record of all the files the package provides, however it also includes file permissions and a basic checksum:

$ less pkgmap
: 1 112659
1 i copyright 10175 24223 1361287043
1 i depend 214 18268 1361287043
...
1 d none riak 0700 riak riak
1 d none riak/bin 0700 riak riak
1 f none riak/bin/riak 0755 riak riak 9041 51698 1361286795
...
1 e preserve riak/etc/app.config 0600 riak riak 14214 8625 1361286647

The last two lines, the important fields are:

i is an SVR4 metadata file, f or d denote whether it is a file or a directory, e are configuration files.
none means no special handling, preserve does just that, and the next field is the full path relative to reloc/
0700 and 0755 are the file/directory permissions
riak riak are the user and group ownership

We will need to ensure at least the file entries are handled correctly.

`reloc/` sub-directory

This directory contains the binaries etc. which make up the actual package. The contents of this directory would normally be installed under BASEDIR from the pkginfo file, so in Riak’s case:

: This..
$ ls reloc
riak
$ ls reloc/riak
bin         erts-5.9.1  etc         lib         releases

: ..would result in this
$ ls /opt/riak
bin         erts-5.9.1  etc         lib         releases

This concludes the examination of the SVR4 package. Let’s turn it into a useful pkgsrc package.

Creating pkgsrc binary package

For more information on creating binary pkgsrc packages from scratch, see this post.

pkgsrc metadata

Create the necessary pkgsrc metadata files.

$ mkdir /var/tmp/pkgsrc-riak
$ cd /var/tmp/pkgsrc-riak

: Standard build-info section.  Change MACHINE_ARCH to x86_64 if you are
: using a base64 image.
$ cat >build-info <<EOF
MACHINE_ARCH=i386
OPSYS=SunOS
OS_VERSION=5.11
PKGTOOLS_VERSION=20091115
EOF

: Generate comment file directly from the DESC field in pkginfo
$ awk -F= '/DESC/ {print $2}' < /var/tmp/BASHOriak/pkginfo >comment

: Generate PLIST directly from pkgmap
$ awk '$2 ~ /[ef]/ {print $4}' < /var/tmp/BASHOriak/pkgmap >plist

: For now just re-use DESC for the description file, however it would normally
: be longer
$ cp comment descr

pkgsrc INSTALL script

To handle the Riak preinstall script, we will create a pkgsrc INSTALL script.

The existing script can be mostly used as-is, we just need to put the entire contents of preinstall inside a PRE-INSTALL case statement so that it is executed prior to installing the package:

: Start with the existing preinstall script
$ cp /var/tmp/BASHOriak/install/preinstall inst

: Alter the script to create the 'riak' user/group during PRE-INSTALL, and
: after install to chown everything to 'riak' (which 
$ vi inst

PKGNAME="$1"
STAGE="$2"

case ${STAGE} in
PRE-INSTALL)
	# Existing preinstall script goes here, changing /opt references
	# to $PKG_PREFIX
	;;

If we recall from the pkgmap file, the entries there contained a user/group that each file should be owned by, and we can handle that in the INSTALL script too with a POST-INSTALL action:

POST-INSTALL)
	chown -R riak:riak ${PKG_PREFIX}/riak
	;;
esac

pkgsrc files

First we simply copy everything from the reloc/ directory to a files/ directory we will use for pkgsrc:

$ mkdir files
$ rsync -a /var/tmp/BASHOriak/reloc/ files/
$ chown -R root:root files

Next we can use the pkgmap file to ensure that the file modes are set correctly with a quick and dirty script:

while read line
do
    set -- $line
    case "$3" in
    [def])
        chmod $5 files/$4
        ;;
    esac
done < /var/tmp/BASHOriak/pkgmap

Create the package

We should now have everything necessary to create a binary package, taking the version from the pkgmap file.

$ pkg_create -B build-info -c comment -d descr -f plist -I /opt/local -i inst -p files -U riak-1.3.0.tgz

Testing

If all went well then we should be able to install the package:

$ pkg_add riak-1.3.0.tgz

and we will find it under /opt/local/riak as expected. If we try to run the binary, we get:

$ /opt/local/riak/bin/riak
/opt/local/riak/bin/riak: line 30: whoami: not found
sudo doesn't appear to be installed and your EUID isn't riak

This nicely proves my earlier point about packages often not working unmodified on SmartOS, in this case because whoami is no longer provided. Thankfully this is an easy fix, and we can simply change whoami to id -un.

Making that change and trying again, but this time as the riak user:

$ su - riak
$ /opt/local/riak/bin/riak
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
Usage: riak {start|stop|restart|reboot|ping|console|attach|chkconfig|escript|version|getpid}
$ /opt/local/riak/bin/riak start
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
$ pgrep -fl riak
20628 /opt/local/riak/erts-5.9.1/bin/epmd -daemon
20650 /opt/local/riak/erts-5.9.1/bin/beam.smp -K true -A 64 -W w -- -root /opt/local/
20648 /opt/local/riak/erts-5.9.1/bin/run_erl -daemon /tmp//opt/local/riak/ /opt/local
20718 /opt/local/riak/lib/os_mon-2.2.9/priv/bin/cpu_sup
20716 /opt/local/riak/lib/os_mon-2.2.9/priv/bin/memsup

This seems to work about as well as one can hope, and concludes my basic example.

Further work

I’ve covered the basics here, but there are additional things you could do to tidy up the conversion:

Fold the whoami fix back into the source file and re-generate the package.
Turn this into a real pkgsrc package, which would simplify some areas such as metadata and user creation.
Come up with a script to automate a lot of this work.
Turn the riak script into an SMF service.

Also note that Basho very helpfully already provide a native SmartOS package on their download page, so this example is somewhat pointless, however I hope it has still proven useful ;)

SmartOS is Not GNU/Linux

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 27 Feb 2013 00:00:00 +0000

One of the requests we get from time to time is for SmartOS to look more like GNU/Linux in layout and behaviour. For example, config files in /etc instead of /opt/local/etc, binaries under /usr instead of /opt/local/{s,}bin, GNU userland by default, etc.

Whilst we believe in the technical merits of our current implementation and the clean separation and upgrade possibilities it provides, we do recognise that some users just don’t care about those things and would prefer a system which looks as close to the GNU/Linux environments they are used to.

Ordinarily this simply wouldn’t be possible given that /usr is a read-only mount from the global zone, however with the highly flexible SmartOS Zones architecture, coupled with Joyent employing Zones guru Jerry Jelinek, we are able to provide you with an option to do exactly this. As Jerry says, at Sun there was even a native Linux brand, so pretty much anything is possible!

We call it ‘SNGL’ (pronounced ‘snuggle’), which is an acronym for ‘SmartOS is Not GNU/Linux’. Currently it is somewhat experimental, but we’d love for people to try it out and provide feedback.

Here’s how you can get it running.

Install the latest platform

You need to be running SmartOS 20130222 or later. Older platforms can be coerced into working, you will just need to work around the lack of this commit by creating an empty sngl_base.tar.gz or so.

As usual, follow the instructions here to upgrade an existing install.

Get the SNGL dataset

: Fetch the dataset image and manifest files.  The image is 107MB.
$ mkdir -p /usbkey/images
$ cd /usbkey/images
$ curl -O http://pkgsrc.smartos.org/datasets/sngl-0.99.0.dsmanifest
$ curl -O http://pkgsrc.smartos.org/datasets/sngl-0.99.0.zfs.bz2

: Import it
$ imgadm install -m sngl-0.99.0.dsmanifest -f sngl-0.99.0.zfs.bz2

Create a new dataset

The important point to note here is that brand is set to sngl.

: Create a new zone using the dataset (change your json to suit).
$ vmadm create <<EOF
{
  "brand": "sngl",
  "image_uuid": "4bf9530a-7ae5-11e2-bb4e-3bad5fbc3de9",
  "ram": 256,
  "quota": 10,
  "alias": "sngl-0.99.0",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}
EOF

At this point you should be able to log in and start using pkgin etc to install new software (there are over 2,000 packages available) as normal, but notice that:

binaries are running from /usr/bin
configuration files are in /etc
the default userland tools are GNU variants (ls, sed, awk, grep, etc.)

For those that are interested, here is some further detail on how this is all implemented.

Brand configuration

The main setup is in /usr/lib/brand/sngl. Firstly, platform.xml defines the mount points to be used inside the zone, and here you can see how we are able to use /usr:

        <global_mount special="/lib" directory="/system/lib"
            opt="ro,nodevices" type="lofs" />
        <global_mount special="/sbin" directory="/system/sbin"
            opt="ro,nodevices" type="lofs" />
        <global_mount special="/usr" directory="/system/usr"
            opt="ro,nodevices" type="lofs" />

We are transplanting the main system directories and mounting them under /system. This leaves /usr free for us to write packages to.

In order to support having the OS under /system there is some additional configuration in config.xml.

        <initname>/system/sbin/init</initname>
        <login_cmd>/system/usr/bin/login -z %Z %u</login_cmd>
        <forcedlogin_cmd>/system/usr/bin/login -z %Z -f %u</forcedlogin_cmd>
        <user_cmd>/system/usr/bin/getent passwd %u</user_cmd>

This is where the flexibility of Zones really shines. We are able to redefine the path to init(1M) and others so that the zone can boot correctly.

In addition, we copy in the crle configuration files ld.sys.config and ld.sys64.config so that binaries will look in /system/usr/lib for their runtime libraries.

Runtime and packages

The brand configuration is enough to set the zone up, but in order to make it boot we need additional files available under /usr, there are simply too many hardcoded paths. For this we just symlink back to /system/usr from /usr any files required.

Finally, we are able to perform a full pkgsrc bulk build with LOCALBASE set to /usr within a chroot which emulates this layout, and when those packages are installed they overwrite the compatability symlinks we have configured and replace them with files from the packages.

Not all symlinks will be overwritten, though, which is why standard SmartOS utilities such as prstat(1M) are still available, as the symlink for it still exists.

Reporting issues

As I mentioned, this is currently experimental, and there will be plenty of problems. However, at least from some initial testing, a reasonable amount of things appear to work fine, and for users who want this particular layout it may be good enough.

Please feel free to give it a try and report issues against our GitHub project. Once we have it working with a reasonable amount of stability we may be able to offer it as an option in the Joyent Public Cloud.

SmartOS development preview dataset

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 18 Feb 2013 00:00:00 +0000

The datasets we produce for SmartOS are usually released on a quarterly cadence, matching the upstream pkgsrc release branches. This is often enough to ensure that people can get up-to-date software with all the usual improvements and bug fixes.

Occasionally though, users want the very latest and don’t really want to wait for 3 months to get it, and so to satisfy those users who crave the bleeding edge I have produced a new dataset which is based upon pkgsrc trunk.

The package repository for this dataset will be constantly updated with the very latest that pkgsrc has to offer, and so you will occasionally see breakage as we integrate updates for core libraries and add new features. Think of it as being similar to Debian ‘unstable’, you get the very latest stuff but you may need to do some maintenance every so often.

Here are some reasons why you may want to use this dataset:

It is based upon the multiarch code, so only one image is necessary. No more needing to decide between the 32-bit or 64-bit options.
It is a full bulk build, so there are in the region of 9,000 packages to choose from (depending on the current state of pkgsrc trunk).
It has all the Joyent specific changes integrated, so that means SMF support for many packages which we have written manifests for, static UID/GID allocation, various improvements, and uses the standard /opt/local prefix.
There are some nice improvements only available in pkgsrc trunk, for example OpenSSL 1.0.1e with proper AES-NI support, and libpcap fixes from @postwait with a slew of networking utilities now available and working (these will be documented in a future post).

How to install

Here is a quick start guide to getting the preview dataset up and running:

: Fetch the dataset image and manifest files.  The image is 82MB.
$ mkdir -p /usbkey/images
$ cd /usbkey/images
$ curl -O http://pkgsrc.smartos.org/datasets/trunk-0.99.0.dsmanifest
$ curl -O http://pkgsrc.smartos.org/datasets/trunk-0.99.0.zfs.bz2

: Import it
$ imgadm install -m trunk-0.99.0.dsmanifest -f trunk-0.99.0.zfs.bz2

: Create a new zone using the dataset (change your json to suit).
$ vmadm create <<EOF
{
  "brand": "joyent",
  "image_uuid": "c91b3752-79c5-11e2-ad33-67667b9ee2c2",
  "max_physical_memory": 256,
  "alias": "trunk-0.99.0",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}
EOF

then login and start using it as you would with any other dataset.

Known issues

With this being a bleeding-edge distribution, there will undoubtedly be problems.The main one I am currently aware of is dependencies upon the GCC runtime from the /opt/pbulk prefix used to build the packages, which looks like this:

# pkgin in samba
calculating dependencies... done.
/opt/pbulk/gcc47/lib/./libgcc_s.so.1, needed by samba-3.6.12nb1 is not present in this system.
/opt/pbulk/gcc47/lib/./libgcc_s.so.1, needed by tdb-1.2.11 is not present in this system.
[...]

We are actively working on fixing these dependencies, but in the meantime if you require a package which is broken in this way you can work around it by installing the /opt/pbulk bootstrap like this:

# curl http://pkgsrc.smartos.org/packages/SmartOS/bootstrap/bootstrap-pbulk.tar.gz \
    | gtar -zxf - -C /
# /opt/pbulk/bin/pkgin -y install gcc47

The other issue to note is that not all packages have been converted to multiarch. Again we are actively working on this, but it will take some time to go through all of the 12,000 or so packages in pkgsrc.

If you find any other problems, please feel free to raise them against our GitHub project here. Or, even better, follow my guides on building pkgsrc (you will want the joyent/release/trunk branch) and have a go at fixing things yourself, we very much welcome patches and pull requests!

pkgsrc on SmartOS - fixing broken builds

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 17 Jan 2013 00:00:00 +0000

This is the second in a series of posts looking at pkgsrc on SmartOS. In the previous post I got us up and running with building packages. This post will focus on what to do when the build fails.

Currently pkgsrc is able to build around 9,500 packages on SmartOS, however pkgsrc carries over 12,000 packages in total, so there is a reasonable chance that you will come across a package which will not produce a binary package.

Let’s have a look at some of the most common failure modes and the facilities pkgsrc provides for them to be fixed.

Adjusting the environment

Probably the most common failures on Solaris are those caused by incorrect compiler or linker flags, where the author of the software has not taken into account differences across Unix platforms. Examples include:

the usage of u_int* types (e.g. u_int32_t) instead of the portable uint* C99 types (e.g. uint32_t).
missing -lsocket -lnsl when using the socket interface.

pkgsrc provides an easy way to pass CFLAGS, LDFLAGS and other common environment variables to the build, and these are often enough to resolve such issues.

Taking net/nsd as an example:

# cd net/nsd
# bmake package
...
gcc -I/opt/local/include -I/opt/local/include -I. -I. -O -I/opt/local/include -c ./util.c
./util.c:745:1: error: unknown type name 'u_int32_t'
*** [util.o] Error code 1

If we edit the Makefile and add the following line below CONFIGURE_ARGS:

CFLAGS.SunOS+=	-Du_int32_t=uint32_t

then a rebuild resolves the problem and results in a binary package:

# bmake clean
# bmake package
...
=> Creating binary package /content/packages/All/nsd-3.2.14.tgz

By using CFLAGS.SunOS rather than the global CFLAGS this is only performed on systems where uname is SunOS.

We can resolve missing libraries in a similar way, taking net/rootprobe as an example:

# cd net/rootprobe
# bmake package
...
gcc -Wl,-R/opt/local/lib  rootprobe.o -o rootprobe
Undefined                       first referenced
 symbol                             in file
recv                                rootprobe.o
send                                rootprobe.o
getsockname                         rootprobe.o
socket                              rootprobe.o
getdomainname                       rootprobe.o
connect                             rootprobe.o
recvfrom                            rootprobe.o
inet_aton                           rootprobe.o
inet_ntoa                           rootprobe.o
shutdown                            rootprobe.o
ld: fatal: symbol referencing errors. No output written to rootprobe
collect2: error: ld returned 1 exit status
*** [rootprobe] Error code 1

Add the following to the Makefile:

LDFLAGS.SunOS+=	-lsocket -lnsl

and hey presto, out comes a binary package:

# bmake clean
# bmake package
...
gcc -lsocket -lnsl -Wl,-R/opt/local/lib  rootprobe.o -o rootprobe
/bin/rm -f cctldprobe;  ln rootprobe cctldprobe
...
=> Creating binary package /content/packages/All/rootprobe-200301.tgz

Not all packages can be fixed directly like this, only those which obey the normal ${CC} and ${LD} environment variables, as pkgsrc creates wrappers for those commands where it can insert these alterations. For packages which directly call e.g. gcc, some additional digging will be required to see how the arguments can be passed.

Taking net/3proxy as an example, the build fails with missing socket libraries:

# cd net/3proxy
# bmake package
...
gcc -opop3p -Wall -O2 -pthread  sockmap.o pop3p.o sockgetchar.o myalloc.o common.o  
Undefined                       first referenced
 symbol                             in file
bind                                pop3p.o
send                                sockgetchar.o
getsockname                         common.o
accept                              pop3p.o
listen                              pop3p.o
gethostbyname                       common.o
sendto                              sockmap.o
socket                              pop3p.o
setsockopt                          pop3p.o
connect                             common.o
recvfrom                            sockmap.o
shutdown                            sockmap.o
ld: fatal: symbol referencing errors. No output written to pop3p
collect2: error: ld returned 1 exit status
*** [pop3p] Error code 1

but adding LDFLAGS.SunOS+= -lsocket -lnsl to the Makefile as before does not resolve the problem. Delving further into the Makefile we can see that the build is driven from a custom Makefile rather than through autoconf/automake:

MAKE_FILE=      Makefile.unix

and it is there we will need to perform the changes. First, let’s find the work area, which we can get from the WRKSRC variable:

# bmake show-var VARNAME=WRKSRC
/var/tmp/pkgsrc-build/net/3proxy/work
# cd /var/tmp/pkgsrc-build/net/3proxy/work
# ls -l Makefile.unix 
-rw-r--r-- 1 11001 10512 675 Apr 30  2005 Makefile.unix

Picking out the relevant bits from Makefile.unix:

CC = gcc
...
CFLAGS = -Wall -g -O2 -c -pthread -D_THREAD_SAFE -D_REENTRANT -DNOODBC -DWITH_STD_MALLOC -DFD_SETSIZE=4096 -DWITH_POLL
...
LDFLAGS = -Wall -O2 -pthread
...

Here, the author has provided no functionality for adding to the environment, and has instead chosen to hardcode a specific compiler and build flags, significantly reducing the portability of their software - good luck to Clang/LLVM or SunStudio users!

If the author had provided a way in to add to these flags, for example:

LDFLAGS = -Wall -O2 -pthread ${USER_LDFLAGS}

then we could have fixed that in the main pkgsrc Makefile with

MAKE_ENV+=	USER_LDFLAGS="-lsocket -lnsl"

or similar, however we now have no choice but to directly edit Makefile.unix. Thankfully, pkgsrc provides a couple of ways to easily do this, which we will look at over the next few sections.

The substitution framework

The subst framework allows basic editing of files within the work area. I will show the solution for the above problem, and then discuss it:

.include "../../mk/bsd.prefs.mk"
.if ${OPSYS} == "SunOS"
SUBST_CLASSES+=		libs
SUBST_STAGE.libs=	pre-build
SUBST_MESSAGE.libs=	Adding SunOS socket libraries
SUBST_FILES.libs=	Makefile.unix
SUBST_SED.libs=		-e '/^LDFLAGS/s/$$/ -lsocket -lnsl/'
.endif

Firstly, we need to limit this to SunOS systems, else we would break platforms which do not have libsocket or libnsl. Including ../../mk/bsd.prefs.mk gives us access to the OPSYS variable so we can test the platform we are running on.

Next we set up the substitution framework:

SUBST_CLASSES creates a new class, which I have named libs. This class name is then appended to the remaining SUBST_* variables to assign them to that class.
SUBST_STAGE defines the make stage when the substitution will be run, and should almost always be pre-build, to ensure it is done after any configuration stage which could rewrite files itself.
SUBST_MESSAGE is optional, and is simply a line which will be printed to the user when the substitution takes places.
SUBST_FILES is a list of files the substitution operates on.
SUBST_SED is the actual substitution, in the form of a sed(1) operation. Here we append -lsocket -lnsl to any line beginning with LDFLAGS. Note $$ is required to get make to escape a $.

# bmake clean
# bmake package
...
===> Building for 3proxy-0.5.3.11nb1
=> Adding SunOS socket libraries
...
=> Creating binary package /content/packages/All/3proxy-0.5.3.11nb1.tgz

Two other subst features you may want are:

SUBST_VARS.foo= VARNAME is a shortcut for the -e 's,@VARNAME@,${VARNAME},g' operation, common with autoconf-based packages.
SUBST_FILTER_CMD.foo= <cmd> allows you to run an arbitrary command, rather than the default of sed.

For more information, see the implementation in pkgsrc/mk/subst.mk.

Patches

While CFLAGS, LDFLAGS and the substitution framework allow for simple one-liner fixes, often more significant changes are required, and in those cases the only sensible option is to create patches. Thankfully, there are some tools provided in pkgsrc to make this a relatively easy process.

First, let’s take a broken package, devel/bglibs:

# cd devel/bglibs
# bmake package
...
./ltcompile net/cork.c
net/cork.c: In function 'socket_cork':
net/cork.c:39:27: error: 'SOL_TCP' undeclared (first use in this function)
net/cork.c:39:27: note: each undeclared identifier is reported only once for each function it appears in
*** [net/cork.lo] Error code 1

which comes from the following function:

/*
 * ..It is known to work on Linux (with the TCP_CORK option) and to at least
 * compile on BSD (with the TCP_NOPUSH option).  On OS's which lack either of
 * these two options, this function is essentially a no-op.
 */
int socket_cork(int sock)
{
#if defined(TCP_CORK)
  int flag = 1;
  return setsockopt(sock, SOL_TCP, TCP_CORK, &flag, sizeof flag) == 0;
#elif defined(TCP_NOPUSH)
  int flag = 1;
  return setsockopt(sock, SOL_SOCKET, TCP_NOPUSH, &flag, sizeof flag) == 0;
#else
  return 1;
#endif
}

Unfortunately, while SmartOS provides the TCP_CORK flag, it does not understand SOL_TCP, and so we need to modify the test and add an additional one.

Before doing this, we should install the pkgtools/pkgdiff package, which contains a pkgvi wrapper. This allows you to edit a file, and if you make changes, it will save the resulting diff in the pkgsrc patches directory ready for use. Much easier than delving into the work area and creating patches yourself.

: Install via pkgin..
# pkgin in pkgdiff

: ..or through pkgsrc
# (cd ../../pkgtools/pkgdiff && bmake install)

Now edit the offending file:

# pkgvi $(bmake show-var VARNAME=WRKSRC)/net/cork.c

This is what I changed the function to:

int socket_cork(int sock)
{
#if defined(TCP_CORK) && defined(SOL_TCP)
  int flag = 1;
  return setsockopt(sock, SOL_TCP, TCP_CORK, &flag, sizeof flag) == 0;
#elif defined(TCP_CORK) && defined(SOL_SOCKET)
  int flag = 1;
  return setsockopt(sock, SOL_SOCKET, TCP_CORK, &flag, sizeof flag) == 0;
#elif defined(TCP_NOPUSH)
  int flag = 1;
  return setsockopt(sock, SOL_SOCKET, TCP_NOPUSH, &flag, sizeof flag) == 0;
#else
  return 1;
#endif
}

This still isn’t ideal, and would be better converted to a proper autoconf test where we can test functionality instead of definitions, but it will do for example purposes.

After writing, you should get output such as:

pkgvi: File was modified. For a diff, type:
pkgdiff "/var/tmp/pkgsrc-build/devel/bglibs/work/bglibs-1.106/net/cork.c"

and running the pkgdiff command will show you the diff.

The final step is storing the diff into a patch file, and that is accomplished with:

# mkpatches
# bmake mps

The mkpatches command creates a new file in the patches directory called patch-net_cork.c, and the bmake mps (short for the makepatchsum target) regenerates the distinfo file with the correct checksum for that patch.

Finally, you can rebuild

# bmake clean
# bmake package

and, hey presto .. another failure!

--- net/uncork.lo ---
net/uncork.c: In function 'socket_uncork':
net/uncork.c:30:27: error: 'SOL_TCP' undeclared (first use in this function)
net/uncork.c:30:27: note: each undeclared identifier is reported only once for each function it appears in

Now that you know how to fix this, I’ll leave it to you to come up with a patch ;)

One final word on this, you will perhaps notice during the build some warnings:

=> Applying pkgsrc patches for bglibs-1.106nb1
=> Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-ab.orig
=> Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-ac.orig
=> Ignoring patchfile /content/pkgsrc/devel/bglibs/patches/patch-net_cork.c.orig

These files are left by mkpatches, and to remove them you can run

# mkpatches -c

Also note that both patch-ab and patch-ac changed, even though we only modified the net/cork.c file. This is due to differences in diff(1) output, and is ultimately harmless.

Summary

pkgsrc provides a number of ways to fix up broken software:

CFLAGS, LDFLAGS, and other environment variables.
The subst.mk framework for simple file changes.
Patch files for more substantial changes.

The important thing, after using any of these features, is to feed back your changes so that we can integrate them into pkgsrc and everyone can benefit. Probably the easiest way to do that is simply raise an issue against our GitHub pkgsrc fork, and either myself or Filip can commit the patch upstream.

pkgsrc on SmartOS - zone creation and basic builds

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 15 Jan 2013 00:00:00 +0000

Our goal at Joyent is that our binary packages for SmartOS fulfill all of our users’ needs. This isn’t always possible, however - users may want packages we do not yet provide, or build with different options.

To satisfy those demands, it should instead be reasonably straight-forward for users to build their own packages, and this guide hopefully provides all the information for them to do just that.

This is the first in a series of posts, and will focus on getting pkgsrc set up in a clean SmartOS zone and then building some packages. Other posts currently available in the series are:

How to fix build failures

Create a build machine

For this guide I use a base:1.8.1 SmartMachine, but any recent dataset should be fine, and it shouldn’t matter whether it’s 32-bit or 64-bit.

If you are a Joyent customer:

: Fill in your SDC details.  You can use the web interface too, if you prefer.
$ export SDC_CLI_ACCOUNT=account name
$ export SDC_CLI_IDENTITY=/path/to/ssh/key
$ export SDC_CLI_KEY_ID=key identifier
: .. or us-west-1 or us-sw-1 or eu-ams-1
$ export SDC_CLI_URL=https://us-east-1.api.joyentcloud.com

: You really really want json(1), it's awesome
$ sudo npm -g install jsontool

: Get list of 'base' datasets, we'll use the latest
$ sdc-listdatasets | json -a urn id | grep :base: | head -1
sdc:sdc:base:1.8.1 55330ab4-066f-11e2-bd0f-434f2462fada

: You can use sdc-listpackages to show available machine types.  I want one
: with good CPU performance.
$ sdc-createmachine --dataset 55330ab4-066f-11e2-bd0f-434f2462fada \
>   --name "pkgsrc-build-machine" --package "XL  8GB High CPU"

If you are using SmartOS:

{
  "brand": "joyent",
  "image_uuid": "55330ab4-066f-11e2-bd0f-434f2462fada",
  "max_physical_memory": 1024,
  "quota": 24,
  "alias": "pkgsrc-build-machine",
  "hostname": "pkgsrc-build-machine",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}

Put the above json into a file, you may want to tweak some values, then

# imgadm update
# imgadm import 55330ab4-066f-11e2-bd0f-434f2462fada
# vmadm create -f /path/to/json/file

Finally, log into the freshly-created zone.

Install dependencies

First thing you’ll need in the zone are build tools, so:

# pkgin -y up
# pkgin -y in gcc47 scmgit-base

Fetch pkgsrc

We maintain a fork of pkgsrc on github, which includes some additional SmartOS fixes and improvements, so we recommend using that rather than upstream.

I put everything under /content, adjust to your own tastes.

# mkdir /content
# cd /content
# git clone git://github.com/joyent/pkgsrc.git

Next, check out the latest stable branch. Our naming scheme is joyent/release/YYYYQQ, and the latest branch at time of writing is pkgsrc_2012Q4, so:

# cd pkgsrc
# git checkout joyent/release/2012Q4

If you want to be particularly adventurous, you could checkout trunk and get the very latest pkgsrc code, however pkgsrc is a very fast-moving target and you may run into unexpected breakages, and you will certainly end up rebuilding packages very often.

Fetch pkgsrc-wip

pkgsrc-wip is a set of additional work-in-progress packages for pkgsrc, and can be useful for trying out the latest packages. It is designed to be extracted directly into pkgsrc, and we maintain a git submodule of it in our release trees, so all you need to do, assuming you are using a joyent/release/* tree, is:

# git submodule init
# git submodule update

This will also get you a joyent/ directory which is a module of our pkgsrc-joyent tree of additional packages.

If you are on trunk or another tree which does not have the wip submodule, you can fetch it manually with:

# git clone git://github.com/joyent/pkgsrc-wip.git wip

Configure mk.conf

The primary method of configuring pkgsrc is through the /opt/local/etc/mk.conf file. The base image comes with one populated to handle the basic layout, but there are some additional basic variables you may want to set prior to building your first package:

DISTDIR determines where source tarballs are stored, default pkgsrc/distfiles.
PACKAGES is where pkgsrc will store binary packages it has built, default pkgsrc/packages.
WRKOBJDIR is where pkgsrc performs the builds, default pkgsrc/<category>/<package>/work.
ALLOW_VULNERABLE_PACKAGES and SKIP_LICENSE_CHECK should both be set to yes if you just want the package to be built, and don’t care whether it is currently vulnerable to security issues or is released under a specific license.
MAKE_JOBS determines the argument to make -j, i.e. the number of concurrent make threads, default 1.
FETCH_USING sets the default program to use for fetching source code, defaulting to an in-tree version of NetBSD’s ftp program. Setting this to curl on SmartOS, which is included in the base platform, will avoid any dependencies.
BINPKG_SITES is a URL pointing to binary packages which can be used by this tree rather than building everything from source, and settings DEPENDS_TARGET to bin-install enables using these binary packages as dependencies.

Thus, this is how I would configure mk.conf:

# mkdir -p /content/{distfiles,packages}
# vi /opt/local/etc/mk.conf

DISTDIR=	/content/distfiles
PACKAGES=	/content/packages
WRKOBJDIR=	/var/tmp/pkgsrc-build
#
ALLOW_VULNERABLE_PACKAGES=	yes
SKIP_LICENSE_CHECK=		yes
#
MAKE_JOBS=	8
#
FETCH_USING=	curl
#
# Adjust BINPKG_SITES, depending upon the dataset chosen, it should be the URL
# from /opt/local/etc/pkgin/repositories.conf without the trailing 'All'.
BINPKG_SITES=	http://pkgsrc.joyent.com/sdc6/2012Q2/i386/
DEPENDS_TARGET=	bin-install

Building a package

Ok, let’s get started and build a package. pkgsrc is organised into categories, with all packages following the pkgsrc/<category>/<package> layout, and everything is driven with bmake, the BSD implementation of make(1). I’m going to choose nmap as an example, as it will show a couple of things I want to cover.

First, finding it. I tend to just do a simple:

# ls -d */*nmap*
net/nmap

If you want a more featured search, you can do:

# bmake search key=nmap

though the first time you run this it creates the INDEX file it requires, and that can take a long time. Another option is to use the pkgsrc.se web interface.

Once you have found the package, cd into the package directory.

# cd net/nmap

Next, let’s look at any options the package supports.

# bmake show-options

If the package supports build options, as net/nmap does, you’ll see something like:

Any of the following general options may be selected:
	inet6	 Enable support for IPv6.
	ndiff	 Enable tool to compare Nmap scans.
	zenmap	 Enable nmap GUI frontend.

These options are enabled by default:
	inet6

These options are currently enabled:
	inet6

You can select which build options to use by setting PKG_DEFAULT_OPTIONS
or PKG_OPTIONS.nmap.

The place to configure these options is again in /opt/local/etc/mk.conf, so if you want to enable ndiff support for example, then you’d add this to mk.conf:

PKG_OPTIONS.nmap=	ndiff

Re-running the show-options command should now print:

...
These options are currently enabled:
        inet6 ndiff
...

And finally, we can go ahead and build the package. The output from this will be long, so you may want to tee it to a file for reviewing:

# bmake install 2>&1 | tee /var/tmp/nmap.log

Assuming this completes ok, you should note the main stages that make up a package build:

bootstrap-depends comes first, and installs all the dependencies required for pkgsrc to get started. pkgtools/digest is required to calculate the SHA1 and RMD160 checksums of the source tarball. If we hadn’t specified FETCH_USING=curl in our mk.conf then it’s likely that net/tnftp would have been pulled in during this stage too and used to fetch the source.

=> Bootstrap dependency digest>=20010302: NOT found
=> Verifying bin-install for ../../pkgtools/digest
===> Binary install for digest>=20010302
=> Installing digest>=20010302 from /content/packages/All;http://pkgsrc.joyent.com/sdc6/2012Q2/i386//All
pkg_add: Can't process file:///content/packages/All/digest*: No such file or directory
digest-20111104 successfully installed.
...

fetch and checksum then run to download the source tarball for this particular package, and then verify the checksum matches that stored by pkgsrc, to ensure it was downloaded from a good source:

=> Fetching nmap-6.01.tar.bz2
=> Total size: 21640157 bytes
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.6M  100 20.6M    0     0   699k      0  0:00:30  0:00:30 --:--:--  721k
=> Checksum SHA1 OK for nmap-6.01.tar.bz2
=> Checksum RMD160 OK for nmap-6.01.tar.bz2

depends then installs all packages required for both build and runtime for the package in question:

=> Build dependency libtool-base>=2.2.6bnb3: NOT found
=> Verifying bin-install for ../../devel/libtool-base
===> Binary install for libtool-base>=2.2.6bnb3
=> Installing libtool-base>=2.2.6bnb3 from /content/packages/All;http://pkgsrc.joyent.com/sdc6/2012Q2/i386//All
libtool-base-2.2.6bnb6 successfully installed.
...

extract and patch then unpack the source and apply any pkgsrc patches to the package. The patches are located in the patches/ sub-directory for each package:

===> Extracting for nmap-6.01nb5
===> Patching for nmap-6.01nb5

The bulk of the build is performed by configure and build which for most software will consist of ./configure && make.

===> Configuring for nmap-6.01nb5
...
checking whether NLS is requested... yes
...
Configuration complete.  Type make (or gmake on some *BSD machines) to compile.
...
===> Building for nmap-6.01nb5
...
gmake[1]: Leaving directory `/var/tmp/pkgsrc-build/net/nmap/work/nmap-6.01'
=> Unwrapping files-to-be-installed.

The final steps are package and install, firstly package does a fake install of the package to a temporary install directory and creates a binary package from that, and then install installs that binary package into place:

===> Building binary package for nmap-6.01nb5
=> Creating binary package /content/packages/All/nmap-6.01nb5.tgz
===> Install binary package of nmap-6.01nb5

You can now verify it is installed, and start using your new software:

# type nmap
nmap is /opt/local/bin/nmap

# nmap -p 22 localhost

Starting Nmap 6.01 ( http://nmap.org ) at 2013-01-15 15:36 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00012s latency).
PORT   STATE SERVICE
22/tcp open  ssh

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

Cleanup

Once everything is working, you may want to clean up the build areas. The pkgsrc way of doing this is:

# bmake clean clean-depends

However, if you just want to blow everything away without prejudice, you can simply remove everything under WRKOBJDIR, so in my case:

# rm -rf /var/tmp/pkgsrc-build/*

This should hopefully be enough to get you started. Over the next few posts we’ll investigate some more advanced topics. Stay tuned!

Multi-architecture package support in SmartOS

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 10 Jan 2013 00:00:00 +0000

[Updated on 2013-01-16 to use the 12.4.1 dataset which fixes some issues.]

Ever since the release of Solaris 7 back in 1998, Solaris has had the ability to run both 32-bit and 64-bit binaries on the same machine. Even now, 15 years later, with much of the world 64-bit only, there are still reasons to retain 32-bit support:

32-bit binaries can be faster in many cases, and if you do not need the additional address space afforded by the 64-bit version then there may be no advantage to running it.
Some software you depend upon may only provide a 32-bit version, or may have better compatability in that mode.
It’s sometimes nice to have a hard 4GB memory limit on a runaway process instead of it completely trashing your machine ;)

Unfortunately, while the base SmartOS platform is set up to provide both 32-bit and 64-bit binaries, the packaging infrastructure we use, pkgsrc, has not traditionally supported building multi-architecture packages. This has meant we have needed to provide both 32-bit and 64-bit versions of each dataset, which is not ideal:

It’s confusing to customers and users, who may think it applies to the kernel and platform version.
Users want the choice to be able to run 32-bit for some applications and 64-bit for others, all on the same machine.
It is additional work and maintenance for us.

In order to resolve this, I have been working on providing multi-architecture support to pkgsrc, and this work is now available for preview testing.

Getting started

Here is a quick start guide to getting the multi-architecture dataset up and running:

: Fetch the dataset image and manifest files.  The image is 85MB.
$ mkdir -p /usbkey/images
$ cd /usbkey/images
$ curl -O http://pkgsrc.smartos.org/datasets/multiarch-12.4.1.dsmanifest
$ curl -O http://pkgsrc.smartos.org/datasets/multiarch-12.4.1.zfs.bz2

: Import it
$ imgadm install -m multiarch-12.4.1.dsmanifest -f multiarch-12.4.1.zfs.bz2

: Create a new zone using the dataset (change your json to suit).
$ vmadm create <<EOF
{
  "brand": "joyent",
  "image_uuid": "ee1fb198-5fe1-11e2-9cce-e319fd47df7b",
  "max_physical_memory": 256,
  "alias": "multiarch-12.4.1",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}
EOF

then login and start using it as you would with any other dataset.

Multi-architecture libraries

Libraries are reasonably straight-forward. For most packages which provide shared libraries, you should find both 32-bit and 64-bit libraries are included, for example:

$ pkg_info -L sqlite3 | grep 'so$'
/opt/local/lib/libsqlite3.so
/opt/local/lib/amd64/libsqlite3.so

Other directories under lib/ should be handled correctly too, such as .pc files for pkg-config:

$ pkg_info -L sqlite3 | grep 'pc$'
/opt/local/lib/pkgconfig/sqlite3.pc
/opt/local/lib/amd64/pkgconfig/sqlite3.pc

Multi-architecture binaries

Binaries are similar, but follow a different layout scheme, and have additional controls to allow the user to select which architecture to use.

The basic layout is:

$ pkg_info -L sqlite3 | grep bin
/opt/local/bin/i86/sqlite3
/opt/local/bin/amd64/sqlite3
/opt/local/bin/sqlite3

In order to explain this, let us look at how the base platform supports multi-architecture binaries, using dtrace as an example:

$ ls -li /usr/sbin/*/dtrace /usr/sbin/dtrace /usr/lib/isaexec
4255 -r-xr-xr-x 72 root bin 12776 Dec 28 02:38 /usr/lib/isaexec
6330 -r-xr-xr-x  1 root bin 52728 Dec 28 02:38 /usr/sbin/amd64/dtrace
4255 -r-xr-xr-x 72 root bin 12776 Dec 28 02:38 /usr/sbin/dtrace
6640 -r-xr-xr-x  1 root bin 41544 Dec 28 02:38 /usr/sbin/i86/dtrace

$ file /usr/sbin/*/dtrace /usr/lib/isaexec
/usr/sbin/amd64/dtrace: ELF 64-bit LSB executable AMD64 Version 1, dynamically linked, not stripped, no debugging information available
/usr/sbin/i86/dtrace:   ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available
/usr/lib/isaexec:       ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available

The i86 and amd64 hold the per-architecture binaries, and the main /usr/sbin/dtrace command is a hardlink to the /usr/lib/isaexec wrapper (as shown by the inode being identical). This wrapper detects whether the running kernel is 32-bit or 64-bit, and calls the appropriate native binary, which nowadays will almost certainly be the 64-bit version.

For the pkgsrc implementation, I needed a way to override this behaviour so that users could select to run the 32-bit version if so desired, without having to munge their $PATH. To do this I took a copy of isaexec and added it to pkgsrc, with additional support for an ABI environment variable.

You can see the behaviour below with the calls to execve().

Default is to run 64-bit

$ truss -t execve sqlite3 -version
execve("/opt/local/bin/sqlite3", 0x08047DA8, 0x08047DB4)  argc = 2
execve("/opt/local/bin/amd64/sqlite3", 0x08047DA8, 0x08047DB4)  argc = 2
3.7.15 2012-12-12 13:36:53 cd0b37c52658bfdf992b1e3dc467bae1835a94ae

Set ABI=32 or ABI=i86 to run the 32-bit version

$ ABI=32 truss -t execve sqlite3 -version
execve("/opt/local/bin/sqlite3", 0x08047D9C, 0x08047DA8)  argc = 2
execve("/opt/local/bin/i86/sqlite3", 0x08047D9C, 0x08047DA8)  argc = 2
3.7.15 2012-12-12 13:36:53 cd0b37c52658bfdf992b1e3dc467bae1835a94ae

Note that not all binaries have been converted to multi-architecture. In fact, the majority have been left as plain 32-bit binaries. While all libraries ultimately have to be provided for both architectures so that users can choose to compile their own software against either, 64-bit binaries only make sense for certain classes of software:

Databases such as SQLite, and other servers which may require >4GB address space.
Language interpreters.
Software which provides a foo-config script with hardcoded references to libdir.

Compiler support

The GCC 4.7.2 package provided has been made aware of this layout, and will add the correct library paths depending upon the ABI you target. For example:

$ cat >test.c <<EOF
int main(){}
EOF

$ gcc -m32 test.c -o test32 -lsqlite3
$ ldd test32 | grep libsqlite3
        libsqlite3.so.0 =>       /opt/local/lib/libsqlite3.so.0

$ gcc -m64 test.c -o test64 -lsqlite3
$ ldd test64 | grep libsqlite3
        libsqlite3.so.0 =>       /opt/local/lib/amd64/libsqlite3.so.0

No additional flags or linker settings should be required (if they are, let me know!)

Interpreter support

For those interpreters which have been converted, their respective module systems should be multi-architecture aware:

Perl already had reasonable support for multi-architecture files, and all perl modules provided should be enabled, for example:

$ pkgin -y in p5-Digest-SHA1
$ pkg_info -L p5-Digest-SHA1 | grep 'so$'
/opt/local/lib/perl5/vendor_perl/5.16.0/i386-solaris-thread-multi/auto/Digest/SHA1/SHA1.so
/opt/local/lib/perl5/vendor_perl/5.16.0/x86_64-solaris-thread-multi-64/auto/Digest/SHA1/SHA1.so
$ ABI=32 perl -MDigest::SHA1 -e 'print'
$ ABI=64 perl -MDigest::SHA1 -e 'print'

Python is similar to Perl, but needed a lot more work to support multi-architecture modules. Modules provided by pkgin should work fine, but there may be issues with locally-built modules - let me know!
node is built as a multi-architecture binary, but npm has received no special handling. This means you need to be careful not to mix-and-match modules. I don’t perceive this to be too much of an issue, as the node community appears to have settled on having one node_modules per application, but again let me know if there are better ways to handle this.

Coverage

This work is incomplete, hence it not being available in the default datasets yet, but a large number of packages have been converted:

Pretty much every package which provides shared libraries.
Python, Perl, Lua and node.js interpreters (and a significant number of modules).
MySQL 5.5, PostgreSQL, SQLite and DB4 databases, Apache 2.2.

Notable exceptions currently are Ruby and PHP, and I will be working on those in due course. There may also be a number of packages which should include multi-architecture binaries, please let me know if I have missed any obvious candidates.

You can raise issues against our GitHub project here.

Enjoy!

Solaris portability - cfmakeraw()

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 09 Jan 2013 00:00:00 +0000

Converting from flock() to fcntl() unfortunately wasn’t enough to get tmux to build on Solaris, there was one additional failure:

client.c: In function 'client_main':
client.c:246:3: warning: implicit declaration of function 'cfmakeraw' [-Wimplicit-function-declaration]

...

Undefined                       first referenced
 symbol                             in file
cfmakeraw                           client.o
ld: fatal: symbol referencing errors. No output written to tmux
collect2: error: ld returned 1 exit status
*** [tmux] Error code 1

Let’s look at the code:

	struct termios tio;
...
	cfmakeraw(&tio);
	tio.c_iflag = ICRNL|IXANY;
	tio.c_oflag = OPOST|ONLCR;
...

Running man cfmakeraw on my OSX laptop I can read what this function does, and where it is implemented (edited for brevity):

SYNOPSIS
     void
     cfmakeraw(struct termios *termios_p);

DESCRIPTION
     The cfmakeraw() function sets the flags stored in the termios structure
     to a state disabling all input and output processing, giving a ``raw I/O
     path''.

STANDARDS
     The cfmakeraw() and cfsetspeed() functions, as well as the TCSASOFT option
     to the tcsetattr() function are extensions to the IEEE Std 1003.1-1988
     (``POSIX.1'') specification.

Ok, so all it’s doing is setting some additional flags in the tio structure, and a quick search in the NetBSD OpenGrok gives us their implementation (again edited for brevity):

void
cfmakeraw(struct termios *t)
{
...
	t->c_iflag &= ~(IMAXBEL|IGNBRK|BRKINT|PARMRK|ISTRIP|INLCR|IGNCR|ICRNL|IXON);
	t->c_oflag &= ~OPOST;
	t->c_lflag &= ~(ECHO|ECHONL|ICANON|ISIG|IEXTEN);
	t->c_cflag &= ~(CSIZE|PARENB);
	t->c_cflag |= CS8;
...
}

All that’s left is to patch the tmux code to set those flags. A good patch would add a feature test to the tmux autoconf setup and/or provide a compatability macro, but in this case I simply add a Solaris-specific section to the code:

	struct termios tio;
...
#ifdef __sun
	tio.c_iflag &= ~(IMAXBEL|IGNBRK|BRKINT|PARMRK|ISTRIP|INLCR|IGNCR|ICRNL|IXON);
	tio.c_oflag &= ~OPOST;
	tio.c_lflag &= ~(ECHO|ECHONL|ICANON|ISIG|IEXTEN);
	tio.c_cflag &= ~(CSIZE|PARENB);
	tio.c_cflag |= CS8;
#else
	cfmakeraw(&tio);
#endif
	tio.c_iflag = ICRNL|IXANY;
	tio.c_oflag = OPOST|ONLCR;
...

Summary

cfmakeraw() is an extension to POSIX and thus not fully portable. If you use it, please consider including a compatability version for systems which do not implement it.

Solaris portability - flock()

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 08 Jan 2013 00:00:00 +0000

This is the first of what I hope will be a regular series of posts looking at software portability when it comes to Solarish (Solaris, illumos, SmartOS etc.) systems. We will begin with flock(), as it’s what I happen to have been fixing today.

The most recent version of tmux fails with the following errors:

client.c: In function 'client_get_lock':
client.c:81:2: warning: implicit declaration of function 'flock' [-Wimplicit-function-declaration]
client.c:81:20: error: 'LOCK_EX' undeclared (first use in this function)
client.c:81:20: note: each undeclared identifier is reported only once for each function it appears in
client.c:81:28: error: 'LOCK_NB' undeclared (first use in this function)

from this section of code:

int
client_get_lock(char *lockfile)
{
        int lockfd;

        if ((lockfd = open(lockfile, O_WRONLY|O_CREAT, 0600)) == -1)
                fatal("open failed");

        if (flock(lockfd, LOCK_EX|LOCK_NB) == -1 && errno == EWOULDBLOCK) {
                while (flock(lockfd, LOCK_EX) == -1 && errno == EINTR)
                        /* nothing */;
                close(lockfd);
                return (-1);
        }

        return (lockfd);
}

This code is reasonably straight-forward, trying to create a lock on a file descriptor, and if it isn’t able to waits until the lock is released then fails, ready for the calling function to retry.

However, the flock() routine is of BSD heritage and does not exist on Solaris. There used to be a compatability version as part of the /usr/ucb environment, but that no longer exists in newer versions.

Thankfully, there is an alternative interface which has existed for as long as flock() but with the added benefit of being standardised by POSIX and thus much more portable - enter fcntl()!

Let’s look at how the two are similar-but-different, and then show how we can change this code to be more portable.

flock() implementation

As we can see from the NetBSD manual page below, flock() is very simple, and this is probably the main reason why people choose to use it over fcntl() for file locking.

NAME
     flock -- apply or remove an advisory lock on an open file

LIBRARY
     Standard C Library (libc, -lc)

SYNOPSIS
     #include <fcntl.h>

     #define   LOCK_SH   1    /* shared lock */
     #define   LOCK_EX   2    /* exclusive lock */
     #define   LOCK_NB   4    /* don't block when locking */
     #define   LOCK_UN   8    /* unlock */

     int
     flock(int fd, int operation);

There isn’t much to it, you call flock() specifying either a shared or exclusive lock, with an option to make the call non-blocking. Afterwards you unlock the previously held lock.

The tmux code in question uses LOCK_NB for the initial call, then if that fails reverts to the default blocking operation so that the client can retry as soon as the lock is released.

Use fcntl() instead!

In contrast to flock(), setting up fcntl() is a bit more involved as it allows finer-grained control over the locking.

Set up struct flock

There is a flock structure which controls the lock, defined as:

struct flock {
        off_t       l_start;    /* starting offset */
        off_t       l_len;      /* len = 0 means until end of file */
        pid_t       l_pid;      /* lock owner */
        short       l_type;     /* lock type: read/write, etc. */
        short       l_whence;   /* type of l_start */
};

So, let’s set that up:

int
client_get_lock(char *lockfile)
{
	int lockfd;
	struct flock lock;

	lock.l_start = 0;
	lock.l_len = 0;
	lock.l_type = F_WRLCK;
	lock.l_whence = SEEK_SET;
...

Setting l_start and l_len both to 0 means ‘lock the entire file’, l_whence of SEEK_SET means we are setting absolute values rather than relative, and we set l_type to be a write lock. If we wanted a read lock, we’d use F_RDLCK here instead.

Note that l_pid is only used for a F_GETLK operation - we are only interested in attempting to set a lock, so it is left unset.

Add fcntl() calls

The synopsis for fcntl() is:

SYNOPSIS
     #include <fcntl.h>

     int
     fcntl(int fd, int cmd, ...);

and the commands available for file locking are (from BSD):

#define F_GETLK         7               /* get record locking information */
#define F_SETLK         8               /* set record locking information */
#define F_SETLKW        9               /* F_SETLK; wait if blocked */

Note there is no ‘clear lock’ command. To clear a lock, you use the F_GETLK command with l_type set to F_UNLCK.

So, to show how we would rewrite the flock() instances to fcntl() instead, I’ve put them together below:

#ifdef __sun
	if (fcntl(lockfd, F_SETLK, &lock) == -1 && errno == EAGAIN) {
		while (fcntl(lockfd, F_SETLKW, &lock) == -1 && errno == EINTR)
			/* nothing */;
		close(lockfd);
		return(-1);
	}
#else
	if (flock(lockfd, LOCK_EX|LOCK_NB) == -1 && errno == EWOULDBLOCK) {
		while (flock(lockfd, LOCK_EX) == -1 && errno == EINTR)
			/* nothing */;
		close(lockfd);
		return (-1);
	}
#endif

As you can see, they are very similar, making it relatively straight-forward to rewrite code to use the more portable fcntl().

An important note on semantics

There are two important difference between flock() and fcntl() you need to be aware of which may affect a simple conversion:

fcntl() locks are not held across a fork(), so you cannot pass locks down to child processes.
The semantics of fcntl() are such that any closure of a file descriptor in your application will release the locks held against that file. This is best illustrated with a lock on /etc/passwd that gets released if you call getpwname() as that opens and closes the /etc/passwd file.

Generally such semantics do not apply, but you should be aware of them, and it always pays to carefully read the manual pages, preferably those from BSD.

flock() -> fcntl() cheat sheet

To aid your own conversions, here are some further examples (without error checking for clarity) of flock() and fcntl() equivalents.

#if defined(USING_FLOCK)

	/* Blocking */
	flock(lockfd, LOCK_SH);  // Shared (read) lock
	flock(lockfd, LOCK_EX);  // Exclusive (write) lock

	/* Non-blocking */
	flock(lockfd, LOCK_SH|LOCK_NB);  // Shared (read) lock
	flock(lockfd, LOCK_EX|LOCK_NB);  // Exclusive (write) lock

	/* Release */
	flock(lockfd, LOCK_UN):

#elif defined(USING_FCNTL)

	struct flock lock;

	lock.l_start = 0;
	lock.l_len = 0;
	lock.l_whence = SEEK_SET;

	/* Blocking */
	lock.l_type = F_RDLCK;
	fcntl(lockfd, F_SETLKW, &lock);  // Shared (read) lock
	lock.l_type = F_WRLCK;
	fcntl(lockfd, F_SETLKW, &lock);  // Exclusive (write) lock

	/* Non-blocking */
	lock.l_type = F_RDLCK;
	fcntl(lockfd, F_SETLK, &lock);  // Shared (read) lock
	lock.l_type = F_WRLCK;
	fcntl(lockfd, F_SETLK, &lock);  // Exclusive (write) lock

	/* Release */
	lock.l_type = F_UNLCK;
	fcntl(lockfd, F_GETLK, &lock);

#endif

flock() wrapper

Alternatively, there is a flock() wrapper which is used by the NetBSD toolchain, which you could include in a compatability library or so:

int flock(int fd, int op) {
	int rc = 0;

#if defined(F_SETLK) && defined(F_SETLKW)
	struct flock fl = {0};

	switch (op & (LOCK_EX|LOCK_SH|LOCK_UN)) {
	case LOCK_EX:
		fl.l_type = F_WRLCK;
		break;

	case LOCK_SH:
		fl.l_type = F_RDLCK;
		break;

	case LOCK_UN:
		fl.l_type = F_UNLCK;
		break;

	default:
		errno = EINVAL;
		return -1;
	}

	fl.l_whence = SEEK_SET;
	rc = fcntl(fd, op & LOCK_NB ? F_SETLK : F_SETLKW, &fl);

	if (rc && (errno == EAGAIN))
		errno = EWOULDBLOCK;
#endif

	return rc;
}

It might be handy if the illumos folks merged this, it would immediate fix at least 19 packages in the pkgsrc collection :)

Summary

flock() is simpler, and retains locks across fork() and concurrent access boundaries, but at the cost of portability. Please try to use fcntl() where possible, it isn’t much harder to use, and makes your software run on more platforms.

pkgsrc-2012Q4 illumos packages now available

jonathan@perkin.org.uk (Jonathan Perkin) — Sun, 06 Jan 2013 00:00:00 +0000

In keeping with the regular quarterly pkgsrc releases, I’m pleased to announce that packages from the pkgsrc-2012Q4 branch are now available for general illumos platforms.

As usual, the quick start instructions are:

$ curl -s http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2012Q4-illumos.tar.gz \
    | gtar -zxf - -C /
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
$ pkgin -y up
$ pkgin avail | wc -l
    9518
$ pkgin search <package>
$ pkgin -y install <package> <package...>

For those interested in helping to increase the number of packages available, the bulk build report for this set is available here. It would be great to get past the 10,000 mark!

SmartOS and the global zone

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 23 Nov 2012 00:00:00 +0000

One of the most common issues new users of SmartOS face is understanding the role and design of the global zone. Often they will download SmartOS and try to start using it as they would any other Unix operating system, but quickly run into basic problems such as installing packages or adding users. However, SmartOS is not your usual operating system, and it is imperative that you understand two key principles:

SmartOS is specifically designed as an OS for running Virtual Machines, not as a general purpose OS.
The global zone is effectively a read-only hypervisor, and should only be used for creating and managing Virtual Machines. Everything else should be performed inside Virtual Machines.

I should be clear at this point that I am specifically addressing the issue of downloading and running your own SmartOS installation. If you provision a SmartMachine from Joyent, then you are running inside a Virtual Machine, and this post does not apply - you are free to start serving up awesome applications!

Let’s look at a few aspects of the global zone which make it a great fit for its intended purpose, and help explain why it doesn’t work as you might expect.

The global zone is a ramdisk

This is the key reason why things don’t work as you might expect. SmartOS does not install to disk like other operating systems, instead it boots directly from USB/CD/PXE into a mostly read-only environment.

Why is it done this way? There are a number of good reasons:

Upgrades are trivial. No more patching, just reboot into a new image!
Increased disk space. No wasted space on disk to hold the OS, all the space is dedicated to VMs and user data.
Increased disk performance. It’s common with other systems to have your OS installed to a pair of mirrored disks and then pool the remaining disks for data. With SmartOS you can have all your disks in the same RAIDZ pool, increasing performance.
Additional security. Most of the system files are read-only, and /etc is re-created on each boot, making it much harder to exploit.
Increased stability. Ever had your root disks start to fail and system commands no longer run? This doesn’t happen on SmartOS.
Much simpler to install and provision, especially when you have a large number of machines.

So, what does this look like, and what are the implications?

# zonename
global
# df -h / /usr
Filesystem             size   used  avail capacity  Mounted on
/devices/ramdisk:a     264M   219M    45M    83%    /
/devices/pseudo/lofi@0:1
                       376M   354M    22M    95%    /usr
# lofiadm
Block Device             File                           Options
/dev/lofi/1              /usr.lgz                       Compressed(gzip)

The root file system is a ramdisk, and /usr is a read-only loopback mount of a single compressed file held on the ramdisk.

Apart from a few specific files and directories listed below, this means that you cannot change the global zone. To emphasise this point further:

You cannot add users.
You cannot write anywhere under /usr.
You cannot permanently store or change files under /etc, /root, ..
Changes to SMF services will be reset each reboot.

There are plenty of other things you cannot do, but hopefully this gives you some idea of the restricted nature of the global zone environment.

If you are looking for a general purpose OS to serve as a NAS etc, then you can do worse than give OmniOS a spin - it has most of the same features as SmartOS, but in a normal install-to-disk configuration.

Having said all that, let’s look at some ways you can manage the global zone, if you have settled on running SmartOS.

So what can I do?

Firstly, let’s look at the key writeable areas:

Filesystem             size   used  avail capacity  Mounted on
zones                  2.0T   143M   1.8T     1%    /zones
zones/var              2.0T   4.0G   1.8T     1%    /var
zones/opt              2.0T    55K   1.8T     1%    /opt
zones/usbkey           2.0T   154K   1.8T     1%    /usbkey
/usbkey/shadow         1.8T   154K   1.8T     1%    /etc/shadow
/usbkey/ssh            1.8T   154K   1.8T     1%    /etc/ssh

zones is the big zpool which is spread across all your disks, and SmartOS will create a number of file systems on it for use in the global zone:

/zones

This is where your Virtual Machines are stored, the datasets used to create them, and some other key files related to zones. Don’t mess around in here unless you know what you are doing.

/var

We all like logs, so those are retained in /var as normal, as well as various state files such as the current list of imgadm datasets and the SSH host keys.

/opt

/opt is your main hook into more advanced setup of the global zone. SmartOS will import any SMF manifests it finds in /opt/custom/smf, which allows you to implement rc.local functionality as demonstrated in this gist.

As /opt is writeable you can also install packages as per this wiki page, however you need to bear in mind that, as explained earlier, things such as adding users are not possible, so you may see various errors.

/usbkey

When you first boot SmartOS and go through the rudimentary installer the details you provide are stored in /usbkey/config, which is used during boot to configure the machine. If you need to change any of those variables, this is the file you should edit.

There are also some variables which aren’t set up by default, so if you want to

Install an authorized_keys file for the root user.
Set a keyboard map.

then have a read of this post I wrote a while ago to find out how to configure those.

/etc/shadow and /etc/ssh

In order to support some very basic configuration, a few /etc files are their own mount points onto /usbkey so that changes to them are saved.

/etc/shadow so that you can change the root password.
/etc/ssh is initially where the SSH host keys were stored, however they are now stored under /var/ssh so don’t be surprised if this mount point disappears at some point.

Implementation

While we’re on the subject, let’s complete this post with a look at how the global zone is implemented, for interested readers.

GRUB

There is a good wiki guide which describes how to mount the USB key. Once you have done that you can take a look at the GRUB configuration:

: Assumes your current working directory is the USB key mountpoint.
# less boot/grub/menu.lst
...
title Live 64-bit (text)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='<crypt>',smartos=true
   module /platform/i86pc/amd64/boot_archive

title Live 64-bit (noinstall)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='<crypt>',standalone=true,noimport=true
   module /platform/i86pc/amd64/boot_archive
...

There are a few boot configuration variables which alter how the system is started, and you can see how they are used by grepping for them in the SMF init scripts under /lib/svc.

root_shadow=’crypt string’

This configures the default root password prior to /etc/shadow being mounted, and changes for each release. If you need to know what it is, then browse the download site and look at the SINGLE_USER_ROOT_PASSWORD.release.txt which correlates with your uname -v output, or if you wish you could generate your own password and paste the new crypted string in here.

Alternatively as a convenience you can find the default root password inside the platform directory:

# cat platform/root.password
seitee3oome4aiPh

smartos=true

This variable defines whether to perform the normal SmartOS global zone initialisation, such as mounting /usbkey and configuring the system from the /usbkey/config file (see /lib/svc/method/fs-joyent and /lib/svc/method/smartdc-config.

If left unset (like in the ‘Live 64-bit (noinstall)’ boot option), these will not be performed. standalone=true should be set in those cases to avoid trying to be a Joyent compute node.

noimport=true

Setting this will skip any configuration and mounting of zpools, and is useful for if you have issues during the first installation and need to check disks, etc.

It’s likely that you do not need to change any of these variables, and if you hit problems simply use the noinstall boot option and figure things out from there.

Platform image

The only other files on the USB key are:

# find platform -type f
platform/i86pc/kernel/amd64/unix
platform/i86pc/amd64/boot_archive
platform/i86pc/amd64/boot_archive.gitstatus
platform/i86pc/amd64/boot_archive.manifest

These are mostly self-explanatory:

unix is the SmartOS kernel.
boot_archive is the ramdisk image containing the entire OS.
boot_archive.gitstatus contains the tip revisions of the github repositories used to build that particular image.
boot_archive.manifest contains MD5 checksums of the OS files.

Upgrades

Given we have just explained about the platform directory, it is also worth pointing out that you do not have to reflash your USB key every time, which is great if your server is inaccessible. Instead, assuming your USB key is big enough, you can simply download the newest platform-latest.tgz file, move the existing platform directory out of the way, and unpack platform-latest.tgz there instead. Doing this as an atomic operation is preferred, for example:

: As of writing this is 20121115T191935Z
# gtar zxf ~/platform-latest.tgz
# mv platform platform-$(uname -v) && mv platform-20121115T191935Z platform

This avoids having no platform directory, just in case you have a power cut at that exact point in time!

Summary

If you want to set up a file server or similar, then SmartOS is probably not for you. However, if you are interested in running Virtual Machines and are able to do all of your work inside them, then SmartOS is perfect for that purpose, and has a number of advantages over other operating systems.

Hopefully this has been useful, even if it is to deter you from using SmartOS!

Setting up Samba on SmartOS

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 24 Oct 2012 00:00:00 +0100

A frequent question on #smartos IRC is how to set up SmartOS as a file server. Due to the architecture of SmartOS and its focus on virtualisation, this isn’t as easy as on other systems, and some parts are simply not supported at this time (e.g. NFS).

This guide, therefore, is a simple way to get Samba up and running so that you can at least use a SmartOS machine as a file server, even if it’s perhaps not the protocol you would initially choose.

Create new virtual machine

Let’s start by creating a VM, here is what I use locally, you may want to tweak the settings for your environment. If you already have a suitable VM configured you can just skip this step.

# Import the base1.8.1 image, though any base image will suffice.
$ imgadm update
$ imgadm import 55330ab4-066f-11e2-bd0f-434f2462fada

# I store all my VM configs in this directory
$ mkdir /usbkey/vmcfg

# vmadm configuration for 'store' VM
$ vi /usbkey/vmcfg/store.json

{
  "brand": "joyent",
  "zfs_io_priority": 30,
  "image_uuid": "55330ab4-066f-11e2-bd0f-434f2462fada",
  "max_physical_memory": 256,
  "quota": 750,
  "alias": "store",
  "hostname": "store",
  "resolvers": [
    "193.178.223.141",
    "208.72.84.24"
  ],
  "dns_domain": "adsl.perkin.org.uk",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.1.11",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.1"
    }
  ]
}

# Create the VM
$ vmadm create -f /usbkey/vmcfg/store.json

Install Samba package

Next, log in to the VM and install the required packages. These are not currently available from the default repository, so we will use the generic illumos package set:

$ zlogin <uuid>

# Download and unpack bootstrap kit
$ curl -s http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2012Q3-illumos.tar.gz \
    | gtar -zxf - -C /
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH

# Install latest Samba package (others are available if necessary)
$ pkgin -y up
$ pkgin -y install samba-3.6

Configure Samba

This part will differ based on your requirements, here are a couple of simple examples.

Shared guest mount with full read/write access

For if you just want somewhere to share stuff on a local network with no security.

# Create shared mount user and mountpoint.
$ groupadd -g 500 store
$ useradd -u 500 -g 500 -c "Store user" -s /usr/bin/false -d /store store
$ mkdir /store
$ chown store:store /store

# Configure Samba
$ vi /etc/opt/pkg/samba/smb.conf

[global]
  security = share
  load printers = no
  guest account = store

; Comment out [homes] section

[store]
  path = /store
  public = yes
  only guest = yes
  writable = yes
  printable = no

Shared mount with per-user access

For a bit more fine-grained control.

# Create per-user accounts and mountpoint.
$ groupadd -g 1000 alice
$ useradd -u 1000 -g 1000 -c "Alice" -s /usr/bin/false -d /store alice
$ groupadd -g 1001 bob
$ useradd -u 1001 -g 1001 -c "Bob" -s /usr/bin/false -d /store bob
$ mkdir /store
$ chmod 1777 /store
$ /opt/pkg/bin/smbpasswd -a alice
$ /opt/pkg/bin/smbpasswd -a bob

# Configure Samba
$ vi /etc/opt/pkg/samba/smb.conf

[global]
  security = user
  load printers = no

; Comment out [homes] section

[store]
  path = /store
  valid users = alice bob
  public = no
  writable = yes
  printable = no

Access will be via username and the password set with smbpasswd. Users will be able to create files and read other user files, but will only be able to modify files they created.

Startup scripts

Okay, so I’m lazy and whilst I should provide some SMF scripts and manifests for this, it’s simpler to just write an rc.d script! :-)

$ vi /etc/rc2.d/S99samba

#!/bin/sh

case "$1" in
start)
	# Start up Samba daemons
	/opt/pkg/sbin/nmbd -D
	/opt/pkg/sbin/smbd -D
	;;
stop)
	pkill -9 nmbd
	pkill -9 smbd
	;;
reload)
	pkill -HUP nmbd
	pkill -HUP smbd
	;;
esac

Finally, start it up!

$ chmod +x /etc/rc2.d/S99samba
$ /etc/rc2.d/S99samba start

Multicast DNS

In order for shares to automatically show up in e.g. the OSX Finder, you will need to be running some kind of mDNS service on the server.

The easiest solution is to simply enable dns/multicast in the zone, i.e.:

$ svcadm enable dns/multicast

This will then show up based on the hostname of the server, and clicking on it should show the store mount we created.

All done

I hope this proves useful!

pkgsrc-2012Q3 packages for illumos

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 10 Oct 2012 00:00:00 +0100

Continuing from my previous bulk build for pkgsrc-2012Q2 I’m pleased to announce that packages from the pkgsrc-2012Q3 branch are now available.

Quick start:

$ curl -s http://pkgsrc.smartos.org/packages/illumos/bootstrap/bootstrap-2012Q3-illumos.tar.gz \
    | gtar -zxf - -C /
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
$ pkgin -y up
$ pkgin avail | wc -l
    9542
$ pkgin search <package>
$ pkgin -y install <package> <package...>

As usual there have been many hundreds of updates since the previous branch, and hopefully the software you need is included. If not, please get involved! We welcome new contributors.

The bootstrap kit and a number of packages have been tested successfully on OpenIndiana 151a and OmniOS.

Enjoy!

Creating local SmartOS packages

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 23 Aug 2012 00:00:00 +0100

Ok, so you have a SmartOS machine set up, and are using pkgin to install packages. Now you may want to also handle your local software using the same tools, so that you can take advantage of easily installing and upgrading packages. This post describes how you can do this.

We will be creating a hypothetical package meminfo, containing a script of the same name which prints basic memory information, and just enough metadata for it to be understood by the packaging tools. This guide assumes you are running a standard dataset which includes packaging tools in /opt/local/sbin, that is:

$ /opt/local/sbin/pkg_info

should give you a list of currently installed packages.

Files to install

First, let’s start in a clean directory, and then create a files sub-directory which will hold the files we want to package. The files in this directory will be installed relative to the packaging prefix, which for most SmartOS installs will be /opt/local.

$ mkdir package
$ cd package
$ mkdir -p files/bin

Now we create the meminfo script. You will probably just copy your binaries into place at this stage.

$ vi files/bin/meminfo

#!/usr/bin/bash
#
# This is just a subset of 'sm-summary' from the 'smtools' package.
#
SM_UUID=$(zonename);
SM_ID=$(zoneadm list -p | awk -F: '{ print $1 }');
SM_MEMCAP=$(kstat -p -c zone_caps -n lockedmem_zone_${SM_ID} -s value | awk '{print $2/1024/1024 }');
SM_MEMUSED=$(prstat -Z -s rss 1 1 | awk -v zone=${SM_UUID} '$8 ~ zone { printf("%d", $4 ) }');
SM_MEMFREE=$(echo "${SM_MEMCAP}-${SM_MEMUSED}" | bc);
cat <<EOF
Memory (RSS) Cap    ${SM_MEMCAP}M
Memory (RSS) Used   ${SM_MEMUSED}M
Memory (RSS) Free   ${SM_MEMFREE}M
EOF

Ensure the script is executable!

$ chmod +x files/bin/meminfo

Finally for this part, generate a packlist file which simply contains a list of files we want packaged, relative to the files directory:

$ (cd files; find * -type f | sort) >packlist

The packlist file supports many directives which are lines beginning with @<cmd>. These let you do things such as change file permissions once the file has been installed. See the ‘PACKING LIST DETAILS’ section of pkg_create(1) for more information.

Package metadata

In order to successfully create a package, we need a few metadata files which describe the package.

build-info

This file contains basic information about the package, and is primarily used to ensure that the package can be installed on the target machine. The minimum information required is:

MACHINE_ARCH. On SmartOS this is either i386 or x86_64 depending upon whether you chose a base/smartos or base64/smartos64 dataset. Attempting to install a package intended for one architecture with package tools built for a different architecture will work due to SmartOS being able to run both 32-bit and 64-bit applications, but you will get warnings when installing.
OPSYS. On SmartOS this is SunOS, i.e. the output of uname -s.
OS_VERSION. On SmartOS this is 5.11, i.e. the output of uname -r.
PKGTOOLS_VERSION. An integer describing the version of the pkg_install tools required to understand this package. At the current time pkgsrc sets a base version of 20091115 so just use this.

The easiest way to generate this file is simply to get it from an existing package, for example:

$ pkg_info -X pkg_install \
  | egrep '^(MACHINE_ARCH|OPSYS|OS_VERSION|PKGTOOLS_VERSION)' >build-info
$ cat build-info
MACHINE_ARCH=i386
OPSYS=SunOS
OS_VERSION=5.11
PKGTOOLS_VERSION=20091115

comment

This is a short string describing the package which is shown in the default pkg_info or pkgin list output. Keep this short, preferably under 60 characters, so that it displays correctly in 80-column terminals.

$ echo "Show basic memory information on SmartOS" >comment

description

This is a longer multi-line description of the package which is output by pkg_info <package name> or pkgin pkg-descr <package name>. Format this to 80-columns.

$ cat >description <<EOF
meminfo prints basic information about available memory on a SmartOS machine,
listing the total memory size, used, and free.  The information provided
should be understood in light of how memory is used and reported on SmartOS,
for more information refer to:

http://wiki.smartos.org/display/DOC/About+Memory+Usage+and+Capping
EOF

So we should now have:

build-info          # basic package information
comment             # one-line comment
description         # multi-line comment
files/bin/meminfo   # the file we wish to package
packlist            # package listing

Create the package

We now have enough information to create a basic package. The magic invocation to perform the operation is:

$ pkg_create -B build-info -c comment -d description -f packlist \
  -I /opt/local -p files -U meminfo-1.0.tgz

The -B, -c, -d and -f arguments simply pull in the metadata files we’ve written.

The -I argument specifies the destination prefix. As we are creating a package outside of this prefix we need to tell pkg_create of the ultimate destination.

The -p argument is used in conjunction with -I to tell pkg_create where our files to be packaged can be found.

The -U argument means we should just create the package, and not register it.

And finally, we provide the package file. The packaging tools understand version numbers, so we can just provide the file name and it will determine the package name and version from that.

To verify the package, you can run:

$ pkg_info -X meminfo-1.0.tgz

PKGNAME=meminfo-1.0
COMMENT=Show basic memory information on SmartOS
MACHINE_ARCH=i386
OPSYS=SunOS
OS_VERSION=5.11
PKGTOOLS_VERSION=20091115
FILE_NAME=meminfo-1.0.tgz
FILE_SIZE=922
DESCRIPTION=meminfo prints basic information about available memory on a SmartOS machine,
DESCRIPTION=listing the total memory size, used, and free.  The information provided
DESCRIPTION=should be understood in light of how memory is used and reported on SmartOS,
DESCRIPTION=for more information refer to:
DESCRIPTION=
DESCRIPTION=http://wiki.smartos.org/display/DOC/About+Memory+Usage+and+Capping

Installing

For a single machine, you can now simply install the package with:

$ pkg_add meminfo-1.0.tgz

If you want to publish the package so that it is available to pkgin you need to create a pkg_summary file and put it alongside the package for download. This file is usually compressed to speed things up, and pkgin supports both .gz and .bz2.

$ pkg_info -X meminfo-1.0.tgz | gzip -9 >pkg_summary.gz

For example purposes I will simply use file://, but of course you can use http:// instead if you put the packages and pkg_summary file somewhere accessible:

$ mkdir /var/tmp/packages
$ cp meminfo-1.0.tgz pkg_summary.gz /var/tmp/packages

Add this repository to pkgin and reload:

$ echo "file:///var/tmp/packages" >>/opt/local/etc/pkgin/repositories.conf
$ sudo pkgin -fy up

Now the package should show up:

$ pkgin avail | grep meminfo
meminfo-1.0          Show basic memory information on SmartOS

and be installable:

$ sudo pkgin -y install meminfo
$ /opt/local/bin/meminfo
Memory (RSS) Cap    8192M
Memory (RSS) Used   92M
Memory (RSS) Free   8100M

This covers the basics and should be enough to get started. As mentioned, there are many other options available, and I suggest that if you need additional functionality you take a look at the pkg_create manual page, or simply use pkg_info -X on more complicated packages to see what they do.

7,000 binary packages for OSX Lion

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 10 Jul 2012 00:00:00 +0100

For my day job I build packages for our SmartOS operating system using pkgsrc. pkgsrc is a cross-platform package manager which began life as the FreeBSD ports system and was ported to NetBSD back in 1997. Since then it has been ported to many other different systems, 19 at the current count, one of which is OSX/Darwin.

For the recent pkgsrc-2012Q2 release, I have performed a full bulk build on OSX Lion, and the result is 7,374 binary packages for you to use as an alternative to brew/macports/etc.

To use them, run the following:

$ curl http://pkgsrc.smartos.org/packages/Darwin/2012Q2/bootstrap.tar.gz \
    | (cd /; sudo gnutar -zxpf -)
$ PATH=/usr/pkg/sbin:/usr/pkg/bin:$PATH
$ sudo pkgin -y update
$ pkgin avail | wc -l
    7374
$ pkgin search ...
$ sudo pkgin -y install ...

pkgin is a tool similar to apt-get and allows you to easily search for and install/upgrade packages.

Please let me know if you find them useful, and if you have any feedback.

Enjoy!

9,000 packages for SmartOS and illumos

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 09 Jul 2012 00:00:00 +0100

At Joyent, we provide SmartOS users with a set of regularly updated binary packages which are tuned to work well on our operating system. The package manager we use, pkgsrc, does not yet come with native support for e.g. SMF, and so we provide a limited set of packages, ensuring that those we do we provide include such features.

However, often users will want packages which are not yet included in our build set, and so to assist those users we have performed a full bulk build of the unmodified pkgsrc-2012Q2 release, which resulted in well over 9,000 binary packages to choose from.

In addition, thanks to binary compatibility, these packages can also be used on your illumos distribution of choice (please let me know if this is not the case!) if your native package manager does not provide the software you want.

Note that these packages do not come with SMF support, and may be different from our official packages in terms of build options, etc. Also, to avoid conflict with our official packages, these have been built under a different prefix:

/opt/pkg	# main prefix
/etc/opt/pkg	# configuration files
/var/opt/pkg	# transient files

compared to our official packages:

/opt/local	# main prefix
/opt/local/etc	# configuration files
/var		# transient files

To use them, run the following:

# For SmartOS use the original build.
$ curl http://pkgsrc.smartos.org/packages/SmartOS/2012Q2/bootstrap.tar.gz \
    | gzip -dc | (cd /; sudo tar -xpf -)

# For other illumos, use the updated build which should work, but has fewer
# packages, and you will need to ignore warnings about x86_64 != i386.
$ curl http://pkgsrc.smartos.org/packages/SmartOS/2012Q2j1/bootstrap.tar.gz \
    | gzip -dc | (cd /; sudo tar -xpf -)

$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH
$ sudo pkgin -y update
$ pkgin avail | wc -l
    9401
$ pkgin search ...
$ sudo pkgin -y install ...

What’s included? Here are just a few highlights:

Developer tools

PHP: 5.4.4, 5.3.14 + ~100 modules for each
Perl: 5.14.2 + 1,780 modules
Python: 3.2.3, 3.1.5, 2.7.3, 2.6.8, 2.5.6 + ~260 modules for 2.x
Ruby: 1.9.3p194, 1.9.2pl320, 1.8.7.358 + ~450 modules for each
Editors: (vim 7.2, emacs 23.4 and 24.1, xemacs 21.4)
Revision control: (git 1.7.10.5, mercurial 2.2.2, bzr 2.5, svn 1.6.17)

Web stack

Apache: 2.4.2, 2.2.22, 2.0.64, 1.3.42 + modules
MySQL: 5.5.25, 5.1.63, 5.0.96
PostgreSQL: 9.1.3, 9.0.7, 8.3.18
Wordpress 3.4.1

Desktop

GNOME 2.32.1 (not complete)
KDE 4.8.4 (not complete)
XFCE 4.6.1
evilwm 1.1.0

plus thousands of miscellaneous utilities.

The long-term goal of course is to integrate all of our changes back into the upstream pkgsrc repository and eventually provide a single package repository, and we will continue to work on that endeavour.

Enjoy!

Goodbye Oracle, Hello Joyent!

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 07 May 2012 00:00:00 +0100

Back in 2001 when I started working for the BBC I was given a Sun Ultra 10 workstation running Solaris 8 and CDE to do my job. As a Linux/FreeBSD user at the time, and someone accustomed to things such as a working vi implementation, recursive grep and Window Maker, this came as something of a culture shock, and it wasn’t long before I was compiling my own software in a rapid attempt to build and configure a more usable desktop environment - or at least one I was more comfortable in.

Building all that software by hand quickly became tedious, and so I was delighted to find that Christos Zoulas had written a bunch of portability glue, known as Zoularis, which allowed pkgsrc (similar to the FreeBSD ports system I was used to) to be used on Solaris.

It was still early days, however, and so over the next few years I and others helped to improve Solaris support in pkgsrc. By 2004 we had over 1500 packages building on Solaris 9/SPARC using the Sun Studio compiler, and in 2010 there were 5000 packages for Solaris 10/x86, along with a native SFW package including pkgin (an “apt-get” clone), making it easy to get up and running.

It was always something of a disappointment to me though that pkgsrc on Solaris wasn’t more widely deployed as I felt it provided major benefits over Blastwave or IPS. So it was great to see Filip Hajný of Joyent join the pkgsrc developers in 2009 for his work on Solaris with Joyent using pkgsrc internally.

Since then they have:

hired a number of fantastic engineers from the ashes of Sun Microsystems, who have
ported KVM to their SmartOS descendant of Solaris, making it a clear differentiator in the cloud computing arena (DTrace+ZFS+Zones+KVM is the perfect combination for multi-tenancy environments), as well as
taken stewardship of node.js and again hired the most prominent engineers involved in that project

and thus over that time my desire to work there has only increased.

So, I am obviously delighted to say that today is my first day working for Joyent, focusing on making pkgsrc work even better on SmartOS.

It isn’t just the people and the technology which attracted me, though. A recent Twitter post by ex-Sun CEO and co-founder Scott McNealy says it well:

As a long time fan of Sun (I got over the initial shock of Solaris) who was fortunate enough to work there, and as someone who experienced the fantastic company culture of MySQL AB, it is this chance to again join an engineering-driven company that “kicks butt, has fun, doesn’t cheat, loves its customers, changes computing forever” which excites me the most.

Let’s just hope Joyent don’t turn out exactly the same as Sun Microsystems ;)

SmartOS global zone tweaks

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 13 Apr 2012 00:00:00 +0100

The basic premise behind SmartOS is to provide a netbooted global zone, and then do actual work inside KVM or zone-based virtual machines. This has a number of advantages, not least that it’s trivial to upgrade to a newer release - you simply boot a newer image.

However, the read-only nature of the global zone means that if you want to make changes to the global zone, then they need to be read from permanent storage, rather than making changes as you would normally.

SmartOS creates a /usbkey file system on stable storage which holds configuration for the global zone, and an init script /lib/svc/method/smartdc-config which sources the file /usbkey/config for configuration details. By default this will mostly include details you provided at install time, however there are additional values you can set to customise the global zone further.

Here are a couple that I have found, and need for my personal use.

Upload a root authorized_keys file

The root user’s home directory is provided from the ramdisk, so any changes you make will be wiped out on the next boot. This is presumably so that newer images can make changes to the shell rc files, for example to update $PATH without worrying about all users having to merge these changes.

However, I’m not one for typing in passwords all day, so in order to store a /root/.ssh/authorized_keys file you can do this:

mkdir -p /usbkey/config.inc
# paste your key into this file
vi /usbkey/config.inc/authorized_keys
echo "root_authorized_keys_file=authorized_keys" >>/usbkey/config

The root_authorized_keys_file variable points to a file in /usbkey/config.inc, you can of course change the name if you wish.

Set a keyboard map

By default a US keymap will be loaded for the console, if you want to use a different one then find a suitable layout in /usr/share/lib/keytables/type_6 (in my case uk) and then:

echo "default_keymap=uk" >>/usbkey/config

Look for more tweaks

As of time of writing (joyent_20120405T204624Z), these two appear to be the only tweaks available which aren’t already used by default, however this may change in the future. To see if there are any newer ones available, you can:

grep CONFIG_ /lib/svc/method/smartdc-config

and then have a deeper look into the script to see how they are used ;-)

Run ad-hoc scripts

Note that while you may be tempted to think “aha, it’s just a shell script, I can use it like rc.local”, you can’t - it’s explicitly parsed into variables, and trying to put commands in will just break the init script.

However, the /lib/svc/method/manifest-import script does import any SMF manifests it finds in /opt/custom/smf, so if you want to run arbitrary scripts, then have a look at @ryancnelson’s example and modify to suit your tastes.

Automated VirtualBox SmartOS installs

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 12 Apr 2012 00:00:00 +0100

This is a quick script I wrote to:

Download the latest live ISO image of SmartOS
Create a VirtualBox VM, or update an existing VM with the latest ISO
Configure the VM with a zones disk, and boot it!

For the first install you’ll need to:

Configure the network (likely just “dhcp”)
Add c0d0 as the zones disk
Set a root password

After that it will just update the live ISO and use existing settings. By default it will create a port forward so that you can ssh -p 8322 root@localhost into the VM.

Here’s the script in full (or you can download it here):

#!/bin/sh
#
# Configurables:
#
#  - Disk size is in GB
#  - Memory size is in MB
#  - SSH port is the local forwarded port to the VM:22
#
disksize="32"
memsize="1024"
sshport="8322"
vmname="SmartOS"
dlsite="https://download.joyent.com/pub/iso"

vboxdir=$(VBoxManage list systemproperties \
            | awk '/^Default.machine.folder/ { print $4 }')

#
# Find a suitable md5sum program.
#
if type md5 >/dev/null 2>&1; then
    md5sum='md5'
    column='NF'
elif type digest >/dev/null 2>&1 &&
     digest md5 /dev/null >/dev/null 2>&1; then
    md5sum='digest md5'
    column='NF'
elif type digest >/dev/null 2>&1 &&
     digest -a md5 /dev/null >/dev/null 2>&1; then
    md5sum='digest -a md5'
    column='1'
elif type md5sum >/dev/null 2>&1; then
    md5sum='md5sum'
    column='1'
elif type openssl >/dev/null 2>&1 &&
     openssl md5 -hex /dev/null >/dev/null 2>&1; then
    md5sum='openssl md5 -hex'
    column='NF'
else
    echo "ERROR: Sorry, could not find an md5 program" 1>&2
    exit 1
fi

#
# Download MD5 file and parse it for the latest ISO image and checksum
#
curl -o smartos-sums.txt ${dlsite}/md5sums.txt 2>/dev/null
latest_md5=$(awk '/latest.iso/ { print $1 }' smartos-sums.txt)
smartos_version=$(sed -ne "/^${latest_md5}/s/.*-\(.*\).iso/\1/p" \
                    smartos-sums.txt)
if [ -z "${smartos_version}" ]; then
  echo "ERROR: Couldn't determine latest version"
  exit 1
fi

#
# Download the latest ISO image and verify
#
mkdir -p "${vboxdir}/${vmname}"
if [ ! -f "${vboxdir}/${vmname}/smartos-${smartos_version}.iso" ]; then
  echo "Downloading ${dlsite}/smartos-${smartos_version}.iso"
  curl -o "${vboxdir}/${vmname}/smartos-${smartos_version}.iso" \
          ${dlsite}/smartos-${smartos_version}.iso
  dl_md5=$(${md5sum} "${vboxdir}/${vmname}/smartos-${smartos_version}.iso" \
             | awk '{ print $'${column}' }')
  if [ -z "${dl_md5}" ]; then
    echo "ERROR: Couldn't fetch ISO image"
    exit 1
  fi
  if [ "${latest_md5}" != "${dl_md5}" ]; then
    echo "ERROR: md5 checksums do not match"
    exit 1
  fi
fi

#
# Create VirtualBox VM
#
echo "Creating/Updating Virtual Machine"
VBoxManage showvminfo "${vmname}" >/dev/null 2>&1
if [ $? -eq 0 ]; then
    # VM already exists, just update the ISO image
    VBoxManage storageattach "${vmname}" --storagectl "IDE Controller" \
      --port 1 --device 0 --type dvddrive \
      --medium "${vboxdir}/${vmname}/smartos-${smartos_version}.iso"
else
    # Create the VM
    VBoxManage createvm --name "${vmname}" --ostype OpenSolaris_64 --register
    VBoxManage storagectl "${vmname}" --name "IDE Controller" --add ide

    # Attach the ISO image
    VBoxManage storageattach "${vmname}" --storagectl "IDE Controller" \
      --port 1 --device 0 --type dvddrive \
      --medium "${vboxdir}/${vmname}/smartos-${smartos_version}.iso"

    # Create and attach the zone disk
    VBoxManage createhd --filename "${vboxdir}/${vmname}/smartos-zones.vdi" \
      --size $(echo "${disksize}*1024" | bc)
    VBoxManage storageattach "${vmname}" --storagectl "IDE Controller" \
      --port 0 --device 0 --type hdd \
      --medium "${vboxdir}/${vmname}/smartos-zones.vdi"

    # Set misc settings
    VBoxManage modifyvm "${vmname}" --boot1 dvd --boot2 disk --boot3 none
    VBoxManage modifyvm "${vmname}" --memory ${memsize}
    VBoxManage modifyvm "${vmname}" --natpf1 "SSH,tcp,,${sshport},,22"
fi

#
# Start it up
#
echo "Starting Virtual Machine"
VirtualBox --startvm "${vmname}" &

Hopefully there will be a follow-up post which updates my pkgsrc on Solaris for SmartOS, including zone setup etc.

iptables script for Debian / Ubuntu

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 30 Mar 2012 00:00:00 +0100

Most Linux distributions seem to have their own way of handling iptables. Red Hat based distributions come with an init script /etc/init.d/iptables which saves/restores configuration and allows you to check the status.

Debian / Ubuntu come with .. nothing.

So, there is a plethora of advice and ways of setting iptables up them, and this is mine.

It’s a simple shell script which is installed to /etc/network/if-pre-up.d/iptables, meaning it is executed prior to an interface being brought up - better to do it then than afterwards :)

I provide a couple of shell functions to make it easy write rules which are to be applied to both IPv4 and IPv6.

Here it is in its entirety, feel free to use/copy/whatever, it’s public domain.

#!/bin/sh
#
# IPTables firewall script.  There are many.  This is mine.
#


#
# Ensure sane path
#
PATH=/sbin:/usr/sbin:/bin:/usr/bin

#
# When running from the command line, provide a -v option to print the
# installed rules at the end.
#
verbose=
if [ "$1" = "-v" ]; then
    shift
    verbose=on
fi

#
# Rather than duplicate entries for iptables and ip6tables, have some small
# wrapper functions do it for us.
#
# ip4tbl - apply ruleset for just iptables
# ip6tbl - apply ruleset for just ip6tables
# iptbl  - apply ruleset for both iptables and ip6tables
#
ip4tbl()
{
    iptables "$@"
}
ip6tbl()
{
    ip6tables "$@"
}
iptbl()
{
    ip4tbl "$@"
    ip6tbl "$@"
}

#
# Flush all rulesets
#
iptbl -F
iptbl -X

#
# Block by default except outgoing traffic
#
iptbl -P INPUT DROP
iptbl -P FORWARD DROP
iptbl -P OUTPUT ACCEPT

#
# Allow everything on loopback
#
iptbl -A INPUT -i lo -j ACCEPT

#
# Permit established connections
#
iptbl -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

#
# Permit allowed services on all interfaces.  DNS is restricted to my public
# DNS servers, this just runs a hidden master.
#
iptbl  -A INPUT -p tcp -m tcp --dport 22   -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 25   -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 85.158.46.77    -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 85.158.46.77    -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 193.108.199.128 -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 193.108.199.128 -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 193.108.199.130 -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 193.108.199.130 -j ACCEPT
ip4tbl -A INPUT -p tcp -m tcp --dport 53   -s 213.5.89.46     -j ACCEPT
ip4tbl -A INPUT -p udp -m udp --dport 53   -s 213.5.89.46     -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 80   -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 113  -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 443  -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 465  -j ACCEPT
iptbl  -A INPUT -p tcp -m tcp --dport 993  -j ACCEPT

#
# Permit ICMP and traceroute
#
ip4tbl -A INPUT -p icmp -j ACCEPT
ip6tbl -A INPUT -p ipv6-icmp -j ACCEPT
iptbl  -A INPUT -p udp -m udp --dport 33434:33523 -j ACCEPT

#
# Log denied connections
#
LOGCOMMON="-m limit --limit 5/min -j LOG --log-prefix 'iptables: ' --log-level 7"
iptbl  -A INPUT -p tcp       ${LOGCOMMON}
iptbl  -A INPUT -p udp       ${LOGCOMMON}
ip4tbl -A INPUT -p icmp      ${LOGCOMMON}
ip6tbl -A INPUT -p ipv6-icmp ${LOGCOMMON}

#
# Finally, reject to keep open connections down
#
iptbl -A INPUT -j REJECT

#
# Display INPUT chain if verbose
#
if [ -n "${verbose}" ]; then
    iptables  -L INPUT -vn --line-numbers
    ip6tables -L INPUT -vn --line-numbers
fi

Just remember to make it executable ;-)

New site design

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 20 Feb 2012 00:00:00 +0000

I am a minimalist at heart, and I love command line interfaces and text files.

However, I’m also pragmatic, and so for a long time my blog has been powered by Wordpress, which made it relatively easy to manage content and design. I just didn’t want the hassle of maintaining something myself.

Somewhere along the way recently I stumbled across Jekyll, and the idea of generating a static site from templates.

I’d seen a few sites I liked the look of, and I loved the idea of simplifying my site (and editing posts in vim!), and so the end result is the site you see now.

I pondered implementing something like disqus for comments, but honestly I get so few proper responses, compared to the hundreds and hundreds of spam attempts, that it’s not worth it - if you want to comment on a post, just use twitter (I’ll probably implement a little button to do that at some point).

Also, I got rid of categories, simpler to just have tags.

It’s likely I’ve messed up conversion of my old blog posts, as I did them all manually. If you spot anything obviously broken, please let me know. One of the great things of the Jekyll approach is that you can simply manage your site via github, and so if you prefer you can comment on breakages here!

Set up anonymous FTP upload on Oracle Linux

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 11 Jan 2012 00:00:00 +0000

Just because this took a little longer than I expected, here’s a quick howto for setting up an anonymous FTP drop-off on Oracle Linux, which I use as a simple way to transfer files out of my Virtual Machines.

Install vsftpd

$ sudo yum -y install vsftpd

Configure iptables

As FTP is a more complicated protocol than most, there is a special netfilter module required in order to correctly keep track of connections.

# You will perhaps want to change the insert number here.
$ sudo iptables -I INPUT 4 -m state --state NEW -p tcp --dport 21 -j ACCEPT
$ sudo /etc/init.d/iptables save

Add nf_conntrack_ftp to IPTABLES_MODULES

$ sudo vi /etc/sysconfig/iptables-config

Then load the module rather than reboot

$ sudo modprobe nf_conntrack_ftp

Create /incoming

Create /incoming area and ensure it has the correct file permissions and SELinux context. This is the bit which had me stumped for a little while, as I didn’t know about allow_ftpd_anon_write, and while I normally just disable SELinux, I do also like to know how things should work (and be able to write about them!):

$ sudo mkdir /var/ftp/incoming
$ sudo chown ftp:ftp /var/ftp/incoming

# This allows anonymous users to upload, but not see what is in the directory
$ sudo chmod 750 /var/ftp/incoming

$ sudo chcon -u system_u -t public_content_rw_t /var/ftp/incoming
$ sudo setsebool allow_ftpd_anon_write=1

Configure vsftpd

$ sudo vi /etc/vsftpd/vsftpd.conf

anon_upload_enable=YES

Startup

Finally, enable and start vsftpd:

$ sudo chkconfig vsftpd on
$ sudo /etc/init.d/vsftpd start

And that’s it, you should now be able to FTP as anonymous and upload files into /incoming.

Kickstart Oracle Linux in VirtualBox

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 09 Jan 2012 00:00:00 +0000

In my previous post I configured an Ubuntu laptop as a Kickstart install server for a physical machine I wanted to build.

Now that everything is configured for automated installs, it makes sense to use the same infrastructure to install virtual machines too. Everything can be done from the command line, and new virtual machines can be provisioned very quickly.

Again, I will use Oracle Linux 6.2 as an example.

Create Virtual Machine

These commands create a new virtual machine, including disk, and configures it for network booting. If you don’t already have VirtualBox installed:

$ sudo apt-get install virtualbox

Set the name of the virtual machine as it appears in VirtualBox, this variable is then used throughout:

$ VM="Oracle Linux 6.2"

Create the VM:

$ VBoxManage createvm --name "${VM}" --ostype "Oracle_64" --register

Create the hard disk (32GB expanding) and attach it via SATA. Note that I store my VMs in ${HOME}/VirtualBox rather than the default ${HOME}/VirtualBox VMs.

$ VBoxManage createhd --filename "VirtualBox/${VM}/${VM}.vdi" --size 32768

$ VBoxManage storagectl "${VM}" --name "SATA Controller" --add sata \
>   --controller IntelAHCI

$ VBoxManage storageattach "${VM}" --storagectl "SATA Controller" --port 0 \
>   --device 0 --type hdd --medium "VirtualBox/${VM}/${VM}.vdi"

Default RAM is 128MB, Oracle Linux installer requires at least 512MB however. Once installed we can drop back down to 256MB or so:

$ VBoxManage modifyvm "${VM}" --memory 512

Configure boot order. Put disk first, as on the first boot there is nothing on it so it falls through to PXE for install, then after the install the disk is bootable.

$ VBoxManage modifyvm "${VM}" --boot1 disk --boot2 net --boot3 none --boot4 none

Change NIC type from the default e1000, as a vanilla VirtualBox install does not include the firmware necessary to network boot from that device – it is available in the “VirtualBox Extension Pack” add-on. Switch to a plain PCNet Fast III which does include PXE firmware.

$ VBoxManage modifyvm "${VM}" --nictype1 Am79C973

VirtualBox does have the ability to serve TFTP directly from the file system by placing files inside ${HOME}/.VirtualBox/TFTP/ but I prefer to just use the network as it’s already configured.

$ VBoxManage modifyvm "${VM}" --nattftpserver1 10.0.2.2
$ VBoxManage modifyvm "${VM}" --nattftpfile1 pxelinux.0

Configure pxelinux/kickstart

We just need a couple of tweaks to the configs from the last blog entry, as the network addresses are different inside VirtualBox, and we also may want a different kickstart configuration.

$ sudo vi /var/lib/tftpboot/pxelinux.cfg/default

I just amended the existing entry to point to 10.0.2.2 which is the address of the machine running VirtualBox, and a different ks-vm.cfg kickstart configuration file, but you could also create a new label if you wanted to regularly switch between different configurations:

LABEL ol6.2
    KERNEL /ol6.2/vmlinuz
    APPEND initrd=/ol6.2/initrd.img ks=http://10.0.2.2/ks-vm.cfg

For virtual machines I use a slightly different configuration compared to previously. I’ve only shown the changes below, not the full file:

$ sudo cp /usr/share/nginx/www/ks.cfg /usr/share/nginx/www/ks-vm.cfg

# Update network configuration for DHCP instead of static
network --bootproto=dhcp
url --url=http://10.0.2.2/ol6.2

# Don't specify disks, just use default layout
bootloader --location=mbr --driveorder=sda
clearpart --all --initlabel

# Just add 'screen' to the default set of @base and @core packages.
%packages
screen
%end

Start Virtual Machine

All that’s left to do is to boot up the VM, and everything else should run automatically.

$ VirtualBox --startvm "${VM}"

All done!

Kickstart Oracle Linux from Ubuntu

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 09 Jan 2012 00:00:00 +0000

As my new job involves working on Oracle Linux, I figured I should migrate my home server to it, which would also mean I could move it to a proper RAID10 configuration rather than relying on multiple RAID1s.

My laptop runs Ubuntu, and I wanted to install the server from it using PXE and Kickstart, so here’s how I did it.

Configure dhcpd

DHCP is required for two things, to give the server its network configuration, and to point it at the PXE boot loader we want to use.

$ sudo apt-get install isc-dhcp-server
$ sudo vi /etc/dhcp/dhcpd.conf

Here’s the configuration I used, which says to configure 192.168.2.0/24 with a dynamic DHCP range between 192.168.2.100 – 192.168.2.200, and to boot machines using pxelinux.0 which is relative to the TFTP root directory (configured in the next section):

subnet 192.168.2.0 netmask 255.255.255.0 {
    range 192.168.2.100 192.168.2.200;
    filename "pxelinux.0";
}

The isc-dhcp-server install automatically tries to start the server, but will fail as it isn’t configured, so we restart it now that there is a working configuration installed

$ sudo /etc/init.d/isc-dhcp-server restart

Configure tftpd

TFTP is a simple protocol used to transfer files over the network, and due to its simplicity it is the primary way to network boot, as it can be easily embedded into firmware.

All we need to do is install the TFTP daemon and syslinux which includes the PXE boot loader, then put the pxelinux file into the tftproot area:

$ sudo apt-get install syslinux tftpd-hpa
$ sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot

Configure Oracle Linux DVD

To save space on the laptop we can just mount the DVD read-only and install from that:

$ sudo mkdir /media/ol6.2
$ sudo mount -o loop,ro /path/to/OracleLinux-R6-U2-Server-x86_64-dvd.iso /media/ol6.2

However, we do need to copy the kernel and initrd image from the DVD into the tftproot as they are required for booting:

$ sudo mkdir /var/lib/tftpboot/ol6.2 /var/lib/tftpboot/pxelinux.cfg
$ sudo cp -a /mnt/images/pxeboot/{initrd.img,vmlinuz} /var/lib/tftpboot/ol6.2/

Configure pxelinux

All that’s left for the PXE stage is to configure the boot loader, and tell it what kernel and initrd we want to use:

$ sudo vi /var/lib/tftpboot/pxelinux.cfg/default

This configuration has just one entry which is booted after a short wait, but pxelinux has many more options, including the ability to boot from local disk.

DEFAULT	ol6.2
PROMPT 1
TIMEOUT 5

LABEL ol6.2
    KERNEL /ol6.2/vmlinuz
    APPEND initrd=/ol6.2/initrd.img ks=http://192.168.2.1/ks.cfg

Note the ks argument which specifies the kickstart file we will use, that and the web server required to serve it will be set up next.

Configure nginx

I chose nginx as it is small and simple to configure, but any web server will do.

$ sudo apt-get install nginx
$ sudo vi /etc/nginx/sites-available/default

The install will be performed over HTTP, so we need to make the DVD we mounted earlier available. This entry in the server { } section makes the DVD available via http://192.168.2.1/ol6.2/:

server {
[...]
    location /ol6.2 {
        root /media;
        autoindex on;
        allow all;
    }

Then start nginx (unlike isc-dhcp-server this isn’t done automatically):

$ sudo /etc/init.d/nginx start

Configure kickstart

Finally, we create a kickstart configuration which specifies exactly how our target machine is to be installed, and this allows a completely unattended installation.

Ideally I should create a specific area for holding files like this, but as a quick hack I simply put it into the default nginx web root (and thus available as http://192.168.2.1/ks.cfg as configured earlier in the pxelinux.cfg/default file:

sudo vi /usr/share/nginx/www/ks.cfg

Here is my ks.cfg file in full. The only thing missing is a rootpw line to automatically set a root password, however for maximum security I am happy to forego a completely unattended installation and instead have the installer prompt me to type it in during the install.

Some notes:

UK keyboard language and timezone selected.
Automatically reboot when the installer is finished.
Point to the Oracle Linux DVD using the url directive.
I have 4 disks configured with RAID1 for /boot, and RAID10 for swap and /.
Disks are referred to by path, to ensure correct ordering.
A small set of packages are installed, containing just the functionality I require.
A small %post section is used to perform any fixes I want for the first boot.

#
# Miscellaneous options.
#
install
keyboard uk
lang en_GB.UTF-8
reboot
selinux --enforcing
timezone Europe/London

#
# User setup.  I'd create my local user here and configure sudoers, but
# kickstart doesn't yet support creating a user with uid/gid of 1000 (the
# gid is always 500, even if you add the named group first).
#
authconfig --enableshadow --passalgo=sha512

#
# Networking.  The 'network' line needs to be on a single line
# for kickstart to work - it is only split here for the blog.
#
firewall --service=ssh
network --bootproto=static \
        --hostname=gromit.adsl.perkin.org.uk \
        --ip=192.168.2.10 \
        --netmask=255.255.255.0 \
        --gateway=192.168.2.1 \
        --nameserver=193.178.223.141,208.72.84.24 \
        --ipv6=auto
url --url=http://192.168.2.1/ol6.2

#
# Disk configuration.
#
bootloader --location=mbr --driveorder=sda,sdb,sdc,sdd
clearpart --all --initlabel
#
# /boot (RAID1 necessary as booting from RAID10 isn't supported)
#
part raid.00 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.01 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.02 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.03 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-3*0
raid /boot --level=1 --device=md0 --fstype=ext4 raid.00 raid.01 raid.02 raid.03
#
# swap, RAID10 of size RAM+2GB, give or take..
#
part raid.10 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.11 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.12 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.13 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-3*0
raid swap --level=10 --device=md1 --fstype=swap raid.10 raid.11 raid.12 raid.13
#
# /, RAID10 of remainder (have to specify an arbitrary --size even with --grow)
#
part raid.20 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.21 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.22 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.23 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-3*0
raid / --level=10 --device=md2 --fstype=ext4 raid.20 raid.21 raid.22 raid.23

#
# Packages.  @base and @core are pre-selected.
#
%packages
@cifs-file-server
@console-internet --optional
@development
@legacy-unix --optional
@mail-server
@network-server --optional
@network-tools
@nfs-file-server
@web-server
screen
%end

#
# Post-install fix-ups.
#
%post
#
# The 'network' directive doesn't support DNS search paths, so set those
# manually, and disable Network Manager.
#
printf "/^NM_CONTROLLED/s/yes/no/\nw\nq\n" \
  | ed /etc/sysconfig/network-scripts/ifcfg-eth0
printf "/^#/s/.*/search adsl.perkin.org.uk perkin.org.uk/\nw\nq\n" \
  | ed /etc/resolv.conf
#
# Disable unwanted services
#
chkconfig --del cups
%end

All done.

Last day at MySQL

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 22 Dec 2011 00:00:00 +0000

Today is my final day working on MySQL. It has been an amazing 4.5 years, and I’ve loved working on the technical challenges involved in producing a piece of software which runs on so many different platforms, as well as working with so many talented individuals. I have learned a lot, which is just how I like it.

Times have changed since I first joined MySQL AB, of course. But despite going through two acquisitions, the day to day work hasn’t changed much at all. I still get to work from home, which is something I am passionate about and feel many companies are missing out on massively by still being stuck in an outdated 20th century mindset. Oracle have continued to invest in MySQL, providing additional headcount and extra hardware. And the vast majority of people I started working with back in 2007 are still with the company (don’t believe the FUD, folks) and working harder than ever.

I’m most proud of the work we have done internally to ensure our MySQL releases are in excellent shape. When I first joined, our internal “PushBuild” system was very limited, only able to handle a small number of pushes with reduced testing across a few platforms, and the builds produced were completely different to those that were released.

Nowadays, thanks to the investment made by Sun and Oracle, we have a large server farm producing thousands of builds and hundreds of gigabytes of data every day across all our supported platforms. Those which are based on MySQL 5.5 or newer have additional package verification tests, using chroots and virtual machines, to ensure that the RPM/MSI/etc packages can be installed, run, and uninstalled, all automatically and on every push by developers. We run additional nightly and weekly tests which extend the default set of test suites. Our 5.5+ releases are produced directly in PushBuild. And we are looking to extend this to all MySQL products, not just the Server.

If the above sounds fun, keep an eye out for job posts, as the Release Engineering team is looking to expand, and they are an amazing group of people to work with.

I certainly found it fun, which is why I’m going to continue in that line of work, and stay within Oracle, but move to the Linux group. To be honest, databases have never been a passion of mine, however I am rather fond of Operating Systems, and so I am really looking forward to continuing to work on continuous integration and testing, but with Oracle Linux and Oracle VM instead of MySQL. Plus, it will be nice to go back to a purely technical role – I have learned over the last year or so that management isn’t really my thing :)

To all my colleagues, past and present, thank you for the wonderful ride. I have many, many good memories, and hopefully we will keep in touch.

Installing OpenBSD with softraid

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 15 Dec 2011 00:00:00 +0000

This is more of a log for me than anything else, but perhaps someone will find this useful.

OpenBSD includes a software RAID implementation which supports booting in newer snapshots, and I was itching to install the latest version and use it as my file server, which has 4 750GB disks. There is a small bit of preparation work to do prior to installing, which is the bulk of this entry, most of which is based on this undeadly.org article.

Grab latest amd64 snapshot, boot it, drop into (S)hell mode and set up the disks:

#!/bin/sh

cd /dev
sh MAKEDEV sd1 sd2 sd3 sd4 sd5
for disk in 0 1 2 3
do
  # Clear beginning of disks..
  dd if=/dev/zero of=/dev/rsd${disk}c bs=1m count=10

  # ..and initialise new partition table
  fdisk -iy sd${disk}

  #
  # Create BSD disklabel:
  #
  # - 128m partitions at start to hold kernels for booting
  # - 4g spare raid on each disk for testing
  # - rest raid for main OS and data
  #   - OS and /home on first two mirrored disks
  #   - /store on second two mirrored disks
  #
  print "a a\n\n128m\n\na d\n\n4g\nraid\na e\n\n\nraid\nw\nq\n" \
    | disklabel -E sd${disk}

  # Clear beginning of raid partitions
  dd if=/dev/zero of=/dev/rsd${disk}d bs=1m count=10
  dd if=/dev/zero of=/dev/rsd${disk}e bs=1m count=10
done

# Create RAID1 mirrors
bioctl -c 1 -l sd0e,sd1e softraid0
bioctl -c 1 -l sd2e,sd3e softraid0

# Exit shell and start the (I)nstall
exit

As for the install, go with the sensible defaults, other than:

change keyboard layout to “uk”
manually configure network, enable rtsol
start ntpd
do not expect to run X

When it comes to disk selection, choose sd4 as the root disk, and use the following layout:

# partition  size  mount
  sd4a       1G    /
  sd4b       8G    swap
  sd4d       1G    /tmp
  sd4e       8G    /var
  sd4f       16G   /usr
  sd4h       rest  /home

then initialise sd5 with:

# partition  size  mount
  sd5a       2G    /altroot
  sd5d       8G    /scratch
  sd5e       rest  /store

Install the full OS, set the correct timezone, then before rebooting initialize the boot partitions and copy the kernels to them. Doing this on all of them means we can boot from any disk.

#!/bin/sh

for disk in 0 1 2 3
do
  newfs sd${disk}a
  mount /dev/sd${disk}a /mnt2
  cp /mnt/bsd* /mnt2
  umount /mnt2
done
eject cd0a
reboot

Job done.

Create VirtualBox VM from the command line

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 21 Sep 2011 00:00:00 +0100

As something of a follow-up post to the previous entry, here’s a quick recipe for creating a Virtual Machine using the VirtualBox command line tools:

We’re using Windows Server 2008 64bit as an example, modify to taste.

$ VM='Windows-2008-64bit'

Create a 32GB “dynamic” disk.

$ VBoxManage createhd --filename $VM.vdi --size 32768

You can get a list of the OS types VirtualBox recognises using:

$ VBoxManage list ostypes

Then copy the most appropriate one into here.

$ VBoxManage createvm --name $VM --ostype "Windows2008_64" --register

Add a SATA controller with the dynamic disk attached.

$ VBoxManage storagectl $VM --name "SATA Controller" --add sata \
>  --controller IntelAHCI
$ VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 \
>  --device 0 --type hdd --medium $VM.vdi

Add an IDE controller with a DVD drive attached, and the install ISO inserted into the drive:

$ VBoxManage storagectl $VM --name "IDE Controller" --add ide
$ VBoxManage storageattach $VM --storagectl "IDE Controller" --port 0 \
>  --device 0 --type dvddrive --medium /path/to/windows_server_2008.iso

Misc system settings.

$ VBoxManage modifyvm $VM --ioapic on
$ VBoxManage modifyvm $VM --boot1 dvd --boot2 disk --boot3 none --boot4 none
$ VBoxManage modifyvm $VM --memory 1024 --vram 128
$ VBoxManage modifyvm $VM --nic1 bridged --bridgeadapter1 e1000g0

Configuration is all done, boot it up! If you’ve done this one a remote machine, you can RDP to the console via vboxhost:3389.

$ VBoxHeadless -s $VM

Once you have configured the operating system, you can shutdown and eject the DVD.

$ VBoxManage storageattach $VM --storagectl "IDE Controller" --port 0 \
>  --device 0 --type dvddrive --medium none

Finally, it’s a good idea to take regular snapshots so that you can always revert back to a known-good state rather than having to completely re-install.

$ VBoxManage snapshot $VM take <name of snapshot>

And, if you need to revert back to a particular snapshot:

$ VBoxManage snapshot $VM restore <name of snapshot>

Enjoy!

Creating chroots for fun and MySQL testing

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 14 Sep 2011 00:00:00 +0100

Virtualisation is all the rage today, however there are still a few cases where good old-fashioned Unix chroot is still applicable, and testing MySQL across multiple platforms and architectures is one of those cases.

At Oracle we do full automated package verification testing of our MySQL server binaries prior to release, which attempts to install the package, start the server, run some functionality testing, then uninstall. It is of course highly desirable that the testing environment this is performed in is as close to a clean install of the target operating system as possible, to avoid problems such as our packages depending upon some local changes or packages we may have installed which won’t be available on a customer system.

Given the large number of platforms and architectures that MySQL supports, going the virtualisation route would mean having to use many different products: VirtualBox for x86, zones for SPARC, qemu for ia64/PA-RISC/others (if it even supports them), and this gets complicated quickly and is not very maintainable. Thus I chose to use chroot as much as possible. In addition, it’s much faster and less intensive on resources to use a chroot than boot up an entire OS image each time.

I built all images directly from the original installation images (DVD, ISO, RPM, etc), to ensure that there was no contamination from our build environment or local install scripts in the image – they should be as close to what a normal user or customer will be running in their setup. From the install image, the packages are installed to a temporary directory, some final modifications are made, then the directory is tarred up ready to be extracted by the test framework and used.

Here are some operating system specific examples, which set up an extracted chroot image into ${CHROOTDIR}. There may be additional steps required to get a fully functioning chroot, such as copying device files (/dev/zero and /dev/null are usually the minimum requirements) and adding users.

FreeBSD

FreeBSD 7 and 8 come as a number of sets in tar format, and for our purposes we only need to extract the base set. You may wish to add more sets if you want to use your chroot for building packages:

#!/bin/sh
mdunit=$(mdconfig -a -t vnode -o readonly -f /path/to/dvd1/of/freebsd.iso)
mount_cd9660 /dev/${mdunit} /mnt
cat /mnt/*/base/base.?? | tar -xpzf - -C ${CHROOTDIR}
umount /mnt
mdconfig -d -u ${mdunit}

HP-UX

HP-UX has since been EOL’d for MySQL, however this information might still prove useful. The HP-UX installation media contains per-directory packages, with the contents representing how they are laid out on the destination file system with each file gzip compressed.

#!/bin/sh
cd /path/to/extracted/hpux-dvd1
for pkg in $LIST_OF_PKGS
do
  for subpkg in ${pkg}/*
  do
    if [ ! -d "${subpkg}" ]; then
      continue
    fi
    for d in $(find ${subpkg} -type d)
    do
      mkdir -p ${CHROOTDIR}/$(echo ${d} \
        | sed -e "s#${subpkg}/##g" \
              -e "s#usr/newconfig/##g")
    done
    for f in $(find ${subpkg} -type f)
    do
      gzip -dc ${f} >${CHROOTDIR}/$(echo ${f} \
        | sed -e "s#${subpkg}/##g" \
              -e "s#usr/newconfig/##g")
    done
  done
done

Once this is done you’ll need to fix up permissions in bin and lib directories (make files executable), as well as create a bunch of symlinks for e.g. /bin and /lib.

OSX

For OSX we don’t actually use a chroot tarball but instead create a sparse disk image. Currently the size of the “chroot” is very large as there’s no easy way to strip down an OSX install, so mounting a disk image is faster than unpacking a chroot, plus it preserves various HFS-specific attributes.

You will likely need at least the BSD, BaseSystem, and Essentials packages.

# Create a sparse image to hold the chroot (which isn't really a directory)
hdiutil create -fs HFS+ -size 8g -type SPARSE -volname osx-chroot ${CHROOTDIR}
hdiutil attach -mountpoint ${CHROOTDIR} ${CHROOTDIR}.sparseimage
# Either attach a DVD image or the real thing
hdiutil attach -mountpoint /Volumes/osx-install /path/to/dvd
# Install packages
for pkg in BSD BaseSystem Essentials
do
  installer -verbose \
   -pkg /Volumes/osx-install/System/Installation/Packages/${pkg}.pkg \
   -target ${CHROOTDIR}
done
# Unmount
hdiutil detach ${CHROOTDIR}
hdiutil detach /Volumes/osx-install

Red Hat / Oracle Linux / SuSE

For RPM-based distributions we use rpm to directly install packages into the chroot. The list of RPMs we install varies quite a lot from release to release, usually by having to increase the number:

RH3: 81
RH4: 85
RH5: 114
RH6: 203

for the same functionality.

# Kludge for 'setup' RPM to install
mkdir -p ${CHROOTDIR}/var/lock/rpm
# If installing from an ISO:
mount -o loop /path/to/iso /mnt
# Path varies from release to release
cd /mnt/path/to/RPMs
rpm --root=${CHROOTDIR} -Uvh ${LIST_OF_RPMS}

Solaris

Similar to RPM, we use the native package manager to install packages directly into the chroot directory:

# Avoid prompts
sed -e "s/ask$/nocheck/" /var/sadm/install/admin/default > /tmp/admin-$$
pkgadd -a /tmp/admin-$$ -R ${CHROOTDIR} -d . ${LIST_OF_PKGS}
rm /tmp/admin-$$

Windows

Ok, so of course we can’t use chroot images for Windows, as it doesn’t have chroot(2). So here we use VirtualBox and its snapshot ability to load a clean snapshot of a basic Windows install, do the tests, then shut down the virtual machine, restore the snapshot, and boot up again.

Graphing memory usage during an MTR run

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 30 Jun 2011 00:00:00 +0100

In order to optimally size the amount of RAM to allocate to a set of new machines for running MTR, I ran a few tests to check the memory usage of an MTR run for mysql-trunk and cluster-7.1. As using a RAM disk considerably speeds things up, I set the vardir to be on /ramdisk and logged the usage of that too.

The tests were performed on an 8-core E5450 @ 3.00GHz with 24GB RAM, with 8GB allocated to /ramdisk. Each branch ran the default.daily collection, which generally contains the most testing we do per-run. Between each run I rebooted the machine to clear the buffer cache and /ramdisk.

I used something like the script below, which saved the per-second usage of /ramdisk, the total RAM used, and the RAM used minus buffers.

#!/bin/bash

BRANCH="mysql-trunk"
BUILDDIR="mysql-5.6.3-m5-linux2.6-x86_64"
TESTDIR="${HOME}/mtr-test/${BRANCH}"

stats()
{
  i=1
  rm -f ${TESTDIR}/stats-${BRANCH}
  while [ -f ${TESTDIR}/running ]; do
    rd=$(df -k /ramdisk | awk '/^\// {print $3}')
    mem=$(free | awk '/^Mem/ {print $3}')
    mem1=$(free | awk '/cache:/ {print $3}')
    echo "${i} ${rd} ${mem} ${mem1}" >>${TESTDIR}/stats-${BRANCH}
    i=$((i+1))
    sleep 1
  done
}

export TMPDIR="${TESTDIR}/tmp"
rm -rf ${TMPDIR}
mkdir -p ${TMPDIR}

>${TESTDIR}/running
stats &

(
  cd ${TESTDIR}/${BUILDDIR}/mysql-test

  perl mysql-test-run.pl ... --parallel=8 --vardir=/ramdisk/mtr-${BRANCH}/...
  mv /ramdisk/mtr-${BRANCH}/* ${TMPDIR}/
  ...
)

sync
rm -f ${TESTDIR}/running
wait

First I graphed a straight run of the two branches, using the following gnuplot script:

set terminal png enhanced font "Times,11" size 640,768
set output "mtr-ram.png"
set title "MTR memory usage (8-core Xeon, 24GB, 8GB RAM disk)"
set xlabel "Time (minutes)"
set ylabel "Memory usage (GB)"
set yrange [0:16]
set xtics 10
set key top box
set grid
plot "stats-mysql-trunk" every 60 using (($1)/60):(($2)/1024/1024) \
        title 'mysql-trunk /ramdisk usage' with lines, \
     "stats-mysql-trunk" every 60 using (($1)/60):(($3)/1024/1024) \
        title 'mysql-trunk RAM (inc buf)' with lines, \
     "stats-mysql-trunk" every 60 using (($1)/60):(($4)/1024/1024) \
        title 'mysql-trunk RAM (exc buf)' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($2)/1024/1024) \
        title 'cluster-7.1 /ramdisk usage' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($3)/1024/1024) \
        title 'cluster-7.1 RAM (inc buf)' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($4)/1024/1024) \
        title 'cluster-7.1 RAM (exc buf)' with lines

I then performed a valgrind run on mysql-trunk using similar scripts. As valgrind takes considerably longer (and uses more RAM) I kept it separate as the combined graph isn’t very clear:

So, based on these results, the host machine (16GB RAM + 8GB RAM disk) is probably a sensible guide for now, and allows for some future growth.

Fix input box keybindings in Firefox

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 29 Jun 2011 00:00:00 +0100

Those of us used to command line editing will no doubt have been frustrated many times in Firefox when editing text in an input box and subconciously hitting ctrl-w to delete-word, only to have the tab close and your work deleted.

Thankfully there is a workaround to this. It used to be a case of adding the following to .gtkrc:

gtk-key-theme-name = "Emacs"

However these days it’s a gconf setting:

$ gconftool-2 --set /desktop/gnome/interface/gtk_key_theme Emacs --type string

This will bind ctrl-w to delete-word when in an input box, but retain the close tab binding elsewhere, a nice implementation of DWIM. See this page for more information.

How to lose weight

jonathan@perkin.org.uk (Jonathan Perkin) — Fri, 24 Jun 2011 00:00:00 +0100

This year I set myself a target to get back to my pre-marital weight. Back in 2001 I was 90kg, but over the past decade I’d steadily gone up and for quite a few years had hovered around 100kg.

I’m a bit of a perfectionist, and like shiny graphs, so from Jan 1st this year I weighed myself every day on the Wii Fit, and kept a record of my progress. This morning I was finally under 90kg. Here’s how things have gone thus far:

And for the geeks, my gnuplot script, assuming an input file named weight.txt containing lines of the format "%Y-%m-%d <weight>":

set terminal png size 640, 400
set output "weight-jun11.png"
set xdata time
set timefmt "%Y-%m-%d"
set format x "%b"
set xlabel 'Date'
set ytics 2
set ylabel 'Weight (kg)'
set multiplot
plot 'weight.txt' using 1:2 title 'Weight' with lines linecolor 2

My three step plan for losing weight

The interwebs are awash with a million different ways to lose weight, and there are thousands of companies who will gladly accept large chunks of your cash in order to provide you with their expert opinion on how to do it. This is my three-step plan, and you can have it for free:

Eat less.
Eat less!
Profit!!!

Really, that’s it. It’s not rocket science, after all. If you eat less stuff (especially saturated fat and sugar), there is less excess for your body to store as fat, and thus over time you weigh less as your body starts to use up the excess you have stored.

Here’s some things I did to achieve this.

Eat less

The basic aim. For me this was mainly about reducing my portion size.

Previously, I’d eat a massive bowl of cereal in the morning. which I thought was a good thing as everyone always says to ensure you have a good breakfast. However I was eating more than my body needed and as a consequence it was likely storing up the excess.

Also at meal times I’d usually serve myself a full plate, and always feel that I needed to finish it. I liked feeling full. However, again, this likely just meant I was eating more than I needed, and I found that just by serving less food I was still satisfying my hunger but with less excess.

After a relatively short period of time I found that by avoiding these large meals, I needed less to feel full – as if my stomach had shrunk and gotten used to the reduced size. This was a great positive feedback loop, as it massively helped avoid the temptation to snack between meals.

Cut down on sugar/fat

I realise this can be hard for some people, I found it relatively easy but as you can see on the graph above there are some upward trends, which were mainly when I went to visit my parents, who have a cupboard full of chocolates, crisps, cakes, sweets, etc, and I find it really hard not to have at least one or two fake Lidl snickers bars per day!

However, also note that after an upward trend, I lost the weight again pretty quickly, so don’t worry too much about a few days of eating junk, I actually noticed how bad I felt afterwards after getting used to a reduced sugar/fat intake, and that provided good incentive to cut down again.

Some practical things I did:

Stopped putting sugar in tea/coffee. I’ve since regressed, but it was helpful to do this for a while, and definitely helped wean my body off desiring sugar.
Avoided snacking on crisps, chocolate, biscuits, etc. When you work from home this can be difficult, but I found it helped to ensure they weren’t in the house to begin with, and that we were stocked up on bananas, apples, and other less sugary/fatty snacks.
Used a fine cheese grater when making sandwiches, beans on toast, etc. Previously I’d put a good few slabs of cheese in, when I didn’t really need that quantity. The fine grater ensured I still got the taste, but with less quantity.
Bought reduced fat mayonnaise, margarine, etc. These actually taste pretty good these days, and in a sandwich with lots of fresh cucumber, tomato, salad etc you don’t notice the difference

All these things helped to lose weight and, in a similar manner to the ‘eating less’ part, trained my body to not require them as much as it used to – and actually, to noticeably feel worse if ever I regressed, which made getting back on track very easy.

Weigh myself daily

It seems this is generally not recommended, and advice is that you should weigh yourself weekly (at the same time each week). Logically this doesn’t make sense to me, as a geek it’s obvious that the more data points you have, the better. Your body weight can fluctuate quite a lot from day to day especially with regards to how much liquid you have drunk, and if you are only weighing yourself once per week you could get caught out by a daily spike.

Aside from the geek factor of a better graph resolution, I also prefer daily weighing as it provides me, as a perfectionist, with incentives both ways. If I weigh less compared to the previous day, I feel good that I am achieving my aim, and am encouraged to continue. If I weigh more, it’s a warning that I may have eaten too much, and I am then motivated to be more careful that day.

Weighing daily also helps to just keep your mind reminded of the task, plus if you’re on the Wii Fit already you might be tempted to do some exercise :)

Next steps

I’m not content to stay here, the next plan is to get closer to my ideal BMI weight which is around 80kg. This is likely to be much harder, as there is less excess fat for me to get rid of now. However, one thing I haven’t done so far is increase the amount of exercise I do, and there is definitely room for improvement there!

Hopefully I can provide another update later this year and perhaps be around 85kg.

How to fix stdio buffering

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 23 Jun 2011 00:00:00 +0100

It’s a common problem. You write some shell command like:

$ tail -f /var/log/foo | egrep -v 'some|stuff' | sed | awk

and wonder why nothing is printed, even though you know some text has matched. The problem is that stdio is being buffered, and there’s a very good write-up of the problem here so I won’t repeat the technical background.

What I will provide though is how to fix it for common cases.

stdbuf

stdbuf is part of GNU coreutils, and is essentially an LD_PRELOAD hack which calls setvbuf() for an application. Thus it is a generic solution to the problem and can be used to fix most applications. Usage looks like this:

$ tail -f /var/log/foo | stdbuf -o0 app ...

which will disable output buffering for app, assuming it does not do something itself to reverse the setvbuf() call. An example of a misbehaving application is mawk, below.

awk

GNU awk needs no modifications, that is it does not buffer when there is no controlling tty.

mawk however (the default awk in Debian/Ubuntu and possibly others) buffers output, and also does not seem to work with stdbuf. It does however provide a -Winteractive option which will turn off buffering.

$ tail -f /var/log/foo | gawk

$ tail -f /var/log/foo | mawk -Winteractive

sed

GNU sed provides the -u option which calls fflush(), thereby providing unbuffered output. You can also use stdbuf as above.

$ tail -f /var/log/foo | sed -u

$ tail -f /var/log/foo | stdbuf -o0 sed

grep

Similar to sed, GNU grep provides a specific option, --line-buffered, to disable buffering, or again you can use stdbuf.

$ tail -f /var/log/foo | grep --line-buffered

$ tail -f /var/log/foo | stdbuf -o0 grep

Serving multiple DNS search domains in IOS DHCP

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 13 Jun 2011 00:00:00 +0100

I have a Cisco router at home which I also use as a DHCP server, and it works pretty well. Today I wanted to fix a long-standing issue on my network, in that I want multiple DNS search domains.

First off, the domain-name DHCP option doesn’t support multiple entries so we can’t use that. So, off to try raw DHCP option codes. You can find the list of options here, thus 119 is the one I want.

Trying a simple:

ip dhcp pool host.net.example.com
   option 119 ascii net.example.com,example.com

didn’t work at all. A quick prod of lazyweb (in this case Simon on IRC) suggested using hex input instead. In order to do that we need to convert the ASCII string into Cisco’s hex sequence, which is as follows:

Split domain name by dot
Prepend each string by its length (in hex)
NUL terminate each domain
Dot-seperate the final string in 16bit chunks

To do this I wrote a quick Python script:

#!/usr/bin/python

import sys

hexlist = []
for domain in sys.argv[1:]:
    for part in domain.split("."):
        hexlist.append("%02x" % len(part))
        for c in part:
            hexlist.append(c.encode("hex"))
    hexlist.append("00")

print "".join([(".%s" % (x) if i and not i % 2 else x) \
               for i, x in enumerate(hexlist)])

which can be used like this:

$ ./cisco.py net.example.com example.com
036e.6574.0765.7861.6d70.6c65.0363.6f6d.0007.6578.616d.706c.6503.636f.6d00

Then back to IOS and paste it in:

ip dhcp pool host.net.example.com
   option 119 hex 036e.6574.0765.7861.6d70.6c65.0363.6f6d.0007.6578.616d.706c.6503.636f.6d00

This seems to do what we want, though IOS appears to append a dot to each domain when serving via DHCP.

One last note, if you use this in addition to domain-name then the option 119 list will be appended to the domain-name name in the search list, so you’d actually want something like this:

ip dhcp pool host.net.example.com
   domain-name net.example.com
   option 119 hex 0765.7861.6d70.6c65.0363.6f6d.00

to generate a resolv.conf containing:

domain net.example.com
search net.example.com example.com.

Fix Firefox URL double click behaviour

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 13 Jun 2011 00:00:00 +0100

One of the best things I find with OS X is the consistent and sane handling of keyboard and mouse bindings. Since having to move back to Linux for work I am constantly frustrated that different applications all have their own idea of what shortcuts to use.

One of the biggest annoyances was the mouse click behaviour in the browser URL bar. OS X behaviour is one click to position cursor, two clicks to select a word, three clicks to select all. I rely on this behaviour a lot as I frequently copy/paste parts of URLs.

Thankfully that’s one that can be fixed easily. Simply navigate to about:config and double click on the browser.urlbar.doubleClickSelectsAll key to set it to false.

One annoyance down, lots more to go..

SSH via HTTP proxy in OSX

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 20 Apr 2011 00:00:00 +0100

If you happen to be stuck behind a corporate firewall with only HTTP proxies for external access, you might still be able to SSH out through them using the built-in nc on OSX.

First, hope that the proxies haven’t disabled the CONNECT method, then simply add a section to your .ssh/config like this:

Host foobar.example.com
    ProxyCommand          nc -X connect -x proxyhost:proxyport %h %p
    ServerAliveInterval   10

This will tunnel the connection through the HTTP proxy to the remote server. The ServerAliveInterval setting is required as most proxies will drop the connection after a period of inactivity.

To avoid issues with trying to connect to the host when not behind the corporate firewall, replace the above with a fake entry for the proxy method like this:

Host foobar-proxy.example.com
    HostName              foobar.example.com
    ProxyCommand          nc -X connect -x proxyhost:proxyport %h %p
    ServerAliveInterval   10

Then use

$ ssh foobar-proxy.example.com

when inside the firewall, and

$ ssh foobar.example.com

when outside.

Simples, no external software required.

How to build MySQL releases

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 09 Nov 2010 00:00:00 +0000

One of the major benefits from the CMake work available in MySQL 5.5 is that in the MySQL Release Engineering team we have been able to make it easy for users and developers to build MySQL exactly as we do for the official releases. For too long there has been a disconnect between the binaries produced as part of a regular “./configure; make” build and what we ship to users and customers.

We’re still not exactly where we want to be, there are still some parts which need to be integrated into the server tree, but for now it’s relatively straightforward to build exactly as we do.

Here are the instructions, using the mysql-5.5.6-rc.tar.gz source tarball as an example.

tar.gz

These are the generic instructions to build a tarball release. Note that we make use of CMake’s out-of-srcdir support to

ensure the source directory is kept pristine
allow us to easily build both debug and release binaries, and package them together

#!/bin/sh

tar -zxf mysql-5.5.6-rc.tar.gz

# Build debug binaries first, they are picked up in the final 'make package'
mkdir debug
(
  cd debug
  cmake ../mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release -DCMAKE_BUILD_TYPE=Debug
  make VERBOSE=1
)

# Build release binaries and create final package
mkdir release
(
  cd release
  cmake ../mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release
  make VERBOSE=1 package
)

Assuming everything goes ok, you should end up with a tarball in the release/ directory.

Some platforms require extra flags to be specified on the CMake command line to ensure the correct compiler etc is used. Here are some that we use.

Avoid libstdc++ dependancy

In order to create “generic” binaries, on GCC platforms we compile using CXX=gcc. This avoids libstdc++ being pulled in, and means that the server will run across a larger range of releases as you do not rely on having the exact version of libstdc++ installed by your system package manager.

Paste this before running cmake.

# CXXFLAGS is required to trick CMake into believing it is a C++ compiler
$ CXX=gcc; CXXFLAGS="-fno-exceptions"; export CXX CXXFLAGS

Specify target architecture

On systems which support multiple targets, you can specify exactly which one you want, rather than relying on the OS default. These strings are meant to be added to the cmake command lines above.

$ -DCMAKE_OSX_ARCHITECTURES="i386"    # 32bit
$ -DCMAKE_OSX_ARCHITECTURES="x86_64"  # 64bit

GCC/Sun Studio

$ -DCMAKE_C_FLAGS="-m32" -DCMAKE_CXX_FLAGS="-m32"  # 32bit
$ -DCMAKE_C_FLAGS="-m64" -DCMAKE_CXX_FLAGS="-m64"  # 64bit

HP/UX

$ -DCMAKE_C_FLAGS="+DD64" -DCMAKE_CXX_FLAGS="+DD64"

RPM

We’ve spent a lot of time improving the RPM builds too. Previously some of the configuration was only available in the commercial RPM builds, but now we have merged them into the community version.

#!/bin/sh

mkdir -p rpm/{BUILD,RPMS,SOURCES,SPECS,SRPMS} tmp

# Create spec file.
# XXX: We should probably just include this in the source tarball.
tar -zxf mysql-5.5.6-rc.tar.gz
(
  mkdir bld; cd bld
  cmake ../mysql-5.5.6-rc
)

cp bld/support-files/*.spec rpm/SPECS
cp mysql-5.5.6-rc.tar.gz rpm/SOURCES

rpmbuild -v --define="_topdir $PWD/rpm" --define="_tmppath $PWD/tmp" \
 -ba rpm/SPECS/mysql.5.5.6-rc.spec

You should end up with nice shiny RPMs in rpm/RPMS. One thing to note is that the RPM spec no longer runs the test suite as part of the build, so you will need to run that separately. On the plus side, you get your RPMs much quicker.

Another nice thing about the improved RPM spec is that you can now build targetted distribution RPMs, as we do. These have extra dependancy information in them tailored to the target distribution. Currently the spec file supports the distributions we build on, but we will accept patches for others.

To enable this, use:

$ rpmbuild -v --define="distro_specific 1" ...

Windows

Our Windows builds have relied on CMake since MySQL 5.0, but the procedure has still changed to ensure that you can build as we do. These instructions use cmd.exe but you can use the Visual Studio front end if you prefer.

unzip mysql-5.5.6-rc.zip

rem There is no separate 'debug' directory on Windows, as the CMake
rem infrastructure doesn't yet know to pull files in from there on Windows.
md release
cd release

rem Choose your target architecture, 32bit
set VSTARGET=Visual Studio 9 2008
rem or 64bit
set VSTARGET=Visual Studio 9 2008 Win64

cmake ..\mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release \
 -DCMAKE_BUILD_TYPE=Debug -G "%VSTARGET%"

devenv MySql.sln /build Debug

cmake ..\mysql-5.5.6-rc -DBUILD_CONFIG=mysql_release \
 -DCMAKE_BUILD_TYPE=RelWithDebInfo -G "%VSTARGET%"

You can now choose which type of package to create. 5.5 includes new code to create minimal MSI packages, these should work ok, and only differ from the official MySQL MSI packages in that they do not include the instance config wizard.

To create the MSI packages you will need to install WiX.

rem Standard zip package
devenv MySql.sln /build RelWithDebInfo /project package

rem Full MSI package
devenv MySql.sln /build RelWithDebInfo /project msi

rem 'Essentials' MSI package
devenv MySql.sln /build RelWithDebInfo /project msi_essentials

Assuming everything goes ok, you should have some packages in the release\ directory.

Work still to be done

We still have a number of scripts only available internally, for example those we use to create SVR4, DMG and DEPOT packages. However, we are looking to integrate these into the MySQL Server source tree so that all users can benefit from them.

Enjoy!

'apt-get' and 5,000 packages for Solaris10/x86

jonathan@perkin.org.uk (Jonathan Perkin) — Thu, 29 Apr 2010 00:00:00 +0100

Here’s how:

# Install pkg_* tools and the 'pkgin' package manager
$ pkgadd -d http://www.netbsd.org/~jperkin/TNFpkgsrc-x86.pkg all

# Add tools to PATH
$ PATH=/opt/pkg/sbin:/opt/pkg/bin:$PATH

# Update package repository (akin to 'apt-get update')
$ pkgin up

# Search for a particular package (you can use regexp)
$ pkgin search ^ap.*python 

# Install it
$ pkgin install ap22-py25-python

# Update all packages (akin to 'apt-get dist-upgrade')
$ pkgin full-upgrade

# How many packages are available?
$ pkgin avail | wc -l
   4970

Ok, so the headline might be slightly mis-leading, this isn’t really apt-get but a tool which is very similar. This is work which builds upon my previous post using pkgsrc to build binary packages on Solaris.

See http://imil.net/pkgin/ for more information on pkgin.

Hopefully this will prove really useful to people still using Solaris 10 and unable to use the new pkg(5) stuff in OpenSolaris.

Please try it out and provide any feedback to pkgsrc-users@netbsd.org. I’m hoping to keep the packages updated for the 2009Q3 branch.

ZFS and NFS vs OSX

jonathan@perkin.org.uk (Jonathan Perkin) — Wed, 16 Sep 2009 00:00:00 +0100

gromit is my Solaris 10 server, chorlton is my OSX desktop. Explain this:

root@gromit# zfs set sharenfs='anon=shared,sec=none' gromit/store

root@gromit# zfs get sharenfs gromit/store
NAME          PROPERTY  VALUE                 SOURCE
gromit/store  sharenfs  anon=shared,sec=none  local

user@chorlton$ ls /net/gromit/store
ls: cannot open directory /net/gromit/store/: Operation not permitted

root@gromit# zfs set sharenfs='anon=shared' gromit/store

user@chorlton$ ls /net/gromit/store
file1  file2

user@chorlton$ touch /net/gromit/store/file3
touch: /net/gromit/store/file3: Permission denied

root@gromit# zfs set sharenfs='anon=shared,sec=none' gromit/store

user@chorlton$ touch /net/gromit/store/file3
user@chorlton$ echo $?
0

pkgsrc on Solaris

jonathan@perkin.org.uk (Jonathan Perkin) — Sat, 12 Sep 2009 00:00:00 +0100

For many years, building and installing third-party software on Solaris has been a huge pain. For people who do not use pkgsrc, that is.

Originating from the NetBSD project, pkgsrc is a source-based cross-platform package manager. If you’ve used FreeBSD ports, then it is very similar as it derives from the same codebase, so the basic premise is that you:

$ cd /usr/pkgsrc/www/apache22
$ make package

and pkgsrc will download the source for Apache 2.2, compile it and all dependancies, install it on the local system and create a package which can be installed on other similar systems.

However, we’ve taken ports further and applied the NetBSD philosophy of portability, meaning that it not only works on NetBSD, but across all *BSD as well as Linux, OSX, HP/UX, AIX, IRIX, QNX, Windows (via Interix), and of course Solaris.

So while apt-get might be awesome, it only really works on Linux. FreeBSD might have way more ports than us, but only runs on FreeBSD and OSX. pkgsrc provides a consistent interface across all the platforms listed above, and in some cases provides a superior package manager than the system provides.

Here’s how I use pkgsrc on Solaris, in this specific case Solaris 10/x86. Paths are specific to my setup, you can of course change them.

Create a chroot/zone environment

I use the zones feature of Solaris 10 to ensure that all packages are built in a sandbox. This has a number of benefits:

The running system is unaffected by the builds, in that they are not writing to the same file system. This is good when you have misbehaving packages.
You can separate the build and install phases, so that you can verify all the packages have been built and are correct before starting any install/upgrade procedure
It’s easier to catch package mistakes, e.g. unpackaged files.
It avoids pollution from the host environment which may produce bad packages

For creating a zone, I wrote the following create-zone script:

#!/bin/sh

DOMAIN="adsl.perkin.org.uk"
MASQDOMAIN="perkin.org.uk"
PHYSIF="e1000g0"
RPOOL="gromit"

if [ $# -eq 1 ]; then
    name=$1; shift
    ipaddr=`getent hosts ${name} | awk '{print $1}'`
    if [ -z "${ipaddr}" ]; then
        echo "ERROR: Could not determine IP address of $name"
        echo "Either add to hosts database or provide on command line"
        echo
        echo "usage: $0 <name> [ <ipaddr> ]"
        exit 2
    fi
elif [ $# -eq 2 ]; then
    name=$1; shift
    ipaddr=$1; shift
else
    echo "usage: $0 <name> [ <ipaddr> ]"
    exit 2
fi

zfs list ${RPOOL}/zones/$name >/dev/null 2>&1
if [ $? -eq 0 ]; then
    echo "${RPOOL}/zones/$name already exists, not continuing"
    exit 1
fi

# Inherited directories are read-only
cat >/tmp/zonecfg.$$ << EOF
    create
    set zonepath=/zones/$name
    set autoboot=true
    add net
        set address=$ipaddr/24
        set physical=${PHYSIF}
    end
    verify
    commit
    exit
EOF

zonecfg -z $name -f /tmp/zonecfg.$$
rm /tmp/zonecfg.$$

zoneadm -z $name install

#
# Automatically configure new zone with most of the settings from the current
# global, grabbing the encrypted root password directly.  Note that the
# service_profile keyword doesn't seem to be used here, so we configure the
# limited_net profile manually and optionally install a custom site.xml which
# is parsed and activated during first boot.
#
cat << EOF >/zones/$name/root/etc/sysidcfg
network_interface=PRIMARY
{
    hostname=$name.${DOMAIN}
}
name_service=DNS
{
    domain_name=${DOMAIN}
    name_server=`awk '/^nameserver/ { print $NF; exit }' /etc/resolv.conf`
}
nfs4_domain=dynamic
root_password=`awk -F: '/^root/ {print $2}' /etc/shadow`
security_policy=NONE
service_profile=limited_net
system_locale=C
terminal=xterm
timezone=Europe/London
EOF

#
# Use the limited_net profile and install custom site.xml if provided.
#
rm /zones/${name}/root/var/svc/profile/generic.xml
ln -s generic_limited_net.xml /zones/${name}/root/var/svc/profile/generic.xml
if [ -f /install/zones/${name}.xml ]; then
    cp /install/zones/${name}.xml /zones/${name}/root/var/svc/profile/site.xml
fi

#
# Disable nscd host cache
#
ex /zones/$name/root/etc/nscd.conf >/dev/null 2>&1 <<EOF
/enable-cache/s/^#//
wq
EOF

#
# Configure sendmail to masquerade and route via smarthost.
#
ex /zones/$name/root/etc/mail/cf/cf/submit.mc >/dev/null 2>&1 <<EOF
/^dnl/
a
FEATURE(\`masquerade_envelope')dnl
MASQUERADE_AS(\`${MASQDOMAIN}')dnl
dnl
.
/^FEATURE.*msp/s/.127.0.0.1./mail.perkin.org.uk/
wq!
EOF
(
  cd /zones/$name/root/etc/mail/cf/cf
  /usr/ccs/bin/make submit.cf >/dev/null 2>&1
)
cp /zones/$name/root/etc/mail/cf/cf/submit.cf \
   /zones/$name/root/etc/mail/submit.cf

#
# Use SHA512 crypted passwords, and change root's home directory.
#
ex /zones/$name/root/etc/security/policy.conf >/dev/null 2>&1 <<EOF
/^CRYPT_DEFAULT/s/__unix__/6/
wq
EOF
mkdir -m 0700 /zones/$name/root/root
ex /zones/$name/root/etc/passwd >/dev/null 2>&1 <<EOF
/^root/s,:/:,:/root:,
/^root/s,:/sbin/sh,:/bin/bash,
wq
EOF

cp /root/.bash_profile /zones/$name/root/root/.bash_profile

#
# Allow root login over SSH (rest of network takes care of external access
# so this isn't a problem) and enforce SSH key authentication.
#
ex /zones/$name/root/etc/ssh/sshd_config >/dev/null 2>&1 <<EOF
/^PermitRootLogin/s/no$/yes/
/^PasswordAuthentication/s/yes$/no/
wq
EOF
mkdir -m 0700 /zones/$name/root/root/.ssh
cp /home/jperkin/.ssh/id_rsa.pub /zones/$name/root/root/.ssh/authorized_keys

#
# Automount shared directories from global zone using direct mounts
#
ex /zones/$name/root/etc/auto_master >/dev/null 2>&1 <<EOF
a
/-        auto_direct
.
wq
EOF
ex /zones/$name/root/etc/auto_direct >/dev/null 2>&1 <<EOF
a
/content    gromit-lan:/content
/install    gromit-lan:/install
.
wq
EOF

#
# All done!  Fire it up...
#
zoneadm -z $name boot

And the corresponding delete-zone script is:

#!/bin/sh

RPOOL="gromit"

if [ $# -ne 1 ]; then
    echo "usage: $0 <zone>"
    exit 2
fi

zoneadm -z $1 halt
zonecfg -z $1 delete -F
zfs destroy ${RPOOL}/zones/$1

If you want to use them then there are some variables to set at the top, and you may want to scan through them for additional bits to change, for example create-zone copies my ssh public key which will most likely be wrong for your setup :-)

One additional piece of configuration for create-zone is an optional SMF xml file. I use this file to disable inetd inside the zone for additional security, like so:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='profile' name='extract'>
  <service name='network/inetd' type='service' version='0'>
    <instance name='default' enabled='false'/>
  </service>
</service_bundle>

The file should be named <yourzonename>.xml. Mine is named vm-generic.xml and I then create symlinks to it for each VM I want with that default configuration.

Fetch pkgsrc

pkgsrc is developed very rapidly. Tracking nearly 9,000 pieces of third-party software means there are always many updates. Thankfully, we provide quarterly branches for people who want more stability, and I recommend using the latest quarterly release. At time of writing, this is known as pkgsrc-2009Q2. Within the next month or so we will release pkgsrc-2009Q3, and you can figure out the names of future releases yourself.

The easiest way to get pkgsrc is using cvs. I keep stuff like this under /content as opposed to the default of /usr, you can use whatever you wish but will need to change all my example scripts to match where you put it.

$ cd /content
$ BRANCH="pkgsrc-2009Q2"
$ cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -r${BRANCH} -d${BRANCH} -P pkgsrc

Alternatively, you can fetch either bzip2 or gzip archives of the current branch. I recommend the cvs method as, with the branch being updated for security fixes and other important changes, you can easily track it using

$ cd /content/pkgsrc-2009Q2
$ cvs update

pkgsrc configuration

pkgsrc is configured using a mk.conf file, this is mine:

#
# Local settings
#
OS_HOST!= uname -n

#
# Per-host settings
#
#    vm0 builds local packages
#
.if !empty(OS_HOST:Mvm0.*) || !empty(OS_HOST:Mgromit.*)
PKGDIRSUFFIX=
PKG_DEFAULT_OPTIONS=    inet6 perl sasl ssl ncursesw
PKG_OPTIONS.mutt+=      mutt-hcache mutt-smtp
PKG_UID.cyrus=          2000
#PKG_GID.cyrus=         mail
PKG_SHELL.cyrus=        /usr/bin/false
PKG_UID.www=            2001
PKG_GID.www=            2001
PYTHON_VERSION_DEFAULT= 24
X11_TYPE=               modular
#
#    vm1 modular X11 bulk builds
#
.elif !empty(OS_HOST:Mvm1.*)
X11_TYPE=       modular
PKGDIRSUFFIX=   -${X11_TYPE}
#
#    vm2 native X11 bulk builds
#
.elif !empty(OS_HOST:Mvm2.*)
X11_TYPE=       native
PKGDIRSUFFIX=   -${X11_TYPE}
#
#    vm3 pkgsrc dev builds
#
.elif !empty(OS_HOST:Mvm3.*)
X11_TYPE=       modular
PKGDIRSUFFIX=   -${X11_TYPE}
.endif

.if exists(../../CVS/Tag)
PKG_BRANCH!=    ${TOOLS_PLATFORM.awk} -F- '{print $$2}' < ../../CVS/Tag
.else
PKG_BRANCH=     HEAD
.endif

ALLOW_VULNERABLE_PACKAGES=  YES

BULKFILESDIR=           ${WRKOBJDIR}
DISTDIR=                /install/pkgsrc/distfiles/
FAILOVER_FETCH=         YES
INSTALL_UNSTRIPPED=     YES
LINTPKGSRC_DB=          ${_PKGSRCDIR}/.lintpkgsrc.db
MASTER_SITE_OVERRIDE=   ftp://ftp.netbsd.org/pub/NetBSD/packages/distfiles/
PACKAGES=               /install/pkgsrc/packages/${PKG_BRANCH}${PKGDIRSUFFIX}
PKGCHK_CONF=            /install/pkgsrc/misc/pkgchk.conf
SKIP_LICENSE_CHECK=     YES
WRKOBJDIR=              /tmp/pkgsrc

#
# Parse pkgchk.conf and supply list of packages for the bulk build framework.
#
.if defined(SPECIFIC_PKGS)
PKGLIST!= ${TOOLS_PLATFORM.awk} '$$1 !~ /^(\#|$$)/ {print $$1}' ${PKGCHK_CONF}
.  for _pkg_ in ${PKGLIST}
HOST_SPECIFIC_PKGS+= ${_pkg_}
.  endfor
.endif

This is the primary configuration file for pkgsrc. Again, you may need to tailor this to your environment, and may find it useful to read the pkgsrc guide to understand what it all means.

As I do a lot of pkgsrc development I have a number of virtual machines up and running doing various bits and pieces. Obviously I don’t want to copy that mk.conf around, so I also have a small fragment file which is appended to each virtual machine’s mk.conf (using the --mk-fragment argument to bootstrap) and includes the global copy:

.include "/install/pkgsrc/misc/mk.conf"

The bulk build setup in pkgsrc requires its own configuration, and for this you will need to edit a file inside pkgsrc. There is an example file provided, so what I usually do is symlink this to the real copy then I can easily keep it up-to-date via cvs.

$ cd /content/pkgsrc-2009Q2/mk/bulk
$ ln -s build.conf-example build.conf
$ vi build.conf

Again, you can find my personal build.conf here.

Finally, there is a configuration file for pkg_chk which is a package inside pkgsrc which makes managing upgrades easier (ideally it should be a part of the main pkgsrc tools but that’s for another day). pkgchk.conf is a list of package directories, relative to the pkgsrc top level, which are to be built and installed for this setup. If you have a large installation then pkg_chk has extra features to make it possible to share pkgchk.conf across a number of machines and configure packages on a per-host, per-OS etc basis.

Thus, a sample pkgchk.conf:

chat/irssi
chat/irssi-icb
converters/xlhtml
databases/lbdb
databases/rrdtool
devel/autoconf
devel/automake
devel/bzr
devel/bzrtools
devel/diffutils
devel/libtool-base
devel/scmcvs
editors/vim
mail/mutt-devel
misc/screen
net/p5-Net-SNMP
net/p5-Net-SNMP-Interfaces
net/rsync
net/samba
net/tnftp
pkgtools/digest
pkgtools/pkg_chk
pkgtools/rc.subr
print/mp
security/cy2-login
security/cy2-plain
security/sudo
sysutils/coreutils
textproc/antiword
textproc/grep
textproc/urlview
www/apache22
www/w3m

It is highly likely you will want to change the pkgchk.conf file from what I use :-)

Build scripts

Once everything is set up, I have two scripts to build then update my packages, intuitively called build-packages:

#!/bin/sh

PATH="/opt/pkg/sbin:/opt/pkg/bin:/sbin:/usr/sbin:/usr/bin:/usr/ccs/bin"
export PATH

BRANCH="2009Q2"
export BRANCH

PDIR=/install/pkgsrc

if [ ! -f ${PDIR}/bootstrap-${BRANCH}.tar ]; then
    cd /content/pkgsrc-${BRANCH}/bootstrap
    ./bootstrap --workdir=/tmp/pkgsrc --prefix=/opt/pkg \
                --varbase=/var/opt/pkg --sysconfdir=/etc/opt/pkg \
                --mk-fragment=${PDIR}/misc/mk-include.conf \
                --binary-kit=${PDIR}/bootstrap-${BRANCH}.tar
    rm -rf /tmp/pkgsrc
fi

if [ ! -f /opt/pkg/bin/bmake ]; then
    (cd /; tar -xf ${PDIR}/bootstrap-${BRANCH}.tar)
fi

cd /content/pkgsrc-${BRANCH}

/bin/ksh mk/bulk/build -s  2>&1 | tee -a ${PDIR}/logs/build-${BRANCH}.log
/bin/ksh mk/bulk/upload -n 2>&1 | tee -a ${PDIR}/logs/upload-${BRANCH}.log

and update-packages:

#!/bin/sh

PATH="/opt/pkg/sbin:/opt/pkg/bin:/sbin:/usr/sbin:/usr/bin:/usr/ccs/bin"
export PATH

BRANCH=2009Q2
export BRANCH

PKGSRCDIR=/content/pkgsrc-${BRANCH}
export PKGSRCDIR

if [ ! -f /opt/pkg/bin/bmake ]; then
    (cd /; tar -xf /install/pkgsrc/bootstrap-${BRANCH}.tar)
fi

# Run this anyway in case there are updates to pkg_chk
env PKG_PATH=/install/pkgsrc/packages/${BRANCH}/All pkg_add -u pkg_chk

# Dry run first
pkg_chk -aurbn

printf "Do you wish to install? [y/N]: "
read ans
case "${ans}" in
[Yy])
    pkg_chk -aurb 2>&1 | tee -a /install/pkgsrc/logs/pkg_chk-${BRANCH}.log
    ;;
esac

These are pretty simple as all the hard work has all been done. build-packages is ran inside the zone, then update-packages on the main host. These scripts hardcode the name of the branch currently used, so you will need to update this when moving to newer releases.

Quick recap

Ok, so here is the stuff I have for my setup and where I keep them:

Path	Description
`/content/pkgsrc-2009Q2`	Checked out pkgsrc tree, "2009Q2" branch
`/content/scripts/create-zone`	Creates Solaris zone
`/content/scripts/delete-zone`	Uninstalls and deletes zone
`/content/scripts/build-packages`	Bulk build packages inside the zone
`/content/scripts/update-packages`	Updates installed packages
`/install/pkgsrc/misc/mk.conf`	Main pkgsrc configuration file
`/install/pkgsrc/misc/mk-include.conf`	Fragment file included in each zone's `mk.conf`, sources the global `mk.conf`
`/install/pkgsrc/misc/pkgchk.conf`	`pkg_chk` configuration file
`/install/zones/vm-generic.xml`	Shared SMF configuration file, symlinked to from e.g. "`vm0.xml`"

And these are the paths where stuff will be created:

Path	Description
`/install/pkgsrc/distfiles`	Source tarballs of packages
`/install/pkgsrc/packages/2009Q2`	Resulting binary packages
`/tmp/pkgsrc`	Temporary build area for packages
`/content/vwww/www.adsl.perkin.org.uk/pkgstat`	Bulk build results directory

It’s definitely harder than it should be to get this all setup, but the good news is that once it’s done there’s very little maintenance.

Kicking it all off

Once everything is setup:

$ /content/scripts/create-zone vm0
$ ssh vm0 /content/scripts/build-packages
$ /content/scripts/update-packages
$ /content/scripts/delete-zone vm0

This should do the lot. Once build-packages has finished you should, if you configured your email address in build.conf, get an email with the bulk build results which looks similar to this:

http://mail-index.netbsd.org/pkgsrc-bulk/2009/08/23/msg006883.html

A fuller report is available if you configure a web server to serve the pkgstat directory created by the bulk build, and this can help debug problems (again see the above URL for an example).

You will need to add /opt/pkg/sbin:/opt/pkg/bin to $PATH. Configuration files are in /etc/opt/pkg, and log files and metadata are kept in /var/opt/pkg.

This stuff should be obsolete

While this all works well for me, it’s pretty lame for users who just want to install packages and have stuff work. I’m working on providing regular updates of binary packages, including a SVR4 package of the bootstrap kit, so that in theory all a user needs to do is

$ pkgadd TNFpkgsrc.pkg
$ pkg_add apache22
# Upgrade all installed packages to latest releases
$ pkg_chk -aurb

I’m almost there, just needs some tidying up and regular builds. Please feel free to help out!

Jumpstart from OSX

jonathan@perkin.org.uk (Jonathan Perkin) — Tue, 09 Dec 2008 00:00:00 +0000

I recently built a new file server on which I planned to install Solaris 10 10/08. I’m not a fan of CD/DVD installs, so wanted to jumpstart via PXE, though I only had OSX handy.

Here’s how I installed my new machine (gromit.adsl.perkin.org.uk/192.168.1.10) from my iMac (192.168.1.30) over the local network.

Note that since writing this piece I’ve updated to Solaris 10 10/09, and have changed the examples to use that instead.

Step 1, Prepare File System

First off, create a dedicated file system which we can export our jumpstart configuration from. You can probably skip this and just use any existing file system but this way everything is self-contained and we avoid NFS exporting more than we need.

We use HFSX to ensure that the file system is case sensitive, HFS+ can cause problems with pkgadd(1M).

$ hdiutil create -size 1g -type SPARSE -fs HFSX -volname "install" install
$ hdiutil attach install.sparseimage -mountpoint /install

Next up, download and mount sol-10-u8-ga-x86-dvd.iso.

$ open sol-10-u8-ga-x86-dvd.iso

Step 2, NFS

Share /install and the DVD via NFS with the correct options. -alldirs allows clients to mount from any point within that file system (which jumpstart requires), and -maproot=root is also required by jumpstart. As this allows root-owned files to be created, make sure you understand the security risks.

$ sudo vi /etc/exports

/install                  -alldirs -maproot=root
/Volumes/SOL_10_1009_X86  -alldirs -maproot=root

$ sudo nfsd checkexports && sudo nfsd enable

Step 3, DHCP

For DHCP I happen to already use my Cisco router as a DHCP server on the local network, so added the following configuration:

ip dhcp pool gromit.adsl.perkin.org.uk
   host 192.168.1.10 255.255.255.0
   hardware-address xxxx.xxxx.xxxx
   bootfile /boot/grub/pxegrub
   next-server 192.168.1.30
   client-name gromit
   domain-name adsl.perkin.org.uk
   dns-server xxx.xxx.xxx.xxx
   default-router 192.168.1.1

however, given this is a guide for setting everything up under OSX I also tried using ISC DHCP on OSX to prove it can be done that way too.

I used pkgsrc to install it (I’ll add another blog some time showing how to set up pkgsrc)

$ cd /usr/pkgsrc/net/isc-dhcpd
$ sudo bmake package

And here is my DHCP configuration file in full:

option domain-name "adsl.perkin.org.uk";
option domain-name-servers xxx.xxx.xxx.xxx;
ddns-update-style none;
authoritative;
log-facility local7;

subnet 192.168.1.0 netmask 255.255.255.0 {
    option routers 192.168.1.1;
}

group {
    filename "/boot/grub/pxegrub";
    next-server 192.168.1.30;

    host gromit {
        hardware ethernet xx:xx:xx:xx:xx:xx;
        fixed-address 192.168.1.10;
        option host-name "gromit.adsl.perkin.org.uk";
    }
}

Finally, start DHCP with:

$ sudo /usr/pkg/sbin/dhcpd

Most parts of these configurations should be self-explanatory. The /boot/grub/pxegrub entry is important for our next step, and I’d recommend using that exact pathname for reasons explained later.

Step 4, TFTP

Now, enable the TFTP server which comes with OSX. I added the -s option so tftpd would chroot to the tftpboot directory, both for security reasons and also to ensure that paths specified as /path/to/file would work correctly (relative to /install/tftpboot).

I also changed the location of the tftpboot directory so that everything was self-contained within the UFS image. In previous attempts I didn’t do this and ran into problems with GRUB which I think are again caused by case-insensitive file systems.

$ sudo vi /System/Library/LaunchDaemons/tftp.plist

[...]
    <key>ProgramArguments</key>
    <array>
        <string>/usr/libexec/tftpd</string>
        <string>-i</string>
        <string>-s</string>
        <string>/install/tftpboot</string>
    </array>
[...]

$ mkdir /install/tftpboot
$ sudo launchctl load -w /System/Library/LaunchDaemons/tftp.plist

You can then create a test file and check that it’s working as you expect, using:

$ echo "testing" >/install/tftpboot/testfile
$ printf "verbose\ntrace\nget testfile\n" | tftp localhost
$ rm /install/tftpboot/testfile

Step 5, GRUB

Next up, configure PXE booting using GRUB. We need to copy the GRUB images and configuration from the Solaris install DVD then modify it for our environment:

$ rsync -av /Volumes/SOL_10_1009_X86/boot/grub /install/tftpboot/boot/
$ rsync -av /Volumes/SOL_10_1009_X86/boot/multiboot /install/tftpboot/sol10u8x/
$ rsync -av /Volumes/SOL_10_1009_X86/boot/x86.miniroot /install/tftpboot/sol10u8x/

As we are copying the boot files from the DVD, they come hardcoded with particular pathnames to e.g. the menu.lst file. While it may be possible to pass extra parameters to pxegrub and load this from a different path, I simply recommend doing as I do and replicating the /boot/grub/ path structure so that everything Just Works.

The menu.lst file includes kernel arguments and allows you to choose which type of install to perform at startup. My file listed below has 3 choices:

Unattended install using a graphical environment (if available). The “install” keyword after the kernel instructs it to perform an unattended install, so long as it can find the necessary settings from sysidcfg etc.
As above, but force the use of the console and do not start a graphical environment (using the “nowin” keyword)
A manual install, so you need to go through the steps of layout out disks, selecting packages, etc.

$ vi /install/tftpboot/boot/grub/menu.lst

default=0
timeout=60

title Solaris PXE Unattended Install
    kernel /sol10u8x/multiboot kernel/unix - install -B \
      install_media=192.168.1.30:/Volumes/SOL_10_1009_X86,\
      sysid_config=192.168.1.30:/install/jumpstart,\
      install_config=192.168.1.30:/install/jumpstart
    module /sol10u8x/x86.miniroot

title Solaris PXE Unattended Install (console)
    kernel /sol10u8x/multiboot kernel/unix - install nowin -B \
      install_media=192.168.1.30:/Volumes/SOL_10_1009_X86,\
      sysid_config=192.168.1.30:/install/jumpstart,\
      install_config=192.168.1.30:/install/jumpstart
    module /sol10u8x/x86.miniroot

title Solaris PXE Manual Install
    kernel /sol10u8x/multiboot kernel/unix -B \
      install_media=192.168.1.30:/Volumes/SOL_10_1009_X86
    module /sol10u8x/x86.miniroot

Anyone used to doing jumpstart but with RARP/bootparams will notice the symmetry between install_config etc in the GRUB configuration and similar options in /etc/bootparams. Make sure that the full kernel arguments are all on one line, and that there are no spaces in between the install_media=..,sysid_config=.. options.

Step 6, Jumpstart

Finally, set up your Jumpstart configuration. Here’s what I personally use, you may want something different:

$ mkdir /install/jumpstart
$ cd /install/jumpstart
$ vi sysidcfg

name_service=DNS
{
    domain_name=adsl.perkin.org.uk
    name_server=xxx.xxx.xxx.xxx
}
network_interface=PRIMARY
{
    default_route=192.168.1.1
    netmask=255.255.255.0
    protocol_ipv6=yes
}
nfs4_domain=dynamic
root_password=xxxxxxxx
terminal=xterm
timeserver=localhost
timezone=Europe/London
security_policy=NONE
service_profile=limited_net
system_locale=C

Ordinarily this file is processed using a check script available in the jumpstart_sample directory on the Solaris DVD, however this only works from a Solaris host. To create the rules.ok file, we need to strip out any comments and put entries on one line, then create the checksum (although this isn’t actually necessary).

$ vi rules # emacs sucks :)

hostname gromit.adsl.perkin.org.uk - profile -

$ cp rules rules.ok
$ echo "# version=2 checksum=$(cksum -o 2 rules | awk '{print $1}')" >> rules.ok

Machine profile. This gives me a full Solaris install (minus OEM stuff) on mirrored ZFS disks with additional dump/swap space (the defaults made dump a bit too small I found).

$ vi profile

install_type    initial_install
pool            store auto 4g 4g mirror c1t0d0s0 c1t1d0s0
bootenv         installbe bename sol10u8x
cluster         SUNWCall

Step 7, Make A Cup Of Tea

With everything set up you should be able to enable PXE booting in your BIOS and watch it automatically install. One small minor problem you may have if you don’t have a BIOS which allows you to hit F12 or similar and choose PXE booting for one boot only is that it will infinitely cycle through installing, rebooting, installing, rebooting.. until you change your boot options.

If this happens, I recommend making more cups of tea until you happen to return in time to change the BIOS settings. If you aren’t able to do this for a while, you may need to add the extra steps 8, 9 and 10 titled “Visit The Bathroom”.

Set up local caching DNS server on OSX 10.4

jonathan@perkin.org.uk (Jonathan Perkin) — Mon, 31 Dec 2007 00:00:00 +0000

OSX 10.4 ships with ISC named, but is not configured correctly out of the box. Follow the instructions below to run a local caching name server.

First, we need to generate an rndc key. The default named.conf comes configured to use one, but if it does not exist will bail with an error. Simply run rndc-confgen(1) to create one.

$ sudo rndc-confgen -a

Next, download the latest named.root file, as the supplied named.ca is rather outdated (as you can see from the diff output).

$ cd /var/named
$ sudo curl -O ftp://ftp.internic.net/domain/named.root
$ diff -u named.ca named.root

Now edit /etc/named.conf, we need to make three changes. Firstly, the rndc port is wrong in the default configuration, so make it:

        inet 127.0.0.1 port 953 allow {any;}

Next, add a listen-on directive in the options section to instruct named to only listen on certain interfaces, ensuring that we keep as secure as possible. I have added the internal 192.168/16 LAN to my configuration so that Parallels VMs and other machines can use the server, but if you do not need this then just keep the localhost entry:

        directory "/var/named";
        listen-on {
                127.0.0.1;
                192.168/16;
        };

Finally, change the root DNS server file from named.ca to named.root as downloaded above. You could simply overwrite the named.ca file instead with named.root, however I like to keep defaults around as much as possible - you never know if an update will blat your changes or not:

zone "." IN {
        type hint;
        file "named.root";
};

All done. All that’s left to do is (re)start the service:

$ sudo service org.isc.named stop
$ sudo service org.isc.named start

then run a quick test to ensure it is working correctly:

$ dig www.perkin.org.uk @localhost

If you get a reply, configure your DNS server to be 127.0.0.1 and you’re all set.

Jonathan Perkin

What's new in pkgsrc-2013Q1

Installing

SmartOS/illumos

OSX

What’s New

OpenSSL 1.0.1 with AES-NI support

GCC Go support for SmartOS/illumos

Networking utilities on SmartOS

Major package versions

Installing SVR4 packages on SmartOS

Unpacking SVR4 packages

install sub-directory

pkginfo

pkgmap

reloc/ sub-directory

Creating pkgsrc binary package

pkgsrc metadata

pkgsrc INSTALL script

pkgsrc files

Create the package

Testing

Further work

SmartOS is Not GNU/Linux

Install the latest platform

Get the SNGL dataset

Create a new dataset

Brand configuration

Runtime and packages

Reporting issues

SmartOS development preview dataset

How to install

Known issues

pkgsrc on SmartOS - fixing broken builds

Adjusting the environment

The substitution framework

Patches

Summary

pkgsrc on SmartOS - zone creation and basic builds

Create a build machine

Install dependencies

Fetch pkgsrc

Fetch pkgsrc-wip

Configure mk.conf

Building a package

Cleanup

Multi-architecture package support in SmartOS

Getting started

Multi-architecture libraries

Multi-architecture binaries

Compiler support

Interpreter support

Coverage

Solaris portability - cfmakeraw()

Summary

Solaris portability - flock()

flock() implementation

Use fcntl() instead!

Set up struct flock

Add fcntl() calls

An important note on semantics

flock() -> fcntl() cheat sheet

flock() wrapper

Summary

pkgsrc-2012Q4 illumos packages now available

SmartOS and the global zone

The global zone is a ramdisk

So what can I do?

/zones

/var

/opt

/usbkey

/etc/shadow and /etc/ssh

Implementation

GRUB

root_shadow=’crypt string’

smartos=true

noimport=true

Platform image

Upgrades

`install` sub-directory

`pkginfo`

`pkgmap`

`reloc/` sub-directory