Set up anonymous FTP upload on Oracle Linux

January 11th, 2012 No comments

Just because this took a little longer than I expected, here’s a quick howto for setting up an anonymous FTP drop-off on Oracle Linux, which I use as a simple way to transfer files out of my Virtual Machines.

Install vsftpd:

yum -y install vsftpd

Configure iptables. As FTP is a more complicated protocol than most, there is a special netfilter module required in order to correctly keep track of connections.

# You will perhaps want to change the insert number here.
iptables -I INPUT 4 -m state --state NEW -p tcp --dport 21 -j ACCEPT
/etc/init.d/iptables save

# Add 'nf_conntrack_ftp' to IPTABLES_MODULES
vi /etc/sysconfig/iptables-config

# Load the module rather than reboot
modprobe nf_conntrack_ftp

Create /incoming area and ensure it has the correct file permissions and SELinux context. This is the bit which had me stumped for a little while, as I didn’t know about allow_ftpd_anon_write, and while I normally just disable SELinux, I do also like to know how things should work (and be able to write about them!):

mkdir /var/ftp/incoming
chown ftp:ftp /var/ftp/incoming

# This allows anonymous users to upload, but not see what's in the directory
chmod 750 /var/ftp/incoming

chcon -u system_u -t public_content_rw_t /var/ftp/incoming
setsebool allow_ftpd_anon_write=1

Configure vsftpd:

# Set 'anon_upload_enable=YES'
vi /etc/vsftpd/vsftpd.conf

Finally, enable and start vsftpd:

chkconfig vsftpd on
/etc/init.d/vsftpd start

And that’s it, you should now be able to FTP as anonymous and upload files into /incoming.

Kickstart Oracle Linux in VirtualBox

January 9th, 2012 3 comments

In my previous post I configured an Ubuntu laptop as a Kickstart install server for a physical machine I wanted to build.

Now that everything is configured for automated installs, it makes sense to use the same infrastructure to install Virtual Machines too. Everything can be done from the command line, and new Virtual Machines can be provisioned very quickly.

Again, I will use Oracle Linux 6.2 as an example.

Create Virtual Machine

These commands create a new Virtual Machine, including disk, and configures it for network booting (if you don’t already have VirtualBox installed, sudo apt-get install virtualbox).

The name of the Virtual Machine as it appears in VirtualBox.

VM="Oracle Linux 6.2"

Create the VM.

VBoxManage createvm --name "${VM}" --ostype "Oracle_64" --register

Create the hard disk (32GB expanding) and attach it via SATA. Note that I store my VMs in ${HOME}/VirtualBox rather than the default ${HOME}/VirtualBox VMs.

VBoxManage createhd --filename "VirtualBox/${VM}/${VM}.vdi" --size 32768

VBoxManage storagectl "${VM}" --name "SATA Controller" --add sata \
  --controller IntelAHCI

VBoxManage storageattach "${VM}" --storagectl "SATA Controller" --port 0 \
  --device 0 --type hdd --medium "VirtualBox/${VM}/${VM}.vdi"

Default RAM is 128MB, Oracle Linux installer requires at least 512MB however. Once installed we can drop back down to 256MB or so.

VBoxManage modifyvm "${VM}" --memory 512

Configure boot order. Put disk first, as on the first boot there is nothing on it so it falls through to PXE for install, then after the install the disk is bootable.

VBoxManage modifyvm "${VM}" --boot1 disk --boot2 net --boot3 none --boot4 none

Change NIC type from the default e1000, as a vanilla VirtualBox install does not include the firmware necessary to network boot from that device – it is available in the “VirtualBox Extension Pack” add-on. Switch to a plain PCNet Fast III which does include PXE firmware.

VBoxManage modifyvm "${VM}" --nictype1 Am79C973

VirtualBox does have the ability to serve TFTP directly from the file system by placing files inside ${HOME}/.VirtualBox/TFTP/ but I prefer to just use the network as it’s already configured.

VBoxManage modifyvm "${VM}" --nattftpserver1 10.0.2.2
VBoxManage modifyvm "${VM}" --nattftpfile1 pxelinux.0

Configure pxelinux/kickstart

We just need a couple of tweaks to the configs from the last blog entry, as the network addresses are different inside VirtualBox, and we also may want a different kickstart configuration.

sudo vi /var/lib/tftpboot/pxelinux.cfg/default

I just amended the existing entry to point to 10.0.2.2 which is the address of the machine running VirtualBox, and a different ks-vm.cfg kickstart configuration file, but you could also create a new label if you wanted to regularly switch between different configurations:

LABEL ol6.2
        KERNEL /ol6.2/vmlinuz
        APPEND initrd=/ol6.2/initrd.img ks=http://10.0.2.2/ks-vm.cfg
sudo cp /usr/share/nginx/www/ks.cfg /usr/share/nginx/www/ks-vm.cfg

For virtual machines I use a slightly different configuration compared to previously. I’ve only shown the changes below, not the full file:

# Update network configuration for DHCP instead of static
network --bootproto=dhcp
url --url=http://10.0.2.2/ol6.2

# Don't specify disks, just use default layout
bootloader --location=mbr --driveorder=sda
clearpart --all --initlabel

# Just add 'screen' to the default set of @base and @core packages.
%packages
screen
%end

Start Virtual Machine

All that’s left to do is to boot up the VM, and everything else should run automatically.

VirtualBox --startvm "${VM}"

All done!

Kickstart Oracle Linux from Ubuntu

January 9th, 2012 No comments

As my new job involves working on Oracle Linux, I figured I should migrate my home server to it, which would also mean I could move it to a proper RAID10 configuration rather than relying on multiple RAID1′s.

My laptop runs Ubuntu, and I wanted to install the server from it using PXE and Kickstart, so here’s how I did it.

Configure dhcpd

DHCP is required for two things, to give the server its network configuration, and to point it at the pxe boot loader we want to use.

sudo apt-get install isc-dhcp-server
sudo vi /etc/dhcp/dhcpd.conf

Here’s the configuration I used, which says to configure 192.168.2.0/24 with a dynamic DHCP range between 192.168.2.100 – 192.168.2.200, and to boot machines using pxelinux.0 which is relative to the TFTP root directory (configured in the next section):

subnet 192.168.2.0 netmask 255.255.255.0 {
    range 192.168.2.100 192.168.2.200;
    filename "pxelinux.0";
}

The isc-dhcp-server install automatically tries to start the server, but will fail as it isn’t configured, so we restart it now that there is a working configuration installed

sudo /etc/init.d/isc-dhcp-server restart

Configure tftpd

TFTP is a simple protocol used to transfer files over the network, and due to its simplicity it is the primary way to network boot, as it can be easily embedded into firmware.

All we need to do is install the TFTP daemon and syslinux which includes the pxe boot loader, then put the pxelinux file into the tftproot area:

sudo apt-get install syslinux tftpd-hpa
sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot

Configure Oracle Linux DVD

To save space on the laptop we can just mount the DVD read-only and install from that:

sudo mkdir /media/ol6.2
sudo mount -o loop,ro /path/to/OracleLinux-R6-U2-Server-x86_64-dvd.iso \
  /media/ol6.2

However, we do need to copy the kernel and initrd image from the DVD into the tftproot as they are required for booting:

sudo mkdir /var/lib/tftpboot/ol6.2 /var/lib/tftpboot/pxelinux.cfg
sudo cp -a /mnt/images/pxeboot/{initrd.img,vmlinuz} /var/lib/tftpboot/ol6.2/

Configure pxelinux

All that’s left for the PXE stage is to configure the boot loader, and tell it what kernel and initrd we want to use:

sudo vi /var/lib/tftpboot/pxelinux.cfg/default

This configuration has just one entry which is booted after a short wait, but pxelinux has many more options, including the ability to boot from local disk.

DEFAULT	ol6.2
PROMPT 1
TIMEOUT 5

LABEL ol6.2
	KERNEL /ol6.2/vmlinuz
	APPEND initrd=/ol6.2/initrd.img ks=http://192.168.2.1/ks.cfg

Note the ks argument which specifies the kickstart file we will use, that and the web server required to serve it will be set up next.

Configure nginx

I chose nginx as it is small and simple to configure, but any web server will do.

sudo apt-get install nginx
sudo vi /etc/nginx/sites-available/default

The install will be performed over HTTP, so we need to make the DVD we mounted earlier available. This entry in the server { } section makes the DVD available via http://192.168.2.1/ol6.2/:

server {
[...]
        location /ol6.2 {
                root /media;
                autoindex on;
                allow all;
        }

Then start nginx (unlike isc-dhcp-server this isn’t done automatically):

sudo /etc/init.d/nginx start

Configure kickstart

Finally, we create a kickstart configuration which specifies exactly how our target machine is to be installed, and this allows a completely unattended installation.

Ideally I should create a specific area for holding files like this, but as a quick hack I simply put it into the default nginx web root (and thus available as http://192.168.2.1/ks.cfg as configured earlier in the pxelinux.cfg/default file:

sudo vi /usr/share/nginx/www/ks.cfg

Here is my ks.cfg file in full. The only thing missing is a rootpw line to automatically set a root password, however for maximum security I am happy to forego a completely unattended installation and instead have the installer prompt me to type it in during the install.

Some notes:

  • UK keyboard language and timezone selected.
  • Automatically reboot when the installer is finished.
  • Point to the Oracle Linux DVD using the url directive.
  • I have 4 disks configured with RAID1 for /boot, and RAID10 for swap and /.
  • Disks are referred to by path, to ensure correct ordering.
  • A small set of packages are installed, containing just the functionality I require.
  • A small %post section is used to perform any fixes I want for the first boot.
#
# Miscellaneous options.
#
install
keyboard uk
lang en_GB.UTF-8
reboot
selinux --enforcing
timezone Europe/London

#
# User setup.  I'd create my local user here and configure sudoers, but
# kickstart doesn't yet support creating a user with uid/gid of 1000 (the
# gid is always 500, even if you add the named group first).
#
authconfig --enableshadow --passalgo=sha512

#
# Networking.  The 'network' line needs to be on a single line
# for kickstart to work - it is only split here for the blog.
#
firewall --service=ssh
network --bootproto=static \
        --hostname=gromit.adsl.perkin.org.uk \
        --ip=192.168.2.10 \
        --netmask=255.255.255.0 \
        --gateway=192.168.2.1 \
        --nameserver=193.178.223.141,208.72.84.24 \
        --ipv6=auto
url --url=http://192.168.2.1/ol6.2

#
# Disk configuration.
#
bootloader --location=mbr --driveorder=sda,sdb,sdc,sdd
clearpart --all --initlabel
#
# /boot (RAID1 necessary as booting from RAID10 isn't supported)
#
part raid.00 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.01 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.02 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.03 --asprimary --size=1024 --ondisk=/dev/disk/by-path/pci-*-3*0
raid /boot --level=1 --device=md0 --fstype=ext4 raid.00 raid.01 raid.02 raid.03
#
# swap, RAID10 of size RAM+2GB, give or take..
#
part raid.10 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.11 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.12 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.13 --asprimary --size=6144 --ondisk=/dev/disk/by-path/pci-*-3*0
raid swap --level=10 --device=md1 --fstype=swap raid.10 raid.11 raid.12 raid.13
#
# /, RAID10 of remainder (have to specify an arbitrary --size even with --grow)
#
part raid.20 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-0*0
part raid.21 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-1*0
part raid.22 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-2*0
part raid.23 --asprimary --size=1024 --grow --ondisk=/dev/disk/by-path/pci-*-3*0
raid / --level=10 --device=md2 --fstype=ext4 raid.20 raid.21 raid.22 raid.23

#
# Packages.  @base and @core are pre-selected.
#
%packages
@cifs-file-server
@console-internet --optional
@development
@legacy-unix --optional
@mail-server
@network-server --optional
@network-tools
@nfs-file-server
@web-server
screen
%end

#
# Post-install fix-ups.
#
%post
#
# The 'network' directive doesn't support DNS search paths, so set those
# manually, and disable Network Manager.
#
printf "/^NM_CONTROLLED/s/yes/no/\nw\nq\n" \
  | ed /etc/sysconfig/network-scripts/ifcfg-eth0
printf "/^#/s/.*/search adsl.perkin.org.uk perkin.org.uk/\nw\nq\n" \
  | ed /etc/resolv.conf
#
# Disable unwanted services
#
chkconfig --del cups
%end

All done.

Last day at MySQL

December 22nd, 2011 5 comments

Today is my final day working on MySQL. It has been an amazing 4.5 years, and I’ve loved working on the technical challenges involved in producing a piece of software which runs on so many different platforms, as well as working with so many talented individuals. I have learned a lot, which is just how I like it.

Times have changed since I first joined MySQL AB, of course. But despite going through two acquisitions, the day to day work hasn’t changed much at all. I still get to work from home, which is something I am passionate about and feel many companies are missing out on massively by still being stuck in an outdated 20th century mindset. Oracle have continued to invest in MySQL, providing additional headcount and extra hardware. And the vast majority of people I started working with back in 2007 are still with the company (don’t believe the FUD, folks) and working harder than ever.

I’m most proud of the work we have done internally to ensure our MySQL releases are in excellent shape. When I first joined, our internal ‘PushBuild’ system was very limited, only able to handle a small number of pushes with reduced testing across a few platforms, and the builds produced were completely different to those that were released.

Nowadays, thanks to the investment made by Sun and Oracle, we have a large server farm producing thousands of builds and hundreds of gigabytes of data every day across all our supported platforms. Those which are based on MySQL 5.5 or newer have additional package verification tests, using chroots and virtual machines, to ensure that the RPM/MSI/etc packages can be installed, run, and uninstalled, all automatically and on every push by developers. We run additional nightly and weekly tests which extend the default set of test suites. Our 5.5+ releases are produced directly in PushBuild. And we are looking to extend this to all MySQL products, not just the Server.

If the above sounds fun, keep an eye out for job posts, as the Release Engineering team is looking to expand, and they are an amazing group of people to work with.

I certainly found it fun, which is why I’m going to continue in that line of work, and stay within Oracle, but move to the Linux group. To be honest, databases have never been a passion of mine, however I am rather fond of Operating Systems, and so I am really looking forward to continuing to work on continuous integration and testing, but with Oracle Linux and Oracle VM instead of MySQL. Plus, it will be nice to go back to a purely technical role – I have learned over the last year or so that management isn’t really my thing :)

To all my colleagues, past and present, thank you for the wonderful ride. I have many, many good memories, and hopefully we will keep in touch.

Tags:

Installing OpenBSD with softraid

December 15th, 2011 No comments

This is more of a log for me than anything else, but perhaps someone will find this useful.

OpenBSD includes a software RAID implementation which supports booting in newer snapshots, and I was itching to install the latest version and use it as my file server, which has 4 750GB disks. There is a small bit of preparation work to do prior to installing, which is the bulk of this entry, most of which is based on this undeadly.org article.

Grab latest amd64 snapshot , boot it, drop into (S)hell mode and set up the disks:

cd /dev
sh MAKEDEV sd1 sd2 sd3 sd4 sd5
for disk in 0 1 2 3
do
  # Clear beginning of disks..
  dd if=/dev/zero of=/dev/rsd${disk}c bs=1m count=10

  # ..and initialise new partition table
  fdisk -iy sd${disk}

  #
  # Create BSD disklabel:
  #
  # - 128m partitions at start to hold kernels for booting
  # - 4g spare raid on each disk for testing
  # - rest raid for main OS and data
  #   - OS and /home on first two mirrored disks
  #   - /store on second two mirrored disks
  #
  print "a a\n\n128m\n\na d\n\n4g\nraid\na e\n\n\nraid\nw\nq\n" \
    | disklabel -E sd${disk}

  # Clear beginning of raid partitions
  dd if=/dev/zero of=/dev/rsd${disk}d bs=1m count=10
  dd if=/dev/zero of=/dev/rsd${disk}e bs=1m count=10
done

# Create RAID1 mirrors
bioctl -c 1 -l sd0e,sd1e softraid0
bioctl -c 1 -l sd2e,sd3e softraid0

# Exit shell and start the (I)nstall
exit

As for the install, go with the sensible defaults, other than:

  • change keyboard layout to ‘uk’
  • manually configure network, enable rtsol
  • start ntpd
  • do not expect to run X

When it comes to disk selection, choose sd4 as the root disk, and use the following layout:

# partition  size  mount
  sd4a       1G    /
  sd4b       8G    swap
  sd4d       1G    /tmp
  sd4e       8G    /var
  sd4f       16G   /usr
  sd4h       rest  /home

then initialise sd5 with:

# partition  size  mount
  sd5a       2G    /altroot
  sd5d       8G    /scratch
  sd5e       rest  /store

Install the full OS, set the correct timezone, then before rebooting initialize the boot partitions and copy the kernels to them. Doing this on all of them means we can boot from any disk.

for disk in 0 1 2 3
do
  newfs sd${disk}a
  mount /dev/sd${disk}a /mnt2
  cp /mnt/bsd* /mnt2
  umount /mnt2
done
eject cd0a
reboot

Job done.

Tags: ,

Create VirtualBox VM from the command line

September 21st, 2011 No comments

As something of a follow-up post to the previous entry, here’s a quick recipe for creating a Virtual Machine using the VirtualBox command line tools:

We’re using Windows Server 2008 64bit as an example, modify to taste.

VM='Windows-2008-64bit'

Create a 32GB ‘dynamic’ disk.

VBoxManage createhd --filename $VM.vdi --size 32768

You can get a list of the OS types VirtualBox recognises using:

VBoxManage list ostypes

Then copy the most appropriate one into here.

VBoxManage createvm --name $VM --ostype "Windows2008_64" --register

Add a SATA controller with the dynamic disk attached

VBoxManage storagectl $VM --name "SATA Controller" --add sata \
  --controller IntelAHCI
VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 \
  --device 0 --type hdd --medium $VM.vdi

Add an IDE controller with a DVD drive attached, and the install ISO inserted into the drive

VBoxManage storagectl $VM --name "IDE Controller" --add ide
VBoxManage storageattach $VM --storagectl "IDE Controller" --port 0 \
  --device 0 --type dvddrive --medium /path/to/windows_server_2008.iso

Misc system settings

VBoxManage modifyvm $VM --ioapic on
VBoxManage modifyvm $VM --boot1 dvd --boot2 disk --boot3 none --boot4 none
VBoxManage modifyvm $VM --memory 1024 --vram 128
VBoxManage modifyvm $VM --nic1 bridged --bridgeadapter1 e1000g0

Configuration is all done, boot it up! If you’ve done this one a remote machine, you can RDP to the console via vboxhost:3389

VBoxHeadless -s $VM

Once you have configured the operating system, you can shutdown and eject the DVD.

VBoxManage storageattach $VM --storagectl "IDE Controller" --port 0 \
  --device 0 --type dvddrive --medium none

Finally, it’s a good idea to take regular snapshots so that you can always revert back to a known-good state rather than having to completely re-install.

VBoxManage snapshot $VM take <name of snapshot>

And, if you need to revert back to a particular snapshot:

VBoxManage snapshot $VM restore <name of snapshot>

Enjoy!

Tags:

Creating chroots for fun and MySQL testing

September 14th, 2011 No comments

Virtualisation is all the rage today, however there are still a few cases where good old-fashioned Unix chroot is still applicable, and testing MySQL across multiple platforms and architectures is one of those cases.

At Oracle we do full automated package verification testing of our MySQL server binaries prior to release, which attempts to install the package, start the server, run some functionality testing, then uninstall. It is of course highly desirable that the testing environment this is performed in is as close to a clean install of the target operating system as possible, to avoid problems such as our packages depending upon some local changes or packages we may have installed which won’t be available on a customer system.

Given the large number of platforms and architectures that MySQL supports, going the virtualisation route would mean having to use many different products: VirtualBox for x86, zones for SPARC, qemu for ia64/PA-RISC/others (if it even supports them), and this gets complicated quickly and is not very maintainable. Thus I chose to use chroot as much as possible. In addition, it’s much faster and less intensive on resources to use a chroot than boot up an entire OS image each time.

I built all images directly from the original installation images (DVD, ISO, RPM, etc), to ensure that there was no contamination from our build environment or local install scripts in the image – they should be as close to what a normal user or customer will be running in their setup. From the install image, the packages are installed to a temporary directory, some final modifications are made, then the directory is tarred up ready to be extracted by the test framework and used.

Here are some operating system specific examples, which set up an extracted chroot image into ${CHROOTDIR}. There may be additional steps required to get a fully functioning chroot, such as copying device files (/dev/zero and /dev/null are usually the minimum requirements) and adding users.

FreeBSD

FreeBSD 7 and 8 come as a number of sets in tar format, and for our purposes we only need to extract the base set. You may wish to add more sets if you want to use your chroot for building packages:

mdunit=$(mdconfig -a -t vnode -o readonly -f /path/to/dvd1/of/freebsd.iso)
mount_cd9660 /dev/${mdunit} /mnt
cat /mnt/*/base/base.?? | tar -xpzf - -C ${CHROOTDIR}
umount /mnt
mdconfig -d -u ${mdunit}

HP-UX

HP-UX has since been EOL’d for MySQL, however this information might still prove useful. The HP-UX installation media contains per-directory packages, with the contents representing how they are laid out on the destination file system with each file gzip compressed.

cd /path/to/extracted/hpux-dvd1
for pkg in $LIST_OF_PKGS
do
  for subpkg in ${pkg}/*
  do
    if [ ! -d "${subpkg}" ]; then
      continue
    fi
    for d in $(find ${subpkg} -type d)
    do
      mkdir -p ${CHROOTDIR}/$(echo ${d} \
        | sed -e "s#${subpkg}/##g" \
              -e "s#usr/newconfig/##g")
    done
    for f in $(find ${subpkg} -type f)
    do
      gzip -dc ${f} >${CHROOTDIR}/$(echo ${f} \
        | sed -e "s#${subpkg}/##g" \
              -e "s#usr/newconfig/##g")
    done
  done
done

Once this is done you’ll need to fix up permissions in bin and lib directories (make files executable), as well as create a bunch of symlinks for e.g. /bin and /lib.

OSX

For OSX we don’t actually use a chroot tarball but instead create a sparse disk image. Currently the size of the “chroot” is very large as there’s no easy way to strip down an OSX install, so mounting a disk image is faster than unpacking a chroot, plus it preserves various HFS-specific attributes.

You will likely need at least the BSD, BaseSystem, and Essentials packages.

# Create a sparse image to hold the chroot (which isn't really a directory)
hdiutil create -fs HFS+ -size 8g -type SPARSE -volname osx-chroot ${CHROOTDIR}
hdiutil attach -mountpoint ${CHROOTDIR} ${CHROOTDIR}.sparseimage
# Either attach a DVD image or the real thing
hdiutil attach -mountpoint /Volumes/osx-install /path/to/dvd
# Install packages
for pkg in BSD BaseSystem Essentials
do
  installer -verbose \
   -pkg /Volumes/osx-install/System/Installation/Packages/${pkg}.pkg \
   -target ${CHROOTDIR}
done
# Unmount
hdiutil detach ${CHROOTDIR}
hdiutil detach /Volumes/osx-install

Red Hat / Oracle Linux / SuSE

For RPM-based distributions we use rpm to directly install packages into the chroot. The list of RPMs we install varies quite a lot from release to release, usually by having to increase the number (RH3: 81, RH4: 85, RH5: 114, RH6: 203 – for the same functionality).

# Kludge for 'setup' RPM to install
mkdir -p ${CHROOTDIR}/var/lock/rpm
# If installing from an ISO:
mount -o loop /path/to/iso /mnt
# Path varies from release to release
cd /mnt/path/to/RPMs
rpm --root=${CHROOTDIR} -Uvh ${LIST_OF_RPMS}

Solaris

Similar to RPM, we use the native package manager to install packages directly into the chroot directory:

# Avoid prompts
sed -e "s/ask$/nocheck/" /var/sadm/install/admin/default > /tmp/admin-$$
pkgadd -a /tmp/admin-$$ -R ${CHROOTDIR} -d . ${LIST_OF_PKGS}
rm /tmp/admin-$$

Windows

Ok, so of course we can’t use chroot images for Windows, as it doesn’t have chroot(2). So here we use VirtualBox and its snapshot ability to load a clean snapshot of a basic Windows install, do the tests, then shut down the virtual machine, restore the snapshot, and boot up again.

Tags: ,

Graphing memory usage during an MTR run

June 30th, 2011 No comments

In order to optimally size the amount of RAM to allocate to a set of new machines for running MTR, I ran a few tests to check the memory usage of an MTR run for mysql-trunk and cluster-7.1. As using a RAM disk considerably speeds things up, I set the vardir to be on /ramdisk and logged the usage of that too.

The tests were performed on an 8-core E5450 @ 3.00GHz with 24GB RAM, with 8GB allocated to /ramdisk. Each branch ran the default.daily collection, which generally contains the most testing we do per-run. Between each run I rebooted the machine to clear the buffer cache and /ramdisk

I used something like the script below, which saved the per-second usage of /ramdisk, the total RAM used, and the RAM used minus buffers.

#!/bin/bash

BRANCH="mysql-trunk"
BUILDDIR="mysql-5.6.3-m5-linux2.6-x86_64"
TESTDIR="${HOME}/mtr-test/${BRANCH}

stats()
{
  i=1
  rm -f ${TESTDIR}/stats-${BRANCH}
  while [ -f ${TESTDIR}/running ]; do
    rd=$(df -k /ramdisk | awk '/^\// {print $3}')
    mem=$(free | awk '/^Mem/ {print $3}')
    mem1=$(free | awk '/cache:/ {print $3}')
    echo "${i} ${rd} ${mem} ${mem1}" >>${TESTDIR}/stats-${BRANCH}
    i=$((i+1))
    sleep 1
  done
}

export TMPDIR="${TESTDIR}/tmp"
rm -rf ${TMPDIR}
mkdir -p ${TMPDIR}

>${TESTDIR}/running
stats &

(
  cd ${TESTDIR}/${BUILDDIR}/mysql-test

  perl mysql-test-run.pl ... --parallel=8 --vardir=/ramdisk/mtr-${BRANCH}/...
  mv /ramdisk/mtr-${BRANCH}/* ${TMPDIR}/
  ...
)

sync
rm -f ${TESTDIR}/running
wait

First I graphed a straight run of the two branches, using the following gnuplot script:

set terminal png enhanced font "Times,11" size 640,768
set output "mtr-ram.png"
set title "MTR memory usage (8-core Xeon, 24GB, 8GB RAM disk)"
set xlabel "Time (minutes)"
set ylabel "Memory usage (GB)"
set yrange [0:16]
set xtics 10
set key top box
set grid
plot "stats-mysql-trunk" every 60 using (($1)/60):(($2)/1024/1024) \
        title 'mysql-trunk /ramdisk usage' with lines, \
     "stats-mysql-trunk" every 60 using (($1)/60):(($3)/1024/1024) \
        title 'mysql-trunk RAM (inc buf)' with lines, \
     "stats-mysql-trunk" every 60 using (($1)/60):(($4)/1024/1024) \
        title 'mysql-trunk RAM (exc buf)' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($2)/1024/1024) \
        title 'cluster-7.1 /ramdisk usage' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($3)/1024/1024) \
        title 'cluster-7.1 RAM (inc buf)' with lines, \
     "stats-mysql-cluster-7.1" every 60 using (($1)/60):(($4)/1024/1024) \
        title 'cluster-7.1 RAM (exc buf)' with lines

I then performed a valgrind run on mysql-trunk using similar scripts. As valgrind takes considerably longer (and uses more RAM) I kept it separate as the combined graph isn’t very clear:

So, based on these results, the host machine (16GB RAM + 8GB RAM disk) is probably a sensible guide for now, and allows for some future growth.

Tags: ,

Fix input box keybindings in Firefox

June 29th, 2011 No comments

Those of us used to command line editing will no doubt have been frustrated many times in Firefox when editing text in an input box and subconciously hitting CTRL-w to delete-word, only to have the tab close and your work deleted.

Thankfully there is a workaround to this. It used to be a case of adding the following to .gtkrc:

gtk-key-theme-name = "Emacs"

However these days it’s a gconf setting:

gconftool-2 --set /desktop/gnome/interface/gtk_key_theme Emacs --type string

This will bind CTRL-w to delete-word when in an input box, but retain the close tab binding elsewhere, a nice implementation of DWIM. See this page for more information.

Tags:

How to lose weight

June 24th, 2011 1 comment

This year I set myself a target to get back to my pre-marital weight. Back in 2001 I was 90kg, but over the past decade I’d steadily gone up and for quite a few years had hovered around 100kg.

I’m a bit of a perfectionist, and like shiny graphs, so from Jan 1st this year I weighed myself every day on the Wii Fit, and kept a record of my progress. This morning I was finally under 90kg. Here’s how things have gone thus far:

My weight loss between 1st January 2011 and 24th June 2011

And for the geeks, my gnuplot script, assuming an input file named weight.txt containing lines of the format “%Y-%m-%d <weight>”:

set terminal png size 640, 400
set output "weight-jun11.png"
set xdata time
set timefmt "%Y-%m-%d"
set format x "%b"
set xlabel 'Date'
set ytics 2
set ylabel 'Weight (kg)'
set multiplot
plot 'weight.txt' using 1:2 title 'Weight' with lines linecolor 2

My three step plan for losing weight

The interwebs are awash with a million different ways to lose weight, and there are thousands of companies who will gladly accept large chunks of your cash in order to provide you with their expert opinion on how to do it. This is my three-step plan, and you can have it for free:

  1. Eat less
  2. ???
  3. Profit!!!

Really, that’s it. It’s not rocket science, after all. If you eat less stuff (especially saturated fat and sugar), there is less excess for your body to store as fat, and thus over time you weigh less as your body starts to use up the excess you have stored.

Here’s some things I did to achieve this:

Eat less

The basic aim. For me this was mainly about reducing my portion size.

Previously, I’d eat a massive bowl of cereal in the morning. which I thought was a good thing as everyone always says to ensure you have a good breakfast. However I was eating more than my body needed and as a consequence it was likely storing up the excess.

Also at meal times I’d usually serve myself a full plate, and always feel that I needed to finish it. I liked feeling full. However, again, this likely just meant I was eating more than I needed, and I found that just by serving less food I was still satisfying my hunger but with less excess.

After a relatively short period of time I found that by avoiding these large meals, I needed less to feel full – as if my stomach had shrunk and gotten used to the reduced size. This was a great positive feedback loop, as it massively helped avoid the temptation to snack between meals.

Cut down on sugar/fat

I realise this can be hard for some people, I found it relatively easy but as you can see on the graph above there are some upward trends, which were mainly when I went to visit my parents, who have a cupboard full of chocolates, crisps, cakes, sweets, etc, and I find it really hard not to have at least one or two fake Lidl snickers bars per day!

However, also note that after an upward trend, I lost the weight again pretty quickly, so don’t worry too much about a few days of eating junk, I actually noticed how bad I felt afterwards after getting used to a reduced sugar/fat intake, and that provided good incentive to cut down again.

Some practical things I did:

  • Stopped putting sugar in tea/coffee. I’ve since regressed, but it was helpful to do this for a while, and definitely helped wean my body off desiring sugar.
  • Avoided snacking on crisps, chocolate, biscuits, etc. When you work from home this can be difficult, but I found it helped to ensure they weren’t in the house to begin with, and that we were stocked up on bananas, apples, and other less sugary/fatty snacks.
  • Used a fine cheese grater when making sandwiches, beans on toast, etc. Previously I’d put a good few slabs of cheese in, when I didn’t really need that quantity. The fine grater ensured I still got the taste, but with less quantity.
  • Bought reduced fat mayonnaise, margarine, etc. These actually taste pretty good these days, and in a sandwich with lots of fresh cucumber, tomato, salad etc you don’t notice the difference

All these things helped to lose weight and, in a similar manner to the ‘eating less’ part, trained my body to not require them as much as it used to – and actually, to noticeably feel worse if ever I regressed, which made getting back on track very easy.

Weigh myself daily

It seems this is generally not recommended, and advice is that you should weigh yourself weekly (at the same time each week). Logically this doesn’t make sense to me, as a geek it’s obvious that the more data points you have, the better. Your body weight can fluctuate quite a lot from day to day especially with regards to how much liquid you have drunk, and if you are only weighing yourself once per week you could get caught out by a daily spike.

Aside from the geek factor of a better graph resolution, I also prefer daily weighing as it provides me, as a perfectionist, with incentives both ways. If I weigh less compared to the previous day, I feel good that I am achieving my aim, and am encouraged to continue. If I weigh more, it’s a warning that I may have eaten too much, and I am then motivated to be more careful that day.

Weighing daily also helps to just keep your mind reminded of the task, plus if you’re on the Wii Fit already you might be tempted to do some exercise :)

Next steps

I’m not content to stay here, the next plan is to get closer to my ideal BMI weight which is around 80kg. This is likely to be much harder, as there is less excess fat for me to get rid of now. However, one thing I haven’t done so far is increase the amount of exercise I do, and there is definitely room for improvement there!

Hopefully I can provide another update later this year and perhaps be around 85kg.

Tags: