Creating an emulab image from scratch

This is based on https://wiki.emulab.net/wiki/Emulab/wiki/ClientSideStuff and https://wiki.emulab.net/Emulab/wiki/ClientSideInstall but now done from start to end for Debian 7.7.

Step 1: Do a fresh install of your new OS

  • take a physical node with a default image, e.g. UBUNTU12-64-STD (which has an emulab MBR v2 of 6GB) or UBUNTU14-64-STD (which has an emulab MBRv3 of 15GB)
  • you need for this access to the console and be able to put installation media in the machine (e.g. done through IPMI at iMinds)
  • load Debian 7.7 netinst iso through IPMI, make this one boot instead of PXE boot, and install this with fairly normal settings
  • install base: expert install - ssh server only (in tasksel) - root login enabled (root password does not matter, is overwritten by emulab)
  • do not change partitions: MBRv2:/dev/sda2 is root and /dev/sda3 is swap, MBRv3: /dev/sda1 is root and /dev/sda3 is swap
  • put interface eth0 on dhcp
  • KEEP EXT3, as FREEDBSD (or the version I used) cannot read EXT4 it seems (for imaging later on)
  • put grub on /dev/sda2 for MBRv2 and on /dev/sda1 for MBRv3 ! (not on /dev/sda !)
  • after install, let it reboot on harddisk (network PXE boot, then /dev/sda2 or /dev/sda1 partititon should be started from the emulab starter)
  • set BIOS settings/boot settings back to normal
  • check that you can login as root (with the password you chose)
  • image boots, but emulab does not recognize that it is up (and doesn’t want to take an image)
  • size is approx. 700MB at this point

Step 2: Add packages

Network card bugfix (seen at iMinds with Nvidia MCP55 forcedeth), see here. Create a file /etc/modprobe.d/forcedeth.conf with the following contents:

options forcedeth msi=0 msix=0 optimization_mode=1 poll_interval=38 max_interrupt_work=40 dma_64bit=0

Execute also:

mv /sbin/mii-tool /sbin/mii-tool_orig

More or less copied from existing UBUNTU12-64-STD:

apt-get install g++ apt-transport-https at autoconf automake autotools-dev bash-completion bc bind9-host byacc bzip2 ca-certificates command-not-found cvs debhelper dnsutils dosfstools dpkg-dev ed emacs emacs23 emacs23-bin-common emacs23-common emacsen-common exuberant-ctags iperf traceroute file flex ftp fuse fuse-utils gdb geoip-database gettext git git-core git-man gnupg-curl gpgv hdparm html2text intltool-debian  iputils-arping iputils-tracepath irqbalance iso-codes jove kernel-package kernel-wedge kexec-tools dstat less lftp libboost-dev  libboost-iostreams-dev libtool lintian linux-headers-3.2.0-4-all-amd64 lockfile-progs lshw lsof ltrace lvm2 lzma m4 make makedev makedumpfile memtest86+ mime-support mlocate mtr-tiny netcat-openbsd nfs-common ntfs-3g ntp ntpdate open-iscsi  openssl openvpn parted patch perl perl-modules po-debconf powermgmt-base ppp pppconfig pppoeconf psmisc  smartmontools ssl-cert strace subversion sudo tcl8.5 tcpdump tshark tcsh time telnet ufw zip unzip uuid-runtime valgrind vim vlan wput w3m xterm zlib1g-dev zsh rpm  libssl-dev libpcap0.8-dev screen xfsprogs

apt-get clean

Size is now approx. 1.9GB.

Run update-command-not-found to fill indexes for command-not-found.

Step 3: Install emulab client side

As root (of course drop http_proxy if you have direct IPv4 internet access):

cd /work
http_proxy=proxy.atlantis.ugent.be:8080 wget http://www.emulab.net/downloads/pubsub-0.99.tar.gz
# or get a release from https://gitlab.flux.utah.edu/emulab/pubsub

tar xvzf pubsub-0.99.tar.gz
cd pubsub-0.99
./configure        (on Debian 8, you need to apt-get install libtool-bin for this)
make client
make install-client

mkdir /work/obj
mkdir /work/src
cd /work/src

On boss, in latest src dir (adapt paths as needed, this part is not available to a regular experimenter, so ask admins, n095-01a is the nodename):

cd /proj/testbed/upgrade/src
scp -rp clientside/ root@n095-01a:/work/src/
scp -rp tools/ root@n095-01a:/work/src/
scp -rp .git/ root@n095-01a:/work/src/
(after the following commands, some git version is written in /etc/emulab/version)

copy defs from boss to your node: scp /root/defs-wall2 root@n095-01a:/work/src/

Need patch to /work/src/clientside/tmcc/osstuff.sh as debian is not featured yet (this is line 96 in recent updates):

--- osstuff.shORIG      2014-07-12 16:36:54.034310350 +0000
+++ osstuff.sh  2014-07-12 20:55:00.817858560 +0000
@@ -51,6 +51,10 @@
         dist=`grep DISTRIB_ID /etc/lsb-release | awk -F = '{ print $2; }'`
         rel=`grep DISTRIB_RELEASE /etc/lsb-release | awk -F = '{ print $2; }'`
     fi
+    if [ -r /etc/os-release ]; then
+        dist="Debian"
+        rel=`cat /etc/debian_version`
+    fi
     if [ -z "$dist" -a -r /etc/redhat-release ]; then
         trel=`grep 'Red Hat' /etc/redhat-release | sed -e 's/Red Hat Linux release \([0-9]\(\.[0-9]\)\?\).*/\1/'`
        if [ -n "$trel" ]; then

Create also a debian7 dir with files:

mkdir /work/src/clientside/tmcc/debian7
cd /work/src/clientside/tmcc/debian7
cp -a /work/src/clientside/tmcc/ubuntu11/* .      (or ubuntu14 dir in recent emulab version)
cp /etc/group .
cp /etc/passwd .
cp /etc/gshadow .
cp /etc/shadow .

Patch to add the debian7 directory:

diff /work/src/clientside/tmcc/GNUmakefile.in /work/src/clientside/tmcc/GNUmakefile.inORIG
102,104d101
< ifeq ($(MDSUBDIR),debian7)
< MDSUBDIR  = debian7
< endif

In /work/src/clientside/configure, add:

tmcc/debian7/GNUmakefile \

On the node again:

cd /work/obj
/work/src/clientside/configure --with-TBDEFS=/work/src/defs-wall2

This is needed for the git version command:

cp -a /work/src/tools/git /work/obj/tools

IPv4 internet access needed for next steps (as it fetches some tar balls from Utah):

cd /work/obj
make
make client-install
  • Check for errors in output of above commands !
  • Check /usr/local/etc/emulab to see if it seems comparable to an existing image. (and /usr/local/etc/testbed is a symlink)
  • /var/emulab should contain files
  • /etc/emulab should also contain files

Step 4: The dirty work

Now comes the dirty work (which also takes most time): finetuning the OS image… I guess this could be all put in the client side make scripts, but as it takes some time, I just document here the manual adjustments:

mkdir /users

In /etc/sudoers, add (so that an experimenter can become root on the image):

# Members of the admin group may gain root privileges
%admin ALL=(ALL) NOPASSWD: ALL
%root ALL=(ALL) NOPASSWD: ALL

in /etc/ntp.conf: remove default servers …

/etc/init.d/testbed: put LSB header:

#
# testbed        Start and stop testbed
#
### BEGIN INIT INFO
# Provides: testbed
# Required-Start:       $all
# Required-Stop:
# Default-Start:        2 3 4 5
# Default-Stop:
# Should-Start:
# Short-Description: Start testbed scripts
# Description: Start testbed scripts for emulab
### END INIT INFO

/etc/init.d/pubsubd: put LSB header (check this as the default one has no start on runlevel 2 !):

#
# pubsubd        Start and stop pubsubd
#
### BEGIN INIT INFO
# Provides: pubsubd
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop: 0 1 6
# Should-Start:
# Short-Description: Start and stop pubsubd
# Description: Start and stop pubsubd, the Emulab publish/subscribe daemon
### END INIT INFO

/etc/init.d/rc.local: add LSB header:

### BEGIN INIT INFO
# Provides:          rc.local
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Run /etc/rc.local if it exist
### END INIT INFO

Debian has now changed from update-rc.d to a new Dependency based boot system , so it is better to remove all testbed and pubsubd and rc.local and tbprepare symlinks in /etc/rc.x/ dirs and use:

insserv pubsubd
insserv testbed
insserv rc.local
insserv tbprepare

In /etc/default/kexec (to make reboot a real reboot):

LOAD_KEXEC=false

Kernel options, in /boot/grub/grub.cfg, add serial console settings:

linux   /boot/vmlinuz-3.2.0-4-amd64 root=UUID=8d681cfb-8fb1-4e45-9ca2-7f5e21858b4a ro console=ttyS0,115200  crashkernel=384M-2G:64M,2G-:128M
linux   /boot/vmlinuz-3.2.0-4-amd64 root=UUID=8d681cfb-8fb1-4e45-9ca2-7f5e21858b4a ro single console=ttyS0,115200
(currently: you have to change /etc/default/grub, GRUB_CMDLINE_LINUX="crashkernel=384M-2G:64M,2G-:128M" as grub.cfg is auto-generated)
update-grub

/etc/fstab: remove cdrom line (/dev/sr0)

Patch for /usr/local/etc/emulab/fixup-fstab-swaps (debian now used /dev/disk paths instead of /dev/sda):

--- fixup-fstab-swapsORIG       2014-07-13 09:13:35.000000000 +0000
+++ fixup-fstab-swaps   2014-07-13 09:27:58.000000000 +0000
@@ -125,7 +125,10 @@
     #
     my @swapdevs;
     my $rdisk;
-    my $rootfs = `df / | grep /dev/`;
+    my $rootfs = `df / | grep /dev/ | cut -d" " -f1`;
+    # convert /dev/disk/by-uuid/04936e2c-914e-4b9c-ab3f-278a9c407cdc to /dev/sda2 or similar
+    print STDERR "rootfs: $rootfs";
+    $rootfs = `/bin/readlink -f $rootfs`;
     if ($rootfs =~ /^(\/dev\/[a-z]+)\d+/) {
        $rdisk = $1;
     } else {

Patch /etc/init.d/ntp (this took a long time to find :-) ):

--- ntpORIG     2009-12-26 17:29:45.000000000 +0000
+++ ntp 2014-07-13 12:42:38.000000000 +0000
@@ -26,6 +26,13 @@
        NTPD_OPTS="$NTPD_OPTS -c /var/lib/ntp/ntp.conf.dhcp"
 fi

+. /etc/emulab/paths.sh
+
+NTPD_PREOPTS=
+if [ -x $BINDIR/ntpstart ]; then
+    DAEMON=$BINDIR/ntpstart
+    NTPD_PREOPTS=/usr/sbin/ntpd
+fi

 LOCKFILE=/var/lock/ntpdate

@@ -58,7 +65,7 @@
                        exit 1
                fi
                lock_ntpdate
-               start-stop-daemon --start --quiet --oknodo --pidfile $PIDFILE --startas $DAEMON -- -p $PIDFILE $NTPD_OPTS
+               start-stop-daemon --start --quiet --oknodo --pidfile $PIDFILE --startas $DAEMON -- $NTPD_PREOPTS -p $PIDFILE $NTPD_OPTS
                status=$?
                unlock_ntpdate
                log_end_msg $status

Try to reboot

After reboot: check /var/emulab/logs for errors which are not seen on e.g. ubuntu12-64-STD

Step 5: Install geni-get

See http://groups.geni.net/geni/wiki/GeniGet (you can skip this if you are not using protogeni):

wget --no-check-certificate https://www.emulab.net/downloads/geni-get_1.0_all.deb
dpkg -i geni-get_1.0_all.deb

Step 6: Clean up and prepare for image

  • Change paths in/etc/apt/sources.list to http://http.debian.net

  • Clean up bash_history of root user

  • Clean up /etc/apt/apt.conf (remove proxyserver e.g.)

  • umount /work

  • rmdir /work

  • Disconnect iso image from IPMI if it is still there

  • create a file /etc/emulab_image_version (to trace easily which version is really running, increase when you create a new one):

    version:2

  • Take image and test, repeat if needed :-)

Step 7: Adapt metadata

This should be done by an admin.

In emulab interface (os_info is now called os_info_versions):

toggle that the image can run on xen.
mysql tbdb
update os_info_versions set description='Debian 7.6 wheezy',version='7.6' where osname='DEB76-64-STD';
(update images set description='Debian 7.6 wheezy' where imagename='DEB76-64-STD';)

If you move it also to emulab-ops:

(update images set pid='emulab-ops',gid='emulab-ops',pid_idx=1,gid_idx=1,path='/usr/testbed/images/DEB76-64-STD.ndz' where imagename='DEB76-64-STD';)
update images set pid='emulab-ops',gid='emulab-ops',pid_idx=1,gid_idx=1 where imagename='DEB77-64-STD';
(update os_info set pid='emulab-ops',pid_idx=1 where osname='DEB76-64-STD'; )
update os_info_versions set pid='emulab-ops',pid_idx=1 where osname='DEB77-64-STD';
update os_info set pid='emulab-ops',pid_idx=1 where osname='DEB77-64-STD';
update image_versions set pid_idx=1,gid_idx=1,pid='emulab-ops',gid='emulab-ops',path='/usr/testbed/images/DEB77-64-STD.ndz',description='Debian 7.7 Wheezy (64 bit)' where imagename='DEB77-64-STD';

mv /groups/wall2-ilabt-iminds-be/bvermeul/images/DEB77-64-STD.* /usr/testbed/images/
chown root:wheel /usr/testbed/images/DEB77-64-STD.*

If you want to get it listed in advertisement RSpec (old table name is os_info):

update os_info_versions set protogeni_export=1 where osname='DEB76-64-STD';

Step 8: Load it on another emulab

See `Install a specific disk image on a node <http://doc.ilabt.iminds.be/ilabt-documentation/urnsrspecs.html#install-a-specific-disk-image-on-a-node> `_ how to load an image through an URL from another emulab.

You can also import it in a local emulab, as described here: https://wiki.emulab.net/Emulab/wiki/ImageImport:

Copy the ndz file to /user/testbed/images
get the XML descriptor:  wget --no-check-certificate https://www.wall2.ilabt.iminds.be/image_metadata.php?uuid=dd795c55-0a8b-11e4-b407-001517becdc1
rename: mv image_metadata.php\?uuid\=dd795c55-0a8b-11e4-b407-001517becdc1 DEB76-64-STD.xml
add a field pid: <attribute name="pid"><value>emulab-ops</value></attribute>
create descriptor: wap load-descriptors DEB76-64-STD.xml

or faster on newer versions:  wap image_import -g -p emulab-ops 'https://www.emulab.net/image_metadata.php?uuid=xxxxxxxx'
Debian 7.7 MBR3: https://www.wall2.ilabt.iminds.be/image_metadata.php?uuid=47472517-5d3c-11e4-bd9f-001517becdc1
(and through webinterface click on which hardware types the image can run)
Advanced Subjects
Usage Policy
Ghent University imec
© Copyright imec and Ghent University