The Raspberry PI in the stacks are workers. The master controls them, assigning them jobs and an image to execute. Due to limitations in both its hardware and the way we want to use them, we must install a minimal software infrastructure on each worker. This section descries such an infrastructure and how to use it.

General overview of the boot process

A main problem with a cluster is how to update the software running in the workers. Usually, this task is performed by netbooting; in this way, every time the workers boot they load an up to date kernel and set of applications. This solution has several problems in our use case:

  1. The Raspberry Pi doesn’t support netboot
  2. The Raspberry Pi has some memory constraint; hence, loading a filesystem with many applications in memory can limit its usability.
  3. Since we want to perform experiments where devices are turned on/off randomly, booting from the network can greatly impact the starting time of a particular device.

As a result of these shortcoming, we devised the following solution:

  1. Every PI has two fully functional system in its local SD Card.
  2. The first system is Tiny Core: a really small Linux distribution that used a ramdisk as file system.
  3. The second system is a normal Raspian distribution (or whatever you decide to install) that is used in the common operation mode.
  4. When a worker boot, it uses uboot, a bootloader, to access the TFTP server in the master. It downloads a couple of files (image_crc32.txt and quick_fix_crc32.sh) and compare them against local copies. If there exist some differences, it means that the system must be updated, uboot boots the PI using Tiny Core. Otherwise, the system is up to date and the normal Raspian distribution is used.

To configure a worker, you must copy an initial image to the SD Card. This image contains three partitions:

  1. The boot partition
  2. The Tiny Core partition
  3. The Raspian partition

An image is provided in the link. Issue the following command:

dd bs=4MB if=image.img of=/path/to/sdcard

Notice that this image only contains: the master boot record, the content of the boot partition, and the content of the Tiny Core partition. In other words, the normal Raspbian partition is not copied because it will be updated the first time you initialize the system.

Understanding the boot partition

It must contain the uboot bootloader and a set of environment variables for uboot. These environment variables are nothing but an implementation of the booting process described in the previous section.

Add a simple line to the file config.txt in order to boot using uboot instead of the normal Raspbian Kernel

echo "kernel=u-boot.bin" >> config.txt

In addition, the boot partition contains two Linux kernels, the one used by Tiny Core and the Raspian kernel. Both kernels can be updated, but we recommend updating only the Raspbian kernel.

Understanding the update process

The following script must be executed at boot time; it simply executes another script as a daemon. To execute it at boot time, you can modify Tiny Core on your own, or you can use this file.

#!/bin/sh

DIR=$(dirname $0)

nohup ${DIR}/update-script-daemon.sh 0<&- &>/dev/null &

The real update process is performed in the following script. Two kind of updates exists, a quick fix and a complete update.

#!/bin/sh

mount /dev/mmcblk0p1 /mnt/mmcblk0p1

IMAGE_CRC32=image_crc32.txt
IMAGE=copy.img.gz
SERVER=192.168.0.13
QUICK_FIX=quick_fix.sh
QUICK_FIX_CRC32=quick_fix_crc32.sh
fifo_image=fifo_image

sleep 5
tftp -l /root/${IMAGE_CRC32} -r ${IMAGE_CRC32} -g ${SERVER}
while [ "$?" -ne "0" ]; do
	sleep 1
	tftp -l /root/${IMAGE_CRC32} -r ${IMAGE_CRC32} -g ${SERVER}
done

diff /mnt/mmcblk0p1/${IMAGE_CRC32} /root/${IMAGE_CRC32}
status=$?

if [ "$status" -ne "0" ]; then

    umount /mnt/mmcblk0p3
    mkfifo ${fifo_image}
    tftp -l ${fifo_image} -r ${IMAGE} -g ${SERVER} &
    gunzip -c ${fifo_image} > /dev/mmcblk0p3
    cp /root/${IMAGE_CRC32} /mnt/mmcblk0p1/

fi

tftp -l /root/${QUICK_FIX_CRC32} -r ${QUICK_FIX_CRC32} -g ${SERVER}
diff /root/${QUICK_FIX_CRC32} /mnt/mmcblk0p1/${QUICK_FIX_CRC32}

if [ "$?" -ne "0" ]; then

    tftp -l /root/${QUICK_FIX} -r ${QUICK_FIX} -g ${SERVER}
    chmod +x /root/${QUICK_FIX}
    /root/${QUICK_FIX} && cp /root/${QUICK_FIX_CRC32} /mnt/mmcblk0p1/${QUICK_FIX_CRC32}

fi

reboot

A complete update simply downloads an compressed image from a TFTP server and copies it to a block device. Observe how the kernel image cannot be replace in this way because it resides in a different block device.

A quick fix is meant to replace the kernel, and the update process itself. However, any modification is possible because this script is executed as supervisor. Notice that the quick fi script is downloaded from the TFTP server and executed as root. It is obvious that any error or ill-intenioned behavior can introduce irreparable errors. Be careful: test your scripts before submitting them to the workers.

Finally, the update process may seems slow, maybe it is too much for the master. However, in our experiments we saw that the bottleneck is not the network bandwidth nor reading from the master SD Card. Instead, the slow part is writing to the workers’ SD Cards. To speed up the update process we compress the image, so only 13 minutes were needed to update an image of 1.3 GiB.