Using a custom Linux OS on your PiBox#
We maintain and distribute a modified 64-bit Raspberry Pi OS image which is pre-configured to be perfect for the PiBox. The rest of this page contains the instructions for making those modifications yourself if you decide to reflash your PiBox. In general, the modifications we make are:
- Fan support (PWM) for quiet fan operation
- SATA Kernel modules
- Install Kubernetes (via k3s)
- Tweaks for stability and performance
- Display kernel modules for the 1.3” front panel display
If you’re using our OS Image, this guide has already been completed for you!
Dedicating an SSD to Kubernetes#
We recommend using one or more SSDs for your Kubernetes installation and storage. You’ll want to format a disk and mount it at
/var/snap before installing Kubernetes. This storage can then be used by Kubernetes for it’s storage system. This piece is optional, but we strongly recommend it!
If you’ve already installed k3s, please first uninstall with
Take a look to see if your disks have been detected:
lsblk | fgrep disk - if not, try updating the OS with
apt update && apt upgrade and restart. If you still don’t see your disks, see the “Enabling the SATA Kernel Module” section below.
# Format the entire /dev/sda disk (WARNING! This will wipe data!) sudo parted -a opt /dev/sda mkpart primary ext4 0% 100% # Format the partition sudo mkfs.ext4 -L kubernetesdata /dev/sda1 # Mount the partition in place! K3s usually lives at "/var/lib/rancher", whereas microk8s uses "/var/snap" sudo mkdir -p /var/lib/rancher sudo bash -c "echo '/dev/sda1 /var/lib/rancher ext4 defaults,nofail,noatime,discard,errors=remount-ro 0 0' >> /etc/fstab"
K3S is our preferred Kubernetes distribution. It’s maintained by Rancher, is updated regularly, and performs well on low-power devices, such as Raspberry Pis. Other distributions of Kubernetes should work fine on the PiBox and with KubeSail.com, but this guide focuses on k3s. We recommend k3s over Microk8s because of its dramatically reduced resource requirements as of the time of writing (11/1/2021)
# On Raspberry PI OS, the path is /boot/cmdline.txt, on Ubuntu it's /boot/firmware/cmdline.txt # TLDR: Add 'cgroup_enable=memory cgroup_memory=1' to that file and reboot sudo bash -c "grep -qxF 'cgroup_enable=memory cgroup_memory=1' /boot/cmdline.txt || sed -i 's/$/ cgroup_enable=memory cgroup_memory=1/' /boot/cmdline.txt" sudo reboot # Install k3s - see https://k3s.io/ for more curl -sfL https://get.k3s.io | sh -
KubeSail helps you manage software on your PiBox, or any other computer running Kubernetes. If you don’t yet have a KubeSail account, creating one is as easy as signing up with GitHub. Once you’ve done that, simply replace your username in the following command and run it:
sudo bash -c "echo YOUR_KUBESAIL_USERNAME > /boot/kubesail-username.txt"
Then install the KubeSail agent:
curl -s https://raw.githubusercontent.com/kubesail/pibox-os/main/agent-installer.sh | sudo bash
After a few minutes, your cluster will appear in the clusters tab of the KubeSail dashboard.
Enabling PWM Fan Support#
To make the fan quiet and only spin as fast as necessary, we install a service that sends the correct signal to the fan using the Pi’s hardware PWM controller. This code can be found in our fork of
alwynallan‘s original gist on GitHub.
git clone https://github.com/kubesail/pibox-os.git cd pibox-os/pwm-fan tar zxvf bcm2835-1.68.tar.gz cd bcm2835-1.68 ./configure && make && sudo make install cd .. make && sudo make install
Enable USB (Raspberry Pi OS only)#
If you are using Raspberry Pi OS, you will need to enable the USB 2.0 ports on the Compute Module 4. Edit the boot config file at
/boot/config.txt and add:
Enabling the SATA Kernel Module#
Raspberry Pi OS will soon ship the SATA modules by default. Use
apt update && apt upgrade to upgrade!
You may want to see if your disks are already detected (
lsblk | fgrep disk) before building these modules, which can take some time. Instructions for Compiling the Ubuntu-raspi kernel can be found at https://askubuntu.com/a/1242267 and cross-compilation instructions can be found at https://github.com/carlonluca/docker-rpi-ubuntu-kernel
Enabling the SATA ports requires compiling the SATA modules into the kernel.
# Install dependencies sudo apt install -y git bc bison flex libssl-dev make libncurses5-dev # Clone source git clone --depth=1 https://github.com/raspberrypi/linux # Apply default configuration cd linux export KERNEL=kernel8 # use kernel8 for 64-bit, or kernel7l for 32-bit make bcm2711_defconfig # Customize the .config further with menuconfig make menuconfig # Enable the following: # Device Drivers: # -> Serial ATA and Parallel ATA drivers (libata) # -> AHCI SATA support nano .config # (edit CONFIG_LOCALVERSION and add a suffix that helps you identify your build) # Build the kernel and copy everything into place make -j4 Image modules dtbs # 'zImage' on 32-bit sudo make modules_install sudo cp arch/arm64/boot/dts/*.dtb /boot/ sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/ sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/ sudo cp arch/arm64/boot/Image /boot/$KERNEL.img # Reboot the system reboot
Further detailed instructions and discussion can be found on Jeff Geerling’s PCI device guide on GitHub
Enabling the 1.3” LCD display#
Raspberry Pi OS only. Ubuntu ships with the SPI interface enabled by default
To enable the front-panel display, the SPI interface needs to be turned on. Edit the boot config file at
/boot/config.txt and add:
After these changes, you will need to
You can then use a Python library or install the kernel module in order to draw images to the display. See below for details.
Modifying the image / stats shown on the display#
Adafruit has a guide which is compatible with the display used in PiBox. We modified Adafruit’s code in combination with prometheus-png to render stats to the display. This code runs as containers within Kubernetes, installed via this template, and the source code lives in two repositories:
- Our fork of prometheus-png built for
- Our python script to render the resulting PNGs to the display: https://github.com/kubesail/pibox-os/tree/main/lcd-display
In our testing, if you are trying the Python “easy way”, and installing the
adafruit-circuitpython-rgb-display library, you will need to install
RPi.GPIO manually via pip3. Once installed, you can try installing the Adafruit library again.
CFLAGS=-fcommon pip3 install RPi.GPIO