Gav's World : Raspberry Pi : Cluster Pi Kubernetes

Setting Up Cluster Pi with Kubernetes

There are a great many things that can be done with a cluster, even a low power one like a Raspberry Pi cluster where the worker nodes are only single core Pi Zeros. I have decided to play around with Kubernetes, with Ansible to administer the whole cluster, just for the sheer fun of it!

Hat's off to Jeff Geerling as I've used his tutorials (links at the bottom) to assist in setting this all up.

However... this guide will end up with a small cluster that is using most of its limited resources to run kubernetes and nothing left over to deploy anything useful - you have been warned :-)

Hardware

I've chosen to use a Pi 3 with 4x Zero Pis (no point in using ZeroW as they are connected in USB Gadget mode anyway, so have networking that way). If you use a Pi 4 as the controller, you can't reboot the controller without power cycling the worker nodes. Of course, you'll also need a Cluster HAT! I'd also recommend the Cluster HAT Case to house the cluster.

Operating Systems

Download and install all necessary images from ClusterCTRL... remember to add an empty "ssh" file in each boot partition to automatically enable SSH on all nodes. I picked the BRIDGE controller - primarily so that I can easily access each worker from my Desktop PC.

Connect everything up and power up the cluster - only the controller Pi will power up, as the ClusterHAT needs to be instructed to apply power to all of the workers.

That's the easy bit! Now for just a few more steps...

Instructions

ssh into the controller Pi and power up the workers:
clusterctrl on
ssh into all nodes to apply all updates, change the default password and node names (if desired, using raspi-config). My nodes called:
cluster-pi-c (controller)
cluster-pi-w1 (worker 1)
cluster-pi-w2 (worker 2)
cluster-pi-w3 (worker 3)
cluster-pi-w4 (worker 4)
On the controller Pi create an RSA key pair:
ssh-keygen -b 2048
Then copy the public key to each of the workers, for example:
ssh-copy-id pi@cluster-pi-w1
(I also do that from my desktop PC to make remote login quicker)
On whichever computer you wish to administer the entire cluster from, install ansible:
sudo apt install ansible
And install kubectl (follow these instructions).
Then download k3s-ansible from rancher's Git repo
git clone https://github.com/rancher/k3s-ansible.git
Enter the k3s-ansible directory
cd k3s-ansible
Copy the sample host inventory file and edit to list the worker hostnames or IP addresses.
cp inventory/sample/hosts.ini inventory/.
nano inventory/hosts.ini
Copy the sample inventory/group_vars/all.yml file and edit to change the ansible_user to pi (for use with a Raspbian based distro)
cp -R inventory/sample/group_vars inventory/.
nano inventory/group_vars/all.yml
Run the installed playbook which will download and install k3s on all nodes and then configure them appropriately:
ansible-playbook site.yml -i inventory/hosts.ini
(I had to run this twice, with a reset.yml playbook in between, I think that the configuration of the ClusterCTRL controller, meant that the workers failed to download the k3s image first time around. The script isn't quite tailored for a ClusterPi bridge network.)
Copy the kubectl configuration from the controller pi to the administration PC:
scp pi@cluster-pi:~/.kube/config ~/.kube/config-cluster-pi

Check that the kubernetes nodes are up and running:

export KUBECONFIG=~/.kube/config-cluster-pi
kubectl get nodes

All well and good, the result should look something like this:

NAME	STATUS	ROLES	AGE	VERSION
cluster-pi	Ready	master	23m	v1.17.5+k3s1
cluster-p1	Ready	<none>	22m	v1.17.5+k3s1
cluster-p2	Ready	<none>	22m	v1.17.5+k3s1
cluster-p3	Ready	<none>	22m	v1.17.5+k3s1
cluster-p4	Ready	<none>	22m	v1.17.5+k3s1

Unfortunately, here's where it breaks down a bit... the Pi zeros are so stretched just running the kubernetes framework that they've got no resources left to run any deployment. You may be able to run a "hello world" type thing though :-)

An interesting experience, but I'll save it for a slightly more capable cluster :-)