date = 2022-02-04 tags = ["kubernetes", "k3s", "harbor", "nginx", "raspberrypi"] title = "Setting up a K3S cluster"
I wanted to set up a Kubernetes cluster on my
that consists of three RPis and a PC, and run some actual workload. I used
, a lightweight Kubernetes distribution, because it seemed to be the one to go to for running Kubernetes on RPis, ex.
Jeff Geerling's video on installing K3S
. For image registry, I tried
Harbor open source registry v2.4.1
. The cluster ingress is fronted by a reverse proxy.
is the simplest way to setup a cluster. The amount of simplication it does is phenomenal. It packages up all the functionalities of etcd3, control plane, networking, DNS, and ingress (Traefik) in one binary and I did not have to interact with any of the components to have a working cluster.
At first, I tried to run a K3S server node on RPi 3 but it did not work well, which I assume is because of the memory constraint. Running K3s agent nodes on RPi 3 instead was not an issue.
On K3S server node
sudo k3s server
On K3S agent nodes
sudo k3s agent --server https://${SERVER_HOST}:6443 --token "${NODE_TOKEN}"
Harbor is an open source registry that has many additional features like access control, vuln scanning, and others that are overkill for my use.
Since I am running the Harbor instance in the same node as the K3S server, I configure it to listen to port 8443 instead of 443.
I used
Certificate Manager to provision an internal cert for image registry.
step ca certificate server --san gauss.lab.jaeyoung.se:8443 server.crt server.key --not-after 31d
Install the root CA in each node and configure K3S nodes to use the root CA to pull images.
sudo cp root_ca.crt /usr/local/share/ca-certificates/root_ca.crt
sudo update-ca-certificates
mkdir -p /etc/rancher/k3s
cat <<EOF > /etc/rancher/k3s/registries.yaml
configs:
"gauss.lab.jaeyoung.se:8443":
tls:
ca_file: /usr/local/share/ca-certificates/root_ca.crt
EOF
Ref.
https://rancher.com/docs/k3s/latest/en/installation/private-registry/
Make changes to harbor.yml to configure the HTTP port and the storage directory, then Harbor can be installed. It installs as several Docker containers.
One thing I find really neat is that, since Harbor v2 supports manifest lists, if you build images for both linux/amd64 and linux/arm64, the images will work on both platforms transparently. I use
to build and push multi-platform images in one command:
KO_DOCKER_REPO=gauss.lab.jaeyoung.se:8443/library ko publish --platform=linux/amd64,linux/arm64 -P .
Yay.