Create a Project - call it something meaningful, I called mine cucumber (got the irony?)
Add a ssh-key to the project and mark it as the default key
Infrastructure
Let’s quickly talk about the infrastructure I’m using for this guide.
You need:
A DNS record pointing to all of your master nodes (even if you only use one master node that makes sense)
Placement Groups: one for workers, one for masters (technically not required but recommended)
Firewall Rules: one for masters, one for workers using the following rules:
most rules are requirements of cilium, which you can checkout here
masters referes to the list of IPs of all master nodes
workers refers to the list of IPs of all worker nodes
masters:
Type
Source
Protocol
Port
Incoming
0.0.0.0/0, ::/0
TCP
22
Incoming
0.0.0.0/0, ::/0
ICMP
-
Incoming
0.0.0.0/0, ::/0
TCP
6443
Incoming
masters`z
TCP
2379-2380
Incoming
masters
TCP
10250
Incoming
masters
TCP
10259
Incoming
masters
TCP
10257
Incoming
masters
TCP
4240
Incoming
workers
TCP
4240
Incoming
workers
UDP
8472
Incoming
masters
UDP
8472
Incoming
workers
UDP
51871
Incoming
masters
UDP
51871
workers:
Type
Source
Protocol
Port
Incoming
0.0.0.0/0, ::/0
TCP
22
Incoming
0.0.0.0/0, ::/0
ICMP
-
Incoming
0.0.0.0/0, ::/0
TCP
30000 - 32768
Incoming
0.0.0.0/0, ::/0
UDP
30000 - 32768
Incoming
masters
TCP
10250
Incoming
workers
TCP
10250
Incoming
workers
TCP
4240
Incoming
masters
TCP
4240
Incoming
masters
UDP
8472
Incoming
workers
UDP
8472
Incoming
workers
UDP
51871
Incoming
masters
UDP
51871
Servers
Kubernetes requires you to have an odd number of master nodes, for true HA. Also many applications you may install require three replicas that are spread accross different nodes or zones, so at least three master and worker nodes are ideal. For lab purposes though, I usually only create one master and one worker (note that this is still an odd number ;)):
Location
Image
Type
Networks
Placement Group
Backups
Name
Labels
Falkenstein
Ubuntu 22.04
CPX11
ipv4
masters
false
hawk
cluster=cucumber,role=master
Helsinki
Ubuntu 22.04
CPX31
ipv4
workers
false
minion-01
cluster=cucumber,role=worker
Some notes before creating the servers:
Expect the size and number of workers to change over time.
You should have a DNS record for your kubeapi where you have added all master node IPs as valid answers (e.g multiple answers for the same domain name). This is the simplest way to avoid an external load balancer in front of your control plane (of course you could do that too and just forward port 6443 to all the master nodes)
Only use ipv4 or ipv6 but not both. Dual-stack is really hard to deploy and since many sites are not rechable over IPV6 (Github for example) I use an IPv4 only mode to avoid any network-related issue that take hours to investigate.
Cloud-init
The OS configuration shown in the next chapter can be masively simplified by using a custom cloud-init file for each server.
Note that this has to be specified at creation time.
#cloud-config <node>locale: en_US.UTF-8timezone: UTCusers:
- name: technatgroups: sudosudo: ALL=(ALL) NOPASSWD:ALL# Allow any operations using sudolock_passwd: True# disable password logingecos: "Admin user created by cloud-init"shell: /bin/bashssh_authorized_keys:
- "ssh-ed25519 ...."apt:
sources:
kubernetes:
source: "deb [signed-by=$KEY_FILE] https://apt.kubernetes.io/ kubernetes-xenial main"keyid: B53DC80D13EDEF05helm:
source: "deb [arch=amd64 signed-by=$KEY_FILE] https://baltocdn.com/helm/stable/debian/ all main"keyid: 294AC4827C1A168A# containerd from ubuntu repo is much older so use docker repodocker:
keyid: 8D81803C0EBFCD88source: "deb [arch=amd64 signed-by=$KEY_FILE] https://download.docker.com/linux/ubuntu jammy stable"package_update: truepackage_upgrade: truepackages:
- vim- git- wget- curl- dnsutils- containerd.io- apt-transport-https- ca-certificates- kubeadm- kubectl- kubelet- helmwrite_files:
- path: /etc/modules-load.d/containerd.confcontent: | overlay
br_netfilter- path: /etc/sysctl.d/99-kubernetes-cri.confcontent: | net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1- path: /etc/systemd/system/kubelet.service.d/20-hcloud.confcontent: | [Service]
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"- path: /etc/ssh/sshd_configcontent: | Port 22
PermitRootLogin no
PermitEmptyPasswords no
PasswordAuthentication no
PubkeyAuthentication yes
Include /etc/ssh/sshd_config.d/*.conf
ChallengeResponseAuthentication no
UsePAM yes
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
X11Forwarding no
PrintMotd no
Subsystem sftp /usr/lib/openssh/sftp-serverruncmd:
- sudo apt-mark hold kubelet kubeadm kubectl - sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="systemd.unified.cgroup_hierarchy=1"/g' /etc/default/grub - sudo update-grub - sudo mkdir -p /etc/containerd - sudo containerd config default | sudo tee -a /etc/containerd/config.toml - sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml - sudo sed -i 's/disabled_plugins = ["cri"]/disabled_plugins = []/g' /etc/containerd/config.toml - wget -O- https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz | tar Oxzf - | sudo dd of=/usr/local/bin/cilium && sudo chmod +x /usr/local/bin/cilium - helm repo add argo https://argoproj.github.io/argo-helm - helm repo add cilium https://helm.cilium.io/power_state:
mode: reboottimeout: 30condition: true
OS Preparations
Now that the servers are up and running, we will need to prepare all nodes according to the install kubeadm docs.
Swap
The first of them is Swap. Swap must be completly disabled on all nodes. Ubuntu 22.04 on Hetzner does this by default, if not, make sure it’s disabled using swapoff -a and removed from /etc/fstab.
Container Runtime
The container runtime must be installed prior to cluster bootstraping. There are various runtimes that fulfil the CRI. I’m using containerd as its simple and minimal.
The steps to install can be checked in the linked documentation or here.
Frist let’s load the br_netfilter module and set some forwarding settings:
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOFsudo modprobe overlay
sudo modprobe br_netfilter
# Setup required sysctl params, these persist across reboots.cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF# Apply sysctl params without rebootsudo sysctl --system
then we need to add the docker repository in order to install containerd in the latest version:
Lastly print the default config for containerd into it’s config file:
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee -a /etc/containerd/config.toml
sudo sed -i 's/^disabled_plugins \=/\#disabled_plugins \=/g' /etc/containerd/config.toml # containerd should enable the CRIsudo systemctl restart containerd
Now containerd should be installed, you can check with ctr version to see if you can access it (some errors are fine, you just want to see a version number).
Cgroups
The docs tell you a lot about cgroups and cgroup drivers. Take a look there if you want to understand why it’s needed and what it’s doing. For the cluster setup, it’s just important to agree on one variant and enforce this in all components. I’m using the systemd driver and cgroup v2. To enforce this on Ubuntu, we need to add a kernel parameter in the GRUB_CMDLINE_LINUX directive of /etc/default/grub:
And run a sudo update-grub followed by a reboot. Then we are sure, the system uses the correct cgroup driver and version.
We repeat the procedure and tell containerd to use systemd’s cgroup as well by modifiying /etc/containerd/config.toml:
cat <<EOF | sudo tee -a /etc/containerd/config.toml
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
EOF
And do a final restart of containerd:
sudo systemctl restart containerd
sudo systemctl status containerd
kubeadm, kubectl, kubelet
Next step is to get kubeadm (cluster bootstraping tool), kubectl and kubelet (cluster agent on systems) installed using package manager.
But to start we need to ensure some network settings are given. They should actually already be set when you used the above commands to install containerd.
Note: We mark kubelet, kubeadm and kubectl on hold so that they don’t get updated when updating other system packages. This is because they have to follow the versioning of Kubernetes and have some versions of room to differ. So it’s best to keep them at the same version as Kubernetes itself and update them when you update Kubernetes.
Make sure you set the cgroupDriver: systemd when bootstraping the cluster so that the kubelet uses the same cgroup driver as the container runtime.
kubelet config flags
If you want to use the hcloud-cloud-controller-manager later on to provision loadbalancers and volumes in Hetzner Cloud you must add the following drop-in for kubelet before initalizing the cluster:
One last thing before we can bootstrap our cluster is to choose a CNI for kubernetes. Although we could wait with that until kubeadm init is started we may need to set some flags in the init config to work with our CNI.
As said I’m using Cilium. Cilium has two options for IPAM, one being kubernetes and one beeing cluster-scoped cilium mode. I’m using the later and also use the kube-proxy replacement of cilium which means I can completly ignore the flags for service and pod CIDRs in kubeadm.
Bootstraping
Finally we can bootstrap our cluster with kubeadm. For this we will create a kubeadm config to customize our installation:
We are going to change some things in this default config. All options can be found in the reference docs. My file usually looks like that:
---
apiVersion: kubeadm.k8s.io/v1beta3kind: ClusterConfigurationclusterName: technat.k8scontrolPlaneEndpoint: admin.technat.dev:6443# DNS record pointing to your master nodes---
apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationcgroupDriver: systemd# must match the value you set for containerd
Some of the options are reasonable, some are only cosmetics and some are performance tweacks.
If you think the config looks good for you, you can start the initial bootstrap using the following command (remove the --skip-phase=addon/kube-proxy argument if you’re not using cilium):
The output will clearly show when you had success initializing your control-plane and when not. Expect this to fail the first time. In my experience custom configs always lead to an error in the first place. But fortunately the commands shows you where to start debugging.
If it worked, you can then get the kubeconfig from the kubernetes directory to your regular user and start kubeing:
See that -f cilium-values.yaml at the end of the last command? That’s the config for cilium. It looks like that for me:
rollOutCiliumPods: truepriorityClassName: "system-node-critical"annotateK8sNode: trueencryption:
enabled: truetype: wireguardoperator:
replicas: 1l7Proxy: false# not compatible with kube-proxy replacement (but better double-check if that's still true)hubble:
enabled: truerelay:
enabled: trueui:
enabled: truerollOutPods: truepolicyEnforcementMode: "always"# you should be using that but it will get you into a lot of work ;)kubeProxyReplacement: "strict"k8sServiceHost: "admin.technat.dev"k8sServicePort: "6443"
You can get the default config using helm show values cilium/cilium and then customize this to your needs.
Join master nodes
To join the other master nodes to the cluster, copy the command from our init output and run it on the other master nodes:
The package etcd-client provides us with an etcdctl that can be used to interact with the etcd cluster that backes the control-plane. However the tool needs some flags to be passed into for usage. I recommend you save yourself the following as an alias:
Make sure the master nodes have their taint on them so that only control-plane components are scheduled on them:
k taint nodes master-1 node-role.kubernetes.io/master:NoSchedule
type: LoadBalancer
You may have noticied that I installed my K8s cluster on Hetzner. To interact with Hetzner Cloud you would now need the Hcloud CCM in order to be able to provision LoadBalancers for Services.
See their repo for a quickstart how to install the manager and note that the prerequisite with the cloud-provider flag in the kubelet has already been done.