Containers

Create Kubernetes clusters with Kind, rootless Docker and rootless Podman

Mohit GoyalLeave a comment

Kind is a tool which can be used for running local multi node Kubernetes clusters. Kind was primarily designed for testing Kubernetes itself and associated extensions, but may be used for local development or CI. With the Kind, you can spin up a multi node Kubernetes cluster in couple of minutes, perform your work and then wind it down. And nobody need to worry about the Bill! Starting with kind 0.11.0, Rootless Docker and Rootless Podman can be used as the node provider of kind. Its not running kubernetes in rootless mode as some component of kubernetes stack cannot run rootless yet, however it is certainly a step in that direction.

Pre-requisites to run rootless Kind

Note that while ability to run rootless containers was always present in the Podman, it is only with the recent version of Docker that it moved to rootless stack. And you can use either Podman or Docker, depending on the container engine you are familiar with.

  • Docker: 20.10 or later
  • Podman: 3.0 or later
  • cgroup v2 should be enabled on the linux host. cgroup v2 is enabled by default on Fedora 31 or RHEL 8.

On other distros, cgroup v2 can be typically enabled by adding GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1" to /etc/default/grub and running sudo update-grub. In more recent versions of distributions, this functionality should be enabled by default. However, check instruction specific to your distribution.

For the purpose of this blog post, we are using Fedora 33, which has cgroup v2 enabled by default:

[cloud_user@217fabcd221c ~]$ cat /etc/*-release
Fedora release 33 (Thirty Three)
NAME=Fedora
VERSION="33 (Cloud Edition)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Cloud Edition)"
...
...

Also, depending on the host configuration, the following steps might be needed:

  • Create /etc/systemd/system/user@.service.d/delegate.conf with the following content, and then run sudo systemctl daemon-reload:
[Service]
Delegate=yes
[cloud_user@217fabcd221c ~]$ sudo mkdir /etc/systemd/system/user@.service.d

[cloud_user@217fabcd221c ~]$ sudo touch /etc/systemd/system/user@.service.d/delegate.conf

[cloud_user@217fabcd221c ~]$ sudo vi /etc/systemd/system/user@.service.d/delegate.conf

[cloud_user@217fabcd221c ~]$ sudo cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes
  • Create /etc/modules-load.d/iptables.conf with the following content:
iptables_nat
ip6tables_nat
[cloud_user@217fabcd221c ~]$ sudo vi /etc/modules-load.d/iptables.conf

[cloud_user@217fabcd221c ~]$ cat vi /etc/modules-load.d/iptables.conf
cat: vi: No such file or directory
iptables_nat
ip6tables_nat

Install and setup Kind

You can review the official installation instructions here. Check for kind releases here to identify latest release version. We can also use below steps to grab and install Kind:

# download the latest version of kind
[cloud_user@217fabcd221c kind]$ curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.10.0/kind-linux-amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    98  100    98    0     0    265      0 --:--:-- --:--:-- --:--:--   265
100   624  100   624    0     0    976      0 --:--:-- --:--:-- --:--:--  9313
100 7277k  100 7277k    0     0  2429k      0  0:00:02  0:00:02 --:--:-- 3852k

# assign execute permissions to binary
[cloud_user@217fabcd221c kind]$ chmod +x ./kind

# move binary to /usr/local/bin or add to $PATH
[cloud_user@217fabcd221c kind]$ sudo mv ./kind /usr/local/bin


# verify kind is working
[cloud_user@217fabcd221c kind]$ kind version
kind v0.10.0 go1.15.7 linux/amd64

Install Rootless Docker 20.10+

The instructions would depend in terms of pre-requisites and setup depending on the underlying linux distribution of choice. For detailed instructions, refer to official docs. TL;DR We can install Docker in rootless mode by grabbing and running script at https://get.docker.com/rootless:

[cloud_user@217fabcd221c kind]$ curl -fsSL https://get.docker.com/rootless | sh
# Installing stable version 20.10.5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 65.9M  100 65.9M    0     0   107M      0 --:--:-- --:--:-- --:--:--  107M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.0M  100 19.0M    0     0   100M      0 --:--:-- --:--:-- --:--:--  100M
+ PATH=/home/cloud_user/bin:/home/cloud_user/.local/bin:/home/cloud_user/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
+ /home/cloud_user/bin/dockerd-rootless-setuptool.sh install
[INFO] Creating /home/cloud_user/.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
+ sleep 3
+ systemctl --user --no-pager --full status docker.service
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/cloud_user/.config/systemd/user/docker.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2021-04-06 18:48:09 UTC; 3s ago
       Docs: https://docs.docker.com/engine/security/rootless/
   Main PID: 1727 (rootlesskit)
      Tasks: 38
     Memory: 26.4M
        CPU: 413ms
     CGroup: /user.slice/user-1001.slice/user@1001.service/docker.service
             ├─1727 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/cloud_user/bin/dockerd-rootless.sh
             ├─1740 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback
...
...
+ systemctl --user enable docker.service
Created symlink /home/cloud_user/.config/systemd/user/default.target.wants/docker.service → /home/cloud_user/.config/systemd/user/docker.service.
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger cloud_user`

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

export PATH=/home/cloud_user/bin:$PATH
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock

As instructed in the last few lines, set the environmental variables:

[cloud_user@217fabcd221c ~]$ echo "export PATH=/home/cloud_user/bin:$PATH" >> ~/.bashrc
[cloud_user@217fabcd221c ~]$ echo "export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock" >> ~/.bashrc

Verify docker is working with docker info:

[cloud_user@217fabcd221c ~]$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.5
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
  cgroupns
 Kernel Version: 5.10.22-200.fc33.x86_64
 Operating System: Fedora 33 (Cloud Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.687GiB
 Name: 217fabcd221c.mylabserver.com
 ID: MLMY:JY3Z:3MEK:EBXY:AVNR:XU73:S7V2:OMYO:HBMQ:JDUQ:AH5Y:JBDM
 Docker Root Dir: /home/cloud_user/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: Support for cgroup v2 is experimental
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

We can see that docker engine is automatically configured to use cgroups v2 by viewing property Cgroup Version: 2 and WARNING: Support for cgroup v2 is experimental in above output. If cgroups v2 is configured on your host OS, you do not need to do any special steps to enable it for docker engine. Same goes with Podman.

Firewall Changes for Fedora 33

For some reason, I kept getting error related to kind cluster nodes not being able to reach each other. Fortunately, its documented in the list of known issues in kind. I was able to work around this by changing the FirewallBackend in the /etc/firewalld/firewalld.conf file from nftables to iptables and restarting firewalld:

[cloud_user@217fabcd221c docker]$ sudo sed -i 's/FirewallBackend=.*/FirewallBackend=iptables/' "/etc/firewalld/firewalld.conf"
[sudo] password for cloud_user: 
[cloud_user@217fabcd221c docker]$ 

[cloud_user@217fabcd221c docker]$ systemctl restart firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ====
Authentication is required to restart 'firewalld.service'.
Authenticating as: cloud_user
Password: 
==== AUTHENTICATION COMPLETE ====

Create Kubernetes Cluster with Kind and Rootless Docker

Creating an cluster in kind is as simple as running kind create cluster.

As we mentioned above, that certain components of Kubernetes stack cannot be run in rootless mode yet. So even though we are using docker engine or podman in rootless mode, we cannot run kind in rootless mode. If we try to run kind in rootless mode, we’ll be greeted with an error like below:

[cloud_user@217fabcd221c docker]$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼
 ✓ Preparing nodes 📦  
 ✗ Writing configuration 📜 
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 1
Command Output: Error response from daemon: Container be1884b745fe628171f7330b50def65e99bf1e69880bb487fc1cdd73fc0a4e83 is not running

However, we can run and launch Kubernetes cluster fine when using kind with sudo privileges:

[cloud_user@217fabcd221c docker]$ sudo kind create cluster
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

[cloud_user@217fabcd221c ~]$ sudo kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:32769
KubeDNS is running at https://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Install Rootless Podman 3.0+

This is an experimental release which is not supported in all linux distributions yet. You can read official instructions here for your distributions here. Once you are done with installation, verify podman is working by using podman version:

[cloud_user@217fabcd221c ~]$ podman version
Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.8
Built:        Fri Feb 19 16:56:17 2021
OS/Arch:      linux/amd64

Create Kubernetes Cluster with Kind and Rootless Podman

To instruct kind to use podman, we have to enable property KIND_EXPERIMENTAL_PROVIDER and set it to podman kind create cluster:

[cloud_user@217fabcd221c ~]$ export KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster

Again as we mentioned above, that certain components of Kubernetes stack cannot be run in rootless mode yet. So even though we are using docker engine or podman in rootless mode, we cannot run kind in rootless mode. However we can launch cluster fine when running kind with sudo privileges:

[cloud_user@217fabcd221c ~]$ sudo kind create cluster
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
[cloud_user@217fabcd221c ~]$ sudo kubectl cluster-info --context kind-kind
[sudo] password for cloud_user: 
Sorry, try again.
[sudo] password for cloud_user: 
Kubernetes control plane is running at https://127.0.0.1:43385
KubeDNS is running at https://127.0.0.1:43385/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

[cloud_user@217fabcd221c ~]$ sudo kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
kind-control-plane   Ready    control-plane,master   4m50s   v1.20.2

How do we know if it is indeed using Podman? By looking for below output in the kubectl cluster-info dump:

[cloud_user@217fabcd221c ~]$ sudo kubectl cluster-info --context kind-kind dump | grep -i podman
                "providerID": "kind://podman/kind/kind-control-plane"

Leave a comment