Create Multi Node Kubernetes Cluster with k3d

k3d is a lighweight wrapper to run k3s with docker. Unlike k3s, docker containers can be used to create Kubernetes nodes. It makes it useful to create local multi node clusters on the same machine like kind. This is particularly useful for devs and testers alike, as they would not need to deal with complication of setting up multi node Kubernetes setup.

Installation and Setup for k3d

One of the pre-requisites for k3d to work is docker. You can refer official instructions for same or one of our previous posts for installing docker in rootless mode.

Distribution specific instructions for installing k3d can be found at official docs. Installing k3d is as simple as grabbing the script at https://raw.githubusercontent.com/rancher/k3d/main/install.sh and running it:

cloud_user@d7bfd02ab81c:~$ wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
Preparing to install k3d into /usr/local/bin
k3d installed into /usr/local/bin/k3d
Run 'k3d --help' to see what you can do with it.
cloud_user@d7bfd02ab81c:~$ k3d --version
k3d version v4.4.2
k3s version v1.20.6-k3s1 (default)

Install Kubectl

We’ll need kubectl to work with Kubernetes cluster, in case its not already installed. For this, we can use below commands:

cloud_user@d7bfd02ab81c:~$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   154  100   154    0     0    564      0 --:--:-- --:--:-- --:--:--   564
100 44.2M  100 44.2M    0     0  42.4M      0  0:00:01  0:00:01 --:--:--  169M

cloud_user@d7bfd02ab81c:~$ chmod +x kubectl 

cloud_user@d7bfd02ab81c:~$ sudo mv kubectl /usr/local/bin

Create Kubernetes Cluster with k3d

Creating a cluster in the k3d is as simple as running k3d cluster create command:

cloud_user@8e8f37ca841c:~$ k3d cluster create mycluster
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-mycluster' (9e707271808c5c08e7fa7ba286fffe7a44d7797844951b115bde05ce89283173) 
INFO[0000] Created volume 'k3d-mycluster-images'        
INFO[0001] Creating node 'k3d-mycluster-server-0'       
INFO[0004] Pulling image 'docker.io/rancher/k3s:v1.20.6-k3s1' 
INFO[0017] Creating LoadBalancer 'k3d-mycluster-serverlb' 
INFO[0020] Pulling image 'docker.io/rancher/k3d-proxy:v4.4.2' 
INFO[0025] Starting cluster 'mycluster'                 
INFO[0025] Starting servers...                          
INFO[0025] Starting Node 'k3d-mycluster-server-0'       
INFO[0033] Starting agents...                           
INFO[0033] Starting helpers...                          
INFO[0033] Starting Node 'k3d-mycluster-serverlb'       
INFO[0034] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
WARN[0038] Failed to patch CoreDNS ConfigMap to include entry '172.18.0.1 host.k3d.internal': Exec process in node 'k3d-mycluster-server-0' failed with exit code '1' 
INFO[0038] Successfully added host record to /etc/hosts in 2/2 nodes 
INFO[0038] Cluster 'mycluster' created successfully!    
INFO[0038] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0038] You can now use it like this:                
kubectl config use-context k3d-mycluster
kubectl cluster-info

By default and contrary to the documentation, it will directly switch the default kubeconfig’s current-context to the new cluster’s context so that ~/.kube/config is automatically updated. We can see the cluster information with kubectl:

cloud_user@d7bfd02ab81c:~$ kubectl get nodes -o wide
NAME                     STATUS   ROLES                  AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION   CONTAINER-RUNTIME
k3d-mycluster-server-0   Ready    control-plane,master   2m25s   v1.20.6+k3s1   172.18.0.2    <none>        Unknown    5.4.0-1038-aws   containerd://1.4.4-k3s1

cloud_user@d7bfd02ab81c:~$ kubectl cluster-info
Kubernetes control plane is running at https://0.0.0.0:39165
CoreDNS is running at https://0.0.0.0:39165/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:39165/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

To avoid this, we can use the --kubeconfig-update-default=false flag while creating cluster. If we run docker ps at this time, we can see that two containers are created for creating single node Kubernetes cluster. Out of this, the other container k3d-mycluster-serverlb, is a load-balancer:

cloud_user@d7bfd02ab81c:~$ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED        STATUS         PORTS                             NAMES
025bfb9d3533   rancher/k3d-proxy:v4.4.2   "/bin/sh -c nginx-pr…"   30 hours ago   Up 9 minutes   80/tcp, 0.0.0.0:39165->6443/tcp   k3d-mycluster-serverlb
b4fc98027ce1   rancher/k3s:v1.20.6-k3s1   "/bin/k3s server --t…"   30 hours ago   Up 9 minutes                                     k3d-mycluster-server-0
cloud_user@d7bfd02ab81c:~$ 

We can also see that local port 39165 is mapped to API server port 6443 for Kubernetes cluster, with the help of load balancer container. The load balancer will then take care of proxying the requests to the appropriate server node as its acting as ingress. This holds true for the case of multi-node cluster as well.

Creating Multi-Server Clusters

Creating a multi node cluster is as easy as passing the number of servers (control plane nodes) and workers (agents) with the k3d cluster create command:

cloud_user@d7bfd02ab81c:~$ k3d cluster create multinode --agents 2 --servers 2
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-multinode' (55da3f39ab800281392fec338e7ab2acb4c6719935d8e5c1eb062dcac1fe0658) 
INFO[0000] Created volume 'k3d-multinode-images'        
INFO[0000] Creating initializing server node            
INFO[0000] Creating node 'k3d-multinode-server-0'       
INFO[0001] Creating node 'k3d-multinode-server-1'       
INFO[0001] Creating node 'k3d-multinode-agent-0'        
INFO[0001] Creating node 'k3d-multinode-agent-1'        
WARN[0001] You're creating 2 server nodes: Please consider creating at least 3 to achieve quorum & fault tolerance 
INFO[0001] Creating LoadBalancer 'k3d-multinode-serverlb' 
INFO[0001] Starting cluster 'multinode'                 
INFO[0001] Starting the initializing server...          
INFO[0001] Starting Node 'k3d-multinode-server-0'       
INFO[0003] Starting servers...                          
INFO[0003] Starting Node 'k3d-multinode-server-1'       
INFO[0027] Starting agents...                           
INFO[0027] Starting Node 'k3d-multinode-agent-0'        
INFO[0036] Starting Node 'k3d-multinode-agent-1'        
INFO[0045] Starting helpers...                          
INFO[0045] Starting Node 'k3d-multinode-serverlb'       
INFO[0047] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
INFO[0055] Successfully added host record to /etc/hosts in 5/5 nodes and to the CoreDNS ConfigMap 
INFO[0055] Cluster 'multinode' created successfully!    
INFO[0055] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0056] You can now use it like this:                
kubectl config use-context k3d-multinode
kubectl cluster-info

cloud_user@d7bfd02ab81c:~$ kubectl get nodes
NAME                     STATUS   ROLES                       AGE   VERSION
k3d-multinode-agent-0    Ready    <none>                      54s   v1.20.6+k3s1
k3d-multinode-agent-1    Ready    <none>                      45s   v1.20.6+k3s1
k3d-multinode-server-0   Ready    control-plane,etcd,master   75s   v1.20.6+k3s1
k3d-multinode-server-1   Ready    control-plane,etcd,master   62s   v1.20.6+k3s1

cloud_user@d7bfd02ab81c:~$ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED         STATUS          PORTS                             NAMES
f9226e70ed06   rancher/k3d-proxy:v4.4.2   "/bin/sh -c nginx-pr…"   3 minutes ago   Up 2 minutes    80/tcp, 0.0.0.0:34821->6443/tcp   k3d-multinode-serverlb
b945f568c333   rancher/k3s:v1.20.6-k3s1   "/bin/k3s agent"         3 minutes ago   Up 3 minutes                                      k3d-multinode-agent-1
df68cdcf55bd   rancher/k3s:v1.20.6-k3s1   "/bin/k3s agent"         3 minutes ago   Up 3 minutes                                      k3d-multinode-agent-0
0321f88031bc   rancher/k3s:v1.20.6-k3s1   "/bin/k3s server --t…"   3 minutes ago   Up 3 minutes                                      k3d-multinode-server-1
75a5a9d2a289   rancher/k3s:v1.20.6-k3s1   "/bin/k3s server --c…"   3 minutes ago   Up 3 minutes                                      k3d-multinode-server-0

We can also dynamically add/remove nodes to the cluster, so as to accommodate the application requirements:

# add an control plane node to cluster multinode
cloud_user@d7bfd02ab81c:~$ k3d node create server --role server --cluster multinode
INFO[0000] Starting Node 'k3d-server-0' 

# add an worker node to cluter multinode                
cloud_user@d7bfd02ab81c:~$ k3d node create myagent --role agent --cluster multinode
INFO[0000] Starting Node 'k3d-myagent-0'  
              
# list existing nodes in the cluster. 
# Alternatively, use kubectl get nodes                      
cloud_user@d7bfd02ab81c:~$ k3d node list 
NAME                     ROLE           CLUSTER     STATUS
k3d-multinode-agent-0    agent          multinode   running
k3d-multinode-agent-1    agent          multinode   running
k3d-multinode-server-0   server         multinode   running
k3d-multinode-server-1   server         multinode   running
k3d-multinode-serverlb   loadbalancer   multinode   running
k3d-myagent-0            agent          multinode   running
k3d-mycluster-server-0   server         mycluster   running
k3d-mycluster-serverlb   loadbalancer   mycluster   running
k3d-server-0             server         multinode   running

# remove the server node from multinode cluster
cloud_user@d7bfd02ab81c:~$ k3d node delete k3d-server-0
INFO[0000] Deleted k3d-server-0  

# verify that the node removal was complete                       
cloud_user@d7bfd02ab81c:~$ k3d node list
NAME                     ROLE           CLUSTER     STATUS
k3d-multinode-agent-0    agent          multinode   running
k3d-multinode-agent-1    agent          multinode   running
k3d-multinode-server-0   server         multinode   running
k3d-multinode-server-1   server         multinode   running
k3d-multinode-serverlb   loadbalancer   multinode   running
k3d-myagent-0            agent          multinode   running
k3d-mycluster-server-0   server         mycluster   running
k3d-mycluster-serverlb   loadbalancer   mycluster   running
cloud_user@d7bfd02ab81c:~$ 

Lifecycle for k3d cluster

We can view the current list of clusters created with k3d cluster list, stop a cluster with k3d cluster stop, start with k3d cluster start and delete one with k3d cluster delete command as below:

# list existing clusters and their status
cloud_user@d7bfd02ab81c:~$ k3d cluster list
NAME        SERVERS   AGENTS   LOADBALANCER
multinode   2/2       3/3      true
mycluster   1/1       0/0      true


# stop cluster mycluster
cloud_user@d7bfd02ab81c:~$ k3d cluster stop mycluster
INFO[0000] Stopping cluster 'mycluster'    

# verify that cluster is stopped and now has no active servers/agents             
cloud_user@d7bfd02ab81c:~$ k3d cluster list
NAME        SERVERS   AGENTS   LOADBALANCER
multinode   2/2       3/3      true
mycluster   0/1       0/0      true


# start cluster mycluster
cloud_user@d7bfd02ab81c:~$ k3d cluster start mycluster
INFO[0000] Starting cluster 'mycluster'                 
INFO[0000] Starting servers...                          
INFO[0000] Starting Node 'k3d-mycluster-server-0'       
INFO[0007] Starting agents...                           
INFO[0007] Starting helpers...                          
INFO[0007] Starting Node 'k3d-mycluster-serverlb' 

# verify that cluster is started and all nodes are running      
cloud_user@d7bfd02ab81c:~$ k3d cluster list
NAME        SERVERS   AGENTS   LOADBALANCER
multinode   2/2       3/3      true
mycluster   1/1       0/0      true


# delete cluster mycluser -- No safety switch here as you can delete a running cluster
cloud_user@d7bfd02ab81c:~$ k3d cluster delete mycluster
INFO[0000] Deleting cluster 'mycluster'                 
INFO[0000] Deleted k3d-mycluster-serverlb               
INFO[0004] Deleted k3d-mycluster-server-0               
INFO[0004] Deleting cluster network 'k3d-mycluster'     
INFO[0004] Deleting image volume 'k3d-mycluster-images' 
INFO[0004] Removing cluster details from default kubeconfig... 
INFO[0004] Removing standalone kubeconfig file (if there is one)... 
INFO[0004] Successfully deleted cluster mycluster! 

# verify that the cluster is removed     
cloud_user@d7bfd02ab81c:~$ k3d cluster list
NAME        SERVERS   AGENTS   LOADBALANCER
multinode   2/2       3/3      true
cloud_user@d7bfd02ab81c:~$ 

Deploy Application Containers

Deploying application container is as simple as normal application workloads deployments:

# create a simple deployment for nginx
cloud_user@d7bfd02ab81c:~$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

# expose deployment with service 
cloud_user@d7bfd02ab81c:~$ kubectl create service clusterip nginx --tcp=8080:80
service/nginx created

# expose service with nginx ingress
cloud_user@d7bfd02ab81c:~$ cat <<EOF | kubectl apply -f -
> apiVersion: networking.k8s.io/v1
> kind: Ingress
> metadata:
>   name: nginx
>   annotations:
>     ingress.kubernetes.io/ssl-redirect: "false"
> spec:
>   defaultBackend:
>     service:
>        name: nginx
>        port:
>           number: 8080
> EOF
ingress.networking.k8s.io/nginx created

cloud_user@d7bfd02ab81c:~$ kubectl get all -n default
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6799fc88d8-9vqg6   1/1     Running   0          16s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP   10.43.0.1       <none>        443/TCP    56m
service/nginx        ClusterIP   10.43.119.164   <none>        8080/TCP   16s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           16s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-6799fc88d8   1         1         1       16s

# verify that service is mapped with the nginx pods
cloud_user@d7bfd02ab81c:~$ kubectl describe svc nginx
Name:              nginx
Namespace:         default
Labels:            app=nginx
Annotations:       <none>
Selector:          app=nginx
Type:              ClusterIP
IP Families:       <none>
IP:                10.43.119.164
IPs:               10.43.119.164
Port:              8080-80  8080/TCP
TargetPort:        80/TCP
Endpoints:         10.42.5.4:80
Session Affinity:  None
Events:            <none>

# verify that ingress is matching the correct service port
cloud_user@d7bfd02ab81c:~$ kubectl describe ingress nginx
Name:             nginx
Namespace:        default
Address:          172.19.0.2
Default backend:  nginx:8080 (10.42.5.4:80)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     nginx:8080 (10.42.5.4:80)
Annotations:  ingress.kubernetes.io/ssl-redirect: false
Events:       <none>


# verify that we can access nginx
cloud_user@d7bfd02ab81c:~$ curl http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Create cluster with Specific Kubernetes Version

One may want to create a Kubernetes cluster with a specific version, to accommodate certain requirements. The list of available node images can be found at k3s docker repository. We can specify image with --image option, at cluster creation time.

cloud_user@d7bfd02ab81c:~$ k3d cluster create v121-dev --port 8080:80@loadbalancer --port 8443:443@loadbalancer --image rancher/k3s:v1.21.0-k3s1
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-v121-dev' (e57e9044185de6f0e1c5c6ff418e29fa9aec5d9a7c4ca8806e891420e39a138d) 
INFO[0000] Created volume 'k3d-v121-dev-images'         
INFO[0001] Creating node 'k3d-v121-dev-server-0'        
INFO[0004] Pulling image 'rancher/k3s:v1.21.0-k3s1'     
INFO[0010] Creating LoadBalancer 'k3d-v121-dev-serverlb' 
INFO[0010] Starting cluster 'v121-dev'                  
INFO[0010] Starting servers...                          
INFO[0010] Starting Node 'k3d-v121-dev-server-0'        
INFO[0019] Starting agents...                           
INFO[0019] Starting helpers...                          
INFO[0019] Starting Node 'k3d-v121-dev-serverlb'        
INFO[0021] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
INFO[0025] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap 
INFO[0025] Cluster 'v121-dev' created successfully!     
INFO[0025] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0026] You can now use it like this:                
kubectl config use-context k3d-v121-dev
kubectl cluster-info

cloud_user@d7bfd02ab81c:~$ kubectl get nodes -o wide
NAME                    STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION   CONTAINER-RUNTIME
k3d-v121-dev-server-0   Ready    control-plane,master   16s   v1.21.0+k3s1   172.20.0.2    <none>        Unknown    5.4.0-1045-aws   containerd://1.4.4-k3s1
cloud_user@d7bfd02ab81c:~$ 

Infrastructure as Code with k3d clusters

Instead of passing multiple command line options with k3d cluster create command, we can define all the things that you defined with CLI flags before in a nice and tidy YAML. Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag:

  • All options in config file: k3d cluster create --config /home/me/my-awesome-config.yaml (must be .yaml/.yml)
  • With CLI override (name): k3d cluster create somename --config /home/me/my-awesome-config.yaml
  • With CLI override (extra volume): k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server[0]'

More details are available at official documentation for same.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s