As we discussed in previous posts, that Docker team re-factored, extracted and donated containerd as container runtime to the CNCF in Mar’2017. As of docker engine 1.11, docker engine switch to containerd as default runtime as well and it has been there since. containerd implements downloading images, managing them, and running containers from images. When it needs to run a container it unpacks the image into an OCI runtime bundle and shells out to runc to run it. Also, containerd is platform agnostic, so you can run it on pretty much any OS.
Initially, when containerd was started, it had single purpose – to manage and run containers. However, over the time, more and more functionality was added such as downloading images, networking interfaces and management etc. One of the reasons for adding more functionality is to make it easier to use in other projects. For example, in projects like Kubernetes, it was beneficial for containerd to do additional things like push and pull images. However, all the extra functionality is modular and optional and is available in the form of plugins such as cri, namespaces, cgroups, images, tasks, etc, meaning you can pick and choose which bits you want. So, it’s possible to include containerd in projects such as Kubernetes, but only to take the pieces your project needs.
To see all the plugins containerd has, use
ctr plugins list. Here are some of the typical plugins which are generally available:
cloud_user@3b36d84d6d1c:~$ sudo ctr plugins list [sudo] password for cloud_user: TYPE ID PLATFORMS STATUS io.containerd.content.v1 content - ok io.containerd.snapshotter.v1 aufs linux/amd64 ok io.containerd.snapshotter.v1 btrfs linux/amd64 error io.containerd.snapshotter.v1 devmapper linux/amd64 error io.containerd.snapshotter.v1 native linux/amd64 ok io.containerd.snapshotter.v1 overlayfs linux/amd64 ok io.containerd.snapshotter.v1 zfs linux/amd64 error io.containerd.metadata.v1 bolt - ok io.containerd.differ.v1 walking linux/amd64 ok io.containerd.gc.v1 scheduler - ok io.containerd.service.v1 introspection-service - ok io.containerd.service.v1 containers-service - ok io.containerd.service.v1 content-service - ok io.containerd.service.v1 diff-service - ok io.containerd.service.v1 images-service - ok io.containerd.service.v1 leases-service - ok io.containerd.service.v1 namespaces-service - ok io.containerd.service.v1 snapshots-service - ok io.containerd.runtime.v1 linux linux/amd64 ok io.containerd.runtime.v2 task linux/amd64 ok io.containerd.monitor.v1 cgroups linux/amd64 ok io.containerd.service.v1 tasks-service - ok io.containerd.internal.v1 restart - ok io.containerd.grpc.v1 containers - ok io.containerd.grpc.v1 content - ok io.containerd.grpc.v1 diff - ok io.containerd.grpc.v1 events - ok io.containerd.grpc.v1 healthcheck - ok io.containerd.grpc.v1 images - ok io.containerd.grpc.v1 leases - ok io.containerd.grpc.v1 namespaces - ok io.containerd.internal.v1 opt - ok io.containerd.grpc.v1 snapshots - ok io.containerd.grpc.v1 tasks - ok io.containerd.grpc.v1 version - ok cloud_user@3b36d84d6d1c:~$
You can see more information about a particular plugin’s load status using option
-d, where -d is for detailed information:
cloud_user@3b36d84d6d1c:~$ sudo ctr plugins list -d id==namespaces id==btrfs Type: io.containerd.snapshotter.v1 ID: btrfs Platforms: linux/amd64 Exports: root /var/lib/containerd/io.containerd.snapshotter.v1.btrfs Error: Code: Unknown Message: path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin Type: io.containerd.grpc.v1 ID: namespaces Requires: io.containerd.service.v1
More information on plugins is available at https://github.com/containerd/containerd/blob/master/docs/PLUGINS.md.
The configuration for containerd can be seen using
containerd config command. Its custom configuration is saved in the
/etc/containerd/config.toml file and can be customized using the same. Below is the default configuration and it contains some of the settings worth pondering over.
cloud_user@3b36d84d6d1c:~$ containerd config NAME: containerd config - information on the containerd config USAGE: containerd config command [command options] [arguments...] COMMANDS: default see the output of the default config dump see the output of the final main config with imported in subconfig files OPTIONS: --help, -h show help cloud_user@3b36d84d6d1c:~$ containerd config default version = 2 root = "/var/lib/containerd" state = "/run/containerd" plugin_dir = "" disabled_plugins =  required_plugins =  oom_score = 0 [grpc] address = "/run/containerd/containerd.sock" tcp_address = "" tcp_tls_cert = "" tcp_tls_key = "" uid = 0 gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 [ttrpc] address = "" uid = 0 gid = 0 [debug] address = "" uid = 0 gid = 0 level = "" [metrics] address = "" grpc_histogram = false [cgroup] path = "" [timeouts] "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s" [plugins] [plugins."io.containerd.gc.v1.scheduler"] pause_threshold = 0.02 deletion_threshold = 0 mutation_threshold = 100 schedule_delay = "0s" startup_delay = "100ms" [plugins."io.containerd.grpc.v1.cri"] disable_tcp_service = true stream_server_address = "127.0.0.1" stream_server_port = "0" stream_idle_timeout = "4h0m0s" enable_selinux = false selinux_category_range = 1024 sandbox_image = "k8s.gcr.io/pause:3.2" stats_collect_period = 10 systemd_cgroup = false enable_tls_streaming = false max_container_log_line_size = 16384 disable_cgroup = false disable_apparmor = false restrict_oom_score_adj = false max_concurrent_downloads = 3 disable_proc_mount = false unset_seccomp_profile = "" tolerate_missing_hugetlb_controller = true disable_hugetlb_controller = true ignore_image_defined_volumes = false [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "overlayfs" default_runtime_name = "runc" no_pivot = false disable_snapshot_annotations = true discard_unpacked_layers = false [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] runtime_type = "" runtime_engine = "" runtime_root = "" privileged_without_host_devices = false base_runtime_spec = "" [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] runtime_type = "" runtime_engine = "" runtime_root = "" privileged_without_host_devices = false base_runtime_spec = "" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" runtime_engine = "" runtime_root = "" privileged_without_host_devices = false base_runtime_spec = "" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" max_conf_num = 1 conf_template = "" [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io"] [plugins."io.containerd.grpc.v1.cri".image_decryption] key_model = "" [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins."io.containerd.internal.v1.opt"] path = "/opt/containerd" [plugins."io.containerd.internal.v1.restart"] interval = "10s" [plugins."io.containerd.metadata.v1.bolt"] content_sharing_policy = "shared" [plugins."io.containerd.monitor.v1.cgroups"] no_prometheus = false [plugins."io.containerd.runtime.v1.linux"] shim = "containerd-shim" runtime = "runc" runtime_root = "" no_shim = false shim_debug = false [plugins."io.containerd.runtime.v2.task"] platforms = ["linux/amd64"] [plugins."io.containerd.service.v1.diff-service"] default = ["walking"] [plugins."io.containerd.snapshotter.v1.devmapper"] root_path = "" pool_name = "" base_image_size = "" async_remove = false cloud_user@3b36d84d6d1c:~$
Linux Runtime’s Plugin
If we carefully go through above output, we can see the following configuration for linux runtime plugin:
[plugins."io.containerd.runtime.v1.linux"] shim = "containerd-shim" runtime = "runc" runtime_root = "" no_shim = false shim_debug = false
Here we can see some of the important configuration like which runtime is spin up to create containers, which is runc in this case. We can also see the shim used to attach with the runc, which is containerd-shim in this case.
containerd’s Role in Container Ecosystem
Here’s one of the old reports but its still significant and upholds today. As we can see that containerd is the primary component to deal with the OCI images. Other container engines such as docker, docker-swarm, AKS, EKS, Mesos, DC/OS etc depends on it.
Container Runtime Interface or CRI
cri is a containerd plugin implementation of the Kubernetes container runtime interface (CRI). With it, we are able to use containerd as the container runtime for a Kubernetes cluster:
Since containerd 1.1, the cri plugin is built into the release binaries and enabled by default. There are many other Kubernetes CRI compatible runtimes available such as rktlet, frakti, virtlet, cri-o, dockershim, etc.
containerd also provides an API and client application that can be used to interact with it. The containerd command line client is
ctr. We can view the options available using
cloud_user@3b36d84d6d1c:~$ ctr --help NAME: ctr - __ _____/ /______ / ___/ __/ ___/ / /__/ /_/ / \___/\__/_/ containerd CLI USAGE: ctr [global options] command [command options] [arguments...] VERSION: 1.4.3 DESCRIPTION: ctr is an unsupported debug and administrative client for interacting with the containerd daemon. Because it is unsupported, the commands, options, and operations are not guaranteed to be backward compatible or stable from release to release of the containerd project. COMMANDS: plugins, plugin provides information about containerd plugins version print the client and server versions containers, c, container manage containers content manage content events, event display containerd events images, image, i manage images leases manage leases namespaces, namespace, ns manage namespaces pprof provide golang pprof outputs for containerd run run a container snapshots, snapshot manage snapshots tasks, t, task manage tasks install install a new package oci OCI tools shim interact with a shim directly help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --debug enable debug output in logs --address value, -a value address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS] --timeout value total timeout for ctr commands (default: 0s) --connect-timeout value timeout for connecting to containerd (default: 0s) --namespace value, -n value namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE] --help, -h show help --version, -v print the version
For example, we can pull images using containerd with
sudo ctr images pull <full image path>. Most of these commands would need to be run with root privileges:
And list images we have using
sudo ctr images list:
Same set of operations can be performed with API provided by containerd. You can read more on the same with official docs. Again as we said previously, you may or may not need to know none/some/all of the details, depending on your job role, interests and time available at hand.