Going Down the Rabbit Hole of Docker Engine… – containerd

As we discussed in previous posts, that Docker team re-factored, extracted and donated containerd as container runtime to the CNCF in Mar’2017. As of docker engine 1.11, docker engine switch to containerd as default runtime as well and it has been there since. containerd implements downloading images, managing them, and running containers from images. When it needs to run a container it unpacks the image into an OCI runtime bundle and shells out to runc to run it. Also, containerd is platform agnostic, so you can run it on pretty much any OS.

containerd Plugins

Initially, when containerd was started, it had single purpose – to manage and run containers. However, over the time, more and more functionality was added such as downloading images, networking interfaces and management etc. One of the reasons for adding more functionality is to make it easier to use in other projects. For example, in projects like Kubernetes, it was beneficial for containerd to do additional things like push and pull images. However, all the extra functionality is modular and optional and is available in the form of plugins such as cri, namespaces, cgroups, images, tasks, etc, meaning you can pick and choose which bits you want. So, it’s possible to include containerd in projects such as Kubernetes, but only to take the pieces your project needs.

To see all the plugins containerd has, use ctr plugins list. Here are some of the typical plugins which are generally available:

cloud_user@3b36d84d6d1c:~$ sudo ctr plugins list
[sudo] password for cloud_user: 
TYPE                            ID                       PLATFORMS      STATUS    
io.containerd.content.v1        content                  -              ok        
io.containerd.snapshotter.v1    aufs                     linux/amd64    ok        
io.containerd.snapshotter.v1    btrfs                    linux/amd64    error     
io.containerd.snapshotter.v1    devmapper                linux/amd64    error     
io.containerd.snapshotter.v1    native                   linux/amd64    ok        
io.containerd.snapshotter.v1    overlayfs                linux/amd64    ok        
io.containerd.snapshotter.v1    zfs                      linux/amd64    error     
io.containerd.metadata.v1       bolt                     -              ok        
io.containerd.differ.v1         walking                  linux/amd64    ok        
io.containerd.gc.v1             scheduler                -              ok        
io.containerd.service.v1        introspection-service    -              ok        
io.containerd.service.v1        containers-service       -              ok        
io.containerd.service.v1        content-service          -              ok        
io.containerd.service.v1        diff-service             -              ok        
io.containerd.service.v1        images-service           -              ok        
io.containerd.service.v1        leases-service           -              ok        
io.containerd.service.v1        namespaces-service       -              ok        
io.containerd.service.v1        snapshots-service        -              ok        
io.containerd.runtime.v1        linux                    linux/amd64    ok        
io.containerd.runtime.v2        task                     linux/amd64    ok        
io.containerd.monitor.v1        cgroups                  linux/amd64    ok        
io.containerd.service.v1        tasks-service            -              ok        
io.containerd.internal.v1       restart                  -              ok        
io.containerd.grpc.v1           containers               -              ok        
io.containerd.grpc.v1           content                  -              ok        
io.containerd.grpc.v1           diff                     -              ok        
io.containerd.grpc.v1           events                   -              ok        
io.containerd.grpc.v1           healthcheck              -              ok        
io.containerd.grpc.v1           images                   -              ok        
io.containerd.grpc.v1           leases                   -              ok        
io.containerd.grpc.v1           namespaces               -              ok        
io.containerd.internal.v1       opt                      -              ok        
io.containerd.grpc.v1           snapshots                -              ok        
io.containerd.grpc.v1           tasks                    -              ok        
io.containerd.grpc.v1           version                  -              ok        

You can see more information about a particular plugin’s load status using option -d, where -d is for detailed information:

cloud_user@3b36d84d6d1c:~$ sudo ctr plugins list -d id==namespaces id==btrfs
Type:          io.containerd.snapshotter.v1
ID:            btrfs
Platforms:     linux/amd64
               root      /var/lib/containerd/io.containerd.snapshotter.v1.btrfs
               Code:        Unknown
               Message:     path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin
Type:          io.containerd.grpc.v1
ID:            namespaces

More information on plugins is available at https://github.com/containerd/containerd/blob/master/docs/PLUGINS.md.

containerd config

The configuration for containerd can be seen using containerd config command. Its custom configuration is saved in the /etc/containerd/config.toml file and can be customized using the same. Below is the default configuration and it contains some of the settings worth pondering over.

cloud_user@3b36d84d6d1c:~$ containerd config
   containerd config - information on the containerd config

   containerd config command [command options] [arguments...]

   default  see the output of the default config
   dump     see the output of the final main config with imported in subconfig files

   --help, -h  show help
cloud_user@3b36d84d6d1c:~$ containerd config default
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0

  address = "/run/containerd/containerd.sock"
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

  address = ""
  uid = 0
  gid = 0

  address = ""
  uid = 0
  gid = 0
  level = ""

  address = ""
  grpc_histogram = false

  path = ""

  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"
    disable_tcp_service = true
    stream_server_address = ""
    stream_server_port = "0"
    stream_idle_timeout = "4h0m0s"
    enable_selinux = false
    selinux_category_range = 1024
    sandbox_image = "k8s.gcr.io/pause:3.2"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    disable_cgroup = false
    disable_apparmor = false
    restrict_oom_score_adj = false
    max_concurrent_downloads = 3
    disable_proc_mount = false
    unset_seccomp_profile = ""
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    ignore_image_defined_volumes = false
      snapshotter = "overlayfs"
      default_runtime_name = "runc"
      no_pivot = false
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
        privileged_without_host_devices = false
        base_runtime_spec = ""
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
        privileged_without_host_devices = false
        base_runtime_spec = ""
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          privileged_without_host_devices = false
          base_runtime_spec = ""
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      max_conf_num = 1
      conf_template = ""
          endpoint = ["https://registry-1.docker.io"]
      key_model = ""
      tls_cert_file = ""
      tls_key_file = ""
    path = "/opt/containerd"
    interval = "10s"
    content_sharing_policy = "shared"
    no_prometheus = false
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
    platforms = ["linux/amd64"]
    default = ["walking"]
    root_path = ""
    pool_name = ""
    base_image_size = ""
    async_remove = false

Linux Runtime’s Plugin

If we carefully go through above output, we can see the following configuration for linux runtime plugin:

    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false

Here we can see some of the important configuration like which runtime is spin up to create containers, which is runc in this case. We can also see the shim used to attach with the runc, which is containerd-shim in this case.

containerd’s Role in Container Ecosystem

Here’s one of the old reports but its still significant and upholds today. As we can see that containerd is the primary component to deal with the OCI images. Other container engines such as docker, docker-swarm, AKS, EKS, Mesos, DC/OS etc depends on it.

Container Runtime Interface or CRI

cri is a containerd plugin implementation of the Kubernetes container runtime interface (CRI). With it, we are able to use containerd as the container runtime for a Kubernetes cluster:

Since containerd 1.1, the cri plugin is built into the release binaries and enabled by default. There are many other Kubernetes CRI compatible runtimes available such as rktlet, frakti, virtlet, cri-o, dockershim, etc.

containerd CLI

containerd also provides an API and client application that can be used to interact with it. The containerd command line client is ctr. We can view the options available using ctr --help:

cloud_user@3b36d84d6d1c:~$ ctr --help
   ctr - 
  _____/ /______
 / ___/ __/ ___/
/ /__/ /_/ /

containerd CLI

   ctr [global options] command [command options] [arguments...]


ctr is an unsupported debug and administrative client for interacting
with the containerd daemon. Because it is unsupported, the commands,
options, and operations are not guaranteed to be backward compatible or
stable from release to release of the containerd project.

   plugins, plugin            provides information about containerd plugins
   version                    print the client and server versions
   containers, c, container   manage containers
   content                    manage content
   events, event              display containerd events
   images, image, i           manage images
   leases                     manage leases
   namespaces, namespace, ns  manage namespaces
   pprof                      provide golang pprof outputs for containerd
   run                        run a container
   snapshots, snapshot        manage snapshots
   tasks, t, task             manage tasks
   install                    install a new package
   oci                        OCI tools
   shim                       interact with a shim directly
   help, h                    Shows a list of commands or help for one command

   --debug                      enable debug output in logs
   --address value, -a value    address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS]
   --timeout value              total timeout for ctr commands (default: 0s)
   --connect-timeout value      timeout for connecting to containerd (default: 0s)
   --namespace value, -n value  namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE]
   --help, -h                   show help
   --version, -v                print the version

For example, we can pull images using containerd with sudo ctr images pull <full image path>. Most of these commands would need to be run with root privileges:

And list images we have using sudo ctr images list:

Same set of operations can be performed with API provided by containerd. You can read more on the same with official docs. Again as we said previously, you may or may not need to know none/some/all of the details, depending on your job role, interests and time available at hand.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s