Configure Pods to run as Systemd services with Podman

This blog post continues from where we left in our earlier blog post, where we discussed how systemd and podman fits in together to run and manage containers as systemd services. We discussed how we can do the same for a specific containers and learned to create generic systemd unit files. We also discussed few use cases where this integration is useful. As we know, pods in podman are a way to group and manage multiple application containers as one. So we can start them together, manage them together and then remove them together once done. If we do need, we can manage them individually as well. Basics of pods are covered here.

However, when we have multiple containers, we often want to start them in a specific order. For example, one of the containers may be app container and another is db container, so you would want to start first the db container and then only start the app container. You may also want the rule to not start app container at all if db container fails to come up. Lets see how we can do that with a wordpress and mariadb container as an example.

Create a Pod with Multiple Containers

Lets first create an empty pod named as wpapp_pod and verify it comes up fine:

[cloud_user@96dd25ebf51c ~]$ podman pod create -p 8080:80 --name wpapp_pod
bdd858cbdfcd7de51a2e0097a46ee10bc983a5c4d6edfb725cea8bce2e5a2263
[cloud_user@96dd25ebf51c ~]$ 
[cloud_user@96dd25ebf51c ~]$ podman pod list
POD ID        NAME       STATUS   CREATED        INFRA ID      # OF CONTAINERS
bdd858cbdfcd  wpapp_pod  Created  6 seconds ago  cb3d1c6f32d1  1
[cloud_user@96dd25ebf51c ~]$ podman ps -a
CONTAINER ID  IMAGE                 COMMAND  CREATED         STATUS   PORTS                 NAMES
cb3d1c6f32d1  k8s.gcr.io/pause:3.2           10 seconds ago  Created  0.0.0.0:8080->80/tcp  bdd858cbdfcd-infra

Now we’ll add mariadb container and wordpress container to above pod:

[cloud_user@96dd25ebf51c ~]$ podman run \
>  -d --restart=always --pod wpapp_pod \
>  -e MYSQL_ROOT_PASSWORD="myrootpass" \
>  -e MYSQL_DATABASE="wp-db" \
>  -e MYSQL_USER="wp-user" \
>  -e MYSQL_PASSWORD="w0rdpr3ss" \
>  --name=wptest-db mariadb
Completed short name "mariadb" with unqualified-search registries (origin: /etc/containers/registries.conf)
Trying to pull registry.access.redhat.com/mariadb:latest...
  name unknown: Repo not found
Trying to pull registry.redhat.io/mariadb:latest...
  unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Trying to pull docker.io/library/mariadb:latest...
Getting image source signatures
Copying blob a70d879fa598 done  
Copying blob 10e6159c56c0 done  
Copying blob 75bac9accf15 done  
Copying blob 6ee25b525863 done  
Copying blob 5553d41e82b1 done  
Copying blob c4394a92d1f8 done  
Copying blob 0df5d11e9502 done  
Copying blob b61f99a83efe done  
Copying blob 6fd69eaf8690 done  
Copying blob 03f7bd4f1dda done  
Copying blob 6ce5fa137d06 done  
Copying blob 689a8c518900 done  
Copying config e76a4b2ed1 done  
Writing manifest to image destination
Storing signatures
8e4813b3218fe8a40f4262e3cfcd2ad646b94fa4e66a01898fe2b6bea074f941

[cloud_user@96dd25ebf51c ~]$ podman run \
>  -d --restart=always --pod=wpapp_pod \
>  -e WORDPRESS_DB_NAME="wp-db" \
>  -e WORDPRESS_DB_USER="wp-user" \
>  -e WORDPRESS_DB_PASSWORD="w0rdpr3ss" \
>  -e WORDPRESS_DB_HOST="127.0.0.1" \
>  --name wptest-web wordpress
Completed short name "wordpress" with unqualified-search registries (origin: /etc/containers/registries.conf)
Trying to pull registry.access.redhat.com/wordpress:latest...
  name unknown: Repo not found
Trying to pull registry.redhat.io/wordpress:latest...
  unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Trying to pull docker.io/library/wordpress:latest...
Getting image source signatures
Copying blob 6da3e75ee2ca done  
Copying blob 038e5b090752 done  
Copying blob 56671971dcc6 done  
Copying blob 854fb08fe050 done  
Copying blob 75646c2fb410 done  
Copying blob d099f6707d86 done  
Copying blob f33a657f956e done  
Copying blob 249520ff71af done  
Copying blob 88fd46807e1d done  
Copying blob 4213c3e42364 done  
Copying blob 4915809df15f done  
Copying blob 2faa4b167ab4 done  
Copying blob 78435232ad8f done  
Copying blob 662883b7bb15 done  
Copying blob bf62eea5448f done  
Copying blob 92a1afd88c46 done  
Copying blob e0f9cda83bc3 done  
Copying blob a01ecf9f410a done  
Copying blob 608ccbf945cb done  
Copying blob fd4a2a57c3c7 done  
Copying blob 0ca288048117 done  
Copying config bfcb597091 done  
Writing manifest to image destination
Storing signatures
41f21efffcf4d2a6839f9a1ac519f20f248bcf2407b1a63e63bbb70604c0ed4b

Verify that all 3 containers are up and running now:

[cloud_user@96dd25ebf51c ~]$ podman pod list
POD ID        NAME       STATUS   CREATED        INFRA ID      # OF CONTAINERS
bdd858cbdfcd  wpapp_pod  Running  6 minutes ago  cb3d1c6f32d1  3

[cloud_user@96dd25ebf51c ~]$ podman ps -a --pod
CONTAINER ID  IMAGE                               COMMAND               CREATED        STATUS            PORTS                 NAMES               POD ID        PODNAME
41f21efffcf4  docker.io/library/wordpress:latest  apache2-foregroun...  4 minutes ago  Up 4 minutes ago  0.0.0.0:8080->80/tcp  wptest-web          bdd858cbdfcd  wpapp_pod
8e4813b3218f  docker.io/library/mariadb:latest    mysqld                5 minutes ago  Up 5 minutes ago  0.0.0.0:8080->80/tcp  wptest-db           bdd858cbdfcd  wpapp_pod
cb3d1c6f32d1  k8s.gcr.io/pause:3.2                                      6 minutes ago  Up 5 minutes ago  0.0.0.0:8080->80/tcp  bdd858cbdfcd-infra  bdd858cbdfcd  wpapp_pod

Generate and Review systemd unit files for pod

We can generate systemd unit files with podman generate systemd command. We’ll also add --files parameter to create generic files and use --new parameter to supply pod name:

[cloud_user@96dd25ebf51c ~]$ podman generate systemd --files --name wpapp_pod
/home/cloud_user/container-wptest-db.service
/home/cloud_user/container-wptest-web.service
/home/cloud_user/pod-wpapp_pod.service

We can see that it created 3 files for 3 containers. Now lets review the systemd unit file for pod i.e. /home/cloud_user/pod-wpapp_pod.service:

[cloud_user@96dd25ebf51c ~]$ cat pod-wpapp_pod.service 
# pod-wpapp_pod.service
# autogenerated by Podman 2.2.1
# Mon Apr  5 09:21:31 UTC 2021

[Unit]
Description=Podman pod-wpapp_pod.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
Requires=container-wptest-db.service container-wptest-web.service
Before=container-wptest-db.service container-wptest-web.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start bdd858cbdfcd-infra
ExecStop=/usr/bin/podman stop -t 10 bdd858cbdfcd-infra
ExecStopPost=/usr/bin/podman stop -t 10 bdd858cbdfcd-infra
PIDFile=/run/user/1001/containers/overlay-containers/cb3d1c6f32d1f8eab036a35e5a4030515654934fc957572bc618fff65d0bbcf5/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

Few properties are of interest here in the unit section:

  • Requires=: This directive lists any units upon which this unit essentially depends. If the current unit is activated, the units listed here must successfully activate as well, else this unit will fail. These units are started in parallel with the current unit by default.
  • Wants=: This directive is similar to Requires=, but less strict. Systemd will attempt to start any units listed here when this unit is activated. If these units are not found or fail to start, the current unit will continue to function. This is the recommended way to configure most dependency relationships. Again, this implies a parallel activation unless modified by other directives.
  • Before=: The units listed in this directive will not be started until the current unit is marked as started if they are activated at the same time. This does not imply a dependency relationship and must be used in conjunction with one of the above directives if this is desired.
  • After=: The units listed in this directive will be started before starting the current unit. This does not imply a dependency relationship and one must be established through the above directives if this is required.

Another interesting property is BindsTo=, which is similar to Requires=, but also causes the current unit to stop when the associated unit terminates.

Now, we’ll review and modify our unit files as per our requirements. In the pod-specific unit file, we do not want to start wptest-web and wptest-db containers in parallel. However we may want to mark it as fail if these are not running. Similarly, we want to start wptest-web after wptest-db container. Let’s modify our files and configure them as below:

[cloud_user@96dd25ebf51c ~]$ cat pod-wpapp_pod.service 
# pod-wpapp_pod.service
# autogenerated by Podman 2.2.1
# Mon Apr  5 09:21:31 UTC 2021

[Unit]
Description=Podman pod-wpapp_pod.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
BindsTo=container-wptest-db.service container-wptest-web.service
Before=container-wptest-db.service container-wptest-web.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start bdd858cbdfcd-infra
ExecStop=/usr/bin/podman stop -t 10 bdd858cbdfcd-infra
ExecStopPost=/usr/bin/podman stop -t 10 bdd858cbdfcd-infra
PIDFile=/run/user/1001/containers/overlay-containers/cb3d1c6f32d1f8eab036a35e5a4030515654934fc957572bc618fff65d0bbcf5/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

[cloud_user@96dd25ebf51c ~]$ cat container-wptest-db.service 
# container-wptest-db.service
# autogenerated by Podman 2.2.1
# Mon Apr  5 09:21:31 UTC 2021

[Unit]
Description=Podman container-wptest-db.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
BindsTo=pod-wpapp_pod.service
After=pod-wpapp_pod.service
Before=container-wptest-web.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start wptest-db
ExecStop=/usr/bin/podman stop -t 10 wptest-db
ExecStopPost=/usr/bin/podman stop -t 10 wptest-db
PIDFile=/run/user/1001/containers/overlay-containers/8e4813b3218fe8a40f4262e3cfcd2ad646b94fa4e66a01898fe2b6bea074f941/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

[cloud_user@96dd25ebf51c ~]$ cat container-wptest-web.service 
# container-wptest-web.service
# autogenerated by Podman 2.2.1
# Mon Apr  5 09:21:31 UTC 2021

[Unit]
Description=Podman container-wptest-web.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
BindsTo=pod-wpapp_pod.service
After=pod-wpapp_pod.service container-wptest-db.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start wptest-web
ExecStop=/usr/bin/podman stop -t 10 wptest-web
ExecStopPost=/usr/bin/podman stop -t 10 wptest-web
PIDFile=/run/user/1001/containers/overlay-containers/41f21efffcf4d2a6839f9a1ac519f20f248bcf2407b1a63e63bbb70604c0ed4b/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target
[cloud_user@96dd25ebf51c ~]$ 

Enables systemd Services for Pod

First we need to copy all the generated files to $HOME/.config/systemd/user for installing as a non-root user:

[cloud_user@96dd25ebf51c ~]$ cp container-wptest-web.service container-wptest-db.service pod-wpapp_pod.service $HOME/.config/systemd/user
[cloud_user@96dd25ebf51c ~]$ ls $HOME/.config/systemd/user
container-wptest-db.service  container-wptest-web.service  pod-wpapp_pod.service

Now we can enable systemd for current user:

[cloud_user@96dd25ebf51c ~]$ systemctl enable --user pod-wpapp_pod.service 
Created symlink /home/cloud_user/.config/systemd/user/multi-user.target.wants/pod-wpapp_pod.service → /home/cloud_user/.config/systemd/user/pod-wpapp_pod.service.
Created symlink /home/cloud_user/.config/systemd/user/default.target.wants/pod-wpapp_pod.service → /home/cloud_user/.config/systemd/user/pod-wpapp_pod.service.

Note that we have not created any services for the dependent application containers. We can verify if service is enabled with below command:

[cloud_user@96dd25ebf51c user]$ systemctl is-enabled --user pod-wpapp_pod.service 
enabled

Verify with systemd in action

Before that, lets stop our pod and verify that all 3 containers are stopped:

[cloud_user@96dd25ebf51c ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

[cloud_user@96dd25ebf51c ~]$ podman ps -a
CONTAINER ID  IMAGE                               COMMAND               CREATED         STATUS                    PORTS                 NAMES
14297e417b1f  docker.io/library/wordpress:latest  apache2-foregroun...  10 minutes ago  Exited (0) 5 minutes ago  0.0.0.0:8080->80/tcp  wptest-web
62b438173484  docker.io/library/mariadb:latest    mysqld                10 minutes ago  Exited (0) 5 minutes ago  0.0.0.0:8080->80/tcp  wptest-db
cf349f0c3d97  k8s.gcr.io/pause:3.2                                      11 minutes ago  Exited (0) 5 minutes ago  0.0.0.0:8080->80/tcp  ddcb4333bec6-infra

Now we can start our application with systemctl enable --user command:

[cloud_user@96dd25ebf51c user]$ systemctl start --user pod-wpapp_pod.service 
[cloud_user@96dd25ebf51c user]$ systemctl status --user pod-wpapp_pod.service 
● pod-wpapp_pod.service - Podman pod-wpapp_pod.service
   Loaded: loaded (/home/cloud_user/.config/systemd/user/pod-wpapp_pod.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-04-05 10:16:50 UTC; 8s ago
     Docs: man:podman-generate-systemd(1)
  Process: 25874 ExecStart=/usr/bin/podman start ddcb4333bec6-infra (code=exited, status=0/SUCCESS)
 Main PID: 25917 (conmon)
   CGroup: /user.slice/user-1001.slice/user@1001.service/pod-wpapp_pod.service
           ├─25892 /usr/bin/fuse-overlayfs -o lowerdir=/home/cloud_user/.local/share/containers/storage/overlay/l/PC6CJP3JJJJLUZCLTJBR7RUHBL,upperdir=/home/cloud_user/>
           ├─25897 /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1001/netns/cni-4>
           ├─25899 containers-rootlessport
           ├─25906 containers-rootlessport-child
           ├─25917 /usr/bin/conmon --api-version 1 -c cf349f0c3d9723ecb8e1c54c4646fbb15e117cf72b3a6bad1a4ed7f010e0b1d9 -u cf349f0c3d9723ecb8e1c54c4646fbb15e117cf72b3a6>
           └─cf349f0c3d9723ecb8e1c54c4646fbb15e117cf72b3a6bad1a4ed7f010e0b1d9
             └─25928 /pause

Apr 05 10:16:50 96dd25ebf51c.mylabserver.com systemd[1550]: Starting Podman pod-wpapp_pod.service...
Apr 05 10:16:50 96dd25ebf51c.mylabserver.com podman[25874]: ddcb4333bec6-infra
Apr 05 10:16:50 96dd25ebf51c.mylabserver.com systemd[1550]: Started Podman pod-wpapp_pod.service.


# verify that now our containers come up fine
[cloud_user@96dd25ebf51c user]$ podman ps -a
CONTAINER ID  IMAGE                               COMMAND               CREATED         STATUS             PORTS                 NAMES
14297e417b1f  docker.io/library/wordpress:latest  apache2-foregroun...  15 minutes ago  Up 23 seconds ago  0.0.0.0:8080->80/tcp  wptest-web
62b438173484  docker.io/library/mariadb:latest    mysqld                15 minutes ago  Up 24 seconds ago  0.0.0.0:8080->80/tcp  wptest-db
cf349f0c3d97  k8s.gcr.io/pause:3.2                                      16 minutes ago  Up 25 seconds ago  0.0.0.0:8080->80/tcp  ddcb4333bec6-infra

Do note that that mariadb container came up before wordpress container, as indicated by the difference in the start time status. Since we have instructed systemd to stop at user logout and start at user login, it will do so. However we cannot show it here due to obvious limits of text based posts :).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s