systemd has long been the de-facto standard for managing services and their dependencies in linux. While its good to run applications within containers, to provide a certain functionality and to avoid installing packages on the host OS, the availability and reliability has been an issue. Before you go ahead and start using an application packaged inside that container, you need to make sure that container is up and running. And what if your application consists of multiple containers which needed to be started in a certain order. In fact, there is a growing set of applications which are available as containers so that users can bypass all the headaches associated with installation and setup. So there is a use case for the systemd to take control and manage containers as native services.

Not only this, you may also want to make sure that your application is up and running inside container itself, it has got all the dependencies, is starting the processes in the right order, etc. You can create a solution by invoking custom scripts using entrypoints. A more better approach would be to use an init system inside containers to take care of this requirement.

So integration with systemd can be approached in two entirely different ways. First one, is to run containers as systemd services for the host OS and second one, is to take care of application requirements inside containers with systemd. We will be focusing on handling the first requirement for this blog post.

systemd integration with Podman and Docker Engine

Docker Engine has been designed to perform many of the tasks that systemd performs such as service initialization and activation, logging and monitoring etc. Also, docker engine is divided into client server architecture, where you would interact with docker daemon (or dockerd) using API calls. The dockerd receives the request and takes appropriate actions. However, this poses a problem for systemd since it has no reliable way to know the state of containers and act on it.

There are also use cases where you would like to start containers at certain points in time (or on-demand), rather than always having to run them and consume system resources. For this, there is a feature called socket activation in the systemd, which allows you to do it, but dockerd does not provides such functionality. There are more use cases but that’s not the focus of this post.

With the serverless architecture of the podman, where you can fork a container by simply instructing runc, it offers a more seamless integration with systemd. Features like sd-notify and socket activation make this integration even more important. The sd-notify service manager allows a service to notify systemd that the process is ready to receive connections, and socket activation permits systemd to launch the containerized process only when a packet arrives from a monitored socket.

This also improves the auditing associated with the containers. This is a valid use case for certain type of business and transactions.

Different options for enabling systemd services

When enabling systemd services, you can choose to enable it system-wide or user specific. For example:

• To enable the service at system start, irrespective of whether if a user is logged in or not, use systemctl enable <service>. For this, we have to copy the systemd unit files to the /etc/systemd/system directory.
• To start a service at user login and stop at user logout, use systemctl --user enable <service>. For this, we have to copy the systemd unit files to the $HOME/.config/systemd/user directory. This option also allows us to run rootless containers.
• To enable users to start a service at system start and persist over logouts, use loginctl enable-linger <user>. This option also allows us to run rootless containers.

Generating systemd unit files with Podman

With the latest version of podman, we can use podman generate systemd $container command to auto-generate the systemd unit files. Infact, it is recommended to use podman generate systemd because the generated units files change frequently (via updates to Podman) and the podman generate systemd ensures that you get the latest version of unit files.

To create a unit file, we first need to pull an image and run a container with Podman, before we can issue podman generate command. Lets do that with the busybox image:

[cloud_user@81f87084a51c ~]$ sudo podman run -d --name busybox docker.io/library/busybox:latest
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 8b3d7e226fab done  
Copying config a9d583973f done  
Writing manifest to image destination
Storing signatures
e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c

[cloud_user@81f87084a51c ~]$ sudo podman ps -a
CONTAINER ID  IMAGE                             COMMAND  CREATED         STATUS                     PORTS   NAMES
e9dd4f366ee2  docker.io/library/busybox:latest  sh       37 minutes ago  Exited (0) 37 minutes ago          busybox

We can now instruct the podman to generate the systemd unit file and view the same:

[cloud_user@81f87084a51c ~]$ sudo podman generate systemd busybox > ~/.config/systemd/user/busybox_cloud_user.service
[cloud_user@81f87084a51c ~]$ cat ~/.config/systemd/user/busybox_cloud_user.service
# container-e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c.service
# autogenerated by Podman 2.2.1
# Sun Mar 28 11:27:56 UTC 2021

[Unit]
Description=Podman container-e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c
ExecStop=/usr/bin/podman stop -t 10 e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c
ExecStopPost=/usr/bin/podman stop -t 10 e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c
PIDFile=/var/run/containers/storage/overlay-containers/e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

Discussing the systemd unit file

In the service parameter block in the above file, the Restart=on-failure line sets the restart policy and instructs systemd to restart when the service cannot be started or stopped cleanly, or when the process exits non-zero.

The ExecStart line describes how we start the container. Note that since we are using the podman start command here, it expects a container with the required container id to be present. A more generic version of systemd unit file, would be using podman run command, instead of podman start.

The ExecStop line describes how we stop and remove the container. Again, this needs more modification for a generic systemd unit file as we cannot expect container id to be the same, if its not already present.

An important part is the Main PID, which points to the correct conmon process. Conmon is a small monitoring tool that podman uses to perform operations such as keeping ports and file descriptors open, streaming container logs and cleaning up once the container is finished. Without explicitly pointing systemd to the correct process via the PIDFile option, systemd might wrongly choose another process in this cgroup as the main process.

Another important part is to set the kill mode to none. Otherwise, systemd will start competing with podman to stop and kill the container processes. which can lead to various undesired issues.

Create more Generic systemd unit file

Generating a more generic systemd unit file is tricky. As we mentioned earlier, we cannot use podman start. So we need to use podman run to create and run a container. We cannot know in advance, what will be the container id that will be created after a container is created. We know that we can use the container id and container name, interchangeably for some commands. What about Conmon? We also need it to control and manage the container.

To solve these kind of issues, we can use podman generate systemd --new command. The --new flag instructs Podman to generate units that create, start, and remove containers. Podman also supports generating units files with the --new flag for pods. Previously, the --new flag was limited to containers. Lets see this in action.

Lets cleanup existing unit files and containers first:

[cloud_user@81f87084a51c ~]$ sudo podman ps -a
[sudo] password for cloud_user: 
CONTAINER ID  IMAGE                             COMMAND  CREATED            STATUS                        PORTS   NAMES
e9dd4f366ee2  docker.io/library/busybox:latest  sh       About an hour ago  Exited (0) About an hour ago          busybox

[cloud_user@81f87084a51c ~]$ sudo podman stop busybox
e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c

[cloud_user@81f87084a51c ~]$ sudo podman rm busybox
e9dd4f366ee26531c9b051309a25a1977a78fd9f500b4b42df47c77a2ad5d23c

[cloud_user@81f87084a51c ~]$ sudo rm ~/.config/systemd/user/busybox_cloud_user.service
[cloud_user@81f87084a51c ~]$ 

Now, we’ll create a container (and not run it) with Podman and then generate a unit file:

[cloud_user@81f87084a51c ~]$ sudo podman create --name busybox docker.io/library/busybox:latest
14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c

[cloud_user@81f87084a51c ~]$ sudo podman generate systemd --new busybox > ~/.config/systemd/user/busybox_cloud_user.service

[cloud_user@81f87084a51c ~]$ cat ~/.config/systemd/user/busybox_cloud_user.service
# container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.service
# autogenerated by Podman 2.2.1
# Sun Mar 28 11:57:39 UTC 2021

[Unit]
Description=Podman container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStartPre=/bin/rm -f %t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.pid %t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.ctr-id
ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.pid --cidfile %t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.ctr-id --cgroups=no-conmon -d --replace --name busybox docker.io/library/busybox:latest
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.ctr-id
PIDFile=%t/container-14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

Unfortunately, we still see the container id getting at all the places. To properly generate the generic unit file, we need to run it along it --files parameter like below:

[cloud_user@81f87084a51c ~]$ sudo podman generate systemd --new --files --name busybox
[sudo] password for cloud_user: 
Sorry, try again.
[sudo] password for cloud_user: 
/home/cloud_user/container-busybox.service

[cloud_user@81f87084a51c ~]$ cat container-busybox.service 
# container-busybox.service
# autogenerated by Podman 2.2.1
# Sun Mar 28 12:03:12 UTC 2021

[Unit]
Description=Podman container-busybox.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStartPre=/bin/rm -f %t/container-busybox.pid %t/container-busybox.ctr-id
ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-busybox.pid --cidfile %t/container-busybox.ctr-id --cgroups=no-conmon -d --replace --name busybox docker.io/library/busybox:latest
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-busybox.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-busybox.ctr-id
PIDFile=%t/container-busybox.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

In the above file, the --conmon-pidfile option points to a path to store the process id for the conmon process running on the host, the --cidfile option points to the path that stores the container id, the %t is the path to the run time directory root, for example /run/user/$UserID and %n is the full name of the service.

See systemd in action with containers

Now that we have generated a proper generic unit file, we can configure it with systemd to run and manage containers. Note that since we are not running podman in rootless mode, we can not enable services at user level i.e. cloud_user in our case. If you want to enable systemd services at user level, follow instructions as mentioned above.

# copy service file to /etc/systemd/system
[cloud_user@81f87084a51c ~]$ sudo cp -Z container-busybox.service /etc/systemd/system

# creates systemd service at system level
[cloud_user@81f87084a51c ~]$ sudo systemctl enable container-busybox.service 
Created symlink /etc/systemd/system/multi-user.target.wants/container-busybox.service → /etc/systemd/system/container-busybox.service.
Created symlink /etc/systemd/system/default.target.wants/container-busybox.service → /etc/systemd/system/container-busybox.service.

# start the service
[cloud_user@81f87084a51c ~]$ sudo systemctl start container-busybox.service
 

# verifies the service status
[cloud_user@81f87084a51c ~]$ sudo systemctl status container-busybox.service 
● container-busybox.service - Podman container-busybox.service
   Loaded: loaded (/etc/systemd/system/container-busybox.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Sun 2021-03-28 12:41:26 UTC; 8s ago
     Docs: man:podman-generate-systemd(1)
  Process: 7921 ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
  Process: 7874 ExecStop=/usr/bin/podman stop --ignore --cidfile /run/container-busybox.ctr-id -t 10 (code=exited, status=0/SUCCESS)
  Process: 7671 ExecStart=/usr/bin/podman run --conmon-pidfile /run/container-busybox.pid --cidfile /run/container-busybox.ctr-id --cgroups=no-conmon -d --replace --na>
  Process: 7668 ExecStartPre=/bin/rm -f /run/container-busybox.pid /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
 Main PID: 7765 (code=exited, status=0/SUCCESS)

Mar 28 12:41:24 81f87084a51c.mylabserver.com systemd[1]: Starting Podman container-busybox.service...
Mar 28 12:41:24 81f87084a51c.mylabserver.com podman[7671]: 14cdc35451d68d83c90f3b6e76fc395fed4d876e4565be769d6a25f5b8c7158c
Mar 28 12:41:25 81f87084a51c.mylabserver.com podman[7671]: 59f0bdd7fd7140895afc95a50a7115de6cde3929420c18d537828aebf3fc07d9
Mar 28 12:41:25 81f87084a51c.mylabserver.com systemd[1]: Started Podman container-busybox.service.
Mar 28 12:41:25 81f87084a51c.mylabserver.com podman[7874]: 59f0bdd7fd7140895afc95a50a7115de6cde3929420c18d537828aebf3fc07d9
Mar 28 12:41:26 81f87084a51c.mylabserver.com podman[7921]: 59f0bdd7fd7140895afc95a50a7115de6cde3929420c18d537828aebf3fc07d9
Mar 28 12:41:26 81f87084a51c.mylabserver.com systemd[1]: container-busybox.service: Succeeded.

In the above output you can note that although the container is successfully created and loaded by systemd, it exited. This is because we failed to pass a argument to our container, that will keep it running. Since our container exited with an status of 0, it means that systemd has detected that operation has been successful and it will not trigger the on-failure action. Such behavioral differences are extremely important to consider when writing systemd services, so we need to be careful when setting the restart policy.

Lets modify our systemd unit file to pass an argument like top to keep our container running and re-copy it to the proper location for unit files and voila:

[cloud_user@81f87084a51c ~]$ sudo systemctl daemon-reload

[cloud_user@81f87084a51c ~]$ sudo systemctl start container-busybox.service
 
[cloud_user@81f87084a51c ~]$ sudo systemctl status container-busybox.service 
● container-busybox.service - Podman container-busybox.service
   Loaded: loaded (/etc/systemd/system/container-busybox.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-03-28 12:56:25 UTC; 12s ago
     Docs: man:podman-generate-systemd(1)
  Process: 9152 ExecStart=/usr/bin/podman run --conmon-pidfile /run/container-busybox.pid --cidfile /run/container-busybox.ctr-id --cgroups=no-conmon -d --replace --na>
  Process: 9150 ExecStartPre=/bin/rm -f /run/container-busybox.pid /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
 Main PID: 9241 (conmon)
    Tasks: 2 (limit: 23192)
   Memory: 2.3M
   CGroup: /system.slice/container-busybox.service
           └─9241 /usr/bin/conmon --api-version 1 -c 112ae3d9d09a0629c8ce55d6ac66f44835529f10b3bd867b2b951fa1ee87deab -u 112ae3d9d09a0629c8ce55d6ac66f44835529f10b3bd86>

Mar 28 12:56:24 81f87084a51c.mylabserver.com systemd[1]: Starting Podman container-busybox.service...
Mar 28 12:56:25 81f87084a51c.mylabserver.com podman[9152]: 112ae3d9d09a0629c8ce55d6ac66f44835529f10b3bd867b2b951fa1ee87deab
Mar 28 12:56:25 81f87084a51c.mylabserver.com systemd[1]: Started Podman container-busybox.service.

We can verify if our container is running indeed with podman ps command:

[cloud_user@81f87084a51c ~]$ sudo podman ps -a
CONTAINER ID  IMAGE                             COMMAND  CREATED             STATUS                 PORTS   NAMES
112ae3d9d09a  docker.io/library/busybox:latest  top      About a minute ago  Up About a minute ago          busybox

If we now kill the container by issuing a SIGTERM, the systemd will notice that it has been killed and triggers on-failure action, which is set to restart in our case:

[cloud_user@81f87084a51c ~]$ sudo kill -9 9252
[cloud_user@81f87084a51c ~]$ 
[cloud_user@81f87084a51c ~]$ sudo systemctl status container-busybox.service 
● container-busybox.service - Podman container-busybox.service
   Loaded: loaded (/etc/systemd/system/container-busybox.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-03-28 13:01:37 UTC; 33s ago
     Docs: man:podman-generate-systemd(1)
  Process: 9486 ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
  Process: 9543 ExecStart=/usr/bin/podman run --conmon-pidfile /run/container-busybox.pid --cidfile /run/container-busybox.ctr-id --cgroups=no-conmon -d --replace --na>
  Process: 9541 ExecStartPre=/bin/rm -f /run/container-busybox.pid /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
 Main PID: 9632 (conmon)
    Tasks: 2 (limit: 23192)
   Memory: 2.1M
   CGroup: /system.slice/container-busybox.service
           └─9632 /usr/bin/conmon --api-version 1 -c 3675e8e29efd21e2e5fac937792e0e8948e6eef1207bcf8ec7e84bf9282d31d7 -u 3675e8e29efd21e2e5fac937792e0e8948e6eef1207bcf>

Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: container-busybox.service: Service RestartSec=100ms expired, scheduling restart.
Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: container-busybox.service: Scheduled restart job, restart counter is at 1.
Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: Stopped Podman container-busybox.service.
Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: Starting Podman container-busybox.service...
Mar 28 13:01:37 81f87084a51c.mylabserver.com podman[9543]: 3675e8e29efd21e2e5fac937792e0e8948e6eef1207bcf8ec7e84bf9282d31d7
Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: Started Podman container-busybox.service.
[cloud_user@81f87084a51c ~]$ 

To stop the container, we can use systemctl stop command. Since we are now using system to terminate it, it will be terminated and removed properly. We can verify the same from output of podman ps as well.

[cloud_user@81f87084a51c ~]$ sudo systemctl stop container-busybox.service 

[cloud_user@81f87084a51c ~]$ sudo systemctl status container-busybox.service 
● container-busybox.service - Podman container-busybox.service
   Loaded: loaded (/etc/systemd/system/container-busybox.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2021-03-28 13:04:21 UTC; 3s ago
     Docs: man:podman-generate-systemd(1)
  Process: 9831 ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
  Process: 9706 ExecStop=/usr/bin/podman stop --ignore --cidfile /run/container-busybox.ctr-id -t 10 (code=exited, status=0/SUCCESS)
  Process: 9543 ExecStart=/usr/bin/podman run --conmon-pidfile /run/container-busybox.pid --cidfile /run/container-busybox.ctr-id --cgroups=no-conmon -d --replace --na>
  Process: 9541 ExecStartPre=/bin/rm -f /run/container-busybox.pid /run/container-busybox.ctr-id (code=exited, status=0/SUCCESS)
 Main PID: 9632 (code=exited, status=143)

Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: Stopped Podman container-busybox.service.
Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: Starting Podman container-busybox.service...
Mar 28 13:01:37 81f87084a51c.mylabserver.com podman[9543]: 3675e8e29efd21e2e5fac937792e0e8948e6eef1207bcf8ec7e84bf9282d31d7
Mar 28 13:01:37 81f87084a51c.mylabserver.com systemd[1]: Started Podman container-busybox.service.
Mar 28 13:04:20 81f87084a51c.mylabserver.com systemd[1]: Stopping Podman container-busybox.service...
Mar 28 13:04:21 81f87084a51c.mylabserver.com podman[9706]: 3675e8e29efd21e2e5fac937792e0e8948e6eef1207bcf8ec7e84bf9282d31d7
Mar 28 13:04:21 81f87084a51c.mylabserver.com systemd[1]: container-busybox.service: Main process exited, code=exited, status=143/n/a
Mar 28 13:04:21 81f87084a51c.mylabserver.com podman[9831]: 3675e8e29efd21e2e5fac937792e0e8948e6eef1207bcf8ec7e84bf9282d31d7
Mar 28 13:04:21 81f87084a51c.mylabserver.com systemd[1]: container-busybox.service: Failed with result 'exit-code'.
Mar 28 13:04:21 81f87084a51c.mylabserver.com systemd[1]: Stopped Podman container-busybox.service.

[cloud_user@81f87084a51c ~]$ sudo podman ps -a
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES