Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It follows the GitOps pattern of using git repositories as the source of truth for defining the desired application state. With ArgoCD, application deployments can be automated and updates to application can be made at the simple git commit events without the need of any complicated Continuous Integration and/or Deployment Pipelines.

This is our sixth post in the series of blog post on deploying and managing application with Kubernetes and Argo CD. You can find the series index here.

While Argo CD works well to manage Kubernetes resources, someone has to go and make a change in the docker image version in the manifests and commit it to Git Repository. This may become annoying for high-frequency deployment pipelines.

Argo CD Image Updater

Argo CD Image Updater is a tool which continuously monitors for new container images Kubernetes workloads that are managed by Argo CD. If the new versions are available, it can instructs the Argo CD to update the resource with the newer image available. Optionally, it can also be used to update the image tag in the application git repository automatically, which in turn will trigger a continuous deployment. This tool has out-of-the-box support for many popular, public container registries, such as Docker Hub, Red Hat Quay, or GitHub Container Registry, and supports private and on-premise registries as well.

Depending on your Automatic Sync Policy for the Application, Argo CD will either automatically deploy the new image version or mark the Application as Out Of Sync, and you can trigger the image update manually by syncing the Application.

Install Argo CD Image Updater

You can install Argo CD Image Updater as a standalone installation away from Kubernetes cluster and then give it access to various Kubernetes clusters that you want to manage. However, the straight forward way is to install it as a Kubernetes resource itself:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/v0.9.0/manifests/install.yaml

Once its done, you can check if the resources are created with:

mohitgoyal@desktop:/mnt/d/mohit/src/kustomize/argocd-example-apps$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/v0.9.0/manifests/install.yaml
serviceaccount/argocd-image-updater created
role.rbac.authorization.k8s.io/argocd-image-updater created
rolebinding.rbac.authorization.k8s.io/argocd-image-updater created
configmap/argocd-image-updater-config created
secret/argocd-image-updater-secret created
deployment.apps/argocd-image-updater created

mohitgoyal@desktop:/mnt/d/mohit/src/kustomize/argocd-example-apps$ kubectl rollout status deploy/argocd-image-updater -n argocd
deployment "argocd-image-updater" successfully rolled out

mohitgoyal@desktop:/mnt/d/mohit/src/kustomize/argocd-example-apps$ kubectl get all -n argocd
NAME                                        READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0         1/1     Running   0          10h
pod/argocd-dex-server-5dd657bd9-8xqsh       1/1     Running   2          10h
pod/argocd-image-updater-744bf747f4-tg8xx   1/1     Running   0          112s
pod/argocd-redis-759b6bc7f4-948j6           1/1     Running   0          10h
pod/argocd-repo-server-6c495f858f-jqnv9     1/1     Running   0          10h
pod/argocd-server-859b4b5578-zmvms          1/1     Running   0          10h

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/argocd-dex-server       ClusterIP   10.96.106.52    <none>        5556/TCP,5557/TCP,5558/TCP   10h
service/argocd-metrics          ClusterIP   10.96.163.130   <none>        8082/TCP                     10h
service/argocd-redis            ClusterIP   10.96.82.37     <none>        6379/TCP                     10h
service/argocd-repo-server      ClusterIP   10.96.80.226    <none>        8081/TCP,8084/TCP            10h
service/argocd-server           NodePort    10.96.80.126    <none>        80:32470/TCP,443:31000/TCP   10h
service/argocd-server-metrics   ClusterIP   10.96.135.142   <none>        8083/TCP                     10h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-dex-server      1/1     1            1           10h
deployment.apps/argocd-image-updater   1/1     1            1           112s
deployment.apps/argocd-redis           1/1     1            1           10h
deployment.apps/argocd-repo-server     1/1     1            1           10h
deployment.apps/argocd-server          1/1     1            1           10h

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-dex-server-5dd657bd9       1         1         1       10h
replicaset.apps/argocd-image-updater-744bf747f4   1         1         1       112s
replicaset.apps/argocd-redis-759b6bc7f4           1         1         1       10h
replicaset.apps/argocd-repo-server-6c495f858f     1         1         1       10h
replicaset.apps/argocd-server-859b4b5578          1         1         1       10h

NAME                                             READY   AGE
statefulset.apps/argocd-application-controller   1/1     10h

More details on the installation process are here.

Create an Application in Argo CD

Argo CD Image Updater can only update container images for applications whose manifests are rendered using either Kustomize or Helm. So we’ll create an application with Kustomize manifests.

As always, it all starts with defining an application. For purpose of this post, we’ll use example repository at https://github.com/goyalmohit/argocd-example-apps/tree/master/kustomize-guestbook.

To start, click the + NEW APP button and fill the upcoming form. After that, we need to submit fields as below:

Application Name – This is the application name inside Argo CD. You can use Argo CD to manage multiple application at once. We’ll set it to guestbook-ui in our case.

Project – This is the project name inside Argo CD. Project can be used to segregate and group applications together. Since this is a new setup for Argo CD, a default project is created for us and we’ll select the same. If you have multiple projects together, you’ll see an auto-populated list and you can choose the same.

Sync Policy – You can choose to auto synchronize the state of application in the Kubernetes with the GitHub repository or you can set it to manual. There are many choices, and we’ll probably discuss it in more details later. For now, leave it as manual.

We now need to provide the GitHub repository details containing the application manifests, in the same form. Submit the details as below:

Repository URL – Provide the url for the GitHub repository containing the application manifests. We can add private as well as public repositories. For private repositories, you’ll need to provide additional details related to the authentication. Since our repository is a public one, we do not need to provide additional details.

Revision – You can choose to provide the specific branch or tag for github repo and sync the same state with Kubernetes details. We’ll leave it as HEAD so that it can get latest commits at various points in the time and compare them with the application state inside the Kubernetes cluster.

Path – This helps in further segregating application manifests inside the GitHub repository. We can ask it to read only specific directories in the repository and read manifests within that path. We’ll select this as kustomize-guestbook.

Once we have provided source details where we described the desired state, we now need to provide the destination Kubernetes cluster details as below:

Cluster URL – Argo CD can be used to connect and deploy application to multiple Kubernetes clusters. Inside the UI, you’ll not get the option to add and connect a different Kubernetes cluster. This feature is restricted to Argo CD CLI. For our case, we’ll use the default in-cluster (where Argo CD itself is deployed).

Namespace – This can be used to select namespace where manifests will be deployed. You can choose a custom namespace and provide the same. Also, you’ll need to create the namespace on the target Kubernetes cluster before you can deploy manifests to it. Alternatively, you can select the checkbox for ‘AUTO-CREATE NAMESPACE’ in the sync options. We’ll leave it as default in our case.

Note that the namespace specified inside Kubernetes manifests overrides this value.

At this point, Argo CD will read the Kustomization.yaml file present in the path and will prompt us to allow override with different values. For our use case, we’ll go with the default configuration mentioned in the github repo:

After filling out the information above, click Create at the top of the UI to create the guestbook-ui application. After this, it will read out all the parameters, read the source Kubernetes manifests. After this, it will go to OutOfSync state since the application has yet to be deployed, and no Kubernetes resources have been created.

Synchronize the Application Manifests / Deploy the Application

As we mentioned above, the application status is initially in OutOfSync state since the application has yet to be deployed, and no Kubernetes resources have been created. To sync (deploy) the application, we can choose the tile and then select SYNC. This will present us with a choice about what we want to synchronize:

For now, we’ll leave the default options and select synchronize button. We’ll wait for few minutes till it deploys the application:

If we now see deployments inside Kubernetes, we can see it was deployed with container image gcr.io/heptio-images/ks-guestbook-demo:0.1:

mohitgoyal@DESKTOP-823COUV:/mnt/d/mohit/src/kustomize/argocd-example-apps$ kubectl get deploy -n default -o wide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS     IMAGES                                       SELECTOR
kustomize-demo-ui01      1/1     1            1           3h58m   demo-ui01      docker.io/mohitgoyal/demo-ui01:1.0.0         app=demo-ui01
kustomize-demo-ui02      1/1     1            1           3h58m   demo-ui02      docker.io/mohitgoyal/demo-ui02:1.0.0         app=demo-ui02
kustomize-demo-ui03      1/1     1            1           3h58m   demo-ui03      docker.io/mohitgoyal/demo-ui03:1.0.0         app=demo-ui03
kustomize-guestbook-ui   1/1     1            1           77s     guestbook-ui   gcr.io/heptio-images/ks-guestbook-demo:0.1   app=guestbook-ui

Mark Application for Image Update

By default, Image Updater would not manage your application resources unless it is asked to do so. For this, we need to annotate the Argo CD application resources with a list of images to be considered for update, along with a version constraint. After this, it will regularly poll the configured applications and queries the corresponding container registry for possible new versions. If a new version of the image is found in the registry, and the version constraint is met, it will instruct Argo CD to update the application with the new image.

Depending on your Automatic Sync Policy for the Application, Argo CD will either automatically deploy the new image version or mark the Application as Out Of Sync, and you can trigger the image update manually by syncing the Application. 

For its annotations, Argo CD Image Updater uses the annotation key of argocd-image-updater.argoproj.io/image-list which contains one or more container images along with version constraint as its value.

Also, the resources can be updated by instructing Argo CD in two different ways:

• By calling the Argo CD API directly (Imperatively) – This method works by calling Argo CD API directly. Argo CD can then follow the work as per sync policies defined. This method is pseudo-persistent. If you delete the Application resource from the cluster and re-create it, changes made by Image Updater will be gone. The same is true if you manage your Application resources using Git, and the version stored in Git is synced over the resource in the cluster. This method is most suitable for Applications also created imperatively, i.e. using the Web UI or CLI.

• By Updating the Git Repository (Declaratively) – This method works by using Git to permanently store its parameter overrides along with the Application’s resource manifests in the form of commit. This will enable persistent storage of the parameters in Git. This is more preferable method as it allows consistent view of sync state and live state and is also more auditable.

To use the Git write-back method, application needs to be tagged with additional annotation of argocd-image-updater.argoproj.io/write-back-method: git. You would also need to enable authentication to commit to the git repository for the application.

With all this information at hand, lets run the below command to tag guestbook-ui application we created above:

mohitgoyal@desktop:/mnt/d/mohit/src/kustomize/argocd-example-apps$ kubectl annotate app guestbook-ui argocd-image-updater.argoproj.io/image-list=gcr.io/heptio-i
mages/ks-guestbook-demo argocd-image-updater.argoproj.io/write-back-method=git -n argocd
application.argoproj.io/guestbook-ui annotated

See Image Updater in Action

The Image Updater checks applications every two minutes. Lets wait a couple of minutes and check the application again in Argo CD. Since we set sync policy as manual, it would mark the application as outofsync and stop further. We can see the difference in manifests:

Lets go and sync our application. Once sync is completed, lets check the deployment status in the Kubernetes and voila:

mohitgoyal@desktop:/mnt/d/mohit/src/kustomize/argocd-example-apps$ kubectl get deploy -n default -o wide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS     IMAGES                                       SELECTOR
kustomize-demo-ui01      1/1     1            1           4h23m   demo-ui01      docker.io/mohitgoyal/demo-ui01:1.0.0         app=demo-ui01
kustomize-demo-ui02      1/1     1            1           4h23m   demo-ui02      docker.io/mohitgoyal/demo-ui02:1.0.0         app=demo-ui02
kustomize-demo-ui03      1/1     1            1           4h23m   demo-ui03      docker.io/mohitgoyal/demo-ui03:1.0.0         app=demo-ui03
kustomize-guestbook-ui   1/1     1            1           26m     guestbook-ui   gcr.io/heptio-images/ks-guestbook-demo:0.2   app=guestbook-ui

We can see that the container image is bumped to gcr.io/heptio-images/ks-guestbook-demo:0.2

We can also go to the git repository for the application and see the commit created by image updater:

Limitations

There are couple of limitations with managing Kubernetes workloads:

• The applications you want container images to be updated must be managed using Argo CD. There is no support for workloads not managed using Argo CD.
• Argo CD Image Updater can only update container images for applications whose manifests are rendered using either Kustomize or Helm and especially in the case of Helm – the templates need to support specifying the image’s tag (and possibly name) using a parameter (i.e. image.tag).
• Image pull secrets must exist in the same Kubernetes cluster where Argo CD Image Updater is running in (or has access to). It is currently not possible to fetch those secrets from other clusters.