Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It follows the GitOps pattern of using git repositories as the source of truth for defining the desired application state. With ArgoCD, application deployments can be automated and updates to application can be made at the simple git commit events without the need of any complicated Continuous Integration and/or Deployment Pipelines.

This is our third post in the series of blog post on deploying and managing application with Kubernetes and Argo CD. In the first two posts, we discussed how to install and setup Argo CD, how to deploy application, view and manage application health and manage application lifecycle. You can find the series index here.

For the purpose of this post, we’ll create a multi node Kubernetes cluster with Kind (created with Docker and WSL2 backend).

Understand Various Sync Options for Applications

As we have noted down earlier, there are quite a few sync options available for the application manifests. We have first observed the same, while we created the application in the Argo CD workspace. It asked us if we want to opt for Manual or Automatic sync policy:

As indicated by name, Manual means the sync operations will be performed only when a human operator will click sync button.

Automated Sync Policy

Argo CD has the ability to automatically sync an application when it detects differences between the desired manifests in Git, and the live state in the cluster. A benefit of automatic sync is that CI/CD pipelines no longer need direct access to the Argo CD API server to perform the deployment. Instead, the developers only need to make commit to tracked branches/tags in the associated git repository.

We can choose to select this policy when creating application or edit it any point in application lifecycle by modifying it. We can also choose to apply automated policy either with UI or CLI.

argocd app set <APPNAME> --sync-policy automated

If we have selected Automated sync policy, it will asks us further details on resource pruning and self-healing capabilities:

Resource Pruning

By default (and as a safety mechanism), automated sync will not delete resources when Argo CD detects the resource is no longer defined in Git. To prune the resources, a manual sync can always be performed (with pruning checked). Pruning can also be enabled to happen automatically as part of the automated sync, by selecting the checkbox for the ‘PRUNE RESOURCES’:

From Argo CD CLI:

argocd app set <APPNAME> --auto-prune

Self-Heal Capability

By default, changes that are made to the live cluster will not trigger automated sync. So even after we manually edited the service type for guestbook-ui from ClusterIP to NodePort, it has not reflected and our application status was marked as synced.

This presents an anomaly, where human operators can login to the Kubernetes clusters and make changes to the applications deployed, either intentionally or un-intentionally. This also presents a potential risks to the application health, as operators cannot rely on the resource state of application presented in the Argo CD and make wrong assumptions. To prevent such issues, we can mark the checkbox for ‘SELF HEAL’:

To enable automatic sync with CLI, run:

argocd app set <APPNAME> --self-heal

Note that if selfHeal flag is set to true then sync will be attempted again after self heal timeout (5 seconds by default) which is controlled by --self-heal-timeout-seconds flag of argocd-application-controller deployment.

Once we enable selfHeal flag, it will look and nullify the deviation from the desired state on its own. In our case, it will reset the service type for guestbook-ui to the ClusterIP type:

NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-85985d774c-69nrr   1/1     Running   0          152m

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.163.151   <none>        80/TCP    152m
service/kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP   5h

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/guestbook-ui   1/1     1            1           152m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/guestbook-ui-85985d774c   1         1         1       152m

Perform Selective Sync

By default, the auto sync applies to every option in the application. For repositories containing multiple manifests, this may take a long time and puts undue pressure on the api server. We can choose to turn on selective sync option, which will sync only out-of-sync resources.

• Add ApplyOutOfSync=true in manifest:

syncPolicy:
  syncOptions:
    - ApplyOutOfSyncOnly=true

• Via Argo CD CLI:

$ argocd app set guestbook --sync-option ApplyOutOfSyncOnly=true

• Selective sync while performing sync manually:

There are other sync options such as auto-create namespace, replace, skip schema validation etc. most of which perform the tasks associated with their name. These are not part of automated policies, as you may want to perform them during certain scenarios or to meet specific requirements and not at all times.

For example, by default, Argo CD will not try to create a Kubernetes namespace, if it is not present. This may result in deployment errors. To prevent such errors, we can use auto-create namespace feature. However, this may also let developers create namespaces on the fly by mentioning them in the application manifests, which may or may not be what is desired by Kubernetes cluster operators for application lifecycle management.

Again by default, Argo CD executes kubectl apply operation to apply the configuration stored in Git. In some cases kubectl apply is not suitable. So this feature lets you replace Kubernetes resources with the new manifest definitions defined in the git repo. However, this may result in un-wanted or un-intended downtime for the application depending on the underlying resource.

Defining Sync Windows

You may want to restrict sync based on the certain windows of time meeting specific criteria. For example, you may deny to perform sync operations only on weekdays as you have no one in office during weekends monitoring application health. Alternatively, you may perform sync operations only on weekends as you do not want any application health change for your customers on weekdays. You may want to disallow any sync for specific days when you have high application usage like cyber monday, etc.

Sync windows are configurable windows of time where syncs will either be blocked or allowed. These are defined by a kind, which can be either allow or deny, a schedule in cron format and a duration along with one or more of either applications, namespaces and clusters. Wildcards are supported.

View Sync Window

We can view the current sync windows associated with an application using argocd app get <app>:

mohitgoyal@desktop:/mnt/d/mohit/src/kind$ argocd app get guestbook-ui
Name:               guestbook-ui
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://localhost:31000/applications/guestbook-ui
Repo:               https://github.com/goyalmohit/argocd-example-apps
Target:             master
Path:               guestbook
SyncWindow:         Sync Allowed
Sync Policy:        Automated
Sync Status:        Synced to master (53e28ff)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     default    guestbook-ui  Synced  Healthy        service/guestbook-ui configured
apps   Deployment  default    guestbook-ui  Synced  Healthy        deployment.apps/guestbook-ui unchanged

Note that deployment windows are applied and managed at project level and not individual application. We can view the same in the UI by going to selected project:

Add New Sync Window

To create new window, click the button for ‘+ADD SYNC WINDOW’ and submit the details in the associated form. Once saved, we can view details of same:

Or using the CLI:

mohitgoyal@desktop:/mnt/d/mohit/src/kind$ argocd proj windows list default
ID  STATUS    KIND  SCHEDULE    DURATION  APPLICATIONS  NAMESPACES  CLUSTERS  MANUALSYNC
0   Inactive  deny  0 17 * * *  12h       *             -           -         Enabled

mohitgoyal@desktop:/mnt/d/mohit/src/kind$ argocd app get guestbook-ui
Name:               guestbook-ui
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://localhost:31000/applications/guestbook-ui
Repo:               https://github.com/goyalmohit/argocd-example-apps
Target:             master
Path:               guestbook
SyncWindow:         Sync Allowed
Assigned Windows:   deny:0 17 * * *:12h
Sync Policy:        Automated
Sync Status:        Synced to master (53e28ff)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     default    guestbook-ui  Synced  Healthy        service/guestbook-ui configured
apps   Deployment  default    guestbook-ui  Synced  Healthy        deployment.apps/guestbook-ui unchanged

These windows affect the running of both manual and automated syncs but allow an override for manual syncs which is useful if you are only interested in preventing automated syncs or if you need to temporarily override a window to perform a sync.

We can allow a manual override permanently for the window by selecting below checkbox, while defining window:

Update / Delete Sync Window

We can update or delete the sync window by selecting the associated window in the project settings. If deleting, it will ask you to confirm:

At CLI, we can delete sync window with argocd proj windows delete command.

For more details, refer here.

Sync Phases and Waves

Argo CD executes a sync operation in a number of steps. At a high-level, there are three phases pre-sync, sync and post-sync.

Within each phase you can have one or more waves, that allows you to ensure certain resources are healthy before subsequent resources are synced.

For more details, refer here.