Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It follows the GitOps pattern of using git repositories as the source of truth for defining the desired application state. With ArgoCD, application deployments can be automated and updates to application can be made at the simple git commit events without the need of any complicated Continuous Integration and/or Deployment Pipelines.
This is our third post in the series of blog post on deploying and managing application with Kubernetes and Argo CD. In the first two posts, we discussed how to install and setup Argo CD, how to deploy application, view and manage application health and manage application lifecycle. You can find the series index here.
For the purpose of this post, we’ll create a multi node Kubernetes cluster with Kind (created with Docker and WSL2 backend).
Understand Various Sync Options for Applications
As we have noted down earlier, there are quite a few sync options available for the application manifests. We have first observed the same, while we created the application in the Argo CD workspace. It asked us if we want to opt for Manual or Automatic sync policy:
As indicated by name, Manual means the sync operations will be performed only when a human operator will click sync button.
Automated Sync Policy
Argo CD has the ability to automatically sync an application when it detects differences between the desired manifests in Git, and the live state in the cluster. A benefit of automatic sync is that CI/CD pipelines no longer need direct access to the Argo CD API server to perform the deployment. Instead, the developers only need to make commit to tracked branches/tags in the associated git repository.
We can choose to select this policy when creating application or edit it any point in application lifecycle by modifying it. We can also choose to apply automated policy either with UI or CLI.
argocd app set <APPNAME> --sync-policy automated
If we have selected Automated sync policy, it will asks us further details on resource pruning and self-healing capabilities:
By default (and as a safety mechanism), automated sync will not delete resources when Argo CD detects the resource is no longer defined in Git. To prune the resources, a manual sync can always be performed (with pruning checked). Pruning can also be enabled to happen automatically as part of the automated sync, by selecting the checkbox for the ‘PRUNE RESOURCES’:
From Argo CD CLI:
argocd app set <APPNAME> --auto-prune
By default, changes that are made to the live cluster will not trigger automated sync. So even after we manually edited the service type for guestbook-ui from
NodePort, it has not reflected and our application status was marked as synced.
This presents an anomaly, where human operators can login to the Kubernetes clusters and make changes to the applications deployed, either intentionally or un-intentionally. This also presents a potential risks to the application health, as operators cannot rely on the resource state of application presented in the Argo CD and make wrong assumptions. To prevent such issues, we can mark the checkbox for ‘SELF HEAL’:
To enable automatic sync with CLI, run:
argocd app set <APPNAME> --self-heal
Note that if
selfHeal flag is set to true then sync will be attempted again after self heal timeout (5 seconds by default) which is controlled by
--self-heal-timeout-seconds flag of
Once we enable selfHeal flag, it will look and nullify the deviation from the desired state on its own. In our case, it will reset the service type for guestbook-ui to the
NAME READY STATUS RESTARTS AGE pod/guestbook-ui-85985d774c-69nrr 1/1 Running 0 152m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/guestbook-ui ClusterIP 10.96.163.151 <none> 80/TCP 152m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/guestbook-ui 1/1 1 1 152m NAME DESIRED CURRENT READY AGE replicaset.apps/guestbook-ui-85985d774c 1 1 1 152m
Perform Selective Sync
By default, the auto sync applies to every option in the application. For repositories containing multiple manifests, this may take a long time and puts undue pressure on the api server. We can choose to turn on selective sync option, which will sync only out-of-sync resources.
syncPolicy: syncOptions: - ApplyOutOfSyncOnly=true
- Via Argo CD CLI:
$ argocd app set guestbook --sync-option ApplyOutOfSyncOnly=true
- Selective sync while performing sync manually:
There are other sync options such as auto-create namespace, replace, skip schema validation etc. most of which perform the tasks associated with their name. These are not part of automated policies, as you may want to perform them during certain scenarios or to meet specific requirements and not at all times.
For example, by default, Argo CD will not try to create a Kubernetes namespace, if it is not present. This may result in deployment errors. To prevent such errors, we can use auto-create namespace feature. However, this may also let developers create namespaces on the fly by mentioning them in the application manifests, which may or may not be what is desired by Kubernetes cluster operators for application lifecycle management.
Again by default, Argo CD executes
kubectl apply operation to apply the configuration stored in Git. In some cases
kubectl apply is not suitable. So this feature lets you replace Kubernetes resources with the new manifest definitions defined in the git repo. However, this may result in un-wanted or un-intended downtime for the application depending on the underlying resource.
Defining Sync Windows
You may want to restrict sync based on the certain windows of time meeting specific criteria. For example, you may deny to perform sync operations only on weekdays as you have no one in office during weekends monitoring application health. Alternatively, you may perform sync operations only on weekends as you do not want any application health change for your customers on weekdays. You may want to disallow any sync for specific days when you have high application usage like cyber monday, etc.
Sync windows are configurable windows of time where syncs will either be blocked or allowed. These are defined by a kind, which can be either
schedule in cron format and a duration along with one or more of either
clusters. Wildcards are supported.
View Sync Window
We can view the current sync windows associated with an application using
argocd app get <app>:
mohitgoyal@desktop:/mnt/d/mohit/src/kind$ argocd app get guestbook-ui Name: guestbook-ui Project: default Server: https://kubernetes.default.svc Namespace: default URL: https://localhost:31000/applications/guestbook-ui Repo: https://github.com/goyalmohit/argocd-example-apps Target: master Path: guestbook SyncWindow: Sync Allowed Sync Policy: Automated Sync Status: Synced to master (53e28ff) Health Status: Healthy GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE Service default guestbook-ui Synced Healthy service/guestbook-ui configured apps Deployment default guestbook-ui Synced Healthy deployment.apps/guestbook-ui unchanged
Note that deployment windows are applied and managed at project level and not individual application. We can view the same in the UI by going to selected project:
Add New Sync Window
To create new window, click the button for ‘+ADD SYNC WINDOW’ and submit the details in the associated form. Once saved, we can view details of same:
Or using the CLI:
mohitgoyal@desktop:/mnt/d/mohit/src/kind$ argocd proj windows list default ID STATUS KIND SCHEDULE DURATION APPLICATIONS NAMESPACES CLUSTERS MANUALSYNC 0 Inactive deny 0 17 * * * 12h * - - Enabled mohitgoyal@desktop:/mnt/d/mohit/src/kind$ argocd app get guestbook-ui Name: guestbook-ui Project: default Server: https://kubernetes.default.svc Namespace: default URL: https://localhost:31000/applications/guestbook-ui Repo: https://github.com/goyalmohit/argocd-example-apps Target: master Path: guestbook SyncWindow: Sync Allowed Assigned Windows: deny:0 17 * * *:12h Sync Policy: Automated Sync Status: Synced to master (53e28ff) Health Status: Healthy GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE Service default guestbook-ui Synced Healthy service/guestbook-ui configured apps Deployment default guestbook-ui Synced Healthy deployment.apps/guestbook-ui unchanged
These windows affect the running of both manual and automated syncs but allow an override for manual syncs which is useful if you are only interested in preventing automated syncs or if you need to temporarily override a window to perform a sync.
We can allow a manual override permanently for the window by selecting below checkbox, while defining window:
Update / Delete Sync Window
We can update or delete the sync window by selecting the associated window in the project settings. If deleting, it will ask you to confirm:
At CLI, we can delete sync window with
argocd proj windows delete command.
For more details, refer here.
Sync Phases and Waves
Argo CD executes a sync operation in a number of steps. At a high-level, there are three phases pre-sync, sync and post-sync.
Within each phase you can have one or more waves, that allows you to ensure certain resources are healthy before subsequent resources are synced.
For more details, refer here.