Argo CD OutOfSync: a command-driven diagnosis and recovery path

An Argo CD application can be OutOfSync while the service is healthy. I see this most often after an incident: someone changed a replica count directly in the cluster, an admission controller added a field, or a Helm value produced a different manifest than the one Argo CD compared on the previous refresh.

That status is not a reason to immediately click Sync. It is a comparison result: the live object does not match the desired object Argo CD rendered from Git. Before I reconcile a production application, I want to know which field changed, which system owns that field, and whether Git or the cluster is wrong.

What OutOfSync tells you—and what it does not

Argo CD uses sync status to answer one narrow question: does the live Kubernetes state match the desired state it is tracking from Git? OutOfSync means at least one managed resource differs. The difference can be one annotation on one Deployment; it does not mean an entire application is unavailable.

Keep sync status separate from health status. A Deployment can be Synced and Degraded because its pods are crash-looping. It can also be OutOfSync and Healthy because a live replica count or label differs from Git while traffic continues to flow. Resyncing the first case will not fix a broken process. Treating the second as an outage can overwrite a deliberate emergency change.

Argo CD application sync statuses

Unknown deserves its own branch in the runbook. It is not confirmed drift; it means Argo CD lacks enough information to compare the desired and live objects. Start with repository credentials, the destination cluster connection, and controller errors rather than trying to force a reconciliation.

Common sources of Git-to-cluster drift

Argo CD is doing useful work when it reports a real difference. The job is to find the writer of that difference. In practice, most cases fall into a few repeatable patterns.

Manual changes bypassed Git

The fastest path to drift is an emergency edit such as kubectl edit deployment payments. Kubernetes accepts the new replica count, environment variable, or pod-disruption-budget value, but the repository remains unchanged. On the next comparison, Argo CD sees exactly what you would expect: live state and desired state disagree.

A controller can create the same result. A HorizontalPodAutoscaler owns a Deployment's replica count; an admission webhook might add labels or annotations. If Git and Argo CD are expected to own that field as well, the difference will recur after every sync. Decide which controller owns the field and encode that decision, rather than teaching operators to ignore a persistent diff.

Hooks and generated resources have a different lifecycle

Hooks are for lifecycle actions: a database migration, a smoke test, or cleanup. They may create a Job and then remove it, or produce an object that is not part of the steady-state application manifest. That is not the same lifecycle as a Deployment. Review the hook annotations, deletion policy, and ownership before retrying a sync.

If an object must persist, put it in the normal desired-state manifests. If it is intentionally ephemeral, make its hook lifecycle explicit. Repeatedly forcing a sync around a transient Job only hides the modeling problem.

The manifest render is not deterministic

GitOps depends on a simple invariant: the same inputs render the same manifests. Current timestamps, randomly generated values, and moving tags such as latest break it. Git may not change, yet the desired output changes on every render and the application never settles.

Pin image digests or immutable version tags, freeze chart versions, and pass stable template inputs. I also make nondeterminism a CI failure: render the same application twice from the same commit and compare the output. If the two renders differ, fix that before Argo CD has to explain it in production.

Sync policy and resource ownership leave an expected gap

When automated sync is disabled, Argo CD can detect a new Git revision and leave the cluster at the old revision by design. That is a pending approval, not a failed reconciler. Likewise, a ConfigMap created by another deployment script or a manually installed ServiceAccount is not automatically repaired by an Argo CD Application unless it is in the desired manifest set and under clear ownership.

Diagnose the diff before reconciling

I start at the CLI, even if the incident begins from the UI. The command output is easier to paste into an incident record, and it forces a distinction between drift and health before someone performs a destructive operation.

1. Read the application summary

argocd app get payments-prod

Record the Sync Status, Health Status, and Sync Revision. A result such as Sync Status: OutOfSync, Health Status: Healthy, and Sync Revision: 8f31c2a points to configuration drift, not necessarily a serving failure. The resource list below the summary tells you whether the difference is a Deployment, Service, ConfigMap, Job, or something else.

If the application is Degraded, inspect workload health as a separate track. Look at the affected namespace, events, and pods; a correct desired manifest does not prevent an image pull error, a failed probe, or a crashing process.

2. Identify the resource and exact field

argocd app diff payments-prod

This is the point where I stop using the word “drift” as a diagnosis. Read the diff. An image tag, replica count, label, annotation, environment variable, generated value, or admission-controller mutation each has a different owner and a different fix.

A resource marked Missing may have been deleted from the cluster or may be affected by pruning. A resource marked Unknown cannot be reliably compared or assessed yet. In either case, write down the affected resource names before changing anything.

3. Check the policy and the previous operation

argocd app get payments-prod -o yaml

Inspect spec.syncPolicy. Is automated sync enabled? Is prune enabled? Is selfHeal enabled? Then read the operation state and messages for a failed hook, admission rejection, RBAC error, or an operation that never completed. A disabled policy explains an expected pending revision; an enabled policy with a failed operation requires you to fix that error before retrying.

The application view in the UI is useful for tracing a resource tree and reviewing operation details, but it should confirm this diagnosis rather than replace it. For the behavior of individual sync flags, use the Argo CD sync-options documentation.

Reconcile with the narrowest safe action

Once the diff has an owner, the action is straightforward. If Git is correct and the change is expected, do a normal sync. If the live change is the new approved configuration, commit it to Git first. A sync operation should restore a reproducible desired state, not preserve a one-off fix that only exists in the cluster.

Standard sync

argocd app sync payments-prod
argocd app get payments-prod

Use a standard sync when the resource can be updated in place and the diff is understood. Confirm both sync and health after the operation. A successful sync that leaves the application OutOfSync usually means a controller, webhook, hook lifecycle, or nondeterministic render changed the field again; go back to the diff rather than syncing repeatedly.

Force only when replacement is the intended recovery

argocd app get payments-prod --hard-refresh
argocd app sync payments-prod --force

A hard refresh can help when the comparison is stale. Force is different: it can replace resources and interrupt workloads. I reserve it for a resource that cannot be reconciled with a normal apply, or for a deliberate replacement during a maintenance window. It is not the default response to an unclear diff.

Prune only when Git owns the full resource set

argocd app sync payments-prod --prune

Prune deletes objects that are absent from the desired manifests. That is correct only when Git is authoritative for the application boundary. A removed manifest, changed tracking label, or incorrect project boundary can otherwise make a legitimate object look orphaned. Review proposed deletions carefully, especially in shared namespaces, app-of-apps deployments, and manually managed infrastructure.

Make dependencies explicit with sync waves

For migrations, operators, and platform services, do not depend on incidental resource discovery order. Use sync-wave annotations to apply a namespace and CRDs before their controllers, then allow workloads that depend on those controllers to proceed. Verify each wave becomes healthy before moving on. That is safer than forcing an entire application because one dependent object raced the resource it needs.

Prevent the next OutOfSync event

The recurring fix is less interesting than the recurring cause. I try to remove the workflow that made direct cluster edits, mutable inputs, or ambiguous ownership possible in the first place.

• Keep Git authoritative. Do not treat kubectl apply -f as an alternate deployment path in an Argo CD-managed namespace. For an emergency override, make the corresponding Git change during the same incident and remove the manual override before closing it.
• Test renders for determinism. Pin digests and chart versions, avoid timestamps and random values in templates, and fail CI when two renders from the same commit differ.
• Use auto-sync and prune deliberately. Enable them where the application owns a stable manifest set. Keep manual sync where you still need to review a change boundary. Prune is powerful precisely because it deletes resources.
• Standardize the delivery path. The same build, render, test, and promotion path should run in every environment. That reduces direct, unreviewed writes to a cluster and supports the control objectives described in NIST SP 800-204D.

Once this becomes a multi-cluster problem, the issue is not how to run argocd app diff—it is finding the same diff across every cluster before it becomes twenty separate incidents. A fleet-management layer such as Plural is useful there: it provides a central place to see sync and health state across clusters while Git remains the desired-state source.

The safe recovery path is always the same: read the diff, assign field ownership, make Git authoritative, then use the least destructive sync operation that restores convergence.