Master GitOps principles and ArgoCD - from understanding why GitOps exists to operating multi-environment deployments, App of Apps, ApplicationSets, and progressive delivery with Argo Rollouts.
Before GitOps, deploying to Kubernetes looked like this at most Indian product companies: ``` Developer merges code | v CI pipeline builds Docker image | v Someone runs: kubectl apply -f deployment.yaml | v It works (maybe) | v Three months later: nobody remembers what is running ``` This process has several serious problems that show up at scale. **Configuration drift** — someone runs `kubectl edit deployment` to "temporarily" increase replicas during a traffic spike. They forget to update the YAML file. Now the cluster state and the Git state are different. Nobody knows which one is correct. The next deployment overwrites the fix. Production breaks on Friday night. **No audit trail** — a security incident happens. Who deployed what, when, and from which commit? Nobody knows. The only record is in someone's terminal history, if they haven't cleared it. **Manual errors** — a developer runs `kubectl apply -f prod-config.yaml` but they are in the wrong terminal window. The staging config just got applied to production. This actually happens. **Credential sprawl** — every CI/CD pipeline needs direct credentials to your Kubernetes cluster. Every developer who deploys needs `kubectl` access. The blast radius of a compromised credential is enormous. GitOps solves all of these problems with one simple idea: **Git is the only source of truth. The cluster must always match what Git says.** ---
The OpenGitOps community defines GitOps through four principles. Every tool, pattern, and practice in this module flows from these four ideas. ### Principle 1 — Declarative Everything about the system is described in files, not commands. ``` Imperative (old way): "First create the namespace, then create the deployment, then create the service, then scale to 3 replicas" Declarative (GitOps way): "Here is a YAML file describing the desired end state. Make it happen." ``` You do not tell Kubernetes HOW to get there. You describe WHAT you want. Kubernetes figures out the steps. This is how Kubernetes was designed — GitOps simply takes the same philosophy and applies it to the entire delivery pipeline. ### Principle 2 — Versioned and Immutable Every change to the desired state is a Git commit. Git commits are immutable — once made, they cannot be changed without creating a new commit. The entire history of every change to your infrastructure is preserved forever. ``` Deployment broke at 3am? git log → find the commit that changed it git revert → the system rolls back automatically Total time: 2 minutes Without GitOps: "What changed? Who changed it? When? Which kubectl command?" Total time: 30 minutes of panic ``` Rollback is a `git revert`. Not a manual process. Not a runbook. One command. ### Principle 3 — Pulled Automatically This is the most important architectural difference between GitOps and traditional CI/CD. **Traditional push model:** The CI pipeline has credentials to your Kubernetes cluster. When code merges, the pipeline runs `kubectl apply` to push changes into the cluster. The pipeline needs cluster admin access. **GitOps pull model:** A software agent (ArgoCD) runs inside your cluster. It watches the Git repository. When it detects a difference between Git and the running cluster, it pulls the change and applies it itself. Nobody outside the cluster needs credentials to deploy. ``` Push model security problem: CI pipeline → needs cluster credentials → stored as secrets If CI system is compromised → attacker has cluster access Pull model security advantage: ArgoCD inside cluster → pulls from Git → no external credentials Git repository credentials are read-only Attacker compromising CI cannot touch the cluster ``` ### Principle 4 — Continuously Reconciled The GitOps agent does not just apply changes once. It runs a continuous reconciliation loop — constantly comparing the actual cluster state to the desired state in Git. If anything diverges (manual change, hardware failure, partial deploy), ArgoCD detects it and corrects it. ``` Reconciliation loop (every ~3 minutes by default): ArgoCD reads desired state from Git | v ArgoCD reads actual state from Kubernetes API | v Are they the same? | YES | NO | | Done Fix it → apply diff to cluster | Done ``` This is what eliminates configuration drift. You cannot have a cluster that diverges from Git because ArgoCD will always bring it back. ---
Most teams transitioning to GitOps come from a push-based CI/CD world. Understanding the architectural difference prevents a lot of confusion. ``` PUSH-BASED (GitHub Actions / Jenkins style): Git commit | v CI pipeline triggers | v Pipeline runs: kubectl apply -f k8s/ | v Cluster updated Problems: * CI system needs cluster credentials (security risk) * If CI is down, no deployments happen * No continuous reconciliation — drift goes undetected * Deployment history lives in CI, not Git PULL-BASED (GitOps / ArgoCD style): Git commit | v ArgoCD detects change (polling or webhook) | v ArgoCD applies change from inside cluster | v Cluster updated Benefits: * No cluster credentials outside the cluster * Continuous reconciliation catches drift * Full deployment history in Git * ArgoCD can recover cluster from scratch using Git ``` In practice many teams use both: CI handles build, test, and image push. ArgoCD handles deployment. CI updates a manifest file in Git. ArgoCD detects the change and deploys it. ``` Best of both worlds: Code commit | v CI: build → test → push image → update image tag in Git | v ArgoCD: detects Git change → deploys new image to cluster ``` ---
ArgoCD is a Kubernetes-native application. It runs as a set of pods inside your cluster. Understanding its three core components helps you debug it when something goes wrong. ``` +------------------------------------------+ | ArgoCD Namespace | | | | +-------------+ +------------------+ | | | API Server | | Repo Server | | | | | | | | | | Web UI | | Clones Git repos | | | | CLI | | Renders manifests| | | | REST/gRPC | | Caches locally | | | +-------------+ +------------------+ | | | | +----------------------------------+ | | | Application Controller | | | | | | | | Watches all ArgoCD Applications | | | | Compares desired vs actual state | | | | Triggers sync when diff detected | | | | Runs reconciliation loop | | | +----------------------------------+ | +------------------------------------------+ | | v v Kubernetes API Git Repository (live state) (desired state) ``` **API Server** — handles all interactions with the outside world. The Web UI, the `argocd` CLI, and any CI/CD integrations talk to the API server. It handles authentication, RBAC enforcement, and exposes REST/gRPC endpoints. **Repository Server** — is the Git reader. It maintains a local cache of all connected Git repositories. When the Application Controller needs to know what the desired state should be, it asks the Repo Server to fetch and render the manifests. The Repo Server can render plain YAML, Helm charts, Kustomize overlays, and Jsonnet. **Application Controller** — is the brain of ArgoCD. It runs the reconciliation loop. For every Application resource, it continuously compares the desired state (from the Repo Server) against the actual state (from the Kubernetes API). When it detects a difference, it marks the application as `OutOfSync` and optionally triggers an automatic sync. ### The Reconciliation Loop in Detail ``` Every ~3 minutes (configurable): App Controller asks Repo Server: "What should payment-service look like?" Repo Server reads Git repo, renders manifests, returns desired state App Controller queries K8s API: "What does payment-service look like now?" App Controller compares the two If same → status: Synced If different → status: OutOfSync If selfHeal: true → automatically sync to fix it If selfHeal: false → alert engineer to review and sync manually ``` > 📌 **Remember:** The reconciliation loop polls Git by default every 3 minutes. For faster response, configure a Git webhook that calls `https://your-argocd-server/api/webhook` after every push. This triggers an immediate reconciliation instead of waiting for the next poll cycle. ---
```bash ## Create dedicated namespace kubectl create namespace argocd ## Install ArgoCD (server-side apply required for large CRDs) kubectl apply -n argocd \ --server-side \ --force-conflicts \ -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml ## Wait for all pods to be ready kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s ## Check what got created kubectl get pods -n argocd ## NAME READY STATUS ## argocd-server-xxx 1/1 Running ## argocd-repo-server-xxx 1/1 Running ## argocd-application-controller-0 1/1 Running ## argocd-dex-server-xxx 1/1 Running ## argocd-redis-xxx 1/1 Running ``` ### Accessing the ArgoCD UI ```bash ## Option 1: Port forward (for local development) kubectl port-forward svc/argocd-server -n argocd 8080:443 ## Open: https://localhost:8080 ## Option 2: Change service type to LoadBalancer (for cloud clusters) kubectl patch svc argocd-server -n argocd \ -p '{"spec": {"type": "LoadBalancer"}}' ## Get the external IP kubectl get svc argocd-server -n argocd ## Get the initial admin password kubectl get secret argocd-initial-admin-secret -n argocd \ -o jsonpath="{.data.password}" | base64 -d ## Username: admin ## Password: (the decoded value above) ``` ### Installing the ArgoCD CLI ```bash ## Linux curl -sSL -o argocd \ https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 chmod +x argocd && sudo mv argocd /usr/local/bin/ ## Login via CLI argocd login localhost:8080 --username admin --password <your-password> ## Change the default admin password argocd account update-password ## List all applications argocd app list ## Get details of a specific app argocd app get payment-service ``` ---
An ArgoCD `Application` is a Kubernetes custom resource that tells ArgoCD: "Watch this Git repository path and keep this Kubernetes namespace in sync with it." ```yaml ## payment-service-app.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: payment-service namespace: argocd ## Always in the argocd namespace finalizers: ## When this Application is deleted, also delete the K8s resources it manages - resources-finalizer.argocd.argoproj.io spec: ## Which AppProject this belongs to (security boundary — covered later) project: default ## WHERE to get the desired state from source: repoURL: https://github.com/razorpay/platform-gitops.git targetRevision: HEAD ## Latest commit on default branch path: services/payment-service/overlays/production ## WHERE to deploy destination: server: https://kubernetes.default.svc ## In-cluster namespace: production ## HOW to sync syncPolicy: automated: prune: true ## Delete K8s resources removed from Git selfHeal: true ## Revert manual changes back to Git state syncOptions: - CreateNamespace=true ## Create namespace if it doesn't exist retry: limit: 3 backoff: duration: 5s factor: 2 maxDuration: 1m ``` ```bash ## Apply the Application to the cluster kubectl apply -f payment-service-app.yaml ## OR create it using the CLI argocd app create payment-service \ --repo https://github.com/razorpay/platform-gitops.git \ --path services/payment-service/overlays/production \ --dest-server https://kubernetes.default.svc \ --dest-namespace production \ --sync-policy automated \ --auto-prune \ --self-heal ``` ### Understanding Sync Policies The sync policy is the most critical configuration decision for each application. ``` syncPolicy: automated: prune: true → delete resources removed from Git (dangerous without testing!) selfHeal: true → revert any manual changes to match Git ``` **prune: true** — if you remove a Deployment from Git, ArgoCD deletes it from the cluster. Without prune, removed resources stay running forever until someone deletes them manually. **selfHeal: true** — if someone runs `kubectl edit` and changes something, ArgoCD reverts it on the next reconciliation. This is the core of drift prevention. > 🔴 **Common Mistake:** Enabling `prune: true` without testing first. If your Git path is wrong and ArgoCD thinks there are no resources, it will delete everything in the namespace. Always test in staging with prune disabled, verify the sync is correct, then enable prune. ### Application Health States ArgoCD tracks two dimensions for every application: ``` Sync Status: Synced → cluster matches Git exactly OutOfSync → cluster differs from Git Unknown → ArgoCD cannot determine the state Health Status: Healthy → all resources are running correctly Progressing → resources are being updated Degraded → resources have failed (pod crash, etc.) Suspended → a Rollout is paused waiting for promotion Missing → resources defined in Git do not exist in cluster ``` > 💡 **Tip:** An application can be `Synced` but `Degraded`. This means the cluster matches Git exactly — but the thing Git describes is broken (for example, a crashing pod). Fix the application code and push to Git, not the ArgoCD sync status. ---
Before GitOps, deploying to Kubernetes looked like this at most Indian product companies: This process has several serio...
The OpenGitOps community defines GitOps through four principles. Every tool, pattern, and practice in this module flows ...
Most teams transitioning to GitOps come from a push-based CI/CD world. Understanding the architectural difference preven...
ArgoCD is a Kubernetes-native application. It runs as a set of pods inside your cluster. Understanding its three core co...
Accessing the ArgoCD UI Installing the ArgoCD CLI ---...
An ArgoCD Application is a Kubernetes custom resource that tells ArgoCD: "Watch this Git repository path and keep this K...
ArgoCD natively understands Kustomize and Helm. You do not need to render manifests manually — ArgoCD does it. Kustomize...
At Hotstar, Zerodha, or Swiggy scale, you might have 50+ services, each needing its own ArgoCD Application. Creating the...
App of Apps requires you to write a separate YAML file for each application. With 50 services across 3 environments, tha...
In a real company, multiple teams share one ArgoCD installation. The payments team should not be able to accidentally de...
Configuration drift is when the live cluster state diverges from the desired Git state. GitOps makes drift detection aut...
Kubernetes' built-in Deployment has one deployment strategy: rolling update. It replaces old pods with new pods graduall...
Repository structure is where most teams make mistakes. The right structure makes everything — promotions, audits, rollb...
❌ selfHeal: true with no ignoreDifferences — fighting HPA 💥 An HPA scales the payment-service to 8 replicas during peak...
When an Application shows OutOfSync: When an Application shows Sync Failed: When Argo Rollouts gets stuck: Decision tree...
This project builds a complete GitOps pipeline. You will deploy a Flask API using ArgoCD, set up multi-environment promo...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.