Build a GitOps Delivery Platform with ArgoCD

Build a multi-environment GitOps platform with ArgoCD: App of Apps, ApplicationSets, PR-based promotion and Argo Rollouts canaries.

Architecture Overview & Problem Statement

This project builds a GitOps delivery platform on Kubernetes with ArgoCD: one bootstrap command installs the platform, ApplicationSets generate an app per service and environment, changes move from staging to production through a pull request, and Argo Rollouts releases each new version as a canary that rolls back automatically when its health check fails.

The problem this platform solves

One app in one cluster is easy. A real team has many services and at least two environments, and the questions pile up. How do you add a service without hand-writing ArgoCD objects for it? How does a version move from staging to production with review? How do you avoid sending a new version to every user at once?

This platform answers each question with one pattern:

Question	Pattern	Tool
How do I avoid repeating YAML per environment?	Base plus overlays	Kustomize
How do I bootstrap everything from one place?	App of Apps	ArgoCD Application
How do I onboard a new service with no extra setup?	Generate apps from folders	ArgoCD ApplicationSet
How do I move a version to production safely?	Promotion by pull request	Git and GitHub
How do I limit the blast radius of a bad release?	Canary release with analysis	Argo Rollouts

flowchart TD A[Root
Application] --> B[Argo Rollouts] A --> C[ApplicationSet
staging] A --> D[ApplicationSet
production] C --> E[podinfo
staging] D --> F[podinfo
production] classDef purple fill:#EDE9FE,stroke:#8B5CF6,stroke-width:2px,color:#1E293B classDef blue fill:#DBEAFE,stroke:#3B82F6,stroke-width:2px,color:#1E293B classDef green fill:#D1FAE5,stroke:#10B981,stroke-width:2px,color:#1E293B classDef orange fill:#FFEDD5,stroke:#F97316,stroke-width:2px,color:#1E293B classDef red fill:#FEE2E2,stroke:#EF4444,stroke-width:2px,color:#1E293B classDef slate fill:#F1F5F9,stroke:#475569,stroke-width:2px,color:#1E293B class A purple class B blue class C,E green class D,F orange linkStyle default stroke:#475569,stroke-width:2px

You apply one root Application by hand. It installs Argo Rollouts and the two ApplicationSets. Each ApplicationSet then scans the services/ folder and creates one Application per service. Adding a folder adds a service.

Repository layout

TEXT

gitops-platform/
  bootstrap/
    root.yaml
  platform/
    argo-rollouts.yaml
    appset-staging.yaml
    appset-production.yaml
  services/
    podinfo/
      base/
        kustomization.yaml
        rollout.yaml
        services.yaml
        analysis-template.yaml
      overlays/
        staging/kustomization.yaml
        production/kustomization.yaml

What you will build

A Kustomize base with staging and production overlays for one service
A root Application that bootstraps the whole platform (App of Apps)
ApplicationSets that create staging and production apps from folders
Automated sync for staging and manual approval for production
A canary rollout with 20 percent then 60 percent steps and an automated health check, including a failing release that rolls itself back

What you need

Tool	Check it works	Install guide
Docker and kind	`kind version`	kind quick start
kubectl	`kubectl version --client`	Install kubectl
ArgoCD CLI	`argocd version --client`	Argo CD releases
Git and GitHub account	`git --version`	GitHub signup
Argo Rollouts kubectl plugin	`kubectl argo rollouts version`	Installed in Milestone 1

This is the advanced project in the ArgoCD series. You should already know what an Application, a sync and selfHeal are. If not, start with Deploy Your First App with ArgoCD and GitOps. Plan for about 4 hours. Replace YOUR_USER with your GitHub username everywhere.

Tip
Each milestone ends with a Checkpoint. This project builds up in layers, so confirm each one before the next.

Security
The canary health check here is a simple in-cluster HTTP probe so the project runs anywhere. Treat it as a learning stand-in. Real teams base the pass or fail decision on measured error rate and latency from a metrics system such as Prometheus, and any thresholds should be targets you tune from your own data.

Go deeper

Argo CD cluster bootstrapping - the App of Apps pattern in the docs
ApplicationSet overview - generators and templates
Argo Rollouts documentation - progressive delivery reference

Milestone 1: Check Your Cluster and Tools

This milestone confirms you have a cluster with ArgoCD running and installs the Argo Rollouts kubectl plugin. Everything after this assumes both are ready.

Why check first

This project bootstraps a lot at once. A broken cluster or a missing CLI would show up as a confusing error three milestones later, so spend five minutes here.

Steps

If you do not already have a cluster with ArgoCD, create one:

Bash

kind create cluster --name gitops-lab
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl wait --for=condition=ready pod --all -n argocd --timeout=300s

In a second terminal, expose the UI, then log in:

Bash

kubectl port-forward svc/argocd-server -n argocd 8080:443

Bash

kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath='{.data.password}' | base64 -d; echo
argocd login localhost:8080 --username admin --password 'PASTE_PASSWORD_HERE' --insecure

Install the Argo Rollouts plugin (the controller itself is installed by GitOps in Milestone 4):

Bash

# macOS
brew install argoproj/tap/kubectl-argo-rollouts
 
# Linux
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

Checkpoint

Bash

kubectl config current-context
kubectl get pods -n argocd
argocd account get-user-info
kubectl argo rollouts version

The context is your kind cluster, all ArgoCD pods are Running, argocd account get-user-info shows Logged In: true, and the plugin prints a version.

Common Mistake
Installing ArgoCD with plain kubectl apply. The ApplicationSet CRD exceeds the client-side annotation limit and the install fails. This platform depends on ApplicationSets, so always use --server-side --force-conflicts.

Go deeper

Argo CD getting started - install and login
Argo Rollouts kubectl plugin - commands you will use later

Milestone 2: Create the Platform Repo and Kustomize Base

This milestone creates the gitops-platform repository and writes the base manifests for one service: the part that is identical in every environment. You will use podinfo, a small public demo app, so no image build is needed.

Why a base plus overlays

Copying manifests per environment makes them drift apart. Kustomize keeps one base and lets each environment add a small overlay that patches only what differs, such as replica count or image tag. Think of the base as a template and an overlay as a few sticky notes on top.

What the base contains

File	Purpose
`rollout.yaml`	The Argo Rollouts `Rollout`, a Deployment replacement with a canary strategy
`services.yaml`	Three Services: one for users and two that Rollouts manages
`analysis-template.yaml`	The health check that decides whether a canary may continue
`kustomization.yaml`	Lists the files above

Steps

Create an empty public GitHub repo named gitops-platform, then:

Bash

mkdir gitops-platform && cd gitops-platform
mkdir -p bootstrap platform
mkdir -p services/podinfo/base
mkdir -p services/podinfo/overlays/staging services/podinfo/overlays/production

Create services/podinfo/base/rollout.yaml:

YAML

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: podinfo
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo
          ports:
            - name: http
              containerPort: 9898
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi
  strategy:
    canary:
      stableService: podinfo-stable
      canaryService: podinfo-canary
      steps:
        - setWeight: 20
        - analysis:
            templates:
              - templateName: canary-health
            args:
              - name: path
                value: /healthz
        - setWeight: 60
        - pause:
            duration: 30s
        - analysis:
            templates:
              - templateName: canary-health
            args:
              - name: path
                value: /healthz

Create services/podinfo/base/services.yaml:

YAML

apiVersion: v1
kind: Service
metadata:
  name: podinfo
spec:
  selector:
    app: podinfo
  ports:
    - name: http
      port: 9898
      targetPort: http
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo-stable
spec:
  selector:
    app: podinfo
  ports:
    - name: http
      port: 9898
      targetPort: http
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo-canary
spec:
  selector:
    app: podinfo
  ports:
    - name: http
      port: 9898
      targetPort: http

Create services/podinfo/base/analysis-template.yaml:

YAML

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-health
spec:
  args:
    - name: path
      value: /healthz
  metrics:
    - name: canary-endpoint
      interval: 10s
      count: 3
      failureLimit: 1
      provider:
        job:
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers:
                  - name: probe
                    image: curlimages/curl:8.10.1
                    command:
                      - sh
                      - -c
                      - curl -fsS http://podinfo-canary:9898{{args.path}}

Create services/podinfo/base/kustomization.yaml:

YAML

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - rollout.yaml
  - services.yaml
  - analysis-template.yaml

How to read the canary strategy

Without an ingress or service mesh, Argo Rollouts splits traffic by pod count. With 5 replicas, setWeight: 20 runs 1 canary pod and 4 stable pods, so about one request in five reaches the canary. The podinfo Service selects all pods, so it spreads users across both versions. podinfo-stable and podinfo-canary are managed by Rollouts and always point at one version each, which lets the health check target only the canary.

Checkpoint

The base is not deployable on its own yet (it has no image tag or namespace), but it must render:

Bash

kubectl kustomize services/podinfo/base | grep -E '^kind:'

TEXT

kind: Service
kind: Service
kind: Service
kind: AnalysisTemplate
kind: Rollout

Common Mistake
Hard-coding the image tag or namespace in the base. Anything that differs per environment belongs in an overlay, otherwise you are back to editing the base for every promotion.

Tip
The first time a Rollout is created it skips the canary steps and goes straight to 100 percent, because there is no previous version to compare against. Canary behaviour starts with the second release.

Go deeper

Argo Rollouts canary strategy - steps, weights and services
Argo Rollouts analysis - how AnalysisTemplates decide success
Kustomize overview - bases and overlays
podinfo endpoints - /healthz, /readyz and /status/{code}

Milestone 3: Add Staging and Production Overlays

This milestone adds the two environment overlays. Staging is small and gets new versions first. Production is larger and runs the version that has already passed staging.

Why environments track different tags

If staging and production both follow the same tag, a bad release hits both at once and there is no safety net. Each overlay pins its own image tag, so promotion becomes a deliberate act: staging moves first, and production moves only when you change its overlay.

Steps

Create services/podinfo/overlays/staging/kustomization.yaml:

YAML

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo-staging
resources:
  - ../../base
images:
  - name: ghcr.io/stefanprodan/podinfo
    newTag: 6.7.0
labels:
  - pairs:
      environment: staging
    includeSelectors: false
patches:
  - target:
      group: argoproj.io
      kind: Rollout
      name: podinfo
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 3

Create services/podinfo/overlays/production/kustomization.yaml:

YAML

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo-production
resources:
  - ../../base
images:
  - name: ghcr.io/stefanprodan/podinfo
    newTag: 6.7.0
labels:
  - pairs:
      environment: production
    includeSelectors: false
patches:
  - target:
      group: argoproj.io
      kind: Rollout
      name: podinfo
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5

Commit and push everything so far:

Bash

git init -b main
git add .
git commit -m "feat: add podinfo base and overlays"
git remote add origin https://github.com/YOUR_USER/gitops-platform.git
git push -u origin main

Checkpoint

Render both overlays and compare:

Bash

kubectl kustomize services/podinfo/overlays/staging | grep -E 'namespace:|replicas:|image:'
kubectl kustomize services/podinfo/overlays/production | grep -E 'namespace:|replicas:|image:'

Staging shows podinfo-staging, replicas: 3 and podinfo:6.7.0. Production shows podinfo-production, replicas: 5 and podinfo:6.7.0.

Common Mistake
A namespace in the overlay that does not match the Application's destination namespace. The overlay sets the namespace of every resource, so the Application destination must be the same, or ArgoCD reports resources as out of sync. Later in this project the names are generated as <service>-staging and <service>-production.

Go deeper

Kustomize images transformer - how newTag works
Kustomize patches - JSON patches, as used for replicas
Argo CD with Kustomize - ArgoCD renders these natively

Milestone 4: Bootstrap the Platform with App of Apps

This milestone creates a single root Application that installs Argo Rollouts and, later, everything else. This is the App of Apps pattern: one parent Application whose source is a folder of other Application manifests.

Why App of Apps

Without it, every Application is a separate kubectl apply that someone must remember to run. With it, you apply one file once, and from then on the platform is managed from Git. Adding or removing a file in platform/ adds or removes things in the cluster.

Steps

Create platform/argo-rollouts.yaml, an Application that installs the Argo Rollouts controller from its official Helm chart:

YAML

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-rollouts
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  project: default
  source:
    repoURL: https://argoproj.github.io/argo-helm
    chart: argo-rollouts
    targetRevision: 2.40.9
  destination:
    server: https://kubernetes.default.svc
    namespace: argo-rollouts
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Create bootstrap/root.yaml:

YAML

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_USER/gitops-platform.git
    targetRevision: main
    path: platform
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Push, then apply the root once:

Bash

git add .
git commit -m "feat: add root app and argo rollouts"
git push
 
kubectl apply -f bootstrap/root.yaml
kubectl get applications -n argocd
kubectl get pods -n argo-rollouts

Checkpoint

TEXT

NAME            SYNC STATUS   HEALTH STATUS
argo-rollouts   Synced        Healthy
platform        Synced        Healthy

The argo-rollouts namespace has a controller pod Running. This kubectl apply is the last manual command you run for the platform.

Setting	Why it is there
`finalizers: resources-finalizer...`	Deleting the root also deletes what it created, instead of orphaning it
`sync-wave: "-1"`	Asks ArgoCD to apply this Application before others in the same sync
`targetRevision: 2.40.9`	Pins the Helm chart so the install is repeatable

Common Mistake
Deleting the root Application without the finalizer. The child Applications and everything they deployed are left running with nothing managing them. Keep the finalizer on parent Applications.

Tip
If chart version 2.40.9 is not found, open the Argo Rollouts chart and pick a current version. Sync waves order resources, but a parent only waits for a child Application if ArgoCD has a health check for it, so later steps also use retries instead of relying on order.

Go deeper

Cluster bootstrapping - the App of Apps pattern
Sync waves - controlling apply order
Argo Rollouts installation - other install methods
argo-helm chart - chart values and versions

Milestone 5: Generate Service Apps with ApplicationSets

This milestone adds two ApplicationSets that scan the services/ folder and create one Application per service per environment. Adding a new service folder will deploy it with no extra ArgoCD setup.

Why ApplicationSets on top of App of Apps

App of Apps needs one Application file per service per environment. With 10 services and 2 environments that is 20 files to keep in sync. An ApplicationSet is a template plus a generator. The Git directory generator finds every folder matching a pattern and stamps out one Application for each match.

Steps

Create platform/appset-staging.yaml. Staging syncs automatically:

YAML

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services-staging
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/YOUR_USER/gitops-platform.git
        revision: main
        directories:
          - path: services/*/overlays/staging
  template:
    metadata:
      name: '{{ index .path.segments 1 }}-staging'
      labels:
        environment: staging
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: https://github.com/YOUR_USER/gitops-platform.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ index .path.segments 1 }}-staging'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - SkipDryRunOnMissingResource=true
        retry:
          limit: 5
          backoff:
            duration: 10s
            factor: 2
            maxDuration: 2m

Create platform/appset-production.yaml. Production has no automated block, so a human must approve every sync:

YAML

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services-production
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/YOUR_USER/gitops-platform.git
        revision: main
        directories:
          - path: services/*/overlays/production
  template:
    metadata:
      name: '{{ index .path.segments 1 }}-production'
      labels:
        environment: production
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: https://github.com/YOUR_USER/gitops-platform.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ index .path.segments 1 }}-production'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - SkipDryRunOnMissingResource=true
        retry:
          limit: 5
          backoff:
            duration: 10s
            factor: 2
            maxDuration: 2m

Push and let the root pick them up:

Bash

git add platform/
git commit -m "feat: add applicationsets for staging and production"
git push
 
argocd app get platform --refresh
kubectl get applications -n argocd

Sync production for the first time (it waits for approval):

Bash

argocd app sync podinfo-production
argocd app wait podinfo-production --health

Now prove the generator works. Add a second service by copying the folder and changing only its namespace:

Bash

cp -r services/podinfo services/hello
sed -i.bak 's/namespace: podinfo-/namespace: hello-/' services/hello/overlays/*/kustomization.yaml
rm services/hello/overlays/*/kustomization.yaml.bak
 
git add services/hello
git commit -m "feat: add hello service"
git push
argocd app get platform --refresh
kubectl get applications -n argocd

Checkpoint

TEXT

NAME                  SYNC STATUS   HEALTH STATUS
argo-rollouts         Synced        Healthy
hello-production      OutOfSync     Missing
hello-staging         Synced        Healthy
platform              Synced        Healthy
podinfo-production    Synced        Healthy
podinfo-staging       Synced        Healthy

hello-staging appeared without you writing an Application for it, and hello-production waits for a manual sync. You can leave hello in place or remove it with git rm -r services/hello; ArgoCD prunes its apps.

Common Mistake
Naming generated Applications with path.basename. For services/podinfo/overlays/staging the basename is staging, so every service would get the same name and collide. Use the path segment that holds the service name: {{ index .path.segments 1 }}.

Common Mistake
Giving production an automated sync policy. A merged PR that removes a NetworkPolicy would be applied instantly with no human review of the diff. Leave production manual until you have the guardrails and monitoring to trust automation.

Tip
SkipDryRunOnMissingResource and retry let a service app recover if the Rollouts CRDs are not installed yet when it first syncs, which can happen during first bootstrap.

Go deeper

Git generator - directory and file generators
Go template mode - the .path.segments syntax used here
Sync options - SkipDryRunOnMissingResource and more

Milestone 6: Promote from Staging to Production

This milestone moves a new version through the environments: auto-deploy to staging by committing to main, then promote to production through a pull request and a manual sync. Promotion means moving a version that has proven itself in one environment to the next.

Why promotion is a pull request

Production is the environment you most want a second pair of eyes on. A pull request that changes one line in the production overlay gives you a review, an approval record, a diff, and an instant rollback by reverting the PR.

flowchart TD subgraph S[Staging] A[Edit
staging tag] --> B[Auto sync] B --> C[Verify
healthy] end subgraph P[Production] D[Open
pull request] --> E[Review
and merge] E --> F[Manual sync] end C --> D style S fill:#F1F5F9,stroke:#10B981,stroke-width:2px,color:#1E293B style P fill:#F1F5F9,stroke:#F97316,stroke-width:2px,color:#1E293B classDef purple fill:#EDE9FE,stroke:#8B5CF6,stroke-width:2px,color:#1E293B classDef blue fill:#DBEAFE,stroke:#3B82F6,stroke-width:2px,color:#1E293B classDef green fill:#D1FAE5,stroke:#10B981,stroke-width:2px,color:#1E293B classDef orange fill:#FFEDD5,stroke:#F97316,stroke-width:2px,color:#1E293B classDef red fill:#FEE2E2,stroke:#EF4444,stroke-width:2px,color:#1E293B classDef slate fill:#F1F5F9,stroke:#475569,stroke-width:2px,color:#1E293B class A,B,C green class D,E,F orange linkStyle default stroke:#475569,stroke-width:2px

Steps

Deploy 6.7.1 to staging. Edit services/podinfo/overlays/staging/kustomization.yaml:

YAML

images:
  - name: ghcr.io/stefanprodan/podinfo
    newTag: 6.7.1

Bash

git add services/podinfo/overlays/staging/kustomization.yaml
git commit -m "deploy: podinfo 6.7.1 to staging"
git push
 
argocd app get podinfo-staging --refresh
kubectl argo rollouts get rollout podinfo -n podinfo-staging --watch

A canary rollout starts in staging. You will examine it closely in the next milestone. Wait until it reports Healthy (about two minutes), then promote to production through a branch and a pull request:

Bash

git switch -c promote-podinfo-6.7.1
# Edit services/podinfo/overlays/production/kustomization.yaml: newTag 6.7.0 -> 6.7.1
git add services/podinfo/overlays/production/kustomization.yaml
git commit -m "deploy: promote podinfo 6.7.1 to production"
git push -u origin HEAD

Open a pull request in GitHub (or run gh pr create --fill), review the one-line diff, and merge it. Then review and sync in ArgoCD:

Bash

git switch main && git pull
argocd app get podinfo-production --refresh
argocd app diff podinfo-production
argocd app sync podinfo-production
kubectl argo rollouts get rollout podinfo -n podinfo-production --watch

Checkpoint

Bash

kubectl get rollout podinfo -n podinfo-staging \
  -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
kubectl get rollout podinfo -n podinfo-production \
  -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

Both print ghcr.io/stefanprodan/podinfo:6.7.1, and argocd app diff showed only the image line changing before you synced production.

Common Mistake
Merging the production change before checking staging. Promotion only protects you if staging was actually healthy on the new version first. Make "staging is Healthy" a rule before anyone opens the production PR.

Tip
Protect main with required reviews so the production overlay cannot change without approval. See the branch protection guide below.

Go deeper

About protected branches - require reviews on main
Argo CD diffing - how argocd app diff compares Git and the cluster
GitHub: creating a pull request - the review flow

Milestone 7: Run a Canary Release with Automated Analysis

This milestone watches a healthy canary complete, then ships a release whose health check fails and watches Argo Rollouts abort it. A canary sends a new version to a small share of traffic first and only continues if checks pass.

Why canaries

A normal Deployment replaces every pod with the new version. If the new version is broken, everyone hits it. A canary limits the blast radius: with 3 staging replicas, setWeight: 20 rounds up to 1 canary pod. Because Kubernetes pod counts are whole numbers, the real traffic share is about 33 percent here, and the smaller your replica count the coarser the split.

flowchart TD A[20 percent
canary] --> B{Health
check} B -->|pass| C[60 percent
canary] C --> D{Pause, then
check} D -->|pass| E[Fully
promoted] B -->|fail| F[Abort and
roll back] D -->|fail| F classDef purple fill:#EDE9FE,stroke:#8B5CF6,stroke-width:2px,color:#1E293B classDef blue fill:#DBEAFE,stroke:#3B82F6,stroke-width:2px,color:#1E293B classDef green fill:#D1FAE5,stroke:#10B981,stroke-width:2px,color:#1E293B classDef orange fill:#FFEDD5,stroke:#F97316,stroke-width:2px,color:#1E293B classDef red fill:#FEE2E2,stroke:#EF4444,stroke-width:2px,color:#1E293B classDef slate fill:#F1F5F9,stroke:#475569,stroke-width:2px,color:#1E293B class A,C blue class B,D purple class E green class F red linkStyle default stroke:#475569,stroke-width:2px

Part 1: a healthy canary

Change the greeting so the new version is visible. In the staging overlay, add this patch under patches::

YAML

  - target:
      group: argoproj.io
      kind: Rollout
      name: podinfo
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/env
        value:
          - name: PODINFO_UI_MESSAGE
            value: "canary demo"

Bash

git add services/podinfo/overlays/staging/kustomization.yaml
git commit -m "deploy: staging greeting for canary demo"
git push
argocd app get podinfo-staging --refresh

Watch it step through, and in another terminal watch the traffic mix from inside the cluster:

Bash

kubectl argo rollouts get rollout podinfo -n podinfo-staging --watch

Bash

kubectl run traffic --rm -it --restart=Never -n podinfo-staging \
  --image=curlimages/curl:8.10.1 -- sh -c \
  'for i in $(seq 1 30); do curl -s http://podinfo:9898/ | grep "\"message\""; sleep 1; done'

You will see a mix of the old and new greeting while the canary is live, then only canary demo once it is fully promoted.

Part 2: a failing canary rolls itself back

Make the next release fail its health check on purpose. Change the greeting value to bad release, and add a second patch that points the health check at an endpoint that returns HTTP 500:

YAML

  - target:
      group: argoproj.io
      kind: Rollout
      name: podinfo
    patch: |-
      - op: replace
        path: /spec/strategy/canary/steps/1/analysis/args/0/value
        value: /status/500

Bash

git add services/podinfo/overlays/staging/kustomization.yaml
git commit -m "deploy: failing canary demo"
git push
argocd app get podinfo-staging --refresh
kubectl argo rollouts get rollout podinfo -n podinfo-staging --watch

The first analysis fails, the rollout aborts, and traffic returns to the stable version. Inspect why:

Bash

kubectl get analysisrun -n podinfo-staging
kubectl describe analysisrun -n podinfo-staging | tail -20

Then clean up with a normal GitOps rollback:

Bash

git revert HEAD --no-edit
git push
argocd app get podinfo-staging --refresh
kubectl argo rollouts status podinfo -n podinfo-staging

Checkpoint

During Part 1 you saw both greetings, then only canary demo
During Part 2 the rollout status became Degraded (aborted), the AnalysisRun shows Failed, and users never saw the bad release greeting on more than the single canary pod
After the revert, the rollout is Healthy again

Command	What it does
`kubectl argo rollouts get rollout NAME -n NS --watch`	Live view of steps, pods and analysis
`kubectl argo rollouts promote NAME -n NS`	Skip a pause and continue the rollout
`kubectl argo rollouts abort NAME -n NS`	Stop the release and return to stable
`kubectl argo rollouts retry rollout NAME -n NS`	Try an aborted rollout again
`kubectl argo rollouts dashboard`	Local web UI for rollouts

Common Mistake
Expecting exact traffic percentages from a small number of replicas. With pod-count based splitting, 10 percent of 2 pods still means 1 pod, which is 50 percent. For precise weights use a traffic router such as an ingress controller or service mesh, and enough replicas that each step means something.

Common Mistake
Rolling forward by editing the live Rollout with kubectl after an abort. ArgoCD self-heals it back to Git. Fix problems by committing to Git.

Tip
This job based check only tests that an endpoint answers. In production, point the AnalysisTemplate at a metrics provider such as Prometheus and base the pass or fail on error rate and latency measured against your own service targets.

Go deeper

Job metric provider - the provider used in this project
Prometheus metric provider - the production-style upgrade
Rollouts traffic management - exact weights with ingress or a mesh
Argo Rollouts canary docs - all step types, including manual gates

Validation & Testing

This final step confirms every layer of the platform works and gives you a checklist you can show in a portfolio or interview.

Verification commands

Bash

# 1. The root and both ApplicationSets exist and are healthy
kubectl get applications -n argocd
kubectl get applicationsets -n argocd
 
# 2. Staging syncs automatically, production does not
kubectl get application podinfo-staging -n argocd \
  -o jsonpath='{.spec.syncPolicy.automated}'; echo
kubectl get application podinfo-production -n argocd \
  -o jsonpath='{.spec.syncPolicy.automated}'; echo
 
# 3. Both environments run the promoted version
kubectl get rollout podinfo -n podinfo-staging \
  -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
kubectl get rollout podinfo -n podinfo-production \
  -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
 
# 4. Rollouts controller is running
kubectl get pods -n argo-rollouts
 
# 5. Rollouts are healthy
kubectl argo rollouts list rollouts -A

Expected: platform, argo-rollouts, and the generated apps are Synced; services-staging and services-production are listed; the staging policy prints {"prune":true,"selfHeal":true} and production prints an empty line; both rollouts show the same image tag; and every rollout is Healthy.

Troubleshooting

Symptom	Likely cause	Fix
`no matches for kind "Rollout"`	Rollouts CRDs not installed yet	Wait for `argo-rollouts` to sync; the `retry` setting will recover
ApplicationSet creates no apps	Folder does not match the pattern	Confirm `services/NAME/overlays/staging/kustomization.yaml` exists, then refresh `platform`
Apps named `staging` or `production` collide	Used `path.basename` in the template	Use `{{ index .path.segments 1 }}`
App `OutOfSync` on namespace	Overlay namespace differs from destination	Make the overlay `namespace` equal `<service>-<env>`
Analysis stuck or failing for no reason	Probe cannot reach `podinfo-canary`	Run `kubectl get svc -n podinfo-staging` and check the AnalysisRun logs
Chart version not found	Old pinned chart version	Pick a current version from the argo-helm chart page

Clean up

Bash

kubectl delete -f bootstrap/root.yaml
kind delete cluster --name gitops-lab

What to try next

Replace the job health check with a Prometheus query and tune thresholds against your own data
Add an ingress controller and switch the Rollout to weight-based traffic routing
Restrict the platform with Argo CD Projects so each team can deploy only to its own namespaces
Add a third environment, such as dev, by adding one more overlay and ApplicationSet

Go deeper

Argo CD documentation home - full reference
Argo Rollouts documentation home - full reference
OpenGitOps - vendor-neutral GitOps principles

Found a mistake, an outdated step or a broken link?

Build a GitOps Delivery Platform with ArgoCD

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem this platform solves

Repository layout

What you will build

What you need

Go deeper

Milestone 1: Check Your Cluster and Tools

Why check first

Steps

Checkpoint

Go deeper

Milestone 2: Create the Platform Repo and Kustomize Base

Why a base plus overlays

What the base contains

Steps

How to read the canary strategy

Checkpoint

Go deeper

Milestone 3: Add Staging and Production Overlays

Why environments track different tags

Steps

Checkpoint

Go deeper

Milestone 4: Bootstrap the Platform with App of Apps

Why App of Apps

Steps

Checkpoint

Go deeper

Milestone 5: Generate Service Apps with ApplicationSets

Why ApplicationSets on top of App of Apps

Steps

Checkpoint

Go deeper

Milestone 6: Promote from Staging to Production

Why promotion is a pull request

Steps

Checkpoint

Go deeper

Milestone 7: Run a Canary Release with Automated Analysis

Why canaries

Part 1: a healthy canary

Part 2: a failing canary rolls itself back

Checkpoint

Go deeper

Validation & Testing

Verification commands

Troubleshooting

Clean up

What to try next

Go deeper