GitOps and Configuration Management Interview Questions

GitOps is a way of running infrastructure and applications where Git holds the desired state, and an automated agent makes the real system match it. Instead of an engineer running kubectl apply or helm upgrade from a laptop, they open a pull request. After review and merge, the agent (Argo CD or Flux) applies the change.

The difference is who acts and where the truth lives. With manual kubectl apply, the truth is "whatever the last person ran", and nobody can easily tell what changed or why. With GitOps, the truth is the Git history, so you get review, an audit trail, easy rollback with git revert, and automatic correction when the cluster is changed by hand.

The common wrong answer is "GitOps is just CI/CD that uses Git". Almost every pipeline uses Git. What makes it GitOps is that an agent continuously reconciles the cluster to Git, not just once when a pipeline runs.

The four principles from the OpenGitOps project are:

1. Declarative: the system is described as the desired end state, not as a list of steps to run.
2. Versioned and immutable: the desired state is stored so that every change is recorded and history cannot be silently rewritten. Git is the usual choice.
3. Pulled automatically: software agents pull the desired state from the source, instead of something pushing it in.
4. Continuously reconciled: agents keep comparing actual state with desired state and fix any difference.

In an interview, link each principle to a real benefit. Declarative gives repeatability, versioned gives audit and rollback, pulled gives better security, and continuous reconciliation gives drift correction.

OpenGitOps principles

A reconciliation loop is a control loop that repeats forever. It reads the desired state from Git, reads the live state from the cluster, compares them, and applies changes to remove any difference. Kubernetes controllers work the same way, which is why GitOps tools fit Kubernetes so naturally.

Argo CD checks Git about every three minutes by default, or immediately when a webhook fires. Flux uses the interval field on each resource. A shorter interval means faster convergence but more load on Git.

Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. It watches a Git repository and keeps a cluster matching it.

Its main components are:

• API server: serves the web UI, CLI and API, and handles login and RBAC.
• Repo server: clones Git and renders manifests from plain YAML, Helm or Kustomize.
• Application controller: compares live state with the rendered desired state and runs syncs.
• Redis: a cache that speeds up the other components.
• Dex (optional): connects single sign-on providers.

By default Argo CD polls Git about every three minutes. You can configure a webhook from GitHub or GitLab so it refreshes immediately after a push.

Argo CD documentation

An Application is a custom resource that links a source (a Git repo, path and revision, or a Helm chart) to a destination (a cluster and namespace). It is the basic unit Argo CD manages.

Because an Application is itself a Kubernetes resource, you can store it in Git too. That is what makes patterns like App of Apps possible.

Sync status answers "does the live state match Git?" and is either Synced or OutOfSync. Health status answers "is it actually working?" and can be Healthy, Progressing, Degraded, Suspended or Missing.

They are independent. An app can be Synced but Degraded, for example when the manifests were applied correctly but the pods crash loop. It can also be OutOfSync but Healthy, for example when someone scaled a Deployment by hand and everything is running fine.

A common mistake is to see a green "Synced" and assume the release is good. Always check health too, and ideally add a PostSync smoke test.

Flux is a CNCF GitOps toolkit made of small, focused controllers, each managing its own custom resources:

• source-controller: fetches Git, Helm and OCI sources.
• kustomize-controller: applies Kustomizations.
• helm-controller: manages HelmReleases.
• notification-controller: handles alerts and incoming webhooks.
• image-reflector and image-automation controllers (optional): scan registries and write new tags to Git.

Everything is configured with Kubernetes resources and the flux CLI. Flux has no built-in UI, which keeps it light, but teams that want a dashboard add a separate UI.

Flux documentation

Helm is the package manager for Kubernetes. It bundles manifests into a reusable package and lets you customize it with values.

• A chart is the package: templates plus default values and metadata.
• A release is one installed instance of a chart in a cluster. You can install the same chart many times, each as a separate release with its own name.
• A repository is a place that hosts charts. It can be an HTTP index or, increasingly, an OCI registry.

A good way to explain it: the chart is the recipe, the values are your choices, and the release is the meal you cooked.

Helm documentation

A chart is a folder with a standard layout:

Chart.yaml has two version fields that people confuse. version is the version of the chart itself, and appVersion is the version of the application it deploys. Change version whenever the chart changes, even if the app does not.

Ansible is an automation tool for configuring servers, deploying software and running repeatable tasks. You describe the desired state in YAML playbooks, and Ansible makes the machines match it.

It is agentless because it needs no software installed on the managed machines. The control node connects over SSH (or WinRM for Windows), copies a small module, runs it and removes it. That makes it easy to start with, and there is no agent to patch or keep running. The cost is that it is push-based, so it only acts when you run it. Ansible does not watch for drift on its own the way Argo CD does.

Ansible documentation

An inventory is the list of machines Ansible manages, organised into groups such as web and db, with variables attached to hosts or groups.

A static inventory is a file (INI or YAML) that you edit by hand. It is fine for a small, stable set of servers.

A dynamic inventory is built at run time from a source of truth, using an inventory plugin such as aws_ec2, azure_rm or gcp_compute. It discovers instances by tags, so autoscaled and short-lived machines are always included. In cloud environments, dynamic inventory avoids the stale-list problem, where a host was added or removed and nobody updated the file.

Idempotent means that running the same playbook once or ten times gives the same end result, and changes nothing after the first run. A task like "ensure nginx is installed" checks the current state and only acts if it is wrong. It reports ok when nothing changed and changed when it acted.

This matters because you can safely re-run playbooks to fix drift, and a second run is a quick check that the machine is correct.

The trap is command and shell. They run whatever you give them every time, so they always report changed. Prefer a purpose-built module (package, service, file, template). When you must use a command, add creates:, removes: or changed_when: so Ansible can tell whether work is needed.

In push-based deployment, the CI pipeline runs kubectl apply or helm upgrade against the cluster at the end of a build. In pull-based deployment, an agent running inside the cluster fetches the desired state from Git and applies it itself.

Pull is safer for two reasons. First, the cluster credentials stay inside the cluster. In push mode your CI system needs an admin-level kubeconfig, so anyone who compromises CI can reach production. In pull mode the cluster only needs read access to Git. Second, a pull agent keeps checking forever, so it detects and repairs drift. A push pipeline only acts when it runs, and it never notices that someone edited production an hour later.

Pull has trade-offs too. Feedback is slower (you wait for the next sync, unless you use a webhook), and errors show up in the agent, not in your pipeline log, so you need good notifications.

Configuration drift is when the live system no longer matches the declared state. A typical cause is someone running kubectl edit deployment in production to fix something quickly, and never writing it down.

GitOps tools detect drift by diffing live objects against what Git renders. Argo CD then shows the app as OutOfSync. You choose the response. You can only report it, or you can heal it automatically (selfHeal in Argo CD, normal reconciliation in Flux), which overwrites the manual change.

The trap is a field that something else legitimately owns, for example spec.replicas controlled by a HorizontalPodAutoscaler. The tool sees a difference every time and either fights the autoscaler or shows OutOfSync forever. You solve that by telling the tool to ignore that field, not by turning drift detection off.

There are three main reasons. A config change such as raising replicas should not trigger a full application build. The config repo gives a clean audit trail of what was deployed and when, without noise from feature commits. And access control is simpler: developers push code, while only the pipeline or release owners write to the config repo.

The usual flow is that CI builds and tests the code, pushes an image, and then commits the new image tag to the config repo. The GitOps agent sees that commit and deploys it.

Small teams sometimes keep manifests in the app repo for simplicity, and that works. It starts to hurt when build history and deploy history get mixed together.

Neither is always right. A monorepo is easy to search, lets one pull request change several services, and needs only one set of permissions. A repo per team or service gives strong isolation, a smaller blast radius and clearer ownership, but costs more to manage and makes cross-service changes harder.

A common middle path is one platform repo, owned by the platform team, for cluster add-ons such as ingress, monitoring and cert-manager, plus one config repo per product team for their apps. Choose based on team size, who needs write access to what, and how often changes span services. A good answer explains the reasoning and does not just name a favourite.

A directory per environment on one main branch is the more recommended approach. Folders such as overlays/dev, overlays/staging and overlays/prod let you see the differences side by side, and promotion is an ordinary pull request that changes a version in one folder.

Branch per environment (dev, staging, prod branches) looks tidy at first, but branches drift apart. Promotion means merging or cherry-picking, merge conflicts appear in files that should be different per environment, and it is hard to answer "what is different between staging and prod?".

The exception is when a tool or compliance rule needs a protected release branch. Even then, keep the environment differences in folders.

Promotion is a Git change. Whatever runs in dev is promoted by changing the image tag or chart version in the next environment's folder.

There are three common ways to do it. A pipeline step opens the pull request. An image automation tool (Flux image automation or Argo CD Image Updater) writes the new tag. Or a promotion tool such as Kargo manages stages and gates. Dev can usually auto-merge, staging should wait for automated tests, and prod normally needs a human approval.

The key point is that you promote the same immutable artifact (the same image digest or tag) through every environment. You do not rebuild per environment.

A loop happens when the commit that CI writes triggers CI again. You avoid it by keeping config in a separate repo from the code, so the tag-bump commit never touches the app repo. If both live in one repo, use path filters so changes under deploy/ do not trigger the build, or add [skip ci] to the bot's commit message. Use a dedicated bot account so these commits are easy to spot.

Race conditions happen when two pipelines try to push to the same config branch at once and one push is rejected. The fix is to fetch, rebase and retry the push a few times, or to open a pull request per change and let the merge queue order them.

For production, many teams do not let CI push directly. CI opens a pull request and a person or policy check approves it.

They are three separate switches:

• automated: Argo CD syncs by itself whenever Git changes. Without it, a person must press Sync.
• prune: resources that were removed from Git are deleted from the cluster. Without it, orphaned objects stay forever.
• selfHeal: if someone changes a live object by hand, Argo CD reverts it to match Git. Without it, only Git changes trigger a sync, and manual edits just show as OutOfSync.

For trusted production workloads you usually enable all three. Be careful with prune when you first onboard an existing namespace, because it can delete resources that were created outside Git. Run a manual sync and review the diff first.

Both control the order of a sync, which normally applies everything together.

Sync waves use the annotation argocd.argoproj.io/sync-wave: "N". Lower numbers go first, and Argo CD waits for each wave to be healthy before starting the next. Typical use: a Namespace or CRD in wave 0, the operator in wave 1, and the resources that need that CRD in wave 2.

Hooks run Jobs at set phases: PreSync, Sync, PostSync and SyncFail. Typical use: a database migration in PreSync and a smoke test in PostSync.

The trap is relying on waves for everything. If your application needs strict startup ordering, it is more robust to make it retry until its dependency is ready, because waves only help during syncs.

Argo CD sync waves

App of Apps is a parent Application whose source folder contains other Application manifests. Syncing the parent creates all the child apps, so you can bootstrap a whole cluster (ingress, monitoring, cert-manager, then team apps) from one entry point.

It is simple and needs no extra feature. The downside is that the child manifests are hand-written YAML, so it does not scale well when you have many clusters or many near-identical apps. That is the problem ApplicationSet solves, because it generates the Applications from a template.

An ApplicationSet generates many Applications from one template and a generator. Generators include list (fixed elements), cluster (one app per registered cluster), git (one app per directory or file in a repo), matrix (combine two generators) and pullRequest (a preview environment per pull request).

Use it for multi-cluster rollouts and multi-tenant setups, so you maintain one template instead of dozens of near-copies.

ApplicationSet docs

An AppProject groups Applications and puts limits on them. A project defines which source repos are allowed, which destination clusters and namespaces are allowed, which resource kinds can be deployed, and which roles can act on its apps.

For example, a team project can be limited to its own namespaces and blocked from creating cluster-scoped objects such as ClusterRole. That stops one team from deploying into another team's namespace, or from escalating its own permissions through a manifest. It is the main multi-tenancy control in Argo CD, together with RBAC policy.

There are two ways, and one is better.

The first is the UI or argocd app rollback <app> <id>, which redeploys a previous synced revision. This only works when auto-sync is off. If auto-sync is on, Argo CD would just sync back to the latest commit, so it refuses the rollback.

The second, and the GitOps way, is git revert of the bad commit and a push. Argo CD then syncs the reverted state. This is preferred because Git again matches what is running, the history shows what happened, and every environment that follows the same branch stays consistent.

Use the UI rollback only as an emergency brake while you prepare the proper revert, and remember to fix Git afterwards.

Some fields are changed by other controllers after you apply a resource. Examples are spec.replicas managed by an HPA, annotations added by a controller, or CA bundles injected by a webhook. Argo CD sees each of these as a difference and reports OutOfSync, or fights the controller if selfHeal is on.

ignoreDifferences tells Argo CD to skip specific fields, selected by JSON pointer or JQ path, when it diffs. For example, you can ignore /spec/replicas on Deployments.

Use it narrowly and only for fields you genuinely do not own in Git. Ignoring a whole resource hides real drift.

Argo CD diffing customization

GitRepository says where to fetch from: the URL, branch, interval and credentials. A Flux Kustomization says what to apply from that source: the path, whether to prune, the interval, health checks and dependencies on other Kustomizations.

The trap is the name. The Flux Kustomization resource is not the same as the Kustomize kustomization.yaml file. The Flux resource points at a folder, and that folder may or may not contain a kustomization.yaml.

You define a source for the chart (a HelmRepository, GitRepository or OCIRepository) and a HelmRelease that names the chart, version and values. The helm-controller then installs and upgrades the release, and also detects drift.

Remediation is what Flux does when an install or upgrade fails. You can ask it to retry a number of times and to roll back to the last good release. Values can also come from ConfigMaps or Secrets using valuesFrom, which is useful for secret values.

Helm starts with the chart's default values.yaml, then applies each override on top. Later sources win:

1. The chart's own values.yaml.
2. A parent chart's values, if the chart is a subchart.
3. Files passed with -f or --values. If you pass several, the last one wins.
4. Individual --set, --set-string and --set-file flags, which are highest.

Maps are merged key by key, but lists are replaced whole. That is a classic trap. If you override one item of a list, you replace the entire list, so you must include every item you want to keep.

In GitOps, keep a base values file plus one small values file per environment, so the differences are easy to read in review.

helm upgrade --install <release> <chart> installs the release if it does not exist and upgrades it if it does. That makes it idempotent, so you can run the same command in every pipeline run.

--wait makes Helm wait until resources such as Deployments are ready (up to --timeout) before reporting success. Without it, Helm says "deployed" as soon as the manifests are accepted, even if the pods never start.

--atomic goes further. If the upgrade fails or times out, Helm automatically rolls back to the previous release. It implies --wait.

Helm templates use Go templating plus the Sprig function library. Inside a template, .Values reads configuration, .Release gives release information, and .Chart gives chart metadata. Pipes chain functions, for example {{ .Values.image.tag | default .Chart.AppVersion | quote }}.

Named templates, stored in _helpers.tpl, are reusable snippets such as standard labels or a full name. You call them with include, not template, because include returns a string you can pipe into functions like nindent.

Common mistakes are wrong indentation in YAML output, forgetting to quote strings that look like numbers or booleans, and using template when you need to pipe the result.

Hooks are templates with an annotation such as helm.sh/hook: pre-upgrade that run at a defined point in the release life cycle. Common hooks are pre-install, post-install, pre-upgrade, post-upgrade, pre-delete and test. They are usually Jobs.

Typical uses are running a database migration before an upgrade, or running a smoke test afterwards. Use helm.sh/hook-weight to order several hooks and helm.sh/hook-delete-policy to clean up finished Jobs.

The trap is that hook resources are not tracked as part of the release. They are not removed by a normal uninstall unless you set a delete policy, and a failing hook fails the whole release. Under Argo CD, Helm hooks are translated into Argo CD hooks, so behaviour can differ from running Helm directly.

Helm chart hooks

Use several layers, from cheapest to most realistic:

• helm lint checks the chart for structure and syntax problems.
• helm template renders the manifests locally with your values, so you can read the exact YAML. Pipe it into a schema validator such as kubeconform.
• helm install --dry-run --debug asks the cluster to validate the manifests without creating them.
• The helm diff plugin shows what an upgrade would change against the live release.
• helm test runs the chart's test hooks against a real install.

A values.schema.json file in the chart also lets Helm reject invalid values early. In CI, run lint, template and a schema check on every pull request, so errors appear before the GitOps agent sees them.

A chart can depend on other charts, declared under dependencies in Chart.yaml with a name, version and repository. helm dependency update downloads them into the charts/ folder, and a Chart.lock file pins the exact versions.

A parent chart configures a subchart by putting values under a key named after the subchart. For example, postgresql.auth.database: shop in the parent's values sets that value in the postgresql subchart. The condition field lets you turn a dependency on or off.

Watch out for the lists-replace rule and for version drift. Commit Chart.lock, and avoid unpinned ranges, so your build today is the same as your build next month.

Helm is a templating and packaging tool. It is best when you distribute software to others, need versioned releases, or the app has many configurable options. Its weakness is that Go templates inside YAML are hard to read and debug.

Kustomize is template-free. You keep plain YAML in a base and layer patches in overlays. It is best for your own apps that differ slightly per environment, because the output is easy to read and review. Its weakness is that it cannot do loops or conditions, so it struggles with very flexible packages.

They are not rivals. A common pattern is to use a third-party Helm chart for infrastructure (ingress, monitoring) and Kustomize overlays for your own apps. You can also render a Helm chart and patch the result with Kustomize, and both Argo CD and Flux support this.

A base holds the common manifests that every environment shares. An overlay is a folder that points at the base and applies environment-specific changes, such as replica counts, resource limits, image tags and extra labels, using patches.

Because overlays contain only the differences, you can review exactly how prod differs from dev. Run kustomize build overlays/prod (or kubectl kustomize) to see the final YAML. Patches can be strategic merge or JSON 6902, and the images field is a clean way to bump a tag without editing the base.

Kustomize

Kubernetes does not restart pods when a ConfigMap or Secret changes. Values injected as environment variables are read only at container start. Files mounted from a ConfigMap are eventually refreshed on disk, but most applications do not re-read them without a signal.

The fix is to make the change alter the pod template, so a rolling update starts. There are two common ways:

• Kustomize configMapGenerator: it adds a hash of the content to the ConfigMap name. Any change creates a new name, which changes the Deployment, which triggers a rollout.
• Helm checksum annotation: put a hash of the rendered ConfigMap in the pod template annotations.

Another option is a tool such as Reloader that watches the object and restarts workloads. The hash approach is better for GitOps, because the rollout is part of the Git change and rollback restores the old config together with the old pods.

A role is a standard folder structure that packages tasks, default variables, templates, files and handlers for one purpose, such as nginx or postgres. You can then reuse it across playbooks and share it, which keeps playbooks short and consistent.

A handler is a task that runs only when notified by another task that changed something, and it runs once at the end of the play. The classic use is restarting a service only if its config file changed.

If nothing changes, nginx is not restarted, so you avoid needless downtime.

Ansible has many places to set a variable, and a fixed order decides which wins. In simple terms, role defaults are the lowest, then inventory and group_vars, then host_vars, then play and role variables, and extra variables passed with -e on the command line always win.

The practical advice is to put sensible defaults in the role, put environment differences in group_vars, and use -e only for one-off overrides. Avoid defining the same variable in many places, because debugging precedence is painful.

Ansible Vault encrypts files or single values so you can commit them to Git. You create one with ansible-vault encrypt or encrypt_string, and supply the password at run time with --ask-vault-pass or a vault ID or password file. Keep the vault password in a proper secret store, not in the repository.

Ansible variable precedence

GitOps is not free. The main costs are these:

• Slower feedback. A change waits for the agent to sync, and a failed sync shows up in the agent, not in your pipeline.
• Secrets are awkward. Git must not hold plain secrets, so you need Sealed Secrets, SOPS or an external secrets operator.
• Manifest sprawl. Many environments and clusters create a lot of YAML, so you need Kustomize, Helm or ApplicationSet to avoid copy and paste.
• Imperative steps do not fit. Database migrations, one-time data jobs and ordered rollouts need hooks or sync waves, which add complexity.
• Cultural change. Everyone must stop making manual fixes. If a team keeps using kubectl edit, you get constant drift fights.
• Git becomes critical. If Git is down, you cannot deploy (the cluster keeps running, but you cannot change it).

It is a poor fit for tiny single-server setups, one-off experiments, or teams without the discipline to treat Git as the only way to change production. For most Kubernetes platforms with several teams, the benefits clearly win.

Work from the evidence outward:

1. Open the diff (UI or argocd app diff <app>) and read exactly which fields differ. Most problems are obvious here.
2. Ask who else changes that field. If an HPA, operator or mutating webhook edits it, either add ignoreDifferences or put the defaulted value into your manifest so the two agree.
3. Check the sync result and events. Look for invalid manifests, a missing CRD, a project restriction or an RBAC denial.
4. Run a hard refresh to clear Argo CD's manifest cache, in case it is rendering an old revision.
5. Confirm the repo server can render the source. A bad Helm value, a missing dependency or a Kustomize error will show as a comparison error.
6. If sync keeps "succeeding" and the app returns to OutOfSync, suspect a loop between Argo CD and another controller that keeps changing the object.

A common wrong answer is "just press Sync again". That treats the symptom. If the cause is a controller fight, it will be OutOfSync again within seconds.

For multi-cluster you register each remote cluster (argocd cluster add), which stores its credentials as a Kubernetes Secret, then point each Application's destination at that cluster. There are two common designs. One central Argo CD manages all clusters, which is easy to operate but is a high-value target and a single point of failure. Or you run one Argo CD per cluster (or per group), which gives isolation and a smaller blast radius, with extra management overhead.

To scale a large installation:

• Run the control plane in HA mode.
• Shard the application controller across several replicas.
• Add repo-server replicas and tune its cache.
• Use webhooks instead of frequent polling.
• Use ApplicationSet to generate apps, and split very large apps into smaller ones.

Reason about the failure modes in the interview. If the central Argo CD is down, running workloads are unaffected, but nobody can deploy or heal drift until it is back.

Both are CNCF graduated projects and both reconcile a cluster from Git, so the choice is about fit, not capability.

Choose Argo CD when teams want visibility and a UI, and when you want a central control plane. Choose Flux when the platform team wants a lightweight, composable toolkit that is configured entirely in Git. Mention that you would also consider existing skills and how each option handles access control, and that either is a defensible answer if you can explain the reasoning.

Flux image automation updates the image tag in Git for you, in three steps:

1. ImageRepository scans a container registry and records the available tags.
2. ImagePolicy picks the newest tag that matches a rule, such as semver range, regex or alphabetical order.
3. ImageUpdateAutomation commits the new tag back to the Git repo. It finds the right line using a marker comment on the field in your manifest.

The deployment itself still happens through normal Git reconciliation. That keeps Git as the truth and keeps every change auditable.

The risks are important. If your policy is too loose (for example latest or a wide semver range), a bad image can reach production automatically. Automatic commits can clutter history, and a compromised registry could push malicious tags. Mitigate by using strict policies, signing and verifying images, auto-updating only dev and staging, and using pull requests for production.

Flux image update guide

Helm stores every release revision as a Kubernetes Secret in the release's namespace by default. helm history <release> lists them. helm rollback <release> <revision> takes the manifest from that old revision and applies it, creating a new revision. The history keeps growing, it does not go backwards.

What it does not restore matters in an interview:

• Data. Persistent volumes, databases and anything written at runtime are not rolled back.
• Side effects of hooks. A pre-upgrade migration that changed a schema stays changed.
• Changes made outside Helm. Helm compares the old manifest, the new manifest and the live state, but it cannot undo things it never managed.

So a rollback is safe for stateless config and image changes, and dangerous when the new version already ran an incompatible migration. Design migrations to be backward compatible so that rolling back the app remains possible.

This happens when an earlier install or upgrade was interrupted, for example a pipeline was cancelled or a node died, and the release is left in a pending-install or pending-upgrade state. Helm locks the release, so new operations are refused.

Recover step by step:

1. Run helm history <release> and helm status <release> to see the stuck revision and the last deployed one.
2. Check that no Helm process is still running. If one is, wait or stop it.
3. Try helm rollback <release> <last-good-revision>. This often clears the state.
4. If rollback is refused, the stuck revision is stored as a Secret named sh.helm.release.v1.<release>.v<N>. Deleting or editing the pending revision's Secret unlocks the release. Do this carefully and only after you are sure nothing is running.
5. Find the root cause (timeout too short, failing hook, quota), then retry.

To prevent it, use --atomic and generous timeouts, and avoid killing deployment jobs midway. Under Flux or Argo CD, let the agent own the release, so fewer people run Helm by hand.

Separate what is shared from what differs, and generate the rest.

1. One base for the app (plain YAML or a Helm chart), with sensible defaults.
2. Environment layer (dev, staging, prod) for things that follow the environment: replicas, resources, feature flags, external URLs.
3. Cluster layer only for things that are truly cluster-specific, such as the cluster name, region, ingress domain and node selectors. Keep it as small as possible.
4. Generate the Applications with an ApplicationSet (Argo CD) or Flux Kustomizations per cluster, using the cluster list as the generator, instead of writing five near-identical files.

A suitable folder layout is base/, overlays/<env>/ and clusters/<cluster>/. Pin versions per environment so promotion is a one-line change. Keep secrets out of all layers and reference them from a secret manager.

Explain the trade-off. More layers mean less duplication but harder debugging, so stop adding layers once the differences are clear. A reviewer should be able to answer "what is different in this cluster?" by reading one small folder.

They solve different problems. Terraform provisions infrastructure (networks, VMs, databases, load balancers). It is declarative and keeps a state file that records what it created, so it can plan changes and destroy things precisely. Ansible configures what runs on that infrastructure (packages, files, services, users). It is agentless, runs tasks in order using idempotent modules, and keeps no state file.

Ansible can create cloud resources, but without state it cannot reliably tell what it created earlier or remove it cleanly. Terraform can run scripts on machines with provisioners, but that is discouraged because it is hard to re-run and debug.

A clean way to combine them:

1. Terraform creates the servers and outputs their addresses or tags.
2. Ansible uses a dynamic inventory based on those tags to configure them.
3. Or you skip long-lived configuration by baking images with Packer (using Ansible as the provisioner) and let Terraform deploy the images.

For Kubernetes workloads, neither is the main tool. GitOps agents handle the cluster, while Terraform builds the cluster and Ansible configures the nodes if you manage them yourself.

Git holds history forever, so you must never commit a plain Kubernetes Secret. Base64 is encoding, not encryption, and anyone can decode it. There are three common approaches.

Trade-offs: Sealed Secrets are tied to one cluster's key, so you must back that key up and re-seal for each cluster. SOPS keeps secrets in Git, so rotation means a new commit and you need key management. External Secrets keeps values out of Git entirely and supports rotation, but depends on the external service being available.

If a secret was ever committed in plain text, rotate it. Deleting the commit is not enough, because it can still exist in clones and caches.

External Secrets Operator

The fastest safe path is still through Git. Run git revert on the bad commit and push it, then trigger the sync immediately instead of waiting for the interval (argocd app sync <app>, or flux reconcile kustomization <name> --with-source). A revert is quick, reviewed after the fact, and leaves Git matching production.

If Git or the agent is itself the problem, an emergency manual change can be acceptable, but follow a strict routine:

1. Pause auto-sync for that app first. Otherwise selfHeal will undo your fix within minutes, and you will think the fix did not work.
2. Make the smallest change that stops the damage, for example kubectl scale or rolling back an image.
3. Immediately commit the same change to Git, so the repo and the cluster agree.
4. Re-enable auto-sync and confirm the app is Synced and Healthy.
5. Record what you did and add a follow-up to the postmortem, including why the normal path was not enough.

The common wrong answer is "just edit it in the cluster, Argo CD will sort it out". It will, by reverting your fix. The interviewer wants to hear that you protect the incident first, but you do not leave Git and production disagreeing afterwards.