CI/CD Interview Questions

CI/CD stands for Continuous Integration and Continuous Delivery (or Deployment) — the practice of automatically building, testing, and shipping code every time it changes, instead of doing those steps by hand before a release.

It matters because manual releases don't scale. Continuous Integration catches a broken change within minutes of it being pushed, while the person who wrote it still remembers the context, instead of weeks later during a manual "integration week." Continuous Delivery then keeps the resulting build in a state that's always ready to release, so shipping becomes a routine, low-risk event rather than a stressful one. Teams that do this well release more often, in smaller batches, which is itself what makes each release lower-risk — small changes are easier to test and easier to roll back than a pile of six months of work merged at once.

Further reading: GitLab CI/CD docs

Continuous delivery means every change that passes the pipeline is ready to release to production, but a human still clicks the button. Continuous deployment removes that click — a passing pipeline deploys to production automatically, with no manual gate.

The distinction matters because it changes what your test suite is allowed to get wrong. If a human reviews the release, a flaky test or a missed edge case is a nuisance someone will catch. If deployment is automatic, that same gap goes straight to users, so continuous deployment only works once you trust your automated tests, monitoring, and rollback path enough to remove the human checkpoint. Most teams start with continuous delivery and only move to full continuous deployment once they have confidence in those safety nets — feature flags, canary rollouts, and fast automated rollback are common prerequisites.

Further reading: GitLab CI/CD docs

A build artifact is the output of the build stage — a compiled binary, a Docker image, a packaged zip, a set of static assets — saved so later stages can use it without regenerating it.

Rebuilding in every stage sounds harmless but breaks a core CI/CD guarantee: that the thing you tested is the exact thing you deploy. If the test stage rebuilds from source and the deploy stage rebuilds again, a non-deterministic build step (a dependency that resolved to a different patch version, a timestamp baked into the binary) can silently produce a different artifact in production than the one your tests ran against. Building once, storing the artifact, and reusing it in every later stage — commonly called "build once, deploy everywhere" — closes that gap and is also just faster, since compiling or bundling is usually the slowest part of a pipeline.

.gitlab-ci.yml is a YAML file at the root of a GitLab repository that defines the project's entire CI/CD pipeline — its stages, the jobs within each stage, and the script each job runs.

GitLab reads this file automatically whenever a pipeline-triggering event happens (a push, a merge request, a schedule) and builds a pipeline from it. Because the file lives in the repository itself, the pipeline configuration is versioned alongside the code it builds — a branch can have a different pipeline than main, and a bad pipeline change can be reviewed and reverted exactly like any other code change.

Further reading: GitLab CI/CD docs

A GitLab Runner is the agent that actually executes the jobs defined in .gitlab-ci.yml — GitLab itself schedules the pipeline, but a runner does the real work of checking out code, running the script, and reporting back the result.

Runners can be shared (GitLab-managed infrastructure available to many projects, the default for GitLab.com) or specific (registered and dedicated to one project or group, typically self-hosted). They also come in different executor types — Docker, shell, Kubernetes, and a few others — which determines what environment a job's script actually runs inside. Picking the right executor matters: a shell executor runs jobs directly on the runner's host with no isolation between jobs, while a Docker executor gives each job a clean, disposable container.

before_script runs before a job's main script, and is typically used for setup — installing dependencies, authenticating to a registry, exporting environment variables the job needs. after_script runs after script finishes, whether it succeeded or failed, and is typically used for cleanup — removing temporary files, sending a notification, tearing down a test container.

One easy-to-miss detail: after_script runs in a new, separate shell session from script and before_script, so any environment variables or shell state set earlier in the job aren't automatically available to it. If cleanup needs that state, it has to be written to a file or a shared variable mechanism the after_script step re-reads, rather than assumed to still be in scope.

These three are nested layers of a GitHub Actions pipeline. A workflow is a YAML file in .github/workflows/ that defines an automated process, triggered by events like a push or a pull request. A workflow contains one or more jobs, which run in parallel by default (unless one explicitly needs another) and each get their own fresh virtual machine or container. A job is made up of steps, which run sequentially within that job and share its filesystem and environment — a step can be a shell command or a reusable action from the marketplace.

Further reading: GitHub Actions docs

GITHUB_TOKEN is a token GitHub automatically generates for every workflow run, scoped to that specific run and repository, used to authenticate API calls the workflow makes back to GitHub — commenting on a pull request, creating a release, pushing a tag — without you having to create and store a personal access token as a secret.

Its permissions are configurable via the permissions: key in the workflow file, and the safe default is to grant only what a given job actually needs (often just contents: read) rather than leaving it at its broader default, since a compromised or overly broad step in that job can only do as much damage as the token allows. It expires automatically when the job finishes, which is what makes it meaningfully safer than a long-lived personal access token sitting in a secret.

Use the schedule trigger with a cron expression under on:. GitHub runs the workflow at the specified times using standard five-field cron syntax, evaluated in UTC:

A couple of practical caveats worth knowing: scheduled workflows only run on the repository's default branch, and GitHub explicitly notes that scheduled runs can be delayed during periods of high load, so a cron trigger is best-effort timing rather than a guarantee of the exact minute.

Azure DevOps is Microsoft's suite of development tools — Boards (work tracking), Repos (Git hosting), Pipelines (CI/CD), Artifacts (package feeds), and Test Plans — and Azure Pipelines is specifically the CI/CD piece: the service that builds, tests, and deploys code, whether that code lives in Azure Repos, GitHub, or another supported source.

A common point of confusion is treating "Azure DevOps" and "Azure Pipelines" as interchangeable — they're not; you can use Azure Pipelines against a GitHub repository without using any other part of Azure DevOps, and conversely use Azure Repos or Boards without touching Pipelines at all. Each service is usable on its own.

Further reading: Azure Pipelines docs

A Classic pipeline is built through Azure DevOps's web-based graphical editor — you click through a UI to add build steps and, separately, define a release pipeline with stages and approvals. A YAML pipeline is a text file (conventionally azure-pipelines.yml) committed to the repository itself, which defines build, test, and — since the introduction of multi-stage YAML pipelines — deployment stages all in one version-controlled file.

The practical difference that matters most: because a YAML pipeline lives in the repo, it's versioned, branchable, and reviewable exactly like application code — a bad pipeline change goes through the same pull request review as anything else, and each branch can have its own pipeline definition. A Classic pipeline configured through the UI has none of that; its configuration lives in Azure DevOps's database, not in git history.

Microsoft's own guidance is to start new projects on YAML pipelines, and Classic pipelines are being phased out — Azure DevOps documentation now frames Classic as the legacy path rather than an equal alternative. One functional gap to know about: Classic release pipelines have a "gates" feature (automated checks like a work-item query or an Azure Monitor alert that must pass before a stage proceeds) that doesn't have a direct one-to-one equivalent in YAML pipelines; the closest replacements are YAML environment checks and approvals, which cover most but not identically all of the same ground.

Further reading: Azure Pipelines docs

The straightforward path is a pipeline variable marked as secret in the pipeline's UI settings, or a variable inside a variable group flagged as secret — either way, Azure DevOps encrypts the value at rest and masks it in the build log automatically.

For anything more sensitive or at organizational scale, link a variable group to Azure Key Vault instead of storing the secret directly in Azure DevOps — the pipeline then pulls the current value from Key Vault at run time, which centralizes rotation and access control in the vault rather than duplicating the secret's lifecycle management across every pipeline that uses it. Either way, secrets should never be written directly into the YAML file itself, since that file is committed to source control and visible to anyone with read access to the repository.

Further reading: Azure Pipelines docs

Separating stages isn't just tidiness — it changes what happens when something fails and how fast feedback arrives.

• Fast failure, cheap failure. A syntax error should fail in seconds during build, not after a 20-minute test suite has already run. Ordering stages from cheapest/fastest to most expensive means a pipeline fails as early — and as cheaply — as possible.
• Parallelism. Independent test suites (unit, integration, lint, security scan) can run at the same time once the build stage produces one artifact, instead of each retesting from source serially.
• Reuse. The same tested artifact can be promoted through dev, staging, and production deploy stages without rebuilding, which is what makes "build once, deploy everywhere" possible in the first place.
• Clear ownership of gates. A deploy stage can require manual approval or a passing security scan without that same friction slowing down every commit's build and test run.

A single monolithic script gets none of this: one failure anywhere reruns everything from scratch, and there's no natural point to insert an approval gate before production.

Trunk-based development means developers merge small changes into the main branch (the "trunk") frequently — often multiple times a day — instead of working for weeks on long-lived feature branches that merge in one large batch.

It changes pipeline design in a few concrete ways. First, the pipeline has to be fast, because it runs on every merge to main and slow feedback defeats the point of merging often. Second, since features aren't finished when they merge, incomplete work is usually hidden behind feature flags rather than a branch, which the pipeline treats as ordinary code — it builds and deploys whether or not the flag is on. Third, because main is always close to release-ready, CI on trunk-based projects tends to run a full test suite on every push rather than only before a release, since "every push" effectively is the release candidate. The common wrong answer here is assuming trunk-based development means no branches at all — it doesn't; short-lived feature branches are fine, the key constraint is how long they live before merging, typically under a day or two.

Both are ways to release a new version without a hard cutover, but they trade risk for speed differently.

Blue-green keeps two identical production environments — "blue" (current) and "green" (new). You deploy the new version fully to green, test it in isolation, then switch all traffic over at once, usually at the load balancer or router. Rollback is just switching traffic back to blue, which is why it's fast, but it doesn't reduce blast radius: once you switch, 100% of users are on the new version immediately.

Canary deploys the new version alongside the old one and shifts a small slice of real traffic to it first — say 5%, then 25%, then 100% — while watching error rates and latency at each step. It catches problems that only show up under real production traffic before most users are affected, at the cost of more operational complexity: you need traffic-splitting infrastructure and metrics good enough to decide automatically (or manually) whether to keep ramping up or roll back.

A rough rule of thumb: blue-green favors fast, clean rollback for changes you're fairly confident in; canary favors limiting blast radius for changes you're less sure about.

Further reading: Martin Fowler — BlueGreenDeployment

A self-hosted runner (GitLab), runner (GitHub Actions), or agent (Azure Pipelines) is a machine you provision and register yourself to execute pipeline jobs, instead of using the platform's managed, shared infrastructure.

You'd reach for one when the hosted option can't meet a real constraint: you need hardware the vendor doesn't offer (GPUs, specific chip architectures, large local caches), you need the runner inside a private network to reach internal systems a hosted runner can't route to, you're hitting cost limits from heavy pipeline usage at scale, or you have compliance requirements that mean build environments can't leave infrastructure you control. The trade-off is that you now own patching, scaling, and securing that machine — a self-hosted runner that executes code from an untrusted pull request is a genuine attack surface, since a malicious workflow step can read anything the runner's credentials can reach.

A job is a single unit of work — a named block in .gitlab-ci.yml with a script to run. A stage is a named phase (like build, test, deploy) that groups jobs together and controls ordering: all jobs in one stage run in parallel, and GitLab won't start the next stage until every job in the current one finishes successfully.

Here, unit_tests and lint both run at the same time because they share the test stage, but neither starts until build_job succeeds, and deploy_prod waits for both of them. This stage-then-parallel-jobs model is the default ordering, though needs (covered separately) lets you override it for jobs that don't actually depend on their whole stage finishing.

Both save files between job runs, but they exist for opposite reasons and behave differently as a result.

Artifacts pass the output of this specific pipeline run from one stage to the next — a compiled binary, a test report, a built Docker image tarball. They're guaranteed to be available to downstream jobs in the same pipeline, stored on the GitLab server, and are what you'd use to hand off a build's result from the build stage to the deploy stage.

Cache speeds up future pipeline runs by reusing files across pipelines — most commonly a dependency directory like node_modules or a package manager's download cache, so the next run doesn't redownload everything from scratch. Cache is best-effort: GitLab doesn't guarantee a cache hit, and a job should still work correctly (just slower) on a cache miss.

The trap: using cache to pass a build's output to a deploy job. It isn't guaranteed to be there, so a deploy job that silently depends on a cache hit for its actual deployable artifact is a pipeline that will occasionally fail — or worse, silently deploy stale content — for no obvious reason. Use artifacts for anything the pipeline actually depends on.

Further reading: GitLab CI/CD docs

only and except are GitLab's original, simpler syntax for controlling when a job runs — only lists the branches or tags a job should run on, except lists ones it shouldn't. rules is the newer, more expressive replacement: a list of conditions, each with an if, that's evaluated in order, and the first one that matches decides whether — and how — the job runs.

The reason rules replaced only/except rather than sitting alongside it as an equal option is that only/except can only branch on a fixed set of built-in conditions, while rules can combine arbitrary CI/CD variable expressions, react to which files changed, and set when: manual or when: never per condition instead of just on/off:

This job only appears — and only as a manual approval step — when the pipeline is running on main; everywhere else it doesn't run at all. GitLab documents rules as the recommended approach for new pipelines, with only/except kept for backward compatibility rather than active recommendation.

Further reading: GitLab CI/CD docs

The safest path is GitLab's built-in CI/CD variables, set in Settings → CI/CD → Variables rather than hardcoded in .gitlab-ci.yml, where two flags matter: masked, which hides the value from job logs, and protected, which restricts the variable to protected branches and tags only — so a pipeline running on an arbitrary feature branch or a fork's merge request can't read a production deploy key.

For values shared across many projects, a group-level or instance-level variable avoids copy-pasting the same secret into every repository, and for anything sensitive at scale, integrating with a dedicated secrets manager (GitLab's own secrets manager integration, or HashiCorp Vault) is generally preferable to long-lived static variables, since it enables rotation and short-lived credentials instead of a value that sits unchanged for years. Plain, non-sensitive configuration — a target environment name, a feature flag — is fine as an ordinary unmasked variable; the masking and protection controls exist specifically for things that would cause harm if exposed.

An environment in GitLab (declared with the environment: keyword on a deploy job) is a named target — production, staging, review/feature-x — that GitLab tracks deployment history against. Every time a job with an environment: block runs, GitLab records it as a deployment to that environment, which is what powers the deployments dashboard, the ability to see exactly which commit is live where, and one-click rollback to a previous successful deployment.

Beyond tracking, environments are also where GitLab enforces protection — a protected environment can require specific users or groups to approve a deployment before it runs, which is the mechanism behind manual production-deploy gates. Dynamic environments (using variables in the environment name, like review/$CI_COMMIT_REF_SLUG) are how GitLab's review-app pattern spins up a temporary, per-merge-request environment that's automatically cleaned up when the branch is deleted or the MR closes.

Both exist to avoid copy-pasting the same YAML into every repository, but they operate at different levels and are called differently.

A reusable workflow is an entire workflow file (with its own jobs:) that another workflow calls with uses: and workflow_call:. It can define its own jobs, run on its own runners, and expose typed inputs, secrets, and outputs — it's the right tool when you want to share a whole multi-job process, like "build, test, and publish this package," across several repositories.

A composite action bundles multiple steps into a single reusable step that gets called from within an existing job, alongside other steps in that job. It's the right tool for a smaller, step-level piece of reuse — "checkout, set up the right Node version, and install dependencies with the right caching" — that a job wants to insert as one line without owning its own separate runner or job-level structure.

The rule of thumb: reusable workflows for sharing whole pipelines across jobs and repos; composite actions for sharing a handful of steps inside one job.

Further reading: GitHub Actions docs

A matrix strategy runs the same job multiple times with different variable combinations — different OS versions, language versions, or any other axis you define — instead of writing a near-identical job block for each combination by hand.

That example runs four jobs — every combination of the two OSes and two Node versions. By default, GitHub Actions treats a matrix as fail-fast: the moment any one combination fails, GitHub cancels all the other still-running combinations, since the assumption is you've already found a problem and don't need to spend more runner time confirming it elsewhere.

Setting fail-fast: false turns that off, and you want it whenever the matrix is testing compatibility rather than just re-running the same test for speed — if Node 18 fails but Node 20 passes, you want to see both results, not have the Node 20 job cancelled before it finishes, because "does this work on 18 but not 20" is exactly the information the matrix exists to surface.

actions/cache saves specified paths (a dependency directory, a build cache folder) keyed by a string you define, and restores them on a later run if a cache with a matching key exists — the classic use is caching node_modules or a package manager's download cache keyed by a hash of the lockfile, so dependencies only get freshly resolved when the lockfile actually changes.

On a cache miss — no exact key match — the step doesn't fail; it simply doesn't restore anything, and the job falls back to running the install step from scratch, then saves a fresh cache under the new key at the end of the job for next time. restore-keys softens a miss further: if the exact key doesn't match, GitHub will restore the most recent cache matching the prefix instead of nothing at all, which is usually still faster than a completely cold install even if it's not a perfect match.

The trap worth naming: caching something that should invalidate on more than just the lockfile — a cache key that never changes will happily serve stale dependencies indefinitely, and a build cache key that's too broad can occasionally cause a corrupted or stale-in-a-bad-way cache to persist across runs it shouldn't.

GitHub-hosted runners are fresh virtual machines GitHub provisions, runs your job on, and destroys — zero setup, but limited to GitHub's available OS images and hardware, subject to job time limits (a 6-hour maximum per job, and a per-plan limit on total workflow duration), and billed per minute beyond the free tier. Self-hosted runners are machines you provision and register yourself, giving you full control over hardware (GPUs, more memory, specific chip architectures), software pre-installed on the image, and network access (reaching internal, non-public systems a GitHub-hosted runner can't route to) — at the cost of you owning patching, scaling, and security.

The security trade-off is the one to actually weigh carefully: a self-hosted runner that picks up jobs from public repository workflows — especially from forks — is executing arbitrary, potentially attacker-controlled code on a machine you own, which is a meaningfully bigger risk than the same workflow running on GitHub's disposable, isolated infrastructure.

The modern answer is OpenID Connect (OIDC) federation instead of a static access key stored as a secret. GitHub Actions can present a short-lived, cryptographically signed OIDC token to the cloud provider, which the provider verifies and exchanges for temporary credentials scoped to that one job run — no long-lived AWS access key, Azure client secret, or GCP service account key sitting in your repository's secrets at all.

This matters because a static credential in a secrets store is a permanent liability — if it leaks, it's valid until someone notices and manually rotates it. A federated, short-lived credential is only ever valid for the duration of one job run, so even a leak has a tiny, self-expiring blast radius. Setting it up requires configuring trust on the cloud side (an IAM role with a trust policy scoped to your specific repository and, ideally, branch) and adding permissions: id-token: write to the workflow so it's allowed to request the token in the first place.

Further reading: GitHub Actions docs

concurrency prevents multiple runs of the same workflow — or a group you define — from executing at the same time, which matters most for deployment workflows where two overlapping runs could race each other and leave production in an inconsistent state.

With cancel-in-progress: true, a new run in the same concurrency group cancels whatever's currently running before starting — useful for CI on a pull request, where you only care about the latest commit's result and want to stop burning runner minutes on an outdated one. Without it (the default, false), new runs queue up and wait their turn instead of cancelling anything, which is usually what you want for a deploy workflow — you don't want a deploy to production silently cancelled mid-flight by a second one starting.

A common pattern uses a dynamic group name tied to the branch or PR, like group: ${{ github.workflow }}-${{ github.ref }}, so concurrency is scoped per-branch rather than accidentally serializing every PR's CI run against every other PR's.

Start with GitHub's built-in debug logging before reaching for anything else: setting the repository secrets ACTIONS_STEP_DEBUG and ACTIONS_RUNNER_DEBUG to true turns on verbose step-by-step and runner-level logging that's hidden by default, and usually surfaces the actual failure point that the normal log output glosses over.

From there, a few targeted checks cover most "CI-only" failures:

• Environment differences. CI runners start from a clean image every time — no locally cached dependencies, no locally-set environment variables, no files left over from a previous run. A test that implicitly relies on local state (a .env file that isn't checked in, a database seeded once and never reset) will pass locally and fail in CI for that reason alone.
• Timing and concurrency. GitHub-hosted runners are typically less powerful than a developer's machine, which can expose race conditions or flaky timing assumptions (a fixed sleep instead of a proper wait condition) that happen to not trigger locally.
• Secrets and permissions. Confirm the failing step actually has access to the secret or token it needs — a workflow triggered by a fork's pull request, for example, won't have repository secrets available at all, which looks like an auth failure rather than an obviously missing-secret error.
• Add inline debug output. ::debug:: and ::error:: workflow commands, or simply extra run: env / run: pwd steps around the failing point, are often faster than reasoning about it than trying to reproduce the exact runner environment locally with something like act.

The general principle: don't assume the code is broken just because it fails in CI — first confirm you're actually comparing the same inputs, environment, and permissions before debugging the logic itself.

These nest in order, similar in spirit to GitLab's stages/jobs or GitHub's jobs/steps, but with an extra layer:

• A stage is a major division of the pipeline — typically representing a phase like Build, Test, or Deploy-to-Production — and stages can depend on each other or run in parallel.
• A job runs on a single agent and is a set of steps executed in sequence. Multiple jobs within a stage can run in parallel across different agents.
• A task is the smallest unit — a single packaged step, like "restore NuGet packages" or "run a shell script" — that runs sequentially within a job.

The reason Azure Pipelines has this extra stage-vs-job distinction (compared to GitHub Actions' simpler workflow → job → step, or GitLab's stage → job → script) is largely historical: it grew out of merging the old separate Classic build and Classic release pipeline concepts into one YAML model, and stages map roughly onto what used to be separate build and release definitions.

Approvals and gates are both mechanisms that pause a pipeline before it proceeds to a stage — most commonly before deploying to production — but they check different things.

Approvals require a specific person or group to manually review and click approve (or reject) before the stage runs. They're configured on an environment in YAML pipelines (or a stage in Classic release pipelines), and typically list required reviewers, an optional timeout, and whether the approver can be the same person who triggered the pipeline.

Gates (a Classic release pipeline feature) are automated, non-human checks — an Azure Monitor alert query that must return no active alerts, a REST API call that must return a specific status, a work-item query that must return zero results — that run automatically and either pass or block the stage without anyone clicking anything.

In practice, teams often combine both: an automated gate verifies system health is currently fine, and a human approval adds a deliberate go/no-go decision on top for anything customer-facing. YAML pipelines replicate most of this through environment checks (which cover much of what Classic gates did) plus approvals, though — as noted above — it isn't a perfect one-to-one feature match with Classic gates.

Further reading: Azure Pipelines docs

A service connection is an authenticated link from Azure DevOps to an external service — an Azure subscription, an AWS account, a Docker registry, a Kubernetes cluster — that a pipeline task uses to actually perform an action against that service, like deploying to an Azure Web App or pushing an image to a container registry. It stores the credentials and connection details centrally so individual pipelines don't each hold their own copy.

A variable group is a named set of key-value variables (which can include secrets, optionally backed by Azure Key Vault) that multiple pipelines can reference, so a value like an environment name, a connection string, or a feature flag doesn't have to be duplicated across every pipeline that needs it.

The distinction that trips people up: a service connection is specifically for authenticating and acting on an external resource through a pipeline task, while a variable group is just a shared bag of values a pipeline's script or task inputs can read — a service connection is never something you'd use to pass a plain configuration value, and a variable group is never something a deploy task authenticates through directly.

An environment in Azure Pipelines is a named collection of resources you're deploying to — dev, staging, production, or more specifically a set of Kubernetes namespaces or virtual machines — and it's where deployment history, approvals, and checks are configured, similar in role to a GitLab environment.

A deployment job (declared with deployment: instead of job: in YAML) is the job type that specifically targets an environment and records a deployment against it, giving you deployment history and rollback visibility that a plain job: doesn't provide. It also supports built-in deployment strategies — runOnce, rolling, and canary — that control how the deployment steps execute against the environment's resources, rather than you having to script that logic by hand:

The reason to use deployment: instead of an ordinary job: for a deploy step, even though a plain job could technically run the same script, is that you'd lose the environment-linked approval gates, deployment history, and built-in rollout strategies that only apply to deployment jobs specifically.

A Microsoft-hosted agent is a fresh virtual machine Microsoft provisions on demand from a fixed set of images (Ubuntu, Windows, macOS), runs your job on, and tears down afterward — no setup required, but bounded by a job time limit and, on the free tier, limited monthly minutes and parallel jobs. A self-hosted agent is a machine — physical, virtual, or containerized — that you install the Azure Pipelines agent software on and register to your Azure DevOps organization or project yourself, giving you control over installed software, hardware, and network placement.

The reasons to choose self-hosted mirror the same trade-off as GitHub's self-hosted runners or GitLab's specific runners: you need it to reach an internal network Microsoft's hosted agents can't route to, you need specific hardware or a large persistent cache Microsoft doesn't offer, or hosted-tier minute limits are a real constraint at your usage volume. The corresponding cost is that you now own patching and securing that machine yourself, and — same as the other platforms — running untrusted pull request code on a self-hosted agent that has broader network or credential access than a disposable hosted VM is a security decision to make deliberately, not by default.

Branch policies on a protected branch (typically main) can require a specific build validation — a designated pipeline — to run and pass before a pull request targeting that branch is allowed to merge. This is what turns "we have a CI pipeline" into an actual enforced gate rather than a pipeline that runs, reports a status, and can still be merged past if someone ignores a red X.

Setting it up has a few pieces working together:

• Build validation policy on the branch, pointing at the pipeline that should gate merges — you can require it to be up to date with the target branch before merging, not just green at some earlier point, which prevents merging a PR whose passing build no longer reflects what main actually looks like after other merges landed.
• Required reviewers, separately, for human sign-off on the code itself, which a pipeline can't substitute for.
• Status checks from the pipeline itself — test results, code coverage thresholds, a security scan's findings — can each be surfaced as pass/fail signals on the PR, and additional policies can require specific ones (like a minimum coverage percentage) rather than just "the pipeline finished."

The combination matters because either piece alone is incomplete: reviewer approval without a required, up-to-date build lets a human approve code that's never actually been verified to build or pass tests against the current state of main, while a required pipeline without reviewer approval lets syntactically fine but poorly designed code merge with nobody having actually read it.

A push trigger runs the pipeline whenever a commit lands on a branch (or tag) that matches the trigger's filter, regardless of whether that branch is part of an open pull or merge request. A pull/merge request trigger runs specifically against the merged result of a proposed change against its target branch — testing what the code would look like if the merge happened, not just the source branch in isolation.

The distinction matters more than it sounds like it should, because the two can give different results for the same commit. A feature branch might pass every push-triggered pipeline on its own, but fail a merge-request-triggered pipeline because merging it with an already-updated main produces a conflict-free-but-broken combination neither branch alone would surface — a second developer's unrelated change to a shared file, for instance. This is exactly why GitLab's "merged results" pipelines and GitHub's pull_request event both build and test the merge commit, not the branch tip, and why relying only on push-triggered CI on feature branches can let integration problems slip through until they hit main.

Push triggers are still useful and necessary on their own — for running a pipeline on every commit to main after merge, or for triggering scheduled or tag-based release pipelines that have nothing to do with a pull request at all.

Linear execution runs a test suite as one long, sequential job — every test in one process, one after another. Parallel execution splits the same suite across multiple jobs or workers that run at the same time, then combines the results, which cuts wall-clock time roughly in proportion to how many parallel slices you run, at the cost of using more runner capacity simultaneously.

The decision comes down to a few practical factors:

• Suite size and runtime. A two-minute test suite gains little from parallelizing — the overhead of spinning up extra jobs can eat most of the theoretical time savings. A twenty-minute suite is a much clearer candidate.
• Test independence. Parallelizing assumes tests don't depend on shared, mutable state — a test suite that writes to a shared database table and expects a specific order will produce flaky, order-dependent failures once split across workers, so parallelizing sometimes has to wait until that coupling is fixed first.
• Runner cost and availability. Parallel jobs consume more concurrent runner minutes or agents at once; on a constrained runner pool, four jobs running in parallel might actually queue and finish slower in wall-clock time than one long job that starts immediately.
• How you split the suite. Splitting by file count is simple but can produce badly unbalanced shards if some test files are much slower than others; splitting by historical timing data (some CI platforms and tools support this) balances load more evenly across parallel jobs.

In practice, most teams start linear, and only introduce parallelism once suite runtime becomes the bottleneck slowing down feedback — introducing it earlier than necessary just adds coordination complexity (aggregating results, managing shared fixtures) for no real benefit yet.

Infrastructure as Code (IaC) means defining infrastructure — servers, networks, load balancers, Kubernetes clusters — in version-controlled configuration files (Terraform, Bicep, CloudFormation, Pulumi) instead of clicking through a cloud console by hand, so infrastructure changes go through the same review, versioning, and automated process as application code.

In a pipeline, IaC typically runs as its own stage or its own separate pipeline, structured around a plan-then-apply pattern rather than applying changes blindly:

1. Plan — the pipeline runs a dry-run (terraform plan, an ARM/Bicep what-if) that shows exactly what would change, without changing anything yet.
2. Review — for anything touching production infrastructure, the plan output is usually posted for human review (often directly on the pull request) before proceeding, since an infrastructure change that looks small in the diff can imply something destructive, like replacing a resource instead of updating it in place.
3. Apply — only after approval does a separate stage actually apply the change, using the exact plan that was reviewed rather than re-planning at apply time, since state can drift between the two steps otherwise.

The reason this is usually a separate pipeline from the application's build/test/deploy pipeline, rather than one more stage bolted onto it, is that infrastructure and application code change on different cadences and carry different blast radii — a bad application deploy is usually recoverable by rolling back a version; a bad infrastructure apply can delete a database or take down a network path that many services depend on at once.

The honest first answer is: decide the rollback strategy before you deploy, not while production is on fire — by the time something breaks you want a button to press, not a decision to make.

A solid strategy usually has three layers:

1. Detection. Automated health checks and monitoring (error rate, latency, saturation) that fire fast enough to catch a bad deploy before most users notice. A deploy that "worked" but silently doubled your 500 rate is still a failure.
2. A fast, tested rollback path. The two common mechanisms are (a) redeploying the last known-good artifact — which is why keeping build artifacts versioned and addressable matters — or (b) traffic-based rollback, where a blue-green or canary setup lets you shift traffic back to the previous version without a new deployment at all. Database migrations are the hard part here: a rollback that reverts code but leaves a backward-incompatible schema change in place will break just as badly as the bug you were rolling back from, so migrations in a CI/CD pipeline should generally be written to be backward-compatible for at least one release (expand-then-contract), so the previous code version still works against the new schema.
3. A practiced runbook. Rollback steps that only exist in someone's head, or that have never actually been run, tend to fail exactly when you need them. Teams that take this seriously test rollback the same way they test forward deploys — including in a game-day exercise, not just in theory.

The trap to name explicitly: treating "redeploy the old code" as sufficient without also accounting for state — database schema, message queue formats, cached data — that the new version may have already changed and the old version can't read.

The short answer is: secrets never live in the pipeline definition file itself, and the pipeline tool's job is to inject them at runtime and mask them from output.

• Never commit them. No API keys or passwords in .gitlab-ci.yml, workflow YAML, or Azure Pipelines YAML — even in a "private" repo, since history, forks, and logs all leak.
• Use the platform's secret store. GitLab CI/CD variables (masked and protected), GitHub Actions repository/environment secrets, and Azure Pipelines variable groups (optionally backed by Azure Key Vault) all exist so secrets are injected as environment variables at job runtime and never appear in the YAML.
• Mask them in logs. All three platforms automatically redact known secret values from job output, but that only works if the value was registered as a secret — a token pasted into a plain variable, or echoed with a string concatenation that defeats the masking pattern, will print in the clear.
• Scope and rotate. Limit a secret to the environment or branch that actually needs it (a production deploy key shouldn't be readable by a pipeline running on a fork's pull request), and rotate on a schedule so a leaked-but-undetected secret has a shelf life.
• Prefer short-lived credentials over static ones where you can. OIDC-based federation to a cloud provider (AWS STS, Azure AD workload identity, GCP Workload Identity Federation) issues a token that's valid for one job run instead of a long-lived static access key sitting in a secret store waiting to be stolen.

The trap: fork-triggered pull request pipelines. On GitHub Actions in particular, pull_request events from forks don't get access to repository secrets by default — that's intentional, and "fixing" it by switching to pull_request_target without understanding the security model is how secrets and write-scoped tokens end up exposed to untrusted code.

Further reading: GitHub Actions docs

This is the classic "works in CI, breaks in prod" bug, and it almost always comes down to an environment difference the pipeline didn't test for, not a code bug the tests missed.

Work through it in this order:

1. Confirm it's the same artifact. Check that the artifact deployed to production is literally the one that passed CI — not a rebuild. If your pipeline rebuilds at deploy time instead of reusing the tested build artifact, that's the first thing to fix, independent of this specific bug.
2. Diff the environment, not the code. Compare config, environment variables, feature flags, and secrets between the CI test environment and production. A very common cause is a config value (a database URL, a feature flag default, a resource limit) that's different in production and was never exercised by CI.
3. Check for missing test coverage of the actual failure path. Load, concurrency, and scale-dependent bugs (race conditions, connection pool exhaustion, N+1 queries that only hurt under real traffic) frequently don't show up in a CI test suite that runs against a small, synthetic dataset with no concurrent load.
4. Check infrastructure dependencies that CI doesn't touch. DNS, network policies, service mesh rules, third-party API rate limits, and database migration state are all things a CI pipeline typically doesn't fully replicate.
5. Look at what changed outside the code. Did a dependency get a new version pulled in unpinned, did an infrastructure change land separately from the app deploy, did a feature flag get flipped by someone unrelated to this release?

The deeper fix, once you find the gap, is usually to close it structurally — add a staging environment that mirrors production config, add the missing test class, or add a canary step that would have caught this before it hit 100% of users — rather than treating each incident as one-off.

By default, GitLab pipelines run stage-by-stage: every job in a stage must finish before the next stage starts, even if a particular job in that next stage doesn't actually depend on most of what just ran. needs breaks that rule for a specific job — it lets a job start as soon as the specific jobs it lists have finished, regardless of what stage they're in or what else in the current stage is still running. That turns the pipeline from a strict sequence of stages into a directed acyclic graph (DAG), where the real dependency chain between jobs — not the stage list — determines execution order.

Without needs, test_frontend would sit idle waiting for build_backend to finish even though it has nothing to do with it. With needs, each test job starts the moment its own build is done. On a large pipeline with many independent components — common in monorepos — this is often the single biggest wall-clock time improvement available, because independent chains of work stop blocking each other. Setting needs: [] on a job explicitly means "start immediately, don't wait for anything."

GitLab supports this natively with the trigger keyword, which lets a job in one project's pipeline kick off a pipeline in a different project:

strategy: depend is the detail that matters most in practice: without it, the upstream pipeline marks the trigger job as successful the instant the downstream pipeline starts, regardless of whether it later passes or fails — which means a broken downstream deploy shows up as a green upstream pipeline. With strategy: depend, the upstream job actually waits for and mirrors the downstream pipeline's final status.

This pattern is common for a few real setups: a shared library repo triggering rebuilds of every service that depends on it, an application repo triggering an infrastructure repo's deploy job after a successful build, or a monorepo splitting genuinely independent components into separate pipelines that still need to coordinate a release. It's worth weighing against needs with cross-project artifact downloads for simpler cases — full multi-project triggering is the right tool when the downstream project has its own independent pipeline and lifecycle, not just when you want to share a build output.

Further reading: GitLab CI/CD docs

This is almost always a difference in who is running the command or what filesystem it's writing to, not a difference in the command itself.

Work through these, roughly in order of likelihood:

• User inside the container. Local Docker runs often use your own user (or root with your files bind-mounted), while a CI runner's Docker executor typically runs the container as root or as whatever user the image defaults to. A script writing to a directory owned by a different UID than the one executing it will fail with exactly this error.
• Docker-in-Docker vs Docker socket mounting. If the job needs to build or run other containers, check whether the runner is configured for Docker-in-Docker (dind) or socket mounting — the permission model, and what the job is even allowed to do, differs between the two, and a runner config mismatch here is a very common cause.
• Read-only or missing volume mounts. A CI runner's working directory and cache paths may not map to the same paths your local setup assumes; the job may be trying to write somewhere the runner never mounted as writable.
• Runner executor configuration. Check the runner's config.toml for privileged mode and volume settings — some operations (certain Docker builds, some system-level installs) require privileged mode that a shared runner may not grant.
• File permissions baked into the image itself, which won't surface locally if you're the same user who built the image but will surface in CI running as a different one.

The general diagnostic move is to add a debug step (whoami, id, ls -la on the target path) right before the failing command in the CI job specifically, rather than trying to reason about it from the local reproduction — "works locally" and "works in CI" are running under different identities and mount configurations even when the command text is identical.

The core risk is that a fork's pull request contains code an attacker fully controls, and if your workflow runs that code with access to your repository's secrets or write permissions, the attacker effectively controls your CI environment.

GitHub's default behavior already provides most of the protection: workflows triggered by the pull_request event from a fork run with a read-only GITHUB_TOKEN and no access to repository secrets, specifically to contain this risk. The danger case is pull_request_target, which runs with the base repository's permissions and secrets and is often combined — incorrectly — with checking out and running the fork's untrusted code, which defeats the entire protection the event type exists to provide.

Beyond that default, a few concrete practices matter:

• Pin third-party actions to a commit SHA, not a mutable tag, since a tag like @v2 can be moved by the action's maintainer (or an attacker who compromises their account) to point at malicious code without you changing anything in your workflow.
• Set permissions: explicitly and minimally at the workflow or job level rather than relying on the default, which is broader than most jobs need.
• Require approval for workflow runs from first-time contributors, a setting GitHub provides specifically for public repositories, so a fork's first PR doesn't execute automatically at all.
• Never echo secrets or untrusted PR content (titles, branch names, comment bodies) directly into a run: shell command — untrusted strings interpolated into shell scripts are a script-injection vector, since GitHub Actions expression injection through something as innocuous as a PR title is a well-documented attack pattern.

Further reading: GitHub Actions docs

Use job outputs. A job can expose a value from one of its steps as a job-level output, and any job that needs it can read that output through the needs context:

Both build and publish declare determine-version in needs, which does two things at once: it makes the value available through needs.determine-version.outputs.version, and it guarantees those jobs don't start until determine-version has actually finished producing it — without the needs dependency, the output simply wouldn't exist yet when the other jobs start, since jobs run in parallel by default.

A step writes its output by appending key=value to the special $GITHUB_OUTPUT file, which is how a shell command's result gets promoted into something referenceable elsewhere in the workflow — a common extension of this pattern is a setup job that computes a matrix dynamically and outputs it as JSON for a later job's strategy: matrix to consume with fromJson().

Azure Pipelines templates let you define a reusable chunk of pipeline YAML — stages, jobs, steps, or variables — in a separate file, then reference it from one or more pipelines with the template keyword, optionally passing parameters:

This is the Azure Pipelines equivalent of what GitLab does with include/extends and what GitHub Actions does with reusable workflows and composite actions — the underlying problem is identical across all three platforms: a growing organization ends up with dozens of pipelines that all need the same build steps, security scan, or deploy pattern, and copy-pasting that YAML into every repository means a fix or a policy change (say, adding a mandatory vulnerability scan) has to be manually applied everywhere instead of once.

Templates can live in the same repository or a separate shared repository referenced across the organization, which is the common pattern for enforcing a standard, centrally-maintained build or deploy process across many teams — a platform team owns the template, and individual project teams reference it with a version tag rather than maintaining their own copy.

Further reading: Azure Pipelines docs

This is a genuinely common migration, and the risky part isn't the build steps — it's making sure nothing about the release safety (who approves what, and what automated checks must pass) quietly gets dropped along the way.

A workable approach:

1. Inventory what the Classic pipeline actually enforces, not just what it builds. List every approval (who, on which stage), every gate (what automated check, on which stage), and any pre/post-deployment conditions — these are easy to overlook because they live in UI settings rather than anywhere visible in a diff.
2. Convert build first, deploy second. Migrate the build definition to YAML and verify it produces an equivalent artifact before touching the release side at all — this isolates build-side bugs from approval-side gaps, so a problem is easier to attribute to one or the other.
3. Map Classic gates to YAML environment checks. Most Classic gate types (Azure Monitor alerts, REST API calls, work item queries) have an equivalent environment check in YAML; a few edge cases don't map one-to-one, and those need a deliberate decision — either an equivalent implemented a different way, or an explicit, documented acceptance that the check is being dropped, not an accidental gap nobody notices until an incident.
4. Recreate approvals on the corresponding environment, and — importantly — test that they actually fire, by running the new pipeline against a non-production environment first and confirming a deployment to the real "production" environment genuinely pauses for approval before it happens for real.
5. Run both pipelines in parallel for at least one release cycle where feasible, comparing outcomes, before decommissioning the Classic one — this is the step most tempting to skip under time pressure, and the one most likely to catch a gate or approval that didn't survive the migration.
6. Decommission the Classic pipeline only after a full cycle has gone through the new one cleanly, including an actual production deployment that exercised every approval and check you inventoried in step 1.

The trap worth naming explicitly: teams that migrate the build side, confirm it produces the same artifact, and declare the migration done — without verifying that every approval gate that existed in Classic actually has a working equivalent in the new YAML environment. That gap doesn't show up as a pipeline failure; it shows up as an unapproved production deployment going out clean, which nobody notices until it causes a problem.

The concept — a deployment can't proceed until two separate, specific approvers from different groups sign off — exists on all three platforms, but through different mechanisms, and the "different teams" part is the detail each platform handles slightly differently.

GitLab CI/CD: Use a protected environment for production, and configure multiple approval rules on it, each scoped to a different group with a required approval count — GitLab supports multiple approval rules on one protected environment, so you can require one approval from the platform team's group and a separate approval from the security team's group, both before the deployment job runs.

GitHub Actions: Use an environment (Settings → Environments → production) with required reviewers. You can list multiple individuals or teams as required reviewers and set the environment to require review from more than one — the job pauses and waits for that many distinct approvals before executing the deployment steps, and the reviewers list is exactly where you'd add representatives from two different teams.

Azure Pipelines: Configure approvals on the environment the deployment job targets, and add multiple approval stages or list approvers from each team, with the option to require all listed approvers rather than just one of them — Azure DevOps also supports sequential vs. any-order approval, which is the extra knob to check if the two teams' sign-off needs to happen in a specific order rather than either order.

The underlying pattern is identical across all three: attach the approval requirement to the deployment target (an environment), not to the pipeline definition itself, since that's what lets the same approval rule apply consistently no matter which pipeline or branch is trying to deploy to production. The trap to avoid on any of the three: setting the required count to two without checking whether the platform's default lets the same person satisfy it twice under two different accounts, or whether one team's lead approving on behalf of both teams technically satisfies a "two approvals" rule that was meant to mean two independent teams — the count needs to be paired with reviewer scoping (require this rule to come from group A and a separate rule from group B), not just a raw number.

The core problem is that a naive migration — change the schema, then deploy the new code that expects it — has a window where either the old code is running against the new schema, or the new code is running against the old schema, and either combination can break requests in flight.

The standard technique is expand and contract, run across at least two separate deploys instead of one:

1. Expand. Add the new schema element (a new column, a new table) without removing or renaming anything the old code still uses. Deploy this migration first, on its own — at this point both the old and new application code can run against the database unchanged, because nothing it previously depended on has gone away.
2. Migrate and dual-write (if needed). Deploy application code that writes to both the old and new structure, and backfill existing data into the new structure, so at every moment the data required by both code versions is present and correct.
3. Cut over. Deploy application code that reads and writes only the new structure, once you're confident the backfill is complete and correct.
4. Contract. Only once every instance of the old code is confirmed gone — no more requests failing over to a version that expects the old column — deploy a final migration that removes the now-unused old structure.

Doing all of this as one migration plus one deploy is the trap: it assumes every server switches from old code to new code atomically, which isn't true during a rolling deployment, where old and new instances briefly serve traffic side by side. A pipeline that runs the schema migration as a separate, earlier step from the application deploy — and treats "add" and "remove" as separate releases — is what actually avoids a downtime window, rather than just hoping the rollout is fast enough that nobody notices.

The honest framing for this question isn't "automate everything you can" — it's a genuine trade-off between speed and risk, and a good answer shows you can reason about where that line sits rather than reciting that automation is always better.

A few practical criteria that tend to separate the two:

• Reversibility. Steps that are easy and fast to undo (redeploying a previous artifact, rolling forward a small fix) are strong automation candidates. Steps that are hard or impossible to reverse (an irreversible data migration, deleting a resource, a one-way schema contraction) are strong candidates for a manual checkpoint, precisely because the cost of an automated mistake is much higher than the time saved.
• Blast radius. A deploy to an internal staging environment used by five engineers has a small blast radius if it's wrong; a deploy to production affecting every customer has a large one. Teams commonly automate deployment fully up through staging, and add a manual approval gate specifically at the production boundary, rather than applying the same policy everywhere.
• Confidence in the safety net. Full automation to production is reasonable once you trust the test suite, monitoring, and rollback path enough that a bad deploy gets caught and reversed automatically before most users are affected — which is really the same prerequisite that separates continuous delivery from continuous deployment. Where that confidence doesn't exist yet, a manual gate is a deliberate, temporary substitute for automated safety, not laziness.
• Regulatory or compliance requirements. Some industries require a documented human sign-off before specific changes reach production regardless of how much you trust the automation — this isn't a technical decision the pipeline can override.

The trap worth naming: treating "we should automate this too" as always the right next step. A team that removes a manual approval gate because it feels slow, without first building the monitoring and fast-rollback capability that gate was implicitly covering for, tends to find out why the gate existed the hard way.