The Platform Engineering Challenge

Extend the Backstage golden path so one form provisions AWS with Terraform, deploys with Argo CD, and enforces policy and cost tracking.

Architecture Overview & Problem Statement

This capstone extends the golden path from the Mini IDP so that one Backstage form provisions everything a service needs. A GitHub Actions workflow runs Terraform to create the namespace, resource quota, ECR repository, and an IAM role for the service. The repository, CI pipeline, and manifests come from the skeleton, Argo CD deploys them, Kyverno blocks non-compliant workloads at admission, Prometheus scrapes the service, and Kubecost shows what the namespace costs. You finish with a working platform, three unguided challenges, and a repository you can walk an interviewer through.

This is the third and last of the connected platform projects. It starts from the template and ApplicationSet you built in Build a Mini Internal Developer Platform, which itself needs the portal from Build an Internal Developer Portal with Backstage. It also needs an EKS cluster with Prometheus, as built in Build Production AWS Infrastructure with Terraform. The earlier projects each build one layer. This one adds the pieces that turn a template into a governed platform: infrastructure provisioning, policy, and cost.

What you build:

A Terraform module and a GitHub Actions workflow that provision per-service resources on demand
Kyverno policies that apply only to platform-managed namespaces
An extended skeleton with an HPA, a PodDisruptionBudget, network policies, and a ServiceMonitor
An updated ApplicationSet that waits for provisioning instead of failing
Cost visibility per namespace, and three challenge scenarios

Who does what

The central design rule is that Backstage orchestrates and never executes. Each system does the work it is best at.

System	Owns	Why
Backstage	The form, repository creation, catalog entry	It is the front door, so it holds no cloud credentials
GitHub Actions + Terraform	Namespace, quota, ECR, IAM role	Per-service state, audit log, and temporary credentials through OIDC
Argo CD	Deploying what is in Git	Every deployment comes from Git and is drift-checked
Kyverno	Rejecting non-compliant workloads	It runs at admission, so kubectl, CI, and Argo CD all face the same rules
Prometheus, Kubecost	Metrics and cost	They observe, and never block

Long-lived platform pieces such as the cluster and VPC stay in their own Terraform state. Per-service resources use a separate state per service, so one service's apply can never lock or corrupt another's.

flowchart LR ARGO[Argo CD sync] --> KYV{Kyverno
admission} KYV -->|compliant| WL[Workload
in cluster] KYV -->|violation| DENY[Sync fails
with policy name] WL --> PROM[Prometheus
scrapes] WL --> KC[Kubecost
cost by namespace] classDef gate fill:#EF4444,color:#fff classDef ok fill:#10B981,color:#fff classDef obs fill:#8B5CF6,color:#fff classDef cd fill:#326CE5,color:#fff class KYV,DENY gate class WL ok class PROM,KC obs class ARGO cd

Tip
Build and test each layer on its own before wiring them together. If you connect the template, Terraform, Argo CD, and Kyverno at once and it fails, every part is a suspect. The milestones below are ordered so each piece can be checked alone.

Milestone 1: Check the Prerequisites and Set Up Access

This milestone confirms the foundations and creates the access that GitHub Actions needs. A failure in a later step is much easier to read when you know this part works.

You need:

The Mini IDP running: Backstage with the Node.js template and the golden-path-services ApplicationSet from that project
An EKS cluster with Argo CD, the Prometheus Operator stack, and metrics-server (the HPA needs it)
A repository named platform-infrastructure in your GitHub organisation
Two IAM roles that trust GitHub through OIDC, with no stored AWS keys: a Terraform role for the platform-infrastructure repository, and a deploy role for service repositories, whose trust policy allows repo:YOUR_ORG/* and whose permissions cover ECR push
An EKS access entry that lets the Terraform role create namespaces, quotas, and limit ranges
Repository variables in platform-infrastructure: TERRAFORM_ROLE_ARN, TF_STATE_BUCKET, EKS_CLUSTER_NAME; and an organisation variable AWS_DEPLOY_ROLE_ARN

Bash

kubectl get nodes
kubectl get applicationset golden-path-services -n argocd
kubectl top nodes
# Expected: nodes Ready, the ApplicationSet exists, and top returns numbers

Security
A deploy role trusted by repo:YOUR_ORG/* is fine for a lab, because any repository in your organisation could assume it. In production, scope each service to its own role, and give the Backstage GitHub token only the access it needs, ideally through a GitHub App.

Milestone 2: Provision Per-Service Resources with Terraform

This milestone builds the module and workflow that Backstage will trigger. Terraform creates the parts of a service that must be controlled by the platform and not by the team: the namespace with its quota, the image repository, and a least-privilege IAM role.

Create services/namespace/main.tf in platform-infrastructure. Backend settings are passed at init time so each service gets its own state file.

HCL

terraform {
  required_version = ">= 1.11.0"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
 
  backend "s3" {}
}
 
variable "service_name" {
  type = string
  validation {
    condition     = can(regex("^[a-z][a-z0-9-]*[a-z0-9]$", var.service_name))
    error_message = "Use lowercase letters, numbers and hyphens, starting with a letter."
  }
}
 
variable "team" {
  type = string
}
 
variable "cluster_name" {
  type = string
}
 
provider "aws" {
  region = "ap-south-1"
}
 
data "aws_eks_cluster" "platform" {
  name = var.cluster_name
}
 
data "aws_eks_cluster_auth" "platform" {
  name = var.cluster_name
}
 
data "aws_caller_identity" "current" {}
 
# Without this block the kubernetes resources below cannot reach the cluster
provider "kubernetes" {
  host                   = data.aws_eks_cluster.platform.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.platform.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.platform.token
}
 
resource "kubernetes_namespace" "service" {
  metadata {
    name = var.service_name
    labels = {
      "platform.devops-network.io/managed" = "true"
      "platform.devops-network.io/service" = var.service_name
      "team"                               = var.team
    }
  }
}
 
resource "kubernetes_resource_quota" "service" {
  metadata {
    name      = "service-quota"
    namespace = kubernetes_namespace.service.metadata[0].name
  }
  spec {
    hard = {
      "pods"            = "20"
      "requests.cpu"    = "4"
      "requests.memory" = "8Gi"
      "limits.cpu"      = "8"
      "limits.memory"   = "16Gi"
    }
  }
}
 
# Gives pods sensible defaults so quota and cost attribution always work
resource "kubernetes_limit_range" "service" {
  metadata {
    name      = "service-defaults"
    namespace = kubernetes_namespace.service.metadata[0].name
  }
  spec {
    limit {
      type = "Container"
      default_request = {
        cpu    = "100m"
        memory = "128Mi"
      }
      default = {
        cpu    = "500m"
        memory = "512Mi"
      }
    }
  }
}
 
resource "aws_ecr_repository" "service" {
  name                 = var.service_name
  image_tag_mutability = "IMMUTABLE"
  force_delete         = true # lab only, so teardown works
 
  image_scanning_configuration {
    scan_on_push = true
  }
 
  tags = {
    Service   = var.service_name
    Team      = var.team
    ManagedBy = "terraform"
  }
}
 
# CI tags images with commit SHAs, so keep the newest 20 of any tag
resource "aws_ecr_lifecycle_policy" "service" {
  repository = aws_ecr_repository.service.name
  policy = jsonencode({
    rules = [{
      rulePriority = 1
      description  = "Keep the last 20 images"
      selection = {
        tagStatus   = "any"
        countType   = "imageCountMoreThan"
        countNumber = 20
      }
      action = { type = "expire" }
    }]
  })
}
 
locals {
  oidc_issuer = replace(data.aws_eks_cluster.platform.identity[0].oidc[0].issuer, "https://", "")
}
 
data "aws_iam_policy_document" "irsa_trust" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.oidc_issuer}"]
    }
    condition {
      test     = "StringEquals"
      variable = "${local.oidc_issuer}:sub"
      values   = ["system:serviceaccount:${var.service_name}:${var.service_name}"]
    }
  }
}
 
resource "aws_iam_role" "service" {
  name               = "${var.service_name}-irsa"
  assume_role_policy = data.aws_iam_policy_document.irsa_trust.json
}
 
# The service may read only its own secrets
resource "aws_iam_role_policy" "own_secrets" {
  name = "${var.service_name}-own-secrets"
  role = aws_iam_role.service.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["secretsmanager:GetSecretValue"]
      Resource = "arn:aws:secretsmanager:*:${data.aws_caller_identity.current.account_id}:secret:services/${var.service_name}/*"
    }]
  })
}
 
output "ecr_repository_url" {
  value = aws_ecr_repository.service.repository_url
}
 
output "irsa_role_arn" {
  value = aws_iam_role.service.arn
}

The IAM role is for the application's own AWS calls, such as reading a secret. It is not for pulling the image: the cluster nodes pull images from ECR using the node role, and a pod's IRSA role plays no part in that.

Create .github/workflows/provision-service.yml. The inputs go through environment variables instead of being pasted into shell commands, so a crafted input cannot inject commands into the runner.

YAML

name: Provision Service
 
on:
  workflow_dispatch:
    inputs:
      service_name:
        description: 'Service name'
        required: true
        type: string
      team:
        description: 'Owning team'
        required: true
        type: string
      action:
        description: 'apply or destroy'
        required: true
        type: choice
        options:
          - apply
          - destroy
        default: apply
 
# Two runs for the same service must never overlap
concurrency:
  group: provision-${{ inputs.service_name }}
  cancel-in-progress: false
 
permissions:
  id-token: write
  contents: read
 
jobs:
  terraform:
    runs-on: ubuntu-latest
    env:
      TF_VAR_service_name: ${{ inputs.service_name }}
      TF_VAR_team: ${{ inputs.team }}
      TF_VAR_cluster_name: ${{ vars.EKS_CLUSTER_NAME }}
      ACTION: ${{ inputs.action }}
      STATE_BUCKET: ${{ vars.TF_STATE_BUCKET }}
    defaults:
      run:
        working-directory: services/namespace
    steps:
      - uses: actions/checkout@v4
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.TERRAFORM_ROLE_ARN }}
          aws-region: ap-south-1
 
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: '^1.11.0'
 
      - name: Terraform init
        run: |
          terraform init \
            -backend-config="bucket=$STATE_BUCKET" \
            -backend-config="key=services/$TF_VAR_service_name/terraform.tfstate" \
            -backend-config="region=ap-south-1" \
            -backend-config="use_lockfile=true"
      - name: Terraform plan
        run: |
          FLAG=""
          if [ "$ACTION" = "destroy" ]; then FLAG="-destroy"; fi
          terraform plan $FLAG -out=tfplan
      - name: Terraform apply
        run: terraform apply -auto-approve tfplan

Test this workflow by hand before Backstage touches it: run it from the Actions tab for a throwaway service name and confirm the namespace, quota, ECR repository, and role appear. Then run it again with destroy.

Common Mistake
Putting the cluster, VPC, and every service namespace in one Terraform state. A new service's apply then locks the whole platform, and one corrupted state file affects everything. One state per service keeps failures contained.

Milestone 3: Enforce Policy with Kyverno

This milestone adds admission-time rules. Kyverno runs as a Kubernetes admission webhook, so every resource is checked whether it arrives from Argo CD, CI, or a person with kubectl. Running the same checks only in CI can be bypassed by anyone who applies a manifest directly.

Bash

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm search repo kyverno/kyverno
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --version <version from the search>
kubectl get pods -n kyverno

The policies below apply only to namespaces carrying the managed label that Terraform sets, so they never touch system namespaces. Create policies/platform-policies.yaml:

YAML

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-run-as-non-root
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: run-as-non-root
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaceSelector:
                matchLabels:
                  platform.devops-network.io/managed: "true"
      validate:
        message: "Pods must set securityContext.runAsNonRoot to true."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: cpu-memory-limits
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaceSelector:
                matchLabels:
                  platform.devops-network.io/managed: "true"
      validate:
        message: "Every container must set CPU and memory limits."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    cpu: "?*"
                    memory: "?*"
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: no-latest
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaceSelector:
                matchLabels:
                  platform.devops-network.io/managed: "true"
      validate:
        message: "Use an explicit image tag, never latest."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Bash

kubectl apply -f policies/platform-policies.yaml
kubectl get clusterpolicy

Kyverno also generates the same checks for Deployments from these Pod rules, so a bad Deployment is rejected when Argo CD applies it, not later when a pod fails to start. Field names change between Kyverno versions, so check the current policy reference if validationFailureAction is reported as deprecated.

Test it alone: create a namespace with the managed label and try to apply a Pod that runs as root. It must be rejected with the policy name in the message.

Security
Never fix a rejected deployment by turning the policy off. Fix the manifest. Use a documented PolicyException only for a justified, narrow case, and write the reason next to it.

Milestone 4: Extend the Skeleton with Platform Guardrails

This milestone upgrades the Mini IDP skeleton so every new service ships with the reliability and observability defaults. Copy the Mini IDP template to templates/platform-engineering-challenge/ and change the files below. The Dockerfile, package.json, catalog-info.yaml, docs, and src/index.js carry over, except that the metrics counter should use the label status and the request counter should be named http_requests_total so SLO queries work later.

The template passes these values: name, description, owner, team, instanceSize, awsAccountId, awsRegion, ecrRegistry, and destination.

k8s/deployment.yaml pulls from ECR and sizes resources from the form. Backstage templates use {% %} for conditions, not ${{ }}.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${{ values.name }}
  labels:
    app: ${{ values.name }}
    team: ${{ values.team }}
spec:
  selector:
    matchLabels:
      app: ${{ values.name }}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: ${{ values.name }}
        team: ${{ values.team }}
    spec:
      serviceAccountName: ${{ values.name }}
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
      containers:
        - name: ${{ values.name }}
          image: ${{ values.ecrRegistry }}/${{ values.name }}
          ports:
            - name: http
              containerPort: 4000
          securityContext:
            allowPrivilegeEscalation: false
{% if values.instanceSize == 'small' %}
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits: { cpu: 500m, memory: 512Mi }
{% elif values.instanceSize == 'medium' %}
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { cpu: 1000m, memory: 1Gi }
{% else %}
          resources:
            requests: { cpu: 500m, memory: 512Mi }
            limits: { cpu: 2000m, memory: 2Gi }
{% endif %}
          livenessProbe:
            httpGet:
              path: /health/live
              port: 4000
            initialDelaySeconds: 15
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 4000
            initialDelaySeconds: 10
            periodSeconds: 5

k8s/serviceaccount.yaml carries the IAM role the Terraform module created. Its name is predictable, so the template needs no outputs from Terraform:

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${{ values.name }}
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${{ values.awsAccountId }}:role/${{ values.name }}-irsa

k8s/service.yaml, k8s/hpa.yaml, k8s/pdb.yaml, and k8s/servicemonitor.yaml follow. The Service must carry the app label and name its port http, or the ServiceMonitor finds nothing.

YAML

apiVersion: v1
kind: Service
metadata:
  name: ${{ values.name }}
  labels:
    app: ${{ values.name }}
spec:
  selector:
    app: ${{ values.name }}
  ports:
    - name: http
      port: 4000
      targetPort: 4000
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ${{ values.name }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ${{ values.name }}
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ${{ values.name }}
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: ${{ values.name }}
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ${{ values.name }}
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: ${{ values.name }}
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

k8s/networkpolicies.yaml isolates ingress to the namespace and allows only the callers that must reach the service. Prometheus runs in another namespace, so it needs its own rule, or scraping silently stops. Network policies are only enforced if your CNI supports them, so confirm that for your cluster.

YAML

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring
spec:
  podSelector:
    matchLabels:
      app: ${{ values.name }}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 4000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
spec:
  podSelector:
    matchLabels:
      app: ${{ values.name }}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - port: 4000

This baseline restricts ingress only. Restricting egress as well is a good hardening step, but it breaks calls to databases and AWS APIs unless you allow them explicitly.

k8s/kustomization.yaml lists every resource and holds the image tag:

YAML

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - serviceaccount.yaml
  - deployment.yaml
  - service.yaml
  - hpa.yaml
  - pdb.yaml
  - networkpolicies.yaml
  - servicemonitor.yaml
images:
  - name: ${{ values.ecrRegistry }}/${{ values.name }}
    newTag: bootstrap

.github/workflows/ci.yml builds to ECR. The first push to a new repository can start CI before Terraform has created the ECR repository, so the job waits for it. Wrap GitHub expressions in {% raw %} so Backstage does not try to fill them in.

YAML

name: CI
 
on:
  push:
    branches: [main]
    paths-ignore:
      - 'k8s/**'
 
permissions:
  id-token: write
  contents: write
 
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: {% raw %}${{ vars.AWS_DEPLOY_ROLE_ARN }}{% endraw %}
          aws-region: ${{ values.awsRegion }}
 
      - name: Wait for the ECR repository
        run: |
          for i in $(seq 1 40); do
            if aws ecr describe-repositories --repository-names ${{ values.name }} >/dev/null 2>&1; then
              exit 0
            fi
            echo "Waiting for provisioning to finish..."
            sleep 15
          done
          echo "ECR repository never appeared"
          exit 1
      - uses: aws-actions/amazon-ecr-login@v2
 
      - name: Build and push
        run: |
          IMAGE=${{ values.ecrRegistry }}/${{ values.name }}:{% raw %}${{ github.sha }}{% endraw %}
          docker build -t "$IMAGE" .
          docker push "$IMAGE"
      - name: Point the manifests at the new image
        run: |
          cd k8s
          kustomize edit set image ${{ values.ecrRegistry }}/${{ values.name }}=${{ values.ecrRegistry }}/${{ values.name }}:{% raw %}${{ github.sha }}{% endraw %}
          git config user.name "platform-bot"
          git config user.email "platform-bot@users.noreply.github.com"
          git commit -am "deploy: {% raw %}${{ github.sha }}{% endraw %}"
          git push

Common Mistake
Granting the pod's IRSA role ECR pull permissions and expecting that to pull images. The kubelet on the node pulls images, using the node's identity. IRSA is for what the running application itself calls in AWS.

Milestone 5: Update the ApplicationSet to Wait for Provisioning

This milestone fixes a timing problem. The namespace now comes from Terraform, so Argo CD must not create it, and it must keep retrying until Terraform has finished. Edit argocd/golden-path-services from the Mini IDP and replace its syncPolicy:

YAML

      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 10
          backoff:
            duration: 30s
            factor: 2
            maxDuration: 5m

Removing CreateNamespace=true matters. If Argo CD created the namespace first, Terraform would fail with "already exists", and the namespace would miss its quota and labels.

Bash

kubectl apply -f argocd/golden-path-services.yaml

Milestone 6: Extend the Template

This milestone adds the provisioning step. It goes first, before the repository exists, so a failure leaves nothing half-built. Backstage's github:actions:dispatch action starts the workflow but does not wait for it, which is why the CI wait loop and the Argo CD retry exist.

templates/platform-engineering-challenge/template.yaml:

YAML

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: platform-engineering-challenge
  title: Production Service on the Platform
  description: Provisions AWS and Kubernetes resources, then creates and deploys a policy-compliant service
  tags:
    - nodejs
    - kubernetes
    - recommended
spec:
  owner: group:platform-team
  type: service
 
  parameters:
    - title: Service details
      required:
        - name
        - description
        - owner
      properties:
        name:
          title: Service name
          type: string
          pattern: '^[a-z][a-z0-9-]*[a-z0-9]$'
          ui:autofocus: true
        description:
          title: Description
          type: string
        owner:
          title: Owner team
          type: string
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group
        instanceSize:
          title: Service size
          type: string
          enum: [small, medium, large]
          enumNames:
            - 'Small (100m CPU, 128Mi)'
            - 'Medium (250m CPU, 256Mi)'
            - 'Large (500m CPU, 512Mi)'
          default: small
    - title: Repository
      required:
        - repoUrl
      properties:
        repoUrl:
          title: Repository location
          type: string
          ui:field: RepoUrlPicker
          ui:options:
            allowedHosts:
              - github.com
 
  steps:
    - id: provision
      name: Provision AWS and Kubernetes resources
      action: github:actions:dispatch
      input:
        repoUrl: github.com?owner=YOUR_GITHUB_ORG&repo=platform-infrastructure
        workflowId: provision-service.yml
        branchOrTagName: main
        workflowInputs:
          service_name: ${{ parameters.name }}
          team: ${{ parameters.owner | parseEntityRef | pick('name') }}
          action: apply
 
    - id: fetch-template
      name: Render service files
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          description: ${{ parameters.description }}
          owner: ${{ parameters.owner }}
          team: ${{ parameters.owner | parseEntityRef | pick('name') }}
          instanceSize: ${{ parameters.instanceSize }}
          awsAccountId: 'YOUR_AWS_ACCOUNT_ID'
          awsRegion: 'ap-south-1'
          ecrRegistry: 'YOUR_AWS_ACCOUNT_ID.dkr.ecr.ap-south-1.amazonaws.com'
          destination: ${{ parameters.repoUrl | parseRepoUrl }}
 
    - id: publish-github
      name: Create GitHub repository
      action: publish:github
      input:
        repoUrl: ${{ parameters.repoUrl }}
        description: ${{ parameters.description }}
        repoVisibility: private
        defaultBranch: main
        topics:
          - golden-path
          - nodejs
 
    - id: register-catalog
      name: Register in catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['publish-github'].output.repoContentsUrl }}
        catalogInfoPath: '/catalog-info.yaml'
 
  output:
    links:
      - title: Repository
        url: ${{ steps['publish-github'].output.remoteUrl }}
      - title: Open in catalog
        icon: catalog
        entityRef: ${{ steps['register-catalog'].output.entityRef }}

The Backstage GitHub token needs the workflow scope to dispatch workflows. A token with that scope can trigger workflows in every repository it can reach, so use a dedicated machine account or a GitHub App.

Because private repositories are used here, keep the repo-creds secret for Argo CD from the Mini IDP, and register the template in catalog.locations as before.

Common Mistake
Building one huge template for every kind of service. Share the small universal part (security context, limits, owner, catalog entry) and give each kind of workload its own template, such as a batch job or a static site. A template that fits nobody gets bypassed, and bypassed services escape every guardrail.

Milestone 7: Add Cost Visibility

This milestone shows what each team's namespace costs. Terraform already labels each namespace with team, and Kubecost reports cost by namespace without further setup. Install Kubecost, or OpenCost, by following the current install instructions for your version, and connect it to your Prometheus.

Bash

kubectl get pods -n kubecost
kubectl port-forward -n kubecost deployment/kubecost-cost-analyzer 9091:9090 &
 
curl -s "http://localhost:9091/model/allocation?window=1h&aggregate=namespace" | jq 'keys'

Allow 15 to 30 minutes before new namespaces show meaningful numbers. To attribute cost by the team label instead of the namespace, check your version's documentation for the label mapping setting.

Tip
Cost is derived from resource requests as well as usage. A service that requests far more than it uses looks expensive and wasteful in Kubecost, which is the problem Challenge C asks you to fix.

Milestone 8: Run the Platform End to End

This milestone proves the whole chain works. Start a timer, open Create in Backstage, choose Production Service on the Platform, and fill in analytics-service, a description, an owner team, and medium size. Click Create.

Verify each link in the order it happens:

Bash

# 1. Provisioning workflow succeeded
gh run list --repo YOUR_GITHUB_ORG/platform-infrastructure --limit 3
 
# 2. Terraform created the namespace, quota and limits
kubectl get namespace analytics-service --show-labels
kubectl get resourcequota,limitrange -n analytics-service
 
# 3. The ECR repository and IAM role exist
aws ecr describe-repositories --repository-names analytics-service --region ap-south-1 \
  --query 'repositories[0].repositoryUri' --output text
aws iam get-role --role-name analytics-service-irsa --query 'Role.RoleName' --output text
 
# 4. The service repository exists and CI pushed an image
gh repo view YOUR_GITHUB_ORG/analytics-service --json name,repositoryTopics
gh run list --repo YOUR_GITHUB_ORG/analytics-service --limit 3
 
# 5. Argo CD created and synced the Application
kubectl get applications -n argocd
 
# 6. Pods run, and the HPA and PDB exist
kubectl get pods,hpa,pdb,networkpolicy -n analytics-service
 
# 7. Prometheus is scraping
curl -s "http://localhost:9090/api/v1/query" \
  --data-urlencode 'query=up{namespace="analytics-service"}' | jq '.data.result[].value[1]'
# Expected: "1"

Stop the timer when the pods are Running, and write the time down. Then open the service in Backstage and check the Kubernetes tab. Finally, push a code change and confirm the loop: CI builds a new image, a deploy: commit changes the tag, and Argo CD rolls it out.

Milestone 9: Solve Three Challenges

These scenarios have no step-by-step instructions. Use everything you have built.

Challenge A: Add a database with Crossplane

Give payment-reconciler a PostgreSQL database that is created from a Kubernetes manifest in the service repository and managed by Argo CD like any other resource.

Install Crossplane and an AWS provider, then write a CompositeResourceDefinition and a Composition that turn a small claim into an RDS instance in your private database subnets, with the connection details written to a Secret in the service's namespace. Add an enableDatabase option to the template, render the claim only when it is set (use {% if %}), and give the Deployment the Secret through envFrom.

Success looks like this:

Bash

kubectl get postgresqlinstance -n payment-reconciler
# Expected: READY True (RDS can take 5 to 10 minutes)
 
kubectl get secret payment-reconciler-db-credentials -n payment-reconciler
# Expected: the Secret exists
 
kubectl get deployment payment-reconciler -n payment-reconciler -o json \
  | jq '.spec.template.spec.containers[0].env // [] | map(select(.name == "DB_PASSWORD"))'
# Expected: [] (no password written directly in the Deployment)

Also search the repository history to confirm no credential was ever committed.

Challenge B: Break and fix a policy

Change k8s/deployment.yaml in analytics-service to runAsNonRoot: false and runAsUser: 0, and push. Observe what happens in Argo CD, find the policy name in the error, then fix the manifest and push again.

Success looks like this:

Bash

argocd app get analytics-service --show-operation
# While broken: the sync fails with an admission webhook denial naming require-run-as-non-root
 
argocd app get analytics-service
# After the fix: Synced and Healthy

The pods that were already running keep serving during the failed sync, because Kyverno rejects the new Deployment version and does not remove the old one. Explain why that is the right behaviour.

Challenge C: Cut the service's cost by at least 40 percent

analytics-service was created as medium and uses far less than it requests. Reduce its requested resources by at least 40 percent without breaking its availability.

Check actual usage with kubectl top pods -n analytics-service. Install the Vertical Pod Autoscaler in recommendation-only mode and read its suggested sizes. Update the requests in k8s/deployment.yaml, push the change through Git, and compare Kubecost's cost for the namespace before and after. Remember that the HPA keeps at least two replicas, so requests are multiplied by two.

Then confirm availability held, using traffic you generate yourself:

Bash

curl -s "http://localhost:9090/api/v1/query" --data-urlencode \
  'query=1 - (sum(rate(http_requests_total{namespace="analytics-service",status!~"2.."}[1h])) / sum(rate(http_requests_total{namespace="analytics-service"}[1h])))' \
  | jq '.data.result[0].value[1]'
# Expected: 0.999 or higher

Milestone 10: Prepare the System Design Review

These are the questions an interviewer asks about a platform like this. Answer them in your own words, from what you built, and note the weak points as well as the strengths.

How would you design this for 50 teams deploying on their own? Namespace per service with quotas, so no team can starve the rest. Policy enforced at admission, so there is no approval queue. A self-service template, so a new team needs no ticket. Cost attributed by namespace and team label. Say where this lab falls short: it uses one Argo CD project, and a real platform gives each team its own project and RBAC.
A new engineer joins. How do they get a running service? They fill in a form and the platform provisions, creates, deploys, and registers the service. Use your measured time, and name the manual steps that still exist, such as first-time repository access.
Production returns 502 errors. How do you debug? Work from the outside in: Argo CD sync and health, pod status and restarts, whether the Service has endpoints, network policies, then error-rate metrics lined up with the last change in Git.
How do you make every service meet security standards? Kyverno at admission, with the template compliant by default so a new service starts with zero violations. Be honest about the limits: a cluster admin can change webhooks, and the policies cover only the rules you wrote.

Validation & Testing

The platform is complete when all of these are true for analytics-service:

The provisioning workflow succeeded, and the namespace has its quota, limit range, and managed label
The ECR repository and IAM role exist, and the Terraform state is separate for this service
CI built an image tagged with the commit SHA and committed that tag to the manifests
Argo CD synced the Application, and it is Healthy with at least two pods Running
The HPA, PDB, and network policies exist, and Prometheus shows the service as up
Kyverno rejects a root-user Deployment in this namespace, and accepts the compliant one
Kubecost or OpenCost lists the namespace and its cost
Backstage shows the service with its owner and live Kubernetes status
Challenges A, B, and C pass their success checks

If a step fails, find the first broken link in the chain and fix it before looking further:

Bash

# Dispatch failed: is the workflow active, and does the token have the workflow scope?
gh workflow list --repo YOUR_GITHUB_ORG/platform-infrastructure
 
# Terraform failed: read the failing step, and check for a stale state lock
gh run view --repo YOUR_GITHUB_ORG/platform-infrastructure --log-failed
 
# Argo CD does not see the service: check the ApplicationSet and the repo topic
kubectl describe applicationset golden-path-services -n argocd | tail -20
 
# Sync rejected: read the denial, then fix the manifest
argocd app get analytics-service --show-operation

Clean up with the same workflow: run Provision Service with action set to destroy for each service, delete the repositories, and remove the Argo CD Applications. For a template that fails halfway, the destroy run is the rollback, which is why provisioning is the first step.

For your portfolio, include the architecture diagrams, a screen recording of the form creating a service, your measured time from form to running pods, the Kyverno denial message from Challenge B, and the cost before and after in Challenge C. Add a one-page write-up of what you would change before running this for real: GitHub App authentication, per-team Argo CD projects, per-service deploy roles, and egress policies.

Found a mistake, an outdated step or a broken link?

The Platform Engineering Challenge

Blueprint Walkthrough

Architecture Overview & Problem Statement

Who does what

Milestone 1: Check the Prerequisites and Set Up Access

Milestone 2: Provision Per-Service Resources with Terraform

Milestone 3: Enforce Policy with Kyverno

Milestone 4: Extend the Skeleton with Platform Guardrails

Milestone 5: Update the ApplicationSet to Wait for Provisioning

Milestone 6: Extend the Template

Milestone 7: Add Cost Visibility

Milestone 8: Run the Platform End to End

Milestone 9: Solve Three Challenges

Challenge A: Add a database with Crossplane

Challenge B: Break and fix a policy

Challenge C: Cut the service's cost by at least 40 percent

Milestone 10: Prepare the System Design Review

Validation & Testing