Toil is often confused with "work I dislike" or "boring administrative tasks." Neither is accurate, and the distinction matters because it changes what you are allowed to count. ### Why toil is not the same as overhead **Toil** is operational work directly tied to running a production service. **Overhead** is necessary but separate work like team meetings, HR paperwork, and status reports. Both can feel tedious, but only toil is the target of elimination efforts. Cleaning up a messy alerting configuration is grungy work, but because it produces a lasting improvement, it is not toil either, it is engineering. Google SRE defines toil by six characteristics. The more of these a task exhibits, the more confidently you can label it toil: * **Manual** - a human has to physically do it, even if it is just clicking "run" on a script * **Repetitive** - the same task recurs, not a one-off or a novel problem * **Automatable** - a machine could do it as well as a human, or the need for it could be designed away * **Tactical** - reactive and interrupt-driven, not part of a deliberate strategy * **No enduring value** - the system is in the same state after the task as before it * **Scales linearly (O(n)) with growth** - double the traffic or fleet size, and the work roughly doubles too > 📌 **Remember:** A task does not need all six traits to count as toil. The more boxes it ticks, the stronger the case for treating it as toil and prioritising its removal. ### Why running a script is still toil A common mistake is assuming that once a task is scripted, it stops being toil. It does not. If a human still has to notice the problem, decide to run the script, execute it, and watch it finish, the hands-on time spent doing that is toil time. The elapsed time saved by scripting is real progress, but it is not elimination. Consider Priya's disk-cleanup script. Running `./cleanup-tmp.sh` takes her ninety seconds instead of the ten minutes manual cleanup used to take. That is a genuine improvement. But she is still the one who has to be paged, log in, and press enter. Full elimination means the system detects the condition and remediates it with no human in the loop at all, or better, the underlying log-rotation misconfiguration is fixed so the condition never occurs. Toil Automation Maturity (least eliminated -> most eliminated) +------------------+ +------------------+ +------------------+ +------------------+ | Fully Manual | -> | Human-Triggered | -> | Auto-Remediated | -> | Root Cause Fixed | | SSH + fix by hand| | Script exists, | | System detects | | Problem cannot | | every time | | human runs it | | + fixes itself | | recur at all | +------------------+ +------------------+ +------------------+ +------------------+ still 100% toil still toil, less time near-zero toil zero toil This progression matters because teams often stop at stage two and declare victory. Stage two is worth doing, but stage three or four is the actual goal. > 🔴 **Common Mistake:** Treating a partially automated script as "solved." If a human still has to trigger it, you have reduced the duration of toil, not the existence of it. Keep pushing toward self-triggering remediation or a root-cause fix.
Not everything that feels tedious is toil, and mislabeling work in either direction breaks your measurement and your team's trust in it. ### The three buckets every task falls into * **Engineering** - novel, requires human judgment, produces a permanent improvement, guided by strategy. Writing a Kubernetes operator, redesigning an alerting pipeline, or building a self-service provisioning API all count. * **Toil** - manual, repetitive, tactical, devoid of enduring value, tied directly to running the service. * **Overhead** - necessary administrative work not directly tied to running the service: sprint planning, performance reviews, hiring, training. A useful gut check, sometimes called the one-line test: could a machine do this exactly as well as a human, does completing it leave the system unchanged, and does the volume of this work grow in lockstep with your fleet or traffic? Three yeses is a strong toil signal. ### Migrations are the trickiest edge case Engineers instinctively call a one-time database migration "project work" because it will only happen once. But halfway through migrating six hundred services from one message queue to another, the actual daily work, writing adapter code, running the cutover script, verifying, rolling back on failure, is repetitive, mostly mechanical, and produces the same business value the old system already provided. It walks and talks like toil even though the initiative as a whole is a project. > 💡 **Tip:** When a "project" involves the same five steps repeated across dozens or hundreds of targets, treat the repeated execution as toil for measurement purposes, even while the overall initiative is planned and tracked as engineering work. ### Is all toil bad? No. Small amounts of toil are not a crisis. Predictable, repetitive tasks can be calming, they give a quick sense of accomplishment, and some engineers genuinely enjoy them. Onboarding engineers often learn a system's real behaviour by doing its toil first. The problem is toil in excess, not toil in existence. > 📌 **Remember:** The goal is not zero toil. The goal is toil that stays under control, typically capped at around half of an SRE's time, so the other half can go toward the engineering that shrinks toil further. ### Borderline cases that trip up beginners The three-bucket model is simple in theory and genuinely confusing in practice. These pairs are where most classification mistakes happen. | Task | Classification | Why | | :--- | :--- | :--- | | Manually restarting a failed pod every day | Toil | Manual, repetitive, no lasting improvement | | Writing the automation that restarts it | Engineering | Novel, produces a permanent capability | | Running that automation by hand every day | Still toil | A human still has to trigger it each time | | A one-time production migration | Depends | Toil if the daily execution is repetitive and mechanical; engineering if you are treating the overall initiative | | Responding to an incident | Usually not toil | Diagnosis requires judgment and is not identical each time | | Writing a runbook | Engineering | Produces a lasting, reusable artifact | | Following the same runbook step-by-step every week | Potential toil | The judgment has already been removed; only mechanical execution remains | The incident response row deserves emphasis because it is the most common misclassification. Incident response itself is not automatically toil, real incidents differ from each other and demand genuine diagnostic judgment. What is toil is the repetitive, mechanical sub-steps that show up inside many incidents: manually pulling the same three dashboards, running the same restart command, or copy-pasting the same status update template. Automate those sub-steps without trying to automate the judgment call of "is this actually the same problem as last time."
You cannot reliably reduce what you have not measured, and "it feels like we're drowning in ops work" does not survive a budget conversation with your engineering director. ### Why intuition alone fails Different engineers on the same team routinely disagree about how bad toil is, because everyone remembers their own painful pages more vividly than their teammates'. Toil-reduction projects can take quarters, during which priorities and personnel shift. Without an objective number, the project loses its justification the moment someone asks "are we sure this is worth it?" ### Choosing a unit of measurement Pick something objective, consistent, and already understood by your team. Good choices include: * Minutes or hours spent per week (the most universal choice) * Completed tickets of a specific category * Manual production changes executed * Pages or alerts responded to, weighted by average resolution time Whatever unit you choose, make the tracking itself lightweight. If measuring toil becomes its own toil, engineers will stop doing it within a month. ### A practical toil audit script Below is a starting point for classifying a week of on-call activity from ticket or page metadata. Adjust the keyword lists to match your own ticket taxonomy. ```python import csv from collections import Counter from datetime import datetime ## Keywords that suggest a ticket is toil versus engineering versus overhead. ## Tune these against your own historical ticket titles for accuracy. TOIL_KEYWORDS = [ "restart", "disk full", "clear logs", "manual failover", "provision vm", "reset password", "rotate cert", "drain node" ] OVERHEAD_KEYWORDS = ["1:1", "standup", "planning", "review", "onboarding doc"] def classify_ticket(title): """ Classify a single ticket title into toil, overhead, or engineering using simple keyword matching. This is a first-pass heuristic, not a substitute for a human sanity check on the results. """ lowered = title.lower() if any(keyword in lowered for keyword in TOIL_KEYWORDS): return "toil" if any(keyword in lowered for keyword in OVERHEAD_KEYWORDS): return "overhead" return "engineering" def audit_toil(csv_path): """ Read a CSV export of tickets (columns: title, minutes_spent, closed_at) and produce a toil percentage for the period covered by the file. """ totals = Counter() with open(csv_path, newline="", encoding="utf-8") as handle: reader = csv.DictReader(handle) for row in reader: category = classify_ticket(row["title"]) # Sum minutes spent per category, not just ticket counts, # since a 5-minute toil ticket and a 3-hour one are not equal totals[category] += int(row["minutes_spent"]) total_minutes = sum(totals.values()) toil_fraction = totals["toil"] / total_minutes if total_minutes else 0 print(f"Toil: {totals['toil']} min ({toil_fraction:.0%})") print(f"Engineering: {totals['engineering']} min") print(f"Overhead: {totals['overhead']} min") return toil_fraction if __name__ == "__main__": audit_toil("oncall_tickets_week32_prod-mumbai.csv") ``` > **Note:** `Counter` from the `collections` module is a dictionary subclass built for tallying totals by key. Here it accumulates minutes spent per category (toil, overhead, engineering) as the script loops through every ticket row. Run this weekly against real on-call data for a quarter before drawing conclusions. A single bad week during a major incident will skew the number badly. > 🔴 **Common Mistake:** Treating keyword classification as an accurate measurement of toil. It is a cheap first-pass heuristic, nothing more. A ticket titled "restart payment-worker" might be routine toil, or it might be part of a genuine incident investigation that happens to include a restart. Keyword matching cannot tell the difference. Treat this script's output as a starting baseline, then have a human spot-check a sample of the classified tickets each week, correct the keyword lists based on what you find, and only trust the trend line once a few weeks of human-validated data agree with it. Real toil measurement is a cycle: classify, sample, validate, refine, repeat, not a single script run. ### The 50% rule and why it exists Google's SRE organisation caps toil at roughly 50% of an SRE's time, with the other half reserved for engineering work that reduces future toil or improves the service. This is not an arbitrary number pulled from a productivity book. It exists because toil expands to fill all available time if left unmanaged, and because SRE teams that quietly become 100% operational stop being SRE teams and start being traditional ops teams with a fancier title. There is also a practical floor. If your team runs a six-person on-call rotation with one week of primary and one week of secondary on-call per cycle, the arithmetic already puts a lower bound of roughly 33% of each person's time into interrupt-driven work before any additional toil is counted. An eight-person rotation lowers that floor to about 25%. Know your rotation's floor before you set a target, or you will chase an unreachable number. > 🔴 **Common Mistake:** Setting a toil target of 0%. Some interrupt-driven work is structurally unavoidable if you run on-call at all. Aim for a sustainable ceiling, not an impossible floor. ### Why toil and error budgets are the same conversation Toil and reliability are not separate topics, they feed each other in a loop worth internalising before you look at elimination tactics. Toil consumes engineering capacity | v Less time for reliability engineering | v Recurring failures take longer to fix at the root | v Higher operational risk, which generates more toil | v (loop repeats, and gets worse each cycle) Breaking the loop starts with knowing which toil is worth eliminating first. Your SLO and error budget answer that question directly: a service comfortably inside its error budget can tolerate some rough edges, so the toil around it is lower priority. A service that is burning its error budget fast is where toil, especially interrupt-driven, tactical toil, is actively costing you reliability right now, and it should jump to the front of your elimination backlog. Use the error budget as your prioritisation signal, not just a release-gating number.
Once you can measure toil, the next question is what to actually do about it. Google's SRE teams use a consistent playbook, and the order below reflects roughly how much leverage each tactic gives you. ### Engineer the toil out of the system The highest-leverage move is not building better tooling around a toil-generating system, it is changing the system so the toil-generating condition cannot occur. If pods keep OOMKilling because a memory limit was copy-pasted from an unrelated service, the fix is not a runbook for restarting them, it is correcting the limit and adding a VPA recommendation to prevent drift. ### Reject the toil Before automating a toil-heavy process, ask whether the underlying request should be honoured at all. Teams often inherit toil because saying yes was easier than pushing back. Rejecting or batching low-value requests, so five similar tickets get handled together instead of five separate context switches, reduces total toil without writing a line of automation. ### Use error budgets to decide what deserves attention If a service is comfortably within its SLO error budget, some categories of operational toil, like an alert for a transient blip that self-resolved, can reasonably be ignored rather than chased every time. An SLO-driven team spends effort proportional to actual user impact, not proportional to how loudly a monitoring system shouts. ### Start with a human behind the curtain, then remove the human For complex requests with many edge cases, jumping straight to full self-service is risky. A workable middle stage accepts structured input through an API or form, but still has an engineer review and execute the resulting action. As request patterns stabilise, you progressively automate the reviewed steps until the human step disappears entirely. ### Build self-service so the request never reaches a human The end state for most business-process toil is a self-service portal, script, or pull-request-based workflow. Instead of filing a ticket asking an SRE to provision a staging namespace, an engineer runs a CLI or opens a PR against a config repository, and automation handles the rest. ```bash ## Self-service namespace provisioning replaces a ticket-driven request. ## Engineer runs this instead of filing a ticket and waiting for an SRE. sre-cli namespace create \ --name checkout-service-staging \ --team payments \ --quota-cpu 4 \ --quota-memory 8Gi \ --owner rahul@company.com ## used for audit trail and Slack notifications ``` > **Note:** A CLI like this typically wraps a Kubernetes `ResourceQuota` and `Namespace` creation, plus an RBAC binding, behind one command. The engineer never touches `kubectl apply` directly, and the SRE team never files or triages a ticket for it. ### GitOps turns operational changes into reviewable, auditable pull requests Instead of an SRE manually editing a config map or Helm values file on request, all operational changes flow through Git. A developer opens a pull request against the infrastructure repository; CI validates it; a reviewer approves it; a controller like Argo CD or Flux applies it to the cluster automatically. GitOps Change Flow (replaces manual apply-on-request toil) Developer CI Pipeline Reviewer GitOps Controller opens PR -> validates yaml -> approves -> detects change to repo + runs policy merge PR in Git, applies checks to cluster | v Cluster now matches desired state in Git This removes the SRE from the critical path for routine changes while keeping a full audit trail: every change to production has a corresponding commit, author, and review. > 💡 **Tip:** GitOps does not remove the need for review, it moves review from an ad hoc Slack message to a structured, versioned pull request that anyone can inspect months later. ### Self-healing operators close the loop entirely A Kubernetes operator extends the control plane's reconcile loop to your own domain-specific logic. Instead of a human noticing that a custom resource is unhealthy and fixing it by hand, the operator watches continuously and remediates automatically. Blindly deleting every crash-looping pod is not safe automation. CrashLoopBackOff has many root causes, a bad image, a missing secret, an unavailable dependency, a resource limit that is too low, a schema mismatch, and only some of those are fixed by a restart. Automation that restarts indiscriminately just adds churn on top of the real problem. Safe automation follows four steps in order: detect the condition, classify why it is happening, check whether it is safe to act, then either remediate a known-safe failure or escalate to a human. ```python """ Self-healing reconcile loop that only auto-remediates a known-safe CrashLoopBackOff cause (OOMKilled) and escalates everything else to a human instead of guessing. """ import time from kubernetes import client, config, watch ## Only these termination reasons are considered safe to auto-remediate. ## Anything else (ImagePullBackOff, config errors, app bugs) escalates. SAFE_TO_AUTO_RESTART = {"OOMKilled"} MAX_AUTO_RESTARTS_PER_POD = 2 # guardrail: stop after repeated failures def reconcile_crashlooping_pods(namespace, restart_threshold_minutes=10): """ Watch pods in the given namespace. Classify each CrashLoopBackOff by its last termination reason. Only auto-remediate reasons known to be safely fixed by a restart; escalate everything else. """ config.load_incluster_config() # runs inside the cluster, uses the pod's service account v1 = client.CoreV1Api() w = watch.Watch() for event in w.stream(v1.list_namespaced_pod, namespace=namespace): pod = event["object"] for container in pod.status.container_statuses or []: waiting = container.state.waiting if not (waiting and waiting.reason == "CrashLoopBackOff"): continue age_minutes = _minutes_since(pod.metadata.creation_timestamp) if age_minutes <= restart_threshold_minutes: continue # Classify: why did the container actually terminate last time? last_reason = _get_last_termination_reason(container) restart_count = container.restart_count if last_reason in SAFE_TO_AUTO_RESTART and restart_count <= MAX_AUTO_RESTARTS_PER_POD: # Safety check passed and cause is known-safe: remediate v1.delete_namespaced_pod(name=pod.metadata.name, namespace=namespace) _log_audit_event(pod.metadata.name, f"auto-restarted (reason={last_reason})") else: # Unknown cause, or already retried too many times: do not guess _escalate_to_human(pod.metadata.name, last_reason, restart_count) def _get_last_termination_reason(container): last_state = container.last_state return last_state.terminated.reason if last_state and last_state.terminated else "Unknown" def _minutes_since(timestamp): return (time.time() - timestamp.timestamp()) / 60 def _log_audit_event(pod_name, action): # Every automated action must be logged. Silent automation # is how a small bug becomes a 3 AM mystery outage. print(f"[AUDIT] {action}: {pod_name} at {time.strftime('%Y-%m-%d %H:%M:%S')}") def _escalate_to_human(pod_name, reason, restart_count): # Paging on an unknown cause is correct. Guessing is not. print(f"[ESCALATE] {pod_name} reason={reason} restarts={restart_count} - needs human review") ``` > **Note:** A reconcile loop is the core Kubernetes pattern of continuously comparing the current state of the cluster against the desired state, and taking action to close any gap. Operators apply this same pattern to custom logic beyond what built-in Kubernetes controllers handle. Notice the pattern here: detect, classify, check safety, then remediate only the known-safe case or escalate. That four-step shape is what separates SRE-grade automation from a script that just deletes things and hopes. > ⚠️ **Security:** Any automation with delete or patch permissions on production resources needs tightly scoped RBAC, a dry-run mode, a restart cap, and an audit log. An operator that restarts blindly, without classifying the cause first, can mask a real bug behind an infinite loop of "successful" auto-remediations while the underlying problem, and user impact, never actually goes away. ### Automated rollback closes the loop on deployments Combine your SLO monitoring with your deployment pipeline so that a burn-rate spike after a release triggers an automatic rollback, instead of paging a human to notice, diagnose, and manually revert. ```yaml ## Simplified Argo Rollouts analysis template. ## Automatically rolls back if error rate exceeds the SLO threshold ## during the canary window, with zero human intervention required. apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: error-rate-slo-gate spec: metrics: - name: error-rate interval: 1m successCondition: result[0] < 0.01 ## fail if error rate exceeds 1% failureLimit: 2 ## rollback after 2 consecutive failures provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{status=~"5..",app="checkout-service"}[1m])) / sum(rate(http_requests_total{app="checkout-service"}[1m])) ``` > 🔴 **Common Mistake:** Deploying automated rollback without first having reliable SLO metrics in place. Automated rollback is only as trustworthy as the signal it watches. A noisy or laggy metric will trigger false rollbacks and erode trust in the automation itself. ### Assess risk before granting automation real power Automation with production-changing permissions needs the same defensive posture a careful human operator would have: input validation, safety limits, and a default to human escalation when conditions look unsafe. A repair automation system at Google, for example, was built with a hard cap on how many devices it could drain simultaneously, specifically to prevent a single bug from taking down excessive capacity at once. * Validate all input, including input arriving from upstream automated systems, not just direct user input * Add safety thresholds, such as a maximum number of resources touched per run, that trigger a human review if exceeded * Default to escalating to a human whenever the automation encounters an unrecognised or ambiguous state * Treat even read-only automation as capable of causing load-related incidents at sufficient scale
Toil elimination projects need a way to prove they worked, beyond "it feels less painful now." The DORA (DevOps Research and Assessment) metrics give you an external, industry-benchmarked way to show impact. ### The four DORA metrics * **Deployment frequency** - how often your team successfully ships to production. Toil reduction in release shepherding directly raises this number. * **Lead time for changes** - the time from a commit landing to it running in production. Manual approval and deployment toil is usually the biggest contributor to a slow lead time. * **Change failure rate** - the percentage of deployments that cause a production incident or require a rollback. Automated rollback gates and safer deploy tooling reduce this. * **Time to restore service (MTTR)** - how long it takes to recover from a production incident. Self-healing operators and automated remediation shrink this dramatically for known failure classes. | Metric | How toil reduction can improve it | Example lever | | :--- | :--- | :--- | | Deployment frequency | Can rise when manual gatekeeping steps are removed | GitOps auto-apply on merge | | Lead time for changes | Can shrink when wait time on human approval disappears | Self-service provisioning | | Change failure rate | Can drop when bad deploys are caught before full rollout | Canary analysis with auto-rollback | | Time to restore (MTTR) | Can shrink when human detection and remediation delay is removed | Self-healing operator | These are not guaranteed outcomes, they hold only when the toil you eliminated was actually a meaningful contributor to that specific metric, and only when the automation you built is itself reliable. Badly designed automation moves these numbers in the wrong direction just as easily. A self-healing operator that restarts pods without classifying the failure first, like the naive version discussed earlier, can quietly increase MTTR by masking a root cause behind repeated "successful" auto-remediations, while an auto-rollback gate wired to a noisy metric can spike your change failure rate with false positives. Measure the DORA numbers before and after any toil-reduction change specifically to confirm the improvement actually happened, rather than assuming it did because the theory says it should. > 💡 **Tip:** Report toil elimination work in DORA terms when talking to leadership, but report the actual measured delta, not the expected one. "We cut MTTR for pod crash-loops from 25 minutes to under 2, measured over the last month" lands better than a general claim, and it protects your credibility if a later change regresses the number.
Everything in this module collapses into one decision-making tool. When you find a candidate piece of toil, walk it through this tree in order, and stop at the first branch that applies. Is this operational work tied to running the service? | No -----------> Overhead or Engineering. Not your target. | Yes | Is it repetitive (happens more than once or twice)? | No -----------> Probably not toil. A novel one-off is project work. | Yes | Can the root cause be fixed so this stops happening? | Yes ----------> Fix the root cause. Highest leverage option. | No | Can the request be rejected, delayed, or batched? | Yes ----------> Reject or simplify. Removes the work entirely. | No | Can this be exposed as self-service (CLI, form, GitOps PR)? | Yes ----------> Build self-service. Removes the human from the path. | No | Can it be automated safely (detect, classify, guardrails)? | Yes ----------> Automate with dry-run, limits, and audit logging. | No | Keep a human in the loop, but measure it and revisit quarterly. Notice that automation is the fifth option, not the first. Every branch above it is cheaper, safer, or more permanent than writing automation, and the tree forces you to rule them out in order instead of reaching for a script by default. > 📌 **Remember:** Run every toil candidate through this tree before writing a single line of automation. Fixing the root cause or rejecting the request outright is often faster to implement and impossible to misconfigure, unlike automation, which introduces a new system that itself needs monitoring, guardrails, and maintenance.
Toil is often confused with "work I dislike" or "boring administrative tasks." Neither is accurate, and the distinction ...
Not everything that feels tedious is toil, and mislabeling work in either direction breaks your measurement and your tea...
You cannot reliably reduce what you have not measured, and "it feels like we're drowning in ops work" does not survive a...
Once you can measure toil, the next question is what to actually do about it. Google's SRE teams use a consistent playbo...
Toil elimination projects need a way to prove they worked, beyond "it feels less painful now." The DORA (DevOps Research...
Everything in this module collapses into one decision-making tool. When you find a candidate piece of toil, walk it thro...
Set up a sample ticket dataset. Create a CSV file named oncallticketsweek32prod-mumbai.csv with columns title,minutesspe...
Concept What it means When to reach for it Toil Manual, repetitive, automatable, tactical work with no lasting value Use...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.