### The same model, two very different answers Here are two prompts sent to the same LLM during a production incident. **Prompt A:** ``` Our service is down. What should I do? ``` **Response A:** ``` Check if the service is running. Look at the logs. Restart if needed. Verify your configuration is correct. ``` Useless. Generic. Could apply to anything. **Prompt B:** ``` You are a senior SRE. The payment service is returning 503 errors. Error rate jumped from 0.1% to 34% at 14:32 UTC. A deployment happened at 14:28 UTC. The last 10 log lines are: [ERROR] Connection pool exhausted: max=50 current=50 [ERROR] Timeout waiting for connection after 5000ms [ERROR] DB query failed: too many connections Based on this data, identify the most likely root cause and give me the next three diagnostic commands to run. ``` **Response B:** ``` Most likely root cause: The deployment at 14:28 introduced a connection leak or increased connection demand beyond the pool limit. Next steps: 1. Check active DB connections: SELECT count(*) FROM pg_stat_activity; 2. Check pool config in deployment diff: git diff HEAD~1 -- config/ 3. Check if a rollback resolves it: kubectl rollout undo deployment/payment ``` Same model. Completely different quality. The difference is the prompt. ### What makes ops prompts different General prompts ask for opinions or explanations. Ops prompts need to: * Drive toward a specific action or diagnosis * Work with structured data — logs, metrics, error messages * Produce output an engineer can act on immediately * Avoid generic advice that wastes time during an incident The templates in this module are designed specifically for ops scenarios. ---
### Four things every ops prompt needs **1. Role** — Tell the model who it is ``` You are a senior SRE with expertise in Kubernetes and distributed systems. ``` This is not just flavour text. It shifts the model's output toward expert-level reasoning and away from beginner explanations. **2. Context** — Give it the actual data ``` The auth service has been throwing 401 errors since 09:14 UTC. CPU is at 12% (normal). Memory is at 89% (threshold: 80%). Last deployment was 3 days ago. No config changes today. Recent logs: [ERROR] JWT validation failed: key not found in cache [ERROR] Redis GET timeout after 3000ms [ERROR] JWT validation failed: key not found in cache ``` No context = generic answer. Real data = specific, actionable answer. **3. Task** — Be precise about what you want Bad: `What is wrong?` Good: `Identify the most likely root cause and list the top three diagnostic steps in order of priority.` **4. Output format** — Tell it how to structure the response ``` Respond in this format: - Root Cause: one sentence - Confidence: HIGH / MEDIUM / LOW - Next Steps: numbered list, max 5 items - What to escalate if steps fail: one sentence ``` Without a format instruction, the model will write an essay. During an incident you need a list. ---
### Template 1: Basic RCA from logs and metrics Use this when an alert fires and you have logs and some metrics. ``` You are a senior SRE. Analyze this incident and identify the root cause. SERVICE: {service_name} ALERT: {alert_name} TIME: {incident_start_time} ERROR RATE: {current_error_rate} (normal: {baseline_error_rate}) RECENT CHANGES: {recent_deployments_or_config_changes} LAST 20 LOG LINES: {paste_log_lines_here} METRICS AT TIME OF INCIDENT: - CPU: {cpu_percent}% - Memory: {memory_percent}% - Latency p99: {latency_ms}ms (normal: {normal_latency_ms}ms) - DB connections: {db_connections} (max: {db_max_connections}) Respond in this format: - Root Cause: one sentence - Confidence: HIGH / MEDIUM / LOW - Supporting Evidence: 2-3 bullet points from the data above - Next 3 Diagnostic Steps: numbered list - Escalate if: one sentence describing when to page someone ``` **Example filled in:** ``` You are a senior SRE. Analyze this incident and identify the root cause. SERVICE: payment-api ALERT: HighErrorRate503 TIME: 2024-11-14 14:32 UTC ERROR RATE: 34% (normal: 0.1%) RECENT CHANGES: Deployment at 14:28 UTC — updated connection pool config LAST 20 LOG LINES: [ERROR] Connection pool exhausted: max=50 current=50 [ERROR] Timeout waiting for connection after 5000ms [ERROR] DB query failed: too many connections [ERROR] Connection pool exhausted: max=50 current=50 [INFO] Request received: POST /payments/process [ERROR] Connection pool exhausted: max=50 current=50 METRICS AT TIME OF INCIDENT: - CPU: 18% - Memory: 45% - Latency p99: 8200ms (normal: 120ms) - DB connections: 50 (max: 50) Respond in this format: - Root Cause: one sentence - Confidence: HIGH / MEDIUM / LOW - Supporting Evidence: 2-3 bullet points from the data above - Next 3 Diagnostic Steps: numbered list - Escalate if: one sentence describing when to page someone ``` ### Template 2: RCA with no obvious cause Use this when you have symptoms but no clear smoking gun. ``` You are a senior SRE investigating a hard-to-diagnose incident. SERVICE: {service_name} SYMPTOM: {what_users_are_experiencing} STARTED: {when_it_started} AFFECTED: {percentage_of_requests_affected} WHAT WE HAVE CHECKED AND RULED OUT: - {ruled_out_1} - {ruled_out_2} - {ruled_out_3} WHAT WE KNOW: - {known_fact_1} - {known_fact_2} Generate three hypotheses for the root cause, ranked by likelihood. For each hypothesis: - What would confirm it (one specific command or check) - What would rule it out (one specific command or check) - Risk level of investigating it: LOW / MEDIUM / HIGH ``` ### Template 3: Five Whys RCA Use this after an incident is resolved to find the actual root cause, not just the symptom. ``` You are a senior SRE running a Five Whys analysis on a resolved incident. INCIDENT: {brief_description} IMMEDIATE CAUSE: {what_actually_failed} IMPACT: {what_users_experienced} DURATION: {how_long_it_lasted} RESOLUTION: {how_it_was_fixed} Run a Five Whys analysis. For each Why, state: - The Why question - The answer based on what we know - Whether this is the root cause or points to another Why Stop at the Why where the answer is a process, system design, or human decision that could be changed to prevent recurrence. Then give one concrete preventive action. ``` ---
### Template 4: Generate a runbook for a new alert Use this when you add a new alert to your monitoring system and need a runbook for it. ``` You are a senior SRE. Write a runbook for the following alert. ALERT NAME: {alert_name} SERVICE: {service_name} TRIGGER CONDITION: {what_causes_this_alert_to_fire} ENVIRONMENT: {kubernetes_version_or_cloud_provider} DEPENDENCIES: {databases_caches_or_services_this_depends_on} The runbook must include: 1. When to use this runbook (exact alert condition) 2. Immediate first check (one command, takes under 30 seconds) 3. Diagnostic steps (numbered, in order, each with the exact command) 4. Common causes and their fixes (table format) 5. When to escalate and who to page 6. How to verify the issue is resolved Write for an on-call engineer who may be unfamiliar with this service. Use exact commands, not descriptions of commands. ``` ### Template 5: Update an existing runbook Use this when a runbook is outdated or a new failure pattern was discovered. ``` You are a senior SRE. Update the following runbook based on what we learned from a recent incident. EXISTING RUNBOOK: {paste_existing_runbook_content} WHAT HAPPENED IN THE RECENT INCIDENT: {brief_incident_description} WHAT THE EXISTING RUNBOOK MISSED: {what_steps_were_wrong_or_missing} NEW INFORMATION TO INCORPORATE: {new_commands_causes_or_procedures} Update the runbook to include this new knowledge. Keep the same structure. Only change what needs changing. Mark updated sections with [UPDATED] and new sections with [NEW]. ``` ---
### Template 6: Real-time incident summary Use this during a live incident to quickly brief stakeholders or a new responder joining. ``` You are an incident commander. Write a brief incident summary for a stakeholder update. INCIDENT: {service_name} — {what_is_broken} STARTED: {time} CURRENT STATUS: {investigating / mitigating / resolved} CUSTOMER IMPACT: {what_users_are_experiencing} TIMELINE SO FAR: {time}: {event} {time}: {event} {time}: {event} Write a 4-sentence update: Sentence 1: What is broken and since when Sentence 2: What the impact is on users Sentence 3: What the team is currently doing Sentence 4: Next update time Use plain English. No technical jargon. Audience is non-technical. ``` ### Template 7: Postmortem first draft Use this after an incident is fully resolved to generate a postmortem draft. ``` You are a senior SRE writing a blameless postmortem. INCIDENT: {title} DATE: {date} DURATION: {how_long} SEVERITY: {P1/P2/P3} SERVICES AFFECTED: {list} CUSTOMERS AFFECTED: {number_or_percentage} TIMELINE: {time} — {event} {time} — {event} {time} — {event} {time} — {resolved} ROOT CAUSE: {one_sentence} RESOLUTION: {what_fixed_it} Write a blameless postmortem with these sections: 1. Summary (3 sentences) 2. Impact (who was affected, for how long, how severely) 3. Timeline (clean version of the above) 4. Root Cause Analysis (explain the chain of events, not blame) 5. What Went Well (at least 2 things) 6. What Could Be Improved (at least 3 things) 7. Action Items (table: item, owner, due date) Blameless means: describe what happened, not who failed. ``` ---
### How to chain these templates in a real incident In a real AIOps system, you do not just use one prompt. You chain them: ``` Alert fires ↓ Template 1: RCA prompt → get hypothesis ↓ Run diagnostic commands from the response ↓ Template 1 again with new data → narrow down cause ↓ Template 6: Stakeholder update → send to Slack ↓ Incident resolved ↓ Template 7: Postmortem draft → review and publish ↓ Template 4 or 5: Update runbook → prevent recurrence ``` Each template hands off to the next. The output of one becomes the input of another. This is what a real AI-assisted incident workflow looks like. ### A simple Python helper to use these templates ```python # prompt_templates.py # A simple helper to fill in and use RCA prompt templates def rca_prompt( service_name, alert_name, incident_time, error_rate, baseline_error_rate, recent_changes, log_lines, cpu, memory, latency_p99, normal_latency, db_connections, db_max ): """ Fill in the basic RCA template (Template 1) with real incident data. Returns a ready-to-send prompt string. """ logs_formatted = "\n".join(log_lines) return f"""You are a senior SRE. Analyze this incident and identify the root cause. SERVICE: {service_name} ALERT: {alert_name} TIME: {incident_time} ERROR RATE: {error_rate}% (normal: {baseline_error_rate}%) RECENT CHANGES: {recent_changes} LAST LOG LINES: {logs_formatted} METRICS AT TIME OF INCIDENT: - CPU: {cpu}% - Memory: {memory}% - Latency p99: {latency_p99}ms (normal: {normal_latency}ms) - DB connections: {db_connections} (max: {db_max}) Respond in this format: - Root Cause: one sentence - Confidence: HIGH / MEDIUM / LOW - Supporting Evidence: 2-3 bullet points from the data above - Next 3 Diagnostic Steps: numbered list - Escalate if: one sentence""" def stakeholder_update_prompt( service, issue, started, status, impact, timeline_events ): """ Fill in Template 6 for a real-time stakeholder update. timeline_events should be a list of (time, event) tuples. """ timeline = "\n".join(f"{t}: {e}" for t, e in timeline_events) return f"""You are an incident commander. Write a brief incident summary for a stakeholder update. INCIDENT: {service} — {issue} STARTED: {started} CURRENT STATUS: {status} CUSTOMER IMPACT: {impact} TIMELINE SO FAR: {timeline} Write a 4-sentence update: Sentence 1: What is broken and since when Sentence 2: What the impact is on users Sentence 3: What the team is currently doing Sentence 4: Next update time Use plain English. No technical jargon. Audience is non-technical.""" # Example usage if __name__ == "__main__": from openai import OpenAI client = OpenAI() # Build the RCA prompt with real data from your monitoring system prompt = rca_prompt( service_name="payment-api", alert_name="HighErrorRate503", incident_time="2024-11-14 14:32 UTC", error_rate=34, baseline_error_rate=0.1, recent_changes="Deployment at 14:28 UTC — updated connection pool config", log_lines=[ "[ERROR] Connection pool exhausted: max=50 current=50", "[ERROR] Timeout waiting for connection after 5000ms", "[ERROR] DB query failed: too many connections", ], cpu=18, memory=45, latency_p99=8200, normal_latency=120, db_connections=50, db_max=50 ) # Send to LLM and get the RCA response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}] ) print(response.choices[0].message.content) ``` > **Note:** These templates are not magic. They work best when you give them real data — actual log lines, real metric values, the actual deployment that happened. Vague inputs still produce vague outputs. The template structure helps, but the data you feed in is what drives quality.
The same model, two very different answers Here are two prompts sent to the same LLM during a production incident. Promp...
Four things every ops prompt needs 1. Role — Tell the model who it is This is not just flavour text. It shifts the model...
Template 1: Basic RCA from logs and metrics Use this when an alert fires and you have logs and some metrics. Example fil...
Template 4: Generate a runbook for a new alert Use this when you add a new alert to your monitoring system and need a ru...
Template 6: Real-time incident summary Use this during a live incident to quickly brief stakeholders or a new responder ...
How to chain these templates in a real incident In a real AIOps system, you do not just use one prompt. You chain them: ...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.