It is 2 AM. An engineer on the acme-payments team gets paged because the payments-api service is throwing errors at peak traffic. CloudWatch shows CPU is fine. Memory looks fine too, or does it - nobody set up memory monitoring. Twenty minutes later someone finds an EC2 instance was terminated by a script an intern ran from their laptop three hours earlier, using credentials nobody rotated. That incident touches three core observability and governance questions, plus one automation layer. Is the system healthy right now? Who did this, and when? Was the instance even configured correctly to begin with? And once you know the answer, what should happen automatically next time? One tool cannot answer all four. AWS gives you separate services because these are separate problems, and conflating them is how root causes stay hidden. * **CloudWatch** answers "is my system healthy right now" - metrics, logs, alarms * **CloudTrail** answers "who did what and when" - a full API call history * **AWS Config** answers "is this resource configured the way it should be" * **EventBridge** ties all three together - it reacts the instant something happens This module builds all four, in the order you actually reach for them during an incident: notice something is wrong, find out who caused it, check if it was ever compliant, then wire up automation so it never pages a human again.
A **metric** is a number tracked over time - CPU utilization, request count, queue depth. Most AWS services publish a default set of metrics to CloudWatch automatically, often at no additional charge - but the exact metrics available, their resolution, and their pricing vary by service, so always check the service's own CloudWatch documentation rather than assuming. Metrics are grouped into **namespaces**, one per service, the same way files are grouped into folders. EC2 metrics live in `AWS/EC2`, Lambda metrics in `AWS/Lambda`. A **dimension** is extra context attached to a metric so you can filter it down - which instance ID, which environment, which Lambda function. Each metric supports up to 30 dimensions. > 📌 **Remember:** RAM is not a default EC2 metric. AWS can see the CPU and > network activity of the hypervisor, but it cannot see inside your guest OS > to know how much memory your application is using. That requires an agent, > covered later in this module. ### Why standard monitoring gives you gaps at exactly the wrong moment EC2's default monitoring pushes metrics every 5 minutes. That is fine for capacity planning, but useless for catching a 90-second CPU spike that crashes a checkout service during a Big Billion Days sale. **Detailed Monitoring** pushes metrics every 60 seconds instead, at a small additional cost per instance. ```bash ## Enable 1-minute detailed monitoring on a running instance ## Without this flag the instance reports every 5 minutes by default aws ec2 monitor-instances --instance-ids i-0a1b2c3d4e5f67890 ``` > **Note:** "Detailed" here just means more frequent, not more metrics. You > still get the same CPU, network, and disk metrics - just sampled 5x faster. ### Custom metrics for anything AWS cannot see If AWS does not track it by default, you push it yourself with `PutMetricData` - from your application code, a cron script, or the Unified Agent. Common examples: RAM used, active DB connections in a pool, queue depth in an internal job system, business KPIs like signups per minute. ```python import boto3 from datetime import datetime cloudwatch = boto3.client("cloudwatch", region_name="ap-south-1") def publish_queue_depth(queue_name, depth): """ Push a custom metric to CloudWatch under a namespace we own. Custom namespaces should NOT start with 'AWS/' - that prefix is reserved for AWS's own service metrics. """ cloudwatch.put_metric_data( Namespace="AcmePayments/PaymentQueue", MetricData=[ { "MetricName": "QueueDepth", "Dimensions": [ {"Name": "QueueName", "Value": queue_name} ], "Value": depth, "Unit": "Count", "Timestamp": datetime.utcnow(), } ], ) ``` > **Note:** `Unit` tells CloudWatch how to label the graph axis - `Count`, > `Percent`, `Bytes`, `Seconds` are common ones. Getting this wrong does not > break anything, it just mislabels your dashboard.
A dashboard is a single screen showing the metrics that actually matter for one system, so an on-call engineer does not have to hunt across a dozen service pages during an incident. Good dashboards follow the **RED method** for anything request-driven: Rate (requests per second), Errors (error rate), Duration (latency). For anything resource-driven, like a database or a queue worker, use the **USE method**: Utilization, Saturation, Errors. * One dashboard per service or system, not one giant dashboard for everything * Put the "is it on fire" metrics (error rate, latency p99) at the top * Group related metrics together - all ALB metrics in one row, all RDS in another * Add a text widget at the top linking to the runbook for this service ```bash ## Create a dashboard from a JSON body describing widgets and layout aws cloudwatch put-dashboard \ --dashboard-name "payments-service-prod" \ --dashboard-body file://payments-dashboard.json \ --region ap-south-1 ``` > 💡 **Tip:** Build dashboards from the CLI or Terraform, not by clicking in > the console. A dashboard that lives in version control survives when > someone accidentally deletes it, and it can be copied for a new > environment in seconds.
An **alarm** watches one metric and changes state when it crosses a threshold you define. It has three states: * `OK` - metric is within the range you defined * `ALARM` - the threshold has been breached * `INSUFFICIENT_DATA` - not enough data points yet to evaluate, common right after creation The **evaluation period** controls how many consecutive breaches are needed before the alarm actually fires. A single 60-second CPU spike should not page anyone - requiring 3 consecutive breaches over 3 minutes filters out noise that would otherwise wake someone up for nothing. ```bash ## Fires only if CPU averages above 80% for 3 straight 1-min periods ## --evaluation-periods 3 with --period 60 means 3 minutes of sustained load aws cloudwatch put-metric-alarm \ --alarm-name "high-cpu-payments-prod" \ --metric-name CPUUtilization \ --namespace AWS/EC2 \ --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f67890 \ --statistic Average \ --period 60 \ --threshold 80 \ --comparison-operator GreaterThanThreshold \ --evaluation-periods 3 \ --alarm-actions arn:aws:sns:ap-south-1:123456789012:payments-oncall \ --region ap-south-1 ``` > **Note:** `--period 60` is how often CloudWatch checks the metric. > `--evaluation-periods 3` is how many checks in a row must breach before > the alarm actually fires. Together they define the 3-minute window above. ### Composite Alarms - the fix for a pager that never stops A single alarm on CPU fires on every brief spike, even ones that resolve themselves in seconds and never affect a real user. A **Composite Alarm** combines several individual alarms with AND / OR logic, and only fires when the combined condition is genuinely a problem. CW Alarm: CPU > 80% ──┐ ├──> Composite Alarm (AND) ──> SNS ──> PagerDuty CW Alarm: p99 latency high ┘ CPU alone spiking is common and often harmless. CPU high *and* latency degraded at the same time is a real incident. This single change is frequently the difference between an on-call rotation people can sustain and one that burns engineers out in three months. > 🔴 **Common Mistake:** Teams create a CloudWatch Alarm per metric per > service and route every single one straight to PagerDuty. Within a month, > engineers start ignoring pages because most of them are noise. The fix is > Composite Alarms for anything with more than one signal, and routing only > genuinely actionable alarms to a page - everything else goes to a Slack > channel instead.
**CloudWatch Logs** is where AWS stores logs from your applications and services. A **Log Group** is a container, typically one per application. A **Log Stream** inside it holds logs from one specific source - one EC2 instance, one Lambda invocation stream, one container task. EC2 sends zero logs to CloudWatch by default. You install the **Unified Agent** and give it IAM permission to write logs, covered in the next section. ### Querying logs with Logs Insights **Logs Insights** is a purpose-built query language for searching stored logs - closer to SQL than to grep. It auto-discovers fields in JSON-formatted logs and can query across multiple log groups, even multiple AWS accounts, in a single request. ```text ## Find every ERROR-level log line in the last search window fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50 ``` ```text ## Count 5xx errors per minute - useful for spotting a slow-building outage fields @timestamp, @message | filter @message like /statusCode=5/ | stats count(*) as errorCount by bin(1m) ``` > 📌 **Remember:** Logs Insights queries stored logs on demand - it is not a > live stream. For real-time reaction to a log line the instant it appears, > you need a Subscription Filter feeding Lambda, covered next. ### Metric Filters - turning raw log text into an alarmable number A **Metric Filter** scans a log group for a pattern and converts matches into a CloudWatch metric you can alarm on. This is how you get paged on "5xx errors exceeded 50 in 5 minutes" without ever writing a Logs Insights query by hand during an incident. ```bash ## Every time "statusCode=5" appears in a log line, increment a metric by 1 aws logs put-metric-filter \ --log-group-name "/ecs/payments-api-prod" \ --filter-name "5xx-error-count" \ --filter-pattern "statusCode=5" \ --metric-transformations \ metricName=PaymentApi5xxCount,metricNamespace=RazorpayOps/Payments,metricValue=1 \ --region ap-south-1 ``` Once that metric exists, put a normal CloudWatch Alarm on it - it behaves exactly like `AWS/EC2` `CPUUtilization` from this point forward. > 💡 **Tip:** Metric Filters are the cleanest way to alert on application > error rates without adding a metrics library to your app code. If your > logs are already structured, this is often less work than instrumenting > custom metrics by hand.
EC2 exposes CPU, network, and disk I/O out of the box - but never memory usage or swap, because those live inside the guest OS, not the hypervisor. The fix is the **Unified Agent**, which pushes both logs and system-level metrics from inside the instance itself. * For new deployments, use the unified CloudWatch agent rather than the legacy CloudWatch Logs agent - it can collect system-level metrics as well as logs, and is typically configured centrally via SSM Parameter Store instead of a local file on every box * It needs an IAM role attached to the instance with permission to call `cloudwatch:PutMetricData` and `logs:PutLogEvents` * The same agent binary works on both EC2 and on-premises servers, though the configuration you give it differs slightly by environment - for example, on-premises hosts need credentials supplied explicitly instead of inheriting them from an instance role ```bash ## Install the agent on Amazon Linux 2023 sudo yum install -y amazon-cloudwatch-agent ## Start it using a config stored in SSM Parameter Store ## -m ec2 tells it it's running on an EC2 instance, not on-prem sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \ -a fetch-config -m ec2 -s \ -c ssm:AmazonCloudWatch-payments-prod-config ``` > **Note:** `AmazonCloudWatch-payments-prod-config` is a Parameter Store key > holding a JSON config file - which metrics to collect, which log files to > tail. Managing it centrally means you update monitoring for a whole fleet > by editing one parameter, not SSHing into every box. > 🔴 **Common Mistake:** Engineers spend an hour looking for a "MemoryUtilization" > metric under `AWS/EC2` that does not exist and never will. Memory is not > an AWS-level metric on any instance type - it always requires the Unified > Agent pushing it as a custom metric under a namespace like `CWAgent`.
It is 2 AM. An engineer on the acme-payments team gets paged because the payments-api service is throwing errors at peak...
A metric is a number tracked over time - CPU utilization, request count, queue depth. Most AWS services publish a defaul...
A dashboard is a single screen showing the metrics that actually matter for one system, so an on-call engineer does not ...
An alarm watches one metric and changes state when it crosses a threshold you define. It has three states: OK - metric i...
CloudWatch Logs is where AWS stores logs from your applications and services. A Log Group is a container, typically one ...
EC2 exposes CPU, network, and disk I/O out of the box - but never memory usage or swap, because those live inside the gu...
Metrics tell you what is happening - error rate went up. Logs tell you what happened in detail - this specific request t...
RED and USE give you the method. The next question every engineer asks is more specific: for this AWS service, which met...
CloudTrail records every API call made in your account - console clicks, CLI commands, SDK calls, even calls AWS service...
AWS Config records how your resources are configured over time, and checks them against rules you define. It is purely a...
CloudWatch tells you something is unhealthy. CloudTrail tells you who did something. Neither one, by itself, sends a Sla...
These four get confused constantly because they all deal with visibility or automation, but each answers a different que...
This lab wires CloudWatch, CloudTrail, Config, and EventBridge together end to end - starting from an empty account and ...
Task Command Create a CloudWatch alarm aws cloudwatch put-metric-alarm Force alarm state for testing aws cloudwatch set-...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.