09 — CloudWatch: Metrics, Logs, and Events — The Nervous System

"The graphs screen in the console" was how I saw CloudWatch, which meant every monitoring task started with me staring at a dashboard. The actual shape: CloudWatch is three primitives — metrics (numbers over time), logs (text events with timestamps), and events (state changes that something can react to) — and alarms are the glue that turns those primitives into automated responses [1]. Once I separated the three, every monitoring question became "which primitive am I looking at, and what should react to it?"

Metrics: numbers over time

A metric is a time-ordered set of data points — a variable I'm watching, sampled over time [2]. CPU utilization, request count, bytes read, error rate — each is a metric. AWS services emit metrics for free (EC2 CPU, S3 bucket size, Lambda invocations), and I can publish my own custom metrics from application code.

The shape worth knowing: a metric is identified by a name, a namespace (which service or app it belongs to), and up to 30 dimensions (key/value pairs that slice it — e.g. InstanceId=i-abc, Environment=prod). I retrieve statistics (average, sum, max, p99) over a time window. The way of thinking: a metric is a labeled time series, and CloudWatch is the store that keeps it.

Logs: text events with timestamps

CloudWatch Logs is the centralized log sink [4]. EC2 instances, Lambda functions, containers, CloudTrail API calls — anything that emits text with a timestamp can ship it here. The two organizing concepts:

• Log groups — the top-level container, usually one per application or service. Retention is set per group (so I can keep prod logs for 90 days and dev logs for 7).
• Log streams — a sequence of logs from one source within a group (one Lambda instance, one EC2 instance).

The feature I lean on hardest is CloudWatch Logs Insights — a query language over my logs. "Show me all ERROR lines from the payments service in the last hour, grouped by request ID" is a few lines of query, not a grep over downloaded files. Logs without a queryable index are just storage; Insights is what makes them usable.

Events: state changes that something reacts to

CloudWatch Events (now folded into EventBridge) captures state changes in my AWS environment [3]. An EC2 instance stopped, a new object landed in an S3 bucket, a scheduled cron fired — each is an event. The shape:

• An event pattern (or a schedule) matches events I care about.
• A target is what runs when the pattern matches — a Lambda function, an SNS notification, a Step Function, an Auto Scaling policy.

This is the automation substrate. "When an instance in my ASG terminates, run a Lambda that deregisters it from the service mesh (the layer that tracks which services are running)" is an event rule plus a target. "Run the nightly ETL (the batch job that crunches the data) at 2am" is a scheduled rule plus a Lambda target. The whole event-driven architectures I build on AWS sit on top of this one primitive.

Alarms: turning observations into actions

An alarm watches a metric over a window and changes state when it crosses a threshold [2]. "If the average CPU on my ASG stays above 80% for 5 minutes, go into ALARM." Once in ALARM, the alarm can:

• Notify an SNS topic, which fans out to email, Slack via a webhook, or a paging system.
• Trigger an Auto Scaling policy (the link between a CloudWatch metric and capacity changes).
• Trigger a Lambda or a Systems Manager action.

Alarms are how a metric becomes a response. Without them, CloudWatch is just dashboards; with them, it's a closed loop — observe, decide, act.

How I use this

CloudWatch is the part of AWS I treat as non-negotiable for anything beyond a tutorial. The discipline I keep: every service emits at least one custom metric for its core health (requests, errors, latency), and every service ships its logs to a CloudWatch Logs group with a defined retention. I write Logs Insights queries for the three questions I'll inevitably ask in an incident (top errors, slowest requests, recent state changes) before the incident, not during it. Alarms are wired to the metrics that matter — and crucially, to the actions that should follow — so that capacity adds itself, and the right person gets paged, without a human in the loop. The three primitives are simple; the value is in closing the loop from observation to automated response.

References

[1] Amazon Web Services, "What is Amazon CloudWatch?," CloudWatch User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html

[2] Amazon Web Services, "CloudWatch metrics," CloudWatch User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html

[3] Amazon Web Services, "Amazon CloudWatch Events," AWS Whitepapers, 2024. [Online]. Available: https://docs.aws.amazon.com/whitepapers/latest/introduction-devops-aws/cloudwatch-events.html

[4] Amazon Web Services, "What is Amazon CloudWatch Logs?," CloudWatch Logs User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html