DevOpsJuly 17, 20266 views0 comments

Prometheus on EKS: What to Actually Alert On (and What to Ignore)

By SeaGit

#prometheus#monitoring#kubernetes#alerting#grafana

Every Kubernetes monitoring journey starts the same way: install kube-prometheus-stack, inherit a few hundred dashboards and recording rules, and drown. The stack is excellent — Prometheus Operator, node-exporter, kube-state-metrics, Grafana, Alertmanager, wired together correctly out of the box. What it cannot decide for you is what deserves a page at 3 a.m. After operating Prometheus across a fleet of EKS clusters, our answer is: far less than the defaults suggest, watched far more strictly.

The alerts that catch real incidents

The alerts that have caught real incidents, roughly in order of value:

Workload health: KubePodCrashLooping and KubeContainerOOMKilled on production namespaces — a crash loop is never fine, and OOM kills are the single most common "it was working yesterday" cause.
Scheduling: pods Pending beyond five minutes, which catches exhausted node groups, broken taints, and PVCs stuck binding in one alert.
Capacity cliffs: PersistentVolume usage above 85 percent with a predict_linear check on the growth rate, and node filesystem or inode pressure — disks are the outages you can see coming days ahead if you look.
Control-plane reachability from the workload's point of view: apiserver request error rate and webhook latency, because on EKS you do not run the control plane but you absolutely feel its bad days.
Certificates: certmanager_certificate_expiration_timestamp_seconds under twenty days, which converts every silent renewal failure into a calm ticket instead of a Saturday outage.
DNS: CoreDNS SERVFAIL/REFUSED rate plus forward latency, because when DNS degrades, every other alert fires at once and this one tells you which page to read first.

The discipline part: cardinality

Now the discipline part: cardinality. Prometheus memory usage is a function of active series, and active series grow with every unique label combination. The classic self-inflicted wounds are putting pod name, request path, or user ID into a histogram's labels. A histogram with 12 buckets, per-pod labels, and 50 replicas is 600 series for one metric on one workload — multiply by a rollout that churns pod names and you have a series explosion that looks exactly like a memory leak.

House rules that keep clusters at a boring couple of million series: no unbounded label values (paths get templated, IDs get dropped), relabel away noisy kube-state-metrics series you never query, and check the ten most expensive metrics (topk by series count) whenever memory trends up.

Sizing and retention

Sizing and retention, the pragmatic version. Local retention on the in-cluster Prometheus stays short — seven to fifteen days — because its job is alerting and recent debugging, and gp3 volumes plus WAL replay time both punish hoarding. Long-term history belongs in remote write to a managed or object-store backend (Amazon Managed Prometheus, Thanos, Mimir — pick by taste and budget). This split also makes the in-cluster Prometheus disposable: it can be rebuilt from manifests without losing anything that matters, which is exactly the property you want when a node group gets recycled.

Alert hygiene that keeps the pager trustworthy

Alert hygiene rules that keep the pager trustworthy:

Route every alert by severity: critical pages a human, warning goes to a channel that is actually read, info does not exist — delete it.
Every page must have a plausible action. "CPU above 80 percent" is not an action, it is a graph.
Use for: durations aggressively — a 30-second CPU spike is weather, not climate.
Inhibit downstream noise: when the DNS alert fires, the forty service-level timeout alerts it causes should be suppressed by an inhibition rule, not acknowledged one by one.
Review the pager log monthly — any alert that fired more than three times without producing an action gets tuned or deleted. Alert fatigue is not a personality trait of on-call engineers; it is a measurable configuration bug.

Three dashboards, not three hundred

Grafana earns its keep with exactly three dashboards per audience: a fleet overview (is anything on fire), a per-cluster drill-down (what is on fire), and a per-workload view developers can self-serve (is MY thing on fire). More dashboards than that and people stop knowing which one is true. Put the vanity metrics — request counts, business KPIs — into the developer view where they motivate, not into the on-call view where they distract.

The meta-lesson

The meta-lesson: Prometheus on EKS fails socially before it fails technically. The stack scales fine; what breaks is a pager nobody believes and dashboards nobody reads. Alert on symptoms users feel, keep series cardinality bounded, ship history off-cluster, and delete anything that talks without saying something. On SeaGit, Prometheus and the metrics server are one-click add-ons, so you start from a working stack and spend your time on the alert rules that matter.

Frequently asked questions

What should I actually alert on in Kubernetes?

Symptoms users feel: pod crash loops and OOM kills, pods Pending beyond five minutes, PersistentVolume usage over 85 percent, apiserver error rate, certificate expiry under twenty days, and CoreDNS failure rate. Skip alerts with no plausible action, like raw CPU thresholds.

Why is my Prometheus using so much memory?

Cardinality. Memory scales with active series, and per-pod or per-path histogram labels explode series counts during rollouts. Template or drop high-cardinality labels, relabel away unused kube-state-metrics series, and audit the top metrics with topk by series count.

How long should Prometheus retain data on EKS?

Keep local retention short — seven to fifteen days for alerting and recent debugging — and remote-write long-term history to Amazon Managed Prometheus, Thanos, or Mimir. That keeps the in-cluster Prometheus disposable and cheap to rebuild.

How do I stop alert fatigue?

Route by severity, require an action per page, use for: durations to ignore brief spikes, add inhibition rules so one root cause doesn't fire forty alerts, and delete any alert that fires repeatedly without producing action.