System design interview · backend

Design a metrics monitoring system

"Build us something like Datadog or Prometheus: ingest 10 billion datapoints a day, keep 2 years of history, and answer 'show me p99 checkout latency for the last 30 days' in under a second."

⚡ The takeaway first

Separate the write path from the read path, and roll data up aggressively at ingest time. Nobody queries raw per-second values from 6 months ago — they query trends. So: ingest everything, immediately pre-compute 1-minute / 5-minute / 1-hour rollups, keep raw data for ~24 hours on fast disks, and serve old queries from tiny pre-aggregated buckets. The whole system is a funnel: wide at the top, narrow at the bottom.

📋 Requirements

Functional

Ingest time-series datapoints: (metric name, labels, timestamp, value) from thousands of hosts.
Query by metric + label filters + time range, with aggregations: avg, sum, min, max, p50/p99.
Downsample on the fly: "give me 1 point per hour for the last 90 days."
Alerting: "page me if error rate > 1% for 5 minutes."

Non-functional

Write throughput: ~10k datapoints/sec sustained, 10× bursts.
Read latency: p99 < 1s for dashboard queries over 30 days.
Retention: 2 years of queryable history.
Durability: losing raw data older than a day is tolerable; losing the last hour is not.
Cardinality: must survive label explosion (e.g. per-pod, per-endpoint labels).

🚫 Common misconception

"We need to store every raw datapoint forever." You don't. A dashboard showing 90 days on a 1200-pixel-wide screen can only display 1200 points — one per ~1.8 hours. Storing per-second resolution for that range is paying for precision nobody can see. Store raw briefly, rollups forever.

🧮 Back-of-the-envelope math

Think of it like a warehouse: every box that arrives must be shelved, and shelf space costs money. Let's count the boxes.

Assumption	Value
Unique time series	100,000 (hosts × services × endpoints)
Sample interval	1 sample / 10 sec per series
Write rate	100,000 ÷ 10 = 10,000 writes/sec
Bytes per datapoint (timestamp + float)	16 bytes
Raw volume	10k × 16 B × 86,400 ≈ 13.8 GB/day

Now the funnel

Tier	Resolution	Retention	Size
🔥 Hot (SSD / memory)	raw (10s)	24 h	~14 GB
🌡️ Warm (cheap disk / object store)	1-min rollup	90 days	100k × 1,440 × 16 B × 90 ≈ 207 GB
❄️ Cold (object store, Parquet)	1-hour rollup	2 years	100k × 24 × 730 × 16 B ≈ 28 GB

Total ≈ 250 GB before compression. Time-series compressors (Gorilla / delta-of-delta) shrink this ~10×, so real disk is on the order of tens of GB — a laptop could hold it. The hard part was never storage; it's write throughput and query fan-out.

Go deeper: the Gorilla compression trick

Facebook's Gorilla paper showed timestamps and floats compress beautifully: store the delta of deltas for timestamps (usually 0 bits when samples are regular) and XOR floats with the previous value (leading zeros dominate). That's why 13.8 GB/day of raw points often lands under 1.5 GB on disk. Interviewers love this reference — it shows you know time-series data is special, not just "rows in a DB."

🏗️ Architecture

The centerpiece: one funnel, three temperatures.

flowchart TD
    A[App hosts & services
100k series] --> B[Ingest gateway
auth, batching, validation]
    B --> C[Stream processor
compute 1m / 5m / 1h rollups]
    C --> D[🔥 Hot store
raw, 24h - SSD]
    C --> E[🌡️ Warm store
1-min rollups, 90d]
    C --> F[❄️ Cold store
1-hour rollups, 2y - object store]
    G[Query API] --> D
    G --> E
    G --> F
    H[Dashboards & alerts] --> G
    I[Compactor
downsamples hot to warm to cold] --> E
    I --> F
    D --> I

Read the funnel like a kitchen: the ingest gateway is the receiving dock (checks the delivery), the stream processor is the prep cook (chops everything into standard portion sizes immediately), and the query API is the waiter who knows exactly which shelf each dish lives on based on how old the order is.

🔍 Component deep-dives

Write path — batch, don't trickle

sequenceDiagram
    participant H as Host agent
    participant G as Ingest gateway
    participant S as Stream processor
    participant T as Hot store
    H->>H: batch 10s of samples
    H->>G: POST /write (1 batch, ~100 series)
    G->>G: validate + hash by series key
    G->>S: route partitions
    S->>S: update 1m/5m/1h rollup windows
    S->>T: append chunk (t,v pairs)
    T-->>H: 202 Accepted

Key insight: agents batch locally so the gateway sees ~1k requests/sec, not 10k. Partition by hash(series key) so one series always lands on the same worker — rollups stay correct without cross-talk.

Read path — fan out by age

sequenceDiagram
    participant U as Dashboard
    participant Q as Query API
    participant W as Warm store
    participant C as Cold store
    U->>Q: GET /query?metric=latency&from=-30d&step=1h
    Q->>Q: split range: [-30d,-1d] cold, [-1d,now] warm
    par fetch in parallel
        Q->>C: scan 1h buckets
    and
        Q->>W: scan 1m buckets, re-roll to 1h
    end
    Q->>Q: merge + sort by time
    Q-->>U: 720 points, p99 320ms

The query planner is the clever bit: it picks the coarsest tier that can answer the question. A 30-day query at 1-hour steps never touches raw data — it reads 720 pre-computed points per series instead of 2.5 million.

🔌 API + data model

API

POST /v1/write
  [{ metric:"http_latency_ms", labels:{svc:"checkout",pod:"a1"},
     points:[[t1,v1],[t2,v2],...] }]

GET /v1/query?metric=http_latency_ms&svc=checkout
        &from=2026-09-03&to=2026-10-03&step=1h&agg=p99

POST /v1/alerts   { metric, threshold, window, notify:"pagerduty" }

Data model

Series key = metric name + sorted label set → hashed to a 64-bit ID. Everything (storage, routing, rollups) keys off this ID.
Chunk = 2 hours of (timestamp, value) pairs for one series, Gorilla-compressed. Immutable once sealed — the unit of storage and replication.
Index = inverted index: label → series IDs (like a search engine), so "svc=checkout" resolves without scanning.

⚖️ Trade-offs

Decision	Option A	Option B	Pick
Collection	Push (agents send)	Pull (server scrapes)	Pull for services (Prometheus-style: dead targets are visible), push for ephemeral jobs
Precision vs cost	Keep raw 30 days	Raw 24h, rollups after	B — dashboards can't render the difference
Query engine	Pre-aggregate everything	Scan raw on demand	Pre-aggregate the common windows; allow raw scans only on the hot tier
Cardinality guard	Reject new series	Charge/rate-limit per tenant	Both: hard cap per tenant + alert before hitting it

🔥 Failure modes

Cardinality explosion — someone adds a user_id label; series count jumps 100k → 50M overnight. Mitigation: per-tenant series caps, alerting on new-series rate.
Ingest spike — deploy storm 10×s the write rate. Mitigation: gateway sheds load (429 + client backoff), stream processor scales horizontally since it's partitioned.
Hot node OOM — a partition gets disproportionate series. Mitigation: consistent hashing with virtual nodes spreads load; chunks flush to warm store on a timer regardless.
Clock skew — hosts disagree about "now" by minutes. Mitigation: accept slightly out-of-order points within a chunk's seal window; reject the rest with a counter.

🛠️ What I'd actually build

Opinionated and concrete: I would not build the storage engine. I'd run VictoriaMetrics cluster (or Mimir if the team knows Prometheus deeply) — it already does the hot/warm split, Gorilla-style compression, and downsampling. Agents: Prometheus remote-write or OpenTelemetry Collector. Long-term cold: VictoriaMetrics' native historic support backed by S3 with Parquet. Query: Grafana on top, alerts via Alertmanager → PagerDuty. The genuinely custom code is thin: the ingest gateway (auth + tenant cardinality caps) and the query planner that routes by age. Everything else is bought, not built.

🎤 Interview tips

Say the funnel sentence early: "raw for a day, rollups forever" — it frames every follow-up answer.
When they ask "how do you handle 10× traffic?", answer partitioning by series hash, not "add more servers."
Drop Gorilla compression by name — it's the canonical reference and signals depth fast.
If they push on exactly-once delivery: concede at-least-once + idempotent chunks; duplicates in metrics are visually harmless, lost data isn't.
Always mention the cardinality guard — interviewers use it to separate people who've operated metrics systems from people who've only drawn them.

Go deeper: when raw data actually matters

Two cases break the "rollups forever" rule: forensics ("what exactly happened at 14:03:22 during the outage?") and ML training on raw signals. The honest answer: keep a sampled raw archive (e.g. 1% of series, or raw only for a golden set of critical metrics) in cold storage. Full raw forever is a cost conversation, not an engineering one — put the number on the table: 13.8 GB/day × 730 days ≈ 10 TB raw, ~1 TB compressed. Sometimes the business says yes.

🎮 Interactive widget: watch raw points roll up

Raw datapoints stream in on the left. Watch them get swallowed into 1-minute buckets in real time — this is exactly what the stream processor does. Then flip retention tiers and drag the downsampling slider.

Retention tiers — pick one, watch the math

Downsampling visualizer — one day of 1-minute data

Same 24 hours of data, re-bucketed. Drag the slider: fewer points, same shape — until it isn't.

Bucket size: 1 min → 1,440 pts