haacked/dotfiles/ops-report

Ops Report

Generate a 24-hour operational health report for a PostHog service by querying Grafana dashboards and Prometheus/VictoriaMetrics metrics.

Arguments (parsed from user input)

• service (optional): The service to report on (default: feature-flags). Other examples: ingestion, capture
• --window (optional): Lookback window: day (24h), week (7d), or month (30d). Default: day
• --region (optional): Grafana region to query: us, eu, or both (default: both)

Example invocations:

• /ops-report - daily report for feature flags (both regions, the default)
• /ops-report feature-flags --window week - weekly report for both regions
• /ops-report ingestion --region eu - daily ingestion report from EU only
• /ops-report --region us - US-only report

MCP Server Mapping

Regional parallelism: wherever a step says "for each active region", fire the region-specific MCP calls in parallel. Query examples in this skill show the US form only; for EU, use the mcp__grafana-eu__* equivalent with the EU datasource UID. Tag every metric, anomaly, and dashboard link with its region ([US] / [EU]) throughout.

Your Task

Follow these steps in order.

Step 1: Parse Arguments and Validate

Extract from user input:

• service - kebab-case service name, default "feature-flags"
• window - one of day, week, month. Default "day"
• region - one of us, eu, both. Default "both"

Determine which MCP servers to use:

• region=us → mcp__grafana__* only
• region=eu → mcp__grafana-eu__* only
• region=both → both mcp__grafana__* and mcp__grafana-eu__* in parallel

Map the window to query parameters:

Compute absolute UTC timestamps for the report window. All Prometheus and Loki queries must use these absolute timestamps; never use relative expressions like now-24h. Absolute timestamps ensure that spike times read from Prometheus responses correspond exactly to the report window, so Loki follow-up queries target the correct time.

For a day report generated on date D, the window is:

window_start = {D - 1 day}T00:00:00Z   (e.g. 2026-03-17T00:00:00Z)
window_end   = {D}T00:00:00Z            (e.g. 2026-03-18T00:00:00Z)

For week and month, anchor the window end to {D}T00:00:00Z and subtract accordingly:

week:  window_start = {D - 7 days}T00:00:00Z  (e.g. 2026-03-11T00:00:00Z)
month: window_start = {D - 30 days}T00:00:00Z (e.g. 2026-02-16T00:00:00Z)

Record these values:

• {window_start} and {window_end}: absolute RFC3339 timestamps for all query start/end parameters
• {window_hours}: 24, 168, or 720, for use in PromQL range selectors like [{window_hours}h]

State the computed {window_start}, {window_end}, and {window_hours} values before proceeding to Step 2.

Record the current session JSONL line count for token tracking:

claude-session-tokens --line-count

Store the output plus one as {token_baseline_line} (i.e., line_count + 1) so that --sum-from starts after the last pre-existing line.

Step 2: Discover Dashboards

Search Grafana for dashboards related to the service, for each active region:

mcp__grafana__search_dashboards(query="{service}")

Filter results to dashboards tagged with the service name or whose title contains the service name. Record each dashboard's UID, title, description, and region. Dashboard UIDs are often the same across regions; query both independently.

If no dashboards are found in either region, tell the user and stop.

Step 3: Discover Datasources

For each active region, discover the Prometheus datasources and the Loki datasources (needed for log queries in Step 8):

mcp__grafana__list_datasources(type="prometheus")
mcp__grafana__list_datasources(type="loki")

For each region, use the datasource named "VictoriaMetrics" (or the default Prometheus datasource). Record each region's Prometheus UID and Loki UID separately. Do not hardcode UIDs; discover them here.

The US Loki datasource is typically named Loki-logs (uid P44D702D3E93867EC), but always verify via discovery rather than assuming.

Step 4: Extract Key Queries from Dashboards

For the most important dashboards (the "general" or overview dashboard first, then latency, cache, and pods dashboards), extract panel queries for each active region:

mcp__grafana__get_dashboard_panel_queries(uid="{dashboard_uid}")

If dashboards share the same UID across regions, the panel structure will be identical, so you only need to extract queries once and reuse them for both regions' metric queries in Step 5.

Identify the key metrics to query. Prioritize these categories:

1. Request rate - throughput over time
2. Success/error rate - 2xx/3xx vs 5xx responses
3. Latency - P50, P95, P99 percentiles
4. Resource usage - CPU, memory relative to requests/limits
5. Pod/scaling - HPA replica count, pod restarts
6. DB/cache performance - pool utilization, cache hit rates, connection times
7. Evaluation errors - service-specific error counters

Step 5: Query Metrics

Run PromQL range queries against each active region's Prometheus datasource, storing results separately per region so they can be compared in the report:

• Use the absolute {window_start} and {window_end} timestamps computed in Step 1 as the query startTime/endTime parameters, never now-{hours}h or now
• Step size: use the value from the window mapping (300s for day, 1800s for week, 7200s for month)

Note: The ban on relative expressions applies to the query's startTime/endTime parameters. PromQL duration expressions inside the query (range selectors like [5m], [{window_hours}h], and subquery windows) remain as durations. These are lookback windows within PromQL, not the time range of the query itself.

Important: Replace any $__rate_interval or $__interval template variables with appropriate values. For day use 5m/1m, for week use 30m/5m, for month use 2h/30m.

Step 5b: Service-Specific Capacity Checks

Read references/capacity-checks.md and run the checks that apply to the service, for each active region. Currently all checks target feature-flags; skip this step if the file has no entries for the service.

When a check produces an anomaly, record its predicate in the shape Step 6 defines below, so Step 6b can re-evaluate it.

Step 6: Analyze Results

For each metric time series, compute:

• Current value (most recent data point)
• Min/max over the window
• Mean over the window
• Notable spikes or dips (values more than 2x the mean, or sudden step changes)

All timestamps reported must come directly from Prometheus data point values, never from log entries, correlated signals, or inference. A log message at time T is not evidence that a metric spike occurred at time T. For each detected spike, record its peak as {spike_peak_utc} (the exact timestamp from the Prometheus data point) for use in Step 8.

Error Spike Count

After querying the 5xx error count time series, compute a spike count:

1. Count the number of data points where 5xx errors exceeded the warning threshold (thresholds apply per data point regardless of step size)
2. Classify each spike by severity:

• Warning: 50–299 errors per data point
• Critical: ≥ 300 errors per data point

1. Record the timestamps of the worst spikes for investigation in Step 8
2. For each run of consecutive Critical data points, record how many points the run spans and its wall-clock duration. The Assessment Criteria read that number to separate Degraded from Unhealthy, and a lone Critical point is a run of 1.

Use the same spike count assessment labels as the Latency Spike Count table below.

Latency Spike Count

After querying the P99 latency time series, compute a spike count:

1. Compute the median of all P99 data points in the window
2. Set the spike threshold to max(2 × median, 200ms)
3. Count the number of sampling intervals where P99 exceeded the threshold
4. Classify each spike by severity:

• Minor: threshold < P99 ≤ 300ms
• Warning: 300ms < P99 ≤ 600ms
• Critical: P99 > 600ms

1. Record the timestamps of the worst spikes for investigation in Step 8

Use these labels for the spike count assessment:

Cross-correlate anomalies:

• Do error spikes correlate with latency spikes?
• Do latency spikes correlate with DB pool saturation?
• Do latency spikes cluster at specific times of day (e.g., peak traffic hours)?
• Do latency spikes correlate with DB pool utilization spikes?
• Do scaling events correlate with traffic surges?
• Is the HPA spending >20% of the window at max replicas? Does headroom correlate with latency?
• Are there any container restarts?
• Are any teams approaching resource or feature limits (e.g., max flag count)?
• Do task duration increases correlate with queue depth growth?
• Are batch refresh failures correlated with worker OOM kills?
• Do sync_feature_flag_last_called success rate drops correlate with worker log errors?
• Do billing aggregator flush errors or stale-flush alarms correlate with Redis latency or flag_request_redis_error spikes? Does pending queue growth precede unflushed_requests_total{cause="cap_drop"}?
• Has this anomaly happened before? Step 6b runs the recurrence check, and a repeat is investigated as a pattern rather than a fresh event.

For each anomaly, attempt to investigate the cause by querying additional metrics, checking for correlated events, and noting what you ruled out. The goal is to hand the reader a partially-investigated issue with clear next steps, not just a raw signal.

Record Each Anomaly's Predicate

When you flag an anomaly, record the exact test that flagged it. Step 6b re-evaluates that test over a longer lookback, and reconstructing it later from prose is how a recurrence count goes wrong:

• Expression: the PromQL that produced the series, keeping its by (...) labels.
• Direction: > when rising is the failure, < when falling is.
• Threshold: an absolute number. Resolve a mean-relative or median-relative rule to the number it produced in this window and carry that number, not the rule. A P99 threshold of max(2 × median, 200) against a 90ms median is 200; record 200. Recomputing the rule over 14 days would measure against a different median and silently change what counts as an occurrence.

Some anomalies have no such test. Record these as having no predicate: sudden step changes, any trend judged from the shape of a series over the window however it was computed (a deriv slope, pending queue depth growth), rules that fire on drift in either direction such as the records-in / entries-flushed ratio, window-wide ratios such as HPA time at max replicas, and log patterns with no counter behind them. Step 6b reports those as not checkable rather than guessing at a query.

Step 6b: Recurrence Check