wquguru/skills/grafana-best-practice

Grafana Best Practice

Use this skill to make Grafana dashboards readable, correctly aggregated, navigable, and safely deployable. Treat the rendered dashboard, Grafana API state, datasource query results, and container health as the source of truth.

Core Workflow

1. Inspect the current dashboard JSON and provisioning files before editing:

• Dashboard JSON: uid, title, tags, templating.list, links, panels.
• Provisioning: provisioning/dashboards/*.yaml, especially folder, provider path, and allowUiUpdates.
• Runtime state when available: /api/search, /api/folders, /api/dashboards/uid/<uid>, /api/ds/query, /api/health.

1. Audit dashboards for:

• Filters: default values, multi/all behavior, variable labels, and whether All views preserve required dimensions.
• Aggregation dimensions: each metric's real labels — a per-instance/shard id, per-event labels (e.g. symbol, reason, side in a trading app), container_name for containers, and host-level metrics that carry none of these.
• Legend placement, values, sorting, and display names.
• Long labels leaking into stat cards or legends, such as environment, project, url, or raw Prometheus family names.
• Bar chart sorting and limits.
• Table column headers — no field-wide defaults.displayName (it overrides every column); columns renamed via the organize transform.
• Multi-bucket / union() queries — every filter() pushed inside each from() (predicate pushdown), never after union().
• Query cost: panel count, target count, default time range, refresh interval, high-cardinality tags, expensive reshape operators, and whether current-value panels scan the whole dashboard range.
• Tag-vs-field modelling on any dimension the dashboard filters or groups by — a churning value used as a tag is a latent datasource outage, not a query-tuning nit (see Tags Versus Fields).
• Anonymous/read-only access if the dashboard is public.

1. Patch dashboard JSON narrowly and preserve existing panel intent. Row nesting differs by state: a collapsed row (collapsed: true) holds its child panels in the row's own panels[], but an expanded row (collapsed: false) keeps them as TOP-LEVEL siblings after the row object — not nested. A programmatic patch that assumes nesting silently skips every panel under an expanded row; index panels by walking both top-level entries and each row's panels[].
2. Validate locally with JSON parsing, repository tests, and static checks.
3. Validate queries against the real datasource before deployment when possible.
4. Deploy through the repository's normal provisioning path.
5. Verify post-deploy using live Grafana API and datasource query results. Do not rely only on file diffs.
6. Then verify what the page actually RENDERS, because step 7 cannot. Transformations and the

display processor run client-side, so the API answers a different question than "what does the reader see". Open the deployed panel (/d-solo/<uid>/?panelId=N isolates one) and read its cells out of the DOM. Do this whenever you touched units, transforms, displayName, value mappings, overrides, or anything that returns more than one frame.

Hard Rules

These fail SILENTLY — the panel renders, the query returns, the alert stays green, and the result is wrong. You will not know to go read the reference file, so they live here. Each points at the file with the worked example.

• Never make a churning value a tag (status, "best route", container name on a shared

host, interface name). It is a latent datasource outage, not a tuning nit — see below.

• `schema.tagValues()` / `schema.fieldKeys()` do not tell you what is being written now.

They read the index and report the old shape after a collector change. Query real points.

• The InfluxDB Flux datasource IGNORES `legendFormat`. Series names come from group-key

labels + displayName. A bare set() that is not in the group key is not a label. → flux.md

• Never `group()` before a pivot that mixes a string field with numeric ones —

schema collision: column "_value" is both of type string and float. Pivot first, group after. (For an all-numeric pivot the opposite holds: group first, or the rowKey tags stay frame labels and pollute every column header.) → flux.md, panels.md

• `aggregateWindow(fn: max|mean)` cannot aggregate a string — use fn: last.
• Aggregators drop every column that is not a group key or the aggregated value. A column

built with map() before spread() vanishes. Map after, or make it a group key. → flux.md

• Push every `filter()` INSIDE each `from()` before `union()`. A filter after union() is

not pushed down: each branch full-scans the whole bucket into memory and can wedge a shared host. → flux.md

• **Never run an unfiltered or heavy ad-hoc Flux query against a shared production

datasource.** Filter by measurement, keep the range tight, validate with count() first.

• Never set `fieldConfig.defaults.displayName` on a table — it overrides EVERY column

header. Rename in the organize transform. → panels.md

• Never set a field-wide `unit` on a table either — it CORRUPTS string columns. Grafana

leaves a string alone only when no unit is set (getDisplayProcessor: else if (!unit && field.type === string) unit = 'string'). With one set it runs Number(value), and JS parses 0x… as a HEX LITERAL — an address column rendered 2.84027890695473e+22 Sept. Set units per column via overrides. → panels.md

• `/api/ds/query` CANNOT verify what a panel displays. Transformations and the display

processor both run in the BROWSER, so the API returns pre-transform, pre-format data. Every server-side check can be green while the page is wrong — read the rendered DOM. → deployment.md

• `noDataState: OK` turns a broken query into permanent silence — never fires, never

errors, never shows NoData. Verify any schema change under such a rule against live data before shipping. → alerting.md

• Current-value panels must use a bounded window (range(start: -15m) + last()), not

the dashboard picker. → performance.md

• A committed config fix is not a deployed one, and the three planes (Grafana image,

collector bind mount, app) ship by different mechanisms. → deployment.md

Tags Versus Fields (Cardinality At The Source)

This is a collector-config decision (telegraf.conf tags vs included_keys), not a dashboard one, but it is the single thing most likely to take the datasource down — so audit it whenever you touch a dashboard that filters or groups by a status-like dimension.

series = distinct tag-sets × field keys. A tag whose value changes for a fixed entity over time — a "churning tag" — multiplies cardinality by every value it has ever taken inside the retention window. Status flags, a "best route"/"cheapest provider" pick, a container name on a shared host, a network interface name, a request id: all churn.

Three properties make this worse than an ordinary sizing mistake:

1. It does not self-heal. InfluxDB's series file is append-only. Retention expiry drops

shard data but never reclaims series entries, so cardinality only ever goes up until someone rebuilds the index.

1. Uptime hides it. The cost is paid on shard-index load, i.e. only on a COLD start. A

container can sit for weeks pegged at 95-100% of its memory cap and look perfectly healthy, then fail to boot forever after the first restart — a deploy, a host reboot, one heavy query. Cold-start capacity is not exercised by uptime; nothing warns you.

1. The estimate in the comment is probably wrong. Measured example (2026-08): a comment

claiming "20-25k series worst case" sat above a measurement holding 166,773 — 7× off, and that staleness is why nobody looked. Another: six route-venue tags added on the belief that "cardinality is bounded by asset × a small venue-pair" minted 897,212 dead series, 79% of the whole bucket, and the resulting boot loop was a full outage.

Rules:

• Keep as tags only what genuinely identifies the entity (venue, asset, instance,

container_name for containers). Put churning values in included_keys as string fields and pivot them back in at query time.

• State the arithmetic (rows × field keys) BEFORE changing a tag set, and re-measure AFTER

deploying. Against a stopped instance: influxd inspect report-db --db-path <data>/<bucket-id> --rollup m. Every cardinality comment must carry a measured number and its date.

• Demoting a tag is a COORDINATED change — collector config, every dashboard panel that

filters/groups on it, every alert rule, and the downsample task's string guard all move in one commit. A partial change breaks silently, not loudly.

• Never trust schema.tagValues() / schema.fieldKeys() to tell you what is being written

NOW. They read the index, which still lists every tag value and field key whose series exists in an overlapping shard, so right after a collector change they report the OLD shape for the new window — a -3m query can return 30 days of dead values. To answer "what is actually landing", query real points: range(start: -90s) |> filter(...) |> keep(columns: ["<tag>"]) |> group() |> distinct(column: "<tag>").

Folder And Tag Rules

Prefer a named Grafana folder when permissions are known to be handled. For provisioned Grafana OSS dashboards with anonymous access, verify folder permissions explicitly because named folders can 403 even when dashboards exist.

Safe folder checklist:

• provisioning/dashboards/default.yaml uses the intended folder, for example folder: Production.
• Startup or deployment code grants Viewer read on every provisioned folder and dashboard when Grafana state is ephemeral.
• /api/folders shows the folder.
• /api/search?type=dash-db&tag=<base-tag> shows every dashboard with the expected folderTitle.
• Anonymous probes return 200 for both /d/<uid>/... and /dashboards.

Use one stable base tag for dashboard links and discovery, then add semantic tags. Keep tags low-cardinality and queryable.

Recommended tag shape — one stable base tag (the service slug) plus semantic axes. The values below are an illustrative set; substitute your own facets:

<service>                     # stable base tag, e.g. `myservice`
area:<facet>                  # e.g. area:overview | area:app | area:risk | area:infra
audience:<role>               # e.g. audience:operator | audience:oncall
plane:<data-plane>            # e.g. plane:prometheus | plane:app | plane:host

Keep dashboard dropdown links on the stable base tag (the service slug, e.g. myservice) so adding semantic tags does not break navigation.

Variables And Filters

Use explicit labels that explain important defaults:

{
  "name": "instance",
  "label": "Instance (All by default)",
  "multi": true,
  "includeAll": true,
  "allValue": ".*"
}

For high-level fleet dashboards, defaulting instance to All is good only if every panel handles All safely. When All is selected:

• Stat panels should either show one compact value per instance or deliberately aggregate fleet-wide.
• Time series panels should not collapse multiple instances into one unlabeled series.
• Legends and display names should stay short.

Detailed References

Load the file that matches what you are doing: