latent-sre/save-toolkit/obs-pipeline

Ship telemetry end to end

Grafana pointer: UI/API access, datasource configuration, dashboards, and alert operations live in grafana. This skill owns emission, collection, and export through Alloy/OTel to the backend. If a bounded backend query finds the signal but a panel does not, carry that evidence to grafana for datasource, variable, query-window, and presentation checks before changing the pipeline.

Treat the pipeline as one path with four independently failing boundaries:

app → SDK/agent → collector/Alloy → backend

Identify the affected Splunk or Wavefront/PCF App Metrics feed and its owner before changing it. Confirmed PCF-platform-owned feeds stay with that owner; use pcf-ops and obs-analysis for platform-side and query evidence. The backend name alone does not establish the ingestion route.

GCP backends are landing with the migration: the documented ingest is OTLP to the Telemetry API (telemetry.googleapis.com — all three signals) through the same otelcol exporter slot. Log ingestion has been GA since 2026-08-14. The Alloy pipeline reference separately documents its public-preview Google-auth component and required stability flag. Exact target project/config and route behavior remain unconfirmed until a canary run proves them.

Where a missing signal gets lost

1. The app never emits it. Exercise one known request locally and inspect SDK diagnostics

or a local console exporter for the expected log, metric, or span before checking the network.

1. The SDK/agent never exports it. Confirm the process has the intended endpoint,

protocol, resource attributes, and credentials, then inspect its export error/drop counters.

1. The collector/Alloy never accepts or keeps it. Check receiver health and accepted,

refused, processed, and dropped counts for that signal; validate the deployed config before edits.

1. The exporter/backend rejects, delays, or misroutes it. Check exporter send/failure

evidence, then issue one time-bounded backend query using the canary's exact service and trace IDs.

A healthy later component does not prove an earlier boundary, and a backend query with no bounded canary does not identify loss.

Read only the reference needed for the task:

The cardinality rule (this is what blows up metric stores)

Bounded dimensions → metric labels/tags. Unbounded identity (user/request/trace IDs, full URLs, emails, raw SQL) → traces and logs, never metric labels. A label with unbounded values creates a new time series per value.

OTel naming — dotted namespaces and instrument metadata

• The OTel name is the source of truth: namespaces delimited by dots, snake_case only within

a multi-word component. Units live in instrument metadata (UCUM), not the name (upstream allows a unit in the name only to resolve ambiguity) — "units do not need to be specified in the names since they are included during instrument creation." Base units (seconds, bytes); dimensions in attributes, not the name. Pluralization follows the unit: a name whose value carries a real unit stays singular, a unitless countable is pluralized (system.paging.faults, system.network.packets), and UpDownCounter names are never pluralized.

• This lands natively on our stack. Wavefront takes dot-delimited names + point tags

(app.http.requests.latency) — the OTel shape, near-unchanged. Keep the bounded-tag discipline; this section owns the emitted naming/tag contract. A Prometheus-style exporter translates dotted names for you by default; do not pre-bake exporter suffixes into the OTel name.

Build the evidence packet

Return per boundary checked: target, UTC window, exact check/result, canary service ID (trace ID for traces), config revision and claim confidence. Name missing proof without inventing checks or broader coverage. Separate observed counters from interpretations. Minimize copied telemetry. Redact credentials, tokens, secrets, personal data, authentication or session values, and user identifiers before any excerpt enters the packet; prefer an access-controlled source link plus the smallest necessary excerpt. Also redact sensitive attribute values. Hand pipeline-config changes to the observability-engineer agent and app-side instrumentation changes to the software-engineer agent; if the missing signal is part of an active unknown-cause incident, hand the time-bounded evidence to the responder with incident-investigation (sre-assistant only for a dispatched read).