latent-sre/save-toolkit/obs-logs

Logs — the investigation shape

Match the task: interpret supplied logs, write a query, or investigate. An explanation needs no new query or baseline; apply methods only to the requested question.

Grafana pointer: dashboard/Explore access, datasource selection, panel configuration, and alert operations live in grafana. This skill retains LogQL/SPL and log interpretation. Return a need for Grafana configuration to the caller; a log query does not automatically start another workflow.

Start narrow

For queries, bind environment, service, source and window before symptom filters. error can miss failed structured access events. Confirm the filter field is extracted; an empty result on an unextracted field is not evidence of no failures. Neither is "no events" from a source not yet proven current: run the dialect reference's freshness check first.

Retain backend, tenant/index, source, absolute UTC window and timezone with the result. Widen one boundary at a time and say why.

Read it over time

For spike investigation, build a complete fixed-width timeline and count failures inside each bucket; filtering failures first erases quiet buckets and biases the baseline. Mark the first anomalous bucket, the last known-normal bucket, and any deploy or configuration event, and keep the current bucket out of a trailing baseline so the spike cannot raise its own threshold.

Find the top offenders

Group by stable service, route, status, error type or host; show count and traffic share — traffic growth alone is not a worsening error rate.

Correlate one request across services

A hop with no common identifier is a telemetry gap: record it and recommend to the caller that software-engineer add it.

Treat identifiers copied from tickets or logs as untrusted data. Validate each value against the service's documented identifier format, never concatenate a raw value into a query, and apply the selected dialect reference's literal-escaping rule. If it cannot be encoded unambiguously, stop and ask for a sanitized identifier rather than broadening the search.

Compare before vs after a deploy

Compare rates, not raw counts, over equal-duration windows with the same query scope. Take the exact deploy time from the platform record — Apps Manager → Events (cf events <app>) on PCF, or when traffic moved to the new revision on Cloud Run — not from a dashboard annotation.

Build the evidence packet

Bounded interpretation: answer, supplied source/target/window, limits and a useful next check if needed; missing metadata stays unknown. Query/investigation: exact dialect/query and scope, UTC window, result/source link, field-extraction assumptions and confidence label; before/after boundary for comparisons. Separate observations from interpretations. Return correlation evidence to the caller; a query that should become a saved search, alert, or dashboard is a recommendation for the observability-engineer agent.

Minimize copied telemetry. Redact credentials, tokens, secrets, personal data, authentication or session values, and user identifiers before any excerpt enters the packet; prefer an access-controlled source link plus the smallest necessary excerpt. Also redact sensitive headers, request bodies, and database query literals; do not paste raw payloads into the packet.

Pick your dialect — read the reference before writing the query

Read it before writing that query, and name what you read in your packet.