chipi/agentic-ai-homelab/telemetry-probe

telemetry-probe

Every "probe it, then clean up" in this repo used to be a hand-written script, and it went wrong in the same places each time (2026-10-16/17):

• deleting with GlitchTip's read-only token (403);
• the legacy /store/ endpoint, which drops debug_meta;
• forgetting the Umami session row;
• calling a delete done before the re-read showed it gone.

This skill does it the tested way.

Run (on the mini, through host-run)

P=~/.claude/skills/telemetry-probe/scripts/probe.py
R=~/.claude/skills/host-run/scripts/run.sh
 
$R homelab $P glitchtip --project player-dev --assert-absent player,player-api
$R homelab $P umami     --site "Player (dev)" --assert-absent Player
$R homelab $P langfuse  --keys '~/podcast-langfuse-dev.env'

Labels: every probe is labelled zz-probe-<utc>, with GlitchTip environment=probe and Umami hostname telemetry-probe.localhost. Nothing real is mistaken for one.

Exit 0 / `PROBE: PASS` only when all three hold:

1. the probe landed in the target;
2. it is absent from every --assert-absent target;
3. unless --keep, the delete was accepted and a re-read shows it gone.

How each service is done

• GlitchTip:
• sends a Sentry envelope to /api/<id>/envelope/ (not /store/), using the project's own

DSN key;

• reads with the fleet's read-only token;
• deletes through the superuser session login (the handbook's curl recipe), then waits for a

404.

• Umami:
• POSTs /api/send with a browser User-Agent. Umami's bot filter answers 200

{"beep":"boop"} and stores nothing for non-browser UAs; the script detects that.

• checks website_event, then deletes the event, its event_data, and the session if

nothing else uses it, in one transaction.

• Langfuse:
• trace-create through /api/public/ingestion, read back by id;
• DELETE is asynchronous, so the script waits for a 404.

Verified

2026-10-17, all on dev targets:

• GlitchTip: a player-dev probe, absent from player and player-api, deleted, 404

confirmed.

• Umami: a Player (dev) probe, absent from Player, deleted with its session.
• Langfuse: a podcast-dev probe, deleted, 404 confirmed.
• Negative control: --assert-absent "Player (dev)" on its own target reported FAIL

(exit 1) and still deleted the probe, leaving 0 zz-probe events.

Rules

• Prefer dev targets. A probe into a prod project or site needs the operator's go, and

the owning agent should know: they watch those dashboards.

• Never --keep a probe in prod unless the owner asked for it.
• The script prints no secrets. It reads credentials on the mini from the files listed in the

handbook's credentials table.

Not covered

• Source-map resolution: it needs a real live JS file and its debug id; the handbook,

section 7, and the deploy-verify skill cover it.

• VictoriaLogs: deletion is disabled there, so a probe line can't be removed. Write probe

log lines only when the operator agrees, and label them.

• LiteLLM key checks: see mint-credential (call, refused alias, trace).
• Fleet noise: the signal-fleet sees GlitchTip probes while they exist. A short-lived,

labelled one is ignored as a probe, but don't leave them around.