getsentry/sentry-cocoa/relay-conformance-audit

Cocoa ↔ Relay conformance audit

The SDK hard-codes many strings and wire formats that must match Relay exactly — header names, envelope item types, data categories, discard reasons, DSC keys, session fields. A mismatch anywhere in that surface fails silently: data is dropped, mis-routed, or miscounted with no error. This skill audits that entire surface. One motivating example of the class: #8322 (a case-sensitive X-Sentry-Rate-Limits header read silently rate-limited all telemetry) — but the skill exists to hunt bugs _like_ it, wherever they occur, not that one bug.

READ-ONLY toward the SDK. Never edit SDK code. Never modify references/findings.md (the human-maintained ignore list). The skill's primary output is `relay-audit:` GitHub issues (created or edited — see "GitHub issues"). The one other permitted write is a draft PR the coverage check opens against getsentry/sentry-cocoa that touches nothing but references/surface-map.md (see "Coverage check"). When the map is out of date, the skill opens that draft PR directly rather than only flagging the gap.

Model

Every run evaluates the full current list of mismatches — no delta tracking, no run-to-run memory. GitHub is the tracker; references/findings.md holds only the ignore list. A mismatch is in one of three states:

• Ignored — listed in references/findings.md with an explicit _ignore-scenario_. Skipped (counted only) while the scenario holds; becomes needs-action when it no longer holds.
• Needs action — everything else, unless cross-SDK corroboration drops it. Matched to a GitHub issue by fingerprint and filed via the pipeline in "GitHub issues" (open match → edit; expired-ignore closed match whose back-ref proves the audit closed it → reopen; other closed match → comment once; no match → create).
• Cross-SDK false positive — a needs-action mismatch whose all-sdks-agree verdict drops it: no issue, no per-finding detail (so a false positive never reads as real), counted only as an aggregate dropped all-sdks-agree <n> in the console summary (see "Cross-SDK corroboration").

A live GitHub search precedes every create, so the audit never opens a duplicate. It also never closes issues or edits the ignore list itself — the reverse checks (Procedure step 6) only surface stale open issues and stale ignore entries for a human to act on.

Procedure

1. Record git rev-parse HEAD (full SHA — needed for permalinks), git rev-parse --short HEAD, and date +%Y-%m-%d via shell (never hardcode); the full SHA builds the code links, the short SHA + date stamp each issue.
2. For each area in references/surface-map.md, spawn one read-only subagent (parallel). Contract: fetch the listed Relay source / develop-docs pages, read the listed Cocoa files, diff per the area's notes. Return mismatches as {area, severity HIGH|MEDIUM|LOW, file + symbol, line range (start–end), exact wire strings, spec citation, failure mode}. The line range is required — it builds the commit-pinned permalink. Severity = silent blast radius, not fix effort. In parallel with these, spawn the coverage-check subagent (see "Coverage check").
3. Classify each mismatch against references/findings.md on fingerprint (area + file + normalized summary — never line numbers/severity; those vary between runs): listed and ignore-scenario still holds → ignored, skip; listed but scenario no longer holds → needs action (expired-ignore), carrying that flag and, if present, the entry's (relay-audit #<n>) back-reference; not listed → needs action. Every expired-ignore finding also means its findings.md row is now stale — always surface that row in the console summary under ignore-list entries expired — verify & prune for a human to remove, independent of which issue path Dedup then takes (Dedup reopens only when the entry's back-ref matches the closed issue it hits; otherwise create / edit-open / comment-closed like any other finding). Without this, entries added before back-refs existed (or that take create/open-edit) would keep the finding permanently flagged expired-ignore and permanently exempt from the all-sdks-agree drop.
4. Cross-SDK corroboration (see below): for every mismatch that is still needs action after step 3, check how the peer SDKs implement the same wire surface. This is the confidence gate — it strengthens cocoa-only findings with exact peer code locations and silently drops all-sdks-agree false positives. Exception: an `expired-ignore` finding (step 3) is never dropped here — it still reproduces and its stale ignore row must be surfaced, so it always proceeds to step 5 regardless of verdict (its cross-SDK result is recorded in whichever issue Dedup files/reopens/comments).
5. For each mismatch that survives the cross-SDK gate, run the GitHub-issue pipeline (see "GitHub issues" → Dedup) to file it.
6. Reverse checks (no-longer-reproduces). Two symmetric checks against this run's raw mismatch set from Procedure step 2 (every mismatch observed — reported, ignored at step 3, and dropped as all-sdks-agree at the cross-SDK gate; all still reproduce, only the pipeline's disposition differs — so match on the raw set, not the filed subset; enumerating a subset and forgetting a category is exactly what mis-flags):

• 6a — open issues. List all open relay-audit: issues (gh issue list --repo getsentry/sentry-cocoa --search "relay-audit: in:title" --state open --limit 1000 --json number,title,body; body is required to reconstruct each issue's fingerprint — file lives in the body, so without --json ...,body the match can't be built; same pagination guard as dedup — if the result equals the limit, page before trusting it). Any open issue with no matching mismatch this run is a no-longer-reproduces candidate: do not close it (a run may miss a finding for benign reasons — a moved file, a flaky peer search); list it in the console summary under "no longer reproduces — verify & close". This backs the Model's promise; without it a fixed mismatch's issue would linger forever.
• 6b — ignore-list entries. Diff each references/findings.md ignored entry's fingerprint against the same raw mismatch set. Any ignored entry with no matching mismatch this run no longer reproduces: do not edit `findings.md` (read-only toward it); list its fingerprint in the console summary under "ignore-list entries no longer reproducing — verify & prune" so a human prunes it. This backs the Model's promise for the ignore list, symmetric to 6a for issues.

1. Print the console summary (see "Console summary") and append the coverage-check result. If gh is unauthenticated or any gh call fails, print everything and state loudly that the issue write FAILED — never report success on a failed write.

Coverage check (surface-map self-audit)

The audit only inspects what the surface map lists, so a stale map is a silent coverage hole. Every run, one extra subagent checks the map itself — from both directions:

• SDK side: list protocol-relevant code the map does NOT reference. Grep the SDK for wire-emitting surfaces: serialize implementations feeding envelopes, envelope item construction, setValue(_:forHTTPHeaderField:) / header reads, files under Sources/Swift/Networking/ and Sources/Swift/Protocol/, new integrations writing envelope items. Compare against the files the map lists; report unlisted ones that put bytes on the wire.
• Relay side: for each Relay/spec source the map cites, check it still exists (note moved paths) and skim Relay's protocol enums (DataCategory, ItemType, discard reasons, DSC keys) for members added since the map's checks were written that no area would catch.
• Ignore non-wire code (SentryCrash/ internals, UI, tests) — the bar is "could a mismatch here silently corrupt what Relay receives?".

Outcome:

• Map up to date → one line in the console summary: coverage: OK.
• Gaps found → list them in the console summary under coverage: GAPS, and directly open a draft PR against getsentry/sentry-cocoa (base main) that updates ONLY references/surface-map.md — don't just describe it, push it. The map drifts as the SDK changes, so this is the mechanism that keeps it current. Mechanically:

1. Edit skills/relay-conformance-audit/references/surface-map.md only — extend an area's file list, fix a moved Relay path, or add a new area (with What it is / Cocoa / Relay-Spec / Check, all links as blob/main (Cocoa) or blob/master (Relay), matching the file's existing format).
2. Branch chore/relay-audit-surface-map-<date>. Commit just that file: chore: update relay-audit surface map with #skip-changelog in the body.
3. Push and gh pr create --draft --base main --title "chore: update relay-audit surface map"; body = the gap list + one line per gap on why it belongs in the map + "Opened by relay-conformance-audit coverage check, <DATE>, @<SHA>." Put the resulting PR URL in the console summary's coverage line.

• This is the skill's only permitted repository write (issue creation/editing aside): never touch SDK code, findings.md, or SKILL.md in that PR; leave it as a draft and never merge — a human reviews. If a draft PR from a previous run is still open, push to that branch and update its body instead of opening a second one. If gh isn't authenticated or the push fails, say so loudly in the console summary and include the diff inline — never report the PR as opened when it wasn't.
• Gaps are NOT mismatches: list them in the coverage section only; the newly-discovered files get audited by the next run after the map PR merges.

Cross-SDK corroboration

Diffing Cocoa against Relay + develop-docs alone produces false positives: when the develop-docs spec is aspirational/wrong, or Relay tolerates a value, the docs-vs-Relay disagreement gets blamed on Cocoa. The peer SDKs are a second source of truth. If Cocoa is the only SDK that diverges, the finding is almost certainly real; if every SDK does the same thing Cocoa does, the "mismatch" is almost certainly a misreading of the spec, not a Cocoa bug.

Run this only on mismatches that are still needs action after the findings.md match (step 3) — never on ignored entries. That bounds the extra searches to exactly the findings that would be reported.

Peer repos, in priority order (mobile first): getsentry/sentry-java, getsentry/sentry-dart, getsentry/sentry-react-native, then getsentry/sentry-javascript, getsentry/sentry-python.

Locate peer code by the wire string, not a file map. The exact literal the mismatch is about (discard-reason string, header name, envelope item-type, DSC key, session field) is the stable locator — search each repo for it, e.g. gh search code --repo getsentry/sentry-java "ratelimit_backoff". This lands directly on the peer's implementation with no per-SDK map to maintain and keep in sync. If the literal search misses (a repo names the constant differently), fall back to the role-analogous file, then move on — a peer you genuinely can't locate is recorded as not-checked, not as agreement.