citypaul/.dotfiles/ci-debugging

CI Debugging

Every CI failure is real until proven otherwise. Never assume flakiness.

This skill owns pipeline-specific evidence and environment deltas. For a local or runtime failure outside CI, use debugging.

Getting the Data

Identify the CI provider from the repository configuration, then pull the actual failure output with that provider's connected tool, CLI, or UI before forming hypotheses. Preserve the run, job, attempt, commit SHA, and timestamps so later comparisons refer to the same execution.

For GitHub Actions:

gh run list --branch <branch> --limit 5        # find the failing run
gh run view <run-id> --log-failed              # only the failed steps' logs
gh run view <run-id> --job <job-id> --log      # one job's full log
gh run download <run-id>                       # artifacts (coverage, reports, screenshots)

Rerunning a workflow or changing a CI variable mutates external state and may repeat deploys, migrations, notifications, or other effects. Do that only with explicit authority after inspecting the workflow and failed job for side effects. When authorized, rerun only the failed jobs with gh run rerun <run-id> --failed, or use gh run rerun <run-id> --debug for step-level logging; remove any temporary debug variable afterwards. For another provider, use its equivalent controls without translating commands by guesswork. Compare the failing run against the last green run on the same branch — the diff in commits, dependency lockfiles, and pipeline files between those two runs is the primary suspect list.

If the system under test emits structured telemetry (canonical events, traces, in-memory exporter output from test runs), pull it as evidence alongside the logs — see the observability skill.

Hypothesis-First Diagnosis

Before editing, list plausible causes, rank them by evidence, and test one falsifiable hypothesis at a time. Do not invent a fixed minimum merely to fill a list.

Example hypotheses for a test timeout:

1. Test relies on network access unavailable in CI
2. Parallel test execution causes resource contention
3. CI runner has less memory/CPU than local machine

Local Reproduction

Reproduce locally when a faithful environment is available before pushing fixes.

• Run the exact failing command, not a close equivalent
• Match the CI environment as closely as possible (Node version, env vars)
• If it passes locally, treat the environment delta as evidence to localize; the cause may still be timing, state, or an external dependency

Environment Delta Analysis

Compare CI vs local:

Read the Full Error

• Read the complete error output, not just the last line
• Check preceding log lines and warnings — they often contain the real cause
• Look at stack traces to identify the actual failure point
• Check for earlier failures that may cascade into the visible error

Fix Verification

If the request is diagnosis-only, report the supported root cause, evidence, affected jobs, uncertainty, and recommended fix, then stop. Diagnosis does not authorize workflow, production, dependency, test, or configuration changes.

When the user requested a fix, after identifying it:

1. Explain why it addresses the root cause (not just the symptom)
2. Run the exact failing command locally
3. Verify the fix doesn't mask the real issue (e.g., adding a retry hides a race condition)

Anti-Patterns

Proving Flakiness

Different outcomes across independent runs with the same controlled inputs are evidence that the outcome is nondeterministic. That proves flakiness even when the source is not yet diagnosed. Record which inputs and environment facts were actually controlled, then localize the cause: race, time, shared state, resource pressure, or an external dependency. A single unexplained failure remains a real failure, not permission to re-run until green.

Handoff

Once the root cause is identified, a diagnosis-only handoff ends with the evidence and recommended guard. For an authorized fix, leave the smallest durable regression guard that fits it. For production behavior, write the failing behavior test before the fix using tdd (or characterisation-tests when legacy behavior first needs capture). For provider configuration, permissions, or an external outage, use a config assertion, dry-run, contract check, or monitored recovery proof as appropriate. If no executable guard is feasible, record the evidence and remaining detection path instead of fabricating a test.