haacked/dotfiles/ci-monitor

Monitor GitHub CI checks for the current PR, wait for completion, classify failures as flaky or legit, and guide fixes for legit failures.

On a repo behind a Trunk merge queue, a green PR is not a finished PR: the queue re-tests it on a branch of its own, and the PR's checks never reflect that. Step 7 covers it. You never cancel or merge, and never enqueue a PR for the first time — those are the developer's calls. The one queue action you may take is re-enqueueing a PR the queue dropped, through Step 7c's gate.

Arguments:

• (no argument) - Detect PR from current branch
• <pr-number> - Monitor checks for a specific PR (e.g., 123)
• <pr-url> - Monitor checks for PR by URL (e.g., https://github.com/org/repo/pull/123)

Optional Flags:

• --no-fix - Monitor and report only; do not attempt fixes. A flaky-drop re-enqueue (Step 7c) involves no code change, so it still happens; use --no-requeue to suppress that too.
• --no-requeue - Never comment /trunk merge, even for a PR the queue dropped; report the decision and hand over the command instead. Fixing is unaffected.
• --timeout <minutes> - Override default 30-minute timeout
• --auto-approve-base-sync - For a re-gated fork PR, auto-approve the gated workflows only when the sole change since your last approval was a base-branch sync (merge/rebase of the base branch) with the contributor's patch unchanged. Off by default; without it, gated runs are always left for you to approve manually.

Usage examples:

• /ci-monitor - Monitor CI for current branch's PR
• /ci-monitor 123 - Monitor CI for PR #123
• /ci-monitor --no-fix - Monitor only, report failures without fixing
• /ci-monitor 123 --no-requeue - Monitor, but never re-enqueue a dropped PR
• /ci-monitor https://github.com/org/repo/pull/123 --timeout 45

Implementation

Step 1: Parse Arguments

Extract the PR identifier and flags from $ARGUMENTS.

Flags to detect:

• --no-fix - Set NO_FIX=true
• --no-requeue - Set NO_REQUEUE=true (default: false)
• --unattended - Set UNATTENDED=true (default: false): nobody is watching this run, so treat this skill's approval prompts as approved and skip attended-only actions (7b's waiting conflict carve-out). babysit-prs passes it on every dispatch.
• --timeout <N> - Set TIMEOUT_MINUTES=N (default: 30)
• --auto-approve-base-sync - Set AUTO_APPROVE_BASE_SYNC=true (default: false)

Remove flags from the argument string, leaving just the PR identifier (number, URL, or empty).

Run the detection script:

~/.dotfiles/bin/detect-pr.sh --json ${PR_IDENTIFIER:+"$PR_IDENTIFIER"} 2>&1

Save the output as PR_DATA. If the error field is non-null, display the error and stop.

Extract and save: PR_NUMBER, ORG, REPO, HEAD_BRANCH.

Tell the user: "Monitoring CI checks for PR #$PR_NUMBER ($ORG/$REPO)…"

Record the current time as START_TIME for timeout tracking.

Initialize RETRY_COUNT=0 and MAX_RETRIES=3.

Initialize FLAKY_RERUN_COUNT=0 and MAX_FLAKY_RERUNS=2. Flaky re-runs happen automatically (Step 4d) and do not count against RETRY_COUNT, so this separate bound stops an over-eager "flaky" classification from re-running the same failures forever.

Initialize AUTO_REQUEUE_COUNT=0 and MAX_AUTO_REQUEUES=2 (mirrors CI_MAX_AUTO_REQUEUES in ci-helpers.sh, the way MAX_RETRIES mirrors CI_MAX_FIX_RETRIES). This is the in-session bound on Step 7c re-enqueues; the cross-session bound is the comment count inside ci-requeue-check.sh.

Initialize EVICTION_FIX=false. Step 7b sets it when an in-session fix — an eviction's fix cycle, or the conflict fix's rebase — must finish through 7c once the PR's own CI is green again.

Initialize CONFLICT_FIX_DONE=false. Step 7b's conflict fix sets it when it pushes a rebase, so a PR that conflicts again in the same session gets a report instead of a second rebase — the base is moving faster than one session should chase.

Initialize ALERTED_APPROVAL_RUNS to an empty set. It tracks the run_ids of awaiting-approval workflows you have already alerted about, so re-polling does not re-alert for the same runs (see Step 6).

Initialize NO_CHECKS_RETRY_SHA to empty. Step 2's no_checks route uses it to give a fresh commit one grace retry before concluding a PR genuinely has no CI.

Step 2: Check CI Status

Run the status check:

~/.dotfiles/ai/skills/ci-monitor/scripts/ci-check-status.sh $PR_NUMBER "$ORG/$REPO" 2>&1

Save the output as CHECK_DATA.

Route based on status:

• If awaiting_approval is greater than 0: Go to Step 6 (Awaiting Approval). Check this first, before every rule below. An outside-contributor (fork) PR can report no_checks or even all_passed in the rollup while its real CI sits gated behind your approval, so this must take precedence.
• If status is "no_checks": A commit can take a few seconds for GitHub to register any check against, and gh pr view (which ci-check-status.sh now reads) returns an empty rollup during that window rather than erroring the way the old gh pr checks call did — so give it one grace retry before concluding there is truly no CI. If CHECK_DATA.head_sha differs from NO_CHECKS_RETRY_SHA (the first no_checks report for this commit), set NO_CHECKS_RETRY_SHA to it, report "No CI checks registered yet for this commit. Checking again in 60 seconds…", wait 60 seconds (respecting the same timeout check as Step 3), and go back to Step 2. Otherwise (a second consecutive no_checks for the same commit) tell the user "No CI checks found for this PR.", apply the fork caveat below, and go to Step 7.
• If all_passed is true: Report "All CI checks passed!" with a summary of check counts, apply the fork caveat below, and go to Step 7.
• If status is "in_progress": Go to Step 3 (Polling Loop).
• If status is "completed" and there are failures: Go to Step 4 (Triage Failures).
• If status is "completed", there are no failures, and all_passed is false (for example, all remaining checks are skipped, cancelled, or neutral): Report that CI checks have completed with no failures, show a final summary including all buckets (passed, skipped, cancelled, neutral, etc.), apply the fork caveat below, and go to Step 7.

The three terminal branches above end at Step 7 rather than stopping, because on a merge-queue repo the PR's own checks are not the last word. Timeouts and retry-limit stops elsewhere in this skill still stop where they are. Step 7 reports no_queue and stops immediately on a repo without a Trunk queue.

Fork caveat: When is_cross_repository is true and you reach Step 7 from one of the three branches above (no_checks, all_passed, or completed-with-no-failures), add: "This is a fork PR. If you expected gated workflows that still need maintainer approval, confirm on the PR's Checks tab; approval detection relies on a runs-API call that could have been missed."

Step 3: Polling Loop

Checks are still running. Report progress:

"CI checks in progress: $PASSED/$TOTAL passed, $PENDING pending. Checking again in 60 seconds…"

Retain only the summary scalars from CHECK_DATA (status, total, passed, failed, pending). Discard the full JSON until status is "completed".

Under Switchboard (SB_SESSION_ID is set, UNATTENDED is false, and no check has failed), hand the wait to sbd instead of sleeping, so the session stops while CI runs. An unattended run keeps the loop below, because its caller (such as babysit-prs) handles other PRs after this one returns and relies on --timeout. sbd resolves a ci wait as soon as any check fails, so once failed is above 0 the loop below is used as well:

sb wait --kind ci --target "$ORG/$REPO#$PR_NUMBER" --note "ci-monitor: CI on #$PR_NUMBER"

If the command succeeds, tell the user "Waiting on CI for #$PR_NUMBER through Switchboard." and end your turn without running another tool. sbd resumes the session with a message such as Switchboard: CI on #123: the checks passed. once the checks finish. When that message arrives, set START_TIME to the current time and go back to Step 2. The wait has no timeout, so --timeout does not apply while it is open, and the loop's timeout counts from the resume. If sb wait fails, use the loop below.

Otherwise wait 60 seconds:

sleep 60

Check timeout: Calculate elapsed time. If elapsed exceeds TIMEOUT_MINUTES * 60 seconds, tell the user "Timeout reached after $TIMEOUT_MINUTES minutes. $PENDING checks still pending." and show the current check status. Stop.

Go back to Step 2 to re-check status.

Step 4: Triage Failures

For each failed check in the failed_checks array:

4a. Fetch failure logs:

If the check has a run_id:

~/.dotfiles/ai/skills/ci-monitor/scripts/ci-fetch-logs.sh $RUN_ID "$ORG/$REPO" 2>&1

Save as LOG_DATA.

If the check does not have a run_id (e.g., a non-GitHub Actions status check):

• Mark it as uncertain — no logs are available to classify it
• Report the check name and link to the user
• Do not attempt to classify, re-run via gh run rerun, or auto-fix this check
• Skip steps 4b and 4c for this check; move on to the next failed check

4b. Classify failure:

Pass the log excerpt via stdin to the classifier:

printf '%s\n' "$LOG_EXCERPT" | ~/.dotfiles/ai/skills/ci-monitor/scripts/ci-classify-failure.sh $PR_NUMBER "$WORKFLOW_NAME" "$ORG/$REPO" 2>&1

Where $LOG_EXCERPT is the combined log_excerpt from all failed jobs in LOG_DATA, and $WORKFLOW_NAME is the check's workflow field.

Save as CLASSIFICATION.

4c. Present findings to the user:

For each failure, report:

• Check name and link
• Classification: flaky / legit / uncertain
• Confidence score
• Reasoning (why classified this way)
• Key log excerpt (first 20-30 relevant lines)

4d. Handle by classification:

All flaky: Report the failures as flaky, then automatically re-run them. Do not prompt for permission.

Flaky-rerun bound: First check FLAKY_RERUN_COUNT. If it is >= MAX_FLAKY_RERUNS, the same failures have been re-run as "flaky" too many times to still be plausibly flaky. Stop auto-re-running: tell the user "These workflows have failed and been re-run as flaky $MAX_FLAKY_RERUNS times; they're likely not flaky. Investigate manually." List the affected checks and their links, and stop.

Otherwise, re-run each failed check that has a run_id:

gh run rerun $RUN_ID --failed --repo "$ORG/$REPO"

Do not report flaky failures to #flakey-tests. Step 4c already gave the user each failure's link, so the user decides whether to report it. Spawn the report-flake agent only when the user asks for a report.

Increment FLAKY_RERUN_COUNT, then go back to Step 2 to monitor the re-run (this does NOT count against RETRY_COUNT).

All legit or mixed (with `--no-fix`): Report the findings and stop. Do not attempt fixes.

All legit or mixed (without `--no-fix`): Build LEGIT_FAILURES (see below), then go to Step 5.

Uncertain classifications: Present your own analysis of the log excerpt alongside the automated classification. Use your judgment to refine the classification before proceeding. Treat uncertain failures you judge to be legit the same as legit failures when building LEGIT_FAILURES.

Building `LEGIT_FAILURES`: Before entering Step 5, construct an array containing one entry per legit or uncertain failure. Each entry carries only compact identifiers:

• check_name — the check's name
• check_link — the check's URL
• run_id — the run ID (may be null for non-Actions checks)
• workflow — the workflow name
• classification — the full CLASSIFICATION object from step 4b (scores and reasoning, no log text)