humansintheloop-dev/humansintheloop-dev-workflow-and-tools/debugging-ci-failures

Watching and Debugging CI Builds

This skill covers two scenarios: watching a CI build to completion after a push, and diagnosing failures when a build fails.

Overview

Follow these steps in order:

1. Detect the CI system the project uses (see "Detect CI system" below).
2. Phase 0: Watch the build — poll until the build completes. If it succeeds, you're done.
3. Phase 1: Gather evidence (only if the build failed) — collect the failed job summary, download test artifacts, and read TEST-*.xml files for the full stack trace and container logs.
4. Phase 2: Analyze evidence — identify the root cause (not the symptom), determine whether the failure is related to your changes, and fix the root cause.
5. Verify the fix by running the same command that failed in CI.

Phases 0 and 1 have CI-specific commands — see the "GitHub Actions" and "CircleCI" sections below. Phase 2 is CI-agnostic.

Detect CI system

Check which CI system the project uses:

• .github/workflows/*.yml → GitHub Actions
• .circleci/config.yml → CircleCI

For CircleCI, extract the org and repo from git remote get-url origin to construct API URLs.

GitHub Actions

Use the gh CLI (no caching issues).

Phase 0: Watch a build

Poll for the latest run on the branch:

gh run list --branch <branch> --limit 1 --json status,conclusion,databaseId,name

Then poll the specific run until it completes:

gh run view <run-id> --json status,conclusion,jobs

Repeat every 10-15 seconds until status is completed. Then check conclusion for success or failure.

If the build failed, proceed to Phase 1.

Phase 1: Gather evidence

Step 1: Get the failed job summary

gh run view <run-id> --log-failed 2>&1 | grep -i "FAILED\|error\|Exception" | head -30

This gives you the failing task/test name and a high-level error.

Step 2: Download test artifacts

Test results are often saved via actions/upload-artifact, frequently inside a composite or reusable action rather than the workflow file itself. Do not decide whether artifacts exist by reading the workflow definition. List the run's artifacts:

gh api repos/<owner>/<repo>/actions/runs/<run-id>/artifacts --jq '.artifacts[] | {name, size_in_bytes}'

Then download the one containing test results:

gh run download <run-id> --name <artifact-name> --dir test-reports

Step 3: Read the TEST-*.xml files — see "Reading TEST-*.xml files" below.

CircleCI

Use the circleci CLI. It has no caching issues (unlike WebFetch) and returns proper exit codes.

The project slug has the form gh/<org>/<repo> (e.g., gh/eventuate-foundation/eventuate-common). When run from inside the project's git checkout, the CLI infers the slug from the git remote and --project can be omitted.

Phase 0: Watch a build

circleci run watch blocks until the run reaches a terminal state and its exit code reflects the outcome:

• 0 — all workflows succeeded
• 1 — one or more workflows failed
• 6 — cancelled
• 8 — timed out

circleci run watch --project gh/<org>/<repo> --branch <branch>

Or, from inside the project after git push:

circleci run watch --sha $(git rev-parse HEAD)

If you need to poll instead of block (e.g., interleaving other work), use run list + run get:

circleci run list --project gh/<org>/<repo> --branch <branch> --limit 1 --json --jq '.[0] | {id, phase, outcome}'
circleci run get <run-id> --json --jq '{phase, workflows: [.workflows[] | {name, phase, jobs: [.jobs[] | {id, name, phase, outcome}]}]}'

If the build failed, proceed to Phase 1.

Phase 1: Gather evidence

Step 1: Identify the failing job(s)

circleci run get <run-id> --json --jq '.workflows[].jobs[] | select(.outcome == "failed")'

Step 2: Get the failed step and its output

List the job's steps and find the one with a non-zero exit code:

circleci job output list <job-id> --json --jq '.steps[] | select(.exit_code != 0) | {num, name, exit_code}'

Then fetch that step's output. Use --condensed for an AI-friendly filtered view:

circleci job output get <job-id> --step-num <N> --condensed

Step 3: Check parsed test results

If the job uses store_test_results, CircleCI has already parsed them — this may be enough to identify the failing test without downloading anything:

circleci testresult list <job-id>

Step 4: Download test artifacts

List artifacts, then download into a project-relative directory:

circleci artifact <job-id> --json
circleci artifact <job-id> --output test-reports

Step 5: Read the TEST-*.xml files — see "Reading TEST-*.xml files" below.

Reading TEST-*.xml files

Find and read the relevant TEST-*.xml file for the failing test. These files contain:

• The full stack trace (not truncated like CI logs)
• Container/service logs (stdout/stderr captured during the test)
• The actual root cause error, not just the symptom

Testcontainers-based tests typically configure .withLogConsumer(new Slf4jLogConsumer(logger).withPrefix("SVC <service-name>")), so container logs appear in the XML prefixed with [SVC <service-name>]. Use this prefix to filter for the specific container's output when searching for errors.

Use the Glob tool to find the relevant test result:

Glob with pattern="**/TEST-*FailingTestName*.xml" path="test-reports"

Use the Grep tool to search within large XML files for the root cause:

Grep with pattern="ERROR|Exception|FATAL|Application run failed" path="test-reports/path/to/TEST-*.xml" output_mode="content"

Do NOT read HTML reports — they are for humans in browsers. The XML files contain the same information in a machine-readable format.

Do NOT download artifacts to `/tmp` or other directories outside the sandbox — download to a project-relative path like test-reports/.

Phase 2: Analyze Evidence and Determine Problem

Only after gathering evidence, analyze it systematically.

Step 1: Identify the root cause error

Look past the test framework wrappers. A typical failure chain looks like:

TestName > testMethod() FAILED
  IllegalStateException          ← symptom (Spring context failed)
    CompletionException          ← symptom (async wrapper)
      ContainerLaunchException   ← symptom (container didn't start)
        THE ACTUAL ERROR         ← root cause (e.g., missing bean, bad config, JDK bug)

Read the full chain — the root cause is at the bottom.

Step 2: Determine if the failure is related to your changes

• Check if the failing test was actually passing before by examining the previous successful run
• A test marked FROM-CACHE in a previous run means it wasn't re-executed then — it may have been broken already
• Don't assume your changes caused the failure just because they were in the same commit

Step 3: Fix the root cause, not the symptom

• If a container crashes on startup, read the container logs in the XML to find out why
• If a Spring context fails to load, find the specific bean creation error
• If a test times out, determine what it was waiting for

Common Mistakes to Avoid

• Don't skip evidence gathering: Never propose a fix based on the one-line CI summary alone
• Don't assume based on recent changes: Just because you recently modified something doesn't mean it's the cause
• Don't confuse file types: A FileNotFoundException for a .yml file is different from a missing .class file
• Don't implement fixes without understanding: If you can't explain exactly why the fix works, you don't understand the problem
• Don't search JAR files or external libraries when the error is in your project code
• Don't launch research agents when a targeted grep or read of the test XML will give you the answer
• CircleCI orb parameters: When a build fails due to wrong JVM/tool version, check the orb's parameters (e.g., java_version_to_install, machine_image). Compare with other upgraded projects that use the same orb.

Verification Steps

1. After implementing a fix, run the same command that failed in CI
2. Verify the specific test or task that failed now passes
3. If CI fails again with the same error, your fix was wrong — go back to Phase 1

Pattern-Based Problems

When fixing issues caused by naming conventions or patterns:

1. Search the entire codebase for similar occurrences before making any changes
2. Fix ALL instances in a single commit
3. Never commit partial fixes for pattern-based problems