metaculus/forecasting-tools/analyze-bot-failures

Analyze Bot Forecasting Failures

Workflow for diagnosing failures in the run-bot-aib-tournament.yaml GitHub Actions workflow, which runs ~30 bot jobs every 30 minutes via run_bots.py.

Step 1: Pull and aggregate the failure logs

A GitHub token is required (the job-log API rejects unauthenticated requests). Check GITHUB_TOKEN env var or gh auth token; if neither works, ask the user to run gh auth login.

uv run python scripts/skills/analyze_bot_failures/analyze_bot_run_failures.py --since 1d

Useful options:

• --since 12h|2d|1w or an ISO datetime (default 1d)
• --run-id <id> to analyze one specific run
• --max-runs <n> cap on runs fetched (default 50)

Output goes to logs/workflow_failure_analysis/<timestamp>/:

• report.md — counts by category/bot/question, plus failure groups (deduped by normalized signature) with an example message, traceback, and the deepest repo code frame
• failures.json — every parsed failure with full messages and signatures (machine-readable)
• raw_logs/run<id>_<bot>.log — complete raw log of each failed job

Step 2: Read the report, then verify against raw logs

Read report.md first. The parser extracts full tracebacks when present (falling back to ❌ Exception: ... | Message: ... summary lines, then to the raw log tail), so each failure group usually includes a "deepest repo frame" — the last traceback frame inside this repo rather than a dependency — which is the first place to look in the code. Still spot-check 2-3 raw logs for the most frequent signatures to see surrounding context (retry attempts, which LLM call failed). Failures of type UnparsedFailure mean nothing parseable was found — the job likely failed at the infrastructure level (uv install failures, token resolution, job-level timeout at 55 min, cancellation) — read their raw logs directly.

Step 3: Triage — transient noise vs real bug

Likely transient (usually not worth fixing, just note frequency):

• Provider 5xx / overloaded / rate-limit errors that hit one bot in one run
• Occasional timeouts that succeeded on the in-run retry (run_bots.py retries reports whose error contains "TimeoutError")

Likely real bugs (investigate the code):

• The same bot failing across most runs (config/model-name/credentials issue in its RunBotConfig in run_bots.py)
• Structured-output or validation errors clustering on one question type (binary/MC/numeric/group) — points at parsing or prompt code
• Prediction validation errors (probabilities out of bounds, CDF/percentile issues) — points at report data models
• The same question ID recurring across bots/runs — candidate for POST_IDS_TO_SKIP or POST_IDS_TO_NOT_RAISE_ERRORS_FOR in run_bots.py
• Tracebacks ending inside forecasting_tools/ rather than a provider SDK

Step 4: Map errors to code

It may not be in any of the above, so be willing to search the codebase.

Step 5: Summarize findings

Report back with:

1. Failure counts by category and bot (from report.md)
2. Real bugs found: each with evidence (log excerpt + code location) and a proposed fix
3. Transient noise: what to ignore and why
4. Question skip candidates: recurring question IDs with failure counts
5. Do not change code or skip lists without confirming the proposed fixes with the user first