ai4scientist/nano-scientist/auto-review-loop

Auto Review Loop: Autonomous Research Improvement

Autonomously iterate: review → implement fixes → re-review, until the external reviewer gives a positive assessment or MAX_ROUNDS is reached.

Context: $ARGUMENTS

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10, or verdict contains "accept", "sufficient", "ready for submission"
• REVIEW_DOC: review-stage/AUTO_REVIEW.md (cumulative log) (fall back to `./AUTO_REVIEW.md` for legacy projects)
• REVIEWER_MODEL = gpt-5.4 — Model used via Codex MCP. Must be an OpenAI model (e.g., gpt-5.4, o3, gpt-4o)
• REVIEWER_BACKEND = `codex` — Default: Codex MCP (xhigh). Override with — reviewer: oracle-pro for GPT-5.4 Pro via Oracle MCP. See shared-references/reviewer-routing.md.
• OUTPUT_DIR = `review-stage/` — All review-stage outputs go here. Create the directory if it doesn't exist.
• HUMAN_CHECKPOINT = false — When true, pause after each round's review (Phase B) and present the score + weaknesses to the user. Wait for user input before proceeding to Phase C. The user can: approve the suggested fixes, provide custom modification instructions, skip specific fixes, or stop the loop early. When false (default), the loop runs fully autonomously.
• COMPACT = false — When true, (1) read EXPERIMENT_LOG.md and findings.md instead of parsing full logs on session recovery, (2) append key findings to findings.md after each round.
• REVIEWER_DIFFICULTY = medium — Controls how adversarial the reviewer is. Three levels:
• medium (default): Current behavior — MCP-based review, Claude controls what context GPT sees.
• hard: Adds Reviewer Memory (GPT tracks its own suspicions across rounds) + Debate Protocol (Claude can rebut, GPT rules).
• nightmare: Everything in hard + GPT reads the repo directly via codex exec (Claude cannot filter what GPT sees) + Adversarial Verification (GPT independently checks if code matches claims).

💡 Override: /auto-review-loop "topic" — compact: true, human checkpoint: true, difficulty: hard

State Persistence (Compact Recovery)

Long-running loops may hit the context window limit, triggering automatic compaction. To survive this, persist state to review-stage/REVIEW_STATE.json after each round:

{
  "round": 2,
  "threadId": "019cd392-...",
  "status": "in_progress",
  "difficulty": "medium",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "pending_experiments": ["screen_name_1"],
  "timestamp": "2026-03-13T21:00:00"
}

Write this file at the end of every Phase E (after documenting the round). Overwrite each time — only the latest state matters.

On completion (positive assessment or max rounds), set "status": "completed" so future invocations don't accidentally resume a finished loop.

Output Protocols

Follow these shared protocols for all output files:

- [Output Versioning Protocol](../shared-references/output-versioning.md) — write timestamped file first, then copy to fixed name

- [Output Manifest Protocol](../shared-references/output-manifest.md) — log every output to MANIFEST.md

- [Output Language Protocol](../shared-references/output-language.md) — respect the project's language setting

Workflow

Initialization

1. Check for `review-stage/REVIEW_STATE.json` (fall back to `./REVIEW_STATE.json` if not found — legacy path):

• If neither path exists: fresh start (normal case, identical to behavior before this feature existed)
• If it exists AND status is "completed": fresh start (previous loop finished normally)
• If it exists AND status is "in_progress" AND timestamp is older than 24 hours: fresh start (stale state from a killed/abandoned run — delete the file and start over)
• If it exists AND status is "in_progress" AND timestamp is within 24 hours: resume
• Read the state file to recover round, threadId, last_score, pending_experiments
• Read review-stage/AUTO_REVIEW.md to restore full context of prior rounds (fall back to `./AUTO_REVIEW.md`)
• If pending_experiments is non-empty, check if they have completed (e.g., check screen sessions)
• Resume from the next round (round = saved round + 1)
• Log: "Recovered from context compaction. Resuming at Round N."

1. Read project narrative documents, memory files, and any prior review documents. When `COMPACT = true` and compact files exist: read findings.md + EXPERIMENT_LOG.md instead of full review-stage/AUTO_REVIEW.md and raw logs — saves context window.
2. Read recent experiment results (check output directories, logs)
3. Identify current weaknesses and open TODOs from prior reviews
4. Initialize round counter = 1 (unless recovered from state file)
5. Create/update review-stage/AUTO_REVIEW.md with header and timestamp

Loop (repeat up to MAX_ROUNDS)

Phase A: Review

Route by REVIEWER_DIFFICULTY:

Medium (default) — MCP Review

Send comprehensive context to the external reviewer:

mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    [Round N/MAX_ROUNDS of autonomous review loop]
 
    [Full research context: claims, methods, results, known weaknesses]
    [Changes since last round, if any]
 
    Please act as a senior ML reviewer (NeurIPS/ICML level).
 
    1. Score this work 1-10 for a top venue
    2. List remaining critical weaknesses (ranked by severity)
    3. For each weakness, specify the MINIMUM fix (experiment, analysis, or reframing)
    4. State clearly: is this READY for submission? Yes/No/Almost
 
    Be brutally honest. If the work is ready, say so clearly.

If this is round 2+, use mcp__codex__codex-reply with the saved threadId to maintain conversation context.

Hard — MCP Review + Reviewer Memory

Same as medium, but prepend Reviewer Memory to the prompt:

mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    [Round N/MAX_ROUNDS of autonomous review loop]
 
    ## Your Reviewer Memory (persistent across rounds)
    [Paste full contents of REVIEWER_MEMORY.md here]
 
    IMPORTANT: You have memory from prior rounds. Check whether your
    previous suspicions were genuinely addressed or merely sidestepped.
    The author (Claude) controls what context you see — be skeptical
    of convenient omissions.
 
    [Full research context, changes since last round...]
 
    Please act as a senior ML reviewer (NeurIPS/ICML level).
    1. Score this work 1-10 for a top venue
    2. List remaining critical weaknesses (ranked by severity)
    3. For each weakness, specify the MINIMUM fix
    4. State clearly: is this READY for submission? Yes/No/Almost
    5. **Memory update**: List any new suspicions, unresolved concerns,
       or patterns you want to track in future rounds.
 
    Be brutally honest. Actively look for things the author might be hiding.

Nightmare — Codex Exec (GPT reads repo directly)

Do NOT use MCP. Instead, let GPT access the repo autonomously via codex exec:

codex exec "$(cat <<'PROMPT'
You are an adversarial senior ML reviewer (NeurIPS/ICML level).
This is Round N/MAX_ROUNDS of an autonomous review loop.
 
## Your Reviewer Memory (persistent across rounds)
[Paste full contents of REVIEWER_MEMORY.md]
 
## Instructions
You have FULL READ ACCESS to this repository. The author (Claude) does NOT
control what you see — explore freely. Your job is to find problems the
author might hide or downplay.
 
DO THE FOLLOWING:
1. Read the experiment code, results files (JSON/CSV), and logs YOURSELF
2. Verify that reported numbers match what's actually in the output files
3. Check if evaluation metrics are computed correctly (ground truth, not model output)
4. Look for cherry-picked results, missing ablations, or suspicious hyperparameter choices
5. Read NARRATIVE_REPORT.md or review-stage/AUTO_REVIEW.md for the author's claims — then verify each against code
 
OUTPUT FORMAT:
- Score: X/10
- Verdict: ready / almost / not ready
- Verified claims: [which claims you independently confirmed]
- Unverified/false claims: [which claims don't match the code or results]
- Weaknesses (ranked): [with MINIMUM fix for each]
- Memory update: [new suspicions and patterns to track next round]
 
Be adversarial. Trust nothing the author tells you — verify everything yourself.
PROMPT
)" --skip-git-repo-check 2>&1

Key difference: In nightmare mode, GPT independently reads code, result files, and logs. Claude cannot filter or curate what GPT sees. This is the closest analog to a real hostile reviewer who reads your actual paper + supplementary materials.

Phase B: Parse Assessment

CRITICAL: Save the FULL raw response from the external reviewer verbatim (store in a variable for Phase E). Do NOT discard or summarize — the raw text is the primary record.

Then extract structured fields:

• Score (numeric 1-10)
• Verdict ("ready" / "almost" / "not ready")
• Action items (ranked list of fixes)

STOP CONDITION: If score >= 6 AND verdict contains "ready" or "almost" → stop loop, document final state.

Phase B.5: Reviewer Memory Update (hard + nightmare only)

Skip entirely if `REVIEWER_DIFFICULTY = medium`.

After parsing the assessment, update REVIEWER_MEMORY.md in the project root:

# Reviewer Memory
 
## Round 1 — Score: X/10
-**Suspicion**: [what the reviewer flagged]
-**Unresolved**: [concerns not yet addressed]
-**Patterns**: [recurring issues the reviewer noticed]
 
## Round 2 — Score: X/10
-**Previous suspicions addressed?**: [yes/no for each, with reviewer's judgment]
-**New suspicions**: [...]
-**Unresolved**: [carried forward + new]

Rules:

• Append each round, never delete prior rounds (audit trail)
• If the reviewer's response includes a "Memory update" section, copy it verbatim
• This file is passed back to GPT in the next round's Phase A — it is GPT's persistent brain

Phase B.6: Debate Protocol (hard + nightmare only)

Skip entirely if `REVIEWER_DIFFICULTY = medium`.

After parsing the review, Claude (the author) gets a chance to rebut:

Step 1 — Claude's Rebuttal:

For each weakness the reviewer identified, Claude writes a structured response:

### Rebuttal to Weakness #1: [title]
-**Accept / Partially Accept / Reject**
-**Argument**: [why this criticism is invalid, already addressed, or based on a misunderstanding]
-**Evidence**: [point to specific code, results, or prior round fixes]

Rules for Claude's rebuttal:

• Must be honest — do NOT fabricate evidence or misrepresent results
• Can point out factual errors in the review (reviewer misread code, wrong metric, etc.)
• Can argue a weakness is out of scope or would require unreasonable effort
• Maximum 3 rebuttals per round (pick the most impactful to contest)

Step 2 — GPT Rules on Rebuttal:

Send Claude's rebuttal back to GPT for a ruling:

Hard mode (MCP):

mcp__codex__codex-reply:
  threadId: [saved]
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    The author rebuts your review:
 
    [paste Claude's rebuttal]
 
    For each rebuttal, rule:
    - SUSTAINED (author's argument is valid, withdraw this weakness)
    - OVERRULED (your original criticism stands, explain why)
    - PARTIALLY SUSTAINED (revise the weakness to a narrower scope)
 
    Then update your score if any weaknesses were withdrawn.

Nightmare mode (codex exec):

codex exec "$(cat <<'PROMPT'
You are the same adversarial reviewer. The author rebuts your review:
 
[paste Claude's rebuttal]
 
VERIFY the author's evidence claims yourself — read the files they reference.
Do NOT take their word for it.
 
For each rebuttal, rule:
- SUSTAINED (verified and valid)
- OVERRULED (evidence doesn't check out or argument is weak)
- PARTIALLY SUSTAINED (partially valid, narrow the weakness)