grind-lab-core/night_owl_research_agent/auto-review-loop

Skill: auto-review-loop

You run adversarial review cycles to iteratively improve research work. The architecture enforces generator-evaluator separation: the entity that wrote a section does NOT score it. Each round: independent review → parse → implement fixes → re-evaluate → decide.

Context: $ARGUMENTS

Core Principle: Generator-Evaluator Separation

Never let the same agent that wrote a section score it.

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10, or verdict contains "accept", "sufficient", "ready for submission"
• REVIEW_DOC: output/AUTO_REVIEW_REPORT.md in project root (cumulative log)
• REVIEWER_MODEL = gpt-5.4 — Preferred external reviewer via Codex MCP (OpenAI models: gpt-5.4, o3, gpt-4o, etc.). If Codex MCP / codex exec is unavailable or misconfigured, fall back to REVIEWER_FALLBACK (see Phase A.0).
• REVIEWER_FALLBACK = claude-opus-4-6 — The most powerful Claude model, invoked via the Agent tool in a separate context as the adversarial reviewer subagent. Used automatically when the external LLM cannot be reached.
• RESEARCH_DOMAIN = auto — One of ml, giscience, remote-sensing, spatial-data-science, or auto (infer from the work under review). Drives the reviewer persona, rubric emphasis, and domain-specific must-checks (see Phase A.5).
• HUMAN_CHECKPOINT = false — When true, pause after each round's review (Phase B) and present the score + weaknesses to the user. Wait for user input before proceeding to Phase C. The user can: approve the suggested fixes, provide custom modification instructions, skip specific fixes, or stop the loop early. When false (default), the loop runs fully autonomously.
• COMPACT = false — When true, (1) read output/EXPERIMENT_LOG.md and output/PROJ_NOTES.md instead of parsing full logs on session recovery, (2) append key findings to output/PROJ_NOTES.md after each round.
• REVIEWER_DIFFICULTY = medium — Controls how adversarial the reviewer is. Three levels:
• medium (default): Current behavior — MCP-based review, Claude controls what context GPT sees.
• hard: Adds Reviewer Memory (GPT tracks its own suspicions across rounds) + Debate Protocol (Claude can rebut, GPT rules).
• nightmare: Everything in hard + GPT reads the repo directly via codex exec (Claude cannot filter what GPT sees) + Adversarial Verification (GPT independently checks if code matches claims).

💡 Override: /auto-review-loop "topic" — compact: true, human checkpoint: true, difficulty: hard

State Persistence (Compact Recovery)

Long-running loops may hit the context window limit, triggering automatic compaction, or crash mid-round. To survive both, persist state to output/REVIEW_STATE.json at every phase boundary (not only end-of-round):

{
  "round": 2,
  "phase": "C_implementing_fixes",
  "threadId": "019cd392-...",
  "status": "in_progress",
  "difficulty": "medium",
  "persona": "remote-sensing",
  "scores_per_round": [
    {"round": 1, "score": 5.0, "verdict": "not ready"}
  ],
  "last_score": 5.0,
  "last_verdict": "not ready",
  "open_weaknesses": [
    {"id": "W1", "title": "no spatial CV", "first_seen_round": 1, "status": "open"}
  ],
  "pending_experiments": ["screen_name_1"],
  "round_started_at": "2026-03-13T21:00:00",
  "loop_started_at": "2026-03-13T18:00:00",
  "timestamp": "2026-03-13T21:14:03"
}

Write rules:

• Overwrite output/REVIEW_STATE.json at each phase transition (A → B → B.5 → B.6 → C → D → E). phase values: A_selecting_backend, A5_selecting_persona, A_reviewing, B_parsing, B5_memory_update, B6_debate, C_implementing_fixes, D_waiting_results, E_documenting.
• scores_per_round is an append-only array — never overwrite prior round scores.
• open_weaknesses entries carry first_seen_round so the circuit breaker in Phase B.7 can detect repeats.

On completion (positive assessment or max rounds or blocked), set "status" to "completed", "max_rounds_reached", or "blocked" so future invocations don't resume a finished loop.

Budget Caps (safety valves)

Hard caps that terminate the loop with status: "blocked" and a clear reason written to output/AUTO_REVIEW_REPORT.md:

• MAX_ROUNDS = 4 (as before)
• MAX_WALL_CLOCK_HOURS = 6 — total elapsed from loop_started_at to current time. Prevents runaway overnight loops. Override via argument (max-hours: N).
• MAX_ROUND_WALL_CLOCK_HOURS = 2 — any single round exceeding this triggers a soft warning in the round log and a user notification; the round is not killed automatically (experiments may legitimately be long), but the cap is enforced cumulatively by MAX_WALL_CLOCK_HOURS.
• MAX_STALLED_ROUNDS = 2 — see Phase B.7.

Log round_started_at at the top of Phase A and compute elapsed at Phase E.

Workflow

Initialization

1. Check for `output/REVIEW_STATE.json` in project root:

• If it does not exist: fresh start (normal case, identical to behavior before this feature existed)
• If it exists AND status is "completed": fresh start (previous loop finished normally)
• If it exists AND status is "in_progress" AND timestamp is older than 24 hours: fresh start (stale state from a killed/abandoned run — delete the file and start over)
• If it exists AND status is "in_progress" AND timestamp is within 24 hours: resume
• Read the state file to recover round, threadId, last_score, pending_experiments
• Read output/AUTO_REVIEW_REPORT.md to restore full context of prior rounds
• If pending_experiments is non-empty, check if they have completed (e.g., check screen sessions)
• Resume from the next round (round = saved round + 1)
• Log: "Recovered from context compaction. Resuming at Round N."

1. Read project narrative documents, memory files, and any prior review documents. When `COMPACT = true` and compact files exist: read output/PROJ_NOTES.md + output/EXPERIMENT_LOG.md instead of full output/AUTO_REVIEW_REPORT.md and raw logs — saves context window.
2. Read recent experiment results (check output directories, logs)
3. Identify current weaknesses and open TODOs from prior reviews
4. Initialize round counter = 1 (unless recovered from state file)
5. Create/update output/AUTO_REVIEW_REPORT.md with header and timestamp

Loop (repeat up to MAX_ROUNDS)

Phase A.0: Select Reviewer Backend

Before sending any review prompt, decide which backend will act as the reviewer. Probe in this order and stop at the first one that works:

1. Codex MCP (`mcp__codex__codex`) — preferred. Check that the MCP server is registered (e.g., entry in .mcp.json) and that a cheap ping call returns without an auth/config error.
2. `codex exec` CLI — required for nightmare difficulty. Check with command -v codex and a --version probe. If missing on nightmare, downgrade to `hard` and note this in EXPERIMENT_LOG.md.
3. Reviewer subagent fallback — if neither of the above is usable, fall back to `REVIEWER_FALLBACK` (Claude Opus 4.6) invoked through the Agent tool in a fresh context. This preserves generator-evaluator separation because the subagent starts cold and sees only the review package we hand it — not this session's writer context.

How to invoke the subagent fallback:

Agent({
  description: "Adversarial reviewer — round N",
  subagent_type: "general-purpose",
  model: "opus",
  prompt: "<persona from Phase A.5> + <context package> + <scoring rubric> + <difficulty-specific instructions: memory, debate, verification>"
})

Rules for the fallback:

• The subagent must receive the same instructions the external reviewer would have received (persona, rubric, output format). Only the transport changes.
• For hard and nightmare: include the full contents of memory/REVIEWER_MEMORY.md in the prompt; the subagent returns an updated memory block we then persist.
• For nightmare: grant the subagent read access to the repo (it already has file tools) and explicitly tell it to verify claims against code/results/logs itself, without trusting the author.
• There is no persistent `threadId` across rounds. Simulate memory by re-pasting memory/REVIEWER_MEMORY.md + the last round's raw response on each invocation.
• Record which backend was used (mcp / codex_exec / subagent) in the round entry of output/AUTO_REVIEW_REPORT.md so the audit trail is explicit.

If all three backends fail, stop the loop and write a clear error to output/AUTO_REVIEW_REPORT.md + output/REVIEW_STATE.json (status: "blocked", reason: "no reviewer backend available"). Do NOT silently self-review — that breaks generator-evaluator separation.

Phase A.5: Select Domain Persona and Rubric Emphasis

Pick the reviewer persona based on RESEARCH_DOMAIN (when auto, infer from the work: presence of spatial data, remote-sensing imagery, spatial statistics, GIScience theory, etc.). The persona string is injected into every Phase A prompt, whether MCP, codex exec, or subagent.

Persist the selected persona in REVIEW_STATE.json (persona field) so a resumed loop does not reclassify the domain mid-run and flip personas between rounds. On resume, read persona from state rather than re-inferring.

Mixed GeoAI (deep learning + spatial/RS data): combine the ML persona with the relevant geo persona; require BOTH sets of must-checks scoped to the actual research question. Example preamble: "You are both a senior ML reviewer and a senior remote-sensing reviewer — hold this work to both bars, but only apply geo-specific checks (MAUP, spatial CV, GWR, residual Moran's I, per-scene transferability) where the paper's claims actually depend on them."

When constructing Phase A prompts below, replace the placeholder <PERSONA> with the persona string, and append the domain-specific must-checks as a conditional checklist: the reviewer should evaluate each item AND state "not applicable — <reason>" when the claim does not depend on it, instead of treating omission as an automatic weakness.

Phase A: Review

Route by REVIEWER_DIFFICULTY:

Medium (default) — MCP Review