yjh051108/mmkit/auto-review-loop

Auto Review Loop: Autonomous Research Improvement

Autonomously iterate: review → implement fixes → re-review, until the external reviewer gives a positive assessment or MAX_ROUNDS is reached.

Context: $ARGUMENTS

⛔ Output Mode (read first, before anything else)

The orchestrator may inject one of two notice blocks into CLAUDE.md:

Behavior rules:

1. If AUTO_REVIEW_DOCX_MODE is in CLAUDE.md:

• Skip every xelatex / pdflatex / bibtex invocation in this skill (Phase D2, Termination step 3).
• Phase C must produce/edit paper/draft_v<round>.md (markdown only) — never write .tex files.
• Termination step 3 reduces to: cp paper/draft_vN.md NARRATIVE_REPORT.md.

1. If AUTO_REVIEW_PDF_MODE is in CLAUDE.md:

• Run LaTeX compile only if paper/main.tex already exists.
• Otherwise behave like markdown mode and warn the user in NARRATIVE_REPORT.md that LaTeX source was missing.

1. Default (markdown mode): the existing flow applies, but treat paper/main.pdf compile blocks as best-effort, fail-soft — do not abort the whole loop just because there is no .tex source.

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10, or verdict contains "accept", "sufficient", "ready for submission"
• REVIEW_DOC: AUTO_REVIEW.md in project root (cumulative log)
• REVIEWER_MODEL: the configured reviewer model, invoked via reviewer_client.py script
• HUMAN_CHECKPOINT = false — When true, pause after each round's review (Phase B) and present the score + weaknesses to the user. Wait for user input before proceeding to Phase C. The user can: approve the suggested fixes, provide custom modification instructions, skip specific fixes, or stop the loop early. When false (default), the loop runs fully autonomously.

💡 Override: /auto-review-loop "topic" — human checkpoint: true

Reviewer Provider Configuration

The external reviewer is invoked via reviewer_client.py (path injected through the REVIEWER_SCRIPT environment variable). The reviewer model is configured with three environment variables: OPENAI_API_KEY, OPENAI_BASE_URL, REVIEWER_MODEL_ID.

The skill works with any OpenAI-compatible API — no code change needed, just point the three variables at your provider:

💡 Why vary the reviewer model? A different provider gives an independent review perspective — cross-model review improves objectivity. For example, using MiniMax as reviewer means configuring Base URL https://api.minimax.chat/v1 and Model ID MiniMax-M2.5. The loop logic is identical regardless of provider.

State Persistence (Compact Recovery)

Long-running loops may hit the context window limit, triggering automatic compaction. To survive this, persist state to REVIEW_STATE.json after each round:

{
  "round": 2,
  "status": "in_progress",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "pending_experiments": ["screen_name_1"],
  "timestamp": "2026-03-13T21:00:00"
}

Write this file at the end of every Phase E (after documenting the round). Overwrite each time — only the latest state matters.

On completion (positive assessment or max rounds), set "status": "completed" so future invocations don't accidentally resume a finished loop.

Workflow

Initialization

1. Check for `REVIEW_STATE.json` in project root:

• If it does not exist: fresh start (normal case, identical to behavior before this feature existed)
• If it exists AND status is "completed": fresh start (previous loop finished normally)
• If it exists AND status is "in_progress" AND timestamp is older than 24 hours: fresh start (stale state from a killed/abandoned run — delete the file and start over)
• If it exists AND status is "in_progress" AND timestamp is within 24 hours: resume
• Read the state file to recover round, last_score, pending_experiments
• Read AUTO_REVIEW.md to restore full context of prior rounds
• If pending_experiments is non-empty, check if they have completed (e.g., check screen sessions)
• Resume from the next round (round = saved round + 1)
• Log: "Recovered from context compaction. Resuming at Round N."

1. Read project narrative documents, memory files, and any prior review documents
2. Read recent experiment results — explicitly check these files in order:

• experiment_results.md (from experiment-bridge, contains structured results summary)
• figures/experiment_data.json (consolidated raw experiment data)
• RESULTS.md (from comp-code, if competition workflow)
• Output directories: results/, outputs/, logs/

1. Identify current weaknesses and open TODOs from prior reviews
2. Initialize round counter = 1 (unless recovered from state file)
3. Create/update AUTO_REVIEW.md with header and timestamp

Loop (repeat up to MAX_ROUNDS)

Phase A: Review

Send comprehensive context to the external reviewer via reviewer_client.py.

If the reviewer script fails (API key not configured): perform the review yourself using your own critical analysis capabilities. Act as a senior ML reviewer (NeurIPS/ICML level) and score the work honestly. The loop can still function without external review, though cross-model review is preferred for objectivity.

When the reviewer script is available, use:

cat << 'REVIEW_EOF' > _review_prompt.txt
[Round N/MAX_ROUNDS of autonomous review loop]
 
[Full research context: claims, methods, results, known weaknesses]
[Changes since last round, if any]
 
Please act as a senior ML reviewer (NeurIPS/ICML level).
 
1. Score this work 1-10 for a top venue
2. List remaining critical weaknesses (ranked by severity)
3. For each weakness, specify the MINIMUM fix (experiment, analysis, or reframing)
4. State clearly: is this READY for submission? Yes/No/Almost
 
Be brutally honest. If the work is ready, say so clearly.
REVIEW_EOF
PYTHON=$(command -v python3 2>/dev/null || command -v python 2>/dev/null)
$PYTHON "$REVIEWER_SCRIPT" --prompt-file _review_prompt.txt --thread-file _reviewer_thread.json

If this is round 2+, use the same _reviewer_thread.json to maintain conversation context (对话历史通过 _reviewer_thread.json 自动保存).

Phase B: Parse Assessment

CRITICAL: Save the FULL raw response from the external reviewer verbatim (store in a variable for Phase E). Do NOT discard or summarize — the raw text is the primary record.

Then extract structured fields:

• Score (numeric 1-10)
• Verdict ("ready" / "almost" / "not ready")
• Action items (ranked list of fixes)

STOP CONDITION: If score >= 6 AND verdict contains "ready" or "almost" → stop loop, document final state.

Human Checkpoint (if enabled)

Skip this step entirely if `HUMAN_CHECKPOINT = false`.

When HUMAN_CHECKPOINT = true, present the review results. In interactive mode, wait for user input. In non-interactive mode (claude -p), treat as "go" and proceed with all suggested fixes automatically. Log: "HUMAN_CHECKPOINT: non-interactive mode, auto-proceeding with all fixes."

📋 Round N/MAX_ROUNDS review complete.
 
Score: X/10 — [verdict]
Top weaknesses:
1. [weakness 1]
2. [weakness 2]
3. [weakness 3]
 
Suggested fixes:
1. [fix 1]
2. [fix 2]
3. [fix 3]
 
Options:
- Reply "go" or "continue" → implement all suggested fixes
- Reply with custom instructions → implement your modifications instead
- Reply "skip 2" → skip fix #2, implement the rest
- Reply "stop" → end the loop, document current state

Wait for the user's response. Parse their input:

• Approval ("go", "continue", "ok", "proceed"): proceed to Phase C with all suggested fixes
• Custom instructions (any other text): treat as additional/replacement guidance for Phase C. Merge with reviewer suggestions where appropriate
• Skip specific fixes ("skip 1,3"): remove those fixes from the action list
• Stop ("stop", "enough", "done"): terminate the loop, jump to Termination

Feishu Notification (if configured)

After parsing the score, check if ~/.claude/feishu.json exists and mode is not "off":

• Send a review_scored notification: "Round N: X/10 — [verdict]" with top 3 weaknesses
• If HUMAN_CHECKPOINT=true and verdict is "almost": send as checkpoint, wait for user reply on whether to continue or stop. In non-interactive mode (claude -p), auto-continue.
• If config absent or mode off: skip entirely (no-op)

Phase C: Implement Fixes (if not stopping)

For each action item (highest priority first):

1. Code changes: Write/modify experiment scripts, model code, analysis scripts
2. Run experiments: Deploy to GPU server via SSH + screen/tmux
3. Analysis: Run evaluation, collect results, update figures/tables
4. Documentation: Update project notes and review document

Prioritization rules:

• Skip fixes requiring excessive compute (flag for manual follow-up)
• Skip fixes requiring external data/models not available
• Prefer reframing/analysis over new experiments when both address the concern
• Always implement metric additions (cheap, high impact)

⛔ 新增 / 修改图表时的图表规范铁律

如果 Phase C 的某个 action item 要新增、修改、补全图表（如 reviewer 反馈"缺消融实验图"/"散点图配色不可读"），先读图表规范再写代码，否则新图风格会跟前面步骤已生成的图不一致：

Read _utils/figure_style_guide.md            # 配色方案 + 选图决策表 + 反模式
Read _utils/figure_recipes_<type>.md         # 按论文类型选: academic / competition / empirical / advanced / basic
                                             # _utils/get_recipe.py <type> <id> 提取完整配方代码

写图代码必须做到的：

• 顶部统一初始化：from _utils.plot_utils import setup_style, save_fig, PALETTE; setup_style()
• 不要硬编码颜色, 用 PALETTE[0] / PALETTE[1] 等; 不要 plt.title()
• 文件名沿用前面 paper-figure 步骤的命名规范 (figures/fig_*.png + figures/fig_*.pdf)
• 跟既有图保持同一套配色方案（看 figures/fig_*.png 已有的颜色, 或读 figure_style_guide.md 选定方案后保持一致）
• 写完跑一遍 bash _utils/figure_check.sh 自检 (anti-pattern 扫描)

如果 reviewer 只让"加一张图", 优先看现有 figures/fig_*.png 的配色取色, 别自己随便挑——风格漂移会让审稿人感觉"补丁感强"。

Phase D: Wait for Results

If experiments were launched:

• Monitor remote sessions for completion
• Collect results from output files and logs

Phase D2: Recompile + Quality Check

⛔ MANDATORY after every round of fixes — but the action depends on the output mode set in the preamble at the top of this skill: