currytang/amadeus/auto-review-loop

Auto Review Loop: Autonomous Research Improvement

Context: $ARGUMENTS

Step 0: Environment Check

Parse $ARGUMENTS for --remote flag:

• If --remote is present → dispatch to remote server (see below), then STOP
• Otherwise → run locally, proceed to Mode Detection

Remote dispatch (only when --remote is specified)

ARIS_CONFIG=".aris/project.json"
if [ -f "$ARIS_CONFIG" ]; then
  PROJECT_ID=$(python3 -c "import json;print(json.load(open('$ARIS_CONFIG'))['projectId'])")
  API_URL=$(python3 -c "import json;print(json.load(open('$ARIS_CONFIG'))['apiUrl'])")
  API_TOKEN=$(python3 -c "import json;print(json.load(open('$HOME/.claude/aris-api.json'))['token'])" 2>/dev/null)
  # Strip --remote from arguments before forwarding
  CLEAN_ARGS=$(echo "$ARGUMENTS" | sed 's/--remote//')
  PROMPT=$(printf '%s' "$CLEAN_ARGS" | python3 -c "import sys,json;print(json.dumps(sys.stdin.read()))")
  if [ -n "$PROJECT_ID" ] && [ -n "$API_TOKEN" ]; then
    RESULT=$(curl -s -X POST "$API_URL/api/aris/runs" \
      -H "Authorization: Bearer $API_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{\"projectId\":\"$PROJECT_ID\",\"workflowType\":\"auto_review_loop\",\"prompt\":$PROMPT,\"title\":\"Auto Review Loop\"}")
    RUN_ID=$(echo "$RESULT" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d.get('run',{}).get('id','') or d.get('error','UNKNOWN'))" 2>/dev/null)
    echo "Dispatched to remote server. ARIS run ID: $RUN_ID"
  fi
fi

After dispatching, STOP. Tell the user the run ID and that they can monitor on the ARIS dashboard.

Mode Detection

Parse $ARGUMENTS for --plan <path>:

• If --plan is present → Plan-Driven Mode (Section A)
• Otherwise → Free-Form Review Mode (Section B)

Section A: Plan-Driven Mode

Execute an implementation plan with dependency-aware task ordering. Each TODO item is implemented, then verified by the Codex MCP reviewer before marking complete.

A.1 Initialization

1. Read the plan file specified by --plan <path>. This is a markdown file with Steps and TODO items.
2. Parse plan structure:

• Steps are top-level groups (### Step N: Title)
• TODO items within steps (#### TODO-N.M: Title)
• Items marked with ✅ are already completed — skip them

1. Load or create `PLAN_STATE.json` in project root:

``json { "planFile": "docs/implementation_plan.md", "status": "in_progress", "currentNode": "TODO-1.1", "completedNodes": ["TODO-1.0"], "failedNodes": [], "threadId": "019cd392-...", "timestamp": "2026-03-17T10:00:00" } ``

• If exists with "in_progress" and within 24h → resume from currentNode
• Otherwise → fresh start

1. Register plan with ARIS API (if ~/.claude/aris-api.json exists):

``bash ARIS_CFG="$HOME/.claude/aris-api.json" if [ -f "$ARIS_CFG" ]; then API_URL=$(python3 -c "import json;print(json.load(open('$ARIS_CFG'))['api_url'])") API_TOKEN=$(python3 -c "import json;print(json.load(open('$ARIS_CFG'))['token'])") # Register run RESULT=$(curl -s -X POST "$API_URL/api/aris/runs/register" \ -H "Authorization: Bearer $API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"projectId":"PROJECT_ID","prompt":"Plan: PLAN_FILE","status":"running","workflowType":"auto_review_loop","title":"Plan Execution"}') ARIS_RUN_ID=$(echo "$RESULT" | python3 -c "import sys,json;print(json.load(sys.stdin).get('run',{}).get('id',''))") # Upload plan PLAN_MD=$(cat PLAN_FILE) curl -s -X POST "$API_URL/api/aris/runs/$ARIS_RUN_ID/plan" \ -H "Authorization: Bearer $API_TOKEN" \ -H "Content-Type: application/json" \ -d "{\"markdown\":$(python3 -c "import json,sys;print(json.dumps(sys.argv[1]))" "$PLAN_MD")}" fi ``

1. Build execution order: Topological sort of the TODO DAG, respecting dependsOn relationships.

A.2 Execution Loop

For each TODO item in topological order:

Phase 1: Check Prerequisites

• Verify all items in dependsOn are in completedNodes
• If any dependency failed → skip this item, mark as skipped
• If dependencies not yet complete → this shouldn't happen in topological order, but wait/skip if it does
• If item is already in completedNodes (from resume or ✅ marker) → skip

Phase 2: Implement

Read the TODO item's description from the plan file. It contains:

• What needs to be built/implemented
• File locations
• Expected behavior
• Any code snippets or references

Implement the TODO item. This may involve:

• Writing new code files
• Modifying existing code
• Running commands (pip install, tests, etc.)
• Creating data adapters
• Running experiments — check GPU availability first (see "Multi-Server Experiment Routing" section) and dispatch to the server with free GPUs

Phase 3: Verify via Codex MCP

After implementing, send the work to the Codex MCP reviewer for verification:

mcp__codex__codex (or mcp__codex__codex-reply if threadId exists):
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    I am executing an implementation plan. I just completed this TODO item:
 
    **{TODO_KEY}: {TODO_TITLE}**
 
    Description from plan:
    {TODO_DESCRIPTION}
 
    What I implemented:
    {SUMMARY_OF_CHANGES}
 
    Files changed:
    {LIST_OF_FILES}
 
    Test results (if any):
    {TEST_OUTPUT}
 
    Please verify:
    1. Does the implementation match what the plan asked for? (Yes/No)
    2. Are there any bugs, missing pieces, or quality issues? (List them)
    3. Is this TODO item COMPLETE? (Yes/No/Partial)
    4. If Partial, what specific remaining work is needed?
 
    Be strict. Only mark complete if the implementation fully satisfies the plan.

Phase 4: Process Verification Result

Parse the Codex response:

• "Complete: Yes" → Mark as completed, add to completedNodes, update ARIS API
• "Complete: Partial" → Implement the remaining work, then re-verify (max 2 retries)
• "Complete: No" → Fix issues, then re-verify (max 2 retries)
• After 2 failed retries → mark as failed, add to failedNodes, continue to next item

Phase 5: Update State

After each TODO item:

1. Update `PLAN_STATE.json`:

``json { "currentNode": "NEXT_TODO_KEY", "completedNodes": ["TODO-1.0", "TODO-1.1", ...], "timestamp": "NOW" } ``

1. Update ARIS API plan node (if registered):

``bash curl -s -X PATCH "$API_URL/api/aris/runs/$ARIS_RUN_ID/plan/TODO_KEY" \ -H "Authorization: Bearer $API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"status":"completed","resultSummary":"CODEX_VERDICT_SUMMARY"}' ``

1. Write per-TODO review report to `review/` folder:

Create review/TODO-X.Y.md (where X.Y is the TODO key, e.g. review/TODO-1.1.md):

```markdown # Review: TODO-X.Y — {TODO_TITLE}

Status: ✅ completed / ❌ failed / ⚠️ needs_redo Timestamp: {ISO_TIMESTAMP} Retries: {0 / 1 / 2}

## Plan Description

{TODO_DESCRIPTION verbatim from plan file}

## What Was Implemented

{SUMMARY_OF_CHANGES — what you actually built/modified}

Files changed:

• path/to/file1.py — description of change
• path/to/file2.py — description of change

Test / command output: `` {TEST_OUTPUT or "N/A"} ``

## Codex Review Response

{FULL raw Codex response verbatim — do not summarize}

## Verdict

• Match plan?: Yes / No
• Complete?: Yes / Partial / No
• Issues found: {list or "None"}
• Remaining work (if Partial): {list or "N/A"}

```

Rules for this file:

• Always write this file immediately after receiving the Codex verdict, even on failure.
• If the item was retried, overwrite the same file each time (latest verdict wins).
• If the item was skipped due to dependency failure, write a minimal report:

```markdown # Review: TODO-X.Y — {TODO_TITLE}

Status: ⏭️ skipped Reason: Dependency {DEP_KEY} failed — skipping downstream item. ```

• This folder is the primary human-readable audit trail. Keep reports factual and complete.

1. Append to `AUTO_REVIEW.md`:

```markdown ## TODO-X.Y: Title (timestamp)

• Status: completed/failed
• Reviewer verdict: [Codex response summary]
• Files changed: [list]
• Report: review/TODO-X.Y.md
• Notes: [any observations]

```

Phase 6: Parallel Execution

When multiple TODO items have canParallel: true and share the same dependencies (all satisfied):

• List all ready parallel items
• Implement them sequentially (Claude can only do one at a time)
• But batch-verify: send all implementations to Codex in a single review prompt
• This is more efficient than individual reviews for independent items

A.3 Termination

When all TODO items are processed:

1. Set PLAN_STATE.json → "status": "completed"
2. Write final summary to AUTO_REVIEW.md:

```markdown ## Plan Execution Summary

Completed: X/Y items Failed: Z items (listed above with reasons)

Individual review reports: `review/` ```

1. Update ARIS API run status → completed or failed

1. Upload review reports to ARIS API so client device can retrieve them:

``bash ARIS_CFG="$HOME/.claude/aris-api.json" if [ -f "$ARIS_CFG" ] && [ -n "$ARIS_RUN_ID" ] && [ -d "review" ]; then python3 -c " import os, sys, json, base64, urllib.request cfg = json.load(open(os.path.expanduser('~/.claude/aris-api.json'))) api_url, token, run_id = cfg['api_url'], cfg['token'], sys.argv[1] files = {} for f in sorted(os.listdir('review')): if f.endswith('.md'): with open(os.path.join('review', f), 'rb') as fp: files[f] = base64.b64encode(fp.read()).decode() if not files: sys.exit(0) payload = json.dumps(files).encode() req = urllib.request.Request( api_url + '/api/aris/runs/' + run_id + '/review-reports', data=payload, headers={'Content-Type': 'application/json', 'Authorization': 'Bearer ' + token}, method='POST' ) urllib.request.urlopen(req, timeout=30) print('[ARIS] Uploaded ' + str(len(files)) + ' review report(s)') " "$ARIS_RUN_ID" 2>/dev/null || true fi ``

1. Feishu notification (if configured): Send pipeline_done with completion stats

Section B: Free-Form Review Mode

(Original behavior when no --plan is specified)

Autonomously iterate: review → implement fixes → re-review, until the external reviewer gives a positive assessment or MAX_ROUNDS is reached.

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10, or verdict contains "accept", "sufficient", "ready for submission"
• REVIEW_DOC: AUTO_REVIEW.md in project root (cumulative log)
• REVIEWER_MODEL = gpt-5.4 — Model used via Codex MCP
• HUMAN_CHECKPOINT = false — When true, pause after each round's review and present the score + weaknesses to the user.

Override: /auto-review-loop "topic" — human checkpoint: true

State Persistence (Compact Recovery)

Persist state to REVIEW_STATE.json after each round:

{
  "round": 2,
  "threadId": "019cd392-...",
  "status": "in_progress",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "pending_experiments": ["screen_name_1"],
  "timestamp": "2026-03-13T21:00:00"
}

Workflow

Initialization