kbanc85/claudia/auto-research

Auto-Research

A hill-climber for any local artifact with a rubric. Inspired by Karpathy's autoresearch pattern, adapted to Claudia's safety principles.

Mental model

Three primitives, one governance file:

1. The artifact — what gets iterated on (a draft email, a brief, a wiki page, a one-pager). Lives in the workspace, never touches the user's original file during the loop.
2. The evaluator — a scalar rubric. Plain English. An independent Checker (the loop-checker agent, Haiku) scores each iteration against it, not Claudia herself. The rubric must produce a number for the original artifact before the loop starts.
3. The budget — iteration count (default 20). Loop stops when budget is exhausted or score plateaus.
4. The program (governance file) — what the user wants, what's off-limits, what counts as "better." Claudia helps the user write this if they don't volunteer it.

Loop: read program + artifact + results history, Claudia (the Maker) proposes one specific change, implements it, then dispatches the loop-checker (the Checker) to score it independently. If the Checker's score beats the running best, ratchet (commit); if worse, revert (git reset). Write the status file. Report each iteration as a one-line update. Repeat until budget is hit.

When to invoke this skill

User explicitly asks:

• "Iterate on this until it's [adjective]"
• "Loop on this draft"
• "Run experiments on this"
• "Auto-research this"
• "Hill-climb this against [criterion]"
• "Make this better, don't stop until it's good"

User implicitly invokes when:

• They've shared an artifact and a quality bar in the same message
• They've tried 2-3 manual revisions of the same draft in the same session and are still unsatisfied

When NOT to invoke

Refuse to start the loop if:

1. External-action artifact. The artifact is, or directly drives, an external action (an email about to be sent, a Slack message in the compose window, a calendar invite). Auto-research on these is too easy to misuse. Hand-iterate with the user instead.
2. No clear evaluator. The user can't articulate what "better" means even after one pass of helping. Without an evaluator, the loop hill-climbs the wrong hill.
3. Sensitive content. Medical, deeply personal, legal-defensible artifacts. Iteration risks introducing fabricated detail that looks plausible. Decline and explain.
4. The artifact is already good. If you score the baseline and it's above the user's stated threshold, tell them and offer one quick polish instead of N iterations.
5. Bold structural change is the actual need. Karpathy's own limitation: RLHF-trained iteration is "cagy and scared." Iterations tend toward safe edits, not bold reframings. If the user wants a fundamentally different angle, iteration will not get them there. Suggest a fresh draft instead.

Workspace layout

Each run gets its own workspace at ~/.claudia/auto-research/<task-id>/:

~/.claudia/auto-research/<task-id>/
├── program.md           the brief (user-authored with Claudia's help)
├── artifact.md          (or .txt, the working copy that gets edited)
├── original.md          immutable copy of the input, for reference and diff
├── results.tsv          one row per iteration: timestamp, score, kept/reverted/contested, change-summary
├── research_status.md   loop control plane (see docs/loop-status-schema.md): iteration, verified, score, checker_verdict, next_action
├── best.md              symlink (or copy on Windows) to the highest-scoring version
└── iterations/
    ├── 01/artifact.md
    ├── 02/artifact.md
    └── ...

The task-id is the slugified user phrase plus a short timestamp: iterate-board-update-20260515-1430.

Critical: The loop edits artifact.md inside the workspace. The user's original file (wherever it lives in their file system) is NEVER modified during the loop. At the end, Claudia asks the user where to put best.md; the user decides.

program.md template

# Program for: <one-line description of the task>
 
## Goal
 
(1-3 sentences. What is the end state Claudia is iterating toward?)
 
## Evaluator (rubric)
 
Each iteration is scored 0-10 on each dimension. Total score = sum. Higher is better.
 
| Dimension | What scores high | What scores low |
|-----------|------------------|-----------------|
| (dimension 1) | (what makes a 10) | (what makes a 0) |
| (dimension 2) | ... | ... |
| (dimension 3) | ... | ... |
 
## Hard constraints (do NOT violate)
 
-Length cap: stay under N words.
-Must contain: specific phrase or fact.
-Must NOT contain: forbidden phrasings, names, claims.
-Tone: must match the user's prior emails to <recipient> (paste examples in references/).
-(etc.)
 
## Budget
 
-Max iterations: 20 (default)
-OR stop when score plateaus (no improvement for 5 iterations in a row)
 
## Out of scope
 
-(anything Claudia might be tempted to do that isn't the goal)

Claudia's first action when invoked: read the artifact, draft a program.md based on what the user said, present it for confirmation, then start the loop.

The loop (Claudia's internal workflow)

For each iteration N:

1. Read the state. Read program.md, artifact.md, last 3 rows of results.tsv.
2. Propose one change. One specific edit, justified in one sentence. Not a rewrite. Not a refactor. One change.
3. Implement. Apply the edit to artifact.md. Save.
4. Score (independent Checker). Dispatch the loop-checker agent (Haiku) with the rubric from program.md, the new artifact.md, and the one change you just made plus your expected effect (your Maker self-score). The Checker returns a verdict JSON: verified, score, max_score, issues, rationale, hard_constraint_violated. You do NOT score it yourself; the Checker's score governs the decision. (See skills/_loop/checker.md.)
5. Decide. If the Checker's score > best score so far and hard_constraint_violated is false: ratchet. Copy artifact.md to iterations/NN/, update best.md to point at the new winner, append a row to results.tsv marked kept. Otherwise: revert. Copy iterations/(best_iter)/artifact.md back to artifact.md, append a row marked reverted.
6. Flag divergence. Compare the Checker's score against your Maker self-score. If they diverge by more than 1.5 points on a 10-point scale, mark the iteration contested in results.tsv. The Checker's verdict still governs; the flag just surfaces the disagreement in the end-of-run summary.
7. Write the status file. Atomically update research_status.md with iteration, verified, score, budget_remaining, last_input, maker_proposal, checker_verdict (the Checker's rationale), and next_action. Write to a temp sibling and rename; never edit it in place. (See docs/loop-status-schema.md.)
8. Report. One line to the user: iter N: score 7.4 (kept), change: tightened lede to one sentence. Append [contested] when flagged.
9. Check stop conditions and self-repair. If the Checker has returned verified: false 3 times in a row, or its score has regressed for 3 straight iterations, enter the self-repair sub-loop (see skills/_loop/repair.md) before giving up: it diagnoses whether the rubric or a brief is the real problem, proposes one fix, and proves it on the exact failing input. If budget exhausted: stop. If score plateaued (no improvement for 5 iterations): stop. Otherwise: next iteration.

The independent Checker (Maker-Checker)

This loop separates the producer from the verifier (Proposal 11, E2). Claudia is the Maker: she reads the state and proposes one change. The Checker is a separate agent (loop-checker, Haiku) that scores the result independently. The two never collapse into one model grading its own work, which is the bias the split exists to remove.

• The Checker's score, not Claudia's, drives keep/revert. Claudia's expected

effect is only a self-score, used to detect disagreement.

• The Checker is adversarial by brief: it hunts for faults and defaults to

verified: false when uncertain. A passing verdict therefore means something.

• A hard_constraint_violated: true verdict forces a revert regardless of score.

Contested iterations. When the Checker's score and Claudia's self-score diverge by more than 1.5 points (10-point scale), the iteration is marked contested. The Checker still governs the decision; the flag makes the disagreement visible. The end-of-run summary reports how many iterations were contested, which is a useful signal that the rubric is ambiguous or that Claudia was over-optimistic.

Cost. The Checker runs on Haiku, so the verification pass is cheap and fast. With the default 20-iteration budget, that is at most 21 Checker dispatches (including the baseline), each scoped to a single small artifact.

Self-repair. If the loop stalls (the Checker fails 3 times running, or scores regress), auto-research enters a bounded self-repair sub-loop that treats the harness, not the artifact, as the suspect: it diagnoses the rubric or a brief, proposes one fix, and proves it on the exact failing input before adopting it. See skills/_loop/repair.md. It is capped at 2 repair attempts and never auto-edits shipped briefs (those changes go to a human approval gate).

Worked example (dry run)

A short run of "tighten this board update", to show the loop and the independent Checker in practice. The artifact is artifact.md; the rubric is the board update rubric below. Budget: 20 iterations.

iter 0: score 6.1 (baseline)            checker: solid update, but the lede buries the one number that matters; the ask is split across two sentences
iter 1: score 7.0 (kept)                maker: moved the revenue number into the opening line | checker: lede now leads with the number; ask still split
iter 2: score 7.0 (reverted)            maker: cut the second paragraph | checker: lost the context the board needs for the ask; net neutral
iter 3: score 7.8 (kept)                maker: merged the ask into one sentence | checker: ask is now one clear line; tone matches prior updates
iter 4: score 8.4 (kept) [contested]    maker self-score 9.5 | checker: tighter, but "significantly" is doing unearned work in the headline
iter 5: score 8.4 (reverted)            maker: swapped the closing line | checker: weaker call to action than iter 4; revert
iter 6: plateau                         no gain for the last 3 iterations; remaining changes are cosmetic

The loop stops at iteration 6 on plateau. research_status.md at that point:

---
loop_id: tighten-board-update-20260613
iteration: 6
verified: true
score: 8.4
budget_remaining: 14
last_input: board update draft v4
maker_proposal: swapped the closing line (reverted)
checker_verdict: best version is iter 4; remaining changes are cosmetic
next_action: hand off iter 4 to the user
updated_at: 2026-06-13T15:02:00Z
---
 
# Loop status: tighten-board-update
 
Ratcheted 6.1 to 8.4 over 4 kept iterations. One contested iteration (iter 4):
Claudia self-scored 9.5, the Checker scored 8.4 and flagged an unearned
intensifier. The Checker's score governed. Plateaued at iter 6.

Hand-off to the user: "Started at 6.1, ended at 8.4 over 6 iterations. The three biggest jumps were moving the revenue number into the lede (iter 1), collapsing the ask into one sentence (iter 3), and the tone pass (iter 4). One iteration was contested: I scored iter 4 a 9.5, the Checker held it to 8.4 because the headline leaned on 'significantly' with no number behind it. I kept the Checker's call. Want iter 4 back at the original location?"

That contested iteration is the whole point of the split. Without an independent Checker, iter 4 ships at the inflated 9.5 self-score. With it, the unearned intensifier gets caught and the honest 8.4 governs.

Safety rules (mandatory, follow without exception)