escapeboy/claude-code-kit/self-improve

self-improve — closing the loop on a convention library

A review comment that recurs is not a comment — it is an undocumented requirement. Once the same guidance appears often enough it belongs in the system, not in a human review. This skill turns that insight into a converging loop: mine the recurring signal → fold it into the generator/conventions → prove the change with a tiered gate. As the library improves it produces fewer repeat comments → less signal → smaller changes → steady state (natural damping, not infinite mutation). When conventions change, the surge of new comments restarts the loop exactly where needed.

Adapted from Salesforce Engineering, "Closing the Loop: How to Build Self-Improving AI Systems with Automated Feedback Loops" (2026-07-17, forcedotcom/sf-skills).

When to Use This Skill (and When NOT to)

Start simple. Do not run the full loop for a single edit. The loop earns its cost only when the same signal repeats — that repetition is the whole trigger.

The loop

mine signal → apply (bounded) → tiered gate → promote rule → measure → converge

1. Mine the signal. Collect recurring corrections (from feedback memories, PR review

threads, repeated session corrections). Apply a frequency threshold: a pattern seen ~3+ times is a requirement, not a one-off. See references/rubric.md.

1. Apply, bounded. Fold the rule into the generator surface (skill body, a global

CLAUDE.md convention, a template). Blast-radius caps: ≤5 improvements and ≤100 changed lines per cycle. Larger → split into cycles. Any regression in the gate aborts.

1. Tiered gate (below). Tiers 1–3 always; Tier 4 when the skill's procedure changed. All that run must pass.
2. Promote the rule to durable memory (a decision-memory store; see integration seams).
3. Deliver transparently. Consequential steps (auto-edit, push PR) pass a governance

gate and produce a draft PR for human review, never an auto-merge.

1. Measure convergence. Track the decline in repeat-signal frequency across cycles (the

direct efficacy metric). When signal dries up, stop; it restarts on the next convention change.

Tiered evaluation

Deterministic checks where quality is objectively measurable; LLM judgement only where it needs context. Neither replaces the other — they answer different questions.

• Tier 1 — Trigger accuracy (semantic, subagent). Does the skill fire when it should and

stay silent when it should not, in the real crowded skill environment? 10 should-trigger + 10 should-not queries via an independent subagent. Protocol → references/rubric.md.

• Tier 2 — Structural validation (deterministic, script). The mechanical CI checks:

``bash python3 ~/.claude/skills/self-improve/scripts/skill-lint.py <skills-dir> [--json] [--strict] ` Validates frontmatter, kebab-case name matching the dir, description bounds, unreferenced bundled files (dead weight), duplicate names, and trigger-collision (near-duplicate descriptions). --json` emits an evidence record. A non-zero exit is a hard gate — treat like a failing CI check that blocks the PR. Ship it as a pre-commit hook.

• Tier 3 — Rubric LLM-as-judge (semantic, subagent). Grade the artifact on the dimensions

in references/rubric.md (scope precision, progressive disclosure, boundary clarity, convention adherence, signal fidelity). For a scripted, CI-runnable judge with a swappable model, run scripts/deepeval_tier3.py (DeepEval G-Eval, one metric per dimension; TIER3_JUDGE=claude-cli judges with your Claude Code login, no API key): setup and the judge-quality caveat in references/deepeval-setup.md.

• Tier 4 — Behavioral eval (runtime, claude plugin eval). Tiers 1–3 read the text; this

checks that an agent WITH the skill does the task better than one WITHOUT it (built-in with/without ablation on a clean baseline). Run it for a new skill or a procedure change, not for wording edits. Cases live in <skill>/evals/; gate with: ``bash python3 ~/.claude/skills/self-improve/scripts/behavior-stats.py <plugin-eval.json> [--json] ` Adds normalized gain, pass^k for critical cases, and flags cases both arms pass (they prove nothing). Fails on any regression (NG < 0). Mean NG ≈ 0 = dead weight → removal candidate for a human. Case authoring, grader validation, cost: references/behavior-eval.md`.

Integration seams (optional)

The loop composes with whatever platforms you run — it does not require them. See references/integration-seams.md:

• Deterministic-execution / evidence platform — run the Tier-2 gate as a sealed node; the

--json record + input digest make each cycle replayable and auditable.

• Decision-memory store (e.g. Svod) — promoted

rules and cycle decisions land here as durable policy, provisional→active.

• Governance gate (e.g. Lattice) — consequential

steps (auto-edit, push PR) route through an approval gate; read-only eval steps pass freely.

• Code-intelligence engine (e.g. codebase-memory-mcp, see `codebase-memory`) — blast-radius ∩ test-coverage of a

code edit: if it touches untested symbols, escalate to a human.

• MCP aggregator (e.g. Harbormaster) — surfaces

all of the above as one MCP surface.

Boundaries

Always

• Enforce the frequency threshold — do not promote a one-off correction into a rule.
• Run Tiers 1–3 (plus Tier 4 when the procedure changed); a red Tier 2 or Tier 4 aborts the cycle. Keep edits within the ≤5 / ≤100 caps.
• Promote durable rules to a decision-memory store; keep ephemeral facts out of it.

Ask first

• Pushing a PR or editing a shared/team skill library (consequential → governance gate + human).
• Widening the blast-radius caps for a single cycle.
• Removing a skill Tier 4 flagged as dead weight, or a Tier-4 run expected to cost over ~$2.

Never

• Auto-merge a self-generated change without human review (stop at draft PR — the blocker is the

reviewer, by design).

• Let the loop mutate on thin signal — below threshold, do nothing and let it converge.
• Duplicate an existing rule/memory instead of superseding it.