version: "1.0.1" name: smith-index description: Build and maintain the project manifest under .smith/index/. Full rebuild scans every source file, runs language parsers, and writes per-file .meta, per-system manifests, and a top-level summary. Supports --check (hash-only staleness), --system (partial rebuild), --incremental (git-diff scope), --describe (per-file LLM descriptions via Task sub-agents — subscription billing), --migrate-templates (constitution.md / CLAUDE.md), --init-system-paths, and --resume. argument-hint: "[--check | --system <name> | --incremental | --describe | --migrate-templates | --init-system-paths | --resume] [--from <ref> --to <ref>] [--batch-size <n>] [--per-method-threshold <n>] [--yes] [--skip-model-probe] [--root <path>] [--system-paths <path>]"

Smith Index

Generate the deterministic project manifest at .smith/index/. The manifest replaces soft natural-language guidance with structured, indexed context — every Smith skill (/smith-new, /smith-bugfix, /smith-debug, /smith-explore, …) consults it through the same retrieval path (/smith-navigate + context-loader.sh).

Arguments: $ARGUMENTS

Manifest is a map, not a fence

.smith/index/ is a navigation aid, not a hard boundary. Skills like /smith-explore still grep the whole codebase when initial signals suggest broader impact. A stale or imprecise manifest must never block the calling session — it should degrade gracefully to vault-only context plus a soft warning.

Behavior

This skill is imperative — running it modifies .smith/index/ and (with --migrate-templates) constitution.md / CLAUDE.md. It does NOT modify any source file in the project. All generated state is confined to .smith/index/ plus optional .bak.<timestamp> files on template migration.

The actual work runs in ~/.smith/scripts/smith-index/run.py (called via ~/.smith/scripts/smith-index/run.sh), installed by scripts/install.sh from this repo. In the smith-repo dev tree these same files live at scripts/smith-index/run.py / run.sh — invoking either works. The skill markdown is the entry point that parses $ARGUMENTS, decides the mode, and shells out.

Modes (flags)

/smith-index — full rebuild (default)

1. Walk the project from the current directory.
2. Honor .gitignore (uses git ls-files when available; falls back to a

manual exclusion list of node_modules/, .git/, .venv/, etc.).

1. For each source file (.py, .js, .jsx, .ts, .tsx, .css,

.html, .sh):

• Resolve the parser via parser-lib.sh resolve_parser <ext> (prefers

.smith/scripts/ over ~/.smith/scripts/ over the in-repo fallback).

• Run the parser; capture JSON.
• Compute SHA-256 of the first 4KB of source content (Q6 — hash field

in .meta).

• Render the .meta file at .smith/index/files/<mirrored>/<file>.meta.
• Resolve the file's system via path-resolver.py (longest-prefix

system-paths.json override → heuristic per spec Requirement 14).

1. Per system: rewrite systems/<sys>.md once with all files bucketed in

that system (sorted by lines desc; truncated at 60 entries with …and N more files per data-model.md section 3). Cap ≤80 lines.

1. Rewrite top-level manifest.md (systems table + Stats). Cap ≤50 lines.
2. Write checkpoint state to .smith/index/.smith-index-checkpoint.json

every 25 files; delete on clean exit.

1. Append one JSONL log line per stage per file to

~/.smith/logs/smith-index-<ISO8601>.jsonl per Rule 4.

1. Write the schema-version marker at .smith/index/.schema-version

containing the current schema version (read from ~/.smith/scripts/meta_schema_version.txt, falls back to scripts/parsers/meta_schema_version.txt in the smith-repo dev tree if the global install is missing). This file lets /smith-update detect projects whose manifest was generated against an older .meta schema and offer to regenerate. The marker is overwritten on every full rebuild (and on --incremental runs that write a fresh manifest); silently skipped if neither source file is found.

1. Print a summary line:

/smith-index: N files indexed (N succeeded, N failed, N skipped) in T.Ts.

Performance budget: <60s p95 for a 100-file project (acceptance criterion from spec).

/smith-index --check

Hash-only staleness scan. No rebuild. For each existing .meta, compute SHA-256 of the first 4KB of the corresponding source file and compare against the Hash: line in the .meta. Reports:

• Fresh count (hashes match)
• Stale list (hash mismatch — file edited externally; manifest hook

missed it; git checkout changed content; etc.)

• Missing-source list (.meta exists but source was deleted/renamed)

No mtime comparison — Q6 is hash-only. Estimated ~5-10s for a 400-file project; acceptable for a maintenance command.

/smith-index --system <name>

Partial rebuild restricted to files mapped to one system. Useful after adding a single feature: refreshes that system's .meta files and systems/<name>.md without re-walking the entire tree. Top-level manifest.md Stats section is also updated.

/smith-index --describe

Generate per-file LLM descriptions in the .meta description layer. Unlike all other modes, --describe is orchestrated BY this skill prose itself — not by run.py. The skill drives a discovery → batched-spawn → write loop, spawning one Task sub-agent per file (or per method on dense files) that returns a MetaDescription JSON. Each spawned Task inherits the session's Claude Code auth → subscription billing, not API-key billing.

Single backend. v3 (PR #23) removed the v2 direct-HTTPS path. The 2am scheduler invokes claude --print -p "/smith-queue process ..." which IS a Claude Code session — Task spawning works there too. No --llm-backend flag, no CLAUDE_HEADLESS env var.

Step 0 — Runtime model probe

Before any bulk work, spawn ONE small Task with:

subagent_type: general
model: claude-haiku-4-5
prompt: "Respond with exactly: MODEL_OK"

If the response doesn't arrive cleanly OR the trimmed response text is not exactly MODEL_OK (heuristic: a Haiku model honoring the override responds crisply; an Opus/Sonnet primary will be more verbose), abort with:

ERROR: Could not verify Haiku model override. Running the bulk

loop on the session's primary model would inflate subscription

cost ~30×. Verify your Task tool subagent type supports the model

parameter. Pass --skip-model-probe to override at your own risk.

--skip-model-probe bypasses this check.

Step 1 — Discovery

python3 ~/.smith/scripts/describe_discover.py \
  --root "$ROOT" \
  ${SYSTEM:+--system "$SYSTEM"} \
  --threshold "${THRESHOLD:-5}"

(Use the repo-relative path scripts/parsers/describe_discover.py if ~/.smith/scripts/ does not resolve.)

Parse the JSON output. Each entry has rel_path, source_hash, parser_output, qualifying_method_ids, existing_description, cache_hit, system. Drop entries with cache_hit=true — these are no-ops (their .meta already matches the current source hash).

Step 2 — Resume filter

If --resume was passed:

python3 ~/.smith/scripts/describe_checkpoint.py load-completed \
  --log-dir ~/.smith/logs \
  --state .smith/index/.smith-index-describe-checkpoint.json

Filter the remaining files to exclude completed rel_path values.

Step 3 — Pre-flight estimate + confirmation gate

After filtering, count files needing description (N) and sum their qualifying_method_ids counts (M). Identify files where len(qualifying_method_ids) > 15 (the per-method-split threshold); each such file contributes that many Tasks instead of 1. Let T be the total Task count.

Print to stderr:

/smith-index --describe pre-flight summary
─────────────────────────────────────────────
  Files needing description: N
  Qualifying methods total: M
  Per-method-split threshold: 15 (configurable via --per-method-threshold)
  Per-method-split files: K
  Estimated Tasks to spawn: T
  Estimated wall time: ~W minutes (5s/Task sequential)

Then ask: Proceed? (y/N):. Read one line from stdin. Accept y/yes (case-insensitive). --yes bypasses the gate (required for the scheduler).

Step 4 — Sequential Task spawning loop

Batch the remaining files in groups of 10 (default; override with --batch-size). For each batch, process files sequentially (one at a time — no parallel tool-use block — simpler per-Task error handling, visible progress logging).

For each file in the batch:

1. Per-method-split decision. If

len(qualifying_method_ids) > 15 (default; override with --per-method-threshold), spawn one Task PER METHOD. Otherwise one Task for the whole file.

1. Build the prompt body. Assemble via the helper (single source

of truth for prompt template): ``bash PROMPT=$(python3 ~/.smith/scripts/describe_write.py build-prompt \ --rel-path "$REL" --root "$ROOT" \ --method-ids "<comma-separated-ids>" --module) ``

1. Spawn the Task.

``yaml subagent_type: general model: claude-haiku-4-5 prompt: | <PROMPT body from step 2> ``

1. Retry on failure (exponential backoff). If the Task call

fails or returns malformed JSON or status="error", retry with backoff 5s → 10s → 20s. Max 3 attempts. After 3, log a failed JSONL record and move to the next file. Do NOT abort the run.

1. STUB MODE. If SMITH_TASK_STUB=1 is set in the env, skip

the Task spawn entirely. Pipe the canned fixture into the writer: ``bash python3 ~/.smith/scripts/describe_write.py apply --from-stub \ tests/fixtures/task-stub-responses.json \ --rel-path "$REL" --root "$ROOT" --hash "$HASH" ` The stub fails loud (exit 4) if any qualifying method id is not in the fixture. Tests set SMITH_TASK_STUB=1`; users never do.

1. Apply the result. Pipe the Task's JSON output into the writer:

``bash echo "$TASK_OUTPUT" | \ python3 ~/.smith/scripts/describe_write.py apply \ --rel-path "$REL" --root "$ROOT" --hash "$HASH" ``

1. Append a checkpoint record. One JSONL line per file:

``bash python3 ~/.smith/scripts/describe_checkpoint.py append \ --log "$LOG_PATH" \ --record "$(printf '{"item_id":"%s","stage":"describe", "status":"ok","backend":"task", "method_count":%d,"module_chars":%d, "batch_index":%d,"retry_count":%d}' \ "$REL" "$N" "$M" "$BATCH_IDX" "$RETRIES")" ``

Then persist checkpoint state: ``bash python3 ~/.smith/scripts/describe_checkpoint.py save \ --path .smith/index/.smith-index-describe-checkpoint.json \ --processed "$REL" ``

Step 5 — Propagate descriptions to manifest tables

/smith-index (full rebuild) populates per-file rows in systems/<id>.md and the top-level manifest.md during its source walk — well before --describe writes the description layer into each .meta. The manifest tables therefore reflect the pre-describe state.

After all batches complete, refresh the manifest tables from the just-updated .meta files:

python3 ~/.smith/scripts/smith-index/run.py --rebuild-manifests \
  --root "$ROOT"

(Falls back to scripts/smith-index/run.py in repo-dev layouts.)

This mode re-reads .smith/index/files/*.meta, salvages the module descriptions from each file's description layer, and re-renders manifest.md + every systems/<id>.md. It does NOT re-parse source or touch .meta files. Skipped if --describe aborted before any descriptions were written.

Step 6 — Summary

After all batches complete (or on abort):

python3 ~/.smith/scripts/describe_checkpoint.py summary \
  --log "$LOG_PATH" --start-iso "$START_ISO"

Format: /smith-index --describe: N files described (succeeded=S failed=F skipped=K) in T.Ts.

On clean completion, remove the checkpoint state file. On Ctrl-C or fatal error, leave it in place so --resume works.

Failure handling

• Per-Task failure. Exponential backoff retry (5s → 10s → 20s,

max 3). After 3, log failed, continue. No run-level abort.

• Helper script failure (non-zero exit). Surface stderr; record

a failed JSONL entry; continue.

• Model probe failure. Hard abort before any bulk work, with the

clear-error message above. --skip-model-probe overrides.

• Missing helper at install location. If

~/.smith/scripts/describe_discover.py is not found, fall through to the repo-relative path scripts/parsers/describe_discover.py. If neither resolves, exit 78 (EX_CONFIG) with: "Smith helpers not installed. Run npx skills add ATTCKDigital/smith to install."

/smith-index --migrate-templates

Non-destructive template migration for existing projects (Q2). For each of constitution.md (or .specify/memory/constitution.md) and CLAUDE.md:

1. Detect missing top-level headers from the template additions:

• ## File Size Policy
• ## Project Manifest
• ## Smith Context System
• ## File Size Awareness

1. If any are missing, write a .bak.<ISO8601> backup of the original.
2. Append the missing sections (sourced from

templates/constitution-additions.md and templates/claude-md-additions.md).

1. Backfill the `base_branch:` frontmatter field on the constitution

(idempotent). If .specify/memory/constitution.md (or constitution.md) lacks a base_branch: key in its YAML frontmatter, add base_branch: main (the backwards-compatible default — older constitutions implicitly meant main). Handle both shapes:

• Frontmatter block present (file starts with a --- fence): insert

base_branch: main as a new line inside the first ---/--- block, after the opening fence.

• No frontmatter block (file starts with # ... Constitution): prepend

a new block: ```markdown

base_branch: main

`` If a base_branch:` key is already present (any value, including a user-customized one), do NOTHING — never overwrite an existing value. This step shares the backup taken in step 2 (take one if not already taken).

1. Skip silently if all sections AND the base_branch: field are already

present (idempotent).

Never overwrites existing user content. Never modifies sections that are already there, even if the template's wording has changed since the section was first added.

/smith-index --incremental

Re-parse only files changed in git diff <from>..<to>. Designed for the post-merge and post-checkout git hooks (per Design Decision 8).

• Default refs: ORIG_HEAD..HEAD.
• Override with --from <ref> --to <ref>.
• Filters changed files to allowed source extensions; runs the same

parse + .meta + per-system + top-level update pipeline as a single PostToolUse hit.

• After re-parsing the diffed subset, rebuilds the full per-system and

top-level manifests from the existing .meta files (so unchanged systems still appear correctly in the regenerated tables).

• Exits 0 silently if git is unavailable or the project has no .git/.

Typical runtime: <2s for normal pulls (5-20 file changes).

/smith-index --init-system-paths

Optional bootstrap helper. Writes a stub .smith/index/config/system-paths.json derived from the project's top-level directories. Per Q7, system-paths.json is OPTIONAL — the heuristic engine handles missing config — so this flag exists only for users who want explicit overrides as a starting point. Does NOT overwrite an existing file.

/smith-index --resume

Continue an interrupted run. Reads the latest smith-index-<ISO>.jsonl log under ~/.smith/logs/, computes the set of files that completed all stages through system-update, and skips them on the resumed run. The checkpoint at .smith/index/.smith-index-checkpoint.json is consulted to recover the in-progress system context.

Per Rule 4: --resume is a no-op if no checkpoint or recent JSONL log exists; it falls back to a fresh run with a warning.

Auto-invocation

/smith init calls /smith-index as its final setup step (per spec Requirement 5). On a fresh project this:

1. Creates .smith/index/ and subdirectories.
2. Copies templates/context-manifest.default.json into

.smith/index/config/context-manifest.json if absent.

1. Does NOT copy system-paths.json (per Q7 — only on

--init-system-paths).

1. Runs the full rebuild.

Outputs

Configuration files (NOT regenerated)

Logging

• One JSONL line per file per stage (parse, meta, system-update,

top-update) to ~/.smith/logs/smith-index-<ISO>.jsonl.

• Summary line to stdout on completion (NOT to JSONL).

Error handling

• Per-file failures are counted, never abort the run.
• Parser timeouts emit a partial .meta with ## Parse Errors populated.
• Missing optional config (system-paths.json) falls back to the

heuristic resolver.

• Missing git short-circuits --incremental to a no-op.

Examples

/smith-index                          # full rebuild
/smith-index --check                  # staleness scan, no rebuild
/smith-index --system system-backend  # rebuild one system
/smith-index --incremental            # re-parse `git diff ORIG_HEAD..HEAD`
/smith-index --incremental --from HEAD~1 --to HEAD
/smith-index --describe               # generate LLM descriptions (Task-spawned)
/smith-index --describe --yes         # skip the pre-flight confirm gate
/smith-index --describe --system foo  # describe one system only
/smith-index --describe --resume      # resume an interrupted describe run
/smith-index --migrate-templates      # patch constitution.md / CLAUDE.md
/smith-index --init-system-paths      # write stub system-paths.json
/smith-index --resume                 # continue interrupted run

Where this skill is invoked from

• `/smith init` — calls /smith-index as the final setup step.
• `post-merge` git hook — calls /smith-index --incremental.
• `post-checkout` git hook — calls `/smith-index --incremental

--from $prev_head --to $new_head`.

• `context-loader.sh` — does NOT auto-invoke; surfaces a soft

warning when .smith/index/manifest.md is absent.

• User, manually — for any of the above modes plus --check and

--system.

Implementation reference

• Entry (all modes except --describe): scripts/smith-index/run.sh →

scripts/smith-index/run.py

• Entry (--describe only): this skill's prose drives the loop directly,

using the helpers below.

• Parsers: scripts/parsers/parse-python.py, scripts/parsers/parse-js.js
• Path resolver: scripts/parsers/path-resolver.py
• Parser-lib helper: scripts/parsers/parser-lib.sh
• v3 description helpers:

scripts/parsers/describe_discover.py, scripts/parsers/describe_write.py, scripts/parsers/describe_checkpoint.py, scripts/parsers/index_common.py (shared utilities), scripts/parsers/meta_describe.py (structural; LLM-call-free).

• Templates: templates/constitution-additions.md,

templates/claude-md-additions.md