attckdigital/smith/smith-index

Smith Index

Generate the deterministic project manifest at .smith/index/. The manifest replaces soft natural-language guidance with structured, indexed context — every Smith skill (/smith-new, /smith-bugfix, /smith-debug, /smith-explore, …) consults it through the same retrieval path (/smith-navigate + context-loader.sh).

Arguments: $ARGUMENTS

Manifest is a map, not a fence

.smith/index/ is a navigation aid, not a hard boundary. Skills like /smith-explore still grep the whole codebase when initial signals suggest broader impact. A stale or imprecise manifest must never block the calling session — it should degrade gracefully to vault-only context plus a soft warning.

Behavior

This skill is imperative — running it modifies .smith/index/ and (with --migrate-templates) constitution.md / CLAUDE.md. It does NOT modify any source file in the project. All generated state is confined to .smith/index/ plus optional .bak.<timestamp> files on template migration.

The actual work runs in ~/.smith/scripts/smith-index/run.py (called via ~/.smith/scripts/smith-index/run.sh), installed by scripts/install.sh from this repo. In the smith-repo dev tree these same files live at scripts/smith-index/run.py / run.sh — invoking either works. The skill markdown is the entry point that parses $ARGUMENTS, decides the mode, and shells out.

Modes (flags)

/smith-index — full rebuild (default)

1. Walk the project from the current directory.
2. Honor .gitignore (uses git ls-files when available; falls back to a

manual exclusion list of node_modules/, .git/, .venv/, etc.).

1. For each source file (.py, .js, .jsx, .ts, .tsx, .css,

.html, .sh):

• Resolve the parser via parser-lib.sh resolve_parser <ext> (prefers

.smith/scripts/ over ~/.smith/scripts/ over the in-repo fallback).

• Run the parser; capture JSON.
• Compute SHA-256 of the first 4KB of source content (Q6 — hash field

in .meta).

• Render the .meta file at .smith/index/files/<mirrored>/<file>.meta.
• Resolve the file's system via path-resolver.py (longest-prefix

system-paths.json override → heuristic per spec Requirement 14).

1. Per system: rewrite systems/<sys>.md once with all files bucketed in

that system (sorted by lines desc; truncated at 60 entries with …and N more files per data-model.md section 3). Cap ≤80 lines.

1. Rewrite top-level manifest.md (systems table + Stats). Cap ≤50 lines.
2. Write checkpoint state to .smith/index/.smith-index-checkpoint.json

every 25 files; delete on clean exit.

1. Append one JSONL log line per stage per file to

~/.smith/logs/smith-index-<ISO8601>.jsonl per Rule 4.

1. Write the schema-version marker at .smith/index/.schema-version

containing the current schema version (read from ~/.smith/scripts/meta_schema_version.txt, falls back to scripts/parsers/meta_schema_version.txt in the smith-repo dev tree if the global install is missing). This file lets /smith-update detect projects whose manifest was generated against an older .meta schema and offer to regenerate. The marker is overwritten on every full rebuild (and on --incremental runs that write a fresh manifest); silently skipped if neither source file is found.

1. Print a summary line:

/smith-index: N files indexed (N succeeded, N failed, N skipped) in T.Ts.

1. Prune stale index state. Runs only when this is a full,

unfiltered, non-resumed rebuild (i.e. no --system and no --resume on this invocation). Removes .smith/index/files/**/*.meta entries whose source file is gone or now resolves to "excluded", and any .smith/index/systems/<id>.md left with zero files after this run. The summary line gains a · <N> pruned clause on every applicable run (including · 0 pruned when nothing was stale) — the clause is absent entirely on --system, --resume, --check, and --incremental runs, none of which prune. See this feature's plan.md (.specify/systems/cross-system/features/ 59-wordpress-aware-index/plan.md) Contracts section for the full per-mode rationale.

Performance budget: <60s p95 for a 100-file project (acceptance criterion from spec).

/smith-index --check

Hash-only staleness scan. No rebuild. For each existing .meta, compute SHA-256 of the first 4KB of the corresponding source file and compare against the Hash: line in the .meta. Reports:

• Fresh count (hashes match)
• Stale list (hash mismatch — file edited externally; manifest hook

missed it; git checkout changed content; etc.)

• Missing-source list (.meta exists but source was deleted/renamed)

No mtime comparison — Q6 is hash-only. Estimated ~5-10s for a 400-file project; acceptable for a maintenance command.

/smith-index --system <name>

Partial rebuild restricted to files mapped to one system. Useful after adding a single feature: refreshes that system's .meta files and systems/<name>.md without re-walking the entire tree. Top-level manifest.md Stats section is also updated.

/smith-index --describe

Generate per-file LLM descriptions in the .meta description layer. Unlike all other modes, --describe is orchestrated BY this skill prose itself — not by run.py. The skill drives a discovery → batched-spawn → write loop, spawning one Task sub-agent per file (or per method on dense files) that returns a MetaDescription JSON. Each spawned Task inherits the session's Claude Code auth → subscription billing, not API-key billing.

Single backend. v3 (PR #23) removed the v2 direct-HTTPS path. The 2am scheduler invokes claude --print -p "/smith-queue process ..." which IS a Claude Code session — Task spawning works there too. No --llm-backend flag, no CLAUDE_HEADLESS env var.

Step 0 — Runtime model probe

Before any bulk work, spawn ONE small Task with:

subagent_type: general
model: claude-haiku-4-5
prompt: "Respond with exactly: MODEL_OK"

If the response doesn't arrive cleanly OR the trimmed response text is not exactly MODEL_OK (heuristic: a Haiku model honoring the override responds crisply; an Opus/Sonnet primary will be more verbose), abort with:

ERROR: Could not verify Haiku model override. Running the bulk

loop on the session's primary model would inflate subscription

cost ~30×. Verify your Task tool subagent type supports the model

parameter. Pass --skip-model-probe to override at your own risk.

--skip-model-probe bypasses this check.

Step 1 — Discovery

python3 ~/.smith/scripts/describe_discover.py \
  --root "$ROOT" \
  ${SYSTEM:+--system "$SYSTEM"} \
  --threshold "${THRESHOLD:-5}"

(Use the repo-relative path scripts/parsers/describe_discover.py if ~/.smith/scripts/ does not resolve.)

Parse the JSON output. Each entry has rel_path, source_hash, parser_output, qualifying_method_ids, existing_description, cache_hit, system. Drop entries with cache_hit=true — these are no-ops (their .meta already matches the current source hash).

Step 2 — Resume filter

If --resume was passed:

python3 ~/.smith/scripts/describe_checkpoint.py load-completed \
  --log-dir ~/.smith/logs \
  --state .smith/index/.smith-index-describe-checkpoint.json

Filter the remaining files to exclude completed rel_path values.

Step 3 — Pre-flight estimate + confirmation gate

After filtering, count files needing description (N) and sum their qualifying_method_ids counts (M). Identify files where len(qualifying_method_ids) > 15 (the per-method-split threshold); each such file contributes that many Tasks instead of 1. Let T be the total Task count.

Print to stderr:

/smith-index --describe pre-flight summary
─────────────────────────────────────────────
  Files needing description: N
  Qualifying methods total: M
  Per-method-split threshold: 15 (configurable via --per-method-threshold)
  Per-method-split files: K
  Estimated Tasks to spawn: T
  Estimated wall time: ~W minutes (5s/Task sequential)

Then ask: Proceed? (y/N):. Read one line from stdin. Accept y/yes (case-insensitive). --yes bypasses the gate (required for the scheduler).

Step 4 — Sequential Task spawning loop

Batch the remaining files in groups of 10 (default; override with --batch-size). For each batch, process files sequentially (one at a time — no parallel tool-use block — simpler per-Task error handling, visible progress logging).

For each file in the batch:

1. Per-method-split decision. If

len(qualifying_method_ids) > 15 (default; override with --per-method-threshold), spawn one Task PER METHOD. Otherwise one Task for the whole file.

1. Build the prompt body. Assemble via the helper (single source

of truth for prompt template): ``bash PROMPT=$(python3 ~/.smith/scripts/describe_write.py build-prompt \ --rel-path "$REL" --root "$ROOT" \ --method-ids "<comma-separated-ids>" --module) ``

1. Spawn the Task.

``yaml subagent_type: general model: claude-haiku-4-5 prompt: | <PROMPT body from step 2> ``

1. Retry on failure (exponential backoff). If the Task call

fails or returns malformed JSON or status="error", retry with backoff 5s → 10s → 20s. Max 3 attempts. After 3, log a failed JSONL record and move to the next file. Do NOT abort the run.

1. STUB MODE. If SMITH_TASK_STUB=1 is set in the env, skip

the Task spawn entirely. Pipe the canned fixture into the writer: ``bash python3 ~/.smith/scripts/describe_write.py apply --from-stub \ tests/fixtures/task-stub-responses.json \ --rel-path "$REL" --root "$ROOT" --hash "$HASH" ` The stub fails loud (exit 4) if any qualifying method id is not in the fixture. Tests set SMITH_TASK_STUB=1`; users never do.

1. Apply the result. Pipe the Task's JSON output into the writer:

``bash echo "$TASK_OUTPUT" | \ python3 ~/.smith/scripts/describe_write.py apply \ --rel-path "$REL" --root "$ROOT" --hash "$HASH" ``

1. Append a checkpoint record. One JSONL line per file:

``bash python3 ~/.smith/scripts/describe_checkpoint.py append \ --log "$LOG_PATH" \ --record "$(printf '{"item_id":"%s","stage":"describe", "status":"ok","backend":"task", "method_count":%d,"module_chars":%d, "batch_index":%d,"retry_count":%d}' \ "$REL" "$N" "$M" "$BATCH_IDX" "$RETRIES")" ``

Then persist checkpoint state: ``bash python3 ~/.smith/scripts/describe_checkpoint.py save \ --path .smith/index/.smith-index-describe-checkpoint.json \ --processed "$REL" ``

Step 5 — Propagate descriptions to manifest tables

/smith-index (full rebuild) populates per-file rows in systems/<id>.md and the top-level manifest.md during its source walk — well before --describe writes the description layer into each .meta. The manifest tables therefore reflect the pre-describe state.

After all batches complete, refresh the manifest tables from the just-updated .meta files:

python3 ~/.smith/scripts/smith-index/run.py --rebuild-manifests \
  --root "$ROOT"

(Falls back to scripts/smith-index/run.py in repo-dev layouts.)

This mode re-reads .smith/index/files/*.meta, salvages the module descriptions from each file's description layer, and re-renders manifest.md + every systems/<id>.md. It does NOT re-parse source or touch .meta files. Skipped if --describe aborted before any descriptions were written.

Step 6 — Summary

After all batches complete (or on abort):

python3 ~/.smith/scripts/describe_checkpoint.py summary \
  --log "$LOG_PATH" --start-iso "$START_ISO"