lambenthan/empiricalwiki/prefill

/prefill

Sediments foundational background (seminal methods, common practice, standard architectures) into wiki/foundations/ as terminal pages.

Foundations are single-direction by design: other pages link to them, foundations write no reverse links.

Trigger

Manual: /prefill [domain] or /prefill --add "concept name".

Inputs

• domain (positional, optional): research domain — one of general, NLP, CV, ML Systems, Robotics. If omitted, infer from wiki/topics/ tags; if wiki/topics/ is empty, prompt the user.
• --add "<concept>": skip the catalog and seed exactly one foundation by name.

Outputs

• wiki/foundations/{slug}.md — one page per seeded concept
• Updated wiki/index.md (foundations section regenerated by rebuild-index)
• wiki/log.md entry

Wiki Interaction

Reads

• wiki/topics/*.md — for domain inference (when domain is omitted)
• wiki/foundations/*.md — to skip already-seeded concepts (idempotent)
• .claude/skills/prefill/foundations-catalog.yaml — seed list

Writes

• wiki/foundations/{slug}.md (new only — never overwrite)
• wiki/index.md (via tools/research_wiki.py rebuild-index)
• wiki/log.md (via tools/research_wiki.py log)

Workflow

Pre-conditions: working directory contains wiki/, tools/, .claude/. Set WIKI_ROOT=wiki/.

Step 1: Resolve domain

1. If domain argument given → use it.
2. Else if --add mode → domain is general unless the user specified one.
3. Else: read all wiki/topics/*.md frontmatter tags; if a single dominant domain is detected, use it; otherwise ask the user.

Step 2: Load seeds

• Catalog mode: read .claude/skills/prefill/foundations-catalog.yaml. Pick all entries under domains.{domain} plus everything under domains.general (general foundations apply to every research field).
• `--add` mode: synthesize a single seed entry {slug: <slugified concept>, title: <concept>, summary: ""}. Use python3 tools/research_wiki.py slug "<concept>" to derive the slug.

For each seed, check wiki/foundations/{slug}.md. If it already exists, skip (do not overwrite, do not warn).

Step 3: Fetch background from Wikipedia

For each remaining seed, call tools/fetch_wikipedia.py:

python3 tools/fetch_wikipedia.py summary "<title>"
python3 tools/fetch_wikipedia.py sections "<title>"
python3 tools/fetch_wikipedia.py section "<title>" --index <N>   # for relevant sections

• The summary call returns {title, extract, url}.
• The sections call returns a list of {index, line, level} — pick sections whose line matches Variants, Types, Architecture, History, Limitations, Applications (case-insensitive substring match).
• Exit code 2 from any call means page not found — fall back to LLM knowledge for that seed and set source_url: "" in the resulting frontmatter.

Step 4: Compose the foundation page

Render each seed into the template below. Distinguish Wikipedia-derived content from LLM-supplied content by appending (LLM analysis) to sections that have no Wikipedia source material.

---
title: "{title}"
slug: "{slug}"
domain: "{domain}"
status: mainstream         # or historical, if the seed is a superseded technique
aliases: []                # list any common aliases the LLM is confident about
first_introduced: "{year if present in Wikipedia summary, else empty}"
date_updated: "{today}"
source_url: "{wikipedia url, or empty if 404}"
---
 
## Definition
{First paragraph of Wikipedia summary, or LLM-supplied definition.}
 
## Intuition
{Plain-language explanation built on the definition.}
 
## Formal notation
{Math/notation extracted from Wikipedia, or LLM-supplied with `(LLM analysis)` tag.}
 
## Key variants
{Bulleted list distilled from Wikipedia "Variants"/"Types"/"Architecture" sections.}
 
## Known limitations
{From Wikipedia + LLM judgment.}
 
## Open problems
{LLM analysis (LLM analysis)}
 
## Relevance to active research
{LLM analysis (LLM analysis)}

Write each file to wiki/foundations/{slug}.md.

Step 5: Refresh navigation and log

python3 tools/research_wiki.py rebuild-index wiki/
python3 tools/research_wiki.py log wiki/ "prefill | {N} foundations created for {domain}"

Step 6: Report

Print a grouped summary:

## Prefill Report — {date}
 
**Domain**: {domain}
**Created**: {N}  **Skipped (already present)**: {M}
 
### mainstream
- foundations/gradient-descent — Gradient Descent
- ...
 
### historical
- foundations/recurrent-neural-networks — Recurrent Neural Networks

Remind the user that subsequent /ingest runs will dedup against these foundations and create wikilinks ([[foundation-slug]]) instead of new concept pages.

Constraints

• foundations are terminal: never write key_papers, related_concepts, or any outbound reference field on a foundation page. Other pages may link in.
• never overwrite an existing wiki/foundations/{slug}.md (idempotent re-runs).
• distinguish sources: Wikipedia-derived content vs. LLM-derived content must be visually distinct in the page body.
• catalog is advisory: the YAML seed list is hand-curated and incomplete. Users may extend it without code changes.
• only writes to `wiki/foundations/`: never creates pages under papers/, concepts/, topics/, etc.

Error Handling

• `wiki/foundations/` does not exist: run python3 tools/research_wiki.py init wiki/ first.
• Wikipedia 404: log the missing page, fall back to LLM knowledge for that seed (source_url: "").
• Network failure: print which seeds failed and continue with the remainder; do not abort the whole batch.
• Catalog file missing: print error pointing to .claude/skills/prefill/foundations-catalog.yaml.

Dependencies

Tools (via Bash)

• python3 tools/fetch_wikipedia.py summary|sections|section|wikitext "<title>" [--index N]
• python3 tools/research_wiki.py slug "<title>"
• python3 tools/research_wiki.py rebuild-index wiki/
• python3 tools/research_wiki.py log wiki/ "<message>"

Catalog

• .claude/skills/prefill/foundations-catalog.yaml