geronimo-iia/llm-wiki-skills/ingest

Ingest

Process source files from the inbox/ queue into synthesized, cross-referenced wiki pages. The engine moves each source to raw/ on successful ingest — the move is the record of completion.

Workflow

Step 0 — Discover the wiki root

wiki_spaces_list()

The response includes the wiki path for each registered space. All paths below (inbox/, raw/) are relative to the default wiki's path.

Step 1 — Scan inbox/

If the user provided a path, use it (must be inside inbox/):

wiki_ingest(path: "<user-provided-path>", dry_run: true)

Otherwise scan the entire queue:

wiki_ingest(path: "inbox/", dry_run: true)

If no files are found, report "No files to process." and stop.

Step 1.5 — Filter the queue

Skip proactively — do not attempt to process:

• Binary files: .pdf, .jpg, .jpeg, .png, .drawio, .gif
• Raw source .txt files when a higher-level analysis of the same source already exists in the queue
• Deprecated documents superseded by a newer version — record as skipped with reason
• Files without a --- YAML frontmatter block when targeting a wiki with skip_no_frontmatter: true (the default) — these are skipped at ingest time; set skip_no_frontmatter: false in [ingest] to index them

Process files one at a time — complete all sub-steps for one file before moving to the next.

Step 2 — Process each file

For each file in inbox/, in order:

2a. Read the source

Read the file and understand what it contains — topics, claims, key concepts, structure.

Natively readable: .md, .txt, .csv, .json, .yaml, .html, .xml, and most text-based files.

If reading fails (binary, unsupported format): note the reason and skip to step 2h. The file stays in inbox/ for manual handling.

Also skip proactively (without attempting to read) if the file matches the skip rules from Step 1.5 — binary, superseded raw extract, or deprecated document. Record the reason in step 2h.

2b. Classify the source

Determine the source type by the material's nature, not its topic:

A blog post about academic research is article, not paper.

Identify the main topic and domain. Ask: what does a reader learn from this? That drives placement, not the document format.

If the source type is ambiguous (e.g. a PDF that could be a paper or documentation), ask the user before proceeding.

2c. Find integration points

For the first file in a session, orient with the full wiki map:

wiki_list(format: "llms")

This produces all pages grouped by type with summaries in a single call — use it to understand what already exists before searching for specific candidates. Subsequent files in the same session can skip this call since the map is already in context.

Search for existing pages on the same topic:

wiki_search(query: "<main topic from source>")
wiki_search(query: "<candidate page title>")

Also check by type to find related source and concept pages:

wiki_search(query: "<topic>", type: "concept")
wiki_search(query: "<topic>", type: "<source-type>")

Read the most relevant results:

wiki_content_read(uri: "<relevant-slug>")

This serves two purposes:

• Duplicate detection — if a page already covers this subject,

prefer updating it over creating a new one

• Context — understand the current state of knowledge so the

new page integrates naturally

If multiple candidate pages could be the integration target, ask the user which one to update.

2d. Analyse the source

Before writing anything, produce an analysis of the source:

Enumerate durable knowledge items: For each concept, claim, finding, or entity in the source, note:

• What — one sentence
• Type — concept, paper, article, documentation, etc.
• Novelty — new (not in wiki), extends (adds to existing page),

contradicts (conflicts with existing page), duplicate (already covered)

• Confidence — estimated score (see calibration table in the frontmatter skill)

Detect contradictions: For each page found in step 2c, compare its claims against the source. Flag explicit contradictions — do not infer or speculate. Report:

• Which source claim contradicts which wiki page
• The slug of the page to update
• Whether the source or the wiki appears more authoritative

Pages with unresolved contradictions should be created or left with status: draft so the review skill picks them up in the next review session.

Produce an ingest plan:

Example:

Source: "Mixtral of Experts" (paper, 2024)
 
1. Mixture of Experts — concept — extends concepts/mixture-of-experts [0.9]
2. Mixtral architecture — concept — new: concepts/mixtral-architecture [0.85]
3. Sparse MoE routing efficiency — claim — extends concepts/sparse-routing [0.8]
4. Contradicts: concepts/moe-compute-cost claims O(n); source claims O(k/n) — update needed
5. Switch Transformer comparison — extends sources/switch-transformer-2021 [0.75]

Present the plan to the user and confirm before writing. If the user redirects (e.g. "skip item 2", "merge 1 and 2"), update the plan before proceeding.

If no user confirmation is available (automated pipeline), proceed with the plan as written — flag any contradictions as status: draft for manual review.

2e. Decide: create or update

• If a concept page already covers this topic → update it with new

claims and sources

• If no existing page → create a new one
• If the source introduces a new concept → create a concept page
• If the source is a summary of a specific document → create a source

page with the classified type

• One source can produce multiple pages — a rich document may warrant

a source page plus new concept pages

2f. Write pages

Work through the ingest plan items in order:

• Each new item → wiki_content_new + direct file write (see below)
• Each extends item → read existing page first (wiki_content_read), then update
• Each contradicts item → see contradiction handling below
• Each duplicate item → skip; note in step 2h outcome

Get the frontmatter scaffold for the target type:

wiki_schema(action: "show", type: "<type>", template: true)

Follow the frontmatter skill for correct field values per type.

Creating a new page — direct write pattern:

If the source has associated assets (images, PDFs, diagrams), create a page bundle so assets live alongside the page:

wiki_content_new(uri: "<slug>", bundle: true)
→ { "path": "/abs/path/to/slug/index.md", "wiki_root": "...", ... }

Otherwise create a regular page:

wiki_content_new(uri: "<slug>", type: "<type>")
→ { "path": "/abs/path/to/slug.md", "wiki_root": "...", ... }

Write content directly to the returned path using your file tools (no MCP round-trip):

# Write the full markdown file directly to result.path

For each page:

• Set the correct type
• Write all required fields for that type (the template shows them)
• Add sources, concepts, tags, aliases, confidence, claims as relevant;

set confidence to 0.5 on new pages (single source, unreviewed) and raise it only as corroborating evidence accumulates

• Write a structured body — synthesize, do not copy verbatim
• Add links to related pages found in step 2c

Edge target types: each edge field (sources, concepts) declares allowed target types via x-graph-edges / target_types in the schema. Check the schema to confirm valid targets before writing:

wiki_schema(action: "show", type: "<type>")

Updating an existing page (accumulation contract):

1. Read the current page first with wiki_content_read
2. Preserve existing list values — do not drop tags, aliases, read_when,

sources, concepts, or claims added by prior ingests

1. Add new values to lists, do not replace them
2. Update scalar fields (summary, tldr, confidence) only when

there is a clear reason

1. Write the complete file

Contradiction handling: when a source contradicts an existing page:

1. Read the existing page with wiki_content_read
2. Add a new entry to claims[] with the source's version and a reference to the source page
3. Lower the page-level confidence if the contradiction is unresolved
4. Add an open question in the body flagging the discrepancy for human review

Do not silently overwrite the existing claim — preserve both versions.

2g. Validate and index

Pre-check the schema:

wiki_schema(action: "validate")

Dry-run to catch errors without committing:

wiki_ingest(path: "<path-relative-to-wiki-root>", dry_run: true)

Dry-run validates frontmatter against the type's JSON Schema and reports edge target type mismatches. Fix any reported issues, then ingest for real:

wiki_ingest(path: "<path-relative-to-wiki-root>")

Redaction (opt-in): when ingesting external content that may contain secrets — web clips, session transcripts, raw notes — pass redact: true:

wiki_ingest(path: "<path>", redact: true)

The engine scans each file body for 6 built-in patterns (GitHub PAT, OpenAI key, Anthropic key, AWS access key, Bearer token, email) and replaces matches with [REDACTED:pattern-name] before commit. The report includes a redacted field with the slug and line numbers of each match. Redaction is lossy — original values are gone after the file is written to disk. Do not use it on pages you have already reviewed and committed.

Per-wiki, you can disable specific patterns or add custom ones in wiki.toml:

[redact]
disable = ["email"]   # keep emails in a people/contacts wiki
 
[[redact.patterns]]
name        = "employee-id"
pattern     = "EMP-[0-9]{6}"
replacement = "[REDACTED:employee-id]"

The engine validates frontmatter, updates the tantivy search index, and commits to git (if ingest.auto_commit is true). If validation fails, the file is rejected with a clear error showing which fields are missing or invalid.

After successful ingest, move the source file from inbox/ to raw/ yourself:

mv <wiki-root>/inbox/<filename> <wiki-root>/raw/<filename>

The move is the record of completion — a file in raw/ means it was successfully ingested.

If validation passes but you want deeper checks (broken wikilinks, orphan pages, style issues), follow the lint skill.

2h. Record outcome

Track the result for this file:

• Pages created (slugs and types)
• Pages updated (what changed)
• Skipped (reason: unreadable, failed validation, etc.)
• Contradictions flagged (slug + which claims conflict)

Then proceed to the next file in inbox/.

Step 3 — Summary

After all files are processed, report to the user:

• Files processed with outcomes
• Pages created (with slugs and types)
• Pages updated (what changed)
• Files skipped with reasons
• Contradictions flagged (slug, conflicting claims, resolution status)
• Suggested follow-ups (e.g. run lint, check graph edges)

After ingest, use wiki_suggest to propose links for new pages, and the graph skill to verify new pages are connected as expected (edges, backlinks, section placement).

Linking policy

When adding links between pages — in frontmatter (sources, concepts) or body ([[wikilinks]]) — apply this test: would a reader of this page benefit from navigating to the linked page?