thoreinstein/gemini-obsidian/wiki-ingest

Wiki Ingest

Distill external sources into vault-native notes following VAULT_CONVENTIONS, with provenance tracking and deduplication.

Workflow

1. Identify Sources

Accept one or more source paths from the user. Sources can be:

• Files on disk (markdown, PDF, text, transcripts)
• Vault-internal notes (Readwise highlights, working notes to promote)
• A directory to scan for ingestible files

For vault-internal sources, use obsidian_read_note. For external files, use the Read tool.

2. Check Manifest

Read .manifest.json from the vault root using Read tool.

For each source path, check if it exists in sources:

• Not present → new source, proceed to extract
• Present but source modified since `ingested_at` → re-ingest
• Present and unchanged → skip (tell user "already ingested, use --force to re-process")

3. Extract Knowledge

Read the source content. Identify:

• Patterns — reusable approaches, techniques, or practices
• Traps — pitfalls, anti-patterns, things that went wrong
• Decisions — architectural or design choices with rationale
• Concepts — definitions, mental models, frameworks

For each extraction, classify its provenance:

• Extracted (unmarked) — directly stated in the source
• Inferred ^[inferred] — synthesized, generalized, or implied but not stated
• Ambiguous ^[ambiguous] — source is unclear or contradicts other sources

4. Deduplicate

For each extracted concept, search the vault:

1. Call obsidian_rag_query with the concept name/summary, limit: 5
2. If a strong match exists, read the existing note with obsidian_read_note
3. Decide: merge into existing note or create new note

Merge when: the existing note covers the same concept and the new source adds information. Create when: no existing note covers this concept, or the angle is sufficiently different.

5. Present Plan

Before creating or modifying any notes, present the plan to the user:

## Ingest Plan for: <source name>
 
### New Notes
- Engineering/Patterns/<name>.md — <one-line summary>
- Engineering/Traps/<name>.md — <one-line summary>
 
### Merge Into Existing
- Engineering/Patterns/<existing>.md — adding: <what's new>
 
### Skipped
- <concept> — already covered in [[existing-note]]

Wait for user confirmation before proceeding.

6. Write Notes

For new notes, use obsidian_create_note with content following VAULT_CONVENTIONS templates:

• Set type: to pattern, trap, or decision
• Set project: to the source project or "general"
• Set tags: from content
• Set created: to today's date
• Set provenance: fractions based on inline marker counts
• Include ## Links section with [[wikilinks]] to related vault notes
• Add ## Sources section citing the ingested source

For merges, use obsidian_insert_at_heading or obsidian_replace_section to add new content to existing notes. Update the provenance: frontmatter if the mix changed, using obsidian_update_frontmatter.

7. Placement Rules

• Cross-project patterns/traps/decisions → Engineering/Patterns/, Engineering/Traps/, Engineering/Decisions/
• Project-specific knowledge → working/<project>/knowledge/
• If unclear, ask the user

8. Update Manifest

After all notes are written, update .manifest.json:

For each processed source, add/update an entry in sources:

{
  "<source-path>": {
    "ingested_at": "<ISO8601 now>",
    "size_bytes": <file size>,
    "modified_at": "<source file mtime>",
    "source_type": "document",
    "pages_created": ["Engineering/Patterns/foo.md"],
    "pages_updated": ["Engineering/Traps/bar.md"]
  }
}

Update stats.total_sources_ingested and stats.total_pages. Set last_updated to now.

Write the updated manifest back using the Write tool.

9. Suggest MOC Updates

For each new Engineering/ note created, check which thematic MOCs it belongs in. Suggest additions but don't auto-edit MOCs — use /moc-update for that.

Arguments

• Source path(s): /wiki-ingest ~/Documents/paper.pdf
• Optional project scope: /wiki-ingest working/appiary/knowledge/auth-notes.md
• Optional force flag: /wiki-ingest --force <path> to re-ingest unchanged sources

Tips

• Run /index first if obsidian_rag_query returns poor results
• For Readwise books, ingest the book's highlight file (e.g., Readwise/Books/<title>.md)
• Large sources may yield many extractions — review the plan carefully before confirming
• Provenance fractions don't need to be exact; approximate to the nearest 0.05