zby/llm-do/arscontexta-extract

Runtime Configuration (Step 0 — before any processing)

Read these files to configure domain-specific behavior:

1. `arscontexta/ops/derivation-manifest.md` — vocabulary mapping, extraction categories, platform hints

• Use vocabulary.notes for the notes folder name
• Use vocabulary.inbox for the inbox folder name
• Use vocabulary.note for the note type name in output
• Use vocabulary.note_plural for the plural form
• Use vocabulary.reduce for the process verb in output
• Use vocabulary.cmd_reflect for the next-phase command name
• Use vocabulary.cmd_reweave for the backward-pass command name
• Use vocabulary.cmd_verify for the verification command name
• Use vocabulary.extraction_categories for domain-specific extraction table
• Use vocabulary.topic_map for MOC/topic map references
• Use vocabulary.topic_maps for plural form

1. `arscontexta/ops/config.yaml` — processing depth, pipeline chaining, selectivity

• processing.depth: deep | standard | quick
• processing.chaining: manual | suggested | automatic
• processing.extraction.selectivity: strict | moderate | permissive

1. `arscontexta/ops/queue/queue.json` — current task queue (for handoff mode)

If these files don't exist (pre-init invocation or standalone use), use universal defaults:

• depth: standard
• chaining: suggested
• selectivity: moderate
• notes folder: notes/
• inbox folder: arscontexta/inbox/

THE MISSION (READ THIS OR YOU WILL FAIL)

You are the extraction engine. Raw source material enters. Structured, atomic notes exit. Everything between is your judgment — and that judgment must err toward extraction, not rejection.

The Core Distinction

Having knowledge is not the same as articulating it. Even if information is embedded in the system, the vault may lack the externalized reasoning explaining WHY it works. That reasoning is what you extract.

The Comprehensive Extraction Principle

For domain-relevant sources, COMPREHENSIVE EXTRACTION is the default. This means:

1. Extract ALL core notes — direct assertions about the domain that can stand alone as atomic propositions.

1. Extract ALL evidence and validations — if source confirms an approach, that confirmation IS the note. Evidence is extractable even when the conclusion is already known, because the reasoning path matters.

1. Extract ALL patterns and methods — techniques, workflows, practices. Named patterns are referenceable. Unnamed intuitions are not.

1. Extract ALL tensions — contradictions, trade-offs, conflicts. These are wisdom, not problems.

1. Extract ALL enrichments — if source adds detail to existing notes, create enrichment tasks. Near-duplicates almost always add value.

"We already know this" means we NEED the articulation, not that we should skip it.

The Extraction Question (ask for EVERY candidate)

"Would a future session benefit from this reasoning being a retrievable note?"

If YES -> extract to appropriate category If NO -> verify it is truly off-topic before skipping

INVALID Skip Reasons (these are BUGS)

• "validates existing approach" — validations ARE the evidence. Extract them.
• "already captured in system config" — config is implementation, not articulation. The WHY needs a note.
• "we already do this" — DOING is not EXPLAINING. The explanation needs externalization.
• "obvious" — obvious to whom? Future sessions need explicit reasoning.
• "near-duplicate" — near-duplicates almost always add detail. Create enrichment task.
• "not a claim" — is it an implementation idea? tension? validation? Those ARE extractable.

VALID Skip Reasons (rare)

• Completely off-topic (unrelated to llm-do)
• Too vague to act on (applies to everything, disagrees with nothing)
• Pure summary with zero extractable insight
• LITERALLY identical text already exists (not "same topic" — IDENTICAL)

For domain-relevant sources: skip rate < 10%. Zero extraction = BUG.

EXECUTE NOW

Target: $ARGUMENTS

Parse immediately:

• If target contains a file path: extract insights from that file
• If target contains --handoff: output RALPH HANDOFF block + task entries at end
• If target is empty: scan arscontexta/inbox/ for unprocessed items, pick one
• If target is "inbox" or "all": process all inbox items sequentially

Execute these steps:

1. Read the source file fully — understand what it contains
2. Source size check: If source exceeds 2500 lines, STOP. Plan chunks of 350-1200 lines. Process each chunk with fresh context. See "Large Source Handling" section below.
3. Hunt for insights that serve the domain (see extraction categories below)
4. For each candidate:

• Use mcp__qmd__vsearch with query "[claim as sentence]", collection="docs-notes", limit=5 to check for duplicates
• If duplicate exists: evaluate for enrichment or skip
• Classify as OPEN (needs more investigation) or CLOSED (standalone, ready)

1. Output extraction report with titles, classifications, extraction rationale
2. Wait for user approval before creating files
3. If --handoff in target: create per-claim task files, update queue, output RALPH HANDOFF block

START NOW. Reference below explains methodology — use to guide, not as output.

Processing Mode Detection

Check the task file (if invoked via queue) for processing_mode:

If no task file or no processing_mode field, default to extract.

Relevance Filter Preamble (applies to ALL modes)

Before extracting any claim or writing any analysis, apply this filter:

"Does this change how we think about or build this project?"

• YES → proceed with extraction/analysis
• NO → discard

This filter is the first gate for ALL content in ALL modes. It replaces the assumption that all domain-adjacent content is worth capturing.

For `extract` mode: Discard off-topic claims (existing category 10), but ALSO discard claims about third-party internals unless they directly inform our design. A claim about how another project implements its internals is not automatically relevant just because we use that project. Ask: "Does knowing this change what WE build?"

For `analyze` mode: The entire document is filtered through relevance. Don't summarize what the document proposes — surface only what matters for our project. The output is an impact analysis, not a summary.

Observation Capture (during work, not at end)

When you encounter friction, surprises, methodology insights, process gaps, or contradictions — capture IMMEDIATELY:

The handoff Learnings section summarizes what you ALREADY logged during processing.

Reduce

Extract composable notes from source material into docs/notes/.

Philosophy

Extract the REASONING behind what works, not just observations about what works.

This is the extraction phase of the pipeline. You receive raw content and extract insights that serve the vault's domain. The mission is building externalized, retrievable reasoning — a graph of atomic propositions that can be traversed, connected, and built upon.

THE CORE DISTINCTION:

The vault is not just an implementation. It is the articulated argument for WHY the implementation works.

THE EXTRACTION QUESTION:

• BASIC thinking: "Is this a standalone composable claim?"
• BETTER thinking: "Does this serve llm-do?"
• BEST thinking: "Would a future session benefit from this reasoning being a retrievable note?"

If YES -> extract to appropriate category (even if "we already know this") If NO -> skip (RARE for domain-relevant sources — verify it is truly off-topic)

THE RULE: Implementation without articulation is incomplete. If we DO something but lack a note explaining WHY it works, that articulation needs extraction.

Extraction Categories

What To Extract

See arscontexta/ops/derivation-manifest.md extraction_categories section

The structural invariant: Every domain's extraction has these universal categories regardless of domain:

IMPORTANT: Categories 1-9 bypass the selectivity gate. They extract directly to the appropriate output type. The selectivity gate exists ONLY for filtering off-topic content from general sources.

Category Detection Signals

Hunt for these signals in every source:

Core domain signals:

• Direct assertions: "the key insight is...", "this means that...", "the pattern is..."
• Evidence: "research shows...", "data indicates...", "studies confirm..."
• Named methods: any named system, technique, or framework relevant to llm-do

Comparison signals:

• "X vs Y", "trade-off between...", "prefer X when...", "unlike Y, this..."
• "choose X when...", "depends on whether..."

Tension signals:

• "contrary to...", "however...", "the problem with...", "fails when..."
• "on the other hand...", "but this conflicts with..."

Anti-pattern signals:

• "systems fail when...", "the anti-pattern is...", "avoid this because..."
• Warnings, cautionary examples, failure postmortems

Enrichment signals:

• Content covering ground similar to an existing note
• New examples, evidence, or framing for an established claim
• Deeper explanation of something already captured shallowly

Implementation signals:

• "we could build...", "would enable...", "a tool that...", "pattern for..."
• Actionable techniques, concrete workflows

Validation signals:

• "this supports...", "evidence shows...", "validates...", "confirms..."
• Research that grounds existing practice in theory

The Mission Lens (REQUIRED)

For EVERY candidate, ask: "Does this serve llm-do?"

• YES -> extract to appropriate category (gate does NOT apply)
• NO -> apply selectivity gate (for off-topic filtering only)

For domain-relevant sources: almost everything is YES. The gate barely applies. Skip rate < 10%.

The Selectivity Gate (for OFF-TOPIC content filtering)