drpwchen/textbook-to-note/textbook-to-md

Textbook-to-Markdown Converter

This skill calls scripts in your clone of the textbook-to-note repo. At

install time, replace {REPO} below with the absolute path of the clone.

Purpose

Convert PDF or EPUB textbooks into searchable markdown that the agent can grep/read directly, eliminating the need for PDF-library extraction at every query.

Output lives outside the note vault, at the path configured by OUTPUT_DIR in shared/config.py (default ./output/), and is for the agent's consumption, not for the user's reading.

Figures are on-demand, not pre-extracted. This skill produces markdown

text only. Figures are extracted one at a time when a note needs them,

via the figure-remap skill's entrypoint (QC-gated). Do not batch-extract

a whole book's figures into a figures/ folder — that approach does not

scale and is unnecessary since the on-demand path already handles it. A

legacy figures/ folder may exist for books converted before this design;

new conversions are markdown-only.

When to use

• User explicitly asks to convert a textbook or chapter
• You need to search textbook content and want to avoid per-query PDF

re-parsing overhead

• Building up a knowledge base from a personal library of reference books
• Before starting work on a new topic, convert the relevant chapters first

Quick reference

# Single file
python {REPO}/converter/convert.py "path/to/chapter.pdf"
 
# Single file with custom output and label
python {REPO}/converter/convert.py "path/to/chapter.pdf" "output.md" --book-label "Author Title 2e — Ch32"
 
# Batch-dir: convert ALL PDFs in a directory tree (auto chapter split + PDF bookmarks)
python {REPO}/converter/convert.py --batch-dir "path/to/your/textbook/folder"
 
# Force re-convert (ignore existing md)
python {REPO}/converter/convert.py --batch-dir "path/to/your/textbook/folder" --force
 
# Force OCR for ALL PDFs in batch-dir (bypass the text-extraction path entirely).
# Use when the text layer "looks" healthy but quality is actually bad (OCR-overlay
# scans, some digitized reprints) — auto-detection won't trigger because the text
# layer passes the shallow check.
python {REPO}/converter/convert.py --batch-dir DIR --force --force-surya
 
# EPUB → markdown (the 2nd arg is a FOLDER, not a .md file)
python {REPO}/converter/convert.py "path/to/book.epub" "<OUTPUT_DIR>/Author_Title_2e_2022"

Batch mode skips files whose markdown already exists and is newer than the source PDF. Batch-dir mode saves progress to batch_progress.json — if interrupted, re-running resumes where it stopped. --batch-dir also picks up .epub files automatically.

EPUB support

• Uses pandoc (epub → gfm). EPUBs are reflowable, so there are no

<!-- page N --> markers; files carry a <!-- SOURCE: epub --> marker instead.

• If pandoc emits proper heading markup, that drives the chapter split. If

the EPUB is CSS-styled with no semantic headings (common), headings are rebuilt from the TOC link table + body anchors before splitting.

• Output is a folder of chNN_*.md + full_text.md (same shape as a PDF

book) — the single-file 2nd argument is the output directory, not a .md path.

• Requires pandoc on PATH. Figures are not extracted from EPUB (the

on-demand figure flow is PDF-only).

Batch-dir features

• Recursively finds all PDFs, auto-outputs to <OUTPUT_DIR>/{PDF_stem}/

(OUTPUT_DIR from shared/config.py; default ./output)

• Produces full_text.md (complete with <!-- page N --> markers) plus a

chapter split

• Chapter-splitting priority: PDF bookmarks → pattern detection →

force-split (every 30 pages)

• Writes PDF bookmarks to source PDFs if none exist
• Auto-routes OCR-needed PDFs to the local OCR engine (GPU). Two

triggers: (1) a page is scan-only (near-zero extractable characters); (2) fitz-silent-failure is detected (see docs/ocr-ladder.md). Falls back to skip-with-explanation if the OCR environment is missing. Output is full_text.md with page markers, and a chapter split is attempted on the OCR'd text too (numbered-heading patterns often survive OCR); it is best-effort and simply yields no chapters when heading detection fails

Output structure

<OUTPUT_DIR>/                       ← default ./output (shared/config.py)
├── Author_Title_Edition_Year/
│   ├── ch01_Chapter_Title.md      ← conversion produces md only
│   ├── ch02_....md
│   ├── full_text.md
│   └── figures/                   ← LEGACY ONLY — not produced by conversion
└── Another_Book/                  ← single-file conversions use the same <book>/full_text.md layout

Figure-registry generation (figure_registry.json) is an optional external hook, not a shipped script — set the FIGURE_REGISTRY_SCRIPT env var to a generator script if you have one; if unset, post_convert.py skips that step.

==Conversion writes .md files only.== A figures/ subdirectory is legacy — present only on books converted before the on-demand switch. New conversions never create one; figures are pulled on demand by figure-remap.

What the converter produces

Each markdown file contains:

• Page markers: <!-- page X --> at every page boundary, to map content

back to the source PDF

• Cleaned text: control characters removed, broken/hyphenated words

rejoined, grep-searchable

• Tables: extracted as markdown tables (a low content-cell threshold is

used to avoid discarding real tables)

• Figure/table reference markers: when the text mentions a figure or

table (Fig. 32.1, Table 31.2, etc.), an HTML comment <!-- REF: Fig. 32.1 → see PDF page X --> is inserted so the agent knows where to look in the original PDF

Searching converted textbooks

Two search methods:

1. Keyword search (grep) — exact match, 0 tokens

grep -r "your search term" "$OUTPUT_DIR"      # default ./output

2. Semantic search (optional) — concept match

If you've set up the optional semantic index (LanceDB + a local embedding model — see docs/architecture.md), a textbook_search tool becomes available for concept-level queries that don't depend on exact wording.

Search strategy

• Known keyword → grep first (fastest, exact)
• Concept/topic exploration → semantic search (finds related content

even with different phrasing)

• Both → grep for precision, semantic search for coverage

Post-conversion pipeline

After converting new books, run the post-pipeline:

# Full pipeline: verify quality + refresh index.md + refresh figure registry + semantic index
python {REPO}/converter/post_convert.py
 
# Or individual steps:
python {REPO}/converter/post_convert.py --verify   # check quality only
python {REPO}/converter/post_convert.py --index    # semantic index only
python {REPO}/converter/post_convert.py --audit    # report index coverage (no backfill)

The pipeline does not extract figures — it only verifies page markers, refreshes index.md + figure_registry.json from whatever figures already exist, and (optionally) builds the semantic index. Figure extraction is never a batch post-step.

Missing page markers do not block indexing by design — EPUBs (reflowable) and some vector-glyph PDFs legitimately lack <!-- page N -->; the indexer handles this gracefully (page metadata = 0). After indexing, a coverage audit compares the markdown corpus against the index and auto-backfills any book that has markdown but no index rows, so every converted book ends up searchable even if a step was skipped.

Figures — do NOT batch-extract

No pre-extraction step. A whole-book batch figure dump is retired and should not be run on new books. When a note needs a figure, figure-remap extracts that single figure on demand with QC. See "Using figures in notes" below.

Chapter splitting (batch mode)

Chapter detection patterns (in priority order)

1. Chapter N / CHAPTER N — standard
2. Part N / PART N — with roman numerals (Part III)
3. Section N / SECTION N / Unit N
4. Numbered headings: 1 Introduction, 23 Shoulder (digit + title-case

text)

Running header deduplication

Many textbooks repeat Chapter N Title as a running header on every page. Only count the first occurrence of each unique chapter number — track a seen_chapters set and skip duplicates.

Force-split rules

• No chapter breaks detected AND >200 pages → split every 30 pages

(pages_0001-0030.md)

• Single output file >500K words → re-split (likely missed chapter breaks)

Output naming

• Folder name = PDF filename without .pdf (use an

Author_Title_Edition_Year convention)

• Chapter files: ch01_Chapter_Title.md, ch00_Front_Matter.md
• Forced splits: pages_0001-0030.md

Quality verification

After conversion, check:

• Total words < 500 for a book → likely scanned; batch-dir auto-routes

to OCR. If it ran the text-extraction path anyway, the OCR-trigger heuristic missed and needs tuning

• Single file > 500K words → chapter detection failed, re-split needed
• 0 chars extracted from sample pages → pure scan; batch-dir handles

this automatically. Single-file mode does not auto-route — use batch-dir on a temp folder for OCR needs

• Garbled glyphs (high char count, mostly private-use-area codepoints)

→ CID-encoded font without a Unicode map; detected by the silent-failure check, auto-routes to OCR in batch-dir

Always cite with year when referencing converted content: (Author 2e, 2021), not just (Author).

Using figures in notes (on-demand)

When writing or supplementing a note that needs a figure (anatomy, classification, imaging, algorithm), extract that one figure on demand — never batch-dump a book. The markdown's <!-- REF: Fig. 5.1 → see PDF page 42 --> markers tell you which figure exists and what PDF page it's on; hand that to the gate.

On-demand workflow — single entry point

Call the figure-remap skill's public entrypoint (figure_remap.py extract — not the internal gate script). The entrypoint runs deterministic geometric matching by default and returns a stable contract:

python {REPO}/figures/figure_remap.py extract \
  --book "{Book}" \
  --fig-id "5-1" \
  --caption "<caption text from the REF marker / md>" \
  --out "path/to/your/vault/attachments/Fig_5-1_{BookShort}.jpeg" \
  --pdf "<source PDF path>" \
  --page {1-indexed PDF page from the REF marker}

Contract: {status: pass|fail|escalate, match_quality: exact|uncertain|failed, hard_fail, file, fig_id, reason, qc_degraded, qc_skipped} — those eight keys exactly (figures/figure_remap.py CONTRACT_KEYS; the validator raises on any extra or missing key, so branching on the engine's internal match_method is not just discouraged, it is impossible). status:fail (exit 1) is a deterministic miss — a correct refusal, not a wrong crop; fix --page, escalate to vision, or leave a <!-- TODO -->. Read the real caption to confirm the figure depicts what you intend.

On pass (exit 0), embed the --out path (result.file) in the note:

![[Fig_5-1_{BookShort}.jpeg|400]]
*Fig 5.1 — description (Author 2e, p.42)*

When writing a fresh note via the note-writing workflow (see workflows/note-writing.md), figure harvest is Phase 3.5 — don't call the gate manually there, the workflow does it. The manual call above is for ad-hoc figure needs outside that workflow. Full fallback ladder + per-book calibration: see the figure-remap skill.

When to include figures

• Always: anatomy diagrams, classification systems, algorithm

flowcharts, key reference images

• Skip: decorative images, author photos, generic stock photos
• Ask the user if unsure whether a figure adds value