undefined-ui/second-brain-os/second-brain-backfill

Backfill an archive

A backfill is where cost and quality both go wrong quietly. Done in the wrong order it also inverts the vault's history, because later sources should update pages that earlier ones created.

Core rule

Oldest first, ten sources per batch, stop for a go-ahead between batches.

Workflow

1. Triage before ingesting. Read titles and first paragraphs. Sort into

worth-ingesting, keep-in-raw, and delete. Most archives are half dead links and things saved but never opened.

1. Order by date, oldest first. Run it the other way and every old source

arrives as a contradiction against a page that already has the final answer.

1. Batch of ten. Ingest following second-brain-ingest.
2. Checkpoint: update index.md, append to log.md, commit with the batch

range in the message, report counts.

1. Stop and wait before the next batch.
2. Full lint at the end. Large imports always leave orphans and duplicates.

Output format

Batch <n>/<total>: <range>
Ingested: <n> sources -> <n> new pages, <n> updated, <n> links
Skipped: <n> (reason)
Running total: <n> of <n>
Next batch: <what it contains>

Calibration

Read every page the first batch produced, all of them. A flaw you let through in batch one is a flaw in four hundred pages by the end, and fixing it afterwards costs more than the original run.

If the archive is over about two hundred sources, propose a filter before starting rather than quoting the full cost.