smontlouis/bible-strong/bible-to-strong

Bible To Strong

Scope

Use this skill only for explicitly requested execution of the deterministic Strong-Bible pipeline for a local version such as nbs, s21, bds, or fmar. Do not load it merely because a conversation mentions Strong numbers, Bible formats, STEP, occurrences, SQLite, JSON, production releases, or mobile integration.

Expected input:

• Bible JSON: data/bibles/bible-<id>.json
• Strong references: data/strongs/Sg1910.csv, data/strongs/Darby.csv, data/strongs/DarbyR.csv
• STEP original sources: data/external/stepbible/amalgamated/TAHOT *.txt and TAGNT *.txt
• Optional audit/provenance sources: data/external/Alignments/data/sources/WLC.tsv and SBLGNT.tsv
• Local lexicons for review and future deterministic evidence: data/dictionaries/strong_fr.sqlite and data/dictionaries/strong_lexicon.full.production.sqlite

Generated full Bible outputs under outputs/ are ignored by Git and must not be committed. The canonical production ledger is now SQLite-first: outputs/strong/<id>/bible-<id>-strong.sqlite.

The canonical ledger and the downloadable Bible are deliberately different artifacts. The ledger keeps annotations, evidence, metrics, views, and audit provenance. A downloadable Bible must be projected from that ledger into a small verse-per-line JSONL artifact; never ship the large authoring ledger or its debug JSON as the product payload.

Default Workflow

1. Confirm the requested Bible id and input file.
2. Generate the canonical Strong ledger. This is the no-LLM production path with reader and advanced views. It also applies the validated deterministic lexical auto-safe layer, then writes the residual lexical candidate report for review:

npm run strong:generate -- --bible <id>

Use the SQLite output as the authoritative production artifact when the user asks for the best workflow. It preserves a complete Strong ledger while keeping reader-visible tags profile-aware. Auto-safe lexical placements are already inserted with source="semantic-lexicon"; they should not remain as residual auto-safe candidates. Do not rely on legacy split verses/*.json files; new generation removes them.

Production full-Bible generation is validated with the SQLite/indexed path. On 2026-07-10, the final full NBS regeneration completed in 156.32 seconds with about 4.33 GB max RSS after streaming the lexical report and releasing consumed source caches before SQLite serialization. It wrote the canonical SQLite ledger and left Auto-safe candidates: 0, Auto-safe items: 0, and Group auto-safe items: 0 in outputs/lexical-candidates/nbs/bible-nbs-lexical-candidates-all.md. The lexical auto-safe loop is intentionally incremental after the first full pass, but it must finish with a full-scope confirmation pass that applies zero placements before the residual lexical report is trusted. If a run leaves any residual auto-safe item, treat generation as incomplete and investigate before using the Bible as a production artifact.

The duplicate lexical group auto-safe rule must continue processing all eligible groups in a report. A regression where that loop returned after the first resolved duplicate group caused slow global cascades; do not reintroduce early returns there.

If an existing output folder still has the old split JSON ledger, migrate it once:

npm run strong:migrate:sqlite -- --bible <id>

This writes bible-<id>-strong.sqlite, keeps metrics/TSV outputs, and removes legacy ledger JSON directories after a successful migration.

For localized deterministic fixes, refresh only the affected books, chapters, or ranges instead of regenerating the whole Bible:

npm run strong:refresh -- --bible <id> --only <Book>
npm run strong:refresh -- --bible <id> --only <Book.Chapter>
npm run strong:refresh -- --bible <id> --only <BookA-BookB>
npm run strong:refresh -- --bible <id> --only <Book1>,<Book2>,<Book3>

strong:refresh generates each requested scope in a temporary output directory, then replaces only those verses in the SQLite ledger. It recalculates affected book metrics, global metrics, and reader/advanced TSV exports without parsing or rewriting a split full-Bible JSON ledger. Use a full strong:generate only when the entire corpus must be rebuilt or when no canonical SQLite ledger exists yet. Every full ledger records a content fingerprint over its Bible, references, STEP sources, dictionary/index files, translation profile, and pipeline code. Scoped refresh must refuse a missing/mismatched fingerprint. If curated decisions changed, every changed override ref must be inside the refresh scope.

Build the local Kaikki SQLite index once when lexical reports need maximum speed:

npm run strong:kaikki:index

When data/external/french-lexical/kaikki/kaikki.org-dictionary-French.sqlite exists, lexical candidate generation uses targeted SQLite lookups instead of streaming the 534 MB JSONL file.

Build the local Strong phrase-lexicon SQLite index once before production generation or repeated scoped refreshes:

npm run strong:phrase:index

When data/derived/strong-phrase-lexicon.sqlite exists and its source fingerprint matches Sg1910, Darby, and DarbyR, generation loads learned multi-word phrase candidates from SQLite instead of rebuilding that large in-memory lexicon for every run.

1. Export reader or advanced views from the canonical ledger when a TSV is needed:

npm run strong:export -- --bible <id> --view reader
npm run strong:export -- --bible <id> --view advanced

3a. Publish a compact STEP-aware JSONL Bible

After the canonical ledger passes its quality and integrity gates, create the downloadable Bible as a separate projection. This applies both to the three reference witnesses and to generated Bibles such as NBS.

For the existing reference witnesses, the compact STEP-aware generation is:

npm run strong:references:jsonl
npm run strong:references:jsonl:dstrong

For generated Bibles, publish directly and incrementally from outputs/strong/<id>/bible-<id>-strong.sqlite. Do not first materialize or parse a gigabyte-scale JSON ledger. Stream the SQLite verse rows in canonical reference order and write one JSON object per line. The default downloadable projection is the reader view; an advanced/debug projection must be a separate explicit artifact and must not silently replace the reader product.

npm run strong:jsonl -- --bible <id> --version <VERSION>

This writes outputs/strong-jsonl/<id>/bible-<id>-strong.jsonl and an immutable sibling manifest.json. Use --only <Book> or --only <Book.Chapter> only for a bounded validation preview, normally with a separate --output-dir; a scoped artifact is not a publishable full Bible. The command refuses to overwrite either artifact.

When the user explicitly prefers recall over conservative placement, generated Bibles also support a deterministic permissive projection:

npm run strong:jsonl -- \
  --bible <id> \
  --version <VERSION> \
  --view permissive

This mode is for the five generated/non-witness Bibles ost, fmar, nvs78p, neg79, and nbs. It must not be applied to the reference witnesses darby, darbyr, or sg1910: those sources already carry their own authored Strong placement and remain unchanged.

The permissive publisher reads outputs/strong/<id>/bible-<id>-strong-permissive-plan.json, verifies that the plan fingerprint matches the SQLite ledger, and projects both reader and advanced annotations. For empty or weakly placed occurrences it applies the plan's stable best carrier when the lexical score reaches 0.48; a relocation must improve the current carrier by at least 0.12. Ties are resolved by the recorded deterministic carrier order. Exact normalized Strong dictionary terms are eligible even when the generic lexical candidate report omitted a proper name. Cases below the threshold remain empty.

The permissive projection is derived only: it never mutates the canonical SQLite annotation, an approved LLM payload, or the conservative reader JSONL. If the same Strong is already present on the selected carrier, the projected occurrence remains empty instead of creating another identical placement. The final compact strong attribute is an ordered set: duplicate values such as strong="H1961 H1961" are serialized once, while different Strong identities on the same French carrier are all retained. The manifest records planned and applied promotions, skipped duplicate carriers, removed redundant values, and remaining empty tags.

This writes outputs/strong-jsonl-permissive/<id>/bible-<id>-strong.jsonl. Publication must still preserve every verse and meaningful inline tag, including notes and divineName, and must still require exact STEP occurrence evidence before emitting estrong, dstrong, or ustrong.

Use the same JSONL verse contract as the reference witnesses:

{
  "ref": "Gen.1.1",
  "version": "NBS",
  "book": 1,
  "bookId": "Gen",
  "chapter": 1,
  "verse": 1,
  "text": "..."
}

The text field keeps meaningful inline Bible markup such as <p>, <note>, <i>, and <divineName>, but strips authoring/debug attributes. Strong and STEP identities belong to the <w> occurrence in the verse:

<w strong="H1254" estrong="H1254a" dstrong="H1254A">créa</w>
<w strong="G1138" ustrong="H1732">David</w>

Compact identity rules:

• always keep strong as the backward-compatible identifier carried by the

Bible word;

• add estrong only when the exact STEP extended identity is not already

represented by strong;

• add dstrong only when the exact STEP disambiguated identity adds precision

beyond estrong/strong;

• add ustrong only when it adds a distinct unified/group identity useful for

alternate forms, spellings, names, Hebrew/Aramaic equivalents, or navigation;

• preserve STEP suffix case exactly: H2148V and H2148v are distinct;
• omit redundant attributes instead of repeating the same value;
• never add inline confidence, method, source, status, or diagnostic

data-* attributes to the downloadable artifact. Put provenance, counts, unresolved cases, source hashes, and methods in a sibling manifest/report.

For generated Bibles, prefer the ledger's exact originalOccurrenceId/STEP-evidence link for each annotation. This is stronger than re-running verse-order heuristics on already generated markup. Resolve eStrong and uStrong through the exact dStrong entry in TBESH.txt or TBESG.txt. When exact occurrence evidence is missing or ambiguous, retain the classical strong unchanged and report the case outside the Bible; never guess an estrong, dstrong, or ustrong during publication.

These attributes normally describe one lexical occurrence at different levels, not four competing analyses. Product lookup should generally use dstrong ?? estrong ?? strong as the primary lexical target. ustrong is a grouping/navigation identity. A French carrier may occasionally represent multiple original occurrences; then the values on <w> form an occurrence-level set. Do not infer positional pairing between whitespace lists unless a future serialization contract explicitly records that pairing; retain the canonical ledger when exact per-occurrence relationships are needed.

Publication validation must fail closed unless all of the following hold: