seanpedrick-case/doc_redaction/doc-redaction-modifications

Goal

Repeatable review → edit CSV → preview → `/review_apply` → download → verify when you already have the original PDF and a matching *_review_file.csv.

Initial redaction only: `../doc-redaction-app/SKILL.md`.

Parallel page-range review (subagents): `../doc-redact-page-review/SKILL.md` — one child per page for Pass 1, parent merges and applies once; Pass 2 (optional VLM) runs after Pass 1 outputs exist.

Two-pass review model

Review is split into a fast, text-based first pass (default) and an optional visual second pass. Do not run VLM on every page unless the user explicitly requests Pass 2 or you are re-checking specific flagged pages.

Default workflow: Pass 1 only → deliver. Run Pass 2 only when needed.

Endpoint semantics (do not get this wrong)

Never implement a custom “true redaction” PyMuPDF script because text_layer_leaks appeared. Fix CSV coverage and coordinates first, then call /review_apply again. Exception: reading PyMuPDF word positions to add normalized CSV rows is allowed — writing the final PDF without /review_apply is not (see task template § Agent anti-confusion rules).

Never run post-apply text-layer checks on *_redactions_for_review.pdf — that file retains text for human review. /review_apply returns both files; only *_redacted.pdf is the deliverable.

Use `tools/verify_redaction_coverage.py` or POST /agent/verify_redaction_coverage — do not reimplement coverage logic ad hoc (pandas/regex scripts). On split-container Pi deployments, pre-apply checks use the CLI on downloaded artifacts (edited CSV is local); post-apply checks use the Agent API with server paths — see § Split-container verify below.

Text-layer leak troubleshooting

When text_layer_leaks appear on *_redacted.pdf:

1. Confirm the PDF basename ends with _redacted.pdf (not _redactions_for_review.pdf).
2. Pre-apply: all xmin/ymin/xmax/ymax must be normalized 0–1 (df[bbox_cols].max().max() <= 1). Never paste PyMuPDF absolute points without dividing by page width/height.
3. Read leak_likely_causes on each page in the coverage report:

Word OCR can show 100% covered while the text layer still leaks — that is usually coordinates or image text, not a broken apply endpoint.

Pass 1 — OCR / CSV / text review

Use artefacts from the same redaction run. No VLM in this pass.

Inputs (discover in output folder)

Pass 1 loop (per page or whole document)

1. Load *_review_file.csv (encoding="utf-8-sig").
2. Policy edits — remove false positives, add missing PII rows, relabel (programmatically, not Excel-only).
3. Word OCR — match page + word_text; merge words on the same line (|Δy0| < ~0.01); separate boxes across lines.
4. Line OCR — use line CSV for phrase context, line numbers, and confirming reading order when word boxes fragment a name or address.
5. Coverage report (mandatory before apply) — run `tools/verify_redaction_coverage.py` or `POST /agent/verify_redaction_coverage` with must_redact / must_not_redact regex lists. Fix policy flags (uncovered_terms, over_redacted, text_layer_leaks); re-run until pass_strict is true.
6. Suspicious-row prune (standard Pass 1 cleanup) — remove short OCR-fragment boxes ("-", ".", "Ho", etc.) that do not match must_redact. CLI: --prune-suspicious --pruned-output merged_pruned.csv or API: auto_prune_suspicious: true. Re-run coverage; target pass_with_cleanup: true.
7. Preview — preview_redaction_boxes or /preview_boxes on edited CSV (spot-check worst pages from the report).
8. Merge full-document CSV (all pages) if reviewing a subset — see page-review skill.
9. One /review_apply → download newest *_redacted.pdf / *_review_file.csv (sort by st_mtime).
10. Coverage report (after apply) — re-run with redacted_pdf_path for text-layer leak checks; optional sample_pixels=true.
11. Term search (optional) — POST /agent/word_level_ocr_text_search or word_level_ocr_text_search to find policy phrases in word OCR and whether each hit is boxed.

Pass 1 is complete when pass_strict is true (policy satisfied). `pass_with_cleanup` also requires no suspicious short rows. Run Pass 2 VLM only on `pages_flagged_for_vlm` (policy/visual risk — not pages_needing_csv_cleanup alone).

Automatic post-redaction QA (optional — main app)

When POST_REDACT_PASS1_QA=True (`tools/config.py`), initial redaction (Gradio / CLI / /doc_redact) runs `tools/post_redaction_pass1_qa.py` after writing *_review_file.csv:

• Emits *_coverage_report.json next to the review CSV
• Optionally emits sibling *_review_file_pruned.csv when POST_REDACT_PASS1_AUTO_PRUNE=True (does not replace the original CSV)
• Maps run deny list → must_redact, allow list → must_not_redact when POST_REDACT_PASS1_USE_DENY_ALLOW_LISTS=True
• Appends a one-line QA summary to the redaction status message

This is pre-review-apply deployment QA only. Agent Pass 1 (policy edits, merge, /review_apply, post-apply coverage) is still required for case-specific review workflows.

Coverage verification (Pass 1 — no VLM)

Programmatic QA replacing per-page visual review for most cases.

CLI:

python tools/verify_redaction_coverage.py merged_review_file.csv ocr_words.csv \
  --must-redact "cora|fuller|fyller" \
  --must-not-redact "dr\\.|macrae|gibson|social worker" \
  --prune-suspicious --pruned-output merged_pruned.csv \
  --redacted-pdf output_redacted.pdf \
  --output-json coverage_report.json

Report fields:

Agent API: POST /agent/verify_redaction_coverage

{
  "review_csv_path": "path/to/doc_review_file.csv",
  "ocr_words_csv_path": "path/to/doc_ocr_results_with_words_local_ocr.csv",
  "must_redact": ["cora|fuller|fyller", "stephen|peter|rhett|yazmin"],
  "must_not_redact": ["dr\\.|doctor|social worker|macrae|gibson"],
  "redacted_pdf_path": "path/to/doc_redacted.pdf",
  "auto_prune_suspicious": true,
  "pruned_output_path": "path/to/doc_review_file_pruned.csv",
  "sample_pixels": false
}

Response includes coverage_pass_strict, coverage_pass_with_cleanup, pruned_csv_path, prune_log, per-page issues, and pages_flagged_for_vlm vs pages_needing_csv_cleanup.

Split-container verify (Pi agent + separate redaction service)

When the Pi agent and doc_redaction run in separate containers (e.g. http://redaction:7860), path validation in secure_path_utils runs on the redaction server. Agent API paths must resolve under repo root, INPUT_FOLDER, or OUTPUT_FOLDER on that server.

Rejected paths (common mistakes):

• Pi workspace paths (e.g. /home/user/app/workspace/sess/...)
• /tmp/gradio_tmp/... from /gradio_api/upload (not under OUTPUT_FOLDER)
• Calling verify_redaction_coverage() from the Pi container with redaction-server path strings

Word search: POST /agent/word_level_ocr_text_search with ocr_words_csv_path, search_text, optional review_csv_path.

covered_by_review_box uses intersecting review boxes (not strict containment). A hit marked false may still be visually redacted if a larger box overlaps — inspect coordinates before adding rows.

Python:

from doc_redaction import verify_redaction_coverage, word_level_ocr_text_search
report = verify_redaction_coverage(
    "doc_review_file.csv",
    "doc_ocr_results_with_words_local_ocr.csv",
    must_redact=[r"cora|fuller"],
    must_not_redact=[r"dr\."],
    redacted_pdf_path="doc_redacted.pdf",
)
hits = word_level_ocr_text_search(
    "doc_ocr_results_with_words_local_ocr.csv",
    "Fuller",
    review_csv_path="doc_review_file.csv",
)