seanpedrick-case/doc_redaction/doc-redaction-app

Scope

This skill covers running an initial redaction and getting artifacts onto the client. For review after a run: Pass 1 (OCR/CSV edits, /review_apply) and optional Pass 2 (visual VLM) — `../doc-redaction-modifications/SKILL.md`. For parallel Pass 1 page review (subagents, merge, one apply), use `../doc-redact-page-review/SKILL.md`.

Starting a new task from chat: copy `../doc-redaction-task-prompt/TASK_PROMPT_TEMPLATE.md`, fill setup placeholders, and put case-specific must redact / must not redact rules in the User redaction requirements section at the end.

Quick start

1) Access path

• Primary: gradio_client
• Fallback: raw /gradio_api/* HTTP
• /agent/* only when inputs resolve on the server (repo root, INPUT_FOLDER, or OUTPUT_FOLDER). If you see path validation errors, the client and app do not share a filesystem—use Gradio upload paths or MCP, not bare agent paths.

2) Prefer /doc_redact over /redact_document

Use /doc_redact for a normal PDF/image first pass. Use /redact_document only when you need the full Gradio control surface. Note that the /doc_redact endpoint may not be visible when calling client.view_api().

2b) /doc_redact empty path list

If the API returns `[]` with "doc_redact completed" but the server log shows Generated Files, common causes are: (1) `SESSION_OUTPUT_FOLDER=True` and a re-run overwrote the same filenames (fixed in recent builds via mtime-based path return); (2) path collectors that ignore Windows C:\... paths. Fallback on shared-disk deployments: glob OUTPUT_FOLDER (including per-user subfolders) for *{document_stem}*, or use remote_redaction.resolve_redaction_output_paths. Do not pass a Pi workspace path as output_dir — only directories under the app OUTPUT_FOLDER are accepted.

2f) AWS ECS / split-container Pi (no shared disk with doc_redaction)

When AGENT_DEPLOYMENT_PROFILE=aws-ecs (or DOC_REDACTION_GRADIO_URL points at an internal service like http://redaction:7860), the Pi agent and doc_redaction run in separate containers.

Split-container verify playbook: pre-apply checks run on local copies of the edited review CSV (it only exists in your session workspace until /review_apply). Post-apply checks use the Agent API on the redaction service with paths that already exist under /home/user/app/output/... on that server. See `../doc-redaction-modifications/SKILL.md` § Split-container verify.

Helper (synced into the workspace): {AGENT_WORKSPACE_DIR}/.pi/helpers/remote_redaction.py. When predict returns [], retry once and parse the status message for paths — do not grep the Pi container filesystem.

2c) /doc_redact vs /redact_document parameter names

They are not interchangeable. Wrong kwargs raise errors such as Parameter is not a valid keyword argument.

• /doc_redact: e.g. document_file, ocr_method, pii_method, redact_entities, handwrite_signature_checkbox (not file_paths, chosen_local_ocr_model, chosen_redact_entities, custom_llm_instructions).
• /redact_document: long-form names (file_paths, chosen_redact_entities, chosen_local_ocr_model, pii_identification_method, etc.). Use client.view_api() when in doubt.

2d) OCR / PII labels on /doc_redact

• ocr_method: high-level modes (Local OCR, AWS Textract, Local text) plus engine shortcuts (tesseract, paddle, hybrid-paddle-vlm, vlm, inference-server, …). Aliases like textract, local often work.
• pii_method: e.g. Local, AWS Comprehend, LLM (AWS Bedrock), Local inference server, Local transformers LLM, None. Prefer exact labels from the deployment.

Optional page window (when exposed): page_min / page_max (1-based; 0 often means first/last page—confirm in view_api()).

AWS Textract extraction (handwrite_signature_checkbox): multiselect list passed when using AWS Textract (or hybrid Textract routes). Values must match the deployment UI labels exactly.

Base deployments expose only the first two (HANDWRITE_SIGNATURE_TEXTBOX_FULL_OPTIONS default). Extra rows appear when the server config sets INCLUDE_FORM_EXTRACTION_TEXTRACT_OPTION, INCLUDE_LAYOUT_EXTRACTION_TEXTRACT_OPTION, INCLUDE_TABLE_EXTRACTION_TEXTRACT_OPTION, or INCLUDE_FACE_IDENTIFICATION_TEXTRACT_OPTION to True. Confirm with the UI “AWS Textract extraction settings” checkbox group or GET /gradio_api/info.

Example:

client.predict(
    api_name="/doc_redact",
    document_file=handle_file("/local/path/document.pdf"),
    ocr_method="AWS Textract",
    handwrite_signature_checkbox=["Extract handwriting", "Extract signatures"],
)

When omitted, server/CLI defaults apply (DEFAULT_HANDWRITE_SIGNATURE_CHECKBOX, often []).

2e) PII entity lists — Local vs AWS Comprehend

Which parameter to use depends on `pii_method`:

/doc_redact maps redact_entities → CLI local_redact_entities only. For Comprehend entity selection, call /redact_document (or cli_redact.py --aws_redact_entities …).

Local PII entities (FULL_ENTITY_LIST in tools/config.py; deployment may extend via env):

TITLES, PERSON, PHONE_NUMBER, EMAIL_ADDRESS, STREETNAME, UKPOSTCODE, CREDIT_CARD, CRYPTO, DATE_TIME, IBAN_CODE, IP_ADDRESS, NRP, LOCATION, MEDICAL_LICENSE, URL, UK_NHS, CUSTOM, CUSTOM_FUZZY

When VLM face/signature detection is enabled on the deployment, the list may also include: CUSTOM_VLM_FACES, CUSTOM_VLM_SIGNATURE.

Textract + `CUSTOM_VLM_SIGNATURE`: when ocr_method is AWS Textract and handwrite_signature_checkbox includes Extract signatures, Textract signature analysis runs and inline/post-pass `CUSTOM_VLM_SIGNATURE` VLM detection is skipped (no duplicate signature finding). Keep CUSTOM_VLM_SIGNATURE in the entity list if you want those regions redacted; Textract supplies the boxes. Without Extract signatures, CUSTOM_VLM_SIGNATURE still uses VLM as before.

Default selection (CHOSEN_REDACT_ENTITIES): TITLES, PERSON, PHONE_NUMBER, EMAIL_ADDRESS, STREETNAME, UKPOSTCODE, CUSTOM.

AWS Comprehend entities (FULL_COMPREHEND_ENTITY_LIST):

BANK_ACCOUNT_NUMBER, BANK_ROUTING, CREDIT_DEBIT_NUMBER, CREDIT_DEBIT_CVV, CREDIT_DEBIT_EXPIRY, PIN, EMAIL, ADDRESS, NAME, PHONE, SSN, DATE_TIME, PASSPORT_NUMBER, DRIVER_ID, URL, AGE, USERNAME, PASSWORD, AWS_ACCESS_KEY, AWS_SECRET_KEY, IP_ADDRESS, MAC_ADDRESS, LICENSE_PLATE, VEHICLE_IDENTIFICATION_NUMBER, UK_NATIONAL_INSURANCE_NUMBER, INTERNATIONAL_BANK_ACCOUNT_NUMBER, SWIFT_CODE, UK_NATIONAL_HEALTH_SERVICE_NUMBER, ALL, CUSTOM, CUSTOM_FUZZY

Default selection (CHOSEN_COMPREHEND_ENTITIES): EMAIL, ADDRESS, NAME, PHONE, PASSPORT_NUMBER, UK_NATIONAL_INSURANCE_NUMBER, UK_NATIONAL_HEALTH_SERVICE_NUMBER, CUSTOM.

Do not mix label namespaces: Local uses PERSON / EMAIL_ADDRESS; Comprehend uses NAME / EMAIL. Pick the list that matches your pii_method.

Custom Presidio recognizers (Local and Comprehend)

These are not generic spaCy NER labels — they are app-specific Presidio recognizers (tools/load_spacy_model_custom_recognisers.py). They appear in both Local and Comprehend entity dropdowns because CUSTOM_ENTITIES is merged into the Comprehend lists (Comprehend can miss titles, UK postcodes, and street fragments).

Default deployment config (CUSTOM_ENTITIES): TITLES, UKPOSTCODE, STREETNAME. Admins may extend via the CUSTOM_ENTITIES env var.

On `/doc_redact`: include in redact_entities. On `/redact_document`: include in chosen_redact_entities (Local) or chosen_redact_comprehend_entities (Comprehend — these three labels are valid there too).

CUSTOM and CUSTOM_FUZZY (deny-list driven)

Both routes support explicit term lists via `deny_list` / `allow_list` on /doc_redact, or deny/allow list files on /redact_document / CLI.

`/doc_redact` inline lists: pass terms directly as deny_list=["term1", "term2"] (not a file path). When deny_list is non-empty, the API auto-appends `CUSTOM` to redact_entities if missing (deny-list matching requires that entity type). If you omit redact_entities entirely, CLI defaults already include CUSTOM.

For fuzzy deny-list matching via API, include CUSTOM_FUZZY in redact_entities explicitly only when the user explicitly requests fuzzy matching. Otherwise prefer CUSTOM with an explicit deny_list on /doc_redact, and use /redact_document with max_fuzzy_spelling_mistakes_num > 0 if you need fuzzy deny-list matching.

⚠️ Warning: on /doc_redact, CUSTOM_FUZZY is expensive and may cause the endpoint to return an empty path list even when the redaction job completes. Use it only when the user asks for typo-tolerant deny-list redaction.

allow_list excludes terms from redaction even when another entity type would match.

Example — Local /doc_redact with UK-specific recognizers + custom deny terms: