carloscape/octorato/multi-specialist-doc-audit

Multi-Specialist Doc Audit

What

A workflow for auditing a long technical document (proposal, RFC, evaluation, RAG output) for hallucinations and unsourced claims. Instead of one agent re-reading the whole thing, dispatch N specialist agents in parallel — each scoped to its domain — and consolidate their findings into one severity-tiered audit report.

Why

A single-agent audit on a 1000+ line document either:

• runs out of attention (skims, misses domain-specific errors), or
• spends most of its time re-reading content outside its specialty.

A multi-specialist parallel audit:

• catches more domain-specific errors (a Compliance Auditor flags BAA wording a generalist would miss; a Security Engineer flags an OAuth scope name that doesn't exist; a Market Research Analyst catches an outdated vendor pricing claim)
• runs in parallel — the wall-clock cost is one agent's time, not N agents in series
• produces consistent severity tiers across all reviewers, so the consolidator can prioritize fixes
• defaults to "find the issues" rather than "approve the doc" — every specialist enters with a "NEEDS WORK" prior

When to Use

• Doc length ≥ 500 lines AND mixes author opinion with vendor / regulatory / technical claims
• Doc is going to an external audience (client, regulator, leadership) where credibility cost is high
• Doc claims metrics, vendor capabilities, regulatory references, or auth/security details that can be objectively checked

Do NOT use for:

• Internal scratchpad docs
• Doc < 200 lines (one agent is fine)
• Doc that is purely subjective (e.g., a vision statement)

Workflow

1. Map domains to specialists

Identify the 3–6 distinct claim domains in the doc. Common pattern:

Do not dispatch a specialist for a domain the doc doesn't actually cover. One unnecessary agent dilutes the consolidated report.

2. Brief each specialist with three things

Each agent gets:

1. Scope — exactly which sections / claim types they own (and which they should NOT verify)
2. Sources of truth — list of [KB] paths, [vendor:URL]s, captured artifacts, git log, etc.
3. Output format — the severity table (below)

Keep prompts under ~600 words each. Specialists do better with focus than with context.

3. Severity tier table (mandatory output format)

Every specialist returns a table:

Severity values are exactly four:

Add NARRATIVE for retrospective/framing sections that aren't falsifiable but acceptable as labeled narrative.

4. Dispatch in parallel

Make all specialist calls in a single tool-use round so they run concurrently. The wall-clock cost is one agent's time, not N agents serialized.

5. Consolidate

The orchestrator receives N reports and produces:

1. A unified severity table (worst first), grouped by tier
2. A count: X HALLUCINATION / Y UNCITED / Z ASSUMPTION / W VERIFIED
3. A recommendation:

• Surgical fix if HALLUCINATION count is small (≤5 over a 1000-line doc)
• Tier 2 fix if UNCITED count is large but no HALLUCINATIONS — label assumptions, cite the rest
• Full rebuild if HALLUCINATION count is high or specialists disagree on basic facts

6. Present and confirm

Show the consolidated table to the user. Recommend a path. Wait for user confirmation before applying fixes.

Anti-patterns

Composability

• source-citation-tagging — pre-tagging makes the audit cheaper; specialists can grep tags and verify each
• peer-review-lifecycle (existing) — multi-specialist audit feeds the peer-review v1.0 → v2.0 cycle
• document-code-review (existing) — adds 9 mechanical defect dimensions; runs in addition to the multi-specialist audit, not instead
• gap-analysis-pattern (existing) — useful for the "what's missing" complement to "what's wrong"

Lessons Learned

• Five specialists in parallel on a 1500-line proposal doc found 15 HALLUCINATIONs in ~10 minutes wall-clock. A single Reality Checker would have taken longer and missed at least 5 of them (the ones in domains it doesn't specialize in).
• Specialist briefs that say "do NOT verify X (other agent has that)" are critical. Without scope boundaries, two specialists waste tokens on the same claim and miss their actual scope.
• The HALLUCINATION/UNCITED/ASSUMPTION distinction is what makes the report actionable. "Issues found: 23" is useless. "5 HALLUCINATIONs, 8 UNCITED, 10 ASSUMPTIONS" tells you exactly what to fix and in what order.
• Always recommend surgical / tier-2 / full-rebuild as the THREE options, then let the user pick. Don't auto-rebuild on a small number of HALLUCINATIONs.