drpwchen/openevidence-tools/oe-verify

/oe-verify — OpenEvidence Citation Verification

Derived from audit-oe-skill,

MIT © 2026 Hsieh-Ting Lin. The origin ROT-1 decode, the corpus→risk tiers, the

transitive-citation framing and trace-back stage, the parallel-per-citation +

single-cross-citation agent architecture, and the CRITICAL/WARNING/NOTE severity

ladder all originate there. See NOTICE for the full licence text.

Why This Skill Exists

OpenEvidence uses RAG (vector search + LLM synthesis) to answer clinical questions. Its structural weaknesses cannot be fixed by improving the prompt:

1. Transitive citation — OE retrieves a chunk of a review's full text in which the review

quotes another trial's number, then credits that number to the review. The claim is real; the attribution is wrong. The origin corpus field (Step 0) tells us exactly when to suspect this, which is why provenance drives the whole Layer 1 flow.

1. Secondary source confusion — embedding search treats review-article claims the same as primary findings
2. Impact-score ranking bias — older high-IF positive results rank above newer negative ones
3. Selective presentation — synthesis picks supporting evidence, not a balanced view.

Layer 2 (external anchor: what did OE omit) and Layer 3 (do OE's own citations contradict each other?) exist for this.

This skill independently verifies OE output before you absorb it.

Transport

OE calls run through the openevidence MCP server (see the server README in this repo), which executes them as page-context fetch() inside a logged-in openevidence.com tab of a dedicated browser it drives over the DevTools Protocol. If OE tools fail with Not authenticated, run python mcp/server.py --login and log in again in the window it opens; for anything else, python tests/smoke_cdp.py checks the transport without touching OE.

Input Modes

Pipeline

Step 0: Obtain & Parse OE Response

If calling OE directly: use ask_openevidence(question, preserve_citations=True).

Parse the response:

1. If ## Citation Data JSON block exists → json.loads() to get citation array
2. Each citation object has: index, title, doi, pmid, date, journal, authors, publication_types, impact_score, recency_days, is_guideline, origin, origin_risk, url
3. The body text uses [N] inline markers matching citation.index

`origin` / `origin_risk` — which corpus the claim was retrieved from:

OE is a RAG system, and origin names the corpus the cited chunk came from. The MCP server decodes it (OE ships it ROT-1 obfuscated) and tiers it:

⚠️ One paper can hold several indices. OE assigns an index per retrieved chunk, so the same PMID legitimately appears under multiple [N] (observed: 10 indices over 7 papers). Two consequences:

• Dedupe by PMID/DOI before spawning agents — verify each paper once, then apply its

verdict to every index it owns. Spawning one agent per index burns tokens re-verifying the same paper.

• Never assume [N] ordering equals the bibliography order.

Build claim-span map:

1. Split body text into sentences (split on . but preserve e.g., vs., et al., numbered decimals)
2. For each sentence containing [N], record: {index: N, claim: "sentence text", context: "preceding sentence for context"}
3. Group by citation index, then merge groups that share a paper → each merged group is one Haiku agent's input
4. If a sentence has multiple [N][M], it maps to both citations
5. Mark a claim quantitative: true if it contains a number that carries an evidential load —

effect size, HR/OR/RR, CI, %, p-value, n. HIGH/MEDIUM origin + quantitative claim is the transitive-citation danger zone (Layer 1 Step 5 below).

Step 0.5: Deterministic Existence Gate (main agent, BEFORE spawning Haiku)

A citation-shaped object is not a source — OE returns well-formatted metadata whose DOI can be fabricated or resolve to a different paper. Existence checking is mechanical; do it deterministically and cheaply BEFORE spending any agent tokens on semantic comparison:

1. For each selected citation with a DOI: WebFetch https://api.crossref.org/works/{doi}

(no key needed; batch all fetches in parallel).

• 404 / no message → verdict FABRICATED immediately. Do NOT spawn a Haiku agent

for it — the semantic layer is meaningless for a nonexistent source.

• Resolves → compare returned title vs OE's citation.title (token-overlap, not

exact-match). Low overlap → keep the citation in the Haiku batch but prepend to that agent's prompt: "⚠️ DOI resolves to a different title than OE reports — verify identity via PMID/title search before trusting any content."

1. No DOI but has PMID → skip the gate here; the Haiku agent's PubMed lookup doubles as

the existence check.

1. Neither DOI nor PMID → note "existence unverified" and let the Haiku agent attempt

title search; failure there → UNVERIFIABLE as usual.

Failure-mode rule: if CrossRef itself is unreachable, do not treat the gate as passed — proceed to Layer 1 but mark those rows "existence gate skipped (CrossRef down)" in the report. A verification-tool outage must degrade to explicit uncertainty, never to a silent pass.

Step 1: Layer 1 — Citation Verification (Parallel Haiku Agents)

Citation cap: Max 10 papers (after PMID/DOI dedupe). If more exist, prioritize by:

1. origin_risk HIGH first — that is where mis-attribution actually lives
2. Citations backing key therapeutic claims (not background/epidemiology)
3. Most recent citations (lower recency_days)
4. Citations with DOI or PMID (verifiable)

For each paper, spawn Agent(model: "haiku") with this prompt template:

You are verifying a citation from OpenEvidence. Your job: check whether the
cited paper actually supports what OE claims.
 
## Citation
- Title: {title}
- DOI: {doi}
- PMID: {pmid}
- Journal: {journal}
- Date: {date}
- Publication types: {publication_types}
- OE retrieved this from corpus: {origin} (transitive-risk tier: {origin_risk})
 
## OE Claims Referencing This Citation
{numbered list of claim sentences; mark each [quantitative] where applicable}
 
## Instructions
1. Look up this paper:
   - First try: `mcp__semantic-scholar__get_paper_details` with DOI or title
   - Fallback: `mcp__claude_ai_PubMed__get_article_metadata` with PMID or title
   - If both fail: mark as UNVERIFIABLE
2. Read the abstract (and `tldr` if available from Semantic Scholar)
3. For each OE claim, compare against what the source actually says
4. Determine study type: is this a PRIMARY study (RCT, cohort, case-control,
   case series) or SECONDARY source (review, meta-analysis, guideline, scoping review)?
5. **Attribution check — run this whenever the corpus tier is HIGH or MEDIUM and the
   claim is quantitative.** The tier means OE pulled this text from the paper's full
   text, and reviews spend most of their full text restating OTHER studies' results.
   So ask specifically: is the number OE quotes **this paper's own finding**, or is
   this paper merely reporting someone else's trial?
   - If the abstract shows the paper generated this data → is_primary_source: true
   - If the claim names or implies another trial (e.g. "the LEADER trial showed…"),
     or the paper is a review/guideline restating pooled results → is_primary_source: false.
     Then TRACE IT: find the original study (Semantic Scholar `get_paper_references`,
     or PubMed search on the trial name / the exact effect size) and check whether the
     number matches the original. Report what the ORIGINAL said.
   - Verdict TRANSITIVE = the number is real but belongs to a different paper than the
     one OE cited for it.
 
## Output Format (use EXACTLY this structure)
CITATION_INDEX: {index — list all indices this paper holds}
VERDICT: [one of: ACCURATE, OVERSTATED, MISREPRESENTED, TRANSITIVE, SECONDARY_UNTRACED, UNVERIFIABLE]
STUDY_TYPE: {RCT|Systematic Review|Meta-analysis|Guideline|Scoping Review|Cohort|Case series|Other}
IS_SECONDARY: {true|false}
IS_PRIMARY_SOURCE: {true|false — did THIS paper generate the data OE attributes to it?}
OE_CLAIMS:
- Claim 1: {what OE said}
  SOURCE_SAYS: {what the paper actually says}
  MATCH: {true|false|partial}
- Claim 2: ...
PRIMARY_TRACE: {if not primary source — the original paper's title/DOI, what IT actually
  found, and whether OE's number matches it. "N/A" if this paper is the primary source}
KEY_FINDINGS_FROM_SOURCE: {what this paper actually concluded — 1-2 sentences. Layer 3
  uses this to cross-check citations against each other, so state findings, not verdicts}
EXPLANATION: {1-3 sentences summarizing the verification result}

Spawn all agents in a single message (parallel execution).

Step 2: Layer 2 — Evidence Completeness Check (Sonnet Agent)

Spawn 1x Agent(model: "sonnet") in the same message as Layer 1 agents (runs in parallel).

Prompt template:

You are checking whether OpenEvidence gave a BALANCED picture of the evidence
for a clinical question. Your job is neutral — find what OE missed in BOTH
directions (positive AND negative), anchored on the highest-level evidence
synthesis available.
 
## Clinical Question
{original question}
 
## Citations OE Used
{numbered list: index, title, DOI, year, study type}