aperivue/medsci-skills/meta-analysis

Meta-Analysis Skill

Meta-Analysis Types

If the type is ambiguous (DTA vs intervention), ask the user to clarify before proceeding.

Workflow Phases

Phase 1: Protocol Development

Goal: Produce a PROSPERO-ready protocol document. Write the protocol, extraction forms and manuscript text in English whatever language the user writes in — PROSPERO records and the target journals are English-language.

1. Research question: PIRD (Population, Index test, Reference standard, Diagnosis) for DTA;

PICO (Population, Intervention, Comparator, Outcome) for intervention.

1. DTA only — do QUADAS-3 phases 1 and 2 now, not at risk-of-bias time. They are

review-level and belong in the protocol: phase 1 states the synthesis question(s) (population, index test(s), target condition — a review may have more than one); phase 2 defines the ideal test accuracy trial for each (objective, participants, index test(s), definition of the target condition, analysis). Every later risk-of-bias and applicability judgement is made against that trial. Write the review-specific guidance for answering each signalling question here too, with clinical and methodological input, and publish it as a web appendix. Defining the ideal trial after seeing the studies is a judgement fitted to the results, not an assessment. See references/checklists/QUADAS3.md.

1. Eligibility criteria: study design, population, index test / intervention, comparator /

reference standard, outcomes (Se/Sp for DTA; effect size for intervention), and exclusion criteria with justification.

1. Search plan: at least 3 databases — PubMed, Embase, and Cochrane CENTRAL (add Scopus / Web

of Science as needed); a Boolean strategy from the PIRD/PICO components; a grey-literature plan (conference abstracts, trial registries); language restrictions and date range stated explicitly, with justification.

1. RoB plan: tool by type (table above), at least 2 independent assessors, and the

disagreement-resolution method (consensus, third reviewer).

1. Synthesis plan: bivariate random-effects (Reitsma) or HSROC (Rutter & Gatsonis) for DTA;

random-effects for intervention (Phase 6); heterogeneity, subgroup / sensitivity, and publication-bias plans.

1. PROSPERO registration document: read ${CLAUDE_SKILL_DIR}/references/PROSPERO_template.md

and follow its field guide, word limits, output format (Markdown + DOCX via pandoc), and Common Pitfalls Checklist. Save to the project's 7_Submission/ or equivalent directory.

• Submit before the formal search, and record the date. The user submits the registration;

record the initial submission date (not the later date the record is published) in the project's status file (STATUS.md, project_state.json or project.yaml, whichever the project keeps). The Phase 2 entry check looks for it. A pilot search is fine before submission; the formal search, screening and extraction, AI first-pass drafts included, are not.

• Registration-ID format gate. A PROSPERO ID is CRD42 + 9 digits (14 characters total),

e.g. CRD42024500001. Validate any ID that appears in the manuscript or registration doc with grep -oE 'CRD42[0-9]+' and assert a 14-character length / ^CRD42\d{9}$ — a 15-character ID (a stray digit) is a transcription error a reviewer will check against the live record.

• Review-type selection. Pick the least-wrong portal review type for the actual design and

state any portal constraint in the protocol. A descriptive single-arm proportion synthesis is not an "Intervention review"; choosing that type only to satisfy a portal field contradicts a later GRADE / effect-certainty statement. Whatever certainty language the protocol commits to (GRADE vs "evidence statements only") must match the manuscript verbatim — a guideline-style "we recommend" is not licensed by a descriptive review type.

Phase 2: Search Strategy

Goal: Develop and validate reproducible search strategies.

Entry check — registration before the formal search. Look for the registration submission date recorded in Phase 1 step 7. If none is recorded, stop and ask the user to choose: submit the registration now and run the search after that date, or proceed without it, in which case Phase 8 item 5 limits what the manuscript may claim. Record the choice. Screening or extraction done before submission, AI first-pass drafts included, cannot be made prospective afterwards. Under /orchestrate --e2e this stop is the REGISTRATION_PENDING halt; no autonomous default passes it.

1. Build search blocks from the PIRD/PICO components and execute them per database with

/search-lit (PubMed: MeSH + free text; Embase: Emtree + free text; further databases as the protocol specifies). Searching only: selecting records and extracting results belong to Phases 3–4, not to /search-lit's own selection and deep-read steps.

1. Report per PRISMA-S (Rethlefsen et al. 2021, PMID:33499930): one section per database with

date of search, number of results, and any limits applied.

1. Merge and deduplicate into a single spreadsheet: by DOI first, then PMID. Save raw counts

for the PRISMA flow.

Phase 3: Screening & Selection

Goal: Systematic title/abstract and full-text screening with two independent reviewers. Read references/phase3_screening_detail.md when executing a round (exclusion-code sets, AI pre-screening template and Methods boilerplate) or when a 3f/3f.5 gate fires (set algebra, reconciliation table).

3a. Round 1 — title/abstract (single reviewer). Define the exclusion codes from the protocol. Mark every record INCLUDE / EXCLUDE / MAYBE with a reason code → round1_{date}.tsv.

3b. Round 2 — dual independent title/abstract. A second independent reviewer (or AI as a documented second-pass tool with human verification) re-screens all R1 records. Report Cohen's κ in Methods. round2_tag = INCLUDE / EXCLUDE / MAYBE (MAYBE = disagreement or either reviewer flagged uncertainty), plus round2_reason.

3c. Round 3 — adjudication (first reviewer). MAYBE records first, then INCLUDE records for a brief confirmation pass → round3_decision (plus round3_reason only when overturning R2). Optional AI pre-screening may compress the effort, but AI suggestions are not decisions: the reviewer independently confirms or overturns every one.

3d. Round 4 — full text (/fulltext-retrieval) for round3_decision = INCLUDE: full-text exclusion codes, two independent reviewers, Cohen's κ, consensus or a third reviewer. Flag comparative studies for priority extraction.

3e. PRISMA flow. Track counts at every stage (R1 → R2 → R3 → R4 → final included); draw it with /make-figures once final.

3f. Post-consensus count reconciliation gate (MANDATORY before Phase 5 write-up). Reconcile counts from the raw ID sets, never from prose summaries, into one source-of-truth file:

python "${CLAUDE_SKILL_DIR}/scripts/screening_reconcile.py" \
  --screening 2_Screening/fulltext_screening.tsv \
  --consensus 2_Screening/consensus_decisions.tsv \
  --table1 6_Tables/table1_studies.csv \
  --output 2_Screening/screening_consensus.json

Downstream stages consume screening_consensus.json for counts and ID sets; the Markdown consensus document remains the human explanation. Three hard rules:

1. List the narrative-only IDs explicitly. The highest-yield red flag is a numeric claim ("10

narrative-only studies") that does not match the enumerable set (A ∪ C) \ B \ T.

1. No "N → M" transition without ID receipts. "k rose from 30 to 32 after FLAG consensus" must

cite the added/removed IDs. A transition claim with no enumerable ID set is a P0 and blocks the Phase 5 hand-off.

1. `STAGE_TRANSFER_LOSS` is a P0. Exit 1 when a record is included at screening but **absent

from the consensus artifact altogether** — no adjudication was ever recorded. An exclusion is a decision; silence is a gap. Never let it settle into narrative-only.

Two more gates at 3f; a non-zero exit from either blocks the Phase 5 write-up:

# every applied exclusion code vs the *registered* eligibility criteria
python3 ${CLAUDE_SKILL_DIR}/scripts/check_exclusion_code_validity.py --protocol 0_Protocol/protocol.md --screening 2_Screening/*.tsv --strict
# DI-6: PRISMA numbers on 5 surfaces vs YAML SSOT; re-run on every revision touching PRISMA numbers
python3 ${CLAUDE_SKILL_DIR}/scripts/prisma_5way_consistency.py --ssot prisma.yaml

Exclusion-code verdicts: CODE_CONTRADICTS_ELIGIBILITY (a code excludes a design the protocol includes — bulk study loss no other gate can see), CODE_NOT_REGISTERED, CODE_RENUMBERED. Verdicts, NOT_ASSESSED cases and known limits of all 3f gates: references/phase3_screening_detail.md §3f.

• Known limit: PRISMA 5-way still skips after_dedup → full_text_assessed; optional prisma2020: counts add report-level identities (§3f).

3f.5 Pool composition lock (MANDATORY at adjudication freeze). Once 3f passes, freeze the pool into a single source-of-truth YAML that every downstream artifact can be checked against:

cp "${CLAUDE_SKILL_DIR}/templates/FINAL_POOL_LOCK.yaml.template" 2_Data/FINAL_POOL_LOCK.yaml
# fill counts + UID lists from 3f, compute the SHA-256 over the sorted UID list,
# and COMMIT THE LOCK before any Phase 4 extraction

• Never re-derive `k included` from the extraction TSV at manuscript build time — always

reference final_pool_n from the lock.

• Aggregate patient/lesion totals are locked too, not just study counts. Distinguish

arm-separable from both-arm rows: a study contributing one arm must not have its full-cohort count folded into a pooled total. A hand-carried headline total that does not re-derive from the locked per-study values is a P0.

• A late post-freeze change to the pool is a formal PROSPERO amendment: file it, re-freeze as

FINAL_POOL_LOCK_v2.yaml, and propagate to every artifact.

Phase 4: Data Extraction

Goal: Create standardized extraction forms and extract 2x2 or effect-size data. Read references/phase4_extraction_detail.md when building the form (DTA / intervention field lists), when an AI draft was shared, for the optional extract_assist.py suggestions (AI_SUGGESTED, a human confirms each before dta_extraction_qc.py), or when a QC flag fires.

4.0 Entry gate (MANDATORY) — pool composition lock ↔ adjudication TSV. Before any extraction work begins, confirm the round-3 adjudication TSV and FINAL_POOL_LOCK.yaml (Phase 3f.5) agree on which UIDs are included:

python "${CLAUDE_SKILL_DIR}/scripts/check_pool_consistency.py" \
    --lock 2_Data/FINAL_POOL_LOCK.yaml \
    --adjudication-tsv 2_Screening/round3_adjudication.tsv \
    --decision-col round3_decision --uid-col uid \
    --include-labels "INCLUDE,INCLUDE_MIXED" \
    --out qc/pool_consistency.json