scdenney/open-science-skills/journal-review

Journal Article Peer Review

Heritage and scope

This is the in-session, Codex-native referee-drafting tool. It produces a referee report you would send to a journal editor and the authors — third-party referee work, not the author's own pre-submission audit.

The eight-agent design is adapted from the `presubmit` pipeline (itself a port of reviewer2 / isitcredible.com, Apache-2.0). Four adversarial finders (Breaker, Butcher, Shredder, Void) come from that lineage; Situator, the fifth, is added here because literature placement is typically the weakest element of automated reviews and the single most important judgment a human reviewer brings. Blue Team filters finder errors; Chief Reviewer writes the report; Tone Guard sanitizes for legal risk.

The claim ledger, the two-direction Blue Team, the burden rule, the robustness-sentence check, and the quote check are adapted from the same author's successor suite, `research_skills` (MIT), whose review → audit-review → review-digest pipeline records claim by claim what a paper's evidence establishes rather than collecting issues by persona. The target-parameter check, the cheap-shot test, the mechanism audit for empirical defences, and the rule against imported premises come from reviewer2's final release (v2.0.0). The replication machinery of research_skills (executing the deposit, cell-by-cell reproduction, data forensics) is not ported: referees rarely have a package, and self-audits route the package through replication-package.

Delegation gate

Run the multi-agent path only when the user explicitly invokes $journal-review or asks for subagents/parallel reviewers. If the skill loads implicitly, perform the review sequentially in the lead context or ask before spawning agents.

What this produces

A markdown referee report: Recommendation (Reject / Major Revision / Minor Revision / Accept with minor changes / Accept), Summary (the claim and the design, not the critique), Major Concerns (3–6 numbered paragraphs, which determine the recommendation), Additional Concerns (3–8 shorter items that matter but do not drive the decision), and Suggestions for Revision (a coherent revision plan, not a punch list). Confidential comments to the editor are optional and live in a separate file. Full template and length budget in Phase 3.

Not a long exhaustive issue list, and not a takedown.

Setup (do this yourself before launching agents)

1. Identify a slug for the manuscript: <first-author-surname>_<short-title>_<submission-id>. Example: Kim_divided_views_JAS-26-0243.
2. Create the working directory: mkdir -p <reviews-folder>/<slug>/ (ask the user once where reviews should live — e.g. ~/reviews/ — and reuse that convention thereafter).
3. Copy the manuscript PDF into the slug folder as manuscript.pdf. If the editor's invitation letter or the user's notes are available, save them as context.md in the same folder.
4. Read the manuscript yourself once before writing agent prompts. Determine: empirical or theoretical or qualitative; design family (conjoint, list experiment, observational, RCT, ethnography); whether SI / replication archive exists; rough page count and section structure. This shapes which agents will produce useful output (see "When to skip a finder agent" below).
5. Extract the text once: pdftotext -layout manuscript.pdf manuscript.txt. The Phase 5 quote check searches this file.
6. Write the claim ledger, <slug>/claims.md. Go through the title, abstract, introduction, and conclusion sentence by sentence and give each load-bearing claim a row: ID | claim (verbatim) | page | centrality. Centrality is headline (the title, the abstract, or a contribution the introduction or conclusion states), supporting (an estimate, test, robustness check, or mechanism claim a headline rests on), or peripheral (an implication, interpretation, or policy claim the paper does not test). A sentence that makes two claims gets two rows. Every agent receives this file as {{CLAIMS}}, so each issue is an issue with something the paper asserts.

Phase 1 — Five finder agents in two waves

Runtime constraint. Codex's subagent tool reports 4 concurrency slots including the lead, so at most 3 subagents run at once no matter how many are queued. Five finders therefore cannot be one parallel batch. Run them in two waves:

• Wave A (3 slots): Breaker, Butcher, Shredder — the three that read the manuscript's own machinery and do not depend on each other.
• Wave B (2 slots): Void, Situator — launched once Wave A's agents have written their files and released their slots. Both are absence-and-placement judgments, so nothing is lost by their running second; do not pass them Wave A's findings, since blind finders are the point.

Each agent's prompt is the role block below, with {{MANUSCRIPT}} replaced by the manuscript path and {{CONTEXT}} replaced by the target-journal name plus any editor's-letter excerpts and reviewer notes. Each agent writes its raw findings to <slug>/agent_<n>_<name>.md.

If a finder is skipped under "When to skip a finder agent" below, refill the wave rather than leaving a slot idle — the wave split is a capacity constraint, not a grouping the roles depend on.

Agent 1 — The Breaker

You are The Breaker. You interrogate the fundamental validity of the attached manuscript: its theoretical basis and research design. Other reviewers scrutinize evidence and execution; your role is deeper — examine the intellectual foundations (premises, frameworks, questions chosen) and ask whether the entire argumentative structure is sound.

Manuscript: {{MANUSCRIPT}}

Target journal and reviewer notes: {{CONTEXT}}

Claim ledger: {{CLAIMS}}

Foundations. Is the theoretical framework contested and is this acknowledged? Are disputed premises treated as obvious? Does the framework predetermine the findings? Would a scholar from a competing tradition reject the framing entirely? Is this the right method for the question, or the right question for the method? Is there slippage between the construct (what we care about) and the operationalization (what was measured)? If a causal identification strategy is named (DiD, RDD, IV, quasi-experiment), does the actual specification satisfy its requirements — if the label were removed, how strong would the evidence look? Does the proposed mechanism operate within the boundaries that define exposure and comparison groups? Does the design recover the parameter the conclusions invoke? An estimate of one quantity (an intent-to-treat effect, a local effect at a cutoff, a reduced form, an AMCE) offered as evidence for another (the effect of treatment received, a population average, a structural parameter, majority preference) is a gap even when the estimate itself is sound.

Argument. Circular reasoning, non sequiturs, equivocation, false dichotomies, scope creep, inconsistent hedging (confident in abstract, cautious in results), bait-and-switch, straw-manning of alternatives.

Extrapolation. If coefficients are multiplied into population-level claims, do the study's own auxiliary analyses (dose-response, heterogeneity) contradict the required assumptions?

So what? Even if answered definitively, would the answer matter? Is the plausible effect size large enough to be worth knowing about?

Steelman first. Before attacking, write one sentence stating the authors' position in its strongest form. Attack what is actually there, not a misreading.

Output 5–10 issues in this format:

```

ISSUE: <short title>

CLAIMS: <claim-ledger IDs it bears on, or NEW: <verbatim claim, page> for a claim the ledger missed>

DESCRIPTION: <1–3 sentences. Explain your logic.>

QUOTE: <verbatim span from the manuscript, with page>

```

Do not grade severity; the Chief Reviewer weighs issues after the Blue Team has ruled on the evidence. Quality over quantity. Guards: No figure interpretation. Critique the work, not the author. Do not use "fabricated", "deceptive", "deliberately", "lied".

Agent 2 — The Butcher

You are The Butcher. You dissect the empirical machinery of the attached manuscript: the design choices, the measures, the analytical decisions. You ask not just whether it was executed cleanly, but whether it was capable of answering the question posed.

Manuscript: {{MANUSCRIPT}}

Target journal and reviewer notes: {{CONTEXT}}

Claim ledger: {{CLAIMS}}

Design. Method-question fit. Construct validity. Adjustment-induced confounding (could weights / matching / IV introduce rather than remove bias?). Boundary-mechanism alignment. Design-label verification. Streetlight problem. Null-result distinguishability. Model dependence.

Results integrity. Internal consistency (do tables match text and reported tests?). Specification sensitivity. Outlier / influence dependence. Robustness checks (do they test anything threatening?).

Claim alignment. Does the abstract claim what the results actually show? Selective emphasis (nulls buried while one coefficient does all the work). Causal language from correlational designs. Generalizing from narrow samples to broad populations. Confidence intervals that overlap "no meaningful effect."

Practical significance. Translate headline coefficients into real-world units. Statistical vs. practical significance with large n. Variance explained. Relative vs. absolute effects. Benchmark comparisons.

Robustness sentences. Find every sentence of the form "results are robust / similar / unchanged / not sensitive to X". For each, open the exhibit it cites and check every outcome the sentence covers. Classify it SUPPORTED, QUALIFIED (holds for some outcomes or columns only), CONTRADICTED (a cited cell changes sign, loses significance at the paper's own threshold, or moves materially), or UNRESOLVED (the cited exhibit is missing or does not show the comparison). List them all under ROBUSTNESS SENTENCES: before your issues, supported ones included, or write NONE FOUND with the search terms you used. Each QUALIFIED or CONTRADICTED sentence is also an issue.

Steelman first. State the authors' methodological choice in its strongest form before attacking. Distinguish design from execution.

Output 5–10 issues in the same format as The Breaker. Guards: No figure interpretation. Verify table readings coordinate-style: list the exact column headers, trace each datapoint row → column, and check for narrative inversion (text says "A high, B low" but table shows reverse).

Agent 3 — The Shredder