sakanaai/ai-scientist-v3/scientific-evidence

scientific-evidence

Use this skill to define or materialize a generic evidence contract for scientific artifacts.

Always use this skill during paper writing whenever the agent adds quantitative content: manuscript numbers, table entries, figure inputs, derived statistics, or plotting/calculation scripts.

What this skill is for

The point of this skill is simple: an agent should not be able to drop important numbers, figures, or derived outputs into a paper without leaving a machine-checkable trail back to real artifacts.

This skill gives the agent one fixed place and one fixed grammar for that trail:

• inline @evidence[[...]] refs in text-like files
• .prov.json sidecars for generated artifacts
• a central scientific-evidence/ledger.jsonl for optional structured rollups

It is deliberately generic. The base contract works even when the workflow spans Python, bash, LaTeX, Markdown, JSON, logs, or other file formats.

Default workflow

1. Read the current paper/results layout, but do not invent a new evidence layout.
2. Use the fixed directory structure rooted at scientific-evidence/.
3. Run the skill script first instead of inventing schemas by hand:

bash src/agents/skills/scientific-evidence/scripts/scientific_evidence.sh --help
bash src/agents/skills/scientific-evidence/scripts/scientific_evidence.sh emit-bundle --root-dir <workspace-root>

1. For every quantitative paper-writing action, use this skill contract:

• adding a number to paper.tex
• filling a table cell
• writing a plotting script
• writing a calculation/statistics script
• generating a figure, table, or derived JSON file

1. Inside plotting or calculation scripts, do not hardcode experiment numbers that should come from artifacts. Read them from files, then leave the corresponding @evidence[[...]] refs near the read/derivation logic.
2. Add inline evidence refs in text-like artifacts and .prov.json sidecars for generated outputs.
3. Run the verifier after the agent writes claims, scripts, figures, or sidecars:

bash src/agents/skills/scientific-evidence/scripts/scientific_evidence.sh verify --root-dir <workspace-root>

1. Inspect scientific-evidence/verification-report.json and fail hard on dangling links.
2. Treat the generated markdown spec and JSON Schemas as canonical for downstream agents.

Keep the verifier behavior explicit:

• resolved: target artifact exists and the verifier can check the locator semantics it understands.
• anchored: target artifact exists, but the locator stays opaque or only hash/path checks were possible.
• dangling: the target or manifest link does not resolve.

How this is used to verify agent work

Verification works by checking whether the agent left a connected evidence trail.

At a minimum, a downstream verifier should be able to:

• scan manuscript text, scripts, and notes for @evidence[[...]] refs
• read .prov.json sidecars for figures, tables, and other generated outputs
• resolve those refs against real files or stable ids
• classify each link as resolved, anchored, or dangling
• fail when a generated artifact has no input trail or when a claimed source does not exist

This does not prove every scientific claim automatically. It does prove whether the agent left enough structure for an auditor or stricter verifier to replay and inspect the work instead of trusting free-form prose.

A core policy for paper writing is: do not hardcode result numbers inside plotting or calculation scripts when those values should be read from real artifacts. Read from files, derive from those inputs, and attach @evidence[[...]] refs that point back to the inputs.

Fixed directory structure

scientific-evidence/
  spec.md
  bundle.manifest.json
  ledger.jsonl
  schemas/
    evidence-reference.schema.json
    artifact-manifest.schema.json
    ledger-entry.schema.json
    verification-report.schema.json

Grammar summary

• Evidence ref token: @evidence[[TARGET]] or @evidence[[TARGET#LOCATOR]]
• TARGET may be a relative path, absolute path, URI, or stable identifier.
• LOCATOR is opaque unless a verifier adapter knows how to interpret it.
• Generated artifact sidecar name: <artifact>.prov.json
• Central ledger file: scientific-evidence/ledger.jsonl

Schema set

Generate the canonical schema bundle with the script. The bundle includes:

• scientific-evidence/spec.md
• scientific-evidence/schemas/evidence-reference.schema.json
• scientific-evidence/schemas/artifact-manifest.schema.json
• scientific-evidence/schemas/ledger-entry.schema.json
• scientific-evidence/schemas/verification-report.schema.json
• scientific-evidence/bundle.manifest.json
• scientific-evidence/ledger.jsonl

Examples

Python:

# Read the metric from disk; do not hardcode 96.7 here.
# @evidence[[results/metrics.json#mnist.accuracy_mean]]
import json
payload = json.loads(open("results/metrics.json", encoding="utf-8").read())
value = payload["mnist"]["accuracy_mean"]

Bash:

# @evidence[[results/run_summary.csv#row(dataset=mnist,variant=proposed).accuracy]]
python scripts/summarize_results.py --input results/run_summary.csv

LaTeX:

Accuracy reached 96.7\%. % @evidence[[generated/stats.json#mnist.accuracy_mean]]

Markdown:

Main result: 96.7%. <!-- @evidence[[generated/stats.json#mnist.accuracy_mean]] -->

R:

# @evidence[[results/summary.tsv#humanoid.edppo.mean]]
summary <- read.delim("results/summary.tsv")
value <- subset(summary, method == "ED-PPO" & env == "Humanoid")$mean[[1]]

YAML:

# @evidence[[generated/stats.json#mnist.accuracy_mean]]
reported_metric: mnist_accuracy_mean

JSON sidecar:

{
  "schema_name": "scientific-evidence.artifact-manifest",
  "artifact": {"path": "figures/main.png", "kind": "figure"},
  "derived_from": [
    "@evidence[[generated/stats.json#mnist.accuracy_mean]]",
    "@evidence[[scripts/build_figure.py]]"
  ]
}

Verification

Smoke test the script, bundle generation, and verifier:

bash src/agents/skills/scientific-evidence/scripts/scientific_evidence.sh --help
bash src/agents/skills/scientific-evidence/scripts/scientific_evidence.sh emit-bundle --root-dir /tmp/scientific-evidence-example
bash src/agents/skills/scientific-evidence/scripts/scientific_evidence.sh verify --root-dir /tmp/scientific-evidence-example