open-h/open-h-rf/open-h-rf-submission-eval

Open-H-RF Submission Evaluator

This skill reviews a contributor submission to the Open-H-RF initiative and produces a structured acceptance report. Each evaluation dimension is independent and is designed to be dispatched to a sub-agent in parallel.

Minimum zea version

Every submission must have been written with zea ≥ v0.1.0a3 — Open-H-RF's acceptance floor. Older alphas (≤ v0.1.0a2) have format drift the current loaders no longer accept; this is an Open-H-RF policy, not a zea spec point. validate_zea_spec.py enforces it automatically: it sets compliant: false and emits a blocker finding when the file's zea_version attribute is missing or below the floor, so dimension 1 catches it without a separate manual check.

What you're evaluating against

Open-H-RF accepts pre-beamformed ultrasound channel data in the zea file format, licensed CC BY 4.0, accompanied by a Hugging Face–style README.md (which is the data card — Hugging Face renders README.md as the dataset landing page; see <https://huggingface.co/docs/hub/datasets-cards>) and a reconstruction pipeline that turns the raw channel data into a B-mode image. There is exactly one user-facing document: `README.md`. There is no separate `DATA_CARD.md`. The full requirements live in:

• references/rfp-summary.md — RFP scope, tiers, eligibility, timeline
• ../open-h-rf-shared/data-card-template.md — exact fields the README.md must contain, including the YAML frontmatter HF parses
• zea's spec classes — zea.data.spec.{DataSpec, ScanSpec, ProbeSpec, MetadataSpec} and <https://zea.readthedocs.io/en/openh-rf-latest/> — what a valid zea file looks like and what fields it must expose (authoritative; no parallel copy maintained here)
• references/phi-checklist.md — canonical HHS Safe Harbor 18 identifiers (used by dimensions 4 and 7)
• references/imaging-artifacts.md — taxonomy of acquisition-induced vs pipeline-induced image artifacts (used by dimension 2)

Read these as needed. Do not try to hold them all in head — pull them in when a specific dimension calls for it.

Sub-agent personas

Each dimension file opens with a short Persona block — a one-paragraph role and disposition. When dispatching a sub-agent for a dimension, the persona is the framing the sub-agent operates under (e.g., "data-format auditor", "ultrasound imaging engineer", "research compliance officer"). The personas are deliberately lightweight: just enough to set tone and rigor, not a full character. They make findings more memorable across dimensions and help calibrate severity ("the compliance officer flagged …" vs "the reproducibility reviewer flagged …").

Do not invent new personas or stretch them. If a dimension's persona feels wrong for a specific submission, fall back to neutral instruction-following — don't drift.

The seven evaluation dimensions

A submission is scored on seven independent dimensions. Each has its own checklist file under references/dimensions/. The dimensions are deliberately decoupled so they can run in parallel as sub-agents:

1. Format compliance (dimensions/01-format.md) — Is the data actually in zea format? Do the on-disk fields, shapes, and dtypes match the spec?
2. Reconstruction (dimensions/02-reconstruction.md) — Run the contributor's pipeline. Does it produce a sensible B-mode image? This is the most expensive check.
3. Metadata sufficiency (dimensions/03-metadata.md) — Are all acquisition hyperparameters (transducer geometry, element count, center frequency, sampling rate, sound speed, transmit sequence, etc.) present and self-consistent? A novice with no contact with the contributor must be able to reconstruct from what's in the file alone.
4. Data card (dimensions/04-data-card.md) — README.md exists at the dataset root, carries valid HF YAML frontmatter (license, pretty_name, task_categories, tags), every template section filled in, no placeholder text, statistics aggregate-only (no PHI).
5. Documentation & clarity (dimensions/05-documentation.md) — Can a graduate student who has never seen this data understand what it is, what task it supports, and how to use it within ~15 minutes of reading? This is the "novice user" check, evaluated against the same README.md plus the pipeline and reference images.
6. Licensing & IP (dimensions/06-licensing.md) — Confirmed CC BY 4.0, no third-party IP encumbrances, consent/IRB status documented.
7. Ethics & compliance (dimensions/07-ethics.md) — HIPAA Safe Harbor de-identification for human data, ARRIVE 2.0 for animal, IRB approval where required.

Each dimension file contains its own pass/fail criteria, severity rubric, and what evidence to record.

Workflow

Step 1: Locate and inventory the submission

The user will hand you a path (local folder, mounted bucket, HF repo). Before evaluating anything, list what's actually there. Expect roughly:

• One or more .hdf5 files in zea format — the channel data; each acquisition is a single HDF5 file, and a submission may contain several
• reconstruct.py — a single runnable script that reconstructs from the raw channel data and outputs a .png
• pipeline.yaml — a saved zea.Pipeline, one per track in the file(s)
• README.md at the dataset root — this is the data card (HF renders it as the dataset landing page). Must have YAML frontmatter with license, pretty_name, task_categories, tags.
• LICENSE file declaring CC BY 4.0 (either spelling accepted; LICENSE is the canonical one the submission guide uses)
• 1–2 reference B-mode images (the .png(s) reconstruct.py produces)
• Possibly: consent/IRB documentation

This is exactly the deliverable set the submission guide requires: zea .hdf5 files, reconstruct.py, pipeline.yaml (one per track), README.md, and LICENSE. A separate DATA_CARD.md is not part of the expected layout — if one exists, treat it as a contributor-side artifact to be merged into README.md and flag the duplication.

Multiple sub-datasets. Some submissions bundle several sub-datasets (distinct acquisition types or geometries) as separate folders under one submission root. The recommended layout gives each sub-dataset folder its own pipeline.yaml + reconstruct.py, rather than sharing one pair at the root — this keeps each reconstruct.py a thin script instead of growing acquisition-detection and branching logic to serve every sub-dataset from one script. A shared root-level pair is still an acceptable choice when the pipeline is genuinely identical across all sub-datasets. Seeing multiple reconstruct.py/pipeline.yaml pairs across sub-dataset folders is the expected layout for this case — do not flag it as duplication the way a stray DATA_CARD.md would be.

Make an inventory.json recording what's present, what's missing, and file sizes. If the inventory is grossly incomplete (e.g., no zea file at all), stop and report — there's nothing to evaluate yet.

Read the proposal first — it is required. The accepted proposal is the contract: the submission is evaluated partly on whether it delivers what the contributor said they would deliver. Before dispatching any dimension sub-agents, read the contributor's accepted proposal from the source location (typically a PDF in the submission tree). If it isn't in the submission, ask the user for it. If no accepted proposal can be produced, stop and mark the evaluation `blocked` — without it you cannot check alignment or honor any negotiated carve-outs. Extract: declared tier, tasks, the data types / modalities / quantities promised, license intent, ethics codes, and any explicit carve-outs (things the contributor said they would not provide). Carve-outs the steering group accepted at proposal review are not failures during evaluation — note them and move on. Each dimension sub-agent prompt must include the carve-out list so they don't double-jeopardy the contributor on something already negotiated.

Common carve-out examples: "3D position information will not be provided", "subject demographics aggregate-only", "annotations limited to N frames per acquisition".

Step 1b — check the submission against the proposal. Open-H-RF wants submissions that match what was proposed and accepted, not something materially different. Compare the inventory to the proposal and record three things:

• Delivered as proposed — promised content that is present. No finding.
• Under-delivered — content the proposal promised that is missing or materially smaller than promised, and that is not an accepted carve-out. This is a major finding (route it to the relevant dimension): the contributor should either deliver it or get a carve-out on record. (Missing raw_data is always a blocker regardless — see below.)
• Added beyond the proposal — content present that the proposal did not mention (extra modalities, tasks, subjects, derived products). This is not a reason to reject — additions are welcome — but each addition must be documented in the data card (README.md). An undocumented addition is a major data-card finding ("you included X beyond your proposal; describe it in the data card"). Also note material scope additions for steering-group awareness.

Pass the alignment summary (under-delivered list + added-beyond-proposal list) into the relevant dimension sub-agents alongside the carve-out list.

Hard requirement — no carve-out applies. Every Open-H-RF submission must include raw pre-beamformed channel capture data (/data/raw_data in zea format). This is the core of the dataset and is non-negotiable: a proposal that omits raw channel data does not qualify, and an evaluation that finds /data/raw_data missing or empty should mark dimension 1 (Format) blocker regardless of what the proposal says.

Step 2: Dispatch the seven dimension checks in parallel

In Claude Code, spawn one sub-agent per dimension. Each sub-agent:

• Reads only its own references/dimensions/0X-*.md
• Reads only the submission artifacts relevant to its dimension (the dimension file says which)
• Returns a structured result: {dimension, status, severity, findings[], evidence[], suggested_fixes[]}

status is one of: pass, pass_with_notes, fail, blocked (couldn't evaluate — usually means a dependency is missing). severity is info, minor, major, blocker. A blocker on any dimension means the submission cannot be accepted as-is.

If you're not in an environment with sub-agents, run them serially in the order above — format first (cheap, gates everything else), reconstruction last (most expensive).

Step 3: Auto-fix the fixable

Several common gaps are fixable without going back to the contributor: