opendcai/dataflow-loopai/reviewing-dataflow-pipeline

Review a DataFlow Pipeline

Use this skill after a candidate pipeline has produced trial artifacts and before reporting mode=trial_run. This is a release gate, not a substitute for pipeline generation.

Read references/rubric.md completely before starting.

Required evidence

Review the candidate pipeline source, trial input and output, intermediate operator outputs or counters, logs, target benchmark metadata and evaluation code, and at least five official benchmark examples. Build a stratified audit sample of three input records per selected source dataset. Preserve dataset and sample_id provenance in every table and finding.

If the existing trial does not cover three records from every selected dataset, do not extrapolate from it. Extend the trial input and rerun the same candidate pipeline on the missing audit records. Do not run the full dataset.

Delegated review

Dispatch six independent subagents, one for each D1-D6 rubric dimension. Give each reviewer only the dimension it owns plus paths to the raw artifacts and the downstream target. Do not give reviewers a proposed score or another reviewer's conclusions. Run reviewers concurrently when capacity permits.

Each reviewer must return structured JSON containing:

{
  "dimension": "D1",
  "raw_score": 0,
  "weighted_score": 0.0,
  "blocked": false,
  "redlines": [],
  "evidence": [{"claim": "...", "numbers": "...", "sample_ids": ["..."]}],
  "findings": ["..."],
  "required_fixes": ["..."]
}

Every raw score is an integer from 0 through 4. Missing evidence scores zero; reviewers must not assume compliance. Evidence must cite measured counts, rates, distributions, or concrete sample IDs. A statement without either is not evidence.

The main agent must not silently change reviewer scores. It checks arithmetic, deduplicates findings, and applies every redline. Resolve a factual conflict by reading the cited artifact; if it cannot be resolved, use the lower supported score and record the conflict.

Scoring and gate

Calculate each weighted score as raw_score / 4 * dimension_weight, retaining at least two decimal places until the final sum. Weights are D1=15, D2=25, D3=20, D4=12, D5=16, and D6=12.

Any redline makes the result Blocked, regardless of total. Otherwise:

• 85-100: release
• 70-84.99: fix_then_release
• 50-69.99: rework
• below 50: reject

Only release permits delivery as mode=trial_run. For every other decision, fix the candidate pipeline when feasible, rerun the affected trial stages, and repeat all six independent reviews. Never relabel an unreviewed repair as passing. If the gate still does not pass, return mode=planned_only with the review report and blockers.

Write pipeline_review.json and a concise pipeline_review.md beside the pipeline. Include the per-dataset funnel, dimension scores, cited evidence, redlines, total, decision, and review iteration. The final DataFlow agent JSON summary must name both report paths and the gate decision.