sumit11402/knights-ae-02/stat-research-orchestrator

Statistical Research Orchestrator

Overview

Coordinates the full statistical research pipeline. This is not a code-first benchmark workflow. The pipeline begins with formal problem formulation and requires theory before final comparisons and conclusions.

Full Pipeline

Topic prompt / topic file / dataset description
  -> [stat-problem-formulator]   formal problem, notation, assumptions, targets
  -> [stat-method-proposer]      proposed method, baselines, diagnostics, ablations
  -> [stat-theory-analyzer]      theoretical properties, proof sketches, predictions
  -> [stat-experiment-designer]  experiments, code, metrics, manifest
  -> [stat-comparison-analyst]   method comparison, theory-vs-experiment check
  -> [stat-result-synthesizer]   final report, conclusions, limitations
  -> [stat-quality-auditor]      formulation/theory/evidence audit

Workflow

Step 0: Invoke stat-problem-formulator

Provide the topic source and any requirements. Wait for:

progress/<TOPIC_ID>/step0_problem_formulation.md

Read:

• Formal data model
• Target parameter or decision target
• Assumptions
• Hypotheses or claims
• Evaluation criteria
• Theory targets

Do not proceed if the target or assumptions are undefined.

Step 1: Invoke stat-method-proposer

Provide the problem formulation. Wait for:

progress/<TOPIC_ID>/step1_method_proposal.md

Read:

• Proposed method
• Baselines
• Oracle references, if any
• Ablations
• Diagnostics
• Implementation requirements

Step 2: Invoke stat-theory-analyzer

Provide the formulation and method proposal. Wait for:

progress/<TOPIC_ID>/step2_theory_analysis.md

Read:

• Theoretical claims
• Required assumptions
• Proof sketches or derivations
• Predicted empirical patterns
• Limitations

Theory can be partial, but the report must honestly label what is proven, heuristic, or only experimentally supported.

Step 3: Invoke stat-experiment-designer

Provide formulation, method, and theory. Wait for:

progress/<TOPIC_ID>/step3_experimental_evaluation.md

Read:

• Config path
• Code paths
• Metrics
• Manifest
• Raw results
• Runtime deviations

Step 4: Invoke stat-comparison-analyst

Provide theory predictions and experiment outputs. Wait for:

progress/<TOPIC_ID>/step4_comparison.md

Read:

• Comparison summary
• Figures and tables
• Claim verdicts
• Theory-experiment agreements and disagreements

Step 5: Invoke stat-result-synthesizer

Provide all previous artifacts. Wait for:

progress/<TOPIC_ID>/step5_result_synthesis.md

Read:

• Paper path
• README path
• Final claims
• Limitations

Step 6: Invoke stat-quality-auditor

Audit the whole research chain:

• Was the problem formulated formally?
• Does the method address that formulation?
• Is there theory or an explicit reason theory is limited?
• Do experiments test theoretical predictions?
• Are comparisons fair?
• Are final conclusions supported?

Wait for:

progress/<TOPIC_ID>/step6_quality_audit.md

Progress File Specification

progress/<TOPIC_ID>/step0_problem_formulation.md

# Step 0: Problem Formulation
## Status: PASS / FAIL
## Topic ID: <TOPIC_ID>
## Research Question
...
## Formal Data Model
...
## Target / Estimand
...
## Assumptions
-...
## Claims / Hypotheses
-...
## Evaluation Criteria
-...
## Theory Targets
-...
## Blocking Ambiguities
-...

progress/<TOPIC_ID>/step1_method_proposal.md

# Step 1: Method Proposal
## Status: PASS / FAIL
## Proposed Method
...
## Baselines
-...
## Diagnostics
-...
## Ablations
-...
## Method-to-Claim Map
-...

progress/<TOPIC_ID>/step2_theory_analysis.md

# Step 2: Theoretical Analysis
## Status: PASS / PARTIAL / FAIL
## Definitions
...
## Main Claims
-...
## Proof Sketches
-...
## Assumptions Required
-...
## Predicted Empirical Patterns
-...
## Limitations
-...

progress/<TOPIC_ID>/step3_experimental_evaluation.md

# Step 3: Experimental Evaluation
## Status: PASS / FAIL
## Config
experiments/<TOPIC_ID>/config.yaml
## Code
-...
## Experiments
-...
## Metrics
experiments/<TOPIC_ID>/results/metrics.json
## Manifest
experiments/<TOPIC_ID>/results/run_manifest.json
## Warnings
-...

progress/<TOPIC_ID>/step4_comparison.md

# Step 4: Comparison
## Status: PASS / FAIL
## Baseline Comparisons
-...
## Ablation Findings
-...
## Theory vs Experiment
-...
## Claim Verdicts
experiments/<TOPIC_ID>/results/claim_verdicts.json

progress/<TOPIC_ID>/step5_result_synthesis.md

# Step 5: Result Synthesis
## Status: PASS / FAIL
## Paper
experiments/<TOPIC_ID>/report/paper.md
## README
experiments/<TOPIC_ID>/README.md
## Final Claims
-...
## Limitations
-...

progress/<TOPIC_ID>/step6_quality_audit.md

# Step 6: Quality Audit
## Status: PASS / WARN / FAIL
## Formulation Check
-...
## Theory Check
-...
## Experiment Check
-...
## Comparison Check
-...
## Blocking Issues
-...

Key Conventions

• Formulation is the gatekeeper. Do not write code before the target,

assumptions, and evaluation criteria are explicit.

• Theory is required as a pipeline stage. If no theorem is possible, write a

clear heuristic or negative analysis and explain why.

• Experiments should test theoretical predictions, not merely produce numbers.
• Comparisons must include meaningful baselines or ablations.
• Final results must connect formulation, method, theory, experiments, and

comparison.