maga2010org/hackathon-run/judge-sim

judge-sim

Input contract

Required:

• repo_root: project root
• .hackathon/state/plan.json (preferred; uses demo_path and KEEP list)
• .hackathon/state/demo.json (preferred; uses one_liner)

Optional:

• time_remaining_minutes: influences the fix priority list
• HACKATHON_JUDGE_BACKEND: optional HTTP endpoint for an LLM judge; the script POSTs the state inputs and uses the returned scores. Any failure falls back to the heuristic scorer.
• HACKATHON_JUDGE_TIMEOUT_SECONDS: request timeout for the backend (default 3).

Execution

1. Score seven dimensions

Each dimension: 0 (catastrophic) to 5 (excellent).

2. For each dimension, output

• score: 0..5
• deduction_reason: one line why not higher
• judge_questions: 2–3 likely questions
• improvements: 1–3 concrete actions

3. Compute fix priorities

Three buckets:

• FIX_NOW: changes that take < 30 min and improve any score
• FIX_LAST_10MIN: cosmetic / typo-level changes only
• DO_NOT_TOUCH: things that look fixable but risk breaking the demo

4. Hard rule

If .hackathon/state/verify.json last status is fail, any dimension scoring above 3 is invalid. Cap them at 3 and explain.

Output contract

Files written:

• .hackathon/state/review.json (matches src/state/schemas/review.schema.json)
• .hackathon/artifacts/judge-feedback.md (human-readable review)

Acceptance criteria

• [ ] Provides per-dimension score (0-5).
• [ ] Provides deduction_reason per dimension.
• [ ] Provides improvement suggestions per dimension.
• [ ] Distinguishes FIX_NOW / FIX_LAST_10MIN / DO_NOT_TOUCH.
• [ ] Caps dimensions at 3 when verify status is fail.
• [ ] Outputs a single overall score (mean).

Failure modes

Trigger phrases

• "how will judges score us"
• "what will the judges ask"
• "is this competitive"
• "final review before submit"
• "what should I fix"