haddock-development/ttc-pi-package/ttc-judge

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Query     │     │  Response   │     │   Judge     │
│             │────▶│   Model     │────▶│   Model     │
└─────────────┘     └─────────────┘     └──────┬──────┘
                                               │
                                               ▼
                                        ┌─────────────┐
                                        │   Score     │
                                        │  0.0 - 1.0  │
                                        └─────────────┘

{"query": "...", "response": "...", "score": 0.9, "rationale": "Correct and well-explained"}
{"query": "...", "response": "...", "score": 0.3, "rationale": "Partially correct but incomplete"}

python scripts/train_judge.py \
  --base-model Qwen/Qwen2.5-0.5B \
  --dataset path/to/judge_data.jsonl \
  --output models/ttc-judge \
  --backend kaggle

python scripts/eval_judge.py \
  --judge models/ttc-judge \
  --benchmark judge-bench \
  --metrics accuracy,correlation

from ttc_judge import JudgeModel
judge = JudgeModel.from_pretrained("models/ttc-judge")
# Score responses
for response in responses:
    score = judge.score(query, response)
    if score > 0.8:
        accept(response)

User: Train a judge to evaluate math solutions
 
Agent will:
1. Prepare judge dataset with scored examples
2. Train judge model
3. Evaluate on held-out set
4. Return judge model path