castform-ai/benchmax/view-progress

View progress

Start with the run ID printed by main.py launch:

castform runs status <run-id>
castform runs scalars <run-id> --mode eval --json
castform runs logs <run-id>

Use eval, not only train reward, to judge generalization. A rising train curve with a flat or falling eval curve is overfitting, not success.

Inspect actual rollouts

Scalar totals are not enough to validate a reward. Read transcripts and per- component scores in the terminal or as JSON:

castform runs rollouts <run-id> --mode eval
castform runs rollout <run-id> <rollout-id>
castform runs rollout <run-id> <rollout-id> --json

runs rollout can join ground truth from local JSONL. Pass --dataset <path> when the project does not use the default eval/train filenames. Use the text or JSON output and the run page printed at launch.

When reviewing stored outcomes, distinguish a valid zero score from execution failure using available logs and stored error fields. benchmax produces a non- finished termination reason locally, but carrying that field faithfully through the trainer and hosted rollout views is pending downstream integration; do not claim a stored run exposes it until the platform does.

Diagnosis

• pending for too long: inspect status and launch/platform logs.
• failed early: inspect environment imports, bundle dependencies and the first

rollout error; fix the project and re-run validation before another launch.

• stalled: inspect recent activity and logs; do not infer model quality from an

incomplete run.

• flat rewards: read correct and incorrect transcripts, then test the reward

locally for discrimination and accidental bonus paths.

• eval peaks then declines: record the best step and verify which checkpoint is

available before treating the final checkpoint as best.

• judge errors: fix auth/provider/runtime reliability; never reinterpret the

zeroed failure reward as the judge's verdict.

<!-- rag:start --> For RAG, separate retrieval from answer quality: gold never retrieved, gold retrieved but not cited, and correct cited answers are different failure modes. Check source-ID canonicalization before changing reward weights. <!-- rag:end -->

Every runs read supports --json. Use it for programmatic comparison, but do not build an unbounded polling loop. Re-run status and scalar reads after returning to a long-running job.

To cancel a run owned by the current account:

castform stop <run-id>

Preserve the run ID, terminal state and decisive log/reward evidence in the handoff.