sahil-ss9/agent-personas/decision-head-training-and-label-supply

Decision head training and label supply

When to use

Every off-the-shelf candidate fails the utility gate on your question.

Procedure

1. Establish the label supply first: real inputs, existing decisions, weak supervision, or active labelling.
2. Freeze a held-out set before training anything.
3. Prefer the cheapest adaptation that can work: freeze the backbone and train a small head on a few examples per class.
4. Fine-tune on your own question and option shape, since the base checkpoint may be far weaker than a task-fine-tuned one.
5. Re-run the frozen exam and compare against the off-the-shelf candidates on the same basis.

Decision rules

• Labels are the constraint, not compute. If you cannot supply labels, stop and choose a different design.
• Never train on the held-out exam set. Its value is that it has never been trained on.
• Compare a fine-tuned small model against the encoder you would otherwise train, on measured cost and agreement, including where the comparison crosses over.
• A fine-tuned model needs the same gate discipline as a purchased one.

Decision-output clarification

For a read-only readiness decision, report prerequisites and the permitted adaptation without initiating training. Preserve held-out isolation. When structured output is requested, return the requested complete object with a short reason rather than a reasoning transcript; never imply that assessment performed training or external checks.

Before returning a structured result, check that each field answers the requested proposition, uses the requested types and vocabulary, and agrees with the explanation. Saved evidence is not an action performed by you; a recommended next step is not an executed action.

Pitfalls

• Starting with training before checking label supply.
• Fine-tuning on the benchmark and reporting the benchmark result.
• Assuming fine-tuning beats a better checkpoint that already matches your question shape.

Done

The label source is stated, the held-out set is untouched, and the trained head is compared to off-the-shelf candidates on the identical frozen exam.