scdenney/open-science-skills/llm-calibration-logprobs

Reading a Model's Own Uncertainty from Token Log-Probabilities

Instructions

This skill covers within-model confidence: how sure one model is about each decision, read from the token log-probabilities it emits, and whether that confidence is calibrated against ground truth. Agreement across several independent models is a different kind of evidence — when a pipeline uses both, report both. Codebook design and human-validation statistics (κ, F1) belong to $text-classification; cross-reference rather than re-derive them.

1. What a Logprob Can and Cannot Tell You

• Use logprobs when you need a per-item, continuous confidence signal. It comes free with the forward pass that produced the label and answers "how sure was the model about this classification?", which a hard 0/1 label cannot.
• A logprob is the model's subjective probability under its own distribution, not a probability of correctness. A model can assign 0.95 to labels that are right 70% of the time. Treat it as a claim about confidence that calibration (§4) tests.
• Applied social science usually takes the final label at face value and discards the distribution it came from; computational linguists treat token probabilities as the native uncertainty signal. Neither is enough alone — the logprob is informative but must be calibrated against external truth before it carries weight. State the choice in the methods section.
• Prefer the emitted-token logprob over a prompted "rate your confidence HIGH/MEDIUM/LOW" string. Verbalized confidence is generated text subject to the same biases (Kadavath et al. 2022; Tian et al. 2023); it can be made roughly calibrated with effort (Tian et al. 2023), but treat it as a separate, weaker signal and cross-check it rather than substitute it. A long reasoning trace is not a confidence measure either.

2. Collecting Logprobs and Aggregating Multi-Token Labels

• Check compatibility before choosing the model. Logprob support depends on model, endpoint, and reasoning setting; confirm with current documentation and a small test response before a batch. Astra does not support reasoning effort none, so a logprob example from a non-reasoning model may not transfer to it. Keep a validated compatible model for calibration, and mark missing logprobs as unavailable rather than filling them with verbal confidence.
• Enable logprob return. OpenAI chat completions take logprobs=True and top_logprobs=K (K ≤ 20 at time of writing) and return per-token .token, .logprob, and .top_logprobs. vLLM's OpenAI-compatible server exposes the same fields (logprobs=K); Ollama returns per-token logprobs too. A typical script sets api_kwargs["logprobs"] = True and api_kwargs["top_logprobs"] = 5, then stores resp.choices[0].logprobs.content alongside the parsed label.
• Fix decoding. Set temperature=0 and a fixed seed, and record both. Temperature 0 makes the label deterministic on most backends but does not make hosted-API logprob values bit-identical (§6).
• Constrain the output so label tokens are findable. A rigid schema (JSON with a "code" field) lets you locate the label; free prose makes the span ambiguous.
• Aggregate label tokens by character span. A label like civic_commitment spans several sub-word tokens. Reconstruct the output string, locate the label (e.g., the value after "code":), map each token to its character offsets, and select exactly the tokens overlapping the label span — never guessed token indices. Then choose an aggregation and state it; the three are not interchangeable:
• Sum = log P(whole label), the joint probability. Principled, but length-confounded: longer labels score lower. Comparable only across labels of similar token length. (logprob_code_sum)
• Mean per-token logprob (geometric-mean probability) removes most of the length confound; the better default when label lengths differ.
• First-token logprob: cheap and length-invariant, informative only when the labels diverge at the first token. Check that before relying on it; shared prefixes make it useless. (logprob_first_token)
• Store the top alternatives at the first label position (top_alternatives_json). The margin to the runner-up is a separate, often more discriminating signal (§3).

3. Confidence Tiers, Margins, and Triage

• Map the chosen-label probability p = exp(logprob) onto action tiers. A starting scheme:
• p ≥ 0.90 → auto-accept
• 0.65 ≤ p < 0.90 → accept, flag for spot-check
• p < 0.65 → human review
• These cut points are house planning defaults, not values from a cited study, and mean nothing until calibration is checked (§4). If the model is overconfident, a 0.90 cut admits too many errors. Re-derive the cuts from your own reliability diagram at the bin where empirical accuracy crosses the tolerated error rate.
• Use the margin as a second trigger. A small gap to the runner-up (e.g., p_top − p_second < 0.10) marks a coin-toss between two specific labels even when p is high. Flagging on either signal catches more errors than absolute confidence alone, and the runner-up label shows which codebook boundary is ambiguous.
• Route low-confidence items to humans; the tiers allocate scarce review time to genuine coin-tosses rather than a random sample. This is the within-model counterpart of disagreement-based flagging in $text-classification.
• Plan for roughly 10-20% of items in the low tier (house default). A much larger share usually means overlapping labels or an under-specified prompt — return to the codebook before spending review hours.

4. Calibration Assessment

A model is calibrated if, among items labeled with confidence p, a fraction p are correct. Tiers are trustworthy only after this is measured against held-out ground truth.

• Ground truth must be human-adjudicated labels (or another external standard), not the model's modal answer or other model runs. Checking logprobs against model output measures self-consistency, not correctness; within-model confidence and cross-model agreement can both look reassuring while the labels are jointly wrong, because model errors are correlated.
• Reliability diagram. Bin by predicted confidence (e.g., 10 equal-width bins) and plot mean confidence against empirical accuracy. Below the diagonal is overconfidence, above is underconfidence. Modern deep networks, including large transformers, tend toward overconfidence (Guo et al. 2017).
• Expected Calibration Error. ECE = Σ_b (n_b / N) · |acc(b) − conf(b)| (Guo et al. 2017). Report it with the bin count, since it is binning-sensitive, and always with the diagram — different curves can share an ECE.
• Brier score. BS = (1/N) Σ (p_i − y_i)² (Brier 1950). A proper scoring rule that captures calibration and resolution together, so it penalizes hedging everything at 0.5. Report it alongside ECE.
• Calibration is not accuracy. Triage depends on calibration: if the 0.90 bin is 70% accurate, the auto-accept tier leaks errors regardless of overall F1.
• Temperature scaling fixes calibration without changing the argmax label: fit one scalar T on a calibration split and rescale before softmax; it often largely corrects ECE (Guo et al. 2017). Report ECE and Brier before and after, and never fit T on the evaluation data.

5. Using Uncertainty in the Pipeline

• Route, don't discard. Dropping unsure items biases the remaining sample toward what the model finds easy, which is rarely random with respect to the construct.
• Propagate uncertainty. When labels feed a regression or prevalence estimate, carry per-item confidence (observation weights, soft labels, or multiple-imputation draws over the label) instead of hard 0/1 labels. At minimum, rerun the downstream analysis without the low-confidence tier and report whether conclusions move. This connects to the measurement-error correction in $text-classification.
• Report the share of the corpus in each tier. It is a corpus-level quality signal and sizes the review budget.

6. Caveats

• Hosted-API logprobs are unstable. Providers may restrict or change logprob support without notice, and values drift across snapshots and even identical calls; batching, kernel non-determinism, and silent updates perturb them at temperature 0. Pin the exact model version, record the run date, and re-measure calibration after any model update. Open-weight models served locally (vLLM, Ollama — e.g., Llama 3.1 8B or Qwen 2.5 3B) give stable, inspectable logprobs.
• Closed-set tasks only, mostly. Logprob confidence is interpretable for a fixed label set. For open-ended generation, per-token logprobs measure local fluency, not global correctness.

7. Reproducibility and Reporting

• Collection recipe: exact model version or snapshot, decoding settings (temperature, seed, top_logprobs K), the output schema used to locate labels, and the aggregation function with its rationale. Archive the raw per-token logprobs and top-K alternatives, not only the derived scalar.
• Calibration: reliability diagram, ECE with bin count, and Brier score on a held-out human-labeled set; direction and size of miscalibration; whether temperature scaling was applied.
• Triage: thresholds and how they were derived from the diagram, per-tier shares, and the human-review rate.
• Ground-truth source for calibration, stated explicitly.
• For the broader methods checklist (model disclosure, archived prompts, DA-RT/JARS), use $methods-reporting; for codebook and human-validation metrics, $text-classification.

Quality Checks

• [ ] Logprobs enabled (logprobs=True, top_logprobs=K); temperature=0 and a fixed seed recorded
• [ ] Label tokens located by character span; aggregation (first-token / sum / mean) stated, with the sum's length confound acknowledged
• [ ] Top-K alternatives stored and margin to the runner-up computed
• [ ] Logprob confidence not reported as probability of correctness without calibration against external ground truth
• [ ] Reliability diagram against human-adjudicated labels; ECE (with bin count) and Brier reported (Guo et al. 2017; Brier 1950)
• [ ] Direction of miscalibration stated; temperature scaling, if used, fit on a separate split
• [ ] Triage thresholds derived from the reliability diagram, with house defaults re-justified on this corpus
• [ ] Low-confidence items routed to review, not discarded; per-tier shares and review rate reported
• [ ] Downstream analysis rerun without the low-confidence tier (or uncertainty propagated), with sensitivity reported
• [ ] Verbalized confidence and reasoning text not substituted for the token logprob
• [ ] Model snapshot pinned, run date recorded, calibration re-measured after model updates
• [ ] Collection recipe and raw per-token logprobs archived; token-probability vs. face-value convention stated in the methods section