christos-antono/npctalk/voice-likeness-eval

Is the voice actually better?

Ears are good at "that's wrong" and bad at "that's 3% better". This repo's numbers all come from one scorer (server/finetune/train.py, class Scorer), run on a GPU in Modal:

• Voice match: cosine similarity of the ECAPA speaker embedding (speechbrain spkrec-ecapa-voxceleb) of the

generated line to the centroid of 60 real clips of the character that were never used as references. Real clips of the same character score ~0.8 against it; 0.6-0.8 is the range where engines differ.

• Word error: Whisper large-v3 transcribes the line; WER against the text, with [tags] and punctuation

removed. Slang ("busta" → "buster", "Ballas" → "ball is") puts a floor of ~5-8% under every engine, so compare WER between engines, not to zero.

• Ten test lines in the character's style with a spread of moods (TEST_LINES in train.py).

modal run server/finetune/train.py::score_extra --model drama3best8a,drama3best8b   # <model>_<line>.wav files
modal run server/finetune/train.py::pick_study_run --models run1,run2,...           # every take, both models

score_extra adds <model>_<line>.wav files from data/finetune/<speaker>/ to scores.json and rebuilds compare.html (every line from every engine, playable, with what Whisper heard).

Rules that kept us honest

1. Compare per line. Average each engine over its runs per line, then a paired t-test over the 10 lines

(|t| > 2.26 is p < 0.05 with 9 degrees of freedom). Report "better on 8 of 10 lines" next to the mean.

1. Run it more than twice. Identical settings, two runs: 0.715 and 0.645. One lucky run made a sampling

change look like +0.08 when it was +0.017. With 10 runs of single takes, the baseline settled at 0.660.

1. Report every run (0.704 (runs 0.702 / 0.706)), and say when a difference isn't significant. A tie is

a result.

1. Correct yourself in the same place you claimed it. If a later run undoes a claim, say so where the

claim was made.

1. Don't let the picker grade itself. If a setting picks takes by a speaker model, check the picks with a

different one too. Here: picks by ECAPA raised ECAPA a lot and WavLM a little; picks by WavLM raised WavLM a lot and ECAPA less; both together kept most of each. If only the picker's own metric moves, it's gaming the metric, not cloning better.

Simulate before you deploy

Rendering N takes per line and scoring every take lets you test picking rules offline. 100 takes (10 per line) gave, at 8 takes per line: one take 0.660; WavLM pick 0.706; ECAPA pick 0.723; both 0.721 with the lowest word error (7.4%); the true best of 8 by real-clip ECAPA 0.732 (the ceiling). The deployed rule measured 0.704 directly (2 runs), close to its simulation.

What was measured (Lamar, 10 lines)

Without training, Drama 3 with best of 8 ties S2 Pro's best zero-shot result; only the fine-tune is clearly higher. Likeness isn't everything: Drama 3 acts [crying] and [laughing] better, and that's a listening test, not a number.