christos-antono/npctalk/fish-voice-server

A Fish Audio voice server for games

server/drama3.py (default) and server/s2pro.py are Modal apps with the same API, so a game switches models by changing one URL. Both run Fish Audio's open weights through SGLang-Omni on one H100 and stream audio as it's generated.

Pick the model

Deploy (Modal)

uv tool install modal && modal setup
modal secret create npcvoice-auth NPCVOICE_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
modal run server/drama3.py::download_weights        # once: ~40 GB into the npcvoice-models volume
npctalk voice make --name Narrator --audio narrator.wav && npctalk voice upload voices/Narrator
modal deploy server/drama3.py                       # prints https://<workspace>--npcvoice-drama.modal.run
export NPCVOICE_URL=https://<workspace>--npcvoice-drama.modal.run NPCVOICE_TOKEN=<the token>
npctalk health                                      # waits through a cold start (1-3 min)
npctalk say "[laughing] Nah, you trippin'." --voice Narrator/clone --out test.wav

Everything is named npcvoice-* (apps, volumes, secrets) so it stays apart from the rest of a Modal workspace. Keep the URL private: /health needs no token and wakes the GPU.

The API

POST /speak    Authorization: Bearer <NPCVOICE_TOKEN>
  {"text": "[angry] Get out of my face!", "voice": "Trevor/clone"}            a voice prepared on the server
  {"text": "...", "reference_audio_url": "https://.../me.wav", "reference_text": "what me.wav says"}
  optional: "temperature", "top_p", "multi" (all clips in the voice's folder), "best_of" (1-8, drama3.py)
→ 200, X-Sample-Rate: 44100, body: 16-bit mono PCM, streamed as it's generated
  (with best_of > 1: the whole line at once, and X-Takes: per-take likeness / word error / length)
GET /health    no auth → {"ok": true, "model": "drama-3", "voices": 21}

• Voices live in the volume at /voices/<Name>/clone.wav + clone.lab (transcript), plus ref_0..2 and mood

folders (/voices/<Name>/angry/ref_0.wav). The server encodes every clip when it starts, so a named voice costs nothing per request; a clip sent by URL costs ~9 s the first time. Restart the server after uploading voices (modal app stop npcvoice-drama).

• Tags go where they happen in the text: "[sigh] Man... [laughing] nah.". Drama 3 and S2 Pro both voice

them; other engines treat them as hints.

• Client: examples/gta-v/mod/Tts.cs (ModalTts, C#) and npctalk/server.py (Python) both read the stream

in 4 KB chunks and start playing at the first chunk.

Settings that were measured (Lamar, 10 lines, voice match = ECAPA similarity to 60 real clips)

So SAMPLING = {"temperature": 0.7, "top_p": 0.8} is the server default, and pre-rendered lines use best of 8.

best_of: for lines made ahead

About 1 take in 10 rambles to the token limit (47.6 s for a 15-word line) or trails off into noises. For lines that don't need streaming (cutscenes, a director's clips) best_of: N renders N takes in one batch and:

1. stops each take just past the longest the words could take (1.5 + 0.45 s per word, + 1 s);
2. drops takes that are too long or that a small ASR (Whisper base, on the GPU) can't hear the line in

(word error above the clearest take's + 0.1, or above 0.5);

1. keeps the one most like the reference by ECAPA plus WavLM similarity, each scaled across the takes.

Picking by one model alone over-fits it: WavLM-only picks scored 0.692 at 4 takes; ECAPA-only raised ECAPA a lot but WavLM less; both together kept most of each. An unchecked pick scored 0.708 but let a 14-second whimper through. Best of 8 takes ~4-6 s per line with the scoring models on the GPU.

Keep-warm, cold starts, cost

• MIN_CONTAINERS = 0, SCALEDOWN = 300: the H100 (about $4/hour) stops 5 minutes after the last request; the

next request waits 1-3 minutes (weights load, kernels warm up, every voice is encoded).

• A game should ping while it runs (the GTA mod sends a one-word line every 90 s of silence) and fall back to

a no-GPU engine (MiniMax on fal) while the GPU wakes.

• For a demo without the game: modal deploy server/keepwarm.py, then

modal run server/keepwarm.py --url $NPCVOICE_URL --hours 12 (it says what that costs: ~$45-50).

Fine-tuning S2 Pro (optional, needs training)

server/finetune/: prepare_dataset.py (Demucs → Whisper word timings → 2-15 s clips → speaker filter → loudness), train.py (LoRA r8, α16, lr 1e-4; step 250 was best), fish_finetune.py (two fixes fish-speech needed for S2 Pro). Read references/finetune.md before starting.

Gotchas for all of this: references/gotchas.md.