christos-antono/npctalk/make-npcs-talk

Make NPCs talk

The finished loop, as built for GTA V (examples/gta-v), with times measured in the game:

player's mic ──► voice activity ──► speech-to-text ──► who? + scene ──► LLM in character ──► voice server ──► NPC speaks
   (open mic,       (end of the       (fal Nemotron,     (the game)      (Cerebras, with      (Fish Audio on     (streamed PCM,
    push-to-talk,    sentence)         0.4-0.8 s)                         persona + lines,    an H100, first     panned, subtitles,
    or typing)                                                            0.3-1.0 s)          audio ~0.4 s)      mouth, action)
                                                                │
                                                                └──► action choice (Jev, ~0.1 s, in parallel) ──► animation

From the end of the player's sentence to the NPC's voice: 1.2-2.2 s warm with push-to-talk (logged exchanges: 1.15 s, 1.20 s, 1.57 s, 2.19 s); the open mic adds the 0.7 s of silence it waits for before deciding the sentence ended. The LLM sometimes spikes to 5-8 s, so log every step's time per exchange.

The loop: build it in this order

1. Recon the game. Version, how code runs inside it (script hook, mod loader, plugin API, or your own engine),

single-player or not, how other mods find nearby characters and play sounds. Read references/game-hooks.md for the engine, and knowledge/ for anyone who already did this game.

1. Pick the engines (table below). Default: Fish Audio Drama 3 on Modal for voices, Cerebras for the LLM, fal

for speech-to-text.

1. Stand the voice server up first, outside the game (skill fish-voice-server): npctalk health, then

npctalk say "..." --voice Name/clone. A game is the worst place to debug a GPU server.

1. Make reference voices for two or three characters (skill voice-references).
2. Vertical slice in the game: one NPC, typed text → LLM → voice → playback. Implement the four hooks below,

nothing else. Log stt / llm / tts / total for every exchange.

1. Voice input (skill voice-input): push-to-talk first, then the open mic.
2. Characters (skill npc-dialogue): persona files, the character's own lines as style samples, scene

facts, per-NPC history, output cleaning.

1. Actions (skill npc-actions): what they physically do, and doing what they're told.
2. Verify in the real game, with a dev mode that shows the target, the timings and the mod's state (below).
3. Show it (skill npc-showcase-clips), and if voice likeness matters, measure it (skill

voice-likeness-eval) before claiming a setting is better.

1. Write it down: a field note in knowledge/games/<game>/ (see knowledge/README.md).

The four hooks a game has to provide

Everything else is game-free code. In the GTA V example about 2,450 of the mod's ~5,500 lines (dialogue, LLM, voice clients, streaming playback, speech-to-text, open mic, action picking) build and run outside the game: examples/gta-v/tools/npcvoice-check compiles exactly those files into a console app.

Keep the hooks thin. If a game exposes them, the rest of this repo's code carries over.

Choosing engines

Costs: an H100 runs about $4/hour while it's up; it scales to zero 5 minutes after the last request, and the next request waits 1-3 minutes for a cold start. The game pings the server while it runs, so it stays warm exactly while someone plays (server/keepwarm.py holds it up for a demo without the game).

Verify like you mean it

• Log every exchange with each step's time and what was heard and said. Most "it doesn't work" reports are

answered by the log (in GTA V: "open mic -> nobody near" 755 times = the target lookup found nobody).

• Show the mod's state on screen. A mod switched off with a hotkey looks exactly like a broken one; say

"off" somewhere visible. Show who would be talked to (a marker) in a dev mode.

• Test the network half without the game (npcvoice-check in the GTA example, npctalk say for the

server).

• Measure, don't eyeball, voice likeness (skill voice-likeness-eval). Two runs of ten lines fooled us

once; ten runs didn't.

Rules

• Single-player, or servers you run. Never inject into online games with anti-cheat (for GTA V: never

launch GTA Online with a script hook installed).

• Voices you have the rights to. Your own, an actor who agreed, a licensed or generated voice. Cloning

a game's characters for your own single-player experiment is one thing; publishing those clones is another. Never commit voice clips, game audio, or the game's dialogue (npctalk check blocks audio and transcripts).

• No secrets in the repo. Keys come from environment variables. A server URL is a secret too when its

/health needs no token: anyone with it can keep your GPU awake.

• Content. NPCs swear if the game's characters do. Slurs are filtered by default and opt-in per persona

(skill npc-dialogue).

• Ask first before driving the user's keyboard and mouse in a running game, installing into game

folders, or publishing anything.

Full reasoning: references/safety.md. Architecture and file map: references/architecture.md.