elumenotion/guideants/audiocpp-extended

audio.cpp extended (experimental)

GuideAnts exposes audio.cpp through curated wrapper services with a deliberately narrow contract ({text, voice, speed} against a fixed model catalog). The engine underneath supports much more. This skill reaches that extra surface without any GuideAnts changes, using the fact that the sandbox runs in the same container as the audio services.

Everything here is experimental. Run the probe first, trust its report over this document, and tell the user plainly when a route is blocked.

Always start with the probe

python3 Skills/audiocpp-extended/scripts/probe.py

It prints a JSON report saying which of the four routes below are open in this deployment, plus the evidence (ports reachable, dirs writable, binary present, HF egress, GPU state). Do not attempt a route the probe marked blocked.

Voice cloning consent rule

Cloning a voice from a reference clip (voice_ref, Routes 1/3/4) is a supported, intended use of this skill when the speaker consents: the user states the clip is their own voice, or that they have the speaker's permission. In that case, proceed — do not refuse consented cloning as a "security risk"; producing a WAV deliverable in the workspace impersonates no live system. Decline only when the request is to imitate a third party without consent (a celebrity, a colleague, a public figure), and say that consent is the missing piece.

The four routes (try in this order)

Route 1 — raw engine API on the already-loaded model (cheapest)

The wrapper services each spawn a real audiocpp_server on localhost: ASR engine at 127.0.0.1:18082, TTS engine at 127.0.0.1:18084 (running only while the wrapper has a model loaded). Calling them directly unlocks, on the model GuideAnts already loaded:

• Voice cloning from a user-supplied reference clip (voice_ref = an absolute

path to a WAV in your workspace — same filesystem, so the engine can read it), optionally with reference_text.

• Deterministic synthesis via seed.
• Language forcing via language (the .NET wire never sends lang_code).
• Voice-design `instructions` (only on a model loaded with the vdes task,

i.e. the VoiceDesign catalog entry).

• ASR on a workspace file by path with a language hint via

/v1/audio/transcriptions — no upload, no wrapper contract.

python3 Skills/audiocpp-extended/scripts/engine_tool.py speech "Hello there" \
  -o cloned.wav --voice-ref user_voice.wav --seed 42
python3 Skills/audiocpp-extended/scripts/engine_tool.py transcribe clip.wav --language de
python3 Skills/audiocpp-extended/scripts/engine_tool.py voices

engine_tool.py speech auto-detects the TTS engine's model id from the wrapper's /health (catalogEntryId). The ASR engine's model id is always qwen3-asr.

Route 2 — sideload a non-catalog ASR model through the wrapper

The ASR wrapper (unlike TTS) accepts a bare directory under /models-local/asr even when it is not in the catalog: /admin/load with a model_id whose leaf matches the directory name. The engine family defaults to qwen3_asr (override requires the GA_ASR_ENGINE_FAMILY env at service start, which a skill cannot set — so this route is only useful for other qwen3_asr-family snapshots).

python3 Skills/audiocpp-extended/scripts/fetch_model.py <hf-repo-id> --dest /models-local/asr/<DirName>
curl -s -X POST http://127.0.0.1:8082/admin/load -H "Content-Type: application/json" -d '{"model_id": "<DirName>"}'

Requires /models-local/asr to be writable from the sandbox — the probe checks. Warning: this replaces the user's loaded ASR model. Ask before doing it, and offer to reload the previous model afterwards ({"model_id": "qwen3_asr_0_6b"}).

Route 3 — spawn a private engine for a deferred/non-catalog TTS model

The TTS wrapper is strictly catalog-gated, but the container ships the full audiocpp_server binary (/usr/local/bin/audiocpp_server) with every released loader — including the families GuideAnts defers (Qwen3 CustomVoice, VibeVoice, MioTTS, VoxCPM2, PocketTTS, Vevo2). The blockers that kept them out of the catalog are download/packaging chores a script can do by hand. So:

1. Download the model with fetch_model.py (per-family recipe in

references/deferred-models.md — composite repos, file scoping, conversions).

1. Spawn your own engine on a free port (default 18099):

python3 Skills/audiocpp-extended/scripts/spawn_engine.py start \
  --path /models-local/tts/Qwen3-TTS-12Hz-1.7B-CustomVoice --family qwen3_tts --task tts
python3 Skills/audiocpp-extended/scripts/spawn_engine.py status   # poll until ready
python3 Skills/audiocpp-extended/scripts/engine_tool.py speech "Hi" \
  --engine-url http://127.0.0.1:18099 --model Qwen3-TTS-12Hz-1.7B-CustomVoice --voice Vivian -o hi.wav
python3 Skills/audiocpp-extended/scripts/spawn_engine.py stop     # always stop when done

Ground rules for this route:

• VRAM: a second engine competes with the wrapper's loaded models. Prefer

asking the user to unload the GuideAnts TTS model first (Settings, or curl -X POST http://127.0.0.1:8084/admin/unload) — ask, never do it silently.

• Script budget is ~5 minutes per call. spawn_engine.py start detaches the

engine and returns; big models may still be warming up — call status in a follow-up script instead of blocking. Downloads of multi-GB models may need several fetch_model.py calls (it resumes; already-complete files are skipped).

• Always `stop` the private engine when the task is done or the user moves on.
• If the detached engine does not survive between script calls (probe reports

this), Route 3 is limited to synth requests issued in the same script call that spawned it — still workable for small models, not for slow-warmup ones.

Route 4 — the user's host-native audiocpp_server

host.docker.internal:8080 (or AUDIOCPP_ENGINE_URL). Same raw API as Route 1, but whatever models and loaders the user's own build has — including families the container binary lacks (e.g. a fork built with Kokoro or Parakeet enabled). TTS-only from the sandbox: its /v1/audio/transcriptions wants host-local paths that sandbox files are not.

Scenario recipe: speaker diarization (who spoke when)

Diarization is a whole task class GuideAnts has no local support for (only the Azure cloud transcription provider does it) — but the container binary ships the sortformer_diar loader and a generic /v1/tasks/run route. This is Route 3 with a small model (~hundreds of MB, usually fits without unloading anything):

python3 Skills/audiocpp-extended/scripts/fetch_model.py nvidia/diar_sortformer_4spk-v1 \
  --dest /models-local/asr/diar_sortformer_4spk-v1 --exclude diar_sortformer_4spk-v1.nemo
python3 Skills/audiocpp-extended/scripts/spawn_engine.py start \
  --path /models-local/asr/diar_sortformer_4spk-v1 --family sortformer_diar --task diar
python3 Skills/audiocpp-extended/scripts/diarize.py meeting.mp3 -o meeting
python3 Skills/audiocpp-extended/scripts/spawn_engine.py stop

(Check whether /models-local/asr/diar_sortformer_4spk-v1/model.safetensors already exists before downloading — the model persists across runs.)

diarize.py converts the input to the 16 kHz mono WAV the engine requires, fetches speaker turns, merges/filters them, and — when the GuideAnts ASR engine is up (Route 1) — transcribes each turn so the deliverable is a speaker-labeled transcript (<base>.transcript.txt + <base>.diarization.json in Output/). Pass --turns-only to skip labeling, --language as an ASR hint. Know the limits: offline only, at most 4 speakers (model variant), and speaker ids are arbitrary labels — ask the user who is who if they want real names. For long recordings the per-turn labeling can hit the script budget; the JSON then says "partial": true — just rerun diarize.py (the diar pass itself is fast).

What a skill cannot do

• Load a non-catalog model through the TTS wrapper (/admin/load hard-rejects

anything outside the manifest) — hence Route 3.

• Use loaders that are not compiled into the container binary: Kokoro

(kokoro_tts) is not in audio.cpp release builds and parakeet_tdt's loader is commented out upstream. No amount of downloading fixes a missing loader; only Route 4 against a custom host build can serve those.

• Set service-level env (GA_ASR_ENGINE_FAMILY, GA_*_CUDA_VISIBLE_DEVICES, …) —

those need a compose/.env change, i.e. actual GuideAnts configuration.

• Show up in GuideAnts' voice-picker UI. Deliverables are WAV/text files in

Output/; the live phone/voice path keeps using the built-in pipeline.

References

• references/engine-api.md — raw engine endpoints, request fields, the engine

config-file schema spawn_engine.py writes, task tokens (tts/clon/vdes).

• references/deferred-models.md — per-family recipes for the deferred models and

the exact blocker each script has to compensate for.

Reporting results

This skill exists to answer "is it possible?". Whatever happens, end by telling the user which route you used, what worked, what was blocked and why (quote the probe/report evidence), so the experiment log stays honest.