elevenlabs/plugin/architect-llm-selection

LLM selection for agents

Resolve constraints in this sequence. Each step narrows the candidate set; never widen it later. Everything is over the ConvAI REST API (host https://api.elevenlabs.io, header xi-api-key: $API_KEY).

1. Explore the agent. Read the current LLM, workflow, and tools with GET /v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID first.
2. Region (infrastructural, non-negotiable). Determine deployment topology. EU and India customers hit isolated deployments with restricted model lists that apply to all tiers. A model hidden by region cannot be enabled; do not offer to.
3. Compliance entitlements (contract-gated). If the workspace requires HIPAA, PCI, or ZRM, filter to compliant models only. These are workspace-level flags managed by sales/CSMs, never self-serve.
4. Capability fit (intelligence / latency / cost). Within whatever survives steps 1-3, recommend on task requirements.

If a constraint in step 2 or 3 eliminates the model the user wanted, say so plainly and route to their CSM or a support form. Do not present a capability recommendation that violates a hard constraint.

Step 1: region gate

Restriction is by deployment topology, not a toggle. Never offer to enable a region-hidden model.

Step 2: compliance gate

Only relevant if the workspace has the corresponding entitlement. You cannot see workspace flags, so never assert an entitlement is active; the safe line is "your CSM can confirm this for your workspace." Never offer to enable HIPAA, PCI, or ZRM; they require contract changes.

• HIPAA requires all three: enterprise tier with force_logging_disabled=true, an LLM in the HIPAA-compliant set, and a signed BAA (a contract artifact you cannot verify; route BAA questions to sales). Approved families include Claude 3.5/3.7 Sonnet, Claude 3 Haiku, Claude Haiku 4.5, Claude Opus 4.7, Claude Sonnet 4 / 4.5 / 4.6, custom-llm, Gemini 1.5/2.0/2.5 flash and pro variants, gemini-3.1-flash-lite, and speech-engine. Models outside the list are not HIPAA-eligible regardless of tier; preview/GA-pending Gemini models are excluded.
• PCI is deny-by-default when on (pci_compliance_required). Only pre-approved telephony providers, webhook domains, MCP URLs, and integration IDs work. Adding an integration requires CSM review; offer a support form, do not promise self-serve.
• ZRM disables audio/transcript storage; effective only when force_logging_disabled=true. Custom LLM plus ZRM additionally needs is_convai_custom_llm_with_zrm_allowed. No Stripe SKU; part of the enterprise contract.

Step 3: capability fit

Recommend within the surviving candidate set. Weigh three axes against the agent's job.

• Latency matters most for voice. In an STT to LLM to TTS loop, LLM time-to-first-token sits on the critical path and is the dominant lever on perceived delay. For real-time agents, bias toward the fastest tier that clears the quality bar.
• Lowest latency: gemini-3.1-flash-lite / nano / small qwen / *-mini tiers and speech-engine.
• Balanced: gemini-3.5-flash, gpt-4o, gpt-5-mini, claude-haiku-4-5.
• Highest capability, higher latency: claude-sonnet-4-6, claude-opus-4-7, gpt-5.x, gemini-3.x-pro (where region/compliance permits).
• Intelligence. Reserve the top tier for genuinely hard reasoning, complex multi-tool orchestration, or nuanced instruction-following. Most transactional voice agents (booking, triage, FAQ, routing) run well on a mid tier and feel snappier. Over-provisioning intelligence usually costs latency the user will notice for quality they will not.
• Cost. Per-token price scales steeply with capability tier. Match tier to task; do not put a frontier model behind a deterministic IVR flow. For high-concurrency deployments, latency and cost compound.

Quick heuristic

Apply the choice

Set the model with PATCH /v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID and body {"conversation_config":{"agent":{"llm":"<model-id>"}}}. This is branch-scoped and ships on merge.

Output discipline

• State the binding constraint explicitly ("In the EU deployment, X is not available, so among allowed models...").
• Give one primary recommendation plus a fallback, with the reason (latency vs intelligence vs cost).
• For anything contract-gated, end with the route: CSM for entitlement confirmation, a support form for new integrations/documents, and https://compliance.elevenlabs.io/ for attestations and signed DPA/BAA/sub-processor requests.
• Never claim an entitlement is active, never offer to enable a contract feature or a region-restricted model.