version: "1.0.0" name: xb-add-ai description: Add AI behavior to an XR Blocks application. Use when implementing Gemini or OpenAI queries, Gemini Live voice or vision, generated images, scene-grounded responses, or model-requested tools with an observable XR outcome.

Add AI behavior

Build complete AI behavior: intentional input, provider, validated response, observable result, and owned cleanup.

1. Write the AI contract

Name the input, provider branch, response modality, application consumer, data sent off-device, user disclosure, concurrency rule, and visible waiting, unavailable, empty, disconnected, and error states. Treat prompts, microphone, camera, screenshots, and scene context as separate disclosures.

Complete this step when success changes scene, UI, speech, or an asset and each disclosed input is necessary for that result.

2. Prove the provider path

Read `../../docs/docs/manual/AI.mdx` and `references/current-api.md`. Verify planned symbols in `../../src/xrblocks.ts`, `../../src/ai/AI.ts`, and the selected provider. For Live, tools, camera, or context grounding, also read `references/live-tools-grounding.md`.

Use `../../templates/06_ai_query/` for bounded queries and `../../templates/07_ai_live_assistant/` for the common Live loop.

Complete this step when every symbol is public and the selected provider supports every required operation and modality.

3. Select one primary branch

• Use xb.ai.query({prompt}) for a bounded provider-neutral text request.
• Use Gemini multipart input when a bounded request needs an image or typed

parts.

• Use Gemini Live for an ongoing audio or video conversation.
• Use xb.ai.generate() when the image result becomes an actual app asset.
• Use a narrow xb.Tool when the model requests a validated application

action.

Complete this step when latency, modality, provider support, data disclosure, and resource ownership match the contract.

4. Implement the complete behavior

Configure and enable the selected provider before initialization. Check availability at use time, catch provider failures, and prevent unintended concurrent starts. Accept missing text, null, rejected requests, interruption, and disconnection as normal observable states.

For Live, register callbacks before connecting, make readiness depend on onopen, use provider-typed nested realtime inputs, and route remote close, local stop, error, and disposal through one idempotent cleanup owner. Stop the Live session, microphone, playback, frame timers, and app-owned tracks.

Validate tool arguments again at execution and resolve targets through an allowlist or current scene context. Keep consequential actions behind application authorization or confirmation.

Complete this step when one intentional input reaches its consumer, duplicate starts are controlled, every failure is visible, and every acquired resource has one cleanup owner.

5. Prove and hand off provider behavior

Build or type-check the app and run startup smoke that does not require a long conversation. Check configuration, availability guards, request and response types, visible states, tool validation, disclosures, and cleanup. Use existing safe prototype credentials only for one narrow connectivity check.

Give the user the trigger, required credentials and permissions, data sent off-device, waiting/success/error states, and recovery steps. For Live, include open, interruption, remote close, local stop, and restart. For tools, include valid, invalid, unknown, denied, and failed outcomes.

Finish when the user can evaluate real provider behavior without discovering setup, disclosure, expected states, or cleanup from source.