Skill v1.0.1
currentAutomated scan100/100+4 new
name: azure-ai-voicelive-java description: Azure AI VoiceLive SDK for Java. Real-time bidirectional voice conversations with AI assistants using WebSocket. risk: critical source: community date_added: '2026-02-27' metadata: version: "1.0.1"
Azure AI VoiceLive SDK for Java
Real-time, bidirectional voice conversations with AI assistants using WebSocket technology.
Installation
<dependency><groupId>com.azure</groupId><artifactId>azure-ai-voicelive</artifactId><version>1.0.0-beta.2</version></dependency>
Environment Variables
AZURE_VOICELIVE_ENDPOINT=https://<resource>.openai.azure.com/AZURE_VOICELIVE_API_KEY=<your-api-key>
Authentication
API Key
import com.azure.ai.voicelive.VoiceLiveAsyncClient;import com.azure.ai.voicelive.VoiceLiveClientBuilder;import com.azure.core.credential.AzureKeyCredential;VoiceLiveAsyncClient client = new VoiceLiveClientBuilder().endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT")).credential(new AzureKeyCredential(System.getenv("AZURE_VOICELIVE_API_KEY"))).buildAsyncClient();
DefaultAzureCredential (Recommended)
import com.azure.identity.DefaultAzureCredentialBuilder;VoiceLiveAsyncClient client = new VoiceLiveClientBuilder().endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT")).credential(new DefaultAzureCredentialBuilder().build()).buildAsyncClient();
Key Concepts
| Concept | Description | |
|---|---|---|
VoiceLiveAsyncClient | Main entry point for voice sessions | |
VoiceLiveSessionAsyncClient | Active WebSocket connection for streaming | |
VoiceLiveSessionOptions | Configuration for session behavior |
Audio Requirements
- Sample Rate: 24kHz (24000 Hz)
- Bit Depth: 16-bit PCM
- Channels: Mono (1 channel)
- Format: Signed PCM, little-endian
Core Workflow
1. Start Session
import reactor.core.publisher.Mono;client.startSession("gpt-4o-realtime-preview").flatMap(session -> {System.out.println("Session started");// Subscribe to eventssession.receiveEvents().subscribe(event -> System.out.println("Event: " + event.getType()),error -> System.err.println("Error: " + error.getMessage()));return Mono.just(session);}).block();
2. Configure Session Options
import com.azure.ai.voicelive.models.*;import java.util.Arrays;ServerVadTurnDetection turnDetection = new ServerVadTurnDetection().setThreshold(0.5) // Sensitivity (0.0-1.0).setPrefixPaddingMs(300) // Audio before speech.setSilenceDurationMs(500) // Silence to end turn.setInterruptResponse(true) // Allow interruptions.setAutoTruncate(true).setCreateResponse(true);AudioInputTranscriptionOptions transcription = new AudioInputTranscriptionOptions(AudioInputTranscriptionOptionsModel.WHISPER_1);VoiceLiveSessionOptions options = new VoiceLiveSessionOptions().setInstructions("You are a helpful AI voice assistant.").setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY))).setModalities(Arrays.asList(InteractionModality.TEXT, InteractionModality.AUDIO)).setInputAudioFormat(InputAudioFormat.PCM16).setOutputAudioFormat(OutputAudioFormat.PCM16).setInputAudioSamplingRate(24000).setInputAudioNoiseReduction(new AudioNoiseReduction(AudioNoiseReductionType.NEAR_FIELD)).setInputAudioEchoCancellation(new AudioEchoCancellation()).setInputAudioTranscription(transcription).setTurnDetection(turnDetection);// Send configurationClientEventSessionUpdate updateEvent = new ClientEventSessionUpdate(options);session.sendEvent(updateEvent).subscribe();
3. Send Audio Input
byte[] audioData = readAudioChunk(); // Your PCM16 audio datasession.sendInputAudio(BinaryData.fromBytes(audioData)).subscribe();
4. Handle Events
session.receiveEvents().subscribe(event -> {ServerEventType eventType = event.getType();if (ServerEventType.SESSION_CREATED.equals(eventType)) {System.out.println("Session created");} else if (ServerEventType.INPUT_AUDIO_BUFFER_SPEECH_STARTED.equals(eventType)) {System.out.println("User started speaking");} else if (ServerEventType.INPUT_AUDIO_BUFFER_SPEECH_STOPPED.equals(eventType)) {System.out.println("User stopped speaking");} else if (ServerEventType.RESPONSE_AUDIO_DELTA.equals(eventType)) {if (event instanceof SessionUpdateResponseAudioDelta) {SessionUpdateResponseAudioDelta audioEvent = (SessionUpdateResponseAudioDelta) event;playAudioChunk(audioEvent.getDelta());}} else if (ServerEventType.RESPONSE_DONE.equals(eventType)) {System.out.println("Response complete");} else if (ServerEventType.ERROR.equals(eventType)) {if (event instanceof SessionUpdateError) {SessionUpdateError errorEvent = (SessionUpdateError) event;System.err.println("Error: " + errorEvent.getError().getMessage());}}});
Voice Configuration
OpenAI Voices
// Available: ALLOY, ASH, BALLAD, CORAL, ECHO, SAGE, SHIMMER, VERSEVoiceLiveSessionOptions options = new VoiceLiveSessionOptions().setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)));
Azure Voices
// Azure Standard Voiceoptions.setVoice(BinaryData.fromObject(new AzureStandardVoice("en-US-JennyNeural")));// Azure Custom Voiceoptions.setVoice(BinaryData.fromObject(new AzureCustomVoice("myVoice", "endpointId")));// Azure Personal Voiceoptions.setVoice(BinaryData.fromObject(new AzurePersonalVoice("speakerProfileId", PersonalVoiceModels.PHOENIX_LATEST_NEURAL)));
Function Calling
VoiceLiveFunctionDefinition weatherFunction = new VoiceLiveFunctionDefinition("get_weather").setDescription("Get current weather for a location").setParameters(BinaryData.fromObject(parametersSchema));VoiceLiveSessionOptions options = new VoiceLiveSessionOptions().setTools(Arrays.asList(weatherFunction)).setInstructions("You have access to weather information.");
Best Practices
- Use async client — VoiceLive requires reactive patterns
- Configure turn detection for natural conversation flow
- Enable noise reduction for better speech recognition
- Handle interruptions gracefully with
setInterruptResponse(true) - Use Whisper transcription for input audio transcription
- Close sessions properly when conversation ends
Error Handling
session.receiveEvents().doOnError(error -> System.err.println("Connection error: " + error.getMessage())).onErrorResume(error -> {// Attempt reconnection or cleanupreturn Flux.empty();}).subscribe();
Reference Links
| Resource | URL | |
|---|---|---|
| GitHub Source | https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-voicelive | |
| Samples | https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-voicelive/src/samples |
When to Use
This skill is applicable to execute the workflow or actions described in the overview.
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
These are beta-version fragments, not a complete audio application. Verify endpoint/model/voice availability and SDK signatures against the deployed resource, supply imports/audio helpers and retain one managed event subscription with shutdown/error handling. Audio/transcription is transmitted to a cloud service: get consent for capture and transmission, minimize retention and protect transcripts. Function-call arguments are untrusted; validate them and authorize any external action separately before invoking it.