Build Real-Time AI Voice Conversations with Java

Azure AI VoiceLive SDK for Java is a client for real-time, bidirectional voice conversations with AI assistants over WebSocket, built around a reactive VoiceLiveAsyncClient that starts a VoiceLiveSessionAsyncClient for a live streaming connection, authenticated with an API key or the recommended DefaultAzureCredential. Audio must be 24kHz, 16-bit signed little-endian PCM, mono.

Starting a session against a realtime model (e.g. gpt-4o-realtime-preview) opens a subscription to server events. Session options configure turn detection (a sensitivity threshold, prefix padding and silence-duration before ending a turn, and whether interruptions are allowed), input/output audio format and sample rate, noise reduction and echo cancellation, Whisper-based input transcription, the assistant's instructions, and the response modalities (text and/or audio). Audio is streamed to the session as raw PCM16 bytes, and server events are handled as they arrive - session created, user speech started/stopped, streaming response-audio deltas (played as they arrive), response complete, or an error event carrying a message. Voices come from either OpenAI's set (Alloy, Ash, Ballad, Coral, Echo, Sage, Shimmer, Verse) or Azure's own voice types - a standard neural voice, a custom voice tied to an endpoint ID, or a personal voice built from a speaker profile ID and a personal-voice model. Function calling defines a named tool with a description and a JSON parameter schema, attached to the session's tools list alongside the assistant's instructions.

Use it when building a real-time, low-latency voice assistant that needs to stream audio both ways over a live WebSocket session, with server-side turn detection and interruption handling - not a batch speech-to-text or text-to-speech pipeline, which would be a different, non-streaming Azure Speech service.

Input is a live audio stream (24kHz 16-bit mono PCM) plus session configuration (instructions, voice, turn detection, tools). Output is a stream of server events - transcription/turn events, streaming audio-response deltas, function-call requests, and completion/error events. Install it with:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-voicelive</artifactId>
    <version>1.0.0-beta.2</version>
</dependency>

Its own best practices: use the async/reactive client throughout since VoiceLive is fundamentally reactive, configure turn detection for natural conversation flow, enable noise reduction for better speech recognition, handle interruptions gracefully via setInterruptResponse(true), use Whisper for input transcription, and close sessions properly when a conversation ends. Errors surface through the event stream's doOnError/onErrorResume reactive operators, where you can attempt reconnection or cleanup.

It targets OpenAI realtime models (like gpt-4o-realtime-preview) through Azure, with OpenAI's own voice set alongside Azure's standard, custom, and personal neural voices, and Whisper for input audio transcription.

Java developers building real-time, streaming voice assistants - natural turn-taking, interruption handling, and function calling over live audio - rather than a batch transcription or text-to-speech workflow.