Build Real-Time Voice AI Assistants

@azure/ai-voicelive is a real-time voice AI SDK for building bidirectional voice assistants with Azure AI in both Node.js (LTS 20+) and modern browsers, authenticating via DefaultAzureCredential (or InteractiveBrowserCredential in-browser) or an AzureKeyCredential. A VoiceLiveClient starts a VoiceLiveSession - a WebSocket connection exposing updateSession(), a subscription-based subscribe() for event handlers, sendAudio() for streaming input, addConversationItem() for messages/function outputs, and sendEvent() for raw protocol events.

Session configuration sets modalities (text and/or audio), system instructions, a voice (Azure Standard, Azure Custom tied to an endpoint ID, Azure Personal built from a speaker-profile ID, or an OpenAI voice like "alloy"), turn detection, input/output audio format, and function-calling tools with a toolChoice policy. Turn detection offers basic server_vad (a threshold plus prefix-padding and silence-duration timing) or Azure's smarter semantic VAD in three variants - default, English-optimized, and multilingual. Audio formats span PCM16 at 24kHz (default, highest quality), 16kHz (voice assistants), and 8kHz (telephony), plus G.711 mu-law and a-law for US/EU telephony respectively. Supported realtime models are gpt-4o-realtime-preview (high-quality conversational AI), gpt-4o-mini-realtime-preview (fast/efficient), and phi4-mm-realtime (cost-effective multimodal).

The event-subscription pattern covers connection lifecycle (connected/disconnected/error), session lifecycle (created/updated), VAD speech start/stop, streaming transcription and text/audio deltas, response lifecycle, and function-call argument completion - the last of which is where a handler parses the call arguments, executes the function, sends the result back as a function_call_output conversation item, and triggers a new response. Errors are typed - VoiceLiveConnectionError, VoiceLiveAuthenticationError, VoiceLiveProtocolError - dispatchable via instanceof inside the onError handler, alongside a separate onServerError for protocol-level server errors. Browser usage needs a bundler (Vite, webpack) and requests microphone access via navigator.mediaDevices.getUserMedia, feeding a 24kHz AudioContext.

Use it when building a real-time, low-latency, bidirectional voice assistant in JavaScript/TypeScript for Node.js or the browser, choosing a turn-detection and voice configuration to match the use case - telephony audio formats and G.711 codecs for phone-based scenarios, semantic VAD for more natural turn-taking than basic VAD. It is not for batch speech-to-text or text-to-speech - that's a separate, non-streaming Azure Speech service.

Input is a live audio stream plus session configuration (voice, turn detection, tools, modalities). Output is a stream of typed server events - transcription, streaming text/audio deltas, function-call requests, and connection/session/response lifecycle events. Install it with:

npm install @azure/ai-voicelive @azure/identity

Its own best practices: always use DefaultAzureCredential rather than hardcoding API keys, include both text and audio modalities for voice assistants, prefer Azure Semantic VAD over basic server VAD for better turn detection, handle all three error types, close subscriptions when done, and use PCM16 at 24kHz for best audio quality.

It targets Azure-hosted realtime models (GPT-4o and Phi multimodal variants), Azure's own neural/custom/personal voices alongside OpenAI voices, and browser microphone access via the Web Audio API and getUserMedia.

JavaScript/TypeScript developers building real-time, bidirectional voice assistants for Node.js or browser apps who need typed events, semantic turn detection, and function calling over a live audio session.