Build Real-Time Voice AI Assistants

Azure.AI.VoiceLive is a real-time voice AI SDK for building bidirectional voice assistants on Azure AI, with a client hierarchy of VoiceLiveClient creating a VoiceLiveSession (a WebSocket connection) that exposes ConfigureSessionAsync(), GetUpdatesAsync() for streamed SessionUpdate events, AddItemAsync() for user messages and function-call outputs, SendAudioAsync(), and StartResponseAsync().

dotnet add package Azure.AI.VoiceLive
dotnet add package Azure.Identity
dotnet add package NAudio                    # For audio capture/playback

The core workflow: start and configure a session with a model (e.g. gpt-4o-mini-realtime-preview), instructions, an AzureStandardVoice, an AzureSemanticVadTurnDetection (with threshold, prefix padding, and silence duration), Pcm16 input/output audio formats, and both Text and Audio modalities for voice assistants; process the streamed events (SessionUpdateResponseAudioDelta for audio chunks, SessionUpdateResponseTextDelta for text chunks, SessionUpdateResponseFunctionCallArgumentsDone for tool calls, SessionUpdateError, and SessionUpdateResponseDone); send a user message via AddItemAsync(new UserMessageItem(...)) followed by StartResponseAsync(); and define function-calling tools via VoiceLiveFunctionDefinition with a JSON parameter schema, added to sessionOptions.Tools, handled in the event loop by deserializing functionCall.Arguments and responding with FunctionCallOutputItem.

Authentication supports Microsoft Entra ID (recommended, via DefaultAzureCredential, requiring the Cognitive Services User role assigned in the Azure Portal) or an API key via AzureKeyCredential. Voice options include Azure Standard (AzureStandardVoice, e.g. "en-US-AvaNeural"), Azure HD (also AzureStandardVoice, e.g. "en-US-Ava:DragonHDLatestNeural"), and Azure Custom voices with an endpoint ID. Supported models are gpt-4o-realtime-preview (GPT-4o with real-time audio), gpt-4o-mini-realtime-preview (lightweight, fast), and phi4-mm-realtime (cost-effective multimodal). Audio configuration recommends Pcm16 input/output, a 24kHz sample rate, and mono channels. Current package versions are stable v1.0.0 and preview v1.1.0-beta.1.

Use it when building a real-time, bidirectional voice assistant on .NET with Azure AI - streaming audio in and out, natural turn detection, and function calling within a live voice conversation.

Best practices: always set both Text and Audio modalities for voice assistants; use AzureSemanticVadTurnDetection for natural conversation flow with a typical 500ms silence duration to avoid premature cutoffs; wrap the session in a using statement for proper disposal; handle every event type including errors; and use DefaultAzureCredential rather than hardcoding API keys. One documented benign error to specifically ignore rather than surface: "Cancellation failed: no active response".

Input is the AZURE_VOICELIVE_ENDPOINT, AZURE_VOICELIVE_MODEL, and AZURE_VOICELIVE_VOICE environment variables (plus an optional AZURE_VOICELIVE_API_KEY if not using Entra ID), session configuration options, and either streamed audio or text user input. Output is a stream of SessionUpdate events - audio deltas to play back, text deltas to display, function-call requests to handle and respond to, errors, and a response-complete signal.

Built on Azure.Identity for Entra ID authentication and pairs with NAudio for audio capture/playback; related SDKs in the same ecosystem include Microsoft.CognitiveServices.Speech for speech-to-text/text-to-speech.

.NET developers building real-time, bidirectional voice AI assistants on Azure that need streaming audio, natural turn detection, and function calling in a live conversation.