Build Real-Time Voice AI Applications

This skill provides concrete implementation patterns for building voice AI applications across four distinct provider/infrastructure choices, each with working code. OpenAI's Realtime API gives native voice-to-voice with GPT-4o over a WebSocket (session config for voice/audio format/VAD, streaming audio deltas, function-calling mid-conversation) when integrated STT/TTS-free voice is wanted. Vapi provides hosted voice agents with webhooks - creating an assistant (model, voice provider like ElevenLabs, transcriber like Deepgram), handling function-call and end-of-call-report webhook events, and placing outbound/web calls - for phone-based agents needing quick deployment. A custom Deepgram STT + ElevenLabs TTS pipeline (real-time transcription with interim results and VAD events, streaming TTS via convert_as_stream or a WebSocket for lowest latency) suits high-quality, fully custom pipelines. LiveKit provides WebRTC real-time infrastructure (room/token creation, subscribing to audio tracks, publishing an agent's synthesized audio) for building bespoke real-time voice apps rather than using a hosted platform.

It also provides a complete reference VoiceAgent class tying these pieces together: an async audio-processing loop that transcribes incoming audio, checks for interruptions mid-response, streams LLM tokens into a ~50-character buffer before sending to TTS (balancing prosody quality against latency), and exposes explicit stop_speaking/interruption handling. Its latency tips reinforce streaming everywhere, starting TTS before the LLM finishes, using raw PCM audio to avoid encoding overhead, keeping WebSocket connections alive, and using regional endpoints near users. It also flags concrete implementation gaps: non-streaming TTS (HIGH latency cost), WebSocket connections without reconnection/error handling (HIGH), blocking (non-async) audio processing (HIGH), hardcoded sample rates (MEDIUM), and missing interruption handling (MEDIUM).

Use this skill when actually implementing a voice AI application - choosing between OpenAI Realtime, Vapi, a custom Deepgram/ElevenLabs pipeline, or LiveKit infrastructure, and wiring up the provider SDKs, webhooks, or WebRTC connections needed. It routes adjacent concerns to sibling skills: complex agent graphs/state behind the voice layer to langgraph, structured tool-response extraction to structured-output, latency/quality monitoring to langfuse, and the web frontend around the voice agent to nextjs-app-router.

It complements rather than duplicates a more theory-focused voice-agent skill covering VAD/turn-taking/conversation design principles - this one is about the concrete provider integration code itself.

Input is a voice AI application requirement and a provider choice - OpenAI Realtime for integrated voice-to-voice, Vapi for a hosted phone agent, Deepgram+ElevenLabs for a custom high-quality pipeline, or LiveKit for custom WebRTC infrastructure. Output is working async Python (or Node.js) code implementing that provider's session/connection setup, audio streaming, and event handling, following the latency-optimization patterns of streaming, buffering, PCM audio, and persistent connections.

# Vapi assistant creation
client.assistants.create(model={...}, voice={"provider": "11labs"}, transcriber={"provider": "deepgram"})

Directly integrates OpenAI Realtime API, Vapi, Deepgram (STT), ElevenLabs (TTS), and LiveKit (WebRTC infrastructure), with telephony via Twilio/SIP-PSTN for phone-based agents. Composes with langgraph, structured-output, langfuse, and nextjs-app-router in workflows like an intelligent tool-using voice agent, a monitored voice agent with Langfuse tracing, or a Twilio-connected phone agent.

Developers implementing a voice AI application who need working provider-specific integration code - OpenAI Realtime, Vapi, Deepgram/ElevenLabs, LiveKit - rather than general voice-agent design theory.