What are the two main architectures for building voice agents covered in this skill?

Speech-to-speech (direct audio-to-audio via OpenAI's Realtime API, achieving sub-500ms latency but harder to control) and pipeline architecture (STT → LLM → TTS using components like Deepgram Nova-3, gpt-4o-mini, and ElevenLabs Flash, achieving 700-1200ms but fully controllable and auditable).

What are the component-level latency budgets in a voice agent pipeline?

Voice Activity Detection ~100ms, STT time-to-first-token ~200ms, LLM time-to-first-token ~300ms, and TTS time-to-first-audio ~150ms, totaling approximately 750ms.

When should I use pipeline architecture instead of speech-to-speech?

Choose pipeline architecture when auditability, logging, or precise control over exact wording matters more than achieving the absolute lowest latency, since speech-to-speech is harder to control and audit.

What are common sharp edges in voice agent design and how are they fixed?

Key issues include exceeding 800ms latency (fixed via model warming and output buffering), naive fixed-silence turn detection (fixed via semantic VAD), missing barge-in handling, text responses that overwhelm working memory (keep under 30 words), and STT hallucinating names (fixed via keyword biasing and confirmation prompts).

Skill

Build Natural Voice Agents

Name: Voice Agents
Availability: OnlineOnly
Author: Antigravity

Builds sub-800ms voice agents: speech-to-speech vs pipeline architecture, VAD, barge-in, and latency optimization.

Get skill

Works with openaideepgramelevenlabspipecatvapi

Antigravity

Own this? Claim it

Spark score

out of 100

Updated today

Version 15.7.0

Models

gpt 4o

Add to Favorites

Why it matters

Develop advanced voice agents capable of natural, low-latency conversations. Integrate speech-to-text, text-to-speech, and conversational AI for seamless human-AI interaction.

Outcomes

What it gets done

Implement speech-to-speech and pipeline architectures for voice agents.

Achieve sub-800ms latency for natural conversation flow.

Handle interruptions, background noise, and emotional nuance.

Integrate best-in-class STT, TTS, and LLM components.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/ag-voice-agents | bash

Overview

Voice Agents

Covers building sub-800ms voice agents: speech-to-speech vs STT-LLM-TTS pipeline architecture, VAD and semantic turn-taking, barge-in handling, and voice-specific conversation/prompt design. Use for voice assistants, real-time customer support, or IVR systems needing natural low-latency conversation; routes telephony integration, audio DSP, and accessibility to other specialist skills.

What it does

This skill covers building conversational voice agents where sub-800ms end-to-end latency is the binding constraint, not just speech recognition/synthesis accuracy. It covers two architectures: speech-to-speech (direct audio-to-audio via OpenAI's Realtime API, sub-500ms and most natural but harder to control/audit) and pipeline (STT -> LLM -> TTS via best-in-class components like Deepgram Nova-3 + gpt-4o-mini + ElevenLabs Flash, 700-1200ms typical but fully controllable and auditable). It documents the component-by-component latency budget (VAD ~100ms, STT TTFT ~200ms, LLM TTFT ~300ms, TTS TTFA ~150ms) and notes that jitter (variance) disrupts conversation rhythm even more than absolute latency - a consistent 800ms beats an unpredictable swing between 400ms and 1200ms.

It covers Voice Activity Detection in depth (energy-based vs model-based like Silero VAD vs semantic VAD that understands meaning rather than just silence duration - critical because "Yes." needs a fast response while "Well, let me think..." needs patience), barge-in/interruption handling (stop TTS immediately, abort pending LLM generation, checkpoint state, distinguish real interruptions from backchannel utterances like "uh-huh"), and voice-specific conversation design (responses under 30 words, front-loaded answers, confirming critical details like names/amounts, no bullets/markdown/digit-formatting since TTS reads everything literally, and SSML for precise pronunciation control). It flags severity-rated sharp edges: exceeding 800ms latency (CRITICAL), high jitter variance (HIGH, fixed via warm models, output buffering, and capped max_tokens), naive fixed-silence turn detection misreading pauses (HIGH, fixed via semantic VAD or adaptive thresholds based on transcript completeness), missing barge-in handling (HIGH), text-length responses that overwhelm working memory in voice (MEDIUM), markdown/formatting artifacts read aloud verbatim (MEDIUM), VAD/STT failing in noisy environments (MEDIUM, fixed via noise reduction, adaptive thresholds, confidence filtering, echo cancellation), and STT hallucinating names/technical terms (MEDIUM, fixed via keyword biasing and confirmation prompts for critical info).

When to use - and when NOT to

Use this skill for building voice assistants, real-time customer support, IVR systems, or any conversational AI that speaks and listens in real time, where the 84%-of-organizations trend toward voice AI budget growth in 2025 makes this increasingly common. It explicitly routes adjacent concerns: phone/telephony integration to backend, DSP/audio processing or music generation to audio-specialist, and accessibility compliance to accessibility-specialist.

It is not the tool for the underlying telephony plumbing (Twilio/SIP integration) or deep audio DSP work - those get delegated to sibling skills - and pipeline architecture specifically should be chosen over speech-to-speech when auditability, logging, or precise control over exact wording matters more than the lowest possible latency.

Inputs and outputs

Input is a voice interaction requirement - a customer support agent, a voice assistant, or a real-time translation use case - with a target latency budget. Output is either a speech-to-speech implementation (OpenAI Realtime API session configuration with turn detection) or a full pipeline (STT streaming client, LLM prompt constrained for spoken delivery, streaming TTS), instrumented with latency measurement at each stage and barge-in handling wired to the VAD.

# Component latency targets
STT TTFT: <200ms, LLM TTFT: <300ms, TTS TTFA: <150ms, total: <800ms

Integrations

Spans OpenAI's Realtime API and Whisper/gpt-4o-transcribe, Deepgram Nova-3 (STT) and Aura-2 (TTS), ElevenLabs (TTS), Pipecat (open-source orchestration with Silero VAD and SmartTurn), and managed platforms Vapi and Retell AI. Composes with agent-tool-builder, multi-agent-orchestration, llm-architect, and backend.

Who it's for

Developers building production voice agents - customer support, assistants, IVR - who need to hit sub-800ms latency with natural turn-taking and interruption handling, not just wire together STT and TTS APIs without regard for conversational timing.

FAQ

Common questions

Discussion