Orchestrate Real-time Voice & Multimodal AI Applications

The skill is a capability reference for Pipecat, a framework for building production-ready voice and multimodal AI applications with 500-800ms round-trip latency by streaming data through a frame-based pipeline rather than waiting for complete responses at each step. Pipelines connect frame processors in sequence, for example:

pipeline = Pipeline([
    transport.input(),              # Receives user audio
    stt,                            # Speech-to-text conversion
    context_aggregator.user(),      # Collect user responses
    llm,                            # Language model processing
    tts,                            # Text-to-speech conversion
    transport.output(),             # Sends audio to user
    context_aggregator.assistant(), # Collect assistant responses
])

It covers 15+ speech-to-text providers (OpenAI, Google Cloud, Deepgram, AssemblyAI, Azure, Whisper) with streaming transcription, VAD, and 125+ languages via Google Cloud; 30+ text-to-speech providers (OpenAI, Google Cloud, ElevenLabs, Cartesia, LMNT, PlayHT) with streaming synthesis and interruption handling; and 20+ LLM providers (OpenAI, Anthropic, Google Gemini, Groq, Perplexity, and open-source models via Ollama) with streaming responses and function calling. Function calling is defined via FunctionSchema and registered with llm.register_function(name, handler), with results automatically stored in conversation context. Context is managed automatically through aggregators or manually via LLMMessagesAppendFrame/LLMMessagesUpdateFrame, with automatic summarization for long conversations. Turn-taking combines VAD-based detection (Silero recommended for low latency), transcription-based fallback, and Smart Turn Detection for AI-driven completion understanding, alongside configurable interruption handling.

Transports include WebRTC (Daily.co, LiveKit, Small WebRTC), WebSocket (FastAPI and generic servers), telephony (Twilio WebSocket/SIP, Telnyx, Plivo, Exotel), and specialized options (HeyGen and Tavus for video, WhatsApp for messaging). Multimodal support covers vision input (Moondream), image generation (DALL-E, Gemini, Fal), video synthesis (HeyGen, Tavus, Simli), and native speech-to-speech via Gemini Live and OpenAI Realtime. Custom FrameProcessor subclasses handle application-specific logic, and Pipecat Flows adds dynamic or static conversation-state management. Client SDKs cover JavaScript/TypeScript, React, React Native, iOS (Swift), Android (Kotlin), and C++, all implementing the RTVI (Real-Time Voice and Video Inference) standard. Deployment targets include Pipecat Cloud (managed scaling), Fly.io, Modal (GPU infrastructure), Cerebrium, or self-managed Docker containers on any cloud provider. Metrics cover real-time latency (TTFB, round-trip), token usage, frame throughput, and OpenTelemetry tracing.

Use it when building a real-time voice or multimodal AI application on Daily- or Pipecat-style transports, when guidance is needed on low-latency audio/video/text/AI-service orchestration in one pipeline, or as a capability reference before choosing services, transports, or workflow patterns for an interactive agent.

Input is user audio, video, text, or images arriving through the configured transport. Output is a real-time pipeline response - transcription, LLM-generated text, synthesized speech, or multimodal content - streamed back through the same transport with sub-second latency.

It integrates with 15+ LLM providers (OpenAI, Anthropic, Google Gemini, Groq, Perplexity, AWS Bedrock, Azure OpenAI), 10+ speech services (Deepgram, ElevenLabs, Google Cloud, Azure, AssemblyAI, Cartesia, LMNT), telephony (Twilio, Telnyx, Plivo, Exotel), video/media platforms (Daily.co, LiveKit, HeyGen, Tavus, Simli), Mem0 for persistent memory, Sentry and Datadog for monitoring, and the RTVI standard plus Pipecat Flows for client/server and conversation structuring.

Developers building interactive voice or multimodal AI agents (voice assistants, phone bots, video-synthesis applications) who need a single reference for the transports, AI services, and orchestration patterns available before wiring the pipeline together.