Process and Analyze Video and Live Streams

Combines ingestion, understanding, retrieval, and media actions for video, audio, live streams, and desktop sessions into one workflow using the VideoDB Python SDK. Desktop perception starts or stops a desktop session capturing screen, mic, and system audio, streams live context, stores episodic session memory, runs real-time alerts on what is spoken or shown on screen, and produces session summaries with a searchable timeline and playable evidence links (macOS only). Video ingest accepts a file, URL, or RTSP feed and returns a playable web stream link, with server-side transcoding for codec, bitrate, fps, resolution, and aspect ratio. Indexing builds visual, spoken, and keyword indexes so searches return exact moments with timestamps and playable evidence, and can auto-create clips from results. Timeline editing generates and translates subtitles, burns them in, adds text/image overlays and motion captions, mixes background music or voiceover/dubbing, and composes and exports programmatically. Live-stream monitoring connects to RTSP feeds and runs real-time visual and spoken understanding, emitting events and alerts for monitoring workflows.

Setup: pip install "videodb[capture]" python-dotenv (drop the capture extra if it fails on Linux), then set VIDEO_DB_API_KEY either as an exported environment variable or in the project's .env file - the user sets the key themselves, since the skill must not read, write, or handle it. A free API key (50 free uploads, no credit card) is available from VideoDB's console. Before running any VideoDB code, load environment variables with load_dotenv(".env") and connect with videodb.connect(), which raises AuthenticationError if the key is missing.

Use it for video or audio perception, indexing, search, or timeline editing from files, URLs, desktop sessions, or live streams - especially when the task involves timestamps, searchable evidence, subtitles, clips, overlays, or real-time monitoring alerts.

Do not fall back to ffmpeg, moviepy, or local encoding tools for operations VideoDB handles server-side: trimming, combining clips, overlaying audio or music, adding subtitles, text/image overlays, transcoding, resolution changes, aspect-ratio conversion, platform resizing, transcription, and media generation. Local tools are only appropriate for operations outside VideoDB's scope, such as transitions, speed changes, crop/zoom, color grading, or volume mixing.

Common inputs are a local file path, a public URL, or an RTSP URL, plus a desired operation (start/stop/summarize a desktop session, a transcode spec, an index spec, a search query, clip ranges, timeline edits, or alert rules). Common outputs are a stream URL, search results with timestamps and evidence links, generated assets (subtitles, audio, images, clips), event/alert payloads for live streams, and desktop session summaries with memory entries.

Uploading accepts a URL, a YouTube link, or a local file path. Spoken-word indexing and transcript/subtitle generation use video.index_spoken_words(force=True) (the force flag skips the error raised when a video is already indexed), video.get_transcript_text(), and video.add_subtitle(). Search raises InvalidRequestError when nothing matches, so calls should be wrapped in try/except treating "No results found" as an empty result. Scene search has no force parameter on index_scenes() - a repeat call raises an error containing the existing scene index id, which can be extracted with a regex and reused; semantic scene search benefits from a score_threshold around 0.3 or higher to filter low-relevance noise. Timeline editing via Timeline, VideoAsset, and TextAsset requires validating timestamps first, since a negative start is silently accepted but produces broken output, and start must be less than end, which must be no greater than the video's length. Transcoding changes resolution, quality, or aspect ratio server-side via conn.transcode() with VideoConfig/AudioConfig. reframe() is a slow server-side operation for social-platform aspect ratios (vertical 9:16, square 1:1, landscape 16:9, or custom dimensions) - it should be limited to a short segment via start/end, or dispatched asynchronously with a callback_url for full-length videos. Generative media covers image generation via coll.generate_image(), plus voice, music, and sound-effect generation.

Desktop capture uses a bundled ws_listener.py script to stream WebSocket events (transcript and visual-index channels) to /tmp/videodb_events.jsonl, queryable by channel and timestamp. Reference documentation covers the full Python SDK API, search, timeline editing, streaming, generative media, live-stream (RTSP/RTMP) ingestion, and desktop capture in detail. Common pitfalls documented include the already-indexed and scene-index-exists errors above, search's "no results" exception, reframe timeouts on long videos, silently broken negative timestamps, and plan-gated features like generate_video() or create_collection() failing with an "Operation not allowed" or limit error.

Developers building video, live-stream, or desktop-session features who need perception, search, and timeline editing without hand-rolling ffmpeg pipelines or local encoding.