Draw diagrams and visuals from voice on infinite canvas

This demo extends tldraw's official Agent starter kit with a speech-driven "capture a region and describe it" workflow: drag a rectangle on an infinite canvas, speak what you want ("a flowchart for user login", "what is a neuron"), and an AI agent draws a complete result inside that region. Dragging several rectangles in a row queues them up - a new capture auto-stops the previous recording, and each is drawn in turn through a serialized queue, since the agent processes one request at a time. Each region's status pill tracks its progress: Recording, then Queued, then Transcribing, then Drawing.

The agent doesn't just dump the transcript as text - it assesses each request and picks the visual form that fits: a named drawing, a single illustration, a diagram, a chart, or literal text. Under the hood, selecting the Agent draw tool (keyboard shortcut a) and dragging starts a capture session and mic recording; clicking Stop (or starting a new capture) sends the audio to a /transcribe worker route that forwards it to Mistral Voxtral for speech-to-text, and the resulting text is sent through the standard agent.prompt() loop along with the captured area's bounds, so the drawing finishes inside the region in a single turn.

Use this demo when you want to prototype or build a voice-driven, spatially-anchored way to generate diagrams, illustrations, or charts on an infinite canvas - explaining a concept out loud and having it appear exactly where you gestured, rather than typing a prompt into a chat panel. It's a good starting point for anyone building on tldraw's Agent starter kit who wants a concrete example of adding a new capture-and-transcribe input modality alongside the existing text-prompt flow.

It requires a Mistral API key for transcription and an OpenRouter (or other LLM provider) API key for the agent model, plus, for any public production deployment, a tldraw SDK license key - the tldraw SDK itself is free for development but proprietary-licensed for production use, separate from this repo's own MIT license. It's a demo/reference app, not a polished production product: it processes captures strictly one at a time via a serialized queue rather than in parallel.

Input is a dragged rectangle region plus a spoken voice request captured through the browser's microphone. Setup requires copying the example environment file and filling in API keys:

cp .dev.vars.example .dev.vars
npm install
npm run dev

Required keys out of the box are MISTRAL_API_KEY (speech-to-text via Mistral) and OPENROUTER_API_KEY (the default agent model, google/gemini-2.5-flash, hosted via OpenRouter); optional keys (ANTHROPIC_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY) apply only if the model picker is switched to a native provider. The chosen model is persisted to localStorage per agent, overriding the code default (DEFAULT_MODEL_NAME in shared/models.ts) on reload.

Output is a set of tldraw shapes drawn directly inside the captured rectangle, matching the visual form the agent judged appropriate for the spoken request. Deployment builds and ships as a Cloudflare Worker (npm run build && npx wrangler deploy), with production secrets set via npx wrangler secret put rather than committed, and a VITE_TLDRAW_LICENSE_KEY env var supplying the required tldraw license for any public deployment.

The app is built directly on tldraw's official Agent starter kit (MIT licensed) and the tldraw SDK itself (free for development, proprietary-licensed for production - a separate license key from tldraw.dev/pricing is required for public deployment). Speech transcription integrates with Mistral's Voxtral model via a Cloudflare Worker route; the agent's language model defaults to Google's Gemini 2.5 Flash routed through OpenRouter, with Anthropic, Google, and OpenAI available as alternative native providers through the model picker. The whole app deploys as a single Cloudflare Worker serving both the built client and the /stream and /transcribe API routes.

Developers exploring or building on tldraw's Agent starter kit who want a working reference implementation of a voice-driven, region-anchored drawing input, and anyone prototyping tools where a spoken description should turn into a diagram, illustration, or chart placed exactly where the user gestured on a canvas.