Automate Computer Tasks with Vision Agents

This skill is a pattern reference for building computer use agents - AI agents that interact with computers like humans do, viewing screens, moving cursors, clicking, and typing. It covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives, with a critical focus on sandboxing, security, and the unique challenges of vision-based control.

Core patterns include: the Perception-Reasoning-Action loop (screenshot capture, vision-language model reasoning, mouse/keyboard execution, and feedback), noting that vision agents are completely still during a 1-5 second "thinking" phase, a detectable pause pattern; the Sandboxed Environment pattern, which mandates running agents in isolated Docker containers with restricted networking, read-only or scoped filesystems, no host credentials, filtered syscalls, and resource limits, aiming for "blast radius minimization"; the official Anthropic Computer Use implementation, noting Claude 3.5 Sonnet was the first frontier model to offer computer use and Claude Opus 4.5 is described as "the best model in the world for computer use," with tool versions computer_20251124 (Opus 4.5, adds a zoom action) and computer_20250124 for other models, exposing screenshot, mouse, keyboard, bash, and text_editor capabilities; the Browser-Use pattern via Playwright, which uses structured DOM/accessibility-tree access instead of pixel-based screenshots for browser-only automation - faster, cheaper on tokens, more precise, and more reliable than vision for web tasks; the User Confirmation pattern, which classifies actions into LOW/MEDIUM/HIGH/CRITICAL sensitivity, from auto-approved navigation up to confirm-and-review for purchases, credential entry, and financial transactions; and the Action Logging pattern, which captures timestamps, sanitized action parameters (redacting passwords, tokens, and keys), before/after screenshots, and model reasoning for debugging, auditing, and compliance.

The skill also documents Sharp Edges with severity ratings. Web content can hijack a browser agent via prompt injection - it cites real incidents where Microsoft Copilot agents were manipulated via malicious emails to extract CRM data, and Google Workspace/Gemini agents were tricked via hidden prompts in calendar invites and emails - countered with defense in depth: network-isolated sandboxing, injection-pattern classifiers, user confirmation on sensitive actions, and scoped, temporary credentials. Vision agents click mathematically exact element centers rather than the Gaussian-distributed clicks humans make, and combined with predictable "thinking" pauses this lets anti-bot systems detect and block the automation, addressed with randomized click offsets, variable delays, and curved mouse movement. Dropdowns, scrollbars, and drag operations are unreliable for vision models since they infer UI structure from pixels rather than the DOM, addressed by preferring keyboard-driven interaction and breaking complex actions into discrete steps.

Use when building or hardening a computer-use or browser-use AI agent: designing the core perception-action loop, sandboxing its execution, wiring up Anthropic's official Computer Use tools, choosing between vision-based and structured-DOM automation, gating sensitive actions behind confirmation, or adding action logging. Prefer the structured Browser-Use pattern over vision-based computer use for web automation, form filling, and data extraction, where speed and cost matter more than visual understanding; reserve vision-based control for desktop apps, complex UIs, and visual verification tasks a DOM can't represent.

Each pattern also documents its own anti-patterns to avoid: running the perception loop without a step limit or inter-action delay, or capturing full-resolution screenshots; running the sandbox directly on the host, with full network access, as root, or with persistent storage; skipping the computer-use beta flag or command sanitization on the bash tool; using screenshots when structured DOM access would work, or hardcoding selectors; auto-approving all confirmations or displaying full passwords in a confirmation prompt; and logging without sanitizing sensitive fields or rotating log files.

The skill provides reference Python implementations rather than a single tool contract: a ComputerUseAgent class running the perception-reasoning-action loop against Anthropic's Claude API, a Dockerfile and docker-compose.yml for a sandboxed execution environment, an AnthropicComputerUse class wrapping the official computer/bash/text_editor tools, a Playwright-based BrowserUseAgent for structured DOM automation, a ConfirmationGate class for severity-based action approval, and an ActionLogger for session and action-level logging. Its recommended sandboxing invocation:

# Docker with strict isolation
docker run \
    --security-opt no-new-privileges \
    --cap-drop ALL \
    --network none \  # No internet!
    --read-only \
    computer-use-agent

Anthropic's Claude API and its official Computer Use tool beta, OpenAI's Operator/CUA, Docker for sandboxing, Playwright for structured browser automation, and pyautogui/xdotool for direct mouse and keyboard control. The Action Logging pattern writes per-action logs as JSONL plus a JSON session summary to a configurable log directory with its own screenshots subdirectory.

Engineers building autonomous computer-use or browser-use agents who need concrete, security-aware patterns for the perception-action loop, sandboxing, confirmation gating, and defending against prompt injection and anti-bot detection.