Claude Computer Use agent with sandboxed desktop environment.
Part of the Psyche AI Virtual Assistant Platform.
Overview#
The Computer Use service provides autonomous desktop interaction capabilities powered by Claude's computer use API. It can observe, reason about, and interact with desktop environments to complete complex tasks.
Features#
- Claude Agent Loop: Observe-reason-act cycle with vision and tool use
- Virtual Desktop: Xvfb-based sandboxed environment (1920x1080)
- Browser Automation: Chromium via Playwright for web interactions
- Vision System: Screenshots, OCR, UI element detection, accessibility
- Action System: Mouse, keyboard, file operations, browser control
- Screen Sharing: H.264/VP9/AV1 streaming with WebRTC
- Demonstration Mode: Cursor effects, step-by-step tutorials
- Benchmarking: OSWorld evaluation framework
Architecture#
text
computer_use/
├── agent/ # Claude agent integration
│ ├── loop.py # Observe-reason-act loop
│ ├── planning.py # Task planning with caching
│ ├── dead_end.py # Dead-end detection/recovery
│ ├── recovery.py # Failure recovery planning
│ ├── undo.py # Multi-step undo support
│ └── explain.py # Action explanation
├── vision/ # Screen understanding
│ ├── screenshot.py # High-performance capture
│ ├── ocr.py # Multi-backend OCR
│ ├── ui_detection.py # Interactive element detection
│ ├── accessibility.py # ATSPI tree extraction
│ └── semantics.py # LLM-based understanding
├── actions/ # Desktop automation
│ ├── input.py # Mouse and keyboard
│ ├── browser.py # Chromium automation
│ ├── files.py # File operations
│ └── cookies.py # Browser cookies
├── environment/ # Sandboxed execution
│ ├── container.py # Docker management
│ ├── display.py # Xvfb/VNC setup
│ └── windows_vm.py # Windows VM support
├── screen_sharing/ # Real-time streaming
│ ├── capture.py # 30+ FPS capture
│ ├── encoding.py # H.264/VP9/AV1
│ ├── overlay.py # Cursor/annotation
│ └── streaming.py # WebRTC/RTP
├── demonstration/ # Interactive demos
│ ├── cursor_effects.py # Spotlight, trails
│ └── step_mode.py # Step-by-step execution
└── benchmark/ # OSWorld evaluation
├── runner.py # Task execution
├── tasks.py # Task factory
└── scoring.py # Error recovery scoring
Agent Loop#
The agent operates in a continuous observe-reason-act cycle:
text
1. OBSERVE → Take screenshot, extract UI state
↓
2. THINK → Claude processes with vision, decides action
↓
3. ACT → Execute mouse/keyboard/browser action
↓
4. VERIFY → Screenshot confirms action completed
↓
5. REPEAT → Continue until task complete (max 50 iterations)
Agent States#
| State | Description |
|---|---|
| IDLE | Agent ready for new task |
| OBSERVING | Taking screenshot and analyzing |
| THINKING | Claude processing with extended thinking |
| ACTING | Executing specified action |
| VERIFYING | Confirming action success |
| COMPLETED | Task finished successfully |
| FAILED | Task failed after recovery attempts |
| PAUSED | Waiting for user confirmation |
Action Types#
Mouse Actions#
| Action | Description |
|---|---|
| LEFT_CLICK | Single left click |
| RIGHT_CLICK | Context menu click |
| DOUBLE_CLICK | Open file/select word |
| TRIPLE_CLICK | Select line/paragraph |
| DRAG | Click and drag |
| SCROLL | Scroll up/down/left/right |
| MOUSE_MOVE | Move cursor |
Keyboard Actions#
| Action | Description |
|---|---|
| TYPE | Type text string |
| KEY | Press single key |
| HOLD_KEY | Hold modifier key |
| CLIPBOARD | Copy/paste operations |
Browser Actions#
| Action | Description |
|---|---|
| NAVIGATE | Go to URL |
| NEW_TAB | Open new tab |
| CLOSE_TAB | Close current tab |
| SWITCH_TAB | Switch to tab N |
| REFRESH | Reload page |
| BACK/FORWARD | Navigation history |
File Actions#
| Action | Description |
|---|---|
| UPLOAD | Upload file to web form |
| DOWNLOAD | Download file |
| LIST_DIRECTORY | List directory contents |
| READ | Read file contents |
| WRITE | Write file |
| CONVERT | Convert file format |
System Actions#
| Action | Description |
|---|---|
| BASH_COMMAND | Execute shell command |
| WAIT | Wait for condition |
Vision Pipeline#
- Screenshot Capture: High-performance X11 capture
- OCR Processing: Tesseract/EasyOCR/PaddleOCR
- UI Detection: Clickable elements, modals, menus
- Accessibility: ATSPI tree for semantic structure
- Semantic Understanding: LLM-based scene analysis
OCR Backends#
| Backend | Speed | Accuracy | GPU |
|---|---|---|---|
| Tesseract | Fast | Medium | No |
| EasyOCR | Medium | High | Yes |
| PaddleOCR | Medium | High | Yes |
Quick Start#
Installation#
bash
# Using Nx
nx install psyche-computer-use
# With all optional dependencies
nx install-all psyche-computer-use
# Install Playwright browsers
nx install-playwright psyche-computer-use
# Or directly with Poetry
cd apps/psyche/computer-use
poetry install
poetry install --extras all
poetry run playwright install chromium --with-deps
Environment Variables#
bash
# Claude API
ANTHROPIC_API_KEY=your-api-key
# Agent Config
CLAUDE_MODEL=claude-sonnet-4-20250514
CLAUDE_MAX_TOKENS=4096
AGENT_MAX_STEPS=50
# Display (auto-configured in Docker)
DISPLAY=:99
SANDBOX_RESOLUTION=1920x1080
# Service Config
COMPUTER_USE_PORT=8005
COMPUTER_USE_HOST=0.0.0.0
LOG_LEVEL=INFO
# Optional: Redis for caching
REDIS_URL=redis://localhost:6379
# Optional: Database for persistence
DATABASE_URL=postgresql://...
Basic Usage#
python
from computer_use.agent import ComputerUseAgent, AgentConfig
# Create agent
config = AgentConfig(
model="claude-sonnet-4-20250514",
max_iterations=50,
enable_thinking=True,
)
agent = ComputerUseAgent(config)
# Run a task
result = await agent.run_task(
"Open Chrome and navigate to google.com, "
"then search for 'weather' and click the first result."
)
print(f"Task status: {result.status}")
print(f"Steps taken: {len(result.steps)}")
API Endpoints#
Agent#
POST /tasks- Submit new taskGET /tasks/{id}- Get task statusPOST /tasks/{id}/pause- Pause executionPOST /tasks/{id}/resume- Resume executionPOST /tasks/{id}/cancel- Cancel task
Vision#
POST /screenshot- Capture screenshotPOST /ocr- Extract text from imagePOST /ui/detect- Detect UI elements
Actions#
POST /actions/execute- Execute actionGET /actions/history- Get action historyPOST /actions/undo- Undo last action
Screen Sharing#
GET /stream/start- Start WebRTC streamGET /stream/stop- Stop streamWS /stream/ws- WebSocket stream
Health#
GET /health- Health checkGET /ready- Readiness checkGET /metrics- Prometheus metrics
Development#
Running the Server#
bash
# Development mode with virtual display
nx serve psyche-computer-use
# Production mode
nx serve-prod psyche-computer-use
Running Tests#
bash
nx test psyche-computer-use
nx test-unit psyche-computer-use
nx test-integration psyche-computer-use
nx test-cov psyche-computer-use
Running Benchmarks#
bash
nx benchmark psyche-computer-use
Linting & Formatting#
bash
nx lint psyche-computer-use
nx format psyche-computer-use
Docker#
bash
# Build image
nx docker-build psyche-computer-use
# Run container
nx docker-run psyche-computer-use
# Run with VNC access
nx docker-run-vnc psyche-computer-use
# Connect to VNC: localhost:5900
Nx Integration#
bash
# Available targets
nx serve psyche-computer-use # Development server
nx serve-prod psyche-computer-use # Production server
nx build psyche-computer-use # Build package
nx install psyche-computer-use # Install dependencies
nx install-all psyche-computer-use # Install all extras + Playwright
nx install-playwright psyche-computer-use # Install browsers
nx lint psyche-computer-use # Run linters
nx format psyche-computer-use # Format code
nx test psyche-computer-use # Run all tests
nx test-unit psyche-computer-use # Unit tests only
nx test-integration psyche-computer-use # Integration tests
nx test-cov psyche-computer-use # Tests with coverage
nx benchmark psyche-computer-use # Run OSWorld benchmarks
nx docker-build psyche-computer-use # Build Docker image
nx docker-run psyche-computer-use # Run container
nx docker-run-vnc psyche-computer-use # Run with VNC
OSWorld Benchmark#
The service includes an OSWorld benchmark evaluation framework:
Task Categories#
| Category | Description | Examples |
|---|---|---|
| File Management | File operations | Create, move, rename files |
| Web Browsing | Browser tasks | Navigate, search, fill forms |
| Document Editing | Office tasks | Create/edit documents |
| Multi-App | Cross-application | Copy data between apps |
| System Settings | OS configuration | Change settings, install apps |
| Enterprise | Business workflows | Process invoices, reports |
Difficulty Levels#
| Level | Steps | Complexity |
|---|---|---|
| Easy | 1-3 | Simple, direct actions |
| Medium | 4-7 | Multi-step with verification |
| Hard | 8-15 | Complex workflows |
| Expert | 15+ | Enterprise scenarios |
Performance#
System Requirements#
- Minimum: 4 CPU cores, 8GB RAM
- Recommended: 8 CPU cores, 16GB RAM
- GPU: Optional (improves OCR speed)
Processing Times#
| Task | Time |
|---|---|
| Screenshot | 10-50ms |
| OCR (Tesseract) | 100-500ms |
| UI Detection | 50-200ms |
| Claude API | 1-5s per step |
| Action Execution | 50-200ms |
Agent Loop Latency#
- Simple task (5 steps): ~30s
- Medium task (15 steps): ~90s
- Complex task (30 steps): ~3min
Security#
Sandbox Features#
- Isolated container: Separate network namespace
- Non-root execution: psyche user (UID 1000)
- Resource limits: CPU, memory, disk quotas
- Action confirmation: High-risk actions require approval
- Application whitelist: Limit allowed applications
Best Practices#
- Always run in Docker container for production
- Set AGENT_MAX_STEPS to prevent runaway agents
- Enable action confirmation for sensitive tasks
- Use application whitelisting when possible
- Monitor agent actions via metrics endpoint
License#
Proprietary - Oshun Platform