Eve library

Psyche Computer Use

Other

Revision 1

On this page

Claude Computer Use agent with sandboxed desktop environment.

Part of the Psyche AI Virtual Assistant Platform.

Overview#

The Computer Use service provides autonomous desktop interaction capabilities powered by Claude's computer use API. It can observe, reason about, and interact with desktop environments to complete complex tasks.

Features#

  • Claude Agent Loop: Observe-reason-act cycle with vision and tool use
  • Virtual Desktop: Xvfb-based sandboxed environment (1920x1080)
  • Browser Automation: Chromium via Playwright for web interactions
  • Vision System: Screenshots, OCR, UI element detection, accessibility
  • Action System: Mouse, keyboard, file operations, browser control
  • Screen Sharing: H.264/VP9/AV1 streaming with WebRTC
  • Demonstration Mode: Cursor effects, step-by-step tutorials
  • Benchmarking: OSWorld evaluation framework

Architecture#

text
computer_use/
├── agent/              # Claude agent integration
│   ├── loop.py         # Observe-reason-act loop
│   ├── planning.py     # Task planning with caching
│   ├── dead_end.py     # Dead-end detection/recovery
│   ├── recovery.py     # Failure recovery planning
│   ├── undo.py         # Multi-step undo support
│   └── explain.py      # Action explanation
├── vision/             # Screen understanding
│   ├── screenshot.py   # High-performance capture
│   ├── ocr.py          # Multi-backend OCR
│   ├── ui_detection.py # Interactive element detection
│   ├── accessibility.py # ATSPI tree extraction
│   └── semantics.py    # LLM-based understanding
├── actions/            # Desktop automation
│   ├── input.py        # Mouse and keyboard
│   ├── browser.py      # Chromium automation
│   ├── files.py        # File operations
│   └── cookies.py      # Browser cookies
├── environment/        # Sandboxed execution
│   ├── container.py    # Docker management
│   ├── display.py      # Xvfb/VNC setup
│   └── windows_vm.py   # Windows VM support
├── screen_sharing/     # Real-time streaming
│   ├── capture.py      # 30+ FPS capture
│   ├── encoding.py     # H.264/VP9/AV1
│   ├── overlay.py      # Cursor/annotation
│   └── streaming.py    # WebRTC/RTP
├── demonstration/      # Interactive demos
│   ├── cursor_effects.py # Spotlight, trails
│   └── step_mode.py    # Step-by-step execution
└── benchmark/          # OSWorld evaluation
    ├── runner.py       # Task execution
    ├── tasks.py        # Task factory
    └── scoring.py      # Error recovery scoring

Agent Loop#

The agent operates in a continuous observe-reason-act cycle:

text
1. OBSERVE  → Take screenshot, extract UI state
     ↓
2. THINK    → Claude processes with vision, decides action
     ↓
3. ACT      → Execute mouse/keyboard/browser action
     ↓
4. VERIFY   → Screenshot confirms action completed
     ↓
5. REPEAT   → Continue until task complete (max 50 iterations)

Agent States#

State Description
IDLE Agent ready for new task
OBSERVING Taking screenshot and analyzing
THINKING Claude processing with extended thinking
ACTING Executing specified action
VERIFYING Confirming action success
COMPLETED Task finished successfully
FAILED Task failed after recovery attempts
PAUSED Waiting for user confirmation

Action Types#

Mouse Actions#

Action Description
LEFT_CLICK Single left click
RIGHT_CLICK Context menu click
DOUBLE_CLICK Open file/select word
TRIPLE_CLICK Select line/paragraph
DRAG Click and drag
SCROLL Scroll up/down/left/right
MOUSE_MOVE Move cursor

Keyboard Actions#

Action Description
TYPE Type text string
KEY Press single key
HOLD_KEY Hold modifier key
CLIPBOARD Copy/paste operations

Browser Actions#

Action Description
NAVIGATE Go to URL
NEW_TAB Open new tab
CLOSE_TAB Close current tab
SWITCH_TAB Switch to tab N
REFRESH Reload page
BACK/FORWARD Navigation history

File Actions#

Action Description
UPLOAD Upload file to web form
DOWNLOAD Download file
LIST_DIRECTORY List directory contents
READ Read file contents
WRITE Write file
CONVERT Convert file format

System Actions#

Action Description
BASH_COMMAND Execute shell command
WAIT Wait for condition

Vision Pipeline#

  1. Screenshot Capture: High-performance X11 capture
  2. OCR Processing: Tesseract/EasyOCR/PaddleOCR
  3. UI Detection: Clickable elements, modals, menus
  4. Accessibility: ATSPI tree for semantic structure
  5. Semantic Understanding: LLM-based scene analysis

OCR Backends#

Backend Speed Accuracy GPU
Tesseract Fast Medium No
EasyOCR Medium High Yes
PaddleOCR Medium High Yes

Quick Start#

Installation#

bash
# Using Nx
nx install psyche-computer-use

# With all optional dependencies
nx install-all psyche-computer-use

# Install Playwright browsers
nx install-playwright psyche-computer-use

# Or directly with Poetry
cd apps/psyche/computer-use
poetry install
poetry install --extras all
poetry run playwright install chromium --with-deps

Environment Variables#

bash
# Claude API
ANTHROPIC_API_KEY=your-api-key

# Agent Config
CLAUDE_MODEL=claude-sonnet-4-20250514
CLAUDE_MAX_TOKENS=4096
AGENT_MAX_STEPS=50

# Display (auto-configured in Docker)
DISPLAY=:99
SANDBOX_RESOLUTION=1920x1080

# Service Config
COMPUTER_USE_PORT=8005
COMPUTER_USE_HOST=0.0.0.0
LOG_LEVEL=INFO

# Optional: Redis for caching
REDIS_URL=redis://localhost:6379

# Optional: Database for persistence
DATABASE_URL=postgresql://...

Basic Usage#

python
from computer_use.agent import ComputerUseAgent, AgentConfig

# Create agent
config = AgentConfig(
    model="claude-sonnet-4-20250514",
    max_iterations=50,
    enable_thinking=True,
)
agent = ComputerUseAgent(config)

# Run a task
result = await agent.run_task(
    "Open Chrome and navigate to google.com, "
    "then search for 'weather' and click the first result."
)

print(f"Task status: {result.status}")
print(f"Steps taken: {len(result.steps)}")

API Endpoints#

Agent#

  • POST /tasks - Submit new task
  • GET /tasks/{id} - Get task status
  • POST /tasks/{id}/pause - Pause execution
  • POST /tasks/{id}/resume - Resume execution
  • POST /tasks/{id}/cancel - Cancel task

Vision#

  • POST /screenshot - Capture screenshot
  • POST /ocr - Extract text from image
  • POST /ui/detect - Detect UI elements

Actions#

  • POST /actions/execute - Execute action
  • GET /actions/history - Get action history
  • POST /actions/undo - Undo last action

Screen Sharing#

  • GET /stream/start - Start WebRTC stream
  • GET /stream/stop - Stop stream
  • WS /stream/ws - WebSocket stream

Health#

  • GET /health - Health check
  • GET /ready - Readiness check
  • GET /metrics - Prometheus metrics

Development#

Running the Server#

bash
# Development mode with virtual display
nx serve psyche-computer-use

# Production mode
nx serve-prod psyche-computer-use

Running Tests#

bash
nx test psyche-computer-use
nx test-unit psyche-computer-use
nx test-integration psyche-computer-use
nx test-cov psyche-computer-use

Running Benchmarks#

bash
nx benchmark psyche-computer-use

Linting & Formatting#

bash
nx lint psyche-computer-use
nx format psyche-computer-use

Docker#

bash
# Build image
nx docker-build psyche-computer-use

# Run container
nx docker-run psyche-computer-use

# Run with VNC access
nx docker-run-vnc psyche-computer-use
# Connect to VNC: localhost:5900

Nx Integration#

bash
# Available targets
nx serve psyche-computer-use          # Development server
nx serve-prod psyche-computer-use     # Production server
nx build psyche-computer-use          # Build package
nx install psyche-computer-use        # Install dependencies
nx install-all psyche-computer-use    # Install all extras + Playwright
nx install-playwright psyche-computer-use  # Install browsers
nx lint psyche-computer-use           # Run linters
nx format psyche-computer-use         # Format code
nx test psyche-computer-use           # Run all tests
nx test-unit psyche-computer-use      # Unit tests only
nx test-integration psyche-computer-use  # Integration tests
nx test-cov psyche-computer-use       # Tests with coverage
nx benchmark psyche-computer-use      # Run OSWorld benchmarks
nx docker-build psyche-computer-use   # Build Docker image
nx docker-run psyche-computer-use     # Run container
nx docker-run-vnc psyche-computer-use # Run with VNC

OSWorld Benchmark#

The service includes an OSWorld benchmark evaluation framework:

Task Categories#

Category Description Examples
File Management File operations Create, move, rename files
Web Browsing Browser tasks Navigate, search, fill forms
Document Editing Office tasks Create/edit documents
Multi-App Cross-application Copy data between apps
System Settings OS configuration Change settings, install apps
Enterprise Business workflows Process invoices, reports

Difficulty Levels#

Level Steps Complexity
Easy 1-3 Simple, direct actions
Medium 4-7 Multi-step with verification
Hard 8-15 Complex workflows
Expert 15+ Enterprise scenarios

Performance#

System Requirements#

  • Minimum: 4 CPU cores, 8GB RAM
  • Recommended: 8 CPU cores, 16GB RAM
  • GPU: Optional (improves OCR speed)

Processing Times#

Task Time
Screenshot 10-50ms
OCR (Tesseract) 100-500ms
UI Detection 50-200ms
Claude API 1-5s per step
Action Execution 50-200ms

Agent Loop Latency#

  • Simple task (5 steps): ~30s
  • Medium task (15 steps): ~90s
  • Complex task (30 steps): ~3min

Security#

Sandbox Features#

  • Isolated container: Separate network namespace
  • Non-root execution: psyche user (UID 1000)
  • Resource limits: CPU, memory, disk quotas
  • Action confirmation: High-risk actions require approval
  • Application whitelist: Limit allowed applications

Best Practices#

  • Always run in Docker container for production
  • Set AGENT_MAX_STEPS to prevent runaway agents
  • Enable action confirmation for sensitive tasks
  • Use application whitelisting when possible
  • Monitor agent actions via metrics endpoint

License#

Proprietary - Oshun Platform