# Psyche Computer Use

Claude Computer Use agent with sandboxed desktop environment.

Part of the Psyche AI Virtual Assistant Platform.

## Overview

The Computer Use service provides autonomous desktop interaction capabilities
powered by Claude's computer use API. It can observe, reason about, and interact
with desktop environments to complete complex tasks.

## Features

- **Claude Agent Loop**: Observe-reason-act cycle with vision and tool use
- **Virtual Desktop**: Xvfb-based sandboxed environment (1920x1080)
- **Browser Automation**: Chromium via Playwright for web interactions
- **Vision System**: Screenshots, OCR, UI element detection, accessibility
- **Action System**: Mouse, keyboard, file operations, browser control
- **Screen Sharing**: H.264/VP9/AV1 streaming with WebRTC
- **Demonstration Mode**: Cursor effects, step-by-step tutorials
- **Benchmarking**: OSWorld evaluation framework

## Architecture

```
computer_use/
├── agent/              # Claude agent integration
│   ├── loop.py         # Observe-reason-act loop
│   ├── planning.py     # Task planning with caching
│   ├── dead_end.py     # Dead-end detection/recovery
│   ├── recovery.py     # Failure recovery planning
│   ├── undo.py         # Multi-step undo support
│   └── explain.py      # Action explanation
├── vision/             # Screen understanding
│   ├── screenshot.py   # High-performance capture
│   ├── ocr.py          # Multi-backend OCR
│   ├── ui_detection.py # Interactive element detection
│   ├── accessibility.py # ATSPI tree extraction
│   └── semantics.py    # LLM-based understanding
├── actions/            # Desktop automation
│   ├── input.py        # Mouse and keyboard
│   ├── browser.py      # Chromium automation
│   ├── files.py        # File operations
│   └── cookies.py      # Browser cookies
├── environment/        # Sandboxed execution
│   ├── container.py    # Docker management
│   ├── display.py      # Xvfb/VNC setup
│   └── windows_vm.py   # Windows VM support
├── screen_sharing/     # Real-time streaming
│   ├── capture.py      # 30+ FPS capture
│   ├── encoding.py     # H.264/VP9/AV1
│   ├── overlay.py      # Cursor/annotation
│   └── streaming.py    # WebRTC/RTP
├── demonstration/      # Interactive demos
│   ├── cursor_effects.py # Spotlight, trails
│   └── step_mode.py    # Step-by-step execution
└── benchmark/          # OSWorld evaluation
    ├── runner.py       # Task execution
    ├── tasks.py        # Task factory
    └── scoring.py      # Error recovery scoring
```

## Agent Loop

The agent operates in a continuous observe-reason-act cycle:

```
1. OBSERVE  → Take screenshot, extract UI state
     ↓
2. THINK    → Claude processes with vision, decides action
     ↓
3. ACT      → Execute mouse/keyboard/browser action
     ↓
4. VERIFY   → Screenshot confirms action completed
     ↓
5. REPEAT   → Continue until task complete (max 50 iterations)
```

### Agent States

| State     | Description                              |
| --------- | ---------------------------------------- |
| IDLE      | Agent ready for new task                 |
| OBSERVING | Taking screenshot and analyzing          |
| THINKING  | Claude processing with extended thinking |
| ACTING    | Executing specified action               |
| VERIFYING | Confirming action success                |
| COMPLETED | Task finished successfully               |
| FAILED    | Task failed after recovery attempts      |
| PAUSED    | Waiting for user confirmation            |

## Action Types

### Mouse Actions

| Action       | Description               |
| ------------ | ------------------------- |
| LEFT_CLICK   | Single left click         |
| RIGHT_CLICK  | Context menu click        |
| DOUBLE_CLICK | Open file/select word     |
| TRIPLE_CLICK | Select line/paragraph     |
| DRAG         | Click and drag            |
| SCROLL       | Scroll up/down/left/right |
| MOUSE_MOVE   | Move cursor               |

### Keyboard Actions

| Action    | Description           |
| --------- | --------------------- |
| TYPE      | Type text string      |
| KEY       | Press single key      |
| HOLD_KEY  | Hold modifier key     |
| CLIPBOARD | Copy/paste operations |

### Browser Actions

| Action       | Description        |
| ------------ | ------------------ |
| NAVIGATE     | Go to URL          |
| NEW_TAB      | Open new tab       |
| CLOSE_TAB    | Close current tab  |
| SWITCH_TAB   | Switch to tab N    |
| REFRESH      | Reload page        |
| BACK/FORWARD | Navigation history |

### File Actions

| Action         | Description             |
| -------------- | ----------------------- |
| UPLOAD         | Upload file to web form |
| DOWNLOAD       | Download file           |
| LIST_DIRECTORY | List directory contents |
| READ           | Read file contents      |
| WRITE          | Write file              |
| CONVERT        | Convert file format     |

### System Actions

| Action       | Description           |
| ------------ | --------------------- |
| BASH_COMMAND | Execute shell command |
| WAIT         | Wait for condition    |

## Vision Pipeline

1. **Screenshot Capture**: High-performance X11 capture
2. **OCR Processing**: Tesseract/EasyOCR/PaddleOCR
3. **UI Detection**: Clickable elements, modals, menus
4. **Accessibility**: ATSPI tree for semantic structure
5. **Semantic Understanding**: LLM-based scene analysis

### OCR Backends

| Backend   | Speed  | Accuracy | GPU |
| --------- | ------ | -------- | --- |
| Tesseract | Fast   | Medium   | No  |
| EasyOCR   | Medium | High     | Yes |
| PaddleOCR | Medium | High     | Yes |

## Quick Start

### Installation

```bash
# Using Nx
nx install psyche-computer-use

# With all optional dependencies
nx install-all psyche-computer-use

# Install Playwright browsers
nx install-playwright psyche-computer-use

# Or directly with Poetry
cd apps/psyche/computer-use
poetry install
poetry install --extras all
poetry run playwright install chromium --with-deps
```

### Environment Variables

```bash
# Claude API
ANTHROPIC_API_KEY=your-api-key

# Agent Config
CLAUDE_MODEL=claude-sonnet-4-20250514
CLAUDE_MAX_TOKENS=4096
AGENT_MAX_STEPS=50

# Display (auto-configured in Docker)
DISPLAY=:99
SANDBOX_RESOLUTION=1920x1080

# Service Config
COMPUTER_USE_PORT=8005
COMPUTER_USE_HOST=0.0.0.0
LOG_LEVEL=INFO

# Optional: Redis for caching
REDIS_URL=redis://localhost:6379

# Optional: Database for persistence
DATABASE_URL=postgresql://...
```

### Basic Usage

```python
from computer_use.agent import ComputerUseAgent, AgentConfig

# Create agent
config = AgentConfig(
    model="claude-sonnet-4-20250514",
    max_iterations=50,
    enable_thinking=True,
)
agent = ComputerUseAgent(config)

# Run a task
result = await agent.run_task(
    "Open Chrome and navigate to google.com, "
    "then search for 'weather' and click the first result."
)

print(f"Task status: {result.status}")
print(f"Steps taken: {len(result.steps)}")
```

## API Endpoints

### Agent

- `POST /tasks` - Submit new task
- `GET /tasks/{id}` - Get task status
- `POST /tasks/{id}/pause` - Pause execution
- `POST /tasks/{id}/resume` - Resume execution
- `POST /tasks/{id}/cancel` - Cancel task

### Vision

- `POST /screenshot` - Capture screenshot
- `POST /ocr` - Extract text from image
- `POST /ui/detect` - Detect UI elements

### Actions

- `POST /actions/execute` - Execute action
- `GET /actions/history` - Get action history
- `POST /actions/undo` - Undo last action

### Screen Sharing

- `GET /stream/start` - Start WebRTC stream
- `GET /stream/stop` - Stop stream
- `WS /stream/ws` - WebSocket stream

### Health

- `GET /health` - Health check
- `GET /ready` - Readiness check
- `GET /metrics` - Prometheus metrics

## Development

### Running the Server

```bash
# Development mode with virtual display
nx serve psyche-computer-use

# Production mode
nx serve-prod psyche-computer-use
```

### Running Tests

```bash
nx test psyche-computer-use
nx test-unit psyche-computer-use
nx test-integration psyche-computer-use
nx test-cov psyche-computer-use
```

### Running Benchmarks

```bash
nx benchmark psyche-computer-use
```

### Linting & Formatting

```bash
nx lint psyche-computer-use
nx format psyche-computer-use
```

### Docker

```bash
# Build image
nx docker-build psyche-computer-use

# Run container
nx docker-run psyche-computer-use

# Run with VNC access
nx docker-run-vnc psyche-computer-use
# Connect to VNC: localhost:5900
```

## Nx Integration

```bash
# Available targets
nx serve psyche-computer-use          # Development server
nx serve-prod psyche-computer-use     # Production server
nx build psyche-computer-use          # Build package
nx install psyche-computer-use        # Install dependencies
nx install-all psyche-computer-use    # Install all extras + Playwright
nx install-playwright psyche-computer-use  # Install browsers
nx lint psyche-computer-use           # Run linters
nx format psyche-computer-use         # Format code
nx test psyche-computer-use           # Run all tests
nx test-unit psyche-computer-use      # Unit tests only
nx test-integration psyche-computer-use  # Integration tests
nx test-cov psyche-computer-use       # Tests with coverage
nx benchmark psyche-computer-use      # Run OSWorld benchmarks
nx docker-build psyche-computer-use   # Build Docker image
nx docker-run psyche-computer-use     # Run container
nx docker-run-vnc psyche-computer-use # Run with VNC
```

## OSWorld Benchmark

The service includes an OSWorld benchmark evaluation framework:

### Task Categories

| Category         | Description        | Examples                      |
| ---------------- | ------------------ | ----------------------------- |
| File Management  | File operations    | Create, move, rename files    |
| Web Browsing     | Browser tasks      | Navigate, search, fill forms  |
| Document Editing | Office tasks       | Create/edit documents         |
| Multi-App        | Cross-application  | Copy data between apps        |
| System Settings  | OS configuration   | Change settings, install apps |
| Enterprise       | Business workflows | Process invoices, reports     |

### Difficulty Levels

| Level  | Steps | Complexity                   |
| ------ | ----- | ---------------------------- |
| Easy   | 1-3   | Simple, direct actions       |
| Medium | 4-7   | Multi-step with verification |
| Hard   | 8-15  | Complex workflows            |
| Expert | 15+   | Enterprise scenarios         |

## Performance

### System Requirements

- **Minimum**: 4 CPU cores, 8GB RAM
- **Recommended**: 8 CPU cores, 16GB RAM
- **GPU**: Optional (improves OCR speed)

### Processing Times

| Task             | Time          |
| ---------------- | ------------- |
| Screenshot       | 10-50ms       |
| OCR (Tesseract)  | 100-500ms     |
| UI Detection     | 50-200ms      |
| Claude API       | 1-5s per step |
| Action Execution | 50-200ms      |

### Agent Loop Latency

- Simple task (5 steps): ~30s
- Medium task (15 steps): ~90s
- Complex task (30 steps): ~3min

## Security

### Sandbox Features

- **Isolated container**: Separate network namespace
- **Non-root execution**: psyche user (UID 1000)
- **Resource limits**: CPU, memory, disk quotas
- **Action confirmation**: High-risk actions require approval
- **Application whitelist**: Limit allowed applications

### Best Practices

- Always run in Docker container for production
- Set AGENT_MAX_STEPS to prevent runaway agents
- Enable action confirmation for sensitive tasks
- Use application whitelisting when possible
- Monitor agent actions via metrics endpoint

## License

Proprietary - Oshun Platform