# Psyche Perception Engine

Face analysis, emotion detection, and screen understanding engine.

Part of the Psyche AI Virtual Assistant Platform.

## Overview

The Perception Engine provides comprehensive computer vision capabilities:

- **Face Detection**: Multiple backends (RetinaFace, MTCNN, MediaPipe, OpenCV)
- **Facial Landmarks**: 68-point detection via MediaPipe FaceMesh
- **Emotion Recognition**: 8-emotion classification with multi-modal fusion
- **Screen Understanding**: OCR, UI element detection, document analysis
- **Intent Detection**: Speech and behavioral intent classification
- **Context Fusion**: Multi-modal aggregation with temporal decay

## Features

- **Multi-Backend Face Detection**: RetinaFace (preferred), MTCNN, MediaPipe,
  OpenCV
- **68-Point Landmarks**: MediaPipe-based with Eye/Mouth Aspect Ratios
- **8-Emotion Classification**: Happiness, Sadness, Anger, Fear, Surprise,
  Disgust, Contempt, Neutral
- **Audio Emotion**: Prosodic and spectral feature extraction
- **OCR Engines**: Tesseract and EasyOCR
- **GPU Acceleration**: CUDA-optimized for real-time performance

## Emotion Detection

### Primary Emotions

| Emotion   | Description             | Audio Features         |
| --------- | ----------------------- | ---------------------- |
| Happiness | Positive high arousal   | High pitch, fast tempo |
| Sadness   | Negative low arousal    | Low pitch, slow tempo  |
| Anger     | Negative high arousal   | Loud, high energy      |
| Fear      | Negative high arousal   | Trembling voice        |
| Surprise  | Positive high arousal   | Pitch spikes           |
| Disgust   | Negative medium arousal | Lowered energy         |
| Contempt  | Negative low arousal    | Unilateral expressions |
| Neutral   | Baseline state          | Normal prosody         |

### Multi-Modal Fusion

The engine fuses video and audio emotion signals:

- **Weighted Average**: 60% video, 40% audio (configurable)
- **Confidence Weighted**: Higher confidence = higher weight
- **Conflict Aware**: Detects and resolves emotional conflicts
- **Temporal Smoothing**: 5-frame EMA with 0.3 alpha

## Face Detection

### Backends

| Backend    | Speed  | Accuracy | GPU      |
| ---------- | ------ | -------- | -------- |
| RetinaFace | Medium | High     | Yes      |
| MTCNN      | Fast   | Medium   | Optional |
| MediaPipe  | Fast   | Medium   | Yes      |
| OpenCV     | Fast   | Low      | No       |

### 68-Point Landmarks

Based on MediaPipe FaceMesh with mapping to standard 68 points:

- **Jaw (0-16)**: Face contour
- **Right Eyebrow (17-21)**: Right brow points
- **Left Eyebrow (22-26)**: Left brow points
- **Nose (27-35)**: Nose bridge and tip
- **Right Eye (36-41)**: Right eye contour
- **Left Eye (42-47)**: Left eye contour
- **Outer Lip (48-59)**: Mouth outer contour
- **Inner Lip (60-67)**: Mouth inner contour

## Screen Understanding

### OCR Engines

- **Tesseract**: System-level OCR with multi-language support
- **EasyOCR**: Neural network-based with GPU acceleration

### Document Types

SPREADSHEET, PRESENTATION, DOCUMENT, CODE, TERMINAL, BROWSER, EMAIL, CHAT,
DASHBOARD, FORM, PDF, IMAGE, VIDEO

### UI Elements

BUTTON, TEXT_FIELD, DROPDOWN, CHECKBOX, RADIO_BUTTON, LINK, IMAGE, ICON, TABLE,
CHART, MENU, TOOLBAR, TAB, MODAL

## Quick Start

### Installation

```bash
# Using Nx
nx install psyche-perception-engine

# With all optional backends
nx install-all psyche-perception-engine

# Or directly with Poetry
cd apps/psyche/perception-engine
poetry install
poetry install --extras all
```

### Environment Variables

```bash
# Service Config
PERCEPTION_ENGINE_PORT=8004
PERCEPTION_ENGINE_HOST=0.0.0.0

# Model Paths
FACE_DETECTION_MODEL_PATH=./models/face
EMOTION_MODEL_PATH=./models/emotion
```

### Basic Usage

```python
from perception_engine import PerceptionEngine

# Create engine
engine = PerceptionEngine()
await engine.initialize()

# Process frame and audio
result = await engine.perceive(
    video_frame=frame,
    audio=audio_samples,
    screen=screen_image,
)

# Get results
print(f"Emotion: {result.emotion.dominant}")
print(f"Intent: {result.intent.current}")
print(f"Context: {result.context.summary}")
```

## API Endpoints

### Perception

- `POST /perceive` - Process video/audio/screen
- `GET /status` - Get engine status

### Emotion

- `POST /emotion/detect` - Detect emotion from frame
- `GET /emotion/current` - Get current emotional state

### Face

- `POST /face/detect` - Detect faces in frame
- `POST /face/landmarks` - Get 68-point landmarks

### Screen

- `POST /screen/analyze` - Analyze screen content
- `POST /screen/ocr` - Extract text from screen

## Development

### Running the Server

```bash
# Development mode with auto-reload
nx serve psyche-perception-engine

# Production mode
nx serve-prod psyche-perception-engine
```

### Running Tests

```bash
nx test psyche-perception-engine
nx test-no-gpu psyche-perception-engine
nx test-cov psyche-perception-engine
```

### Linting & Formatting

```bash
nx lint psyche-perception-engine
nx format psyche-perception-engine
```

### Docker

```bash
nx docker-build psyche-perception-engine
nx docker-run psyche-perception-engine
```

## Nx Integration

```bash
# Available targets
nx serve psyche-perception-engine        # Development server
nx serve-prod psyche-perception-engine   # Production server
nx build psyche-perception-engine        # Build package
nx install psyche-perception-engine      # Install dependencies
nx install-all psyche-perception-engine  # Install all backends
nx lint psyche-perception-engine         # Run linters
nx format psyche-perception-engine       # Format code
nx test psyche-perception-engine         # Run tests
nx test-no-gpu psyche-perception-engine  # Tests without GPU
nx test-cov psyche-perception-engine     # Tests with coverage
nx docker-build psyche-perception-engine # Build Docker image
nx docker-run psyche-perception-engine   # Run Docker container
```

## Performance

### GPU Requirements

- Minimum: NVIDIA RTX 3080 (10GB VRAM)
- Recommended: NVIDIA A100 (40GB+ VRAM)

### Processing Times (RTX 4090)

| Task                   | Time          |
| ---------------------- | ------------- |
| Face Detection         | 5-15ms        |
| Landmark Detection     | 10-20ms       |
| Emotion Classification | 20-30ms       |
| Audio Processing       | 100-200ms/sec |
| Screen OCR             | 500-2000ms    |

## License

Proprietary - Oshun Platform