Face analysis, emotion detection, and screen understanding engine.
Part of the Psyche AI Virtual Assistant Platform.
Overview#
The Perception Engine provides comprehensive computer vision capabilities:
- Face Detection: Multiple backends (RetinaFace, MTCNN, MediaPipe, OpenCV)
- Facial Landmarks: 68-point detection via MediaPipe FaceMesh
- Emotion Recognition: 8-emotion classification with multi-modal fusion
- Screen Understanding: OCR, UI element detection, document analysis
- Intent Detection: Speech and behavioral intent classification
- Context Fusion: Multi-modal aggregation with temporal decay
Features#
- Multi-Backend Face Detection: RetinaFace (preferred), MTCNN, MediaPipe, OpenCV
- 68-Point Landmarks: MediaPipe-based with Eye/Mouth Aspect Ratios
- 8-Emotion Classification: Happiness, Sadness, Anger, Fear, Surprise, Disgust, Contempt, Neutral
- Audio Emotion: Prosodic and spectral feature extraction
- OCR Engines: Tesseract and EasyOCR
- GPU Acceleration: CUDA-optimized for real-time performance
Emotion Detection#
Primary Emotions#
| Emotion | Description | Audio Features |
|---|---|---|
| Happiness | Positive high arousal | High pitch, fast tempo |
| Sadness | Negative low arousal | Low pitch, slow tempo |
| Anger | Negative high arousal | Loud, high energy |
| Fear | Negative high arousal | Trembling voice |
| Surprise | Positive high arousal | Pitch spikes |
| Disgust | Negative medium arousal | Lowered energy |
| Contempt | Negative low arousal | Unilateral expressions |
| Neutral | Baseline state | Normal prosody |
Multi-Modal Fusion#
The engine fuses video and audio emotion signals:
- Weighted Average: 60% video, 40% audio (configurable)
- Confidence Weighted: Higher confidence = higher weight
- Conflict Aware: Detects and resolves emotional conflicts
- Temporal Smoothing: 5-frame EMA with 0.3 alpha
Face Detection#
Backends#
| Backend | Speed | Accuracy | GPU |
|---|---|---|---|
| RetinaFace | Medium | High | Yes |
| MTCNN | Fast | Medium | Optional |
| MediaPipe | Fast | Medium | Yes |
| OpenCV | Fast | Low | No |
68-Point Landmarks#
Based on MediaPipe FaceMesh with mapping to standard 68 points:
- Jaw (0-16): Face contour
- Right Eyebrow (17-21): Right brow points
- Left Eyebrow (22-26): Left brow points
- Nose (27-35): Nose bridge and tip
- Right Eye (36-41): Right eye contour
- Left Eye (42-47): Left eye contour
- Outer Lip (48-59): Mouth outer contour
- Inner Lip (60-67): Mouth inner contour
Screen Understanding#
OCR Engines#
- Tesseract: System-level OCR with multi-language support
- EasyOCR: Neural network-based with GPU acceleration
Document Types#
SPREADSHEET, PRESENTATION, DOCUMENT, CODE, TERMINAL, BROWSER, EMAIL, CHAT, DASHBOARD, FORM, PDF, IMAGE, VIDEO
UI Elements#
BUTTON, TEXT_FIELD, DROPDOWN, CHECKBOX, RADIO_BUTTON, LINK, IMAGE, ICON, TABLE, CHART, MENU, TOOLBAR, TAB, MODAL
Quick Start#
Installation#
bash
# Using Nx
nx install psyche-perception-engine
# With all optional backends
nx install-all psyche-perception-engine
# Or directly with Poetry
cd apps/psyche/perception-engine
poetry install
poetry install --extras all
Environment Variables#
bash
# Service Config
PERCEPTION_ENGINE_PORT=8004
PERCEPTION_ENGINE_HOST=0.0.0.0
# Model Paths
FACE_DETECTION_MODEL_PATH=./models/face
EMOTION_MODEL_PATH=./models/emotion
Basic Usage#
python
from perception_engine import PerceptionEngine
# Create engine
engine = PerceptionEngine()
await engine.initialize()
# Process frame and audio
result = await engine.perceive(
video_frame=frame,
audio=audio_samples,
screen=screen_image,
)
# Get results
print(f"Emotion: {result.emotion.dominant}")
print(f"Intent: {result.intent.current}")
print(f"Context: {result.context.summary}")
API Endpoints#
Perception#
POST /perceive- Process video/audio/screenGET /status- Get engine status
Emotion#
POST /emotion/detect- Detect emotion from frameGET /emotion/current- Get current emotional state
Face#
POST /face/detect- Detect faces in framePOST /face/landmarks- Get 68-point landmarks
Screen#
POST /screen/analyze- Analyze screen contentPOST /screen/ocr- Extract text from screen
Development#
Running the Server#
bash
# Development mode with auto-reload
nx serve psyche-perception-engine
# Production mode
nx serve-prod psyche-perception-engine
Running Tests#
bash
nx test psyche-perception-engine
nx test-no-gpu psyche-perception-engine
nx test-cov psyche-perception-engine
Linting & Formatting#
bash
nx lint psyche-perception-engine
nx format psyche-perception-engine
Docker#
bash
nx docker-build psyche-perception-engine
nx docker-run psyche-perception-engine
Nx Integration#
bash
# Available targets
nx serve psyche-perception-engine # Development server
nx serve-prod psyche-perception-engine # Production server
nx build psyche-perception-engine # Build package
nx install psyche-perception-engine # Install dependencies
nx install-all psyche-perception-engine # Install all backends
nx lint psyche-perception-engine # Run linters
nx format psyche-perception-engine # Format code
nx test psyche-perception-engine # Run tests
nx test-no-gpu psyche-perception-engine # Tests without GPU
nx test-cov psyche-perception-engine # Tests with coverage
nx docker-build psyche-perception-engine # Build Docker image
nx docker-run psyche-perception-engine # Run Docker container
Performance#
GPU Requirements#
- Minimum: NVIDIA RTX 3080 (10GB VRAM)
- Recommended: NVIDIA A100 (40GB+ VRAM)
Processing Times (RTX 4090)#
| Task | Time |
|---|---|
| Face Detection | 5-15ms |
| Landmark Detection | 10-20ms |
| Emotion Classification | 20-30ms |
| Audio Processing | 100-200ms/sec |
| Screen OCR | 500-2000ms |
License#
Proprietary - Oshun Platform