Eve library

Psyche Perception Engine

Other

Revision 1

On this page

Face analysis, emotion detection, and screen understanding engine.

Part of the Psyche AI Virtual Assistant Platform.

Overview#

The Perception Engine provides comprehensive computer vision capabilities:

  • Face Detection: Multiple backends (RetinaFace, MTCNN, MediaPipe, OpenCV)
  • Facial Landmarks: 68-point detection via MediaPipe FaceMesh
  • Emotion Recognition: 8-emotion classification with multi-modal fusion
  • Screen Understanding: OCR, UI element detection, document analysis
  • Intent Detection: Speech and behavioral intent classification
  • Context Fusion: Multi-modal aggregation with temporal decay

Features#

  • Multi-Backend Face Detection: RetinaFace (preferred), MTCNN, MediaPipe, OpenCV
  • 68-Point Landmarks: MediaPipe-based with Eye/Mouth Aspect Ratios
  • 8-Emotion Classification: Happiness, Sadness, Anger, Fear, Surprise, Disgust, Contempt, Neutral
  • Audio Emotion: Prosodic and spectral feature extraction
  • OCR Engines: Tesseract and EasyOCR
  • GPU Acceleration: CUDA-optimized for real-time performance

Emotion Detection#

Primary Emotions#

Emotion Description Audio Features
Happiness Positive high arousal High pitch, fast tempo
Sadness Negative low arousal Low pitch, slow tempo
Anger Negative high arousal Loud, high energy
Fear Negative high arousal Trembling voice
Surprise Positive high arousal Pitch spikes
Disgust Negative medium arousal Lowered energy
Contempt Negative low arousal Unilateral expressions
Neutral Baseline state Normal prosody

Multi-Modal Fusion#

The engine fuses video and audio emotion signals:

  • Weighted Average: 60% video, 40% audio (configurable)
  • Confidence Weighted: Higher confidence = higher weight
  • Conflict Aware: Detects and resolves emotional conflicts
  • Temporal Smoothing: 5-frame EMA with 0.3 alpha

Face Detection#

Backends#

Backend Speed Accuracy GPU
RetinaFace Medium High Yes
MTCNN Fast Medium Optional
MediaPipe Fast Medium Yes
OpenCV Fast Low No

68-Point Landmarks#

Based on MediaPipe FaceMesh with mapping to standard 68 points:

  • Jaw (0-16): Face contour
  • Right Eyebrow (17-21): Right brow points
  • Left Eyebrow (22-26): Left brow points
  • Nose (27-35): Nose bridge and tip
  • Right Eye (36-41): Right eye contour
  • Left Eye (42-47): Left eye contour
  • Outer Lip (48-59): Mouth outer contour
  • Inner Lip (60-67): Mouth inner contour

Screen Understanding#

OCR Engines#

  • Tesseract: System-level OCR with multi-language support
  • EasyOCR: Neural network-based with GPU acceleration

Document Types#

SPREADSHEET, PRESENTATION, DOCUMENT, CODE, TERMINAL, BROWSER, EMAIL, CHAT, DASHBOARD, FORM, PDF, IMAGE, VIDEO

UI Elements#

BUTTON, TEXT_FIELD, DROPDOWN, CHECKBOX, RADIO_BUTTON, LINK, IMAGE, ICON, TABLE, CHART, MENU, TOOLBAR, TAB, MODAL

Quick Start#

Installation#

bash
# Using Nx
nx install psyche-perception-engine

# With all optional backends
nx install-all psyche-perception-engine

# Or directly with Poetry
cd apps/psyche/perception-engine
poetry install
poetry install --extras all

Environment Variables#

bash
# Service Config
PERCEPTION_ENGINE_PORT=8004
PERCEPTION_ENGINE_HOST=0.0.0.0

# Model Paths
FACE_DETECTION_MODEL_PATH=./models/face
EMOTION_MODEL_PATH=./models/emotion

Basic Usage#

python
from perception_engine import PerceptionEngine

# Create engine
engine = PerceptionEngine()
await engine.initialize()

# Process frame and audio
result = await engine.perceive(
    video_frame=frame,
    audio=audio_samples,
    screen=screen_image,
)

# Get results
print(f"Emotion: {result.emotion.dominant}")
print(f"Intent: {result.intent.current}")
print(f"Context: {result.context.summary}")

API Endpoints#

Perception#

  • POST /perceive - Process video/audio/screen
  • GET /status - Get engine status

Emotion#

  • POST /emotion/detect - Detect emotion from frame
  • GET /emotion/current - Get current emotional state

Face#

  • POST /face/detect - Detect faces in frame
  • POST /face/landmarks - Get 68-point landmarks

Screen#

  • POST /screen/analyze - Analyze screen content
  • POST /screen/ocr - Extract text from screen

Development#

Running the Server#

bash
# Development mode with auto-reload
nx serve psyche-perception-engine

# Production mode
nx serve-prod psyche-perception-engine

Running Tests#

bash
nx test psyche-perception-engine
nx test-no-gpu psyche-perception-engine
nx test-cov psyche-perception-engine

Linting & Formatting#

bash
nx lint psyche-perception-engine
nx format psyche-perception-engine

Docker#

bash
nx docker-build psyche-perception-engine
nx docker-run psyche-perception-engine

Nx Integration#

bash
# Available targets
nx serve psyche-perception-engine        # Development server
nx serve-prod psyche-perception-engine   # Production server
nx build psyche-perception-engine        # Build package
nx install psyche-perception-engine      # Install dependencies
nx install-all psyche-perception-engine  # Install all backends
nx lint psyche-perception-engine         # Run linters
nx format psyche-perception-engine       # Format code
nx test psyche-perception-engine         # Run tests
nx test-no-gpu psyche-perception-engine  # Tests without GPU
nx test-cov psyche-perception-engine     # Tests with coverage
nx docker-build psyche-perception-engine # Build Docker image
nx docker-run psyche-perception-engine   # Run Docker container

Performance#

GPU Requirements#

  • Minimum: NVIDIA RTX 3080 (10GB VRAM)
  • Recommended: NVIDIA A100 (40GB+ VRAM)

Processing Times (RTX 4090)#

Task Time
Face Detection 5-15ms
Landmark Detection 10-20ms
Emotion Classification 20-30ms
Audio Processing 100-200ms/sec
Screen OCR 500-2000ms

License#

Proprietary - Oshun Platform