psyche library

Objectives and guardrails management for Tavus (src/index.ts): workflow configuration, compliance templates, and trigger handling for conversation objectives.

Phase 85–86 flywheel producer for the avatar/assistant domain (libs/psyche/training-data/src), split into two pipelines rather than the single-pipeline shape its siblings use: PsycheAvatarSignalPipeline captures avatar-animation signals (typed FlameFrame 3DMM coefficients and VisemeFrame lip-sync frames) and PsychePerceptionSignalPipeline captures perception signals, both under an explicit PsycheTrainingConsent and a shared PsycheTrainingSink seam.

Tiered avatar caching for fast switching/preloading: memory + persistent cache entries, expression caching, preloading, and statistics/event types, for low-latency avatar operations (src/index.ts).

3D Gaussian Splatting avatar rendering core (src/index.ts): FLAME parametric model utilities (neutral state, expression mapping, coefficient blending, pose), viseme/lip-sync application, and the shared avatar type surface for the rest of the avatar-* fleet.

Integration layer bridging @isis/3d-generation and psyche-avatar-core: text-to-3D and photo-to-3D avatar generation requests/results, blend-shape mapping/conversion between the two systems, and an avatar generator (src/index.ts). A cross-domain seam.

Temporal anti-aliasing for Gaussian-splatting avatars: jitter generation, motion vectors, history accumulation, and ghost reduction for stable temporal output, with vector/matrix/color helper types (src/index.ts).

Avatar training pipeline (monocular video → 3D Gaussian Splatting): video preprocessing orchestration, FLAME parameter fitting, Gaussian-splat training, blendshape optimization, and expression calibration (src/index.ts).

Conversation-level sentiment tracking via signal fusion (src/index.ts): aggregates facial/vocal/text/engagement/atmosphere signals into smoothed temporal sentiment arcs with turning-point detection, intervention recommendations, phase analysis, speaker congruence, and cross-conversation comparison.

Dialogue management for natural conversation flow (src/index.ts): turn-taking (floor control, overlap, push-to-talk), interruption detection/handling, conversation-state tracking, intent detection, and multi-participant support in an event-driven design.

Emotion modeling for avatar animation (src/index.ts): primary/extended emotions, the Russell circumplex (valence-arousal), compatibility-based blending, emotion-specific transitions, mood persistence/contamination, social display rules, and FACS/blendshape output.

Multi-dimensional engagement detection (src/index.ts): gaze, behavioral, conversational, temporal, and content engagement combined with configurable weights into smoothed scores, trend detection, and signal detection (attention loss, interest spikes, fatigue) with session history.

Face detection and landmark tracking (src/index.ts): realtime detection, the 478-point MediaPipe face mesh, IoU-based tracking with smoothing, head-pose estimation, and a pluggable detection backend.

Biologically-grounded gaze generation (src/index.ts, cites saccade main-sequence and conversational-gaze research): saccades with main-sequence kinematics, context-aware fixations, microsaccade/drift, smooth pursuit, vergence, conversation-aware coordination, cultural profiles, and ARKit output.

Unlimited archival memory (src/index.ts): vector-based semantic retrieval, importance/tagging, and consolidation, built around an injectable embedding provider (createMockEmbeddingProvider for offline tests; real providers in production).

LLM-callable MemGPT-style memory tools (src/index.ts): tool schema/definitions, an execution engine, and backend integration that let an LLM manage its own memory blocks.

TTS library with ElevenLabs and Cartesia providers (src/index.ts): streaming, voice cloning, SSML/prosody control, and a provider abstraction with voice settings, audio-output, and error types.

Custom-LLM integration for Tavus (src/index.ts): an OpenAI-compatible API proxy for Claude — OpenAI↔Claude message translation, streaming SSE translation, a /chat/completions proxy server, system-prompt sync, and tool-calling — via createProxyServer/buildTavusLLMConfig.

Tavus persona management (src/index.ts): persona templating with variable substitution (e.g. CUSTOMER_SERVICE_TEMPLATE), versioning, cloning, and configuration validation via createPersonaManager.

Video-conferencing bridge for Tavus (libs/psyche/tavus-bridge, ~48K LOC across the package; src/index.ts is the 641-line barrel): connects Tavus avatar streams to virtual camera/audio devices (WebRTC, virtual-camera/-audio types) for use with Zoom, Meet, Teams, etc.

TypeScript API client for the Tavus CVI (Conversational Video Interface) platform (src/index.ts): replica/persona/conversation operations with response caching, request batching, and usage monitoring via createTavusClient. The base the other Tavus libs build on.

Tavus conversation management (src/index.ts): configuration presets (e.g. CUSTOMER_SERVICE_PRESET), recording management, transcript handling, analytics, and pluggable storage (createInMemoryStorage) via createConversationManager.

Hybrid-mode orchestration between Psyche-native and Tavus avatar systems (src/index.ts): a unified avatar interface, mode switching, health monitoring, failover, and graceful degradation — the bridge that lets the two stacks run interchangeably.

Phoenix-4 rendering-pipeline integration for Tavus (src/index.ts): WebRTC video streaming, frame synchronization, and quality management via createRenderPipeline/ createQualityManager with QUALITY_PRESETS.

Pipecat-inspired frame-based pipeline architecture for Tavus (src/index.ts): frame types, processors, pipelines, and services for realtime conversational AI.

Tavus replica lifecycle management (src/index.ts): versioning, quality validation, workflow automation, and metadata storage via createReplicaManager/ createReplicaId.

Tool-calling integration for Tavus (src/index.ts): tool-event handling, execution, security/audit, and built-in tools, with branded tool/execution types.

Raven-1 visual-perception integration for Tavus (src/index.ts): emotion detection, video forwarding, a perception handler, LLM context, a facial-expression pipeline, caching/filtering/analytics, screen-content analysis, gesture recognition, and a learning optimizer, with hybrid Tavus+Psyche signals.

Integration bridging Sophia search into Psyche (src/index.ts): a PsycheSearchAdapter, a fluent PsycheQueryBuilder, a citation-producing PsycheRAGIntegration, and a result mapper translating Sophia results to Psyche form.

The largest Python foundation lib (src/common, ~16.7K LOC): shared utilities across all Psyche services — GPU optimization (quantization, batch inference, TensorRT, memory management), structured logging, success-metrics tracking (human-likeness, conversation quality, cost), performance/streaming pipelines, and horizontal-scaling helpers (per README.md).

End-to-end test suite (*.e2e.test.ts, ~4K LOC across four specs): vitest tests exercising avatar rendering, voice pipeline (STT→LLM→TTS with <500ms latency), conferencing, and performance, using mock providers to simulate without live API calls. A test project, not a runtime library.

AIOps anomaly detection (src/index.ts): statistical detectors (Z-score, modified Z-score, IQR, Grubbs, MAD, EWMA), time-series trend/seasonality/forecasting, CUSUM changepoint detection, metric correlation (Pearson/Spearman/cross-corr), root-cause inference, and alert management.

Attention tracking/analysis (src/index.ts): attention-state tracking, shift and lapse detection with recovery, collective multi-participant attention, attention heatmaps, engagement scoring, topic-attention correlation, pattern detection, and fatigue recommendations.

Behavioral anomaly detection for meeting participants (src/index.ts): inactivity detection, erratic-behavior patterns, conflict-escalation, cross-participant analysis, and meeting-atmosphere scoring with a default config.

Realtime context monitoring (src/index.ts): signal collection/aggregation, pattern and anomaly detection, state tracking, threshold-based alerting, and analytics, with Zod schemas for signals and configuration.

Database sharding (src/index.ts): strategies (hash, range, consistent-hash, directory, geographic, composite), a consistent-hash ring with virtual nodes (FNV-1a/DJB2/Murmur-like), shard-key extraction, query routing (single/ scatter-gather/broadcast/targeted), migration, rebalancing, split/merge, and hotspot detection.

Kubernetes disaster recovery (src/index.ts): backup scheduling/execution/ verification/retention, recovery with validation+rollback, cross-site replication with lag monitoring, RPO/RTO/SLA tracking, failover-plan orchestration, DR test planning, and posture scoring.

Continuous human-evaluation pipeline (src/index.ts): evaluation sessions with counterbalancing, multiple rating types (Likert/binary/ranking/continuous), attention checks, inter-rater reliability (ICC, Fleiss' Kappa, Krippendorff's Alpha), significance testing, trend analysis, and quality alerts.

Kubernetes multi-cluster federation (src/index.ts): cluster registration/ lifecycle/health, federated resource distribution with pluggable placement, cross-cluster service discovery, multi-strategy traffic routing (round-robin, weighted, latency, geo, failover), and automatic failover/recovery.

Kubernetes security (src/index.ts): IRSA (IAM Roles for Service Accounts) for AWS EKS, Pod Security Standards validation, Network Policy management, and RBAC configuration/analysis.

Latency analysis (src/index.ts): component-level latency tracking/breakdown, histograms with streaming percentile estimation, statistical regression detection, latency-threshold/anomaly alert rules, and trace critical-path analysis.

Log-based metrics (src/index.ts): multi-format parsing (JSON, logfmt, CLF, Combined, Syslog, custom regex), level classification, timestamp normalization, pattern matching, rule-based metric/label extraction, time-windowed aggregation, and analytics.

ML log analysis (src/index.ts): log clustering (Drain, cosine, Jaccard, Levenshtein, token-frequency), template extraction, TF-IDF vectorization, frequency anomaly detection, classification, sequential-pattern mining, and root-cause inference.

Net Promoter Score collection/analysis (src/index.ts): survey management, response recording, NPS calculation, and an analytics engine producing reports via createNPSSurveyManager/createNPSAnalyticsEngine/calculateNPS.

Meeting participation tracking (src/index.ts): speaker analysis, balance metrics, a MeetingDynamicsAnalyzer, a ParticipationTrendAnalyzer, and the main ParticipationTracker, correlating participation with engagement.

Predictive alerting (src/index.ts): forecasting (linear regression, exponential smoothing, Holt's, Holt-Winters, moving average, ARIMA-like), trend detection, capacity/exhaustion forecasting, lead-time alert rules, notification routing with rate limiting, suppression/maintenance windows, and accuracy tracking.

Customer satisfaction surveys (CSAT, CES) for AI evaluation (src/index.ts): survey creation/publishing, response recording, and CSAT/CES calculation via createSatisfactionSurveyManager/calculateCSAT/calculateCES.

SLO-based alerting (src/index.ts, cites Google SRE): SLI definition/measurement, SLO management, error-budget tracking, multi-window burn-rate alerting, and Prometheus-compatible alert-rule generation.

Expression/emotion control bridge for Tavus (src/index.ts): a TavusExpressionController (extends EventEmitter) providing realtime expression updates, lip sync, and quality monitoring.

Knowledge-base/RAG integration for Tavus (src/index.ts): document management, retrieval configuration, and cross-system synchronization of knowledge between Tavus and Psyche.

Memory management/synchronization for Tavus (src/index.ts): cross-conversation persistence, tag management, and Psyche memory integration.

Full Tavus CVI pipeline (src/index.ts): WebRTC + Daily.co room management, data channel communication, and conversation-lifecycle management via createCVIPipeline, integrating turn-taking, perception, and tool calling.

TTS integration for Tavus (src/index.ts): Cartesia and ElevenLabs voice selection, configuration, presets, and persona-TTS management via createTavusTTSManager/getTTSConfigForUseCase/buildPersonaTTSUpdate.

Sparrow-1 turn-taking integration for Tavus (src/index.ts): floor manager, turn detector, speaker adaptation, prosodic signals, syllable detection, hesitation handling, cultural timing, and multi-speaker support in hybrid Tavus+Psyche mode.

Time-series optimization (src/index.ts): downsampling, compression, retention policies, time-bucketing, query optimization, storage tiering, materialized-view management, and LRU caching, with Zod schemas.

Uncanny-valley detection/scoring for avatar quality (src/index.ts): per-category collectors (blink, gaze, expression, head/body movement, response timing, lip-sync, visual quality) scored against DEFAULT_BASELINES human baselines with weighted aggregation into an UncannyValleyScore.

Accessibility library for Psyche agents (src/index.ts, ~30K LOC): WCAG 2.2 compliance — contrast checking (4.5:1), pointer alternatives for gestures, minimum target sizes (24×24), redundant-entry prevention — plus captioning, screen-reader, motor, and hearing accessibility.

Action validation and dangerous-action blocking (src/index.ts): a SafetyChecker, confirmation workflows, rate limiting, and a policy engine that classify and gate agent actions (e.g. blocking rm -rf /).

Anticipatory response modeling (src/index.ts): an anticipation modeler over incoming signals (with AnticipationSignalSchema) to pre-compute likely responses, plus utility/constant helpers.

Python authentication library (src/psyche_auth) aligned with @oshun/auth-primitives: JWT signing/verification, in-memory + Redis session stores, API-key management, Argon2/bcrypt/PBKDF2 password utilities, and FastAPI auth middleware (per its README.md).

Expression blendshapes and FACS mapping for avatars: ARKit blendshapes, FACS action units, emotion presets, micro-expressions, transitions, asymmetry, and a controller-state surface with Zod schemas (src/index.ts).

Lip-sync/viseme animation: a 19-class viseme system with 31 blendshape parameters, ARPAbet/IPA phoneme-to-viseme mapping, coarticulation blending, and a real-time StreamingAligner with lookahead buffering targeting <33ms latency (src/index.ts).

Adaptive quality system for avatar rendering: GPU detection, quality levels, dynamic resolution scaling, and Gaussian-count adaptation with performance/Gaussian types and Zod schemas (src/index.ts).

Priority-based behavior coordination for avatar animation (src/index.ts): behavior definitions with categories/priorities/durations, a priority queue with conflict resolution, fade-in/out timing, multi-channel output blending, and state machines via a BehaviorOrchestrator.

User-behavior prediction (src/index.ts): Markov-chain transition modeling, sequence pattern mining, temporal pattern detection, ensemble prediction, accuracy tracking, and analytics, with Zod event/config schemas.

Playwright-backed browser automation (src/index.ts): navigation/interaction, form filling, the Page Object pattern, network interception/mocking, screenshots, cookie/storage management, and retry/wait utilities via createBrowserAutomation.

Python caching library (src/psyche_cache) aligned with @oshun/cache: LRU+TTL in-memory cache, Redis distributed cache, namespaced key building, pub/sub state sync, Redlock-compatible distributed locking, and standard TTL presets.

Realtime translated-caption stream management (src/index.ts): caption-segment ingestion, multi-language output channels, speaker attribution, word-level timing, translation integration, buffering, latency tracking, rendering, synchronization, and transcript aggregation.

Provider-neutral native desktop control (src/index.ts): separately injected planning/vision bindings, governed screenshots, native action/result types, and an observe-reason-act agent. Every run requires exact app/window/action/network/ task-file authority, process-confinement attestation, risk-based confirmation, fresh-frame and focus interlocks, step/action/rate/time/token/interrupt budgets, and non-model end-state verification. The library is not a browser driver and is not yet admitted to Eve/Drawer; ADR-0077 owns browser separation and ADR-0080 owns the per-run native control boundary.

Platform-agnostic video-conferencing abstraction (src/index.ts): the IConferenceSession interface, video injection modes (replace/overlay/PIP/ background), audio injection/mixing (replace/blend/ducking/sidechain) with EQ, compression, and noise gating, plus createMockConferenceSession for testing. The base every conferencing adapter implements.

Crowdsourced evaluation at scale (src/index.ts): worker management with qualification tiers, HIT lifecycle, quality control (gold standards, spam detection), response aggregation (majority vote, weighted average, Dawid-Skene EM, median, trimmed mean), cost tracking, and campaign orchestration.

Translation-aware cultural adaptation (src/index.ts): idiom detection/ adaptation, cultural-reference handling, formality-register mapping, honorific systems, measurement conversion, sensitivity filtering, and dialect selection over regional databases.

Dual-language data layer: a Python async client (src/psyche_database, PostgreSQL via asyncpg, Redis, transactions, migrations, query builder, SQLAlchemy models, Qdrant vector storage) plus a Prisma-generated TypeScript client under src/generated/client/ (models such as AvatarPack, VoiceProfile). Aligned with @oshun/database.

Dialect/accent handling (src/index.ts): regional variant detection, rule-based text adaptation between dialects, accent-profile management, formality registers, multi-dialect conversation coordination, and metrics, with Zod schemas.

Realtime multi-face emotion pipeline (src/index.ts): per-face emotion tracking with persistent identity, temporal pattern + micro-expression analysis, dominant emotion calculation, suppression/genuineness detection, and quality scoring.

Disengagement detection and recovery orchestration (src/index.ts): a disengagement detector feeding recovery-strategy selection, with Zod schemas for engagement snapshots and recovery config.

Facial analysis on landmarks (src/index.ts): FACS action-unit detection, Ekman + extended expression classification, the valence-arousal-dominance circumplex, age/gender estimation, and temporal smoothing.

Facial Action Coding System library (src/index.ts): action-unit definitions, intensity control, AU-to-emotion mapping, blendshape combinations, and expression state management for facial animation.

Follow-up initiation (src/index.ts): types and logic for deciding when and how the assistant should proactively initiate a follow-up. A compact lib (one barrel over a types module and engine).

Gaze-awareness behaviors (src/index.ts): maps gaze points/targets onto content and derives awareness-driven behaviors, with Zod input schemas for gaze points, targets, and configuration.

Eye-gaze estimation from landmarks (src/index.ts): eye-region extraction, iris/pupil tracking, pitch/yaw gaze direction with head-pose correction, calibrated screen mapping, exponential+Kalman filtering, and fixation/saccade/blink and AOI analysis.

Gesture generation/animation (src/index.ts): 54+ predefined gesture types (beats, emblematic, pointing, iconic, metaphoric), keyframe animation with easing, priority-based multi-gesture blending, speech-synced beats, cultural adaptation by locale, and personality variation.

Generative head movement (src/index.ts): conversational nods, disagreement shakes, interest/confusion/empathy/thinking tilts, speech-synced emphasis, idle micro-sway/drift, stimulus responses, and cultural/personality adaptation.

Head-pose estimation from landmarks (src/index.ts): PnP-based pose with a geometric fallback, multi-format landmark support (MediaPipe/dlib/OpenPose), movement detection (nods/shakes/tilts/turns), stability and range-of-motion tracking, and fatigue indicators.

Help-offer triggering (src/index.ts): a help-signal aggregator over HelpSignalInputSchema inputs with HelpOfferConfigSchema to decide when to surface help. A small, focused trigger lib.

Automated human-likeness estimation via multi-modal signal fusion (src/index.ts): behavioral, conversational, emotional, linguistic, and consistency signals with feature-weighted scoring, artifact detection, ground-truth calibration, trend analysis, A/B testing, and benchmarking.

Python integration test suite (tests/test_*.py, ~3.3K LOC across three modules): pytest tests covering cross-service event flows, pipeline integration, and service communication for the Python foundation. A test project, not a runtime library.

RAG context assembly (src/index.ts): a ContextAssembler with strategies, a TokenBudgetManager, SourceAttribution, and a ContextFormatter for citation-aware context construction within a token budget.

RAG ingestion pipeline (src/index.ts): multiple chunking strategies (fixed-size, semantic, recursive, token-based), an embedding-service abstraction with caching/batching, vector-index management, document versioning, and progress reporting via createIngestionPipeline.

RAG retrieval (src/index.ts): semantic vector search (cosine/euclidean/dot/ manhattan), BM25/TF-IDF keyword search, hybrid retrieval (RRF, weighted, cascade), and re-ranking (MMR, weighted, recency decay) with query/result caching.

Language detection (src/index.ts): text statistical analysis, script detection, n-gram profiling, audio spectral/phonemic analysis, multi-language content detection, language-change tracking, and realtime streaming detection, with Zod schemas.

Python structured-logging library (src/psyche_logging) aligned with @oshun/logging: JSON log schema, contextvars-based context propagation, log sampling strategies, PII redaction, optional OpenTelemetry trace enrichment, and child loggers.

Google Meet adapter (src/index.ts): browser-automation (Puppeteer) + Calendar API integration for joining meetings with raw audio/video, via createMeetConferenceSession. Requires Google OAuth credentials.

AI meeting summarization (src/index.ts): processes transcripts, participant data, events, and engagement metrics into structured summaries (topics, action items, decisions, questions, engagement analysis) with input/output models.

Memory consolidation (src/index.ts): time-based decay (exponential, power-law, stepped), multi-factor importance scoring, similar-memory merging, low-value pruning, and automated consolidation scheduling.

MemGPT-style hierarchical memory (src/index.ts): multi-tier storage with automatic paging and capacity management via a VirtualContextManager (add, search, context-window assembly, background monitoring). The anchor of the memory-* cluster.

Bridge between Psyche memory and @sophia/indexing embeddings (src/index.ts): memory-specific embedding strategies, tier-aware TTL caching, a batch pipeline with progress, and similarity search over memory blocks. A cross-domain seam.

8K-token in-context memory (src/index.ts): per-message extraction of key facts, topics, and entities, with buildContext assembling an optimized, token-estimated context for the LLM.

Cross-session memory persistence (src/index.ts): multi-backend storage (memory, file, PostgreSQL, Redis, S3), gzip/brotli compression, AES-256-GCM encryption at rest, serialization, session management, and integrity verification.

Advanced memory retrieval (src/index.ts): hybrid semantic (vector) + keyword (BM25/TF-IDF) search, re-ranking (RRF, MMR, importance, recency, hybrid-score), and deduplication, via a MemoryRetrievalService.

32K-token working memory (src/index.ts): LRU eviction with auto promotion/ demotion between tiers, importance-weighted entries, and statistics via a WorkingMemoryManager.

Python event bus (src/psyche_messaging) aligned with @oshun/event-bus: pattern/wildcard pub-sub, event persistence with replay TTL, retry with exponential/linear/fixed backoff, dead-letter queue, correlation tracking, and cross-domain event routing.

Micro-expression generation (src/index.ts): brief involuntary expressions (40–200ms), natural blink patterns with variability, eye-moisture/tearing simulation, subtle tics/twitches/saccades, and suppression/leakage and deception simulation.

Mobile SDK surface (src/index.ts): iOS and Android SDK modules, cross-platform wrappers, white-label capabilities, edge/offline deployment, and multi-agent collaboration (Phase 24.18), re-exported from per-platform modules.

Audio pre-processing for voice (src/index.ts): noise suppression, echo cancellation, and audio normalization, composable individually or via a unified createAudioProcessor operating on sample buffers.

Per-participant language preference management (src/index.ts): source-priority preference resolution, dialect identifiers, auto-detection integration, fallback chains, conflict resolution, meeting language-policy enforcement, change history, and translation routing.

Multi-participant stream management (src/index.ts, ~85K LOC, one of the largest libs): participant tracking, gallery pagination with active-speaker follow, processing queues, and feature degradation for large meetings via createParticipantTracker/ createGalleryManager/createProcessingQueueManager/createDegradationManager.

Root Poetry/pytest aggregator for the Python tier (libs/psyche/pyproject.toml, conftest.py, sourceRoot: libs/psyche); ships no package of its own (packages = []) but owns shared test/lint config and the Nx implicitDependencies that wire the foundation and feature libs into one graph.

Posture animation (src/index.ts): breathing, weight shifting, and social mirroring, exposing the core posture types and a posture controller.

Predictive performance modeling (src/index.ts): multi-metric performance tracking, model training (linear regression, exponential smoothing, Holt-Winters, moving average, ensemble), degradation/capacity-exhaustion/anomaly forecasting, risk assessment, and orchestrated monitoring sessions.

Proactive-assistance trigger detection (src/index.ts): a need detector over indicator inputs (NeedIndicatorInputSchema) plus constants, utilities, and factory functions for firing assistance offers.

Proactive suggestion engine (src/index.ts): signal analysis, trigger-based suggestion generation, suppression/deduplication, user-feedback tracking, pattern learning, and analytics, with Zod schemas.