Eve library

Psyche Conferencing

Other

Revision 1

On this page

Video conferencing platform integrations with virtual camera/audio injection.

Part of the Psyche AI Virtual Assistant Platform.

Overview#

The Conferencing service provides unified access to major video conferencing platforms, enabling AI avatar injection into live meetings.

Supported Platforms#

Platform Integration Type Features
Zoom SDK + Browser Hybrid Audio, video, screen share, breakout rooms, webinars
Teams Graph API + Bot Framework Audio, video, screen share, presence, calendar
Meet WebRTC + Browser Audio, video, screen share, chat
Webex SDK Audio, video, screen share

Architecture#

text
video_conferencing/
├── gateway/                    # Unified meeting adapter interface
│   ├── interface.py            # Abstract MeetingAdapter base
│   └── session.py              # Session lifecycle management
├── zoom/                       # Zoom integration (29 files)
│   ├── sdk.py                  # Native Zoom SDK
│   ├── browser.py              # Browser-based fallback
│   ├── hybrid_adapter.py       # SDK-preferred hybrid approach
│   ├── api.py                  # REST API client
│   ├── auth.py                 # OAuth authentication
│   ├── phone.py                # Zoom Phone integration
│   ├── webinar.py              # Webinar support
│   └── ...
├── teams/                      # Microsoft Teams (16 files)
│   ├── adapter.py              # Teams adapter
│   ├── graph.py                # Graph API client
│   ├── auth.py                 # Azure AD authentication
│   ├── bot/                    # Bot framework
│   │   ├── manifest.py         # App manifest generation
│   │   └── registration.py     # Bot registration
│   └── media/                  # Application-hosted media
│       ├── streaming.py        # RTP/RTCP streaming
│       ├── sdp.py              # SDP negotiation
│       ├── codecs.py           # Opus, VP8, H.264
│       └── ...
├── meet/                       # Google Meet (10 files)
│   ├── adapter.py              # Meet adapter
│   ├── webrtc.py               # WebRTC Media API
│   ├── browser.py              # Browser automation
│   └── ...
├── webex/                      # Webex (5 files)
│   ├── adapter.py              # Webex adapter
│   └── sdk.py                  # Webex SDK
├── webrtc/                     # Generic WebRTC support
├── virtual_devices/            # Virtual camera/audio
│   ├── camera.py               # v4l2loopback virtual camera
│   └── audio.py                # PulseAudio routing
├── optimization/               # Performance tuning
│   ├── quality.py              # Adaptive quality selection
│   └── large_meeting.py        # Large meeting optimization
├── multiparticipant/           # Multi-participant handling
│   ├── quality.py              # Per-participant quality tiers
│   ├── buffering.py            # Frame buffering
│   ├── thumbnails.py           # Thumbnail generation
│   └── audio_separation.py     # Audio source separation
├── perception/                 # Meeting perception
│   ├── body_language.py        # Body language analysis
│   ├── emotion.py              # Emotion tracking
│   └── behavior.py             # Behavior analysis
├── meeting_intelligence/       # Analytics
│   ├── calendar.py             # Calendar integration
│   ├── analytics.py            # Meeting analytics
│   └── proactive_assistance/   # Proactive features
└── accessibility/              # Accessibility features
    └── captioning.py           # Real-time captions

Gateway Interface#

All platform adapters implement the unified MeetingAdapter interface:

python
from video_conferencing.gateway import MeetingAdapter

class MeetingAdapter(ABC):
    @abstractmethod
    async def join(self, meeting_url: str, **options) -> Session: ...

    @abstractmethod
    async def leave(self) -> None: ...

    @abstractmethod
    async def send_video(self, frame: np.ndarray) -> None: ...

    @abstractmethod
    async def send_audio(self, samples: np.ndarray) -> None: ...

    @abstractmethod
    async def receive_events(self) -> AsyncIterator[MeetingEvent]: ...

    @abstractmethod
    async def share_screen(self, enable: bool) -> None: ...

Virtual Device Injection#

Virtual Camera (v4l2loopback)#

python
from video_conferencing.virtual_devices import VirtualCamera

# Create virtual camera device
camera = VirtualCamera(device_path="/dev/video10")
await camera.initialize()

# Inject avatar frames
while True:
    frame = await avatar_engine.render_frame()
    await camera.inject_frame(frame)

Virtual Audio (PulseAudio)#

python
from video_conferencing.virtual_devices import VirtualAudio

# Create virtual audio device
audio = VirtualAudio()
await audio.initialize()

# Inject synthesized speech
while True:
    samples = await voice_engine.synthesize()
    await audio.inject_samples(samples)

Quick Start#

Installation#

bash
# Using Nx
nx install psyche-conferencing

# With all platform SDKs
nx install-all psyche-conferencing

# Platform-specific
nx install-zoom psyche-conferencing
nx install-teams psyche-conferencing
nx install-meet psyche-conferencing

# Or directly with Poetry
cd apps/psyche/conferencing
poetry install
poetry install --extras all

Environment Variables#

bash
# Zoom OAuth
ZOOM_CLIENT_ID=your-client-id
ZOOM_CLIENT_SECRET=your-client-secret
ZOOM_ACCOUNT_ID=your-account-id

# Microsoft Teams / Azure AD
AZURE_TENANT_ID=your-tenant-id
AZURE_CLIENT_ID=your-client-id
AZURE_CLIENT_SECRET=your-client-secret

# Google Meet
GOOGLE_CLIENT_ID=your-client-id
GOOGLE_CLIENT_SECRET=your-client-secret
GOOGLE_REFRESH_TOKEN=your-refresh-token

# Webex
WEBEX_CLIENT_ID=your-client-id
WEBEX_CLIENT_SECRET=your-client-secret

# Service Config
CONFERENCING_PORT=8006
CONFERENCING_HOST=0.0.0.0
LOG_LEVEL=INFO

# Virtual Devices
VIRTUAL_CAMERA_DEVICE=/dev/video10
PULSEAUDIO_SINK=virtual_speaker

Basic Usage#

python
from video_conferencing import gateway

# Select adapter based on meeting URL
adapter = gateway.get_adapter("https://zoom.us/j/123456789")

# Join meeting
session = await adapter.join(
    meeting_url="https://zoom.us/j/123456789",
    password="abc123",
    display_name="AI Assistant",
)

# Inject avatar video
async for event in adapter.receive_events():
    if event.type == "participant_speaking":
        # React to speaker
        frame = await avatar.render_reaction(event.participant_id)
        await adapter.send_video(frame)

API Endpoints#

Gateway#

  • POST /meetings/join - Join a meeting
  • POST /meetings/leave - Leave current meeting
  • GET /meetings/status - Get meeting status
  • GET /meetings/participants - List participants

Media#

  • POST /media/video - Send video frame
  • POST /media/audio - Send audio samples
  • GET /media/stream - WebSocket media stream
  • POST /media/screen-share - Toggle screen share

Platform-Specific#

  • POST /zoom/schedule - Schedule Zoom meeting
  • GET /zoom/recordings - Get Zoom recordings
  • POST /teams/presence - Update Teams presence
  • GET /meet/calendar - Get Google Calendar events

Health#

  • GET /health - Health check
  • GET /ready - Readiness check
  • GET /metrics - Prometheus metrics

Development#

Running the Server#

bash
# Development mode
nx serve psyche-conferencing

# Production mode
nx serve-prod psyche-conferencing

Running Tests#

bash
nx test psyche-conferencing
nx test-unit psyche-conferencing
nx test-integration psyche-conferencing
nx test-cov psyche-conferencing

Docker#

bash
# Build image
nx docker-build psyche-conferencing

# Run with virtual device access
nx docker-run psyche-conferencing

Nx Integration#

bash
# Available targets
nx serve psyche-conferencing          # Development server
nx serve-prod psyche-conferencing     # Production server
nx build psyche-conferencing          # Build package
nx install psyche-conferencing        # Install dependencies
nx install-all psyche-conferencing    # Install all platform SDKs
nx install-zoom psyche-conferencing   # Install Zoom SDK
nx install-teams psyche-conferencing  # Install Teams SDK
nx install-meet psyche-conferencing   # Install Meet SDK
nx lint psyche-conferencing           # Run linters
nx format psyche-conferencing         # Format code
nx test psyche-conferencing           # Run all tests
nx test-unit psyche-conferencing      # Unit tests only
nx test-integration psyche-conferencing  # Integration tests
nx test-cov psyche-conferencing       # Tests with coverage
nx docker-build psyche-conferencing   # Build Docker image
nx docker-run psyche-conferencing     # Run container

Platform-Specific Features#

Zoom#

  • Hybrid Adapter: Prefers SDK, falls back to browser automation
  • Breakout Rooms: Create and manage breakout sessions
  • Webinars: Full webinar support with panelist management
  • Zoom Phone: PSTN integration
  • Reactions: Emoji reactions and hand raise
  • Recording: Cloud and local recording management

Microsoft Teams#

  • Bot Framework: Teams app with calling capabilities
  • Graph API: Full Microsoft Graph integration
  • Application-Hosted Media: RTP/RTCP with SDP negotiation
  • Presence: Real-time presence updates
  • Calendar: Meeting scheduling via Graph

Google Meet#

  • WebRTC: Native WebRTC Media API
  • Browser Fallback: Playwright-based automation
  • Screen Share: Full screen sharing support
  • Chat: Real-time chat integration

Webex#

  • SDK Integration: Native Webex SDK
  • WebRTC Media: Standard WebRTC handling

Multi-Participant Optimization#

Adaptive Quality Selection#

python
from video_conferencing.multiparticipant import QualityManager

quality = QualityManager()

# Configure quality tiers
quality.set_tier("speaker", resolution=1080, fps=30)
quality.set_tier("active", resolution=720, fps=24)
quality.set_tier("passive", resolution=480, fps=15)
quality.set_tier("thumbnail", resolution=180, fps=5)

# Auto-assign based on activity
quality.update_participant_tier(participant_id, speaking=True)

Large Meeting Optimization#

  • Participant quality rotation
  • Bandwidth budgeting
  • CPU throttling
  • Roster sampling
  • Cascading fallback

Performance#

Latency Targets#

Operation Target
Video injection < 16ms (60 FPS capable)
Audio injection < 10ms
Platform join < 5s
Reconnection < 2s

Resource Usage#

  • CPU: 2-4 cores per meeting
  • Memory: 512MB - 2GB per meeting
  • Network: 2-10 Mbps per meeting

Security#

  • OAuth 2.0: Platform-specific OAuth flows
  • Token Management: Secure credential storage
  • Sandboxed Execution: Isolated per-meeting contexts
  • Audit Logging: All meeting actions logged

License#

Proprietary - Oshun Platform