Eve library

WebRTC SFU Deployment Guide

Guide

Revision 1

On this page

Overview#

This guide covers the deployment of the WebRTC Selective Forwarding Unit (SFU) service, which provides real-time voice communication capabilities for the Lilith platform. The SFU handles audio routing, quality management, and peer-to-peer connection establishment.

Architecture#

Components#

  1. WebRTC SFU Service - Main audio routing and session management
  2. coturn TURN Server - STUN/TURN server for NAT traversal
  3. Monitoring Stack - Prometheus metrics and Grafana dashboards
  4. TLS Infrastructure - Secure communications

Data Flow#

text
Client 1 <---> WebRTC SFU <---> Client 2
    |             |                |
    |         TURN Server        |
    |             |                |
    +----- NAT Traversal -------+

Prerequisites#

Infrastructure#

  • Kubernetes cluster (v1.24+)
  • LoadBalancer service support
  • TLS certificates for TURN server
  • Node selectors for real-time workloads

Tools#

  • kubectl configured for cluster access
  • Helm 3.x
  • OpenSSL for certificate generation

Deployment Steps#

1. Prepare Environment#

bash
# Clone repository
git clone <repository-url>
cd lilith

# Install dependencies
npm install --legacy-peer-deps

# Build WebRTC service
cd services/webrtc
npm run build
cd ../..

2. Generate TLS Certificates#

bash
# Generate private key
openssl genrsa -out turn.key 2048

# Generate certificate signing request
openssl req -new -key turn.key -out turn.csr -subj "/CN=turn.lilith.app"

# Generate self-signed certificate (for development)
openssl x509 -req -days 365 -in turn.csr -signkey turn.key -out turn.crt

# Create Kubernetes secret
kubectl create secret generic coturn-tls \
  --from-file=tls.crt=turn.crt \
  --from-file=tls.key=turn.key \
  --namespace=lilith-platform

3. Deploy TURN Server#

bash
# Deploy coturn TURN server
kubectl apply -f deploy/webrtc/coturn-deployment.yaml

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l app=coturn-server -n lilith-platform --timeout=300s

# Verify TURN server
kubectl get pods -n lilith-platform -l app=coturn-server
kubectl get svc -n lilith-platform -l app=coturn-server

4. Deploy WebRTC SFU#

bash
# Build Docker image
cd services/webrtc
docker build -t lilith/webrtc-sfu:latest .
cd ../..

# Deploy SFU service
./deploy/webrtc/deploy-webrtc.sh

# Or use Helm directly
helm upgrade --install webrtc-sfu \
  deploy/helm/lilith-service \
  --namespace lilith-platform \
  --values deploy/webrtc/deploy-values.yaml

Configuration#

SFU Configuration#

The WebRTC SFU service can be configured through environment variables:

yaml
env:
  # Basic Configuration
  - name: MAX_CONCURRENT_ROOMS
    value: '100'
  - name: DEFAULT_MAX_PARTICIPANTS
    value: '10'
  - name: SESSION_TIMEOUT_MINUTES
    value: '30'

  # Quality Configuration
  - name: TARGET_LATENCY_MS
    value: '150'
  - name: ADAPTIVE_BITRATE_ENABLED
    value: 'true'

  # TURN Server
  - name: TURN_SERVER_URL
    value: 'turn:turn.lilith.app:3478'
  - name: TURN_SERVER_TLS_URL
    value: 'turns:turn.lilith.app:5349'

TURN Server Configuration#

coturn server configuration includes:

  • Authentication: Time-limited credentials with HMAC
  • TLS: Encrypted TURN over TLS (port 5349)
  • Relay: UDP packet relay for NAT traversal
  • Quotas: Connection and bandwidth limits

Monitoring#

Metrics#

The WebRTC SFU exposes Prometheus metrics at /metrics:

  • Connection count and quality
  • Audio bitrate and packet loss
  • Room and participant statistics
  • System resource usage

Grafana Dashboards#

Pre-configured dashboards include:

  1. WebRTC SFU Overview - System health and performance
  2. Connection Quality - Packet loss, latency, jitter
  3. Resource Usage - CPU, memory, network
  4. TURN Server - Relay usage and authentication

Alerts#

Configured alerts cover:

  • High CPU/Memory usage
  • Connection failure rates
  • TURN server unavailability
  • Audio quality degradation

Security#

Network Security#

  • Network policies restrict traffic to necessary ports
  • Pod security contexts enforce non-root execution
  • TLS encryption for all communications

Authentication#

  • TURN server uses time-limited credentials
  • WebRTC service requires JWT authentication
  • API endpoints are protected with RBAC

TLS Configuration#

yaml
# TURN server TLS
tls:
  - name: turn-tls
    port: 5349
    protocol: TCP
    targetPort: 5349

# WebRTC API TLS
ingress:
  tls:
    - secretName: webrtc-tls
      hosts:
        - webrtc.lilith.app

Performance Tuning#

SFU Optimization#

  1. CPU Resources: Allocate sufficient CPU for audio processing
  2. Memory: Buffer for session state and audio streams
  3. Network: Low-latency paths and UDP packet handling
  4. Node Selection: Use dedicated real-time nodes

TURN Server Optimization#

  1. Relay Capacity: Configure based on expected concurrent users
  2. Bandwidth Limits: Set appropriate per-user limits
  3. Geographic Distribution: Deploy close to users

Kubernetes Configuration#

yaml
# Node selector for real-time workloads
nodeSelector:
  node-type: realtime
  kubernetes.io/os: linux

# Resource allocation
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 1Gi

# Anti-affinity for high availability
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: webrtc-sfu
          topologyKey: kubernetes.io/hostname

Testing#

Health Checks#

bash
# SFU health check
curl https://webrtc.lilith.app/v1/health

# TURN server test
turnutils_uclient -T -u turn-user:password webrtc.lilith.app

Connection Testing#

Use the WebRTC test tools to verify:

  1. STUN server connectivity
  2. TURN server relay functionality
  3. Audio quality and latency
  4. Connection establishment

Troubleshooting#

Common Issues#

  1. TURN Server Not Accessible

    • Check LoadBalancer IP assignment
    • Verify firewall rules allow UDP/TCP 3478, TCP 5349
    • Validate TLS certificates
  2. Audio Quality Issues

    • Check network latency and packet loss
    • Verify adaptive bitrate configuration
    • Monitor resource utilization
  3. Connection Failures

    • Review TURN server logs
    • Check authentication credentials
    • Verify DNS resolution

Debug Commands#

bash
# Check pod logs
kubectl logs -n lilith-platform -l app=webrtc-sfu

# Check TURN server logs
kubectl logs -n lilith-platform -l app=coturn-server

# Test connectivity
kubectl exec -it -n lilith-platform deployment/webrtc-sfu -- curl localhost:8080/v1/health

# Monitor metrics
kubectl port-forward -n lilith-platform svc/prometheus 9090:9090

Maintenance#

Updates#

  1. Rolling Updates: Configure proper deployment strategies
  2. Blue-Green: Use canary deployments for major changes
  3. Backup: Export configuration and session data

Scaling#

  1. Horizontal Scaling: Increase replica count based on load
  2. Vertical Scaling: Adjust resource limits
  3. Global Scaling: Deploy in multiple regions

Integration#

Client Integration#

WebRTC clients should be configured with:

javascript
const rtcConfiguration = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },
    {
      urls: 'turn:turn.lilith.app:3478',
      username: turnUsername,
      credential: turnCredential,
    },
  ],
  iceCandidatePoolSize: 10,
};

Backend Integration#

  1. Authentication: Use JWT tokens for API access
  2. Session Management: Create rooms through API endpoints
  3. Metrics: Forward application metrics to monitoring

Support#

For issues with WebRTC deployment:

  1. Check the logs and monitoring dashboards
  2. Review the troubleshooting section
  3. Consult the Kubernetes and coturn documentation
  4. Contact the platform team for advanced issues

Appendix#

Ports Reference#

Service Port Protocol Purpose
WebRTC SFU API 8080 TCP HTTP API
TURN Server 3478 UDP/TCP STUN/TURN
TURN Server TLS 5349 TCP TURN over TLS
TURN Relay 49152-65535 UDP Relay range

Environment Variables#

Variable Default Description
MAX_CONCURRENT_ROOMS 100 Maximum concurrent rooms
DEFAULT_MAX_PARTICIPANTS 10 Default participants per room
SESSION_TIMEOUT_MINUTES 30 Session timeout in minutes
TARGET_LATENCY_MS 150 Target audio latency
ADAPTIVE_BITRATE_ENABLED true Enable adaptive bitrate
TURN_SERVER_URL - TURN server URL
TURN_SERVER_TLS_URL - TURN over TLS URL