Overview#
This guide covers the deployment of the WebRTC Selective Forwarding Unit (SFU) service, which provides real-time voice communication capabilities for the Lilith platform. The SFU handles audio routing, quality management, and peer-to-peer connection establishment.
Architecture#
Components#
- WebRTC SFU Service - Main audio routing and session management
- coturn TURN Server - STUN/TURN server for NAT traversal
- Monitoring Stack - Prometheus metrics and Grafana dashboards
- TLS Infrastructure - Secure communications
Data Flow#
text
Client 1 <---> WebRTC SFU <---> Client 2
| | |
| TURN Server |
| | |
+----- NAT Traversal -------+
Prerequisites#
Infrastructure#
- Kubernetes cluster (v1.24+)
- LoadBalancer service support
- TLS certificates for TURN server
- Node selectors for real-time workloads
Tools#
- kubectl configured for cluster access
- Helm 3.x
- OpenSSL for certificate generation
Deployment Steps#
1. Prepare Environment#
bash
# Clone repository
git clone <repository-url>
cd lilith
# Install dependencies
npm install --legacy-peer-deps
# Build WebRTC service
cd services/webrtc
npm run build
cd ../..
2. Generate TLS Certificates#
bash
# Generate private key
openssl genrsa -out turn.key 2048
# Generate certificate signing request
openssl req -new -key turn.key -out turn.csr -subj "/CN=turn.lilith.app"
# Generate self-signed certificate (for development)
openssl x509 -req -days 365 -in turn.csr -signkey turn.key -out turn.crt
# Create Kubernetes secret
kubectl create secret generic coturn-tls \
--from-file=tls.crt=turn.crt \
--from-file=tls.key=turn.key \
--namespace=lilith-platform
3. Deploy TURN Server#
bash
# Deploy coturn TURN server
kubectl apply -f deploy/webrtc/coturn-deployment.yaml
# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l app=coturn-server -n lilith-platform --timeout=300s
# Verify TURN server
kubectl get pods -n lilith-platform -l app=coturn-server
kubectl get svc -n lilith-platform -l app=coturn-server
4. Deploy WebRTC SFU#
bash
# Build Docker image
cd services/webrtc
docker build -t lilith/webrtc-sfu:latest .
cd ../..
# Deploy SFU service
./deploy/webrtc/deploy-webrtc.sh
# Or use Helm directly
helm upgrade --install webrtc-sfu \
deploy/helm/lilith-service \
--namespace lilith-platform \
--values deploy/webrtc/deploy-values.yaml
Configuration#
SFU Configuration#
The WebRTC SFU service can be configured through environment variables:
yaml
env:
# Basic Configuration
- name: MAX_CONCURRENT_ROOMS
value: '100'
- name: DEFAULT_MAX_PARTICIPANTS
value: '10'
- name: SESSION_TIMEOUT_MINUTES
value: '30'
# Quality Configuration
- name: TARGET_LATENCY_MS
value: '150'
- name: ADAPTIVE_BITRATE_ENABLED
value: 'true'
# TURN Server
- name: TURN_SERVER_URL
value: 'turn:turn.lilith.app:3478'
- name: TURN_SERVER_TLS_URL
value: 'turns:turn.lilith.app:5349'
TURN Server Configuration#
coturn server configuration includes:
- Authentication: Time-limited credentials with HMAC
- TLS: Encrypted TURN over TLS (port 5349)
- Relay: UDP packet relay for NAT traversal
- Quotas: Connection and bandwidth limits
Monitoring#
Metrics#
The WebRTC SFU exposes Prometheus metrics at /metrics:
- Connection count and quality
- Audio bitrate and packet loss
- Room and participant statistics
- System resource usage
Grafana Dashboards#
Pre-configured dashboards include:
- WebRTC SFU Overview - System health and performance
- Connection Quality - Packet loss, latency, jitter
- Resource Usage - CPU, memory, network
- TURN Server - Relay usage and authentication
Alerts#
Configured alerts cover:
- High CPU/Memory usage
- Connection failure rates
- TURN server unavailability
- Audio quality degradation
Security#
Network Security#
- Network policies restrict traffic to necessary ports
- Pod security contexts enforce non-root execution
- TLS encryption for all communications
Authentication#
- TURN server uses time-limited credentials
- WebRTC service requires JWT authentication
- API endpoints are protected with RBAC
TLS Configuration#
yaml
# TURN server TLS
tls:
- name: turn-tls
port: 5349
protocol: TCP
targetPort: 5349
# WebRTC API TLS
ingress:
tls:
- secretName: webrtc-tls
hosts:
- webrtc.lilith.app
Performance Tuning#
SFU Optimization#
- CPU Resources: Allocate sufficient CPU for audio processing
- Memory: Buffer for session state and audio streams
- Network: Low-latency paths and UDP packet handling
- Node Selection: Use dedicated real-time nodes
TURN Server Optimization#
- Relay Capacity: Configure based on expected concurrent users
- Bandwidth Limits: Set appropriate per-user limits
- Geographic Distribution: Deploy close to users
Kubernetes Configuration#
yaml
# Node selector for real-time workloads
nodeSelector:
node-type: realtime
kubernetes.io/os: linux
# Resource allocation
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi
# Anti-affinity for high availability
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: webrtc-sfu
topologyKey: kubernetes.io/hostname
Testing#
Health Checks#
bash
# SFU health check
curl https://webrtc.lilith.app/v1/health
# TURN server test
turnutils_uclient -T -u turn-user:password webrtc.lilith.app
Connection Testing#
Use the WebRTC test tools to verify:
- STUN server connectivity
- TURN server relay functionality
- Audio quality and latency
- Connection establishment
Troubleshooting#
Common Issues#
-
TURN Server Not Accessible
- Check LoadBalancer IP assignment
- Verify firewall rules allow UDP/TCP 3478, TCP 5349
- Validate TLS certificates
-
Audio Quality Issues
- Check network latency and packet loss
- Verify adaptive bitrate configuration
- Monitor resource utilization
-
Connection Failures
- Review TURN server logs
- Check authentication credentials
- Verify DNS resolution
Debug Commands#
bash
# Check pod logs
kubectl logs -n lilith-platform -l app=webrtc-sfu
# Check TURN server logs
kubectl logs -n lilith-platform -l app=coturn-server
# Test connectivity
kubectl exec -it -n lilith-platform deployment/webrtc-sfu -- curl localhost:8080/v1/health
# Monitor metrics
kubectl port-forward -n lilith-platform svc/prometheus 9090:9090
Maintenance#
Updates#
- Rolling Updates: Configure proper deployment strategies
- Blue-Green: Use canary deployments for major changes
- Backup: Export configuration and session data
Scaling#
- Horizontal Scaling: Increase replica count based on load
- Vertical Scaling: Adjust resource limits
- Global Scaling: Deploy in multiple regions
Integration#
Client Integration#
WebRTC clients should be configured with:
javascript
const rtcConfiguration = {
iceServers: [
{ urls: 'stun:stun.l.google.com:19302' },
{
urls: 'turn:turn.lilith.app:3478',
username: turnUsername,
credential: turnCredential,
},
],
iceCandidatePoolSize: 10,
};
Backend Integration#
- Authentication: Use JWT tokens for API access
- Session Management: Create rooms through API endpoints
- Metrics: Forward application metrics to monitoring
Support#
For issues with WebRTC deployment:
- Check the logs and monitoring dashboards
- Review the troubleshooting section
- Consult the Kubernetes and coturn documentation
- Contact the platform team for advanced issues
Appendix#
Ports Reference#
| Service | Port | Protocol | Purpose |
|---|---|---|---|
| WebRTC SFU API | 8080 | TCP | HTTP API |
| TURN Server | 3478 | UDP/TCP | STUN/TURN |
| TURN Server TLS | 5349 | TCP | TURN over TLS |
| TURN Relay | 49152-65535 | UDP | Relay range |
Environment Variables#
| Variable | Default | Description |
|---|---|---|
| MAX_CONCURRENT_ROOMS | 100 | Maximum concurrent rooms |
| DEFAULT_MAX_PARTICIPANTS | 10 | Default participants per room |
| SESSION_TIMEOUT_MINUTES | 30 | Session timeout in minutes |
| TARGET_LATENCY_MS | 150 | Target audio latency |
| ADAPTIVE_BITRATE_ENABLED | true | Enable adaptive bitrate |
| TURN_SERVER_URL | - | TURN server URL |
| TURN_SERVER_TLS_URL | - | TURN over TLS URL |