# WebRTC SFU Deployment Guide

## Overview

This guide covers the deployment of the WebRTC Selective Forwarding Unit (SFU)
service, which provides real-time voice communication capabilities for the
Lilith platform. The SFU handles audio routing, quality management, and
peer-to-peer connection establishment.

## Architecture

### Components

1. **WebRTC SFU Service** - Main audio routing and session management
2. **coturn TURN Server** - STUN/TURN server for NAT traversal
3. **Monitoring Stack** - Prometheus metrics and Grafana dashboards
4. **TLS Infrastructure** - Secure communications

### Data Flow

```
Client 1 <---> WebRTC SFU <---> Client 2
    |             |                |
    |         TURN Server        |
    |             |                |
    +----- NAT Traversal -------+
```

## Prerequisites

### Infrastructure

- Kubernetes cluster (v1.24+)
- LoadBalancer service support
- TLS certificates for TURN server
- Node selectors for real-time workloads

### Tools

- kubectl configured for cluster access
- Helm 3.x
- OpenSSL for certificate generation

## Deployment Steps

### 1. Prepare Environment

```bash
# Clone repository
git clone <repository-url>
cd lilith

# Install dependencies
npm install --legacy-peer-deps

# Build WebRTC service
cd services/webrtc
npm run build
cd ../..
```

### 2. Generate TLS Certificates

```bash
# Generate private key
openssl genrsa -out turn.key 2048

# Generate certificate signing request
openssl req -new -key turn.key -out turn.csr -subj "/CN=turn.lilith.app"

# Generate self-signed certificate (for development)
openssl x509 -req -days 365 -in turn.csr -signkey turn.key -out turn.crt

# Create Kubernetes secret
kubectl create secret generic coturn-tls \
  --from-file=tls.crt=turn.crt \
  --from-file=tls.key=turn.key \
  --namespace=lilith-platform
```

### 3. Deploy TURN Server

```bash
# Deploy coturn TURN server
kubectl apply -f deploy/webrtc/coturn-deployment.yaml

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l app=coturn-server -n lilith-platform --timeout=300s

# Verify TURN server
kubectl get pods -n lilith-platform -l app=coturn-server
kubectl get svc -n lilith-platform -l app=coturn-server
```

### 4. Deploy WebRTC SFU

```bash
# Build Docker image
cd services/webrtc
docker build -t lilith/webrtc-sfu:latest .
cd ../..

# Deploy SFU service
./deploy/webrtc/deploy-webrtc.sh

# Or use Helm directly
helm upgrade --install webrtc-sfu \
  deploy/helm/lilith-service \
  --namespace lilith-platform \
  --values deploy/webrtc/deploy-values.yaml
```

## Configuration

### SFU Configuration

The WebRTC SFU service can be configured through environment variables:

```yaml
env:
  # Basic Configuration
  - name: MAX_CONCURRENT_ROOMS
    value: '100'
  - name: DEFAULT_MAX_PARTICIPANTS
    value: '10'
  - name: SESSION_TIMEOUT_MINUTES
    value: '30'

  # Quality Configuration
  - name: TARGET_LATENCY_MS
    value: '150'
  - name: ADAPTIVE_BITRATE_ENABLED
    value: 'true'

  # TURN Server
  - name: TURN_SERVER_URL
    value: 'turn:turn.lilith.app:3478'
  - name: TURN_SERVER_TLS_URL
    value: 'turns:turn.lilith.app:5349'
```

### TURN Server Configuration

coturn server configuration includes:

- **Authentication**: Time-limited credentials with HMAC
- **TLS**: Encrypted TURN over TLS (port 5349)
- **Relay**: UDP packet relay for NAT traversal
- **Quotas**: Connection and bandwidth limits

## Monitoring

### Metrics

The WebRTC SFU exposes Prometheus metrics at `/metrics`:

- Connection count and quality
- Audio bitrate and packet loss
- Room and participant statistics
- System resource usage

### Grafana Dashboards

Pre-configured dashboards include:

1. **WebRTC SFU Overview** - System health and performance
2. **Connection Quality** - Packet loss, latency, jitter
3. **Resource Usage** - CPU, memory, network
4. **TURN Server** - Relay usage and authentication

### Alerts

Configured alerts cover:

- High CPU/Memory usage
- Connection failure rates
- TURN server unavailability
- Audio quality degradation

## Security

### Network Security

- Network policies restrict traffic to necessary ports
- Pod security contexts enforce non-root execution
- TLS encryption for all communications

### Authentication

- TURN server uses time-limited credentials
- WebRTC service requires JWT authentication
- API endpoints are protected with RBAC

### TLS Configuration

```yaml
# TURN server TLS
tls:
  - name: turn-tls
    port: 5349
    protocol: TCP
    targetPort: 5349

# WebRTC API TLS
ingress:
  tls:
    - secretName: webrtc-tls
      hosts:
        - webrtc.lilith.app
```

## Performance Tuning

### SFU Optimization

1. **CPU Resources**: Allocate sufficient CPU for audio processing
2. **Memory**: Buffer for session state and audio streams
3. **Network**: Low-latency paths and UDP packet handling
4. **Node Selection**: Use dedicated real-time nodes

### TURN Server Optimization

1. **Relay Capacity**: Configure based on expected concurrent users
2. **Bandwidth Limits**: Set appropriate per-user limits
3. **Geographic Distribution**: Deploy close to users

### Kubernetes Configuration

```yaml
# Node selector for real-time workloads
nodeSelector:
  node-type: realtime
  kubernetes.io/os: linux

# Resource allocation
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 1Gi

# Anti-affinity for high availability
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: webrtc-sfu
          topologyKey: kubernetes.io/hostname
```

## Testing

### Health Checks

```bash
# SFU health check
curl https://webrtc.lilith.app/v1/health

# TURN server test
turnutils_uclient -T -u turn-user:password webrtc.lilith.app
```

### Connection Testing

Use the WebRTC test tools to verify:

1. STUN server connectivity
2. TURN server relay functionality
3. Audio quality and latency
4. Connection establishment

## Troubleshooting

### Common Issues

1. **TURN Server Not Accessible**
   - Check LoadBalancer IP assignment
   - Verify firewall rules allow UDP/TCP 3478, TCP 5349
   - Validate TLS certificates

2. **Audio Quality Issues**
   - Check network latency and packet loss
   - Verify adaptive bitrate configuration
   - Monitor resource utilization

3. **Connection Failures**
   - Review TURN server logs
   - Check authentication credentials
   - Verify DNS resolution

### Debug Commands

```bash
# Check pod logs
kubectl logs -n lilith-platform -l app=webrtc-sfu

# Check TURN server logs
kubectl logs -n lilith-platform -l app=coturn-server

# Test connectivity
kubectl exec -it -n lilith-platform deployment/webrtc-sfu -- curl localhost:8080/v1/health

# Monitor metrics
kubectl port-forward -n lilith-platform svc/prometheus 9090:9090
```

## Maintenance

### Updates

1. **Rolling Updates**: Configure proper deployment strategies
2. **Blue-Green**: Use canary deployments for major changes
3. **Backup**: Export configuration and session data

### Scaling

1. **Horizontal Scaling**: Increase replica count based on load
2. **Vertical Scaling**: Adjust resource limits
3. **Global Scaling**: Deploy in multiple regions

## Integration

### Client Integration

WebRTC clients should be configured with:

```javascript
const rtcConfiguration = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },
    {
      urls: 'turn:turn.lilith.app:3478',
      username: turnUsername,
      credential: turnCredential,
    },
  ],
  iceCandidatePoolSize: 10,
};
```

### Backend Integration

1. **Authentication**: Use JWT tokens for API access
2. **Session Management**: Create rooms through API endpoints
3. **Metrics**: Forward application metrics to monitoring

## Support

For issues with WebRTC deployment:

1. Check the logs and monitoring dashboards
2. Review the troubleshooting section
3. Consult the Kubernetes and coturn documentation
4. Contact the platform team for advanced issues

## Appendix

### Ports Reference

| Service         | Port        | Protocol | Purpose       |
| --------------- | ----------- | -------- | ------------- |
| WebRTC SFU API  | 8080        | TCP      | HTTP API      |
| TURN Server     | 3478        | UDP/TCP  | STUN/TURN     |
| TURN Server TLS | 5349        | TCP      | TURN over TLS |
| TURN Relay      | 49152-65535 | UDP      | Relay range   |

### Environment Variables

| Variable                 | Default | Description                   |
| ------------------------ | ------- | ----------------------------- |
| MAX_CONCURRENT_ROOMS     | 100     | Maximum concurrent rooms      |
| DEFAULT_MAX_PARTICIPANTS | 10      | Default participants per room |
| SESSION_TIMEOUT_MINUTES  | 30      | Session timeout in minutes    |
| TARGET_LATENCY_MS        | 150     | Target audio latency          |
| ADAPTIVE_BITRATE_ENABLED | true    | Enable adaptive bitrate       |
| TURN_SERVER_URL          | -       | TURN server URL               |
| TURN_SERVER_TLS_URL      | -       | TURN over TLS URL             |
