# Aphrodite Chaos Engineering Test Suite

Comprehensive chaos engineering tests for the Aphrodite adult live streaming
platform. These tests validate system resilience, failure recovery, and graceful
degradation under adverse conditions.

## Overview

This test suite implements chaos engineering principles to verify that the
Aphrodite platform can:

- Maintain service availability during partial failures
- Recover gracefully from service outages
- Prevent cascade failures across dependent services
- Maintain data consistency during network partitions
- Handle latency injection without data loss

## Test Categories

### Service Failure Simulation (`service-failure.test.ts`)

Tests system behavior when individual services become unavailable:

| Test                        | Description                                    |
| --------------------------- | ---------------------------------------------- |
| Streaming Service Failure   | Validates viewer handling during stream outage |
| Payment Service Failure     | Tests tip processing during payment downtime   |
| Chat Service Failure        | Verifies stream continuation without chat      |
| Device Service Failure      | Tests tip-triggered device queue handling      |
| Broadcaster Service Failure | Validates viewer notification and cleanup      |

**Key validations:**

- Graceful degradation behavior
- Dependent service isolation
- Circuit breaker activation
- Queue preservation during outages
- Error message quality

### Network Partition Tests (`network-partition.test.ts`)

Tests system behavior during network isolation scenarios:

| Test                       | Description                                    |
| -------------------------- | ---------------------------------------------- |
| Payment Isolation          | Isolates payment service from other services   |
| Chat-Streaming Split Brain | Tests message synchronization during partition |
| Full Partition             | Simulates complete service isolation           |
| Intermittent Connectivity  | Tests retry logic and reconnection             |
| Latency Injection          | Validates timeout handling                     |

**Key validations:**

- Split-brain prevention
- Data reconciliation after partition heals
- Timeout handling
- Retry behavior
- Message ordering preservation

### Recovery Validation Tests (`recovery-validation.test.ts`)

Tests system recovery after failures:

| Test                   | Description                                   |
| ---------------------- | --------------------------------------------- |
| Time-to-Recovery (TTR) | Measures recovery time for each service       |
| Data Consistency       | Validates transaction logs and event ordering |
| Queue Drainage         | Verifies pending messages are processed       |
| Metrics Normalization  | Confirms return to baseline performance       |
| Dependency Recovery    | Tests dependent service restoration           |

**Key validations:**

- Recovery within SLA timeouts
- No data loss during recovery
- Queue depth returns to zero
- Response times normalize
- All dependencies healthy post-recovery

## Service Configuration

Services are configured in `config.ts` with the following properties:

```typescript
{
  name: 'Streaming',
  url: 'http://localhost:3101',
  healthEndpoint: '/health',
  criticalityLevel: 'critical',  // critical | high | medium | low
  dependencies: ['payment', 'chat', 'devices'],
  recoveryTimeoutMs: 30000,
  gracefulDegradation: true,
}
```

### Criticality Levels

| Level    | Services                        | Recovery SLA | Approval Required |
| -------- | ------------------------------- | ------------ | ----------------- |
| Critical | streaming, broadcaster, payment | 30-60s       | Yes (payment)     |
| High     | viewer, chat                    | 20-30s       | No                |
| Medium   | devices                         | 15s          | No                |

## Pre-defined Experiments

The suite includes pre-configured chaos experiments:

### Service Failure Experiments

- `streaming_service_failure` - Complete streaming outage
- `payment_service_failure` - Payment service unavailable
- `chat_service_failure` - Chat service down
- `device_service_failure` - Device service unavailable

### Network Partition Experiments

- `network_partition_payment` - Isolate payment service
- `network_partition_chat_streaming` - Split chat and streaming

### Latency Injection Experiments

- `latency_payment` - 2000ms latency with 500ms jitter
- `latency_devices` - 1000ms latency with 300ms jitter

### Advanced Experiments

- `cascade_streaming_failure` - Gradual degradation cascade
- `database_payment_failure` - Payment database unavailability
- `cache_failure` - Redis cache failure

## Running Tests

### Prerequisites

1. Ensure all Aphrodite services are running:

   ```bash
   # Start services
   docker compose -f docker/docker-compose.dev.yml up -d
   ```

2. Install dependencies:
   ```bash
   pnpm install
   ```

### Run All Chaos Tests

```bash
# Run all chaos engineering tests
pnpm nx test testing-chaos-aphrodite

# Or using vitest directly
pnpm vitest run testing/chaos/aphrodite/
```

### Run Specific Test Suites

```bash
# Service failure tests only
pnpm vitest run testing/chaos/aphrodite/service-failure.test.ts

# Network partition tests only
pnpm vitest run testing/chaos/aphrodite/network-partition.test.ts

# Recovery validation tests only
pnpm vitest run testing/chaos/aphrodite/recovery-validation.test.ts
```

### Run with Verbose Output

```bash
pnpm vitest run testing/chaos/aphrodite/ --reporter=verbose
```

## Environment Configuration

Configure service URLs via environment variables:

```bash
# Core services
export APHRODITE_STREAMING_URL=http://localhost:3101
export APHRODITE_BROADCASTER_URL=http://localhost:3102
export APHRODITE_VIEWER_URL=http://localhost:3103
export APHRODITE_CHAT_URL=http://localhost:3104
export APHRODITE_PAYMENT_URL=http://localhost:3105
export APHRODITE_DEVICES_URL=http://localhost:3106

# Chaos API (if using external chaos orchestrator)
export CHAOS_API_URL=http://localhost:3500/v1/chaos
export OBSERVABILITY_URL=http://localhost:3500
```

## Safety Controls

All experiments include safety controls:

```typescript
safetyControls: {
  automaticRollback: true,      // Auto-rollback on critical errors
  monitoringRequired: true,      // Require observability
  approvalRequired: true,        // Manual approval (critical services)
  rollbackTriggers: [            // Automatic rollback conditions
    'error_rate > 50%',
    'cascade_detected',
    'data_inconsistency',
  ],
}
```

### Rollback Triggers

| Trigger              | Description                  |
| -------------------- | ---------------------------- |
| `error_rate > N%`    | Error rate exceeds threshold |
| `cascade_detected`   | Cascade failure detected     |
| `data_inconsistency` | Data integrity violation     |
| `timeout > Ns`       | Operation timeout exceeded   |
| `queue_depth > N`    | Message queue overflow       |
| `revenue_loss`       | Payment processing affected  |

## Success Criteria

Each experiment defines success criteria:

```typescript
successCriteria: {
  maxErrorRate: 0.1,           // Max 10% error rate during recovery
  maxResponseTimeMs: 2000,     // Max response time
  recoveryTimeMs: 30000,       // Time to full recovery
  dataConsistencyCheck: true,  // Require data consistency validation
}
```

## Integration with CI/CD

### GitHub Actions Example

```yaml
chaos-tests:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Start services
      run: docker compose up -d

    - name: Run chaos tests
      run: pnpm vitest run testing/chaos/aphrodite/

    - name: Upload results
      uses: actions/upload-artifact@v4
      with:
        name: chaos-test-results
        path: coverage/chaos/
```

### Scheduled Chaos Testing

Run chaos tests on a schedule to validate ongoing resilience:

```yaml
on:
  schedule:
    - cron: '0 2 * * *' # Daily at 2 AM
```

## Metrics and Observability

Tests emit metrics for monitoring:

- `chaos.experiment.started` - Experiment start event
- `chaos.experiment.completed` - Experiment completion
- `chaos.recovery.time_ms` - Time to recovery
- `chaos.error.rate` - Error rate during experiment
- `chaos.cascade.detected` - Cascade failure events

## Best Practices

1. **Start Small**: Begin with single-service failures before testing cascades

2. **Monitor During Tests**: Always have observability active during chaos tests

3. **Document Findings**: Record unexpected behaviors and system limits

4. **Iterate**: Use findings to improve system resilience

5. **Production Chaos**: Graduate to production chaos testing only after
   extensive staging validation

## Troubleshooting

### Tests Failing to Connect

Verify services are running:

```bash
curl http://localhost:3101/health
curl http://localhost:3105/health
```

### Timeout Errors

Increase test timeout in vitest config or individual tests:

```typescript
it('should recover', async () => { ... }, 120000);
```

### Cascade Detection False Positives

Adjust cascade detection thresholds in `config.ts`:

```typescript
rollbackTriggers: ['error_rate > 60%'],  // Increase threshold
```

## Related Documentation

- [Load Testing Suite](../../performance/k6/aphrodite/README.md)
- [E2E Tests](../../e2e/tests/api/aphrodite/)
- [Observability Setup](../../../docs/observability.md)
