Comprehensive chaos engineering tests for the Aphrodite adult live streaming platform. These tests validate system resilience, failure recovery, and graceful degradation under adverse conditions.
Overview#
This test suite implements chaos engineering principles to verify that the Aphrodite platform can:
- Maintain service availability during partial failures
- Recover gracefully from service outages
- Prevent cascade failures across dependent services
- Maintain data consistency during network partitions
- Handle latency injection without data loss
Test Categories#
Service Failure Simulation (service-failure.test.ts)#
Tests system behavior when individual services become unavailable:
| Test | Description |
|---|---|
| Streaming Service Failure | Validates viewer handling during stream outage |
| Payment Service Failure | Tests tip processing during payment downtime |
| Chat Service Failure | Verifies stream continuation without chat |
| Device Service Failure | Tests tip-triggered device queue handling |
| Broadcaster Service Failure | Validates viewer notification and cleanup |
Key validations:
- Graceful degradation behavior
- Dependent service isolation
- Circuit breaker activation
- Queue preservation during outages
- Error message quality
Network Partition Tests (network-partition.test.ts)#
Tests system behavior during network isolation scenarios:
| Test | Description |
|---|---|
| Payment Isolation | Isolates payment service from other services |
| Chat-Streaming Split Brain | Tests message synchronization during partition |
| Full Partition | Simulates complete service isolation |
| Intermittent Connectivity | Tests retry logic and reconnection |
| Latency Injection | Validates timeout handling |
Key validations:
- Split-brain prevention
- Data reconciliation after partition heals
- Timeout handling
- Retry behavior
- Message ordering preservation
Recovery Validation Tests (recovery-validation.test.ts)#
Tests system recovery after failures:
| Test | Description |
|---|---|
| Time-to-Recovery (TTR) | Measures recovery time for each service |
| Data Consistency | Validates transaction logs and event ordering |
| Queue Drainage | Verifies pending messages are processed |
| Metrics Normalization | Confirms return to baseline performance |
| Dependency Recovery | Tests dependent service restoration |
Key validations:
- Recovery within SLA timeouts
- No data loss during recovery
- Queue depth returns to zero
- Response times normalize
- All dependencies healthy post-recovery
Service Configuration#
Services are configured in config.ts with the following properties:
{
name: 'Streaming',
url: 'http://localhost:3101',
healthEndpoint: '/health',
criticalityLevel: 'critical', // critical | high | medium | low
dependencies: ['payment', 'chat', 'devices'],
recoveryTimeoutMs: 30000,
gracefulDegradation: true,
}
Criticality Levels#
| Level | Services | Recovery SLA | Approval Required |
|---|---|---|---|
| Critical | streaming, broadcaster, payment | 30-60s | Yes (payment) |
| High | viewer, chat | 20-30s | No |
| Medium | devices | 15s | No |
Pre-defined Experiments#
The suite includes pre-configured chaos experiments:
Service Failure Experiments#
streaming_service_failure- Complete streaming outagepayment_service_failure- Payment service unavailablechat_service_failure- Chat service downdevice_service_failure- Device service unavailable
Network Partition Experiments#
network_partition_payment- Isolate payment servicenetwork_partition_chat_streaming- Split chat and streaming
Latency Injection Experiments#
latency_payment- 2000ms latency with 500ms jitterlatency_devices- 1000ms latency with 300ms jitter
Advanced Experiments#
cascade_streaming_failure- Gradual degradation cascadedatabase_payment_failure- Payment database unavailabilitycache_failure- Redis cache failure
Running Tests#
Prerequisites#
-
Ensure all Aphrodite services are running:
bash# Start services docker compose -f docker/docker-compose.dev.yml up -d -
Install dependencies:
bashpnpm install
Run All Chaos Tests#
# Run all chaos engineering tests
pnpm nx test testing-chaos-aphrodite
# Or using vitest directly
pnpm vitest run testing/chaos/aphrodite/
Run Specific Test Suites#
# Service failure tests only
pnpm vitest run testing/chaos/aphrodite/service-failure.test.ts
# Network partition tests only
pnpm vitest run testing/chaos/aphrodite/network-partition.test.ts
# Recovery validation tests only
pnpm vitest run testing/chaos/aphrodite/recovery-validation.test.ts
Run with Verbose Output#
pnpm vitest run testing/chaos/aphrodite/ --reporter=verbose
Environment Configuration#
Configure service URLs via environment variables:
# Core services
export APHRODITE_STREAMING_URL=http://localhost:3101
export APHRODITE_BROADCASTER_URL=http://localhost:3102
export APHRODITE_VIEWER_URL=http://localhost:3103
export APHRODITE_CHAT_URL=http://localhost:3104
export APHRODITE_PAYMENT_URL=http://localhost:3105
export APHRODITE_DEVICES_URL=http://localhost:3106
# Chaos API (if using external chaos orchestrator)
export CHAOS_API_URL=http://localhost:3500/v1/chaos
export OBSERVABILITY_URL=http://localhost:3500
Safety Controls#
All experiments include safety controls:
safetyControls: {
automaticRollback: true, // Auto-rollback on critical errors
monitoringRequired: true, // Require observability
approvalRequired: true, // Manual approval (critical services)
rollbackTriggers: [ // Automatic rollback conditions
'error_rate > 50%',
'cascade_detected',
'data_inconsistency',
],
}
Rollback Triggers#
| Trigger | Description |
|---|---|
error_rate > N% |
Error rate exceeds threshold |
cascade_detected |
Cascade failure detected |
data_inconsistency |
Data integrity violation |
timeout > Ns |
Operation timeout exceeded |
queue_depth > N |
Message queue overflow |
revenue_loss |
Payment processing affected |
Success Criteria#
Each experiment defines success criteria:
successCriteria: {
maxErrorRate: 0.1, // Max 10% error rate during recovery
maxResponseTimeMs: 2000, // Max response time
recoveryTimeMs: 30000, // Time to full recovery
dataConsistencyCheck: true, // Require data consistency validation
}
Integration with CI/CD#
GitHub Actions Example#
chaos-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Start services
run: docker compose up -d
- name: Run chaos tests
run: pnpm vitest run testing/chaos/aphrodite/
- name: Upload results
uses: actions/upload-artifact@v4
with:
name: chaos-test-results
path: coverage/chaos/
Scheduled Chaos Testing#
Run chaos tests on a schedule to validate ongoing resilience:
on:
schedule:
- cron: '0 2 * * *' # Daily at 2 AM
Metrics and Observability#
Tests emit metrics for monitoring:
chaos.experiment.started- Experiment start eventchaos.experiment.completed- Experiment completionchaos.recovery.time_ms- Time to recoverychaos.error.rate- Error rate during experimentchaos.cascade.detected- Cascade failure events
Best Practices#
-
Start Small: Begin with single-service failures before testing cascades
-
Monitor During Tests: Always have observability active during chaos tests
-
Document Findings: Record unexpected behaviors and system limits
-
Iterate: Use findings to improve system resilience
-
Production Chaos: Graduate to production chaos testing only after extensive staging validation
Troubleshooting#
Tests Failing to Connect#
Verify services are running:
curl http://localhost:3101/health
curl http://localhost:3105/health
Timeout Errors#
Increase test timeout in vitest config or individual tests:
it('should recover', async () => { ... }, 120000);
Cascade Detection False Positives#
Adjust cascade detection thresholds in config.ts:
rollbackTriggers: ['error_rate > 60%'], // Increase threshold