Overview#
This document demonstrates how the model routing fallback system works in practice, including automatic failover, circuit breaking, and distilled model fallback.
Example Scenario: API Completion with Multiple Provider Failures#
Setup#
typescript
// Providers configured with priorities:
// 1. OpenAI (primary)
// 2. Anthropic (secondary)
// 3. Azure OpenAI (tertiary)
// 4. Local distilled models (emergency fallback)
Normal Operation#
http
POST /v1/ai/complete
Content-Type: application/json
{
"messages": [
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Explain quantum entanglement"}
],
"max_tokens": 500,
"temperature": 0.7
}
Response (Success):
json
{
"content": "Quantum entanglement is a phenomenon...",
"role": "assistant",
"provider": "openai",
"fallbackUsed": false,
"attemptedProviders": ["openai"],
"totalLatency": 234,
"usage": {
"prompt_tokens": 25,
"completion_tokens": 150,
"total_tokens": 175
}
}
Scenario 1: Primary Provider Failure#
OpenAI experiences an outage. The system automatically fails over to Anthropic:
- Attempt: OpenAI → FAIL (503 Service Unavailable)
- Fallback: Anthropic → SUCCESS
Response:
json
{
"content": "Quantum entanglement is a phenomenon...",
"role": "assistant",
"provider": "anthropic",
"fallbackUsed": true,
"attemptedProviders": ["openai", "anthropic"],
"totalLatency": 1245,
"usage": {
"prompt_tokens": 25,
"completion_tokens": 150,
"total_tokens": 175
}
}
System Actions:
- OpenAI failure recorded
- Circuit breaker failure count incremented
- Health monitoring alert generated
- Fallback event logged
Scenario 2: Circuit Breaker Opens#
After 5 consecutive failures, the circuit breaker opens for OpenAI:
http
GET /v1/ai/providers/status
Response:
json
{
"timestamp": "2025-10-17T10:30:00Z",
"providers": [
{
"name": "openai",
"status": "unhealthy",
"available": false,
"latency_ms": 0,
"error_rate": 1.0,
"success_rate": 0.0,
"circuit_state": "open",
"uptime_percent": 0,
"last_check": "2025-10-17T10:29:55Z",
"consecutive_failures": 5
},
{
"name": "anthropic",
"status": "healthy",
"available": true,
"latency_ms": 234,
"error_rate": 0.02,
"success_rate": 0.98,
"circuit_state": "closed",
"uptime_percent": 98.5,
"last_check": "2025-10-17T10:29:58Z",
"consecutive_failures": 0
}
]
}
Subsequent Requests:
- OpenAI is skipped automatically
- Anthropic becomes the primary provider
- OpenAI circuit breaker will attempt reset after 60 seconds
Scenario 3: Multiple Provider Failures#
Both OpenAI and Anthropic are down. System falls back to Azure, then to local distilled models:
- Attempt: OpenAI → SKIP (Circuit open)
- Attempt: Anthropic → FAIL (503 Service Unavailable)
- Fallback: Azure OpenAI → FAIL (429 Rate Limited)
- Emergency Fallback: Distilled Model (Llama 3.1 8B) → SUCCESS
Response:
json
{
"content": "Quantum entanglement is a phenomenon where particles...",
"role": "assistant",
"provider": "distilled-local",
"fallbackUsed": true,
"attemptedProviders": ["anthropic", "azure", "distilled-local"],
"totalLatency": 3456,
"distilledModel": "llama-3.1-8b-instruct",
"modelFamily": "llama",
"modelSize": "8b",
"usage": {
"prompt_tokens": 25,
"completion_tokens": 120,
"total_tokens": 145
}
}
System Actions:
- All cloud providers marked as unavailable
- Health monitoring alerts escalated to CRITICAL
- Local distilled model automatically selected
- Model downloaded and cached if not already available
- Lower quality response but system remains operational
Scenario 4: Monitoring and Recovery#
Check Fallback History#
http
GET /v1/ai/fallback/history?limit=10
Response:
json
{
"history": [
{
"timestamp": "2025-10-17T10:30:15Z",
"attemptedProvider": "openai",
"fallbackProvider": "anthropic",
"reason": "Provider openai is unavailable (circuit breaker open)",
"latency": 1245,
"success": true
},
{
"timestamp": "2025-10-17T10:31:22Z",
"attemptedProvider": "anthropic",
"fallbackProvider": "azure",
"reason": "503 Service Unavailable",
"latency": 2134,
"success": false
},
{
"timestamp": "2025-10-17T10:31:25Z",
"attemptedProvider": "anthropic",
"fallbackProvider": "distilled-local",
"reason": "All cloud providers failed",
"latency": 3456,
"success": true
}
],
"count": 3,
"summary": {
"total_events": 3,
"successful_fallbacks": 2,
"failed_fallbacks": 1,
"success_rate": 0.667
}
}
Manual Circuit Breaker Reset#
Once OpenAI is back online, manually reset the circuit breaker:
http
POST /v1/ai/providers/openai/reset-circuit
Response:
json
{
"success": true,
"provider": "openai",
"message": "Circuit breaker reset successfully"
}
Scenario 5: Distilled Model Management#
List Available Open-Weight Models#
http
GET /v1/ai/models/open-weight?max_memory_gb=10
Response:
json
{
"models": [
{
"id": "llama-3.1-8b-instruct",
"name": "Llama 3.1 8B Instruct",
"family": "llama",
"size": "8b",
"context_length": 8192,
"tasks": ["chat", "completion", "instruction"],
"quality_score": 0.85,
"speed_score": 0.8,
"memory_requirement_gb": 6,
"cached": true
},
{
"id": "mistral-7b-instruct",
"name": "Mistral 7B Instruct",
"family": "mistral",
"size": "7b",
"context_length": 8192,
"tasks": ["chat", "completion", "instruction"],
"quality_score": 0.82,
"speed_score": 0.85,
"memory_requirement_gb": 5,
"cached": false
}
],
"count": 2
}
Download a Distilled Model#
http
POST /v1/ai/models/open-weight/mistral-7b-instruct/download
Response:
json
{
"success": true,
"model_id": "mistral-7b-instruct",
"status": "downloading",
"message": "Model download initiated"
}
Check Download Progress#
http
GET /v1/ai/models/open-weight/mistral-7b-instruct/download-status
Response:
json
{
"modelId": "mistral-7b-instruct",
"downloaded": 2500000000,
"total": 5000000000,
"percentage": 50,
"status": "downloading"
}
Scenario 6: Health Monitoring Alerts#
http
GET /v1/ai/health-monitoring/alerts?severity=critical
Response:
json
{
"alerts": [
{
"id": "alert-1729157400-xyz",
"timestamp": "2025-10-17T10:30:00Z",
"providerName": "openai",
"severity": "critical",
"type": "circuit-open",
"message": "Circuit breaker opened for provider openai",
"metrics": {
"status": "unhealthy",
"consecutiveFailures": 5,
"error_rate": 1.0
},
"acknowledged": false
}
],
"count": 1,
"unacknowledged_count": 1
}
Benefits Demonstrated#
- Zero-Downtime: System continues operating even with multiple provider failures
- Automatic Recovery: Circuit breakers automatically reset after timeout
- Cost Optimization: Falls back to cheaper providers when primary is unavailable
- Quality Graceful Degradation: Uses distilled models as last resort
- Observability: Complete visibility into fallback events and provider health
- Manual Control: Operators can manually reset circuits and configure priorities
Performance Impact#
- Normal Operation: +5-10ms overhead for health tracking
- Single Failover: +1-2 seconds for retry logic
- Multiple Failover: +3-5 seconds for cascading attempts
- Distilled Fallback: +5-10 seconds (first time, includes download/cache)
- Cached Distilled: +2-3 seconds (subsequent uses)
Configuration Best Practices#
- Circuit Breaker Threshold: 3-5 failures (balance between sensitivity and stability)
- Health Check Interval: 30-60 seconds (frequent enough to detect issues quickly)
- Alert Cooldown: 5-15 minutes (avoid alert fatigue)
- Distilled Models: Pre-cache frequently used models
- Provider Priorities: Order by cost/performance tradeoff