Eve library

Model Routing Fallback System - Practical Example

Other

Revision 1

On this page

Overview#

This document demonstrates how the model routing fallback system works in practice, including automatic failover, circuit breaking, and distilled model fallback.

Example Scenario: API Completion with Multiple Provider Failures#

Setup#

typescript
// Providers configured with priorities:
// 1. OpenAI (primary)
// 2. Anthropic (secondary)
// 3. Azure OpenAI (tertiary)
// 4. Local distilled models (emergency fallback)

Normal Operation#

http
POST /v1/ai/complete
Content-Type: application/json

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Explain quantum entanglement"}
  ],
  "max_tokens": 500,
  "temperature": 0.7
}

Response (Success):

json
{
  "content": "Quantum entanglement is a phenomenon...",
  "role": "assistant",
  "provider": "openai",
  "fallbackUsed": false,
  "attemptedProviders": ["openai"],
  "totalLatency": 234,
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}

Scenario 1: Primary Provider Failure#

OpenAI experiences an outage. The system automatically fails over to Anthropic:

  1. Attempt: OpenAI → FAIL (503 Service Unavailable)
  2. Fallback: Anthropic → SUCCESS

Response:

json
{
  "content": "Quantum entanglement is a phenomenon...",
  "role": "assistant",
  "provider": "anthropic",
  "fallbackUsed": true,
  "attemptedProviders": ["openai", "anthropic"],
  "totalLatency": 1245,
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}

System Actions:

  • OpenAI failure recorded
  • Circuit breaker failure count incremented
  • Health monitoring alert generated
  • Fallback event logged

Scenario 2: Circuit Breaker Opens#

After 5 consecutive failures, the circuit breaker opens for OpenAI:

http
GET /v1/ai/providers/status

Response:

json
{
  "timestamp": "2025-10-17T10:30:00Z",
  "providers": [
    {
      "name": "openai",
      "status": "unhealthy",
      "available": false,
      "latency_ms": 0,
      "error_rate": 1.0,
      "success_rate": 0.0,
      "circuit_state": "open",
      "uptime_percent": 0,
      "last_check": "2025-10-17T10:29:55Z",
      "consecutive_failures": 5
    },
    {
      "name": "anthropic",
      "status": "healthy",
      "available": true,
      "latency_ms": 234,
      "error_rate": 0.02,
      "success_rate": 0.98,
      "circuit_state": "closed",
      "uptime_percent": 98.5,
      "last_check": "2025-10-17T10:29:58Z",
      "consecutive_failures": 0
    }
  ]
}

Subsequent Requests:

  • OpenAI is skipped automatically
  • Anthropic becomes the primary provider
  • OpenAI circuit breaker will attempt reset after 60 seconds

Scenario 3: Multiple Provider Failures#

Both OpenAI and Anthropic are down. System falls back to Azure, then to local distilled models:

  1. Attempt: OpenAI → SKIP (Circuit open)
  2. Attempt: Anthropic → FAIL (503 Service Unavailable)
  3. Fallback: Azure OpenAI → FAIL (429 Rate Limited)
  4. Emergency Fallback: Distilled Model (Llama 3.1 8B) → SUCCESS

Response:

json
{
  "content": "Quantum entanglement is a phenomenon where particles...",
  "role": "assistant",
  "provider": "distilled-local",
  "fallbackUsed": true,
  "attemptedProviders": ["anthropic", "azure", "distilled-local"],
  "totalLatency": 3456,
  "distilledModel": "llama-3.1-8b-instruct",
  "modelFamily": "llama",
  "modelSize": "8b",
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 120,
    "total_tokens": 145
  }
}

System Actions:

  • All cloud providers marked as unavailable
  • Health monitoring alerts escalated to CRITICAL
  • Local distilled model automatically selected
  • Model downloaded and cached if not already available
  • Lower quality response but system remains operational

Scenario 4: Monitoring and Recovery#

Check Fallback History#

http
GET /v1/ai/fallback/history?limit=10

Response:

json
{
  "history": [
    {
      "timestamp": "2025-10-17T10:30:15Z",
      "attemptedProvider": "openai",
      "fallbackProvider": "anthropic",
      "reason": "Provider openai is unavailable (circuit breaker open)",
      "latency": 1245,
      "success": true
    },
    {
      "timestamp": "2025-10-17T10:31:22Z",
      "attemptedProvider": "anthropic",
      "fallbackProvider": "azure",
      "reason": "503 Service Unavailable",
      "latency": 2134,
      "success": false
    },
    {
      "timestamp": "2025-10-17T10:31:25Z",
      "attemptedProvider": "anthropic",
      "fallbackProvider": "distilled-local",
      "reason": "All cloud providers failed",
      "latency": 3456,
      "success": true
    }
  ],
  "count": 3,
  "summary": {
    "total_events": 3,
    "successful_fallbacks": 2,
    "failed_fallbacks": 1,
    "success_rate": 0.667
  }
}

Manual Circuit Breaker Reset#

Once OpenAI is back online, manually reset the circuit breaker:

http
POST /v1/ai/providers/openai/reset-circuit

Response:

json
{
  "success": true,
  "provider": "openai",
  "message": "Circuit breaker reset successfully"
}

Scenario 5: Distilled Model Management#

List Available Open-Weight Models#

http
GET /v1/ai/models/open-weight?max_memory_gb=10

Response:

json
{
  "models": [
    {
      "id": "llama-3.1-8b-instruct",
      "name": "Llama 3.1 8B Instruct",
      "family": "llama",
      "size": "8b",
      "context_length": 8192,
      "tasks": ["chat", "completion", "instruction"],
      "quality_score": 0.85,
      "speed_score": 0.8,
      "memory_requirement_gb": 6,
      "cached": true
    },
    {
      "id": "mistral-7b-instruct",
      "name": "Mistral 7B Instruct",
      "family": "mistral",
      "size": "7b",
      "context_length": 8192,
      "tasks": ["chat", "completion", "instruction"],
      "quality_score": 0.82,
      "speed_score": 0.85,
      "memory_requirement_gb": 5,
      "cached": false
    }
  ],
  "count": 2
}

Download a Distilled Model#

http
POST /v1/ai/models/open-weight/mistral-7b-instruct/download

Response:

json
{
  "success": true,
  "model_id": "mistral-7b-instruct",
  "status": "downloading",
  "message": "Model download initiated"
}

Check Download Progress#

http
GET /v1/ai/models/open-weight/mistral-7b-instruct/download-status

Response:

json
{
  "modelId": "mistral-7b-instruct",
  "downloaded": 2500000000,
  "total": 5000000000,
  "percentage": 50,
  "status": "downloading"
}

Scenario 6: Health Monitoring Alerts#

http
GET /v1/ai/health-monitoring/alerts?severity=critical

Response:

json
{
  "alerts": [
    {
      "id": "alert-1729157400-xyz",
      "timestamp": "2025-10-17T10:30:00Z",
      "providerName": "openai",
      "severity": "critical",
      "type": "circuit-open",
      "message": "Circuit breaker opened for provider openai",
      "metrics": {
        "status": "unhealthy",
        "consecutiveFailures": 5,
        "error_rate": 1.0
      },
      "acknowledged": false
    }
  ],
  "count": 1,
  "unacknowledged_count": 1
}

Benefits Demonstrated#

  1. Zero-Downtime: System continues operating even with multiple provider failures
  2. Automatic Recovery: Circuit breakers automatically reset after timeout
  3. Cost Optimization: Falls back to cheaper providers when primary is unavailable
  4. Quality Graceful Degradation: Uses distilled models as last resort
  5. Observability: Complete visibility into fallback events and provider health
  6. Manual Control: Operators can manually reset circuits and configure priorities

Performance Impact#

  • Normal Operation: +5-10ms overhead for health tracking
  • Single Failover: +1-2 seconds for retry logic
  • Multiple Failover: +3-5 seconds for cascading attempts
  • Distilled Fallback: +5-10 seconds (first time, includes download/cache)
  • Cached Distilled: +2-3 seconds (subsequent uses)

Configuration Best Practices#

  1. Circuit Breaker Threshold: 3-5 failures (balance between sensitivity and stability)
  2. Health Check Interval: 30-60 seconds (frequent enough to detect issues quickly)
  3. Alert Cooldown: 5-15 minutes (avoid alert fatigue)
  4. Distilled Models: Pre-cache frequently used models
  5. Provider Priorities: Order by cost/performance tradeoff