# Model Routing Fallback System - Practical Example

## Overview

This document demonstrates how the model routing fallback system works in practice, including automatic failover, circuit breaking, and distilled model fallback.

## Example Scenario: API Completion with Multiple Provider Failures

### Setup

```typescript
// Providers configured with priorities:
// 1. OpenAI (primary)
// 2. Anthropic (secondary)
// 3. Azure OpenAI (tertiary)
// 4. Local distilled models (emergency fallback)
```

### Normal Operation

```http
POST /v1/ai/complete
Content-Type: application/json

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Explain quantum entanglement"}
  ],
  "max_tokens": 500,
  "temperature": 0.7
}
```

**Response (Success):**
```json
{
  "content": "Quantum entanglement is a phenomenon...",
  "role": "assistant",
  "provider": "openai",
  "fallbackUsed": false,
  "attemptedProviders": ["openai"],
  "totalLatency": 234,
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}
```

### Scenario 1: Primary Provider Failure

OpenAI experiences an outage. The system automatically fails over to Anthropic:

1. **Attempt**: OpenAI → **FAIL** (503 Service Unavailable)
2. **Fallback**: Anthropic → **SUCCESS**

**Response:**
```json
{
  "content": "Quantum entanglement is a phenomenon...",
  "role": "assistant",
  "provider": "anthropic",
  "fallbackUsed": true,
  "attemptedProviders": ["openai", "anthropic"],
  "totalLatency": 1245,
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}
```

**System Actions:**
- OpenAI failure recorded
- Circuit breaker failure count incremented
- Health monitoring alert generated
- Fallback event logged

### Scenario 2: Circuit Breaker Opens

After 5 consecutive failures, the circuit breaker opens for OpenAI:

```http
GET /v1/ai/providers/status
```

**Response:**
```json
{
  "timestamp": "2025-10-17T10:30:00Z",
  "providers": [
    {
      "name": "openai",
      "status": "unhealthy",
      "available": false,
      "latency_ms": 0,
      "error_rate": 1.0,
      "success_rate": 0.0,
      "circuit_state": "open",
      "uptime_percent": 0,
      "last_check": "2025-10-17T10:29:55Z",
      "consecutive_failures": 5
    },
    {
      "name": "anthropic",
      "status": "healthy",
      "available": true,
      "latency_ms": 234,
      "error_rate": 0.02,
      "success_rate": 0.98,
      "circuit_state": "closed",
      "uptime_percent": 98.5,
      "last_check": "2025-10-17T10:29:58Z",
      "consecutive_failures": 0
    }
  ]
}
```

**Subsequent Requests:**
- OpenAI is skipped automatically
- Anthropic becomes the primary provider
- OpenAI circuit breaker will attempt reset after 60 seconds

### Scenario 3: Multiple Provider Failures

Both OpenAI and Anthropic are down. System falls back to Azure, then to local distilled models:

1. **Attempt**: OpenAI → **SKIP** (Circuit open)
2. **Attempt**: Anthropic → **FAIL** (503 Service Unavailable)
3. **Fallback**: Azure OpenAI → **FAIL** (429 Rate Limited)
4. **Emergency Fallback**: Distilled Model (Llama 3.1 8B) → **SUCCESS**

**Response:**
```json
{
  "content": "Quantum entanglement is a phenomenon where particles...",
  "role": "assistant",
  "provider": "distilled-local",
  "fallbackUsed": true,
  "attemptedProviders": ["anthropic", "azure", "distilled-local"],
  "totalLatency": 3456,
  "distilledModel": "llama-3.1-8b-instruct",
  "modelFamily": "llama",
  "modelSize": "8b",
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 120,
    "total_tokens": 145
  }
}
```

**System Actions:**
- All cloud providers marked as unavailable
- Health monitoring alerts escalated to CRITICAL
- Local distilled model automatically selected
- Model downloaded and cached if not already available
- Lower quality response but system remains operational

### Scenario 4: Monitoring and Recovery

#### Check Fallback History

```http
GET /v1/ai/fallback/history?limit=10
```

**Response:**
```json
{
  "history": [
    {
      "timestamp": "2025-10-17T10:30:15Z",
      "attemptedProvider": "openai",
      "fallbackProvider": "anthropic",
      "reason": "Provider openai is unavailable (circuit breaker open)",
      "latency": 1245,
      "success": true
    },
    {
      "timestamp": "2025-10-17T10:31:22Z",
      "attemptedProvider": "anthropic",
      "fallbackProvider": "azure",
      "reason": "503 Service Unavailable",
      "latency": 2134,
      "success": false
    },
    {
      "timestamp": "2025-10-17T10:31:25Z",
      "attemptedProvider": "anthropic",
      "fallbackProvider": "distilled-local",
      "reason": "All cloud providers failed",
      "latency": 3456,
      "success": true
    }
  ],
  "count": 3,
  "summary": {
    "total_events": 3,
    "successful_fallbacks": 2,
    "failed_fallbacks": 1,
    "success_rate": 0.667
  }
}
```

#### Manual Circuit Breaker Reset

Once OpenAI is back online, manually reset the circuit breaker:

```http
POST /v1/ai/providers/openai/reset-circuit
```

**Response:**
```json
{
  "success": true,
  "provider": "openai",
  "message": "Circuit breaker reset successfully"
}
```

### Scenario 5: Distilled Model Management

#### List Available Open-Weight Models

```http
GET /v1/ai/models/open-weight?max_memory_gb=10
```

**Response:**
```json
{
  "models": [
    {
      "id": "llama-3.1-8b-instruct",
      "name": "Llama 3.1 8B Instruct",
      "family": "llama",
      "size": "8b",
      "context_length": 8192,
      "tasks": ["chat", "completion", "instruction"],
      "quality_score": 0.85,
      "speed_score": 0.8,
      "memory_requirement_gb": 6,
      "cached": true
    },
    {
      "id": "mistral-7b-instruct",
      "name": "Mistral 7B Instruct",
      "family": "mistral",
      "size": "7b",
      "context_length": 8192,
      "tasks": ["chat", "completion", "instruction"],
      "quality_score": 0.82,
      "speed_score": 0.85,
      "memory_requirement_gb": 5,
      "cached": false
    }
  ],
  "count": 2
}
```

#### Download a Distilled Model

```http
POST /v1/ai/models/open-weight/mistral-7b-instruct/download
```

**Response:**
```json
{
  "success": true,
  "model_id": "mistral-7b-instruct",
  "status": "downloading",
  "message": "Model download initiated"
}
```

#### Check Download Progress

```http
GET /v1/ai/models/open-weight/mistral-7b-instruct/download-status
```

**Response:**
```json
{
  "modelId": "mistral-7b-instruct",
  "downloaded": 2500000000,
  "total": 5000000000,
  "percentage": 50,
  "status": "downloading"
}
```

### Scenario 6: Health Monitoring Alerts

```http
GET /v1/ai/health-monitoring/alerts?severity=critical
```

**Response:**
```json
{
  "alerts": [
    {
      "id": "alert-1729157400-xyz",
      "timestamp": "2025-10-17T10:30:00Z",
      "providerName": "openai",
      "severity": "critical",
      "type": "circuit-open",
      "message": "Circuit breaker opened for provider openai",
      "metrics": {
        "status": "unhealthy",
        "consecutiveFailures": 5,
        "error_rate": 1.0
      },
      "acknowledged": false
    }
  ],
  "count": 1,
  "unacknowledged_count": 1
}
```

## Benefits Demonstrated

1. **Zero-Downtime**: System continues operating even with multiple provider failures
2. **Automatic Recovery**: Circuit breakers automatically reset after timeout
3. **Cost Optimization**: Falls back to cheaper providers when primary is unavailable
4. **Quality Graceful Degradation**: Uses distilled models as last resort
5. **Observability**: Complete visibility into fallback events and provider health
6. **Manual Control**: Operators can manually reset circuits and configure priorities

## Performance Impact

- **Normal Operation**: +5-10ms overhead for health tracking
- **Single Failover**: +1-2 seconds for retry logic
- **Multiple Failover**: +3-5 seconds for cascading attempts
- **Distilled Fallback**: +5-10 seconds (first time, includes download/cache)
- **Cached Distilled**: +2-3 seconds (subsequent uses)

## Configuration Best Practices

1. **Circuit Breaker Threshold**: 3-5 failures (balance between sensitivity and stability)
2. **Health Check Interval**: 30-60 seconds (frequent enough to detect issues quickly)
3. **Alert Cooldown**: 5-15 minutes (avoid alert fatigue)
4. **Distilled Models**: Pre-cache frequently used models
5. **Provider Priorities**: Order by cost/performance tradeoff
