# Milvus Vector Database Operations Runbook

**Purpose**: Operational procedures for Milvus vector database including
configuration, monitoring, collection management, and disaster recovery.

## Milvus Architecture Overview

### Development Environment

```
Milvus Configuration:
├── Image: milvusdb/milvus:v2.3.3
├── Ports:
│   ├── 19530 (gRPC API)
│   └── 9091 (Metrics/Health)
├── Memory: 4GB limit, 2GB reservation
├── Mode: Standalone
└── Dependencies:
    ├── etcd (metadata storage)
    └── MinIO (object storage)

Storage Architecture:
├── etcd: Metadata, schema, collection info
├── MinIO: Vector data segments, indexes
└── Local: WAL, temporary files
```

### Services Using Milvus

| Service              | Purpose                 | Embedding Dimensions |
| -------------------- | ----------------------- | -------------------- |
| ai                   | Inference embeddings    | 768-1536             |
| rag                  | Document embeddings     | 768-1536             |
| spiritual-embeddings | Sacred text embeddings  | 768                  |
| vectordb             | Vector operations proxy | Various              |

## Configuration Details

### Docker Compose Configuration

```yaml
milvus:
  image: milvusdb/milvus:v2.3.3
  ports:
    - '19530:19530' # gRPC API
    - '9091:9091' # Metrics/Health
  environment:
    - ETCD_ENDPOINTS=etcd:2379
    - MINIO_ADDRESS=minio:9000
    - MINIO_ACCESS_KEY_ID=minioadmin
    - MINIO_SECRET_ACCESS_KEY=minioadmin
  deploy:
    resources:
      limits:
        memory: 4G
      reservations:
        memory: 2G
  command: ['/tini', '--', 'milvus', 'run', 'standalone']
```

### Dependencies

**etcd (Metadata Store)**

```yaml
etcd:
  image: quay.io/coreos/etcd:v3.5.0
  environment:
    - ETCD_AUTO_COMPACTION_MODE=revision
    - ETCD_AUTO_COMPACTION_RETENTION=1000
    - ETCD_QUOTA_BACKEND_BYTES=4294967296
    - ETCD_SNAPSHOT_COUNT=50000
```

**MinIO (Object Storage)**

```yaml
minio:
  image: minio/minio:RELEASE.2023-11-15T20-43-25Z
  environment:
    - MINIO_ACCESS_KEY=minioadmin
    - MINIO_SECRET_KEY=minioadmin
  command: minio server /data --console-address ":9001"
```

## Collection Configuration

### Recommended Collection Settings

```python
from pymilvus import Collection, FieldSchema, CollectionSchema, DataType, utility

# Define fields for knowledge embeddings
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="source_id", dtype=DataType.VARCHAR, max_length=100),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=768),
    FieldSchema(name="metadata", dtype=DataType.JSON)
]

schema = CollectionSchema(fields, description="Knowledge embeddings")
collection = Collection("knowledge_embeddings", schema)

# Create HNSW index (recommended for high-quality search)
index_params = {
    "metric_type": "COSINE",
    "index_type": "HNSW",
    "params": {
        "M": 16,                # Number of edges per node
        "efConstruction": 256   # Build-time precision
    }
}
collection.create_index("embedding", index_params)

# Load collection for search
collection.load()
```

### Index Type Recommendations

| Index Type | Use Case                              | Memory   | Search Speed | Accuracy |
| ---------- | ------------------------------------- | -------- | ------------ | -------- |
| HNSW       | General purpose                       | High     | Fast         | High     |
| IVF_FLAT   | Small collections                     | Medium   | Medium       | High     |
| IVF_SQ8    | Large collections, constrained memory | Low      | Fast         | Medium   |
| AUTOINDEX  | Let Milvus decide                     | Variable | Variable     | High     |

## Operational Procedures

### Starting Milvus

```bash
# Development environment
cd /home/eaa/lilith

# Start dependencies first
docker compose -f docker-compose.dev.yml up -d etcd minio

# Wait for dependencies
sleep 10

# Start Milvus
docker compose -f docker-compose.dev.yml up -d milvus

# Verify Milvus is running
docker compose -f docker-compose.dev.yml ps milvus
curl -s http://localhost:9091/healthz
# Expected: {"status":"ok"}
```

### Checking Milvus Health

```bash
# Health check
curl -s http://localhost:9091/healthz | jq

# Metrics
curl -s http://localhost:9091/metrics | grep -E "milvus_(proxy|querynode|datanode)"

# Check collections via Python
python3 << 'EOF'
from pymilvus import connections, utility

connections.connect("default", host="localhost", port="19530")
print("Collections:", utility.list_collections())
for name in utility.list_collections():
    collection = Collection(name)
    print(f"  {name}: {collection.num_entities} entities")
connections.disconnect("default")
EOF
```

### Collection Management

```python
from pymilvus import connections, Collection, utility

# Connect
connections.connect("default", host="localhost", port="19530")

# List collections
print(utility.list_collections())

# Get collection info
collection = Collection("knowledge_embeddings")
print(f"Schema: {collection.schema}")
print(f"Entities: {collection.num_entities}")
print(f"Indexes: {collection.indexes}")

# Check if collection is loaded
print(f"Loaded: {utility.load_state('knowledge_embeddings')}")

# Compact collection (merge small segments)
collection.compact()

# Flush to persist data
collection.flush()

# Release from memory
collection.release()

# Load back
collection.load()

# Drop collection (CAUTION: data loss)
# utility.drop_collection("collection_name")
```

### Search Operations

```python
from pymilvus import connections, Collection

connections.connect("default", host="localhost", port="19530")
collection = Collection("knowledge_embeddings")
collection.load()

# Search parameters for HNSW
search_params = {
    "metric_type": "COSINE",
    "params": {"ef": 64}  # Higher = more accurate, slower
}

# Perform search
results = collection.search(
    data=[query_embedding],  # Your query vector
    anns_field="embedding",
    param=search_params,
    limit=10,
    output_fields=["source_id", "text"]
)

for hits in results:
    for hit in hits:
        print(f"ID: {hit.id}, Distance: {hit.distance}")
        print(f"Source: {hit.entity.get('source_id')}")
```

## Troubleshooting

### Common Issues

#### Issue: Milvus Not Starting

**Symptoms:**

- Container in CrashLoopBackOff
- etcd or MinIO connection errors

**Resolution:**

```bash
# Check dependencies
docker compose -f docker-compose.dev.yml logs etcd --tail=20
docker compose -f docker-compose.dev.yml logs minio --tail=20

# Verify etcd is accessible
curl -s http://localhost:2379/health

# Verify MinIO is accessible
curl -s http://localhost:9000/minio/health/live

# Restart in order
docker compose -f docker-compose.dev.yml restart etcd minio
sleep 10
docker compose -f docker-compose.dev.yml restart milvus
```

#### Issue: Collection Not Found

**Symptoms:**

- "collection not found" errors
- Search returns empty

**Resolution:**

```python
from pymilvus import connections, utility

connections.connect("default", host="localhost", port="19530")

# Check if collection exists
print(utility.list_collections())

# Collection might need to be loaded
if "knowledge_embeddings" in utility.list_collections():
    from pymilvus import Collection
    col = Collection("knowledge_embeddings")
    col.load()
    print(f"Loaded. Entities: {col.num_entities}")
```

#### Issue: Slow Search

**Symptoms:**

- High latency on vector search
- `MilvusHighQueryLatency` alert

**Resolution:**

```python
from pymilvus import Collection

collection = Collection("knowledge_embeddings")

# Check index status
print(collection.indexes)

# Rebuild index if corrupted
collection.release()
collection.drop_index()
collection.create_index(
    "embedding",
    {
        "metric_type": "COSINE",
        "index_type": "HNSW",
        "params": {"M": 16, "efConstruction": 256}
    }
)
collection.load()

# Compact to merge segments
collection.compact()
```

#### Issue: Memory Pressure

**Symptoms:**

- OOM kills
- `MilvusHighMemoryUsage` alert

**Resolution:**

```bash
# Check memory usage
docker stats milvus --no-stream

# Release unused collections
python3 << 'EOF'
from pymilvus import connections, Collection, utility

connections.connect("default", host="localhost", port="19530")
for name in utility.list_collections():
    col = Collection(name)
    # Check if truly needed
    if col.num_entities == 0:
        print(f"Empty collection: {name}")
    col.release()  # Unload from memory
EOF

# Consider increasing memory limit in docker-compose
```

## Backup & Recovery

### Backup Procedure

Milvus data is stored in etcd (metadata) and MinIO (vectors). Backup both:

```bash
# Create backup directory
BACKUP_DIR="/backup/milvus/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Backup etcd
docker compose -f docker-compose.dev.yml exec etcd \
  etcdctl snapshot save /tmp/etcd-backup.db
docker cp $(docker compose ps -q etcd):/tmp/etcd-backup.db "$BACKUP_DIR/etcd.db"

# Backup MinIO (using mc client)
docker run --rm --network=lilith-dev \
  minio/mc alias set myminio http://minio:9000 minioadmin minioadmin
docker run --rm --network=lilith-dev \
  -v "$BACKUP_DIR:/backup" \
  minio/mc mirror myminio/milvus-bucket /backup/minio-data

echo "Backup completed to $BACKUP_DIR"
```

### Restore Procedure

```bash
BACKUP_DIR="/backup/milvus/20240115-120000"

# Stop Milvus
docker compose -f docker-compose.dev.yml stop milvus

# Restore etcd
docker cp "$BACKUP_DIR/etcd.db" $(docker compose ps -q etcd):/tmp/etcd-backup.db
docker compose -f docker-compose.dev.yml exec etcd \
  etcdctl snapshot restore /tmp/etcd-backup.db --data-dir=/etcd-restore
# Then manually move restored data

# Restore MinIO
docker run --rm --network=lilith-dev \
  -v "$BACKUP_DIR:/backup" \
  minio/mc mirror /backup/minio-data myminio/milvus-bucket

# Restart Milvus
docker compose -f docker-compose.dev.yml start milvus
```

### Disaster Recovery

For complete data loss, collections need to be recreated and re-indexed from
source data:

```python
# Assuming source data is available
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType

connections.connect("default", host="localhost", port="19530")

# Recreate collection schema
# ... (use original schema definition)

# Re-embed and insert data
# This requires the original text data and embedding model
```

## Performance Tuning

### Recommended Settings for Production

```yaml
milvus:
  deploy:
    resources:
      limits:
        memory: 16G
      reservations:
        memory: 8G
  environment:
    # Additional performance settings via config
    - MILVUS_LOG_LEVEL=info
```

### Index Parameters by Collection Size

| Collection Size   | M   | efConstruction | ef (search) |
| ----------------- | --- | -------------- | ----------- |
| < 100K vectors    | 16  | 128            | 32          |
| 100K - 1M vectors | 16  | 256            | 64          |
| 1M - 10M vectors  | 32  | 256            | 128         |
| > 10M vectors     | 48  | 512            | 256         |

### Key Metrics to Monitor

| Metric               | Warning Threshold | Critical Threshold |
| -------------------- | ----------------- | ------------------ |
| Query Latency (P95)  | > 2 seconds       | > 5 seconds        |
| Insert Latency (P95) | > 5 seconds       | > 10 seconds       |
| Memory Usage         | > 85%             | > 95%              |
| etcd Disk Usage      | > 3GB             | > 3.5GB            |
| MinIO Disk Usage     | > 80%             | > 90%              |

## Integration with Services

### Sophia Vector Store Integration

```typescript
// libs/sophia/vectordb/src/stores/milvus-store.ts
import { MilvusClient } from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient({
  address: process.env.MILVUS_ADDRESS || 'localhost:19530',
});

export async function searchSimilar(embedding: number[], limit: number = 10) {
  return await client.search({
    collection_name: 'knowledge_embeddings',
    vector: embedding,
    limit,
    metric_type: 'COSINE',
    params: { ef: 64 },
  });
}
```

### AI Service Integration

```typescript
// services/ai/src/embeddings.ts
import { MilvusClient } from '@zilliz/milvus2-sdk-node';

export async function storeEmbedding(
  sourceId: string,
  text: string,
  embedding: number[],
  metadata: object
) {
  const client = new MilvusClient({
    address: process.env.MILVUS_ADDRESS || 'localhost:19530',
  });

  await client.insert({
    collection_name: 'knowledge_embeddings',
    data: [
      {
        source_id: sourceId,
        text,
        embedding,
        metadata,
      },
    ],
  });
}
```

---

**Last Updated**: 2025-01-XX **Owner**: Platform Team **Review Cycle**:
Quarterly
