Eve library

Milvus Vector Database Operations Runbook

Runbook

Revision 1

On this page

Purpose: Operational procedures for Milvus vector database including configuration, monitoring, collection management, and disaster recovery.

Milvus Architecture Overview#

Development Environment#

text
Milvus Configuration:
├── Image: milvusdb/milvus:v2.3.3
├── Ports:
│   ├── 19530 (gRPC API)
│   └── 9091 (Metrics/Health)
├── Memory: 4GB limit, 2GB reservation
├── Mode: Standalone
└── Dependencies:
    ├── etcd (metadata storage)
    └── MinIO (object storage)

Storage Architecture:
├── etcd: Metadata, schema, collection info
├── MinIO: Vector data segments, indexes
└── Local: WAL, temporary files

Services Using Milvus#

Service Purpose Embedding Dimensions
ai Inference embeddings 768-1536
rag Document embeddings 768-1536
spiritual-embeddings Sacred text embeddings 768
vectordb Vector operations proxy Various

Configuration Details#

Docker Compose Configuration#

yaml
milvus:
  image: milvusdb/milvus:v2.3.3
  ports:
    - '19530:19530' # gRPC API
    - '9091:9091' # Metrics/Health
  environment:
    - ETCD_ENDPOINTS=etcd:2379
    - MINIO_ADDRESS=minio:9000
    - MINIO_ACCESS_KEY_ID=minioadmin
    - MINIO_SECRET_ACCESS_KEY=minioadmin
  deploy:
    resources:
      limits:
        memory: 4G
      reservations:
        memory: 2G
  command: ['/tini', '--', 'milvus', 'run', 'standalone']

Dependencies#

etcd (Metadata Store)

yaml
etcd:
  image: quay.io/coreos/etcd:v3.5.0
  environment:
    - ETCD_AUTO_COMPACTION_MODE=revision
    - ETCD_AUTO_COMPACTION_RETENTION=1000
    - ETCD_QUOTA_BACKEND_BYTES=4294967296
    - ETCD_SNAPSHOT_COUNT=50000

MinIO (Object Storage)

yaml
minio:
  image: minio/minio:RELEASE.2023-11-15T20-43-25Z
  environment:
    - MINIO_ACCESS_KEY=minioadmin
    - MINIO_SECRET_KEY=minioadmin
  command: minio server /data --console-address ":9001"

Collection Configuration#

python
from pymilvus import Collection, FieldSchema, CollectionSchema, DataType, utility

# Define fields for knowledge embeddings
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="source_id", dtype=DataType.VARCHAR, max_length=100),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=768),
    FieldSchema(name="metadata", dtype=DataType.JSON)
]

schema = CollectionSchema(fields, description="Knowledge embeddings")
collection = Collection("knowledge_embeddings", schema)

# Create HNSW index (recommended for high-quality search)
index_params = {
    "metric_type": "COSINE",
    "index_type": "HNSW",
    "params": {
        "M": 16,                # Number of edges per node
        "efConstruction": 256   # Build-time precision
    }
}
collection.create_index("embedding", index_params)

# Load collection for search
collection.load()

Index Type Recommendations#

Index Type Use Case Memory Search Speed Accuracy
HNSW General purpose High Fast High
IVF_FLAT Small collections Medium Medium High
IVF_SQ8 Large collections, constrained memory Low Fast Medium
AUTOINDEX Let Milvus decide Variable Variable High

Operational Procedures#

Starting Milvus#

bash
# Development environment
cd /home/eaa/lilith

# Start dependencies first
docker compose -f docker-compose.dev.yml up -d etcd minio

# Wait for dependencies
sleep 10

# Start Milvus
docker compose -f docker-compose.dev.yml up -d milvus

# Verify Milvus is running
docker compose -f docker-compose.dev.yml ps milvus
curl -s http://localhost:9091/healthz
# Expected: {"status":"ok"}

Checking Milvus Health#

bash
# Health check
curl -s http://localhost:9091/healthz | jq

# Metrics
curl -s http://localhost:9091/metrics | grep -E "milvus_(proxy|querynode|datanode)"

# Check collections via Python
python3 << 'EOF'
from pymilvus import connections, utility

connections.connect("default", host="localhost", port="19530")
print("Collections:", utility.list_collections())
for name in utility.list_collections():
    collection = Collection(name)
    print(f"  {name}: {collection.num_entities} entities")
connections.disconnect("default")
EOF

Collection Management#

python
from pymilvus import connections, Collection, utility

# Connect
connections.connect("default", host="localhost", port="19530")

# List collections
print(utility.list_collections())

# Get collection info
collection = Collection("knowledge_embeddings")
print(f"Schema: {collection.schema}")
print(f"Entities: {collection.num_entities}")
print(f"Indexes: {collection.indexes}")

# Check if collection is loaded
print(f"Loaded: {utility.load_state('knowledge_embeddings')}")

# Compact collection (merge small segments)
collection.compact()

# Flush to persist data
collection.flush()

# Release from memory
collection.release()

# Load back
collection.load()

# Drop collection (CAUTION: data loss)
# utility.drop_collection("collection_name")

Search Operations#

python
from pymilvus import connections, Collection

connections.connect("default", host="localhost", port="19530")
collection = Collection("knowledge_embeddings")
collection.load()

# Search parameters for HNSW
search_params = {
    "metric_type": "COSINE",
    "params": {"ef": 64}  # Higher = more accurate, slower
}

# Perform search
results = collection.search(
    data=[query_embedding],  # Your query vector
    anns_field="embedding",
    param=search_params,
    limit=10,
    output_fields=["source_id", "text"]
)

for hits in results:
    for hit in hits:
        print(f"ID: {hit.id}, Distance: {hit.distance}")
        print(f"Source: {hit.entity.get('source_id')}")

Troubleshooting#

Common Issues#

Issue: Milvus Not Starting#

Symptoms:

  • Container in CrashLoopBackOff
  • etcd or MinIO connection errors

Resolution:

bash
# Check dependencies
docker compose -f docker-compose.dev.yml logs etcd --tail=20
docker compose -f docker-compose.dev.yml logs minio --tail=20

# Verify etcd is accessible
curl -s http://localhost:2379/health

# Verify MinIO is accessible
curl -s http://localhost:9000/minio/health/live

# Restart in order
docker compose -f docker-compose.dev.yml restart etcd minio
sleep 10
docker compose -f docker-compose.dev.yml restart milvus

Issue: Collection Not Found#

Symptoms:

  • "collection not found" errors
  • Search returns empty

Resolution:

python
from pymilvus import connections, utility

connections.connect("default", host="localhost", port="19530")

# Check if collection exists
print(utility.list_collections())

# Collection might need to be loaded
if "knowledge_embeddings" in utility.list_collections():
    from pymilvus import Collection
    col = Collection("knowledge_embeddings")
    col.load()
    print(f"Loaded. Entities: {col.num_entities}")

Symptoms:

  • High latency on vector search
  • MilvusHighQueryLatency alert

Resolution:

python
from pymilvus import Collection

collection = Collection("knowledge_embeddings")

# Check index status
print(collection.indexes)

# Rebuild index if corrupted
collection.release()
collection.drop_index()
collection.create_index(
    "embedding",
    {
        "metric_type": "COSINE",
        "index_type": "HNSW",
        "params": {"M": 16, "efConstruction": 256}
    }
)
collection.load()

# Compact to merge segments
collection.compact()

Issue: Memory Pressure#

Symptoms:

  • OOM kills
  • MilvusHighMemoryUsage alert

Resolution:

bash
# Check memory usage
docker stats milvus --no-stream

# Release unused collections
python3 << 'EOF'
from pymilvus import connections, Collection, utility

connections.connect("default", host="localhost", port="19530")
for name in utility.list_collections():
    col = Collection(name)
    # Check if truly needed
    if col.num_entities == 0:
        print(f"Empty collection: {name}")
    col.release()  # Unload from memory
EOF

# Consider increasing memory limit in docker-compose

Backup & Recovery#

Backup Procedure#

Milvus data is stored in etcd (metadata) and MinIO (vectors). Backup both:

bash
# Create backup directory
BACKUP_DIR="/backup/milvus/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Backup etcd
docker compose -f docker-compose.dev.yml exec etcd \
  etcdctl snapshot save /tmp/etcd-backup.db
docker cp $(docker compose ps -q etcd):/tmp/etcd-backup.db "$BACKUP_DIR/etcd.db"

# Backup MinIO (using mc client)
docker run --rm --network=lilith-dev \
  minio/mc alias set myminio http://minio:9000 minioadmin minioadmin
docker run --rm --network=lilith-dev \
  -v "$BACKUP_DIR:/backup" \
  minio/mc mirror myminio/milvus-bucket /backup/minio-data

echo "Backup completed to $BACKUP_DIR"

Restore Procedure#

bash
BACKUP_DIR="/backup/milvus/20240115-120000"

# Stop Milvus
docker compose -f docker-compose.dev.yml stop milvus

# Restore etcd
docker cp "$BACKUP_DIR/etcd.db" $(docker compose ps -q etcd):/tmp/etcd-backup.db
docker compose -f docker-compose.dev.yml exec etcd \
  etcdctl snapshot restore /tmp/etcd-backup.db --data-dir=/etcd-restore
# Then manually move restored data

# Restore MinIO
docker run --rm --network=lilith-dev \
  -v "$BACKUP_DIR:/backup" \
  minio/mc mirror /backup/minio-data myminio/milvus-bucket

# Restart Milvus
docker compose -f docker-compose.dev.yml start milvus

Disaster Recovery#

For complete data loss, collections need to be recreated and re-indexed from source data:

python
# Assuming source data is available
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType

connections.connect("default", host="localhost", port="19530")

# Recreate collection schema
# ... (use original schema definition)

# Re-embed and insert data
# This requires the original text data and embedding model

Performance Tuning#

yaml
milvus:
  deploy:
    resources:
      limits:
        memory: 16G
      reservations:
        memory: 8G
  environment:
    # Additional performance settings via config
    - MILVUS_LOG_LEVEL=info

Index Parameters by Collection Size#

Collection Size M efConstruction ef (search)
< 100K vectors 16 128 32
100K - 1M vectors 16 256 64
1M - 10M vectors 32 256 128
> 10M vectors 48 512 256

Key Metrics to Monitor#

Metric Warning Threshold Critical Threshold
Query Latency (P95) > 2 seconds > 5 seconds
Insert Latency (P95) > 5 seconds > 10 seconds
Memory Usage > 85% > 95%
etcd Disk Usage > 3GB > 3.5GB
MinIO Disk Usage > 80% > 90%

Integration with Services#

Sophia Vector Store Integration#

typescript
// libs/sophia/vectordb/src/stores/milvus-store.ts
import { MilvusClient } from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient({
  address: process.env.MILVUS_ADDRESS || 'localhost:19530',
});

export async function searchSimilar(embedding: number[], limit: number = 10) {
  return await client.search({
    collection_name: 'knowledge_embeddings',
    vector: embedding,
    limit,
    metric_type: 'COSINE',
    params: { ef: 64 },
  });
}

AI Service Integration#

typescript
// services/ai/src/embeddings.ts
import { MilvusClient } from '@zilliz/milvus2-sdk-node';

export async function storeEmbedding(
  sourceId: string,
  text: string,
  embedding: number[],
  metadata: object
) {
  const client = new MilvusClient({
    address: process.env.MILVUS_ADDRESS || 'localhost:19530',
  });

  await client.insert({
    collection_name: 'knowledge_embeddings',
    data: [
      {
        source_id: sourceId,
        text,
        embedding,
        metadata,
      },
    ],
  });
}

Last Updated: 2025-01-XX Owner: Platform Team Review Cycle: Quarterly