Purpose: Operational procedures for Milvus vector database including configuration, monitoring, collection management, and disaster recovery.
Milvus Architecture Overview#
Development Environment#
text
Milvus Configuration:
├── Image: milvusdb/milvus:v2.3.3
├── Ports:
│ ├── 19530 (gRPC API)
│ └── 9091 (Metrics/Health)
├── Memory: 4GB limit, 2GB reservation
├── Mode: Standalone
└── Dependencies:
├── etcd (metadata storage)
└── MinIO (object storage)
Storage Architecture:
├── etcd: Metadata, schema, collection info
├── MinIO: Vector data segments, indexes
└── Local: WAL, temporary files
Services Using Milvus#
| Service | Purpose | Embedding Dimensions |
|---|---|---|
| ai | Inference embeddings | 768-1536 |
| rag | Document embeddings | 768-1536 |
| spiritual-embeddings | Sacred text embeddings | 768 |
| vectordb | Vector operations proxy | Various |
Configuration Details#
Docker Compose Configuration#
yaml
milvus:
image: milvusdb/milvus:v2.3.3
ports:
- '19530:19530' # gRPC API
- '9091:9091' # Metrics/Health
environment:
- ETCD_ENDPOINTS=etcd:2379
- MINIO_ADDRESS=minio:9000
- MINIO_ACCESS_KEY_ID=minioadmin
- MINIO_SECRET_ACCESS_KEY=minioadmin
deploy:
resources:
limits:
memory: 4G
reservations:
memory: 2G
command: ['/tini', '--', 'milvus', 'run', 'standalone']
Dependencies#
etcd (Metadata Store)
yaml
etcd:
image: quay.io/coreos/etcd:v3.5.0
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
MinIO (Object Storage)
yaml
minio:
image: minio/minio:RELEASE.2023-11-15T20-43-25Z
environment:
- MINIO_ACCESS_KEY=minioadmin
- MINIO_SECRET_KEY=minioadmin
command: minio server /data --console-address ":9001"
Collection Configuration#
Recommended Collection Settings#
python
from pymilvus import Collection, FieldSchema, CollectionSchema, DataType, utility
# Define fields for knowledge embeddings
fields = [
FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
FieldSchema(name="source_id", dtype=DataType.VARCHAR, max_length=100),
FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=768),
FieldSchema(name="metadata", dtype=DataType.JSON)
]
schema = CollectionSchema(fields, description="Knowledge embeddings")
collection = Collection("knowledge_embeddings", schema)
# Create HNSW index (recommended for high-quality search)
index_params = {
"metric_type": "COSINE",
"index_type": "HNSW",
"params": {
"M": 16, # Number of edges per node
"efConstruction": 256 # Build-time precision
}
}
collection.create_index("embedding", index_params)
# Load collection for search
collection.load()
Index Type Recommendations#
| Index Type | Use Case | Memory | Search Speed | Accuracy |
|---|---|---|---|---|
| HNSW | General purpose | High | Fast | High |
| IVF_FLAT | Small collections | Medium | Medium | High |
| IVF_SQ8 | Large collections, constrained memory | Low | Fast | Medium |
| AUTOINDEX | Let Milvus decide | Variable | Variable | High |
Operational Procedures#
Starting Milvus#
bash
# Development environment
cd /home/eaa/lilith
# Start dependencies first
docker compose -f docker-compose.dev.yml up -d etcd minio
# Wait for dependencies
sleep 10
# Start Milvus
docker compose -f docker-compose.dev.yml up -d milvus
# Verify Milvus is running
docker compose -f docker-compose.dev.yml ps milvus
curl -s http://localhost:9091/healthz
# Expected: {"status":"ok"}
Checking Milvus Health#
bash
# Health check
curl -s http://localhost:9091/healthz | jq
# Metrics
curl -s http://localhost:9091/metrics | grep -E "milvus_(proxy|querynode|datanode)"
# Check collections via Python
python3 << 'EOF'
from pymilvus import connections, utility
connections.connect("default", host="localhost", port="19530")
print("Collections:", utility.list_collections())
for name in utility.list_collections():
collection = Collection(name)
print(f" {name}: {collection.num_entities} entities")
connections.disconnect("default")
EOF
Collection Management#
python
from pymilvus import connections, Collection, utility
# Connect
connections.connect("default", host="localhost", port="19530")
# List collections
print(utility.list_collections())
# Get collection info
collection = Collection("knowledge_embeddings")
print(f"Schema: {collection.schema}")
print(f"Entities: {collection.num_entities}")
print(f"Indexes: {collection.indexes}")
# Check if collection is loaded
print(f"Loaded: {utility.load_state('knowledge_embeddings')}")
# Compact collection (merge small segments)
collection.compact()
# Flush to persist data
collection.flush()
# Release from memory
collection.release()
# Load back
collection.load()
# Drop collection (CAUTION: data loss)
# utility.drop_collection("collection_name")
Search Operations#
python
from pymilvus import connections, Collection
connections.connect("default", host="localhost", port="19530")
collection = Collection("knowledge_embeddings")
collection.load()
# Search parameters for HNSW
search_params = {
"metric_type": "COSINE",
"params": {"ef": 64} # Higher = more accurate, slower
}
# Perform search
results = collection.search(
data=[query_embedding], # Your query vector
anns_field="embedding",
param=search_params,
limit=10,
output_fields=["source_id", "text"]
)
for hits in results:
for hit in hits:
print(f"ID: {hit.id}, Distance: {hit.distance}")
print(f"Source: {hit.entity.get('source_id')}")
Troubleshooting#
Common Issues#
Issue: Milvus Not Starting#
Symptoms:
- Container in CrashLoopBackOff
- etcd or MinIO connection errors
Resolution:
bash
# Check dependencies
docker compose -f docker-compose.dev.yml logs etcd --tail=20
docker compose -f docker-compose.dev.yml logs minio --tail=20
# Verify etcd is accessible
curl -s http://localhost:2379/health
# Verify MinIO is accessible
curl -s http://localhost:9000/minio/health/live
# Restart in order
docker compose -f docker-compose.dev.yml restart etcd minio
sleep 10
docker compose -f docker-compose.dev.yml restart milvus
Issue: Collection Not Found#
Symptoms:
- "collection not found" errors
- Search returns empty
Resolution:
python
from pymilvus import connections, utility
connections.connect("default", host="localhost", port="19530")
# Check if collection exists
print(utility.list_collections())
# Collection might need to be loaded
if "knowledge_embeddings" in utility.list_collections():
from pymilvus import Collection
col = Collection("knowledge_embeddings")
col.load()
print(f"Loaded. Entities: {col.num_entities}")
Issue: Slow Search#
Symptoms:
- High latency on vector search
MilvusHighQueryLatencyalert
Resolution:
python
from pymilvus import Collection
collection = Collection("knowledge_embeddings")
# Check index status
print(collection.indexes)
# Rebuild index if corrupted
collection.release()
collection.drop_index()
collection.create_index(
"embedding",
{
"metric_type": "COSINE",
"index_type": "HNSW",
"params": {"M": 16, "efConstruction": 256}
}
)
collection.load()
# Compact to merge segments
collection.compact()
Issue: Memory Pressure#
Symptoms:
- OOM kills
MilvusHighMemoryUsagealert
Resolution:
bash
# Check memory usage
docker stats milvus --no-stream
# Release unused collections
python3 << 'EOF'
from pymilvus import connections, Collection, utility
connections.connect("default", host="localhost", port="19530")
for name in utility.list_collections():
col = Collection(name)
# Check if truly needed
if col.num_entities == 0:
print(f"Empty collection: {name}")
col.release() # Unload from memory
EOF
# Consider increasing memory limit in docker-compose
Backup & Recovery#
Backup Procedure#
Milvus data is stored in etcd (metadata) and MinIO (vectors). Backup both:
bash
# Create backup directory
BACKUP_DIR="/backup/milvus/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
# Backup etcd
docker compose -f docker-compose.dev.yml exec etcd \
etcdctl snapshot save /tmp/etcd-backup.db
docker cp $(docker compose ps -q etcd):/tmp/etcd-backup.db "$BACKUP_DIR/etcd.db"
# Backup MinIO (using mc client)
docker run --rm --network=lilith-dev \
minio/mc alias set myminio http://minio:9000 minioadmin minioadmin
docker run --rm --network=lilith-dev \
-v "$BACKUP_DIR:/backup" \
minio/mc mirror myminio/milvus-bucket /backup/minio-data
echo "Backup completed to $BACKUP_DIR"
Restore Procedure#
bash
BACKUP_DIR="/backup/milvus/20240115-120000"
# Stop Milvus
docker compose -f docker-compose.dev.yml stop milvus
# Restore etcd
docker cp "$BACKUP_DIR/etcd.db" $(docker compose ps -q etcd):/tmp/etcd-backup.db
docker compose -f docker-compose.dev.yml exec etcd \
etcdctl snapshot restore /tmp/etcd-backup.db --data-dir=/etcd-restore
# Then manually move restored data
# Restore MinIO
docker run --rm --network=lilith-dev \
-v "$BACKUP_DIR:/backup" \
minio/mc mirror /backup/minio-data myminio/milvus-bucket
# Restart Milvus
docker compose -f docker-compose.dev.yml start milvus
Disaster Recovery#
For complete data loss, collections need to be recreated and re-indexed from source data:
python
# Assuming source data is available
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
connections.connect("default", host="localhost", port="19530")
# Recreate collection schema
# ... (use original schema definition)
# Re-embed and insert data
# This requires the original text data and embedding model
Performance Tuning#
Recommended Settings for Production#
yaml
milvus:
deploy:
resources:
limits:
memory: 16G
reservations:
memory: 8G
environment:
# Additional performance settings via config
- MILVUS_LOG_LEVEL=info
Index Parameters by Collection Size#
| Collection Size | M | efConstruction | ef (search) |
|---|---|---|---|
| < 100K vectors | 16 | 128 | 32 |
| 100K - 1M vectors | 16 | 256 | 64 |
| 1M - 10M vectors | 32 | 256 | 128 |
| > 10M vectors | 48 | 512 | 256 |
Key Metrics to Monitor#
| Metric | Warning Threshold | Critical Threshold |
|---|---|---|
| Query Latency (P95) | > 2 seconds | > 5 seconds |
| Insert Latency (P95) | > 5 seconds | > 10 seconds |
| Memory Usage | > 85% | > 95% |
| etcd Disk Usage | > 3GB | > 3.5GB |
| MinIO Disk Usage | > 80% | > 90% |
Integration with Services#
Sophia Vector Store Integration#
typescript
// libs/sophia/vectordb/src/stores/milvus-store.ts
import { MilvusClient } from '@zilliz/milvus2-sdk-node';
const client = new MilvusClient({
address: process.env.MILVUS_ADDRESS || 'localhost:19530',
});
export async function searchSimilar(embedding: number[], limit: number = 10) {
return await client.search({
collection_name: 'knowledge_embeddings',
vector: embedding,
limit,
metric_type: 'COSINE',
params: { ef: 64 },
});
}
AI Service Integration#
typescript
// services/ai/src/embeddings.ts
import { MilvusClient } from '@zilliz/milvus2-sdk-node';
export async function storeEmbedding(
sourceId: string,
text: string,
embedding: number[],
metadata: object
) {
const client = new MilvusClient({
address: process.env.MILVUS_ADDRESS || 'localhost:19530',
});
await client.insert({
collection_name: 'knowledge_embeddings',
data: [
{
source_id: sourceId,
text,
embedding,
metadata,
},
],
});
}
Last Updated: 2025-01-XX Owner: Platform Team Review Cycle: Quarterly