This document describes usage patterns for the @lilith/vectordb package and
Milvus integration in the Lilith platform.
Overview#
The Lilith platform uses Milvus as its vector database for semantic search, RAG
(Retrieval-Augmented Generation), and recommendation systems. The
@lilith/vectordb package provides:
- Collection schema definitions
- Vector utility functions
- Text chunking utilities
- Batch processing helpers
Installation#
typescript
import {
COLLECTIONS,
WISDOM_EMBEDDINGS_SCHEMA,
cosineSimilarity,
normalizeVector,
chunkText,
} from '@lilith/vectordb';
Collection Schemas#
Available Collections#
| Collection | Dimension | Index Type | Use Case |
|---|---|---|---|
wisdom_embeddings |
768 | HNSW | Spiritual/philosophical content RAG |
content_embeddings |
768 | HNSW | Platform content recommendations |
user_preference_embeddings |
384 | HNSW | User personalization |
conversation_embeddings |
768 | IVF_FLAT | Chat history and context |
persona_embeddings |
384 | HNSW | AI mentor personas |
knowledge_chunks |
768 | HNSW | RAG document chunks |
semantic_search |
768 | IVF_FLAT | Platform-wide search |
Collection Name Constants#
typescript
import { COLLECTIONS } from '@lilith/vectordb';
// Use constants for type safety
const wisdomCollection = COLLECTIONS.WISDOM; // 'wisdom_embeddings'
const contentCollection = COLLECTIONS.CONTENT; // 'content_embeddings'
const userPrefsCollection = COLLECTIONS.USER_PREFERENCES;
const conversationCollection = COLLECTIONS.CONVERSATION;
const personaCollection = COLLECTIONS.PERSONA;
const chunksCollection = COLLECTIONS.KNOWLEDGE_CHUNKS;
const searchCollection = COLLECTIONS.SEMANTIC_SEARCH;
Schema Definitions#
Each schema includes dimension, metric type, index configuration, and metadata fields:
typescript
import { WISDOM_EMBEDDINGS_SCHEMA } from '@lilith/vectordb';
// Schema structure
console.log(WISDOM_EMBEDDINGS_SCHEMA);
// {
// name: 'wisdom_embeddings',
// dimension: 768,
// metricType: 'cosine',
// indexType: 'hnsw',
// indexParams: { m: 16, efConstruction: 256 },
// metadataFields: [
// { name: 'document_id', type: 'string', maxLength: 100, indexed: true },
// { name: 'tradition', type: 'string', maxLength: 50, indexed: true },
// // ...
// ]
// }
Similarity Calculations#
Cosine Similarity#
Best for normalized embeddings (most common):
typescript
import { cosineSimilarity } from '@lilith/vectordb';
const vectorA = [0.1, 0.2, 0.3, ...]; // 768 dimensions
const vectorB = [0.15, 0.25, 0.35, ...];
const similarity = cosineSimilarity(vectorA, vectorB);
// Returns value between -1 and 1 (1 = identical)
Euclidean Distance#
Best for spatial relationships:
typescript
import { euclideanDistance } from '@lilith/vectordb';
const distance = euclideanDistance(vectorA, vectorB);
// Returns non-negative value (0 = identical)
Dot Product#
Best for unnormalized vectors:
typescript
import { dotProduct } from '@lilith/vectordb';
const product = dotProduct(vectorA, vectorB);
// Returns scalar value
Generic Similarity Computation#
typescript
import { computeSimilarity } from '@lilith/vectordb';
// Use metric type from schema
const similarity = computeSimilarity(vectorA, vectorB, 'cosine');
const distance = computeSimilarity(vectorA, vectorB, 'euclidean');
const dot = computeSimilarity(vectorA, vectorB, 'dot_product');
Vector Transformations#
Normalization#
Normalize vectors for cosine similarity:
typescript
import { normalizeVector, vectorMagnitude } from '@lilith/vectordb';
const rawVector = [1.0, 2.0, 3.0, ...];
const normalized = normalizeVector(rawVector);
// Unit vector with magnitude 1.0
const magnitude = vectorMagnitude(rawVector);
// Original magnitude
Vector Arithmetic#
typescript
import {
addVectors,
subtractVectors,
scaleVector,
averageVectors,
weightedAverageVectors,
} from '@lilith/vectordb';
// Add vectors
const sum = addVectors(vectorA, vectorB);
// Subtract vectors
const diff = subtractVectors(vectorA, vectorB);
// Scale by scalar
const scaled = scaleVector(vectorA, 0.5);
// Average multiple vectors
const avg = averageVectors([vectorA, vectorB, vectorC]);
// Weighted average (for combining embeddings)
const weighted = weightedAverageVectors(
[vectorA, vectorB, vectorC],
[0.5, 0.3, 0.2] // Weights must sum to 1 for normalized result
);
Text Chunking#
Character-Based Chunking#
typescript
import { chunkText } from '@lilith/vectordb';
const longText = 'Your long document text here...';
// Chunk with overlap for context preservation
const chunks = chunkText(longText, 1000, 100);
// Returns array of ~1000 character chunks with 100 char overlap
Sentence-Based Chunking#
Better for natural text boundaries:
typescript
import { chunkBySentences } from '@lilith/vectordb';
const document = 'First sentence. Second sentence. Third sentence.';
const chunks = chunkBySentences(document, 500);
// Returns chunks respecting sentence boundaries
Batch Processing#
Array Batching#
typescript
import { batchArray } from '@lilith/vectordb';
const items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const batches = batchArray(items, 3);
// [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
Rate-Limited Batch Processing#
For API rate limits:
typescript
import { batchArray, processBatchesWithDelay } from '@lilith/vectordb';
async function generateEmbeddings(texts: string[]) {
const batches = batchArray(texts, 10); // 10 texts per batch
const embeddings = await processBatchesWithDelay(
batches,
async (batch) => {
// Call embedding API
return await embeddingService.embed(batch);
},
100 // 100ms delay between batches
);
return embeddings;
}
Validation#
Vector Validation#
typescript
import {
validateVectorDimension,
validateVectorValues,
sanitizeVector,
} from '@lilith/vectordb';
const vector = [0.1, 0.2, NaN, 0.4]; // Invalid vector
// Check dimension
const validDim = validateVectorDimension(vector, 768); // false
// Check for NaN/Infinity
const validValues = validateVectorValues(vector); // false
// Sanitize (returns null if invalid)
const sanitized = sanitizeVector(vector, 768); // null
Score Conversions#
typescript
import { distanceToScore, scoreToDistance } from '@lilith/vectordb';
// Convert Milvus distance to 0-1 similarity score
const distance = 0.3;
const score = distanceToScore(distance, 'cosine'); // ~0.7
// Convert score back to distance for queries
const queryDistance = scoreToDistance(0.8, 'cosine'); // 0.2
Common Patterns#
RAG Retrieval Pattern#
typescript
import {
COLLECTIONS,
KNOWLEDGE_CHUNKS_SCHEMA,
cosineSimilarity,
chunkBySentences,
batchArray,
} from '@lilith/vectordb';
async function ragPipeline(query: string, milvusClient: any) {
// 1. Generate query embedding
const queryEmbedding = await embeddingService.embed(query);
// 2. Search knowledge chunks
const results = await milvusClient.search({
collection_name: COLLECTIONS.KNOWLEDGE_CHUNKS,
vectors: [queryEmbedding],
top_k: 5,
params: { ef: 64 },
});
// 3. Return chunks for context
return results.map((r) => ({
content: r.content,
score: r.score,
documentId: r.document_id,
}));
}
Content Recommendation Pattern#
typescript
import {
COLLECTIONS,
weightedAverageVectors,
normalizeVector,
} from '@lilith/vectordb';
async function getRecommendations(
userPreferenceVector: number[],
recentContentVectors: number[][],
milvusClient: any
) {
// Combine user preferences with recent interactions
const combinedVector = weightedAverageVectors(
[userPreferenceVector, ...recentContentVectors],
[0.6, ...recentContentVectors.map(() => 0.4 / recentContentVectors.length)]
);
const normalized = normalizeVector(combinedVector);
// Search for similar content
const results = await milvusClient.search({
collection_name: COLLECTIONS.CONTENT,
vectors: [normalized],
top_k: 10,
filter: 'is_premium == false',
});
return results;
}
Document Indexing Pattern#
typescript
import {
chunkBySentences,
batchArray,
processBatchesWithDelay,
validateVectorValues,
} from '@lilith/vectordb';
async function indexDocument(
documentId: string,
content: string,
milvusClient: any
) {
// 1. Chunk the document
const chunks = chunkBySentences(content, 1000);
// 2. Generate embeddings in batches
const batches = batchArray(chunks, 10);
const embeddings = await processBatchesWithDelay(
batches,
async (batch) => embeddingService.embed(batch),
100
);
// 3. Validate and prepare records
const records = chunks.map((chunk, index) => {
const vector = embeddings[index];
if (!validateVectorValues(vector)) {
throw new Error(`Invalid embedding at index ${index}`);
}
return {
document_id: documentId,
chunk_id: `${documentId}_${index}`,
chunk_index: index,
total_chunks: chunks.length,
content: chunk,
vector,
};
});
// 4. Insert into Milvus
await milvusClient.insert({
collection_name: 'knowledge_chunks',
data: records,
});
}
Semantic Search Pattern#
typescript
import {
COLLECTIONS,
SEMANTIC_SEARCH_SCHEMA,
normalizeVector,
} from '@lilith/vectordb';
async function semanticSearch(
query: string,
filters: Record<string, any>,
milvusClient: any
) {
const queryEmbedding = await embeddingService.embed(query);
const normalized = normalizeVector(queryEmbedding);
// Build filter expression
const filterExpr = Object.entries(filters)
.map(([key, value]) => `${key} == "${value}"`)
.join(' && ');
const results = await milvusClient.search({
collection_name: COLLECTIONS.SEMANTIC_SEARCH,
vectors: [normalized],
top_k: 20,
filter: filterExpr || undefined,
output_fields: ['item_id', 'item_type', 'title', 'content'],
});
return results;
}
Index Configuration#
HNSW Index (Default)#
Best for most use cases - high recall, fast queries:
typescript
// From schema
indexParams: {
m: 16, // Connections per layer (higher = better recall, more memory)
efConstruction: 256 // Build-time search width (higher = better index quality)
}
// Query-time parameter
params: {
ef: 64 // Search-time width (higher = better recall, slower)
}
IVF_FLAT Index#
Best for large-scale data with good recall:
typescript
// From schema
indexParams: {
nlist: 1024; // Number of clusters (sqrt(n) to 4*sqrt(n) recommended)
}
// Query-time parameter
params: {
nprobe: 16; // Clusters to search (higher = better recall, slower)
}
Environment Configuration#
See Environment Variables for Milvus configuration:
bash
MILVUS_HOST=localhost
MILVUS_PORT=19530
MILVUS_USERNAME=
MILVUS_PASSWORD=
MILVUS_SECURE=false
MILVUS_DATABASE=lilith
Best Practices#
- Always normalize vectors before insertion when using cosine similarity
- Use appropriate chunk sizes - 500-1000 characters for most text content
- Include overlap in chunking to preserve context across boundaries
- Batch operations to avoid rate limits and improve throughput
- Validate vectors before insertion to catch NaN/Infinity values
- Use filters to narrow search scope and improve performance
- Choose index parameters based on dataset size and recall requirements
- Monitor collection stats for partition management and optimization