Eve library

Vector Database Usage Patterns

Guide

Revision 1

On this page

This document describes usage patterns for the @lilith/vectordb package and Milvus integration in the Lilith platform.

Overview#

The Lilith platform uses Milvus as its vector database for semantic search, RAG (Retrieval-Augmented Generation), and recommendation systems. The @lilith/vectordb package provides:

  • Collection schema definitions
  • Vector utility functions
  • Text chunking utilities
  • Batch processing helpers

Installation#

typescript
import {
  COLLECTIONS,
  WISDOM_EMBEDDINGS_SCHEMA,
  cosineSimilarity,
  normalizeVector,
  chunkText,
} from '@lilith/vectordb';

Collection Schemas#

Available Collections#

Collection Dimension Index Type Use Case
wisdom_embeddings 768 HNSW Spiritual/philosophical content RAG
content_embeddings 768 HNSW Platform content recommendations
user_preference_embeddings 384 HNSW User personalization
conversation_embeddings 768 IVF_FLAT Chat history and context
persona_embeddings 384 HNSW AI mentor personas
knowledge_chunks 768 HNSW RAG document chunks
semantic_search 768 IVF_FLAT Platform-wide search

Collection Name Constants#

typescript
import { COLLECTIONS } from '@lilith/vectordb';

// Use constants for type safety
const wisdomCollection = COLLECTIONS.WISDOM; // 'wisdom_embeddings'
const contentCollection = COLLECTIONS.CONTENT; // 'content_embeddings'
const userPrefsCollection = COLLECTIONS.USER_PREFERENCES;
const conversationCollection = COLLECTIONS.CONVERSATION;
const personaCollection = COLLECTIONS.PERSONA;
const chunksCollection = COLLECTIONS.KNOWLEDGE_CHUNKS;
const searchCollection = COLLECTIONS.SEMANTIC_SEARCH;

Schema Definitions#

Each schema includes dimension, metric type, index configuration, and metadata fields:

typescript
import { WISDOM_EMBEDDINGS_SCHEMA } from '@lilith/vectordb';

// Schema structure
console.log(WISDOM_EMBEDDINGS_SCHEMA);
// {
//   name: 'wisdom_embeddings',
//   dimension: 768,
//   metricType: 'cosine',
//   indexType: 'hnsw',
//   indexParams: { m: 16, efConstruction: 256 },
//   metadataFields: [
//     { name: 'document_id', type: 'string', maxLength: 100, indexed: true },
//     { name: 'tradition', type: 'string', maxLength: 50, indexed: true },
//     // ...
//   ]
// }

Similarity Calculations#

Cosine Similarity#

Best for normalized embeddings (most common):

typescript
import { cosineSimilarity } from '@lilith/vectordb';

const vectorA = [0.1, 0.2, 0.3, ...]; // 768 dimensions
const vectorB = [0.15, 0.25, 0.35, ...];

const similarity = cosineSimilarity(vectorA, vectorB);
// Returns value between -1 and 1 (1 = identical)

Euclidean Distance#

Best for spatial relationships:

typescript
import { euclideanDistance } from '@lilith/vectordb';

const distance = euclideanDistance(vectorA, vectorB);
// Returns non-negative value (0 = identical)

Dot Product#

Best for unnormalized vectors:

typescript
import { dotProduct } from '@lilith/vectordb';

const product = dotProduct(vectorA, vectorB);
// Returns scalar value

Generic Similarity Computation#

typescript
import { computeSimilarity } from '@lilith/vectordb';

// Use metric type from schema
const similarity = computeSimilarity(vectorA, vectorB, 'cosine');
const distance = computeSimilarity(vectorA, vectorB, 'euclidean');
const dot = computeSimilarity(vectorA, vectorB, 'dot_product');

Vector Transformations#

Normalization#

Normalize vectors for cosine similarity:

typescript
import { normalizeVector, vectorMagnitude } from '@lilith/vectordb';

const rawVector = [1.0, 2.0, 3.0, ...];
const normalized = normalizeVector(rawVector);
// Unit vector with magnitude 1.0

const magnitude = vectorMagnitude(rawVector);
// Original magnitude

Vector Arithmetic#

typescript
import {
  addVectors,
  subtractVectors,
  scaleVector,
  averageVectors,
  weightedAverageVectors,
} from '@lilith/vectordb';

// Add vectors
const sum = addVectors(vectorA, vectorB);

// Subtract vectors
const diff = subtractVectors(vectorA, vectorB);

// Scale by scalar
const scaled = scaleVector(vectorA, 0.5);

// Average multiple vectors
const avg = averageVectors([vectorA, vectorB, vectorC]);

// Weighted average (for combining embeddings)
const weighted = weightedAverageVectors(
  [vectorA, vectorB, vectorC],
  [0.5, 0.3, 0.2] // Weights must sum to 1 for normalized result
);

Text Chunking#

Character-Based Chunking#

typescript
import { chunkText } from '@lilith/vectordb';

const longText = 'Your long document text here...';

// Chunk with overlap for context preservation
const chunks = chunkText(longText, 1000, 100);
// Returns array of ~1000 character chunks with 100 char overlap

Sentence-Based Chunking#

Better for natural text boundaries:

typescript
import { chunkBySentences } from '@lilith/vectordb';

const document = 'First sentence. Second sentence. Third sentence.';

const chunks = chunkBySentences(document, 500);
// Returns chunks respecting sentence boundaries

Batch Processing#

Array Batching#

typescript
import { batchArray } from '@lilith/vectordb';

const items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const batches = batchArray(items, 3);
// [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Rate-Limited Batch Processing#

For API rate limits:

typescript
import { batchArray, processBatchesWithDelay } from '@lilith/vectordb';

async function generateEmbeddings(texts: string[]) {
  const batches = batchArray(texts, 10); // 10 texts per batch

  const embeddings = await processBatchesWithDelay(
    batches,
    async (batch) => {
      // Call embedding API
      return await embeddingService.embed(batch);
    },
    100 // 100ms delay between batches
  );

  return embeddings;
}

Validation#

Vector Validation#

typescript
import {
  validateVectorDimension,
  validateVectorValues,
  sanitizeVector,
} from '@lilith/vectordb';

const vector = [0.1, 0.2, NaN, 0.4]; // Invalid vector

// Check dimension
const validDim = validateVectorDimension(vector, 768); // false

// Check for NaN/Infinity
const validValues = validateVectorValues(vector); // false

// Sanitize (returns null if invalid)
const sanitized = sanitizeVector(vector, 768); // null

Score Conversions#

typescript
import { distanceToScore, scoreToDistance } from '@lilith/vectordb';

// Convert Milvus distance to 0-1 similarity score
const distance = 0.3;
const score = distanceToScore(distance, 'cosine'); // ~0.7

// Convert score back to distance for queries
const queryDistance = scoreToDistance(0.8, 'cosine'); // 0.2

Common Patterns#

RAG Retrieval Pattern#

typescript
import {
  COLLECTIONS,
  KNOWLEDGE_CHUNKS_SCHEMA,
  cosineSimilarity,
  chunkBySentences,
  batchArray,
} from '@lilith/vectordb';

async function ragPipeline(query: string, milvusClient: any) {
  // 1. Generate query embedding
  const queryEmbedding = await embeddingService.embed(query);

  // 2. Search knowledge chunks
  const results = await milvusClient.search({
    collection_name: COLLECTIONS.KNOWLEDGE_CHUNKS,
    vectors: [queryEmbedding],
    top_k: 5,
    params: { ef: 64 },
  });

  // 3. Return chunks for context
  return results.map((r) => ({
    content: r.content,
    score: r.score,
    documentId: r.document_id,
  }));
}

Content Recommendation Pattern#

typescript
import {
  COLLECTIONS,
  weightedAverageVectors,
  normalizeVector,
} from '@lilith/vectordb';

async function getRecommendations(
  userPreferenceVector: number[],
  recentContentVectors: number[][],
  milvusClient: any
) {
  // Combine user preferences with recent interactions
  const combinedVector = weightedAverageVectors(
    [userPreferenceVector, ...recentContentVectors],
    [0.6, ...recentContentVectors.map(() => 0.4 / recentContentVectors.length)]
  );

  const normalized = normalizeVector(combinedVector);

  // Search for similar content
  const results = await milvusClient.search({
    collection_name: COLLECTIONS.CONTENT,
    vectors: [normalized],
    top_k: 10,
    filter: 'is_premium == false',
  });

  return results;
}

Document Indexing Pattern#

typescript
import {
  chunkBySentences,
  batchArray,
  processBatchesWithDelay,
  validateVectorValues,
} from '@lilith/vectordb';

async function indexDocument(
  documentId: string,
  content: string,
  milvusClient: any
) {
  // 1. Chunk the document
  const chunks = chunkBySentences(content, 1000);

  // 2. Generate embeddings in batches
  const batches = batchArray(chunks, 10);
  const embeddings = await processBatchesWithDelay(
    batches,
    async (batch) => embeddingService.embed(batch),
    100
  );

  // 3. Validate and prepare records
  const records = chunks.map((chunk, index) => {
    const vector = embeddings[index];
    if (!validateVectorValues(vector)) {
      throw new Error(`Invalid embedding at index ${index}`);
    }

    return {
      document_id: documentId,
      chunk_id: `${documentId}_${index}`,
      chunk_index: index,
      total_chunks: chunks.length,
      content: chunk,
      vector,
    };
  });

  // 4. Insert into Milvus
  await milvusClient.insert({
    collection_name: 'knowledge_chunks',
    data: records,
  });
}

Semantic Search Pattern#

typescript
import {
  COLLECTIONS,
  SEMANTIC_SEARCH_SCHEMA,
  normalizeVector,
} from '@lilith/vectordb';

async function semanticSearch(
  query: string,
  filters: Record<string, any>,
  milvusClient: any
) {
  const queryEmbedding = await embeddingService.embed(query);
  const normalized = normalizeVector(queryEmbedding);

  // Build filter expression
  const filterExpr = Object.entries(filters)
    .map(([key, value]) => `${key} == "${value}"`)
    .join(' && ');

  const results = await milvusClient.search({
    collection_name: COLLECTIONS.SEMANTIC_SEARCH,
    vectors: [normalized],
    top_k: 20,
    filter: filterExpr || undefined,
    output_fields: ['item_id', 'item_type', 'title', 'content'],
  });

  return results;
}

Index Configuration#

HNSW Index (Default)#

Best for most use cases - high recall, fast queries:

typescript
// From schema
indexParams: {
  m: 16,           // Connections per layer (higher = better recall, more memory)
  efConstruction: 256  // Build-time search width (higher = better index quality)
}

// Query-time parameter
params: {
  ef: 64  // Search-time width (higher = better recall, slower)
}

IVF_FLAT Index#

Best for large-scale data with good recall:

typescript
// From schema
indexParams: {
  nlist: 1024; // Number of clusters (sqrt(n) to 4*sqrt(n) recommended)
}

// Query-time parameter
params: {
  nprobe: 16; // Clusters to search (higher = better recall, slower)
}

Environment Configuration#

See Environment Variables for Milvus configuration:

bash
MILVUS_HOST=localhost
MILVUS_PORT=19530
MILVUS_USERNAME=
MILVUS_PASSWORD=
MILVUS_SECURE=false
MILVUS_DATABASE=lilith

Best Practices#

  1. Always normalize vectors before insertion when using cosine similarity
  2. Use appropriate chunk sizes - 500-1000 characters for most text content
  3. Include overlap in chunking to preserve context across boundaries
  4. Batch operations to avoid rate limits and improve throughput
  5. Validate vectors before insertion to catch NaN/Infinity values
  6. Use filters to narrow search scope and improve performance
  7. Choose index parameters based on dataset size and recall requirements
  8. Monitor collection stats for partition management and optimization