Canonical source: the canonical features document for this domain is
deep-dive/features.md, rendered in the docs center. This file is its operational companion (service topology, APIs, schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).
Knowledge Management and Research Platform
Sophia is the knowledge backbone of the Oshun ecosystem. It turns documents, research papers, scriptures, and any textual source into a searchable, graph-connected, citation-verified knowledge base. Whether you are a researcher building a bibliography, a worldbuilder grounding your fiction in historical fact, or a content creator who needs AI answers backed by real sources, Sophia provides the infrastructure.
Executive Summary#
Sophia provides a full knowledge lifecycle -- from document ingestion through semantic search to citation-verified AI answers. Key capabilities include:
- Document Ingestion -- Upload documents in 14 types (books, articles, theses, scriptures, encyclopedias, and more) through a 10-stage processing pipeline with entity extraction, summary generation, and quality scoring.
- Intelligent Chunking -- Eight chunking strategies (fixed, sliding window, paragraph, sentence, recursive, markdown, code, semantic) to split documents into searchable segments optimized for your content type.
- Semantic Search -- Hybrid search combining vector similarity and keyword matching (BM25), with cross-encoder reranking for high-precision results.
- RAG Pipeline -- Full Retrieval-Augmented Generation: ask a question, get an AI-generated answer with inline citations to the source documents.
- Knowledge Graph -- Build and query a graph of entities and relationships with BFS/DFS traversal, shortest path finding, subgraph extraction, and entity resolution.
- Vector Database Support -- Four interchangeable backends (in-memory, Qdrant, Pinecone, Milvus) with 10 pre-defined collection schemas.
- Citation Management -- Five citation styles (APA, MLA, Chicago, Harvard, IEEE) with claim detection, verification, and bibliography generation.
- Research Workflows -- Project management for research with hypothesis tracking, evidence linking, and methodology frameworks.
- Retrieval Evaluation -- Measure search quality with faithfulness, relevancy, precision/recall metrics, citation integrity checks, and groundedness scoring.
- Cultural Research -- Sensitivity analysis, localization management, accessibility checking, and a cultural knowledge database.
- Predictive Analytics -- Time series forecasting, schedule optimization, and content performance prediction.
- Visual Workbench -- A web-based UI with six pages for dashboard overview, content curation, document review, annotations, document management, and search.
1. Document Ingestion#
1.1 Supported Document Types#
| Type | Description |
|---|---|
| Book | Full-length publications, monographs |
| Article | Journal articles, magazine pieces |
| Thesis | Academic dissertations and theses |
| Manuscript | Unpublished or pre-print works |
| Scripture | Religious and philosophical sacred texts |
| Commentary | Scholarly commentary and exegesis |
| Essay | Short-form analytical writing |
| Letter | Correspondence and epistles |
| Speech | Transcribed speeches and lectures |
| Translation | Translated works with source attribution |
| Anthology | Curated collections of multiple works |
| Encyclopedia | Reference entries and knowledge compilations |
| Dictionary | Lexical definitions and etymological references |
| Other | Any document that does not fit the above types |
1.2 Ingestion Pipeline#
The ingestion service processes documents through a 10-stage pipeline:
- Upload / URL Fetch -- Accept the document from a file upload, URL, or raw text submission.
- Format Detection -- Identify the document format and select the appropriate parser.
- Text Extraction -- Extract raw text content, preserving structure where possible.
- Metadata Extraction -- Pull publication metadata (title, authors, date, publisher, DOI, ISBN).
- Chunking -- Split the document into segments using the configured chunking strategy.
- Embedding Generation -- Generate vector embeddings for each chunk using the configured provider.
- Entity Extraction -- Identify named entities (people, places, organizations, concepts) within the text.
- Summary Generation -- Produce a concise summary of the document.
- Quality Scoring -- Assign a quality score based on completeness, readability, and source credibility.
- Indexing -- Store chunks, embeddings, metadata, and entities in the database and vector store.
1.3 Ingestion Job Management#
- Configurable concurrency (default 5 simultaneous jobs)
- Retry support for failed stages
- Job status tracking through all 10 stages
- Progress reporting per stage
1.4 Ingestion via SDK#
sophia.ingestion.ingestUrlAndWait({ url: '...' })
sophia.ingestion.ingestDocument({ title: '...', content: '...' })
2. Intelligent Chunking#
2.1 Eight Chunking Strategies#
| Strategy | Best For | How It Works |
|---|---|---|
| Fixed Size | Uniform processing | Fixed character or token windows |
| Sliding Window | Overlapping context | Sliding window with configurable overlap |
| Paragraph | Prose documents | Splits on paragraph boundaries |
| Sentence | Fine-grained retrieval | Splits on sentence boundaries |
| Recursive | Mixed-format documents | Hierarchical splitting with ordered separators |
| Markdown | Technical documentation | Splits on Markdown headings and structure |
| Code | Source code and scripts | Language-aware splitting on function/class boundaries |
| Semantic | Maximum retrieval quality | Uses embeddings to find natural topic boundaries |
2.2 Chunking Configuration#
Every chunking strategy accepts:
- Chunk size -- Target size in characters or tokens (default 1000)
- Chunk overlap -- Amount of overlap between consecutive chunks (default 200)
- Token estimation -- Character-based or tokenizer-based counting
- Metadata preservation -- Carry document metadata into each chunk
2.3 Semantic Chunking#
The semantic chunker uses embedding similarity to detect natural topic boundaries:
- Split the document into sentences
- Generate embeddings for each sentence
- Compute similarity between consecutive sentences
- Detect breakpoints where similarity drops below a threshold
- Group sentences between breakpoints into chunks
This produces chunks that align with actual topic shifts rather than arbitrary length boundaries.
3. Semantic Search#
3.1 Hybrid Search#
Sophia's search combines two retrieval strategies in a single query:
- Dense vector search -- Finds semantically similar content using embedding similarity (cosine, dot product, or Euclidean distance)
- Sparse keyword search -- Finds exact term matches using BM25 scoring
You control the blend with configurable weights:
dense: 0.7, sparse: 0.3
3.2 Cross-Encoder Reranking#
After initial retrieval, results pass through a cross-encoder model that evaluates each result against the original query for improved relevance ranking. This catches results that are semantically relevant but were scored lower by the initial retrieval pass.
3.3 Search Builder#
The SDK provides a fluent search builder for constructing queries:
sophia.search
.query('enlightenment practices')
.hybrid({ dense: 0.7, sparse: 0.3 })
.topK(10)
.filter({ tradition: 'buddhism' })
.includeHighlights()
.execute()
3.4 Multi-Collection Search#
Search across 10 pre-defined collection schemas simultaneously:
| Collection | Purpose |
|---|---|
| Knowledge Chunks | Document chunk vectors for general search |
| Research Documents | Full document vectors for document retrieval |
| Citations | Citation vectors for source finding |
| Entities | Entity vectors for knowledge graph search |
| Wisdom Embeddings | Philosophical and spiritual content vectors |
| Content Embeddings | General content vectors |
| User Preferences | User preference vectors for personalization |
| Conversation Embeddings | Dialogue and conversation vectors |
| Semantic Search | Optimized for semantic similarity queries |
| Persona Embeddings | Character and personality vectors |
3.5 Context Retrieval#
Retrieve optimized context windows for LLM prompts:
- Configurable context size limits
- Chunk overlap handling (no duplicate content)
- Source tracking for citation generation
- Relevance-ordered context assembly
4. RAG Pipeline#
4.1 Ask a Question#
The RAG pipeline turns a natural language question into a sourced answer:
- Query Embedding -- Convert the question to a vector
- Retrieval -- Find the most relevant document chunks via hybrid search
- Reranking -- Reorder results using a cross-encoder
- Context Assembly -- Combine top chunks into a coherent context window
- Answer Generation -- Send the question and context to an LLM
- Citation Extraction -- Identify which sources support each claim in the answer
- Response -- Return the answer with inline citations linked to source documents
4.2 RAG Configuration#
- Choice of embedding provider (OpenAI, Cohere, Local)
- Choice of embedding model and dimensions
- Top-K retrieval count
- Reranking model selection
- LLM model selection for answer generation
- Citation style for the response
4.3 Quick Search vs. RAG#
| Feature | Quick Search | RAG Pipeline |
|---|---|---|
| Input | Search query | Natural language question |
| Output | Ranked results | Generated answer + citations |
| Speed | Faster | Slower (LLM generation) |
| Depth | Surface matching | Deep understanding |
| Citations | Source metadata | Inline with claims |
5. Knowledge Graph#
5.1 Entity Types#
The knowledge graph supports 15 entity types:
| Type | Description | Example |
|---|---|---|
| Tradition | Religious or philosophical tradition | Buddhism, Stoicism |
| Text | Significant written work | Dhammapada, Meditations |
| Concept | Abstract idea or principle | Dharma, Eudaimonia |
| Practice | Meditative or spiritual practice | Vipassana, Lectio Divina |
| Figure | Historical or mythological person | Siddhartha Gautama, Epictetus |
| Place | Significant location | Bodh Gaya, Lumbini |
| School | Philosophical or religious school | Zen, Theravada |
| Doctrine | Formal teaching or set of beliefs | Four Noble Truths |
| Event | Historical event | First Buddhist Council |
| Term | Technical vocabulary | Nibbana, Ataraxia |
| Symbol | Sacred or meaningful symbol | Dharma Wheel, Lotus |
| Ritual | Ceremonial practice | Puja, Sabbath |
| Artifact | Physical object of significance | Rosetta Stone, Dead Sea Scrolls |
| Organization | Formal group or institution | Sangha, Franciscan Order |
| Custom | User-defined entity type | Any domain-specific type |
5.2 Relationship Types#
23 relationship types for connecting entities:
| Category | Relationships |
|---|---|
| Knowledge | TAUGHT, STUDIED_WITH, INFLUENCED, REFERENCES, QUOTES, CITES |
| Authorship | WROTE, EDITED, TRANSLATED |
| Organization | FOUNDED, MEMBER_OF, BELONGS_TO |
| Location | LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN |
| Temporal | PRECEDED, SUCCEEDED, CONTEMPORARY_WITH |
| Semantic | RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM |
| Structural | PART_OF, PRACTICES, PRESCRIBES |
| Custom | CUSTOM (user-defined) |
5.3 Graph Operations#
| Operation | Description |
|---|---|
| Create entity | Add a new entity with type, properties, and confidence |
| Query entities | Find entities by type, name, properties, or text search |
| Create relation | Connect two entities with a typed, directed relationship |
| Get neighbors | Find all entities connected to a given entity |
| BFS traversal | Breadth-first traversal from a starting entity |
| DFS traversal | Depth-first traversal from a starting entity |
| Shortest path | Find the shortest path between two entities |
| All paths | Find all paths between two entities up to a maximum depth |
| Extract subgraph | Extract a connected subgraph around one or more entities |
| Entity resolution | Merge duplicate entities based on name and property similarity |
| Find duplicates | Identify potential duplicate entities for review |
5.4 Graph Query Builder#
The SDK provides a fluent graph query builder:
sophia.graph
.from(entityId)
.via('TAUGHT', 'INFLUENCED')
.toType('figure', 'concept')
.neighbors()
5.5 Entity Resolution#
Automatically detect and merge duplicate entities:
- Configurable similarity thresholds
- Name-based matching with alias support
- Property-based matching
- Merge tracking (which entities were combined)
- Confidence levels: VERY_LOW, LOW, MEDIUM, HIGH, VERY_HIGH
5.6 Graph Visualization and Export#
Export the knowledge graph for visualization:
| Format | Description |
|---|---|
| D3 | D3.js-compatible JSON (nodes + links) |
| DOT | Graphviz DOT language |
| GEXF | Graph Exchange XML Format (Gephi compatible) |
Graph statistics and centrality analysis:
- Node count, edge count, density
- Average degree, clustering coefficient
- Connected component detection
- Node centrality scores (degree, betweenness, closeness)
6. Vector Database Support#
6.1 Four Interchangeable Backends#
| Backend | Type | Best For | Port |
|---|---|---|---|
| Memory | In-memory | Development and testing | -- |
| Qdrant | Open source | Self-hosted production | 6333 |
| Pinecone | Managed | Cloud-first, zero-ops production | Cloud |
| Milvus | Open source | Large-scale, distributed deployment | 19530 |
6.2 Unified Interface#
All backends share the same interface:
connect()/disconnect()-- Lifecycle managementcreateCollection()/deleteCollection()-- Schema managementupsert()/delete()-- Vector CRUDsearch()-- Standard vector similarity searchhybridSearch()-- Combined vector + keyword search with RRF rerankingbatchUpsert()/batchDelete()-- Bulk operationsgetCollectionStats()-- Collection metricshealthCheck()-- Backend health verification
6.3 Collection Schemas#
10 pre-defined collection schemas with optimized index configurations:
- Dimension settings matched to embedding model output
- HNSW and IVF index parameters tuned per use case
- Metadata field definitions for filtering
- Distance metric selection (cosine, dot product, Euclidean)
6.4 Hybrid Search with RRF#
Reciprocal Rank Fusion (RRF) combines vector and keyword results:
- Execute vector search to get top-N semantically similar results
- Execute keyword search to get top-N term-matching results
- Apply RRF formula to merge and re-rank the two result sets
- Return the fused results with combined scores
7. Embedding Generation#
7.1 Three Embedding Providers#
| Provider | Models | Features |
|---|---|---|
| OpenAI | text-embedding-3-small, text-embedding-3-large | High quality, cloud-hosted |
| Cohere | embed-english-v3, embed-multilingual-v3 | Multilingual support |
| Local | Custom models | Self-hosted, no external API |
7.2 Embedding Service#
- Automatic provider selection based on configuration
- Batch embedding for bulk document processing
- Content-hash-based caching to avoid re-embedding identical text
- Configurable cache with TTL and size limits
- Environment-based configuration for zero-code setup
7.3 Vector Math Utilities#
Built-in similarity and vector operations:
- Cosine similarity, dot product, Euclidean distance, Manhattan distance
- Top-K nearest neighbor finding
- Pairwise similarity matrices
- Vector normalization, addition, subtraction, scaling
- Centroid computation
- Dimensionality reduction
8. Citation Management#
8.1 Five Citation Styles#
| Style | Format Example |
|---|---|
| APA | Smith, J. (2024). Title. Publisher. |
| MLA | Smith, John. Title. Publisher, 2024. |
| Chicago Notes | John Smith, Title (Place: Publisher, 2024), 42. |
| Harvard | Smith, J., 2024. Title. Place: Publisher. |
| IEEE | [1] J. Smith, "Title," Publisher, 2024. |
8.2 Claim Detection#
Automatically identify claims in text that need citations:
- Statistical assertions ("studies show that...")
- Attribution statements ("according to...")
- Factual claims ("the population of... is...")
- Quote detection
- Citation insertion point suggestions
8.3 Citation Verification#
Verify that citations accurately support their claims:
- Quote accuracy checking (does the source actually say this?)
- Entailment verification (does the source support the claim?)
- Attribution verification (is the author/source correctly identified?)
- Verification status tracking (verified, unverified, disputed, retracted)
8.4 Source Management#
- Source records with author, title, date, publisher, DOI, ISBN, URL
- Source authority scoring (peer review status, citation count, H-index)
- Source type classification (book, journal, conference, web, etc.)
8.5 Citation Graph#
Build and analyze a citation network:
- Nodes represent sources and documents
- Edges represent citation relationships
- Graph statistics (most-cited sources, citation chains)
- Visualization export to D3, DOT, and GEXF formats
8.6 Bibliography Generation#
Generate formatted bibliographies from citation collections:
- Filter by date range, source type, or tag
- Sort alphabetically, by date, or by citation order
- Deduplicate entries
- Export to multiple citation styles
9. Research Workflows#
9.1 Research Projects#
Create and manage research projects with:
- Project title, description, and methodology type
- Status tracking (planning, active, review, completed)
- Project timeline and milestones
9.2 Hypothesis Management#
- Register hypotheses within a research project
- Link evidence (documents, citations, search results) to hypotheses
- Track hypothesis status (proposed, testing, supported, refuted)
9.3 Research Notes#
- Attach notes to projects, documents, or entities
- Tag and categorize notes
- Link notes to evidence and hypotheses
9.4 Methodology Types#
Supported research methodology frameworks:
- Comparative analysis
- Historical analysis
- Textual criticism
- Hermeneutic analysis
- Phenomenological study
- Ethnographic research
10. Retrieval Quality Evaluation#
10.1 Retrieval Metrics#
| Metric | What It Measures |
|---|---|
| Faithfulness | Does the answer only contain information from the context? |
| Answer Relevancy | Does the answer address the question? |
| Context Precision | Are the retrieved contexts relevant to the question? |
| Context Recall | Are all necessary contexts retrieved? |
10.2 Citation Integrity Checking#
Verify the integrity of citations in generated answers:
- Quote accuracy -- Does the cited passage match the source?
- Entailment -- Does the source logically support the claim?
- Attribution -- Is the correct source credited?
- Integrity report -- Aggregated report across all citations in a response
10.3 Groundedness Scoring#
Measure how well an answer is grounded in source material:
- Semantic approach -- Embedding-based comparison between claims and evidence
- Lexical approach -- Term overlap and n-gram matching
- Claim-level scoring -- Individual scores for each claim in the answer
- Multi-dimensional -- Factual accuracy, source support, and consistency
10.4 Benchmark System#
Run standardized benchmarks against your RAG pipeline:
- Built-in benchmark datasets for common evaluation scenarios
- Custom benchmark creation with test cases and expected results
- Statistical analysis (mean, median, standard deviation, percentiles)
- Benchmark comparison across pipeline configurations
- Evaluation pipeline stages for systematic testing
11. Cultural Research#
11.1 Cultural Sensitivity Analysis#
Check content for cultural sensitivity issues:
- Risk level assessment (low, moderate, high, critical)
- Concern type classification (stereotyping, appropriation, sacred content, historical sensitivity, language, visual representation)
- Content location pinpointing (exact text spans with issues)
- Actionable suggestions for improvement
- Representation analysis across multiple dimensions
11.2 Cultural Database#
A structured knowledge base of cultural information:
| Category | Coverage |
|---|---|
| Cultural Profiles | Geographic regions, time periods, values, social structures |
| Belief Systems | Religious and philosophical traditions |
| Holidays | Cultural and religious celebrations |
| Symbols | Sacred and cultural symbols with meanings |
| Traditions | Cultural practices and customs |
| Taboos | Cultural sensitivities and restrictions |
| Visual Styles | Culturally specific aesthetics, clothing, color meanings |
| Narrative Patterns | Cultural storytelling traditions and archetypes |
| Mythological Figures | Major mythological and religious figures |
11.3 Localization Management#
Manage multilingual content:
- Localization project creation and tracking
- Translation unit management with status tracking
- Glossary management with per-locale translations
- Translation memory for consistent terminology
- Machine translation integration (configurable service)
- Linguistic Quality Assurance (LQA) with issue categorization
11.4 Accessibility#
Check content for accessibility compliance:
- WCAG level targeting (A, AA, AAA)
- Visual accessibility (colorblind modes, high contrast, font sizing)
- Audio accessibility (captions, descriptions, volume management)
- Motor accessibility (keyboard navigation, switch access, voice control)
- Cognitive accessibility (reading level, plain language, clear structure)
12. Predictive Analytics#
12.1 Revenue Forecasting#
- Time series analysis with multiple forecasting methods
- Seasonal pattern detection and factor modeling
- External factor incorporation (events, market conditions)
- Peak period identification
- Risk factor assessment
- Revenue recommendations based on historical trends
12.2 Schedule Optimization#
- Analyze historical streaming and publishing performance
- Competitor schedule awareness
- Optimal time slot recommendations
- Weekly schedule generation
- Improvement metrics projection
12.3 Content Performance Prediction#
- Predict engagement metrics for new content before publication
- Analyze content features (format, length, topic, style)
- Audience alignment scoring
- Market trend incorporation
- Performance factor breakdown (what drives success)
- Content comparison (A vs. B prediction)
13. Workbench (Web Application)#
13.1 Dashboard#
Overview of the knowledge base:
- Total documents, entities, and relationships
- Recent ingestion activity
- Search query volume and trends
- System health status
13.2 Documents Page#
Manage the document library:
- Browse all ingested documents with metadata
- Filter by type, date, author, quality score
- View document details, chunks, and extracted entities
- Delete or re-ingest documents
13.3 Search Page#
Interactive search interface:
- Full-text and semantic search with hybrid options
- Result previews with highlighted matches
- Faceted filtering (document type, date range, entity type)
- Search history
13.4 Curation Page#
Curate knowledge packs:
- Create themed collections of documents and entities
- Add/remove documents and entities from packs
- Version and publish knowledge packs
- Community features (starring, sharing)
13.5 Review Page#
Quality review workflow:
- Review queue for newly ingested documents
- Accept, reject, or request revision
- Quality score adjustment
- Annotation tools
13.6 Annotations Page#
Manage annotations across the knowledge base:
- View and create annotations on documents and entities
- Tag and categorize annotations
- Collaborative annotation workflows
14. TypeScript SDK#
14.1 Unified Client#
The SophiaClient provides a single entry point to all services:
sophia.search-- Search client with query buildersophia.ingestion-- Ingestion client for document submissionsophia.graph-- Knowledge graph client with graph query builder
14.2 Search Client#
quickSearch(query)-- Simple text searchask(question)-- RAG pipeline (answer + citations)query(text).hybrid().topK().filter().execute()-- Fluent buildersearch(request)-- Full search request with all options
14.3 Ingestion Client#
ingestDocument(doc)-- Submit a document for processingingestUrl(url)-- Ingest from a URLingestUrlAndWait(url)-- Ingest and wait for completiongetJob(id)-- Check ingestion job status
14.4 Knowledge Graph Client#
createEntity(type, name, properties)-- Add an entitycreateFigure(name, properties)-- Shorthand for figure entitiescreateRelation(from, to, type)-- Connect entitiesfrom(id).via(types).toType(types).neighbors()-- Fluent traversalfindShortestPath(from, to)-- Path findingresolveEntity(candidates)-- Duplicate detection and merging
14.5 Error Handling#
Comprehensive error hierarchy:
ApiError-- Base error with status code and messageNetworkError-- Connection failuresTimeoutError-- Request timeoutsNotFoundError-- Resource not found (404)ValidationError-- Invalid request data (400)UnauthorizedError/ForbiddenError-- Auth failuresRateLimitError-- Too many requests (429)
15. Cross-Domain Integration#
15.1 Events Published#
| Event | Trigger | Consumers |
|---|---|---|
sophia.document.ingested |
Document processing complete | Hathor (world lore grounding) |
sophia.document.updated |
Document metadata changed | Dependent search indices |
sophia.document.deleted |
Document removed | Index cleanup |
sophia.index.updated |
New vectors added to index | Search cache invalidation |
sophia.index.rebuilt |
Full index rebuild complete | Search service refresh |
sophia.search.performed |
Search query executed | Analytics |
sophia.entity.extracted |
New entity added to graph | Hathor knowledge integration |
sophia.relation.discovered |
New relationship found | Knowledge graph consumers |
sophia.citation.created |
Citation record created | Citation tracking |
15.2 Integration with Hathor#
Sophia serves as the research foundation for Hathor worldbuilding:
- Citation grounding -- World lore can be linked to real research sources
- Research-based generation -- World content generated from verified knowledge
- Lore fact-checking -- Validate world facts against the research corpus
- Entity discovery -- Suggest real-world entities relevant to world themes
Platform Availability#
| Feature | API | SDK | Web UI | CLI |
|---|---|---|---|---|
| Document ingestion | Yes | Yes | Yes | -- |
| Semantic search | Yes | Yes | Yes | -- |
| RAG pipeline | Yes | Yes | Yes | -- |
| Knowledge graph | Yes | Yes | Yes | -- |
| Citation management | -- | Yes | -- | -- |
| Research projects | -- | Yes | -- | -- |
| Evaluation and benchmarks | -- | Yes | -- | -- |
| Cultural research | -- | Yes | -- | -- |
| Predictive analytics | -- | Yes | -- | -- |
| Document curation | -- | Yes | Yes | -- |
| Annotations | -- | Yes | Yes | -- |
Feature Comparison: Search Modes#
| Capability | Quick Search | Hybrid Search | RAG Pipeline |
|---|---|---|---|
| Input | Text query | Text query | Natural language question |
| Vector matching | Yes | Yes | Yes |
| Keyword matching | -- | Yes (BM25) | Yes (BM25) |
| Weight control | -- | Yes | Yes |
| Cross-encoder rerank | -- | Optional | Yes |
| LLM generation | -- | -- | Yes |
| Inline citations | -- | -- | Yes |
| Speed | Fastest | Fast | Slower (LLM latency) |
| Best for | Quick lookup | Precise search | Comprehensive answers |
Feature Comparison: Chunking Strategies#
| Strategy | Overlap | Structure-Aware | Best Content Type | Setup Complexity |
|---|---|---|---|---|
| Fixed Size | No | No | Any | Low |
| Sliding Window | Yes | No | Any | Low |
| Paragraph | No | Yes | Prose, essays | Low |
| Sentence | No | Yes | Short-form, Q&A | Low |
| Recursive | Yes | Yes | Mixed format | Medium |
| Markdown | No | Yes | Technical docs, wikis | Medium |
| Code | No | Yes | Source code | Medium |
| Semantic | No | Yes | Any (highest quality) | High |
Feature Comparison: Vector Backends#
| Capability | Memory | Qdrant | Pinecone | Milvus |
|---|---|---|---|---|
| Persistence | No | Yes | Yes | Yes |
| Distributed | No | Yes | Yes | Yes |
| Self-hosted | N/A | Yes | No | Yes |
| Managed cloud | No | Yes | Yes | Yes |
| Hybrid search | Yes | Yes | Yes | Yes |
| Metadata filtering | Yes | Yes | Yes | Yes |
| Batch operations | Yes | Yes | Yes | Yes |
| Health checks | Yes | Yes | Yes | Yes |
| Best for | Dev/Test | Self-host | Cloud-first | Large-scale |