# Sophia -- Product Features

> **Canonical source:** the canonical features document for this domain is
> [`deep-dive/features.md`](./deep-dive/features.md), rendered in the docs
> center. This file is its operational companion (service topology, APIs,
> schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).

> Knowledge Management and Research Platform

Sophia is the knowledge backbone of the Oshun ecosystem. It turns documents,
research papers, scriptures, and any textual source into a searchable,
graph-connected, citation-verified knowledge base. Whether you are a researcher
building a bibliography, a worldbuilder grounding your fiction in historical
fact, or a content creator who needs AI answers backed by real sources, Sophia
provides the infrastructure.

---

## Executive Summary

Sophia provides a full knowledge lifecycle -- from document ingestion through
semantic search to citation-verified AI answers. Key capabilities include:

- **Document Ingestion** -- Upload documents in 14 types (books, articles,
  theses, scriptures, encyclopedias, and more) through a 10-stage processing
  pipeline with entity extraction, summary generation, and quality scoring.
- **Intelligent Chunking** -- Eight chunking strategies (fixed, sliding window,
  paragraph, sentence, recursive, markdown, code, semantic) to split documents
  into searchable segments optimized for your content type.
- **Semantic Search** -- Hybrid search combining vector similarity and keyword
  matching (BM25), with cross-encoder reranking for high-precision results.
- **RAG Pipeline** -- Full Retrieval-Augmented Generation: ask a question, get
  an AI-generated answer with inline citations to the source documents.
- **Knowledge Graph** -- Build and query a graph of entities and relationships
  with BFS/DFS traversal, shortest path finding, subgraph extraction, and entity
  resolution.
- **Vector Database Support** -- Four interchangeable backends (in-memory,
  Qdrant, Pinecone, Milvus) with 10 pre-defined collection schemas.
- **Citation Management** -- Five citation styles (APA, MLA, Chicago, Harvard,
  IEEE) with claim detection, verification, and bibliography generation.
- **Research Workflows** -- Project management for research with hypothesis
  tracking, evidence linking, and methodology frameworks.
- **Retrieval Evaluation** -- Measure search quality with faithfulness,
  relevancy, precision/recall metrics, citation integrity checks, and
  groundedness scoring.
- **Cultural Research** -- Sensitivity analysis, localization management,
  accessibility checking, and a cultural knowledge database.
- **Predictive Analytics** -- Time series forecasting, schedule optimization,
  and content performance prediction.
- **Visual Workbench** -- A web-based UI with six pages for dashboard overview,
  content curation, document review, annotations, document management, and
  search.

---

## 1. Document Ingestion

### 1.1 Supported Document Types

| Type         | Description                                     |
| ------------ | ----------------------------------------------- |
| Book         | Full-length publications, monographs            |
| Article      | Journal articles, magazine pieces               |
| Thesis       | Academic dissertations and theses               |
| Manuscript   | Unpublished or pre-print works                  |
| Scripture    | Religious and philosophical sacred texts        |
| Commentary   | Scholarly commentary and exegesis               |
| Essay        | Short-form analytical writing                   |
| Letter       | Correspondence and epistles                     |
| Speech       | Transcribed speeches and lectures               |
| Translation  | Translated works with source attribution        |
| Anthology    | Curated collections of multiple works           |
| Encyclopedia | Reference entries and knowledge compilations    |
| Dictionary   | Lexical definitions and etymological references |
| Other        | Any document that does not fit the above types  |

### 1.2 Ingestion Pipeline

The ingestion service processes documents through a 10-stage pipeline:

1. **Upload / URL Fetch** -- Accept the document from a file upload, URL, or raw
   text submission.
2. **Format Detection** -- Identify the document format and select the
   appropriate parser.
3. **Text Extraction** -- Extract raw text content, preserving structure where
   possible.
4. **Metadata Extraction** -- Pull publication metadata (title, authors, date,
   publisher, DOI, ISBN).
5. **Chunking** -- Split the document into segments using the configured
   chunking strategy.
6. **Embedding Generation** -- Generate vector embeddings for each chunk using
   the configured provider.
7. **Entity Extraction** -- Identify named entities (people, places,
   organizations, concepts) within the text.
8. **Summary Generation** -- Produce a concise summary of the document.
9. **Quality Scoring** -- Assign a quality score based on completeness,
   readability, and source credibility.
10. **Indexing** -- Store chunks, embeddings, metadata, and entities in the
    database and vector store.

### 1.3 Ingestion Job Management

- Configurable concurrency (default 5 simultaneous jobs)
- Retry support for failed stages
- Job status tracking through all 10 stages
- Progress reporting per stage

### 1.4 Ingestion via SDK

```
sophia.ingestion.ingestUrlAndWait({ url: '...' })
sophia.ingestion.ingestDocument({ title: '...', content: '...' })
```

---

## 2. Intelligent Chunking

### 2.1 Eight Chunking Strategies

| Strategy       | Best For                  | How It Works                                          |
| -------------- | ------------------------- | ----------------------------------------------------- |
| Fixed Size     | Uniform processing        | Fixed character or token windows                      |
| Sliding Window | Overlapping context       | Sliding window with configurable overlap              |
| Paragraph      | Prose documents           | Splits on paragraph boundaries                        |
| Sentence       | Fine-grained retrieval    | Splits on sentence boundaries                         |
| Recursive      | Mixed-format documents    | Hierarchical splitting with ordered separators        |
| Markdown       | Technical documentation   | Splits on Markdown headings and structure             |
| Code           | Source code and scripts   | Language-aware splitting on function/class boundaries |
| Semantic       | Maximum retrieval quality | Uses embeddings to find natural topic boundaries      |

### 2.2 Chunking Configuration

Every chunking strategy accepts:

- **Chunk size** -- Target size in characters or tokens (default 1000)
- **Chunk overlap** -- Amount of overlap between consecutive chunks
  (default 200)
- **Token estimation** -- Character-based or tokenizer-based counting
- **Metadata preservation** -- Carry document metadata into each chunk

### 2.3 Semantic Chunking

The semantic chunker uses embedding similarity to detect natural topic
boundaries:

1. Split the document into sentences
2. Generate embeddings for each sentence
3. Compute similarity between consecutive sentences
4. Detect breakpoints where similarity drops below a threshold
5. Group sentences between breakpoints into chunks

This produces chunks that align with actual topic shifts rather than arbitrary
length boundaries.

---

## 3. Semantic Search

### 3.1 Hybrid Search

Sophia's search combines two retrieval strategies in a single query:

- **Dense vector search** -- Finds semantically similar content using embedding
  similarity (cosine, dot product, or Euclidean distance)
- **Sparse keyword search** -- Finds exact term matches using BM25 scoring

You control the blend with configurable weights:

```
dense: 0.7, sparse: 0.3
```

### 3.2 Cross-Encoder Reranking

After initial retrieval, results pass through a cross-encoder model that
evaluates each result against the original query for improved relevance ranking.
This catches results that are semantically relevant but were scored lower by the
initial retrieval pass.

### 3.3 Search Builder

The SDK provides a fluent search builder for constructing queries:

```
sophia.search
  .query('enlightenment practices')
  .hybrid({ dense: 0.7, sparse: 0.3 })
  .topK(10)
  .filter({ tradition: 'buddhism' })
  .includeHighlights()
  .execute()
```

### 3.4 Multi-Collection Search

Search across 10 pre-defined collection schemas simultaneously:

| Collection              | Purpose                                      |
| ----------------------- | -------------------------------------------- |
| Knowledge Chunks        | Document chunk vectors for general search    |
| Research Documents      | Full document vectors for document retrieval |
| Citations               | Citation vectors for source finding          |
| Entities                | Entity vectors for knowledge graph search    |
| Wisdom Embeddings       | Philosophical and spiritual content vectors  |
| Content Embeddings      | General content vectors                      |
| User Preferences        | User preference vectors for personalization  |
| Conversation Embeddings | Dialogue and conversation vectors            |
| Semantic Search         | Optimized for semantic similarity queries    |
| Persona Embeddings      | Character and personality vectors            |

### 3.5 Context Retrieval

Retrieve optimized context windows for LLM prompts:

- Configurable context size limits
- Chunk overlap handling (no duplicate content)
- Source tracking for citation generation
- Relevance-ordered context assembly

---

## 4. RAG Pipeline

### 4.1 Ask a Question

The RAG pipeline turns a natural language question into a sourced answer:

1. **Query Embedding** -- Convert the question to a vector
2. **Retrieval** -- Find the most relevant document chunks via hybrid search
3. **Reranking** -- Reorder results using a cross-encoder
4. **Context Assembly** -- Combine top chunks into a coherent context window
5. **Answer Generation** -- Send the question and context to an LLM
6. **Citation Extraction** -- Identify which sources support each claim in the
   answer
7. **Response** -- Return the answer with inline citations linked to source
   documents

### 4.2 RAG Configuration

- Choice of embedding provider (OpenAI, Cohere, Local)
- Choice of embedding model and dimensions
- Top-K retrieval count
- Reranking model selection
- LLM model selection for answer generation
- Citation style for the response

### 4.3 Quick Search vs. RAG

| Feature   | Quick Search     | RAG Pipeline                 |
| --------- | ---------------- | ---------------------------- |
| Input     | Search query     | Natural language question    |
| Output    | Ranked results   | Generated answer + citations |
| Speed     | Faster           | Slower (LLM generation)      |
| Depth     | Surface matching | Deep understanding           |
| Citations | Source metadata  | Inline with claims           |

---

## 5. Knowledge Graph

### 5.1 Entity Types

The knowledge graph supports 15 entity types:

| Type         | Description                          | Example                         |
| ------------ | ------------------------------------ | ------------------------------- |
| Tradition    | Religious or philosophical tradition | Buddhism, Stoicism              |
| Text         | Significant written work             | Dhammapada, Meditations         |
| Concept      | Abstract idea or principle           | Dharma, Eudaimonia              |
| Practice     | Meditative or spiritual practice     | Vipassana, Lectio Divina        |
| Figure       | Historical or mythological person    | Siddhartha Gautama, Epictetus   |
| Place        | Significant location                 | Bodh Gaya, Lumbini              |
| School       | Philosophical or religious school    | Zen, Theravada                  |
| Doctrine     | Formal teaching or set of beliefs    | Four Noble Truths               |
| Event        | Historical event                     | First Buddhist Council          |
| Term         | Technical vocabulary                 | Nibbana, Ataraxia               |
| Symbol       | Sacred or meaningful symbol          | Dharma Wheel, Lotus             |
| Ritual       | Ceremonial practice                  | Puja, Sabbath                   |
| Artifact     | Physical object of significance      | Rosetta Stone, Dead Sea Scrolls |
| Organization | Formal group or institution          | Sangha, Franciscan Order        |
| Custom       | User-defined entity type             | Any domain-specific type        |

### 5.2 Relationship Types

23 relationship types for connecting entities:

| Category     | Relationships                                               |
| ------------ | ----------------------------------------------------------- |
| Knowledge    | TAUGHT, STUDIED_WITH, INFLUENCED, REFERENCES, QUOTES, CITES |
| Authorship   | WROTE, EDITED, TRANSLATED                                   |
| Organization | FOUNDED, MEMBER_OF, BELONGS_TO                              |
| Location     | LIVED_IN, BORN_IN, DIED_IN, LOCATED_IN                      |
| Temporal     | PRECEDED, SUCCEEDED, CONTEMPORARY_WITH                      |
| Semantic     | RELATED_TO, SIMILAR_TO, OPPOSITE_OF, DERIVED_FROM           |
| Structural   | PART_OF, PRACTICES, PRESCRIBES                              |
| Custom       | CUSTOM (user-defined)                                       |

### 5.3 Graph Operations

| Operation         | Description                                                    |
| ----------------- | -------------------------------------------------------------- |
| Create entity     | Add a new entity with type, properties, and confidence         |
| Query entities    | Find entities by type, name, properties, or text search        |
| Create relation   | Connect two entities with a typed, directed relationship       |
| Get neighbors     | Find all entities connected to a given entity                  |
| BFS traversal     | Breadth-first traversal from a starting entity                 |
| DFS traversal     | Depth-first traversal from a starting entity                   |
| Shortest path     | Find the shortest path between two entities                    |
| All paths         | Find all paths between two entities up to a maximum depth      |
| Extract subgraph  | Extract a connected subgraph around one or more entities       |
| Entity resolution | Merge duplicate entities based on name and property similarity |
| Find duplicates   | Identify potential duplicate entities for review               |

### 5.4 Graph Query Builder

The SDK provides a fluent graph query builder:

```
sophia.graph
  .from(entityId)
  .via('TAUGHT', 'INFLUENCED')
  .toType('figure', 'concept')
  .neighbors()
```

### 5.5 Entity Resolution

Automatically detect and merge duplicate entities:

- Configurable similarity thresholds
- Name-based matching with alias support
- Property-based matching
- Merge tracking (which entities were combined)
- Confidence levels: VERY_LOW, LOW, MEDIUM, HIGH, VERY_HIGH

### 5.6 Graph Visualization and Export

Export the knowledge graph for visualization:

| Format | Description                                  |
| ------ | -------------------------------------------- |
| D3     | D3.js-compatible JSON (nodes + links)        |
| DOT    | Graphviz DOT language                        |
| GEXF   | Graph Exchange XML Format (Gephi compatible) |

Graph statistics and centrality analysis:

- Node count, edge count, density
- Average degree, clustering coefficient
- Connected component detection
- Node centrality scores (degree, betweenness, closeness)

---

## 6. Vector Database Support

### 6.1 Four Interchangeable Backends

| Backend  | Type        | Best For                            | Port  |
| -------- | ----------- | ----------------------------------- | ----- |
| Memory   | In-memory   | Development and testing             | --    |
| Qdrant   | Open source | Self-hosted production              | 6333  |
| Pinecone | Managed     | Cloud-first, zero-ops production    | Cloud |
| Milvus   | Open source | Large-scale, distributed deployment | 19530 |

### 6.2 Unified Interface

All backends share the same interface:

- `connect()` / `disconnect()` -- Lifecycle management
- `createCollection()` / `deleteCollection()` -- Schema management
- `upsert()` / `delete()` -- Vector CRUD
- `search()` -- Standard vector similarity search
- `hybridSearch()` -- Combined vector + keyword search with RRF reranking
- `batchUpsert()` / `batchDelete()` -- Bulk operations
- `getCollectionStats()` -- Collection metrics
- `healthCheck()` -- Backend health verification

### 6.3 Collection Schemas

10 pre-defined collection schemas with optimized index configurations:

- Dimension settings matched to embedding model output
- HNSW and IVF index parameters tuned per use case
- Metadata field definitions for filtering
- Distance metric selection (cosine, dot product, Euclidean)

### 6.4 Hybrid Search with RRF

Reciprocal Rank Fusion (RRF) combines vector and keyword results:

1. Execute vector search to get top-N semantically similar results
2. Execute keyword search to get top-N term-matching results
3. Apply RRF formula to merge and re-rank the two result sets
4. Return the fused results with combined scores

---

## 7. Embedding Generation

### 7.1 Three Embedding Providers

| Provider | Models                                         | Features                     |
| -------- | ---------------------------------------------- | ---------------------------- |
| OpenAI   | text-embedding-3-small, text-embedding-3-large | High quality, cloud-hosted   |
| Cohere   | embed-english-v3, embed-multilingual-v3        | Multilingual support         |
| Local    | Custom models                                  | Self-hosted, no external API |

### 7.2 Embedding Service

- Automatic provider selection based on configuration
- Batch embedding for bulk document processing
- Content-hash-based caching to avoid re-embedding identical text
- Configurable cache with TTL and size limits
- Environment-based configuration for zero-code setup

### 7.3 Vector Math Utilities

Built-in similarity and vector operations:

- Cosine similarity, dot product, Euclidean distance, Manhattan distance
- Top-K nearest neighbor finding
- Pairwise similarity matrices
- Vector normalization, addition, subtraction, scaling
- Centroid computation
- Dimensionality reduction

---

## 8. Citation Management

### 8.1 Five Citation Styles

| Style         | Format Example                                  |
| ------------- | ----------------------------------------------- |
| APA           | Smith, J. (2024). Title. Publisher.             |
| MLA           | Smith, John. Title. Publisher, 2024.            |
| Chicago Notes | John Smith, Title (Place: Publisher, 2024), 42. |
| Harvard       | Smith, J., 2024. Title. Place: Publisher.       |
| IEEE          | [1] J. Smith, "Title," Publisher, 2024.         |

### 8.2 Claim Detection

Automatically identify claims in text that need citations:

- Statistical assertions ("studies show that...")
- Attribution statements ("according to...")
- Factual claims ("the population of... is...")
- Quote detection
- Citation insertion point suggestions

### 8.3 Citation Verification

Verify that citations accurately support their claims:

- Quote accuracy checking (does the source actually say this?)
- Entailment verification (does the source support the claim?)
- Attribution verification (is the author/source correctly identified?)
- Verification status tracking (verified, unverified, disputed, retracted)

### 8.4 Source Management

- Source records with author, title, date, publisher, DOI, ISBN, URL
- Source authority scoring (peer review status, citation count, H-index)
- Source type classification (book, journal, conference, web, etc.)

### 8.5 Citation Graph

Build and analyze a citation network:

- Nodes represent sources and documents
- Edges represent citation relationships
- Graph statistics (most-cited sources, citation chains)
- Visualization export to D3, DOT, and GEXF formats

### 8.6 Bibliography Generation

Generate formatted bibliographies from citation collections:

- Filter by date range, source type, or tag
- Sort alphabetically, by date, or by citation order
- Deduplicate entries
- Export to multiple citation styles

---

## 9. Research Workflows

### 9.1 Research Projects

Create and manage research projects with:

- Project title, description, and methodology type
- Status tracking (planning, active, review, completed)
- Project timeline and milestones

### 9.2 Hypothesis Management

- Register hypotheses within a research project
- Link evidence (documents, citations, search results) to hypotheses
- Track hypothesis status (proposed, testing, supported, refuted)

### 9.3 Research Notes

- Attach notes to projects, documents, or entities
- Tag and categorize notes
- Link notes to evidence and hypotheses

### 9.4 Methodology Types

Supported research methodology frameworks:

- Comparative analysis
- Historical analysis
- Textual criticism
- Hermeneutic analysis
- Phenomenological study
- Ethnographic research

---

## 10. Retrieval Quality Evaluation

### 10.1 Retrieval Metrics

| Metric            | What It Measures                                           |
| ----------------- | ---------------------------------------------------------- |
| Faithfulness      | Does the answer only contain information from the context? |
| Answer Relevancy  | Does the answer address the question?                      |
| Context Precision | Are the retrieved contexts relevant to the question?       |
| Context Recall    | Are all necessary contexts retrieved?                      |

### 10.2 Citation Integrity Checking

Verify the integrity of citations in generated answers:

- **Quote accuracy** -- Does the cited passage match the source?
- **Entailment** -- Does the source logically support the claim?
- **Attribution** -- Is the correct source credited?
- **Integrity report** -- Aggregated report across all citations in a response

### 10.3 Groundedness Scoring

Measure how well an answer is grounded in source material:

- **Semantic approach** -- Embedding-based comparison between claims and
  evidence
- **Lexical approach** -- Term overlap and n-gram matching
- **Claim-level scoring** -- Individual scores for each claim in the answer
- **Multi-dimensional** -- Factual accuracy, source support, and consistency

### 10.4 Benchmark System

Run standardized benchmarks against your RAG pipeline:

- Built-in benchmark datasets for common evaluation scenarios
- Custom benchmark creation with test cases and expected results
- Statistical analysis (mean, median, standard deviation, percentiles)
- Benchmark comparison across pipeline configurations
- Evaluation pipeline stages for systematic testing

---

## 11. Cultural Research

### 11.1 Cultural Sensitivity Analysis

Check content for cultural sensitivity issues:

- Risk level assessment (low, moderate, high, critical)
- Concern type classification (stereotyping, appropriation, sacred content,
  historical sensitivity, language, visual representation)
- Content location pinpointing (exact text spans with issues)
- Actionable suggestions for improvement
- Representation analysis across multiple dimensions

### 11.2 Cultural Database

A structured knowledge base of cultural information:

| Category             | Coverage                                                    |
| -------------------- | ----------------------------------------------------------- |
| Cultural Profiles    | Geographic regions, time periods, values, social structures |
| Belief Systems       | Religious and philosophical traditions                      |
| Holidays             | Cultural and religious celebrations                         |
| Symbols              | Sacred and cultural symbols with meanings                   |
| Traditions           | Cultural practices and customs                              |
| Taboos               | Cultural sensitivities and restrictions                     |
| Visual Styles        | Culturally specific aesthetics, clothing, color meanings    |
| Narrative Patterns   | Cultural storytelling traditions and archetypes             |
| Mythological Figures | Major mythological and religious figures                    |

### 11.3 Localization Management

Manage multilingual content:

- Localization project creation and tracking
- Translation unit management with status tracking
- Glossary management with per-locale translations
- Translation memory for consistent terminology
- Machine translation integration (configurable service)
- Linguistic Quality Assurance (LQA) with issue categorization

### 11.4 Accessibility

Check content for accessibility compliance:

- WCAG level targeting (A, AA, AAA)
- Visual accessibility (colorblind modes, high contrast, font sizing)
- Audio accessibility (captions, descriptions, volume management)
- Motor accessibility (keyboard navigation, switch access, voice control)
- Cognitive accessibility (reading level, plain language, clear structure)

---

## 12. Predictive Analytics

### 12.1 Revenue Forecasting

- Time series analysis with multiple forecasting methods
- Seasonal pattern detection and factor modeling
- External factor incorporation (events, market conditions)
- Peak period identification
- Risk factor assessment
- Revenue recommendations based on historical trends

### 12.2 Schedule Optimization

- Analyze historical streaming and publishing performance
- Competitor schedule awareness
- Optimal time slot recommendations
- Weekly schedule generation
- Improvement metrics projection

### 12.3 Content Performance Prediction

- Predict engagement metrics for new content before publication
- Analyze content features (format, length, topic, style)
- Audience alignment scoring
- Market trend incorporation
- Performance factor breakdown (what drives success)
- Content comparison (A vs. B prediction)

---

## 13. Workbench (Web Application)

### 13.1 Dashboard

Overview of the knowledge base:

- Total documents, entities, and relationships
- Recent ingestion activity
- Search query volume and trends
- System health status

### 13.2 Documents Page

Manage the document library:

- Browse all ingested documents with metadata
- Filter by type, date, author, quality score
- View document details, chunks, and extracted entities
- Delete or re-ingest documents

### 13.3 Search Page

Interactive search interface:

- Full-text and semantic search with hybrid options
- Result previews with highlighted matches
- Faceted filtering (document type, date range, entity type)
- Search history

### 13.4 Curation Page

Curate knowledge packs:

- Create themed collections of documents and entities
- Add/remove documents and entities from packs
- Version and publish knowledge packs
- Community features (starring, sharing)

### 13.5 Review Page

Quality review workflow:

- Review queue for newly ingested documents
- Accept, reject, or request revision
- Quality score adjustment
- Annotation tools

### 13.6 Annotations Page

Manage annotations across the knowledge base:

- View and create annotations on documents and entities
- Tag and categorize annotations
- Collaborative annotation workflows

---

## 14. TypeScript SDK

### 14.1 Unified Client

The `SophiaClient` provides a single entry point to all services:

- `sophia.search` -- Search client with query builder
- `sophia.ingestion` -- Ingestion client for document submission
- `sophia.graph` -- Knowledge graph client with graph query builder

### 14.2 Search Client

- `quickSearch(query)` -- Simple text search
- `ask(question)` -- RAG pipeline (answer + citations)
- `query(text).hybrid().topK().filter().execute()` -- Fluent builder
- `search(request)` -- Full search request with all options

### 14.3 Ingestion Client

- `ingestDocument(doc)` -- Submit a document for processing
- `ingestUrl(url)` -- Ingest from a URL
- `ingestUrlAndWait(url)` -- Ingest and wait for completion
- `getJob(id)` -- Check ingestion job status

### 14.4 Knowledge Graph Client

- `createEntity(type, name, properties)` -- Add an entity
- `createFigure(name, properties)` -- Shorthand for figure entities
- `createRelation(from, to, type)` -- Connect entities
- `from(id).via(types).toType(types).neighbors()` -- Fluent traversal
- `findShortestPath(from, to)` -- Path finding
- `resolveEntity(candidates)` -- Duplicate detection and merging

### 14.5 Error Handling

Comprehensive error hierarchy:

- `ApiError` -- Base error with status code and message
- `NetworkError` -- Connection failures
- `TimeoutError` -- Request timeouts
- `NotFoundError` -- Resource not found (404)
- `ValidationError` -- Invalid request data (400)
- `UnauthorizedError` / `ForbiddenError` -- Auth failures
- `RateLimitError` -- Too many requests (429)

---

## 15. Cross-Domain Integration

### 15.1 Events Published

| Event                        | Trigger                      | Consumers                     |
| ---------------------------- | ---------------------------- | ----------------------------- |
| `sophia.document.ingested`   | Document processing complete | Hathor (world lore grounding) |
| `sophia.document.updated`    | Document metadata changed    | Dependent search indices      |
| `sophia.document.deleted`    | Document removed             | Index cleanup                 |
| `sophia.index.updated`       | New vectors added to index   | Search cache invalidation     |
| `sophia.index.rebuilt`       | Full index rebuild complete  | Search service refresh        |
| `sophia.search.performed`    | Search query executed        | Analytics                     |
| `sophia.entity.extracted`    | New entity added to graph    | Hathor knowledge integration  |
| `sophia.relation.discovered` | New relationship found       | Knowledge graph consumers     |
| `sophia.citation.created`    | Citation record created      | Citation tracking             |

### 15.2 Integration with Hathor

Sophia serves as the research foundation for Hathor worldbuilding:

- **Citation grounding** -- World lore can be linked to real research sources
- **Research-based generation** -- World content generated from verified
  knowledge
- **Lore fact-checking** -- Validate world facts against the research corpus
- **Entity discovery** -- Suggest real-world entities relevant to world themes

---

## Platform Availability

| Feature                   | API | SDK | Web UI | CLI |
| ------------------------- | --- | --- | ------ | --- |
| Document ingestion        | Yes | Yes | Yes    | --  |
| Semantic search           | Yes | Yes | Yes    | --  |
| RAG pipeline              | Yes | Yes | Yes    | --  |
| Knowledge graph           | Yes | Yes | Yes    | --  |
| Citation management       | --  | Yes | --     | --  |
| Research projects         | --  | Yes | --     | --  |
| Evaluation and benchmarks | --  | Yes | --     | --  |
| Cultural research         | --  | Yes | --     | --  |
| Predictive analytics      | --  | Yes | --     | --  |
| Document curation         | --  | Yes | Yes    | --  |
| Annotations               | --  | Yes | Yes    | --  |

---

## Feature Comparison: Search Modes

| Capability           | Quick Search | Hybrid Search  | RAG Pipeline              |
| -------------------- | ------------ | -------------- | ------------------------- |
| Input                | Text query   | Text query     | Natural language question |
| Vector matching      | Yes          | Yes            | Yes                       |
| Keyword matching     | --           | Yes (BM25)     | Yes (BM25)                |
| Weight control       | --           | Yes            | Yes                       |
| Cross-encoder rerank | --           | Optional       | Yes                       |
| LLM generation       | --           | --             | Yes                       |
| Inline citations     | --           | --             | Yes                       |
| Speed                | Fastest      | Fast           | Slower (LLM latency)      |
| Best for             | Quick lookup | Precise search | Comprehensive answers     |

---

## Feature Comparison: Chunking Strategies

| Strategy       | Overlap | Structure-Aware | Best Content Type     | Setup Complexity |
| -------------- | ------- | --------------- | --------------------- | ---------------- |
| Fixed Size     | No      | No              | Any                   | Low              |
| Sliding Window | Yes     | No              | Any                   | Low              |
| Paragraph      | No      | Yes             | Prose, essays         | Low              |
| Sentence       | No      | Yes             | Short-form, Q&A       | Low              |
| Recursive      | Yes     | Yes             | Mixed format          | Medium           |
| Markdown       | No      | Yes             | Technical docs, wikis | Medium           |
| Code           | No      | Yes             | Source code           | Medium           |
| Semantic       | No      | Yes             | Any (highest quality) | High             |

---

## Feature Comparison: Vector Backends

| Capability         | Memory   | Qdrant    | Pinecone    | Milvus      |
| ------------------ | -------- | --------- | ----------- | ----------- |
| Persistence        | No       | Yes       | Yes         | Yes         |
| Distributed        | No       | Yes       | Yes         | Yes         |
| Self-hosted        | N/A      | Yes       | No          | Yes         |
| Managed cloud      | No       | Yes       | Yes         | Yes         |
| Hybrid search      | Yes      | Yes       | Yes         | Yes         |
| Metadata filtering | Yes      | Yes       | Yes         | Yes         |
| Batch operations   | Yes      | Yes       | Yes         | Yes         |
| Health checks      | Yes      | Yes       | Yes         | Yes         |
| Best for           | Dev/Test | Self-host | Cloud-first | Large-scale |
