# Sophia Domain — Architecture > Knowledge Management and Research Platform --- Sophia is the domain responsible for turning raw documents into a searchable, graph-connected, citation-verified knowledge base. Where most domains in the Oshun monorepo _create_ content, Sophia _grounds_ it: it ingests research papers, scriptures, essays, and any textual source; indexes them for hybrid semantic search; builds a knowledge graph from the entities and relationships it finds; and evaluates how well AI-generated answers are supported by the retrieved evidence. The real-world problem Sophia solves is **hallucination and attribution**. Without a system like Sophia, AI responses can sound authoritative while citing nothing real. Sophia's RAG (Retrieval-Augmented Generation) pipeline ensures every claim in a generated answer can be traced back to a specific chunk of a specific document, with a formatted citation in the style the reader expects. Sophia is consumed by Lilith (to ground spiritual-guidance AI conversations), Hathor (to validate worldbuilding lore against historical research), and any other domain that needs reliable, attributed knowledge retrieval. Internally, Sophia is organized around **4 applications** (`apps/sophia/`) and **27 libraries** (`libs/sophia/`) that together form a complete research knowledge pipeline. --- ## 1. System Overview ### Core Responsibilities - **Document Ingestion** — Multi-format document intake with configurable chunking strategies, entity extraction, and summary generation - **Semantic Search** — Hybrid search combining vector similarity and keyword matching with cross-encoder reranking - **RAG Pipeline** — Retrieval-augmented generation with citation tracking and groundedness scoring - **Knowledge Graph** — Entity extraction, relationship mapping, entity resolution, and graph traversal - **Evaluation** — Retrieval quality metrics, citation integrity checking, and benchmark management - **Research Management** — Research project tracking, hypothesis management, and citation formatting --- ## 2. Service Architecture ### 2.1 Applications The four applications divide Sophia's responsibilities cleanly: the **Search API** handles queries; the **Ingestion** worker handles document intake; the **Knowledge Graph** service manages the entity graph; and the **Workbench** provides a human operator interface. ``` External Clients | +-------------+-------------+ | | +------+------+ +--------+--------+ | search-api | | knowledge-graph| | Port 3000 | | Port 3001 | | | | | | SearchAPI | | KnowledgeGraph | | Service | | Service | | | | | | - hybrid | | - EntityService | | search | | - RelationSvc | | - RAG | | - GraphService | | - reranking | | - EntityResolver| +------+-------+ +--------+--------+ | | v v +------+------------------------------+--------+ | @sophia/database | | PostgreSQL (sophia schema) | +------+------------------------------+--------+ | +------+------+ | ingestion | (pipeline worker, no fixed port) | | | pipeline: | | fetch → | | extract → | | enrich → | | chunk | | (+ optional | | kalika KG) | +------+------+ | +------+------+ | workbench | (React SPA, Vite dev server) | | | Dashboard | | Documents | | Search | | Curation | | Review | | Annotations | +-------------+ ``` ### 2.2 Application Details #### Search API (`apps/sophia/search-api`, Port 3000) The primary search service providing semantic search with RAG context generation. Implemented as a `SearchAPIService` class with framework-agnostic request handlers via `createRequestHandlers(service)`. By keeping the service logic separate from the HTTP layer, the same handlers can be mounted on Hono, Express, or any other framework without modification. - **Dependencies:** `@oshun/logging`, `RetrievalService`, `CitationService`, `CrossEncoderReranker` - **Dev mode:** Uses `MockEmbeddingProvider`, `MockVectorStore`, `MockTextIndex` #### Ingestion Pipeline (`apps/sophia/ingestion`) The ingestion worker (`IngestionPipeline`) accepts documents or URLs and drives them through a multi-stage processing pipeline. It has no fixed HTTP port because it is designed to run as a background worker, not a request handler — it processes jobs from a queue and emits domain events when done. - **Modes:** CLI (single document argument) or service mode; no fixed HTTP port - **Pipeline stages:** Fetch → Extract → Enrich → Chunk, with an optional Kalika knowledge-graph sync stage afterward for math/physics documents - **Concurrency:** `maxConcurrentJobs` option, default 5 - **Pipeline events:** `job:created`, `job:started`, `job:progress`, `job:completed`, `job:failed`, `stage:started`, `stage:completed`, `stage:failed` - **Domain events:** publishes `sophia.document.ingested` and per-entity `sophia.entity.extracted` via `@sophia/event-publisher` #### Knowledge Graph (`apps/sophia/knowledge-graph`, Port 3001) A standalone knowledge graph service (`KnowledgeGraphService`) that manages the entity and relationship graph extracted from ingested documents. The service runs as its own process (separate from the search API) so it can be scaled independently — knowledge graph queries tend to be computationally different from search queries. It is backed by an in-memory `MemoryGraphStore` and exposes a full REST API covering entity and relationship CRUD, graph traversal and path finding, entity resolution, and a curator review/approval subsystem. - **Internal services:** `EntityService`, `RelationService`, `GraphService`, `EntityResolver` (when `enableResolution` is true), and `CuratorService` #### Workbench (`apps/sophia/workbench`) A React SPA for document management, search testing, knowledge curation, annotation workflows, and operator tooling. The Workbench is aimed at knowledge workers — researchers, curators, and operators — who need to interact with the Sophia knowledge base without writing code. - **Technology:** Vite, Tailwind CSS, React Router - **Pages/routes:** Dashboard (`/`), Curation (`/curation`), Review (`/review`), Annotations (`/annotations`), Documents (`/documents`), Search (`/search`), Operator Workbench (`/operator`) - **API Layer:** `AuthService`, `DashboardService`, `DocumentsService`, `AnnotationsService`, `SearchService`, `ReviewService`, `OperatorWorkbenchService` - **Pattern:** Singleton service instances with `getXService()` / `resetXService()` for testing --- ## 3. Library Architecture ### 3.1 Layer Overview There are 27 libraries under `libs/sophia/`. They are organized into four layers: contracts (validation schemas), data (database and vector store), domain (business logic), and client (SDK). The domain layer is by far the largest, covering every subdomain from ingestion to evaluation. ``` Contracts layer: @sophia/schemas — Zod schemas with branded ID types Data layer: @sophia/database — Prisma client and schema (21 models) @sophia/vectordb — Vector store abstraction (4 backends) Domain layer: @sophia/indexing — Chunking, embedding, index building @sophia/ingestion — Ingestion connectors, extraction, enrichment @sophia/knowledge-graph — Entity-relationship graph construction/traversal @sophia/evaluation — RAG quality metrics and benchmarks @sophia/theory — Citation management and research workflows @sophia/agents — Research-agent pipeline (decomposition, verification) @sophia/research-engine — Multi-source research query execution @sophia/semantic-search — Educational semantic search with caching @sophia/knowledge-base — Educational entity/curriculum extraction @sophia/trend-curriculum — Curriculum trend signal detection @sophia/citation-analysis / @sophia/citation-graph — Citation quality and network analysis @sophia/credibility / @sophia/verification — Source credibility and fact-checking @sophia/document-parser — Multi-format parsing with structure extraction @sophia/crawling — Web crawling with rate limiting @sophia/corpus — Educational images and diagram templates @sophia/embeddings — Image / multimodal / dense-retriever embeddings @sophia/cultural-research — Cultural sensitivity, localization, accessibility @sophia/predictions — Time-series forecasting and content prediction @sophia/concordia-knowledge — Precedent/template lookup for Concordia cases Integration layer: @sophia/event-publisher — Typed event publishing (9 event types) Client layer: @sophia/client — Unified TypeScript SDK @oshun/sophia-client — Oshun-scoped compatibility re-export of the SDK ``` ### 3.2 Library Descriptions #### `@sophia/client` — TypeScript SDK The unified client provides type-safe access to all Sophia services from any TypeScript consumer in the monorepo. It wraps the HTTP APIs of the search-api and knowledge-graph services behind a clean object interface with sub-clients for each capability. Exports: `SophiaClient` (unified), `SearchClient`, `IngestionClient`, `KnowledgeGraphClient`, `SearchQueryBuilder`, `GraphQueryBuilder`, branded ID types. ```typescript const sophia = new SophiaClient({ searchUrl: 'http://localhost:3000', knowledgeGraphUrl: 'http://localhost:3001', }); ``` #### `@sophia/database` — Data Access Layer The single source of truth for Sophia's persistent state. All 21 PostgreSQL models are defined here as a Prisma schema, and the generated client is what every other Sophia library uses to read and write data. Exports `getSophiaClient()`, `createSophiaClient()`, `disconnectSophiaClient()`, and uppercase string-union enum types. Schema in `libs/sophia/database/prisma/schema.prisma` (1361 lines). #### `@sophia/schemas` — Validation Contracts Zod schemas with branded IDs for documents, citations, claims, knowledge packs, entities, and relations. This library forms the contract layer shared across apps and libraries — it defines _what shape_ data must be in, independently of how it is stored. Note that this layer uses lowercase kebab-cased enum values (`'chicago-notes'`) while the database layer uses uppercase values (`CHICAGO`); both are documented in detail in the specifications. #### `@sophia/indexing` — Chunking and Embedding The heart of the RAG pipeline's preprocessing stage. This library splits documents into retrievable chunks and generates vector embeddings for each chunk. It implements eight chunking strategies (`FixedSizeChunker`, `SlidingWindowChunker`, `ParagraphChunker`, `SentenceChunker`, `RecursiveChunker`, `MarkdownChunker`, `CodeChunker`, `SemanticChunker`), three embedding providers (OpenAI, Cohere, Local), `IndexBuilder`, vector math utilities, and graph export. #### `@sophia/vectordb` — Vector Store Abstraction Rather than coupling the rest of the system to a specific vector database, this library provides a unified interface that four different backends all implement. Switching from an in-memory development store to a production Qdrant cluster requires only a configuration change. The abstraction covers four backends (Memory, Qdrant, Pinecone, Milvus) with 10 pre-defined collection schemas, hybrid search with RRF, batch utilities, and health checks. Key files: `qdrant-store.ts`, `milvus-store.ts`, `pinecone-store.ts`, `memory-store.ts`, `src/utils/batch.ts`, `src/utils/similarity.ts`, `src/index/builder.ts`. #### `@sophia/evaluation` — Quality Assessment Measures how well the RAG pipeline is actually working. Without evaluation, improvements to chunking or retrieval are guesswork. This library provides `RetrievalEvaluator`, `CitationChecker`, `GroundednessScorer`, and `BenchmarkRunner`. It supports faithfulness, relevancy, precision/recall, citation integrity, and groundedness scoring at both semantic and lexical levels. #### `@sophia/theory` — Research Framework Provides scholarly citation management and research project tracking. `CitationManager` supports 12 citation styles — see the `CitationStyle` union in `theory/src/types.ts` — with claim detection, verification, and bibliography generation. `ResearchManager` handles project, hypothesis, and evidence tracking. #### `@sophia/cultural-research` — Cultural Intelligence Enables Sophia to assess whether knowledge content is culturally sensitive and accessible before it is surfaced to users. Contains `CulturalConsultant` (sensitivity analysis), `LocalizationManager` (translation workflows), `AccessibilityManager` (WCAG checking), and `CulturalDatabase` (structured cultural knowledge). #### `@sophia/event-publisher` — Event Publishing `SophiaEventPublisher` with 9 typed event methods. Wraps `@oshun/event-bus` (Kafka). Follows a singleton pattern so all Sophia services share a single publisher instance. #### `@sophia/predictions` — Predictive Analytics Time series forecasting, schedule optimization, and content performance prediction. Also tagged `scope:shared` for reuse across domains outside Sophia. --- ## 4. Data Flow The four primary data flows below show how data moves through Sophia for the most important operations. Each flow is independent — ingestion populates the index, search queries it, RAG extends search with generation, and the knowledge graph uses the same entity data extracted during ingestion. ### 4.1 Document Ingestion Flow Documents enter through the ingestion pipeline and are processed in sequence. Each stage adds information to the document record until the document is fully indexed and searchable. ``` Client / URL | v Ingestion Pipeline (apps/sophia/ingestion) | +-- [Fetch] ────────> Source connector +-- [Extract] ──────> Text and metadata extraction, OCR +-- [Enrich] ───────> Entity extraction, summarization, keyword extraction +-- [Chunk] ────────> @sophia/indexing (selected chunking strategy) +-- [Index] ────────> Embeddings via @sophia/indexing | Vector upsert via @sophia/vectordb | Document + chunk save via @sophia/database | +-- Emit: sophia.document.ingested +-- Emit: sophia.entity.extracted (per entity) +-- Emit: sophia.index.updated ``` ### 4.2 Search Flow A search query is handled entirely by the Search API. The key step is the **RRF fusion**: combining dense (embedding similarity) and sparse (keyword) results into a single ranked list that is better than either alone. ``` Client Query | v Search API (apps/sophia/search-api) | +-- RetrievalService | | | +-- Dense search ──> @sophia/vectordb (vector similarity) | +-- Sparse search ─> PostgreSQL full-text (tsvector) | +-- RRF fusion ────> Merge and re-rank results | +-- CrossEncoderReranker (optional) | | | +-- Rerank top-K results | +-- Response assembly | | | +-- Metadata enrichment from @sophia/database | +-- Highlight generation | +-- Context window assembly | +-- Emit: sophia.search.performed ``` ### 4.3 RAG Flow RAG (Retrieval-Augmented Generation) extends search by feeding the retrieved chunks into an LLM and then verifying which source documents actually support each claim in the generated answer. This is what allows Sophia to produce answers that are both fluent and fully attributable. ``` Natural language question | v Search API /rag | +-- Query embedding (OpenAI / Cohere / Local) +-- Hybrid retrieval (RetrievalService) +-- Cross-encoder reranking +-- Context assembly (top-K chunks) +-- LLM generation (with system prompt) +-- CitationService | | | +-- Extract citations from generated answer | +-- Verify against source chunks | +-- Format in requested citation style | +-- Return: { answer, citations, sources } ``` ### 4.4 Knowledge Graph Flow The knowledge graph is populated as a side effect of document ingestion and is then served by the Knowledge Graph service as a separate queryable graph. The separation matters: ingestion is batch-oriented and write-heavy, while graph queries are read-heavy and interactive. ``` Ingestion pipeline | v Entity Extraction (enrichment stage) | +-- Entities saved via @sophia/database +-- EntityMentions saved (document + chunk position) | v Knowledge Graph Service (apps/sophia/knowledge-graph) | +-- EntityService — CRUD for graph nodes +-- RelationService — CRUD for directed edges +-- GraphService | | | +-- BFS / DFS traversal | +-- Shortest path (Dijkstra-like) | +-- All-paths enumeration | +-- Subgraph extraction | +-- EntityResolver | +-- Name similarity matching +-- Property-based matching +-- Merge with canonicalId tracking ``` --- ## 5. Key Design Patterns Sophia uses a small set of patterns consistently across all its libraries and applications. Understanding these patterns makes it much easier to navigate the codebase. ### 5.1 Strategy Pattern — Chunking All chunking strategies implement a common `ChunkingStrategy` interface (`chunk(text, options): Promise`). The ingestion pipeline selects the appropriate strategy by name at runtime. Adding a new chunking approach requires only implementing the interface and registering it in the factory — no other code needs to change. ### 5.2 Adapter Pattern — Vector Stores All four vector store backends (`MemoryVectorStore`, `QdrantVectorStore`, `PineconeVectorStore`, `MilvusVectorStore`) implement a unified `VectorStore` interface. The rest of the system never calls backend-specific APIs directly, so swapping the vector store for a different product is a configuration-only change. ### 5.3 Builder Pattern — SDK Queries Both `SearchQueryBuilder` and `GraphQueryBuilder` provide fluent chainable APIs. Builders produce typed request objects that are submitted to the respective service clients. This makes complex multi-parameter queries readable and refactorable. ### 5.4 Singleton Services with Test Reset Workbench API services follow a singleton pattern with a `getXService()` factory and a `resetXService()` function for test isolation. The reset function tears down shared state between test cases, preventing one test's side effects from leaking into another. ### 5.5 Framework-Agnostic Request Handlers The `SearchAPIService` exposes its operations as plain async methods rather than being tightly coupled to an HTTP framework. `createRequestHandlers(service)` wraps these for mounting on Hono, Express, or any other framework. This means the core service logic can be unit-tested without spinning up an HTTP server. --- ## 6. Technology Stack The table below summarizes the technology choices for each layer, followed by the rationale for the most consequential decisions. | Layer | Technology | | ------------- | ---------------------------------------------------------------- | | Language | TypeScript (ESM modules) | | API Framework | Standalone service classes over the Node `http` module | | Database | PostgreSQL via Prisma ORM (21 models, 1361-line schema) | | Vector Stores | Qdrant (primary), Pinecone (cloud), Milvus (distributed), Memory | | Embeddings | OpenAI text-embedding-3-\*, Cohere embed-v3, Local | | Validation | Zod schemas with branded types | | Event Bus | `@oshun/event-bus` (Kafka) | | Logging | `@oshun/logging` (Pino-based structured logging) | | Frontend | React, Vite, Tailwind CSS, React Router | | Build | Nx with `@nx/js:tsc` | | Testing | Vitest | ### Technology Rationale - **PostgreSQL with tsvector** enables dual-mode storage: relational metadata and full-text search in a single database, avoiding a separate search engine for keyword retrieval - **Pluggable vector store** allows teams to use in-memory for development, Qdrant for self-hosted production, or Pinecone for cloud-native deployments without changing any application code - **Prisma ORM** provides type-safe database access with migration support and code generation from the schema - **Zod schemas with branded IDs** prevent mixing incompatible ID types at compile time (e.g., passing a `CitationId` where a `DocumentId` is expected) --- ## 7. Project Structure The directory layout maps directly to the application and library taxonomy described above. Each app has a single focused responsibility; each library handles one coherent subdomain. ``` apps/sophia/ search-api/ # Hybrid search, reranking, RAG, and citation API ingestion/ # Document ingestion pipeline (connectors, extraction, enrichment) knowledge-graph/ # Entity/relationship CRUD, graph queries, resolution, curator workbench/ # React web UI for document management, search, and curation libs/sophia/ (27 libraries) agents/ # Research-agent pipeline citation-analysis/ # Citation source quality and linting citation-graph/ # Citation network analysis client/ # TypeScript SDK (@sophia/client) concordia-knowledge/ # Precedent/template lookup for Concordia corpus/ # Educational images and diagram templates crawling/ # Web crawling with rate limiting credibility/ # Source credibility scoring cultural-research/ # Cultural consultation, localization, accessibility database/ # Prisma schema and generated client document-parser/ # Multi-format parsing with structure extraction embeddings/ # Image / multimodal / dense-retriever embeddings evaluation/ # Retrieval quality metrics and benchmarks event-publisher/ # Typed event publishing indexing/ # Document chunking, embedding, and vector indexing ingestion/ # Ingestion connectors, extraction, enrichment knowledge-base/ # Educational entity/curriculum extraction knowledge-graph/ # Entity-relationship graph construction and traversal oshun-client/ # @oshun/sophia-client compatibility re-export predictions/ # Content prediction and schedule optimization research-engine/ # Multi-source research query execution schemas/ # Shared Zod schemas semantic-search/ # Educational semantic search with caching theory/ # Research management and citation tracking trend-curriculum/ # Curriculum trend signal detection vectordb/ # Vector store abstraction (Qdrant, Milvus, Pinecone, in-memory) verification/ # Fact-checking and verification pipelines ``` --- ## 8. Cross-Domain Dependencies Sophia deliberately limits the number of external domains it depends on, keeping its own dependencies minimal while providing services to other domains. This makes Sophia's infrastructure stable — the knowledge layer should not break because a consumer domain changed. ### Libraries Sophia Depends On | Library | Usage | | ------------------ | ---------------------------------- | | `@oshun/event-bus` | Kafka event publishing | | `@oshun/logging` | Structured logging across all apps | ### Domains That Consume Sophia Hathor is the primary active consumer of Sophia events today. Lilith and Bellona are default routing targets for key event types. | Domain | Integration | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Hathor | Consumes `sophia.document.ingested` and `sophia.entity.extracted` events; uses `@hathor/sophia-integration` library for citation grounding, research-based world generation, and lore fact-checking | **Domain boundary rationale:** Sophia owns _research and knowledge infrastructure_ — ingestion, indexing, retrieval, graph, and citation mechanics. It does not own the product workflows built on top of that infrastructure. Metis owns educational product workflows; Hathor owns worldbuilding semantics; Concordia owns bargaining orchestration; Nous owns model infrastructure. Sophia supplies grounded knowledge to all of them but does not decide how that knowledge is used. ### Optional Infrastructure These services are optional at development time but recommended for production. They are enabled via Docker Compose profiles. | Service | Purpose | Profile | | ------------- | ------------------------------------- | ------- | | Qdrant | Vector database (recommended for dev) | vectors | | Milvus | Alternative distributed vector store | vectors | | Neo4j | Alternative graph backend | graph | | Elasticsearch | Alternative full-text search backend | search | --- ## 9. Deployment Considerations - **Search API and Knowledge Graph** run as separate services, enabling independent scaling. The search API is typically read-heavy and latency-sensitive; the knowledge graph service handles graph traversal workloads that may require more memory. - **Ingestion** can run as a standalone service or as a CLI tool for batch processing. It has no HTTP port and is designed for worker-queue deployment. - **Vector store** selection is driven by environment variables; no code change required to switch backends. - **Workbench** produces a static Vite build deployable to any CDN. - **Database migrations** run via `pnpm nx run @sophia/database:prisma:migrate`.