Sophia Domain — Architecture

Knowledge Management and Research Platform

Sophia is the domain responsible for turning raw documents into a searchable, graph-connected, citation-verified knowledge base. Where most domains in the Oshun monorepo create content, Sophia grounds it: it ingests research papers, scriptures, essays, and any textual source; indexes them for hybrid semantic search; builds a knowledge graph from the entities and relationships it finds; and evaluates how well AI-generated answers are supported by the retrieved evidence.

The real-world problem Sophia solves is hallucination and attribution. Without a system like Sophia, AI responses can sound authoritative while citing nothing real. Sophia's RAG (Retrieval-Augmented Generation) pipeline ensures every claim in a generated answer can be traced back to a specific chunk of a specific document, with a formatted citation in the style the reader expects.

Sophia is consumed by Lilith (to ground spiritual-guidance AI conversations), Hathor (to validate worldbuilding lore against historical research), and any other domain that needs reliable, attributed knowledge retrieval. Internally, Sophia is organized around 4 applications (apps/sophia/) and 27 libraries (libs/sophia/) that together form a complete research knowledge pipeline.

1. System Overview#

Core Responsibilities#

• Document Ingestion — Multi-format document intake with configurable chunking strategies, entity extraction, and summary generation
• Semantic Search — Hybrid search combining vector similarity and keyword matching with cross-encoder reranking
• RAG Pipeline — Retrieval-augmented generation with citation tracking and groundedness scoring
• Knowledge Graph — Entity extraction, relationship mapping, entity resolution, and graph traversal
• Evaluation — Retrieval quality metrics, citation integrity checking, and benchmark management
• Research Management — Research project tracking, hypothesis management, and citation formatting

2. Service Architecture#

2.1 Applications#

The four applications divide Sophia's responsibilities cleanly: the Search API handles queries; the Ingestion worker handles document intake; the Knowledge Graph service manages the entity graph; and the Workbench provides a human operator interface.

                  External Clients
                        |
          +-------------+-------------+
          |                           |
   +------+------+           +--------+--------+
   |  search-api  |           |  knowledge-graph|
   |  Port 3000   |           |  Port 3001      |
   |              |           |                 |
   | SearchAPI    |           | KnowledgeGraph  |
   | Service      |           | Service         |
   |              |           |                 |
   | - hybrid     |           | - EntityService |
   |   search     |           | - RelationSvc   |
   | - RAG        |           | - GraphService  |
   | - reranking  |           | - EntityResolver|
   +------+-------+           +--------+--------+
          |                            |
          v                            v
   +------+------------------------------+--------+
   |                @sophia/database              |
   |         PostgreSQL (sophia schema)            |
   +------+------------------------------+--------+
          |
   +------+------+
   |  ingestion  |     (pipeline worker, no fixed port)
   |             |
   | pipeline:   |
   | fetch →     |
   | extract →   |
   | enrich →    |
   | chunk       |
   | (+ optional |
   |  kalika KG) |
   +------+------+
          |
   +------+------+
   |  workbench  |     (React SPA, Vite dev server)
   |             |
   | Dashboard   |
   | Documents   |
   | Search      |
   | Curation    |
   | Review      |
   | Annotations |
   +-------------+

2.2 Application Details#

Search API (apps/sophia/search-api, Port 3000)#

The primary search service providing semantic search with RAG context generation. Implemented as a SearchAPIService class with framework-agnostic request handlers via createRequestHandlers(service). By keeping the service logic separate from the HTTP layer, the same handlers can be mounted on Hono, Express, or any other framework without modification.

• Dependencies: @oshun/logging, RetrievalService, CitationService, CrossEncoderReranker
• Dev mode: Uses MockEmbeddingProvider, MockVectorStore, MockTextIndex

Ingestion Pipeline (apps/sophia/ingestion)#

The ingestion worker (IngestionPipeline) accepts documents or URLs and drives them through a multi-stage processing pipeline. It has no fixed HTTP port because it is designed to run as a background worker, not a request handler — it processes jobs from a queue and emits domain events when done.

• Modes: CLI (single document argument) or service mode; no fixed HTTP port
• Pipeline stages: Fetch → Extract → Enrich → Chunk, with an optional Kalika knowledge-graph sync stage afterward for math/physics documents
• Concurrency: maxConcurrentJobs option, default 5
• Pipeline events: job:created, job:started, job:progress, job:completed, job:failed, stage:started, stage:completed, stage:failed
• Domain events: publishes sophia.document.ingested and per-entity sophia.entity.extracted via @sophia/event-publisher

Knowledge Graph (apps/sophia/knowledge-graph, Port 3001)#

A standalone knowledge graph service (KnowledgeGraphService) that manages the entity and relationship graph extracted from ingested documents. The service runs as its own process (separate from the search API) so it can be scaled independently — knowledge graph queries tend to be computationally different from search queries. It is backed by an in-memory MemoryGraphStore and exposes a full REST API covering entity and relationship CRUD, graph traversal and path finding, entity resolution, and a curator review/approval subsystem.

• Internal services: EntityService, RelationService, GraphService, EntityResolver (when enableResolution is true), and CuratorService

Workbench (apps/sophia/workbench)#

A React SPA for document management, search testing, knowledge curation, annotation workflows, and operator tooling. The Workbench is aimed at knowledge workers — researchers, curators, and operators — who need to interact with the Sophia knowledge base without writing code.

• Technology: Vite, Tailwind CSS, React Router
• Pages/routes: Dashboard (/), Curation (/curation), Review (/review), Annotations (/annotations), Documents (/documents), Search (/search), Operator Workbench (/operator)
• API Layer: AuthService, DashboardService, DocumentsService, AnnotationsService, SearchService, ReviewService, OperatorWorkbenchService
• Pattern: Singleton service instances with getXService() / resetXService() for testing

3. Library Architecture#

3.1 Layer Overview#

There are 27 libraries under libs/sophia/. They are organized into four layers: contracts (validation schemas), data (database and vector store), domain (business logic), and client (SDK). The domain layer is by far the largest, covering every subdomain from ingestion to evaluation.

Contracts layer:
  @sophia/schemas          — Zod schemas with branded ID types

Data layer:
  @sophia/database         — Prisma client and schema (21 models)
  @sophia/vectordb         — Vector store abstraction (4 backends)

Domain layer:
  @sophia/indexing         — Chunking, embedding, index building
  @sophia/ingestion        — Ingestion connectors, extraction, enrichment
  @sophia/knowledge-graph  — Entity-relationship graph construction/traversal
  @sophia/evaluation       — RAG quality metrics and benchmarks
  @sophia/theory           — Citation management and research workflows
  @sophia/agents           — Research-agent pipeline (decomposition, verification)
  @sophia/research-engine  — Multi-source research query execution
  @sophia/semantic-search  — Educational semantic search with caching
  @sophia/knowledge-base   — Educational entity/curriculum extraction
  @sophia/trend-curriculum — Curriculum trend signal detection
  @sophia/citation-analysis / @sophia/citation-graph — Citation quality and network analysis
  @sophia/credibility / @sophia/verification — Source credibility and fact-checking
  @sophia/document-parser  — Multi-format parsing with structure extraction
  @sophia/crawling         — Web crawling with rate limiting
  @sophia/corpus           — Educational images and diagram templates
  @sophia/embeddings       — Image / multimodal / dense-retriever embeddings
  @sophia/cultural-research — Cultural sensitivity, localization, accessibility
  @sophia/predictions      — Time-series forecasting and content prediction
  @sophia/concordia-knowledge — Precedent/template lookup for Concordia cases

Integration layer:
  @sophia/event-publisher  — Typed event publishing (9 event types)

Client layer:
  @sophia/client           — Unified TypeScript SDK
  @oshun/sophia-client     — Oshun-scoped compatibility re-export of the SDK

3.2 Library Descriptions#

@sophia/client — TypeScript SDK#

The unified client provides type-safe access to all Sophia services from any TypeScript consumer in the monorepo. It wraps the HTTP APIs of the search-api and knowledge-graph services behind a clean object interface with sub-clients for each capability.

Exports: SophiaClient (unified), SearchClient, IngestionClient, KnowledgeGraphClient, SearchQueryBuilder, GraphQueryBuilder, branded ID types.

const sophia = new SophiaClient({
  searchUrl: 'http://localhost:3000',
  knowledgeGraphUrl: 'http://localhost:3001',
});

@sophia/database — Data Access Layer#

The single source of truth for Sophia's persistent state. All 21 PostgreSQL models are defined here as a Prisma schema, and the generated client is what every other Sophia library uses to read and write data. Exports getSophiaClient(), createSophiaClient(), disconnectSophiaClient(), and uppercase string-union enum types. Schema in libs/sophia/database/prisma/schema.prisma (1361 lines).

@sophia/schemas — Validation Contracts#

Zod schemas with branded IDs for documents, citations, claims, knowledge packs, entities, and relations. This library forms the contract layer shared across apps and libraries — it defines what shape data must be in, independently of how it is stored. Note that this layer uses lowercase kebab-cased enum values ('chicago-notes') while the database layer uses uppercase values (CHICAGO); both are documented in detail in the specifications.

@sophia/indexing — Chunking and Embedding#

The heart of the RAG pipeline's preprocessing stage. This library splits documents into retrievable chunks and generates vector embeddings for each chunk. It implements eight chunking strategies (FixedSizeChunker, SlidingWindowChunker, ParagraphChunker, SentenceChunker, RecursiveChunker, MarkdownChunker, CodeChunker, SemanticChunker), three embedding providers (OpenAI, Cohere, Local), IndexBuilder, vector math utilities, and graph export.

@sophia/vectordb — Vector Store Abstraction#

Rather than coupling the rest of the system to a specific vector database, this library provides a unified interface that four different backends all implement. Switching from an in-memory development store to a production Qdrant cluster requires only a configuration change. The abstraction covers four backends (Memory, Qdrant, Pinecone, Milvus) with 10 pre-defined collection schemas, hybrid search with RRF, batch utilities, and health checks.

Key files: qdrant-store.ts, milvus-store.ts, pinecone-store.ts, memory-store.ts, src/utils/batch.ts, src/utils/similarity.ts, src/index/builder.ts.

@sophia/evaluation — Quality Assessment#

Measures how well the RAG pipeline is actually working. Without evaluation, improvements to chunking or retrieval are guesswork. This library provides RetrievalEvaluator, CitationChecker, GroundednessScorer, and BenchmarkRunner. It supports faithfulness, relevancy, precision/recall, citation integrity, and groundedness scoring at both semantic and lexical levels.

@sophia/theory — Research Framework#

Provides scholarly citation management and research project tracking. CitationManager supports 12 citation styles — see the CitationStyle union in theory/src/types.ts — with claim detection, verification, and bibliography generation. ResearchManager handles project, hypothesis, and evidence tracking.

@sophia/cultural-research — Cultural Intelligence#

Enables Sophia to assess whether knowledge content is culturally sensitive and accessible before it is surfaced to users. Contains CulturalConsultant (sensitivity analysis), LocalizationManager (translation workflows), AccessibilityManager (WCAG checking), and CulturalDatabase (structured cultural knowledge).

@sophia/event-publisher — Event Publishing#

SophiaEventPublisher with 9 typed event methods. Wraps @oshun/event-bus (Kafka). Follows a singleton pattern so all Sophia services share a single publisher instance.

@sophia/predictions — Predictive Analytics#

Time series forecasting, schedule optimization, and content performance prediction. Also tagged scope:shared for reuse across domains outside Sophia.

4. Data Flow#

The four primary data flows below show how data moves through Sophia for the most important operations. Each flow is independent — ingestion populates the index, search queries it, RAG extends search with generation, and the knowledge graph uses the same entity data extracted during ingestion.

4.1 Document Ingestion Flow#

Documents enter through the ingestion pipeline and are processed in sequence. Each stage adds information to the document record until the document is fully indexed and searchable.

Client / URL
    |
    v
Ingestion Pipeline (apps/sophia/ingestion)
    |
    +-- [Fetch] ────────> Source connector
    +-- [Extract] ──────> Text and metadata extraction, OCR
    +-- [Enrich] ───────> Entity extraction, summarization, keyword extraction
    +-- [Chunk] ────────> @sophia/indexing (selected chunking strategy)
    +-- [Index] ────────> Embeddings via @sophia/indexing
    |                      Vector upsert via @sophia/vectordb
    |                      Document + chunk save via @sophia/database
    |
    +-- Emit: sophia.document.ingested
    +-- Emit: sophia.entity.extracted (per entity)
    +-- Emit: sophia.index.updated

4.2 Search Flow#

A search query is handled entirely by the Search API. The key step is the RRF fusion: combining dense (embedding similarity) and sparse (keyword) results into a single ranked list that is better than either alone.

Client Query
    |
    v
Search API (apps/sophia/search-api)
    |
    +-- RetrievalService
    |       |
    |       +-- Dense search ──> @sophia/vectordb (vector similarity)
    |       +-- Sparse search ─> PostgreSQL full-text (tsvector)
    |       +-- RRF fusion ────> Merge and re-rank results
    |
    +-- CrossEncoderReranker (optional)
    |       |
    |       +-- Rerank top-K results
    |
    +-- Response assembly
    |       |
    |       +-- Metadata enrichment from @sophia/database
    |       +-- Highlight generation
    |       +-- Context window assembly
    |
    +-- Emit: sophia.search.performed

4.3 RAG Flow#

RAG (Retrieval-Augmented Generation) extends search by feeding the retrieved chunks into an LLM and then verifying which source documents actually support each claim in the generated answer. This is what allows Sophia to produce answers that are both fluent and fully attributable.

Natural language question
    |
    v
Search API /rag
    |
    +-- Query embedding (OpenAI / Cohere / Local)
    +-- Hybrid retrieval (RetrievalService)
    +-- Cross-encoder reranking
    +-- Context assembly (top-K chunks)
    +-- LLM generation (with system prompt)
    +-- CitationService
    |       |
    |       +-- Extract citations from generated answer
    |       +-- Verify against source chunks
    |       +-- Format in requested citation style
    |
    +-- Return: { answer, citations, sources }

4.4 Knowledge Graph Flow#

The knowledge graph is populated as a side effect of document ingestion and is then served by the Knowledge Graph service as a separate queryable graph. The separation matters: ingestion is batch-oriented and write-heavy, while graph queries are read-heavy and interactive.

Ingestion pipeline
    |
    v
Entity Extraction (enrichment stage)
    |
    +-- Entities saved via @sophia/database
    +-- EntityMentions saved (document + chunk position)
    |
    v
Knowledge Graph Service (apps/sophia/knowledge-graph)
    |
    +-- EntityService — CRUD for graph nodes
    +-- RelationService — CRUD for directed edges
    +-- GraphService
    |       |
    |       +-- BFS / DFS traversal
    |       +-- Shortest path (Dijkstra-like)
    |       +-- All-paths enumeration
    |       +-- Subgraph extraction
    |
    +-- EntityResolver
            |
            +-- Name similarity matching
            +-- Property-based matching
            +-- Merge with canonicalId tracking

5. Key Design Patterns#

Sophia uses a small set of patterns consistently across all its libraries and applications. Understanding these patterns makes it much easier to navigate the codebase.

5.1 Strategy Pattern — Chunking#

All chunking strategies implement a common ChunkingStrategy interface (chunk(text, options): Promise<Chunk[]>). The ingestion pipeline selects the appropriate strategy by name at runtime. Adding a new chunking approach requires only implementing the interface and registering it in the factory — no other code needs to change.

5.2 Adapter Pattern — Vector Stores#

All four vector store backends (MemoryVectorStore, QdrantVectorStore, PineconeVectorStore, MilvusVectorStore) implement a unified VectorStore interface. The rest of the system never calls backend-specific APIs directly, so swapping the vector store for a different product is a configuration-only change.

5.3 Builder Pattern — SDK Queries#

Both SearchQueryBuilder and GraphQueryBuilder provide fluent chainable APIs. Builders produce typed request objects that are submitted to the respective service clients. This makes complex multi-parameter queries readable and refactorable.

5.4 Singleton Services with Test Reset#

Workbench API services follow a singleton pattern with a getXService() factory and a resetXService() function for test isolation. The reset function tears down shared state between test cases, preventing one test's side effects from leaking into another.

5.5 Framework-Agnostic Request Handlers#

The SearchAPIService exposes its operations as plain async methods rather than being tightly coupled to an HTTP framework. createRequestHandlers(service) wraps these for mounting on Hono, Express, or any other framework. This means the core service logic can be unit-tested without spinning up an HTTP server.

6. Technology Stack#

The table below summarizes the technology choices for each layer, followed by the rationale for the most consequential decisions.

Technology Rationale#

• PostgreSQL with tsvector enables dual-mode storage: relational metadata and full-text search in a single database, avoiding a separate search engine for keyword retrieval
• Pluggable vector store allows teams to use in-memory for development, Qdrant for self-hosted production, or Pinecone for cloud-native deployments without changing any application code
• Prisma ORM provides type-safe database access with migration support and code generation from the schema
• Zod schemas with branded IDs prevent mixing incompatible ID types at compile time (e.g., passing a CitationId where a DocumentId is expected)

7. Project Structure#

The directory layout maps directly to the application and library taxonomy described above. Each app has a single focused responsibility; each library handles one coherent subdomain.

apps/sophia/
  search-api/        # Hybrid search, reranking, RAG, and citation API
  ingestion/         # Document ingestion pipeline (connectors, extraction, enrichment)
  knowledge-graph/   # Entity/relationship CRUD, graph queries, resolution, curator
  workbench/         # React web UI for document management, search, and curation

libs/sophia/         (27 libraries)
  agents/            # Research-agent pipeline
  citation-analysis/ # Citation source quality and linting
  citation-graph/    # Citation network analysis
  client/            # TypeScript SDK (@sophia/client)
  concordia-knowledge/ # Precedent/template lookup for Concordia
  corpus/            # Educational images and diagram templates
  crawling/          # Web crawling with rate limiting
  credibility/       # Source credibility scoring
  cultural-research/ # Cultural consultation, localization, accessibility
  database/          # Prisma schema and generated client
  document-parser/   # Multi-format parsing with structure extraction
  embeddings/        # Image / multimodal / dense-retriever embeddings
  evaluation/        # Retrieval quality metrics and benchmarks
  event-publisher/   # Typed event publishing
  indexing/          # Document chunking, embedding, and vector indexing
  ingestion/         # Ingestion connectors, extraction, enrichment
  knowledge-base/    # Educational entity/curriculum extraction
  knowledge-graph/   # Entity-relationship graph construction and traversal
  oshun-client/      # @oshun/sophia-client compatibility re-export
  predictions/       # Content prediction and schedule optimization
  research-engine/   # Multi-source research query execution
  schemas/           # Shared Zod schemas
  semantic-search/   # Educational semantic search with caching
  theory/            # Research management and citation tracking
  trend-curriculum/  # Curriculum trend signal detection
  vectordb/          # Vector store abstraction (Qdrant, Milvus, Pinecone, in-memory)
  verification/      # Fact-checking and verification pipelines

8. Cross-Domain Dependencies#

Sophia deliberately limits the number of external domains it depends on, keeping its own dependencies minimal while providing services to other domains. This makes Sophia's infrastructure stable — the knowledge layer should not break because a consumer domain changed.

Libraries Sophia Depends On#

Domains That Consume Sophia#

Hathor is the primary active consumer of Sophia events today. Lilith and Bellona are default routing targets for key event types.

Domain boundary rationale: Sophia owns research and knowledge infrastructure — ingestion, indexing, retrieval, graph, and citation mechanics. It does not own the product workflows built on top of that infrastructure. Metis owns educational product workflows; Hathor owns worldbuilding semantics; Concordia owns bargaining orchestration; Nous owns model infrastructure. Sophia supplies grounded knowledge to all of them but does not decide how that knowledge is used.

Optional Infrastructure#

These services are optional at development time but recommended for production. They are enabled via Docker Compose profiles.

9. Deployment Considerations#

• Search API and Knowledge Graph run as separate services, enabling independent scaling. The search API is typically read-heavy and latency-sensitive; the knowledge graph service handles graph traversal workloads that may require more memory.
• Ingestion can run as a standalone service or as a CLI tool for batch processing. It has no HTTP port and is designed for worker-queue deployment.
• Vector store selection is driven by environment variables; no code change required to switch backends.
• Workbench produces a static Vite build deployable to any CDN.
• Database migrations run via pnpm nx run @sophia/database:prisma:migrate.