Sophia — Feature Reference

Sophia is the Knowledge Management and Research Platform of the Oshun ecosystem. It turns documents, research papers, scriptures, and any textual source into a searchable, graph-connected, citation-verified knowledge base. Sophia powers the research backbone for Lilith's AI conversations, Hathor's lore grounding, Yemaya's creative references, and any other domain that needs reliable knowledge retrieval with source attribution.

Platform Overview#

Sophia provides full-stack knowledge infrastructure for the Oshun ecosystem. Its RAG pipeline (Retrieval-Augmented Generation — a technique that grounds AI responses in retrieved documents rather than relying solely on model weights) handles the complete lifecycle from raw document ingestion through vector indexing, semantic search, citation-verified answer generation, and knowledge graph construction. Sophia is consumed by Lilith for spiritual guidance AI, Hathor for research-grounded worldbuilding lore, Yemaya for creative reference research, and any other domain requiring reliable, attributed knowledge retrieval.

Sophia spans 27 libraries, 4 applications, and a 4-backend vector database abstraction layer supporting everything from local development (in-memory) to large-scale distributed deployments (Milvus).

Domain ownership boundary: Sophia owns research and knowledge infrastructure — ingestion, indexing, retrieval, graph construction, and citation mechanics. It does not own the product workflows built on top of that infrastructure. Metis owns educational product workflows; Hathor owns worldbuilding semantics; Concordia owns bargaining orchestration; Nous owns model infrastructure. Sophia supplies grounded knowledge to each of them but does not decide how that knowledge is used.

Application Architecture#

The four applications that make up Sophia's runtime are shown below with their ports and primary responsibilities.

Library Ecosystem#

The 27 libraries under libs/sophia/ cover every capability Sophia exposes. The table below gives a one-line description for each; later sections in this document go into detail on the most significant ones.

Document Ingestion Pipeline#

The ingestion pipeline is the entry point for all knowledge in Sophia. Raw documents arrive in many formats from many sources; the pipeline normalizes them, extracts their content and metadata, and makes them searchable.

14 Supported Document Types#

Sophia models the following 14 document types in its database schema. Each type carries type-appropriate metadata fields; for example, a Translation record captures source language and translator attribution, while a Thesis record captures the degree level and institution.

Ten-Stage Ingestion Pipeline (apps/sophia/ingestion)#

Documents pass through all 10 stages in order; each stage is independently tracked, retried on failure, and reported with granular progress. A document only becomes searchable after completing the final stage, so in-flight documents never return partial or inconsistent results.

1. Upload / URL Fetch — Accept documents from file upload, HTTPS URL, REST API endpoint, database query, Kafka stream, or ZIP/TAR archive; all source types normalized to the same processing pipeline
2. Format Detection — Identify the input format and select the appropriate parser; no manual format specification required
3. Text Extraction — Extract raw text content, preserving structural elements (headings, lists, tables, footnotes) where the format supports it
4. Metadata Extraction — Pull publication metadata: title, authors, publication date, publisher, DOI, ISBN, abstract, journal name, volume/issue; falls back to heuristic extraction for documents without explicit metadata
5. Chunking — Split the extracted text into segments using the configured chunking strategy (see section below); chunk metadata inherits document metadata
6. Embedding Generation — Generate vector embeddings for each chunk using the configured provider (OpenAI, Cohere, or Local); embeddings stored as binary vectors in the database
7. Entity Extraction — Identify and record named entities (people, places, organizations, concepts, traditions, practices) within the text for knowledge graph population
8. Summary Generation — Generate a concise multi-sentence summary of the document using an LLM; stored alongside the full text for quick retrieval
9. Quality Scoring — Assign a quality score based on completeness of metadata, text readability, source credibility signals (peer review status, citation count, publisher reputation), and OCR confidence if applicable
10. Indexing — Store processed chunks, embeddings, metadata, entities, and summary in PostgreSQL and the configured vector store; document becomes searchable immediately

Supported Input Formats#

Sophia can ingest documents in the following file formats. OCR is applied automatically when the input is a scanned image or a PDF without embedded text.

Job Management#

The ingestion pipeline manages concurrent processing jobs, which means multiple documents can be ingested simultaneously without one slow document blocking others.

• Configurable Concurrency — Default 5 simultaneous ingestion jobs; tunable based on available compute resources
• Per-Stage Retry — Each of the 10 stages can independently retry up to 3 times (configurable) on failure; the pipeline resumes from the failed stage rather than starting over
• Granular Status Tracking — Job status reflects the current stage (e.g., "Stage 6/10 — Embedding Generation"); enables accurate progress reporting
• Detailed Error Codes — Each stage reports a structured error code and message on failure; distinguishes transient failures (network timeouts) from persistent failures (corrupt file, unsupported encoding)

Intelligent Chunking#

Splitting documents into chunks is one of the most consequential decisions in a RAG system — chunks that are too large dilute relevance; chunks that are too small lose context. Sophia provides eight strategies so the right split approach can be matched to the document type.

Eight Chunking Strategies (@sophia/indexing)#

Universal Chunking Options#

Every strategy accepts the same base configuration options, so pipeline configuration does not change when switching between strategies.

• Chunk Size — Target chunk size in characters (default: 1000) or tokens (toggled by configuration)
• Overlap — Number of characters/tokens shared between adjacent chunks (default: 200); overlap prevents important context from being split across two chunks with neither half being sufficient
• Measurement Mode — Character-based (faster, less accurate) or tokenizer-based (slower, precise for LLM context window management)
• Metadata Inheritance — Every chunk inherits the parent document's full metadata (title, author, date, source type) for attribution in retrieval results

Semantic Chunking Detail#

The semantic chunker is the most sophisticated option. Unlike the others, it uses embedding similarity to find natural topic boundaries in the text rather than relying on any structural signal. The algorithm works in five steps:

1. Split document into individual sentences
2. Embed each sentence with the configured embedding provider
3. Compute cosine similarity between each consecutive pair of sentences
4. Detect "break points" where similarity drops below a threshold value
5. Group sentences between break points into coherent topical chunks

This produces chunks that align with actual conceptual topic shifts rather than arbitrary length boundaries, maximizing retrieval relevance.

Semantic and Hybrid Search#

Search is the primary real-time capability of Sophia. A query arrives at the Search API, is executed against the vector index and/or full-text index, and returns ranked, highlighted results.

Five Retrieval Strategies (@sophia/indexing)#

Each strategy trades precision for speed and coverage differently. hybrid is the recommended default; rerank gives the best precision at higher latency.

Cross-Encoder Reranking#

After initial retrieval returns a candidate set, results pass through a cross-encoder model that evaluates each (query, document) pair jointly rather than comparing precomputed vectors. This catches results that are semantically relevant but were scored lower in the initial vector pass, significantly improving precision for the top-K results.

Fluent Search Builder#

The TypeScript SDK provides a chainable builder API for constructing search queries. The example below executes a hybrid search with custom blend weights, a metadata filter, and result highlighting.

sophia.search
  .query('enlightenment practices')
  .hybrid({ dense: 0.7, sparse: 0.3 })
  .topK(10)
  .filter({ tradition: 'buddhism', language: 'en' })
  .includeHighlights()
  .execute();

Search Filtering#

Results can be narrowed at query time using any combination of the following filter types.

• Document Filters — Filter by specific document IDs, source identifiers, language codes, publication date range, and document type
• Metadata Filters — Filter by any arbitrary metadata field using equality, range, and contains operators
• Tag Filters — Filter by tags applied during curation
• Result Deduplication — Automatically remove near-duplicate chunks from result sets to prevent the same passage from appearing multiple times
• Context Inclusion — Optionally include surrounding chunks (before and after each match) for broader context in the response

Multi-Collection Search#

Sophia pre-configures 10 vector collection schemas, each tuned for a different content type. The table below shows what each collection stores and its intended query use case.

RAG Pipeline#

The RAG pipeline is what distinguishes Sophia from a plain search engine. It takes a natural language question and returns a grounded answer — not just a list of documents, but a generated response with each claim backed by an inline citation to the specific source chunk that supports it.

Seven-Stage RAG Pipeline#

The pipeline executes these steps in order for every RAG request:

1. Query Embedding — Convert the question to a vector using the configured embedding provider
2. Retrieval — Find the most relevant document chunks via hybrid search; configurable top-K count
3. Reranking — Reorder retrieved chunks using a cross-encoder model for precision improvement
4. Context Assembly — Combine top-ranked chunks into a coherent context window respecting the LLM's maximum context length; overlapping content deduplicated
5. Answer Generation — Send the question plus assembled context to the configured LLM; configurable model, system prompt, and temperature
6. Citation Extraction — Identify which source chunks support each claim in the generated answer; produce inline citation markers with confidence scores
7. Response — Return the answer with inline citations linked to source document records for user exploration

RAG Configuration Options#

Every aspect of the RAG pipeline is configurable at request time, allowing callers to tune the trade-off between speed, cost, and quality.

• Embedding Provider — Choose between OpenAI, Cohere, or Local embedding models
• Embedding Model — Select specific embedding model and dimensionality
• Top-K Retrieval — Number of chunks retrieved before reranking
• Reranking Model — Select the cross-encoder model used for reranking
• LLM Selection — Configure the answer generation model (OpenAI GPT-4, Claude, Gemini, or any OpenAI-compatible endpoint)
• System Prompt — Custom system prompt to shape answer format, tone, and scope
• Citation Style — Chicago, MLA, APA, Turabian, Harvard, SBL, Oxford, IEEE, Vancouver, or AMA for inline citations
• Minimum Citation Confidence — Filter out low-confidence citation assignments
• Maximum Citations — Cap the number of citations per answer to control verbosity

Knowledge Graph#

Sophia's knowledge graph represents the entities and relationships found in the ingested document corpus. Where search retrieves passages, the knowledge graph retrieves connections — answering questions like "what did this figure teach", "which traditions share this practice", or "what is the shortest conceptual path between these two ideas".

15 Entity Types#

The knowledge graph recognizes the following 15 entity types, covering the major categories of philosophical, religious, and historical knowledge.

27 Relationship Types#

Relationships are the edges of the knowledge graph. Every relationship is directed (from a source entity to a target entity) and typed. The knowledge-graph data model (@sophia/database RelationType enum) defines 27 typed relationship kinds, grouped by domain below.

Note that the @sophia/schemas Zod RelationTypeSchema defines a wider 57-member set, and the knowledge-graph service uses its own 29-member RelationshipType union — see the domain specification for the full breakdown. All relationships are directed and typed; a graph edge from Figure A to Text B with type WROTE has a different meaning from a Text-to-Figure edge with the same type.

Graph Operations#

Sophia supports both simple CRUD operations on graph nodes/edges and graph-theoretic traversal algorithms for exploring connectivity.

Fluent Graph Query Builder#

The SDK provides a chainable graph query builder that reads like plain English. The example below finds all figures or concepts that a given entity taught or influenced.

sophia.graph
  .from(entityId)
  .via(['TAUGHT', 'INFLUENCED'])
  .toType(['figure', 'concept'])
  .neighbors();

Entity Resolution#

Because entities are extracted from many documents by automated processes, the same real-world entity often ends up with multiple records (e.g., "Siddhartha Gautama", "Siddhattha Gotama", "The Buddha"). Entity resolution merges these duplicates into a single canonical record.

• Configurable Similarity Thresholds — Set minimum name similarity score for candidate pairs to be considered duplicates
• Alias Matching — Match entities that have the same alternate names even if their canonical names differ
• Property Matching — Use property value overlap (e.g., dates, locations) to increase or decrease merge confidence
• Merge Audit Trail — Track which entities were merged and from which source, enabling rollback if a merge was incorrect
• Confidence Levels — CERTAIN, PROBABLE, POSSIBLE, UNCERTAIN, INFERRED

Graph Statistics and Visualization#

Once the graph is populated, these metrics and export formats help teams understand its structure and feed it into external visualization tools.

• Graph Metrics — Node count, edge count, graph density (ratio of actual to possible edges), average node degree, connected component count, and node centrality scores (PageRank, betweenness)
• Export Formats — D3.js JSON (nodes + links for web visualization), Graphviz DOT (for graph layout tools), GEXF (for Gephi and similar graph analysis tools)

Vector Database Support#

The vector database layer is where chunk embeddings are stored and queried. Sophia abstracts over four different backends so that the same application code works from a laptop in development (in-memory) to a large production cluster (Milvus).

Four Interchangeable Backends (@sophia/vectordb)#

All four backends implement the same unified interface, so switching backends requires only a configuration change with no application code changes.

Unified Interface Operations#

The interface methods common to all four backends are listed below. Backend-specific features (e.g., Qdrant filters) are not exposed through the unified interface.

• connect() / disconnect() — Connection lifecycle management
• createCollection() / deleteCollection() — Collection schema management
• upsert(vectors) / delete(ids) — Vector CRUD
• search(query, topK) — Standard cosine similarity search
• hybridSearch(query, sparseQuery, topK) — Combined vector + keyword search
• batchUpsert(vectors) / batchDelete(ids) — Bulk operations for indexing efficiency
• getCollectionStats() — Vector count, index size, memory usage
• healthCheck() — Backend availability and latency measurement

Hybrid Search with Reciprocal Rank Fusion (RRF)#

The hybrid search algorithm fuses vector and keyword results without making assumptions about either result set's score distribution. The RRF formula is unbiased toward either retrieval method's score range, making it more robust than a simple weighted sum.

1. Execute vector similarity search to get top-N semantically similar results
2. Execute BM25 keyword search to get top-N term-matching results
3. Apply RRF formula to each result: score = Σ 1/(k + rank_i) where k is a smoothing constant (default 60)
4. Return the fused result set ranked by combined RRF score

Embedding Generation#

Embeddings are the numeric representations that make semantic search possible. Sophia supports three provider options to accommodate different cost, latency, and data sovereignty requirements.

Three Embedding Providers (@sophia/indexing)#

Embedding Service Features#

These features apply regardless of which provider is selected.

• Automatic Provider Selection — Configured once; all ingestion and query embedding uses the same provider without explicit specification per call
• Batch Embedding — Generate embeddings for hundreds of chunks in a single API call using provider batch endpoints; dramatically reduces per-chunk cost and latency for bulk ingestion
• Content-Hash Caching — Before generating an embedding, compute a content hash and check the cache; identical text always produces identical embeddings so re-embedding the same content is avoided
• Configurable Cache TTL and Size — Control how long embeddings are cached and how many unique embeddings the cache holds before LRU eviction

Vector Math Utilities#

A comprehensive vector mathematics library supports embedding operations beyond simple similarity search.

• Similarity Metrics — Cosine similarity, dot product, Euclidean distance, Manhattan distance
• Top-K Finding — Efficiently find the K most similar vectors in a set using brute-force scan or approximate nearest neighbor
• Pairwise Similarity Matrix — Compute all-pairs similarity for a set of vectors; useful for clustering and deduplication analysis
• Vector Operations — Normalization, addition, subtraction, scalar multiplication, centroid computation from a set of vectors
• Dimensionality Analysis — Measure effective dimensionality and detect degenerate (zero-variance) dimensions

Citation Management#

Citation management is what distinguishes Sophia from a generic search system. When an AI generates a claim, Sophia can find the source document that supports it, verify the citation is accurate, and format the reference in whichever scholarly style the consumer requires.

Citation Styles (@sophia/theory)#

The @sophia/theory engine renders citations in ten scholarly styles — Chicago, MLA, APA, Turabian, Harvard, SBL (Society of Biblical Literature), Oxford, IEEE, Vancouver, and AMA (American Medical Association). Chicago and Turabian each expose notes-bibliography and author-date variants, so the CitationStyle enum carries 12 members in total. (The same 12-member set is the CitationStyleSchema in @sophia/schemas.) Representative formats:

Claim Detection#

Before citations can be assigned, Sophia identifies which statements in AI-generated text actually need citing. The following claim types are detected automatically:

• Statistical Assertions — Identifies claims stating percentages, counts, or other quantitative facts that require source backing
• Attribution Statements — Detects "According to X" patterns and similar attributions that need to be verified against the cited source
• Factual Claims — Identifies declarative statements about the world that are not common knowledge and require evidential support
• Quote Detection — Finds direct quotations and checks them against source documents for accuracy
• Citation Insertion Points — Suggests optimal locations in the text to insert citation markers without disrupting reading flow

Citation Verification#

Once citations are assigned, Sophia runs three independent verification checks to ensure every citation is accurate.

• Quote Accuracy — Does the cited source actually contain the quoted text, accounting for minor paraphrase?
• Entailment Verification — Does the source document logically support the claim being cited, or does it merely mention the same topic?
• Attribution Verification — Is the cited author or organization correctly identified in the citation record?

Verification statuses: VERIFIED, REVIEWED (human-checked), UNVERIFIED (not yet checked), DISPUTED (inconsistency detected), PENDING (check in progress).

Source Authority Scoring#

Not all sources are equally reliable. Sophia scores source authority using multiple signals combined into a composite score.

• Peer Review Status — Whether the source was peer-reviewed adds significant authority weight
• Citation Count — How many other sources cite this work (H-index for authors)
• Publisher Reputation — Known academic publishers and journals weighted higher than anonymous web sources
• Composite Authority Score — Weighted combination of all signals into a single 0–1 score for ranking sources by reliability

Citation Graph and Network Analysis#

Beyond per-document citations, Sophia builds a citation network across the entire corpus, enabling corpus-level bibliographic analysis.

• Citation Network — Nodes represent documents; edges represent citation relationships; enables traversal of citation chains
• Most-Cited Sources — Identify the most-authoritative sources in a topic area by citation count within the corpus
• Visualization Export — Export the citation network to D3.js JSON, DOT, and GEXF formats for external visualization tools

Bibliography Generation#

Sophia can generate formatted reference lists from any subset of the corpus.

• Filtered Bibliographies — Generate reference lists filtered by date range, source type, author, or custom tag
• Sort Options — Alphabetical by author, chronological, or citation-order
• Deduplication — Automatically remove duplicate entries (same DOI or same author+title+year)
• Style Application — Format the complete bibliography in any supported style

Research Workflows#

Beyond retrieval, Sophia provides tools for managing the research process itself — tracking projects, hypotheses, and evidence over time.

Research Projects (@sophia/theory)#

• Project Creation — Create named research projects with description, methodology type, and status tracking
• Status Lifecycle — Planning → Active → Review → Completed
• Timeline and Milestones — Define expected completion dates and key deliverable milestones with status tracking

Hypothesis Management#

Research hypotheses in Sophia are first-class objects that can be linked to evidence and tracked through a formal lifecycle.

• Hypothesis Registration — Record formal hypotheses within a project with explicit statement, background, and prediction
• Evidence Linking — Attach supporting or contradicting evidence (documents, citations, search results, data) to hypotheses
• Status Tracking — Proposed → Testing → Supported → Refuted lifecycle with timestamps for each transition

Research Notes#

• Contextual Notes — Attach notes to projects, documents, entities, or individual citations with rich text content
• Tagging — Apply custom tags to notes for cross-project thematic organization
• Evidence Linking — Link notes directly to specific pieces of evidence for traceability

Methodology Frameworks#

Pre-supported methodology types keep research records consistent across projects: comparative analysis, historical analysis, textual criticism, hermeneutic analysis (interpretation-focused methodology), phenomenological study, and ethnographic research.

Retrieval Quality Evaluation#

Sophia's evaluation library makes retrieval quality measurable and comparable. Without evaluation metrics, it is impossible to know whether a change to chunking strategy or retrieval model actually improved results.

Retrieval Metrics (@sophia/evaluation)#

Citation Integrity Checking#

Run all three citation verification checks (quote accuracy, entailment, attribution) across all citations in a response and aggregate into a composite integrity report with per-citation breakdown.

Groundedness Scoring#

Measure how well an AI response is grounded in its cited sources using two complementary approaches. Together they surface both cases where the response drifts semantically from the source and cases where it uses different terminology despite covering the same ground.

• Semantic Grounding — Embedding-based comparison between the response claims and retrieved evidence; high semantic similarity indicates the claim aligns with the source
• Lexical Grounding — Term overlap and n-gram matching between response and sources; catches cases where semantic similarity is high but key terms differ
• Claim-Level Scoring — Individual groundedness scores per claim rather than just a document-level average; surfaces specific unsupported claims
• Multi-Dimensional Scores — Factual accuracy (claim matches source), source support (source actually backs the claim), and internal consistency (claims in the response do not contradict each other)

Benchmark System#

The benchmark system enables objective, repeatable quality comparisons between different pipeline configurations.

• Standard Benchmark Datasets — Built-in evaluation datasets for common knowledge retrieval scenarios; enables objective baseline comparison
• Custom Benchmark Creation — Define custom test cases with questions, expected relevant documents, and expected answer content
• Statistical Analysis — Mean, median, standard deviation, and percentile scores across a benchmark run; confidence intervals for comparisons
• A/B Pipeline Testing — Compare two retrieval configurations (different chunking, different embedding model, different top-K) against the same benchmark to select the superior configuration

Cultural Research and Localization#

Sophia's knowledge base covers traditions, texts, and practices from cultures around the world. The cultural research library helps ensure that knowledge content is handled sensitively and made accessible across languages.

Cultural Sensitivity Analysis (@sophia/cultural-research)#

Analyze content for cultural sensitivity issues before publication or use.

• Risk Level Assessment — LOW, MODERATE, HIGH, CRITICAL risk classification per identified issue
• Concern Type Classification — Stereotyping, cultural appropriation, sacred content misuse, historical insensitivity, inappropriate language, and misrepresentative visual description
• Content Location — Exact text spans (character offsets) of problematic content for precise review
• Actionable Suggestions — Specific rewrite recommendations for each identified issue

Cultural Database#

The cultural database underpins sensitivity analysis with structured knowledge about cultures and their practices.

Localization Management#

The localization manager tracks the lifecycle of content translation from initial machine translation draft through human review and final approval.

• Localization Projects — Create and track translation projects from a source locale to one or more target locales
• Translation Unit Management — Track individual string translation status: Untranslated → In Progress → Translated → Reviewed → Approved
• Glossary Management — Maintain approved terminology lists per locale for consistent translation of domain-specific vocabulary
• Translation Memory — Store approved translations for reuse when identical or similar strings appear in future projects
• Machine Translation Integration — Connect to machine translation services for first-pass translation drafts; human review follows
• Linguistic Quality Assurance (LQA) — Structured issue categorization: accuracy, fluency, consistency, style, and locale convention compliance

Accessibility Checking#

• WCAG Level Targeting — Evaluate content against Web Content Accessibility Guidelines (WCAG) at level A, AA, or AAA
• Accessibility Dimensions — Visual, auditory, motor (keyboard navigation), and cognitive accessibility checks
• Issue Reporting — Per-issue severity, WCAG success criterion reference, and remediation guidance

Predictive Analytics#

Sophia's predictions library provides time-series forecasting and scheduling intelligence for content and revenue workflows. It is tagged scope:shared and can be consumed by other domains that need forecasting capabilities.

Revenue Forecasting (@sophia/predictions)#

• Time Series Analysis — Multiple forecasting methods: naive, exponential smoothing, ARIMA, linear regression, and ensemble; selects best-fit method automatically
• Seasonal Pattern Detection — Identify weekly, monthly, and annual seasonality in revenue data for more accurate forecasts
• External Factor Modeling — Incorporate known external factors (holidays, marketing campaigns, competitor launches) as forecast adjustors
• Peak Period Identification — Automatically flag predicted peak and trough periods for staffing and capacity planning
• Forecast Risk Assessment — Quantify forecast uncertainty with confidence intervals and risk factor breakdown

Schedule Optimization#

• Historical Performance Analysis — Analyze when published or streamed content performs best based on historical engagement data
• Competitor Schedule Awareness — Incorporate competitor publishing patterns to identify uncontested time slots
• Optimal Time Slot Recommendations — Recommend specific times and days for content publication to maximize expected reach

Content Performance Prediction#

• Pre-Publication Engagement Prediction — Predict engagement metrics (views, completions, shares) before content is published based on content features and audience patterns
• Audience Alignment Scoring — Score how well a piece of content aligns with the target audience's demonstrated preferences
• A/B Content Comparison — Compare two content variants on predicted performance before selecting which to publish

Knowledge Packs#

Knowledge packs are curated, versioned collections of documents and entities organized around a specific topic, tradition, or use case. They provide a way to package a portion of the Sophia knowledge base for sharing, reuse, or export — for example, a "Theravada Buddhism" pack might contain the key texts, figures, concepts, and practices from that tradition.

• Pack Creation — Create named packs with description, topic tags, and visibility (private, team, organization, public)
• Document Association — Add documents to a pack with per-document importance weighting and relevance notes
• Entity Association — Add knowledge graph entities to a pack with importance scores
• Pack Versioning — Publish numbered versions of a pack; prior versions remain accessible for reproducibility
• Community Features — Star/favorite packs; browse community-contributed packs by topic and rating
• Pack Export — Export a complete pack (documents + entities + metadata) for use in external systems or as a training corpus

Visual Workbench#

The Workbench is a browser-based interface for knowledge workers to interact with the Sophia knowledge base without writing code. It exposes the main Sophia capabilities — document management, search, curation, and annotation — through a set of purpose-built pages.

TypeScript SDK#

The Sophia TypeScript SDK (@sophia/client) is the recommended way to integrate any TypeScript service or application with Sophia. It wraps the HTTP APIs of the search-api and knowledge-graph services behind a single unified client with typed sub-clients.

Unified Client (@sophia/client)#

const sophia = new SophiaClient({
  searchUrl: 'http://localhost:3000',
  knowledgeGraphUrl: 'http://localhost:3001',
});

Sub-clients are accessible via: sophia.search, sophia.ingestion, sophia.graph.

Search Client#

• Quick Search — quickSearch(query, options) — fast semantic search with minimal configuration
• RAG Query — ask(question, options) — full RAG pipeline returning a grounded answer with citations
• Fluent Builder — sophia.search.query(text).hybrid().topK(n).filter(f).execute()
• Result Types — Typed SearchResult, RAGResponse, and Citation objects

Ingestion Client#

• ingestDocument(doc) — Submit a document object directly
• ingestUrl(url) — Submit a URL for fetch-and-ingest
• ingestUrlAndWait(url) — Submit a URL and poll until all 10 stages complete
• getJob(id) — Check ingestion job status and stage progress

Knowledge Graph Client#

• createEntity(type, name, properties) — Create a new graph entity
• createRelation(fromId, toId, type, properties) — Create a typed relationship
• from(entityId).via(relationTypes).toType(entityTypes).neighbors() — Fluent traversal API
• findShortestPath(fromId, toId) — Find the minimum-hop path between two entities
• resolveEntity(candidates) — Merge duplicate entity candidates

Error Handling#

The SDK throws a single SophiaClientError class that carries enough information to decide whether to retry, log, or surface the error to the user. It carries an ApiErrorCode (one of UNAUTHORIZED, FORBIDDEN, NOT_FOUND, BAD_REQUEST, VALIDATION_ERROR, RATE_LIMITED, SERVICE_UNAVAILABLE, INTERNAL_ERROR, TIMEOUT, NETWORK_ERROR, UNKNOWN), an HTTP status, a retryable flag, and optional request ID. Static helpers (fromResponse, networkError, timeout) construct it from common failure cases.

Advanced Research Agents#

Sophia's research agent library (@sophia/agents) provides a complete pipeline for automated research workflows that go well beyond single-document retrieval. Where the search API answers a specific question, the agent pipeline can decompose a complex research topic, gather evidence from multiple sources, and synthesize a comprehensive answer.

• Query Decomposition — Break a complex research question into a set of simpler sub-questions that can be answered independently; the sub-question answers are then synthesized into a coherent response to the original question
• Multi-Source Search — Execute each sub-question against multiple sources simultaneously (knowledge graph, vector index, web crawl, citation database) and merge results with provenance tracking
• Fact Checking — Verify generated claims against retrieved evidence with confidence scoring; claims that cannot be grounded are flagged with a specific reason (insufficient evidence, contradicted by source, not found)
• Contradiction Detection — Identify when two retrieved sources make contradictory claims about the same entity or fact; surface conflicts for human review with both supporting source references
• Temporal Validation — Verify that time-sensitive claims are still current; flags outdated information from sources that pre-date a configurable cutoff
• Statistical Validation — Check numerical claims in research outputs for internal consistency (e.g., percentages summing to more than 100%, implausible magnitudes) and cross-source agreement
• Methodology Evaluation — Assess the research methodology quality of referenced studies: sample size adequacy, control group presence, peer-review status, and replication status
• Synthesis Generation — Produce a coherent research summary that integrates findings from all sub-questions, acknowledges uncertainty, and cites every supporting source with inline references

Citation Network Analysis#

Sophia's citation analysis libraries operate at a higher level than individual citation management — they analyze patterns across the entire citation network in the corpus to surface structural insights about the research literature.

Citation Analysis (@sophia/citation-analysis)#

Deep analysis of the academic citation landscape for documents in the corpus:

• Source Quality Assessment — Multi-factor quality scoring of academic sources: journal impact factor, author H-index, publisher reputation, retraction status, and peer-review confirmation
• Scholarly Consensus Detection — Identify claims that have high agreement across independent citations in the corpus vs. claims that remain actively debated; distinguish scientific consensus from fringe positions
• Citation Linting — Validate citation format correctness (DOI validity, author name formatting, year plausibility, publication venue consistency) across a document's entire bibliography with per-violation diagnostic codes
• Self-Citation Analysis — Detect and flag excessive self-citation patterns that may inflate apparent impact

Citation Graph Analysis (@sophia/citation-graph)#

Network-level analysis of citation relationships across the entire corpus, treating documents as nodes in a directed graph:

• Citation Network Construction — Build a directed graph where nodes are documents and edges are citation relationships; edge weights reflect citation frequency and recency
• Network Metrics — PageRank centrality (which documents are most cited by highly-cited documents), betweenness centrality (which documents bridge distinct research communities), and clustering coefficient (how tightly interconnected a research area's citations are)
• Co-Citation Clustering — Group documents that are frequently cited together into co-citation clusters; clusters correspond to research sub-fields or thematic areas without requiring manual classification
• Temporal Citation Patterns — Track how citation rates change over time; identify emerging influential papers (rapidly increasing citation count), foundational works (consistently high citation rate), and declining relevance
• Anomaly Detection — Detect unusual citation patterns that may indicate coordinated self-citation rings, citation manipulation, or systematic under-citation of relevant prior work

Web Crawling and Document Parsing#

Before documents can be ingested they must be acquired and their content extracted. These two libraries handle those responsibilities.

Web Crawling (@sophia/crawling)#

Programmatic content ingestion from approved web sources. The crawling library is designed to be a polite, auditable crawler that respects server limits while maximizing throughput.

• Domain Tier Configuration — Assign domains to rate-limit tiers (aggressive, standard, polite, conservative) based on the site's stated crawling policies and observed server sensitivity
• Token Bucket Rate Limiting — Per-domain token bucket rate limiter (a traffic shaping algorithm that allows bursts up to a configured token capacity, then enforces a steady maximum rate) prevents overloading target servers while maximizing crawl throughput
• Backoff and Retry — Configurable exponential backoff with jitter on rate-limit (HTTP 429) and server error (HTTP 5xx) responses; each retry reduces the crawl rate for the affected domain temporarily
• Sitemap Discovery — Automatically discover crawlable content from robots.txt and sitemap.xml declarations before starting a crawl
• Content Extraction — Strip navigation, headers, footers, and ads from HTML pages; extract the main content for ingestion into the document pipeline
• Crawl Result Persistence — Store raw crawl results with HTTP status, final URL after redirects, content hash, and crawl timestamp for auditability

Document Parsing (@sophia/document-parser)#

Multi-format document parsing with structural preservation. The parser's goal is to produce structured content that downstream steps (chunking, entity extraction) can work with — not just raw text.

• Format Detection — Automatically detect input format from file headers, MIME type, and content analysis; no manual format specification required
• Structure Extraction — Parse headings, subheadings, paragraphs, lists, tables, footnotes, and figure captions as first-class structural elements rather than raw text; downstream chunking and entity extraction use structure
• Citation Extraction — Identify and extract bibliographic references from document text; parse author, title, year, journal, DOI, and URL fields from inline citations and reference lists in multiple citation formats
• Header Metadata Parsing — Extract publication metadata from document headers: title, abstract, author affiliations, keywords, and publication venue

Educational Knowledge Intelligence#

Several Sophia libraries are purpose-built for educational applications. These libraries understand the concept of a curriculum, learning objectives, and pedagogical structure — not just raw document content.

Knowledge Base (@sophia/knowledge-base)#

Domain knowledge extraction and curriculum mapping for educational applications:

• Entity Extraction — Identify and classify educational domain entities within documents: concepts, skills, learning objectives, prerequisite relationships, and domain-specific vocabulary
• Relationship Extraction — Extract semantic relationships between entities: "concept A is a prerequisite for concept B", "skill X supports learning objective Y", "topic P is a subdomain of domain Q"
• Bloom's Taxonomy Classification — Classify learning objectives against Bloom's Taxonomy levels (Remember, Understand, Apply, Analyze, Evaluate, Create) for instructional design quality assessment
• Curriculum Standards Mapping — Map learning objectives to established educational standards (Common Core, CEFR, IB, and custom standard sets) for compliance verification and curriculum gap analysis
• Coherence Analysis — Measure the internal coherence of a curriculum: are prerequisite concepts covered before dependent concepts, are learning objectives aligned with assessment criteria, are difficulty progressions appropriate?

Research Engine (@sophia/research-engine)#

Multi-source research query execution with verification and synthesis. Unlike the basic Search API which handles a single query, the research engine plans and executes a full multi-source research workflow.

• Research Query Planning — Parse a research question into a structured query plan specifying which sources to consult, in what order, and with what time budget per source
• Multi-Source Execution — Execute the query plan against multiple simultaneous sources: the Sophia knowledge graph, vector index, citation database, web crawl results, and external academic APIs
• Fact Registry — Maintain a session-scoped registry of verified facts discovered during the research session; later sub-questions draw on the registry to avoid redundant lookups
• Verification Workflow — Each discovered fact goes through an automated verification workflow: find corroborating sources, check for contradictions, assess source authority; verified facts are marked CONFIRMED, contradicted facts are marked DISPUTED

Semantic Search for Education (@sophia/semantic-search)#

Educational-specific semantic search with enhanced ranking, designed for scenarios where difficulty level and learning objective relevance are as important as semantic similarity.

• Multi-Factor Ranking — Combine semantic similarity, recency, authority, difficulty level alignment, and learning objective relevance into a single ranking score; configurable weight profiles for different use cases
• Two-Layer Caching — L1 in-memory cache for exact query repeats (sub- millisecond); L2 Redis cache for near-duplicate queries (milliseconds); cache hit rates monitored and reported
• Batch Similarity Computation — Compute similarities between a query and a large candidate set in parallel using vectorized operations; orders of magnitude faster than sequential comparison for large corpora
• Eviction Policies — LRU (Least Recently Used), LFU (Least Frequently Used), and TTL (Time-to-Live) cache eviction strategies configurable per layer and per collection

Curriculum Trend Intelligence (@sophia/trend-curriculum)#

Surface emerging topics and align curricula with current developments by monitoring external signals that indicate which topics are gaining importance.

• Trend Signal Detection — Monitor citation frequency, web search volume, publication rate, and community discussion velocity for domain topics; trend signals identify which topics are growing in importance
• Learning Objective Mapping — Map trend signals to existing learning objectives; identify which objectives cover trending topics and which leave emerging areas unaddressed
• Curriculum Gap Analysis — Compare the trend signal landscape against the current curriculum to identify content gaps where emerging important topics are not yet covered
• Recommendation Generation — Produce prioritized recommendations for curriculum updates based on trend importance, gap severity, and estimated implementation effort

Cross-Domain Integration#

Sophia integrates with other Oshun domains primarily through its event system. When documents are ingested and entities are extracted, Sophia publishes events on the event bus; other domains subscribe to those events to react to new knowledge as it arrives.

Events Published (@sophia/event-publisher)#

The table below lists all 9 Sophia event types, when they are published, and which domains consume them. Default routing targets are pre-configured in @sophia/event-publisher; events without a default target are only delivered when the caller passes explicit targets.

These 9 event types are defined in @oshun/contracts (SophiaEventTypes) and published through @sophia/event-publisher. Events without a default routing target are delivered only when callers pass explicit targets.

Integration with Hathor#

Sophia serves as the research grounding layer for Hathor worldbuilding. The boundary exists because Hathor owns the semantics and creative rules of a fictional world, while Sophia owns the real-world research that informs it. Data crossing the boundary: document-ingested events (triggering Hathor to consider new real-world knowledge for lore validation) and entity-extracted events (surfacing new historical figures, places, and concepts for potential inclusion in world lore).

• Citation Grounding — Hathor elements (locations, factions, cultural practices) can be grounded in real-world research from the Sophia corpus
• Research-Based World Generation — Sophia's knowledge of real history, cultures, and traditions informs AI-generated world content for historical accuracy
• Lore Fact-Checking — Worldbuilding content is checked against the research corpus to ensure plausibility and consistency with established knowledge
• Entity Discovery — New entities identified in research documents are surfaced to Hathor for potential inclusion in world lore

Integration with Lilith#

Lilith's AI conversation features need access to a curated knowledge base to give spiritually accurate, citation-grounded responses. The boundary exists because Lilith owns the conversation and spiritual guidance UX, while Sophia owns the underlying knowledge retrieval mechanics.

• RAG for AI Conversations — Lilith's BFF knowledge routes consume Sophia's search API to ground spiritual guidance responses in the curated knowledge corpus
• Knowledge Graph for Concepts — Sophia's entity graph supplies the interconnected concept network that powers Lilith's knowledge graph feature

Platform Availability#

The table below shows which Sophia capabilities are accessible via the REST API, the TypeScript SDK, and the Workbench web UI.

Concordia Mediation Knowledge Support#

@sophia/concordia-knowledge (libs/sophia/concordia-knowledge/) provides Sophia-backed precedent and template lookup for Concordia mediation cases. Its precedent-lookup module exports a PrecedentQuery / PrecedentMatch schema pair plus scorePrecedent() and rankPrecedents() for scoring and ranking candidate precedents.

Sophia's broader Concordia role — policy and legal-template grounding, source-provenanced and citation-backed issue summaries, jurisdictional research packets, and evidence quality signals — extends this lookup capability. The boundary is clear: Sophia supplies grounded knowledge to Concordia; Concordia owns bargaining orchestration and does not depend on Sophia for its core settlement logic. Sophia does not decide settlement terms or reviewer outcomes.

Training-Data Flywheel (Phases 85–86)#

libs/sophia/training-data implements this domain's side of the ML-sovereignty data flywheel: a training-data pipeline that captures research and knowledge interactions (search sessions, RAG citations, relevance feedback) as passive training signals. Signals are consent-gated, anonymized where required, and emitted in the shared flywheel envelope that Nous dataset management (Phase 87) ingests for training and evaluation. Nous owns the training infrastructure; this domain owns what constitutes a high-quality domain signal.