# Sophia — Systems Deep Dive

> The `libs/sophia/` area: ~27 Nx libraries that make up Sophia, the Oshun
> **Knowledge Engine** — the research, ingestion, indexing, retrieval,
> knowledge-graph, citation/credibility/verification, and evaluation stack that
> turns raw sources into grounded, cited, fact-checked knowledge.

## What this area is

Sophia is Oshun's knowledge and research domain. Its libraries form a pipeline
that takes documents from the open web (or tenant uploads), parses and chunks
them, embeds and indexes them, retrieves over them semantically, links the
extracted entities into a knowledge graph, scores the sources for credibility,
verifies claims against evidence, and finally evaluates the whole pipeline's
groundedness before anything is published. The `@sophia/event-publisher` barrel
names the domain directly: "Sophia (Knowledge Engine) domain." Several of the
heavier libraries note in their headers that they were ported from the
Metis/Minerva research subsystems, so the area is a consolidation of prior
research tooling into one domain.

The libraries split into recognisable layers. **Foundation:** `@sophia/schemas`
(Zod domain schemas + branded IDs) and `@sophia/database` (the Prisma client).
**Acquisition:** `sophia-crawling`, `sophia-document-parser`,
`@sophia/ingestion`, `@sophia/corpus`. **Representation & retrieval:**
`@sophia/embeddings`, `@sophia/indexing`, `@sophia/vectordb`,
`sophia-semantic-search`, `@sophia/knowledge-graph`, `sophia-knowledge-base`.
**Trust & analysis:** `sophia-credibility`, `@sophia/citation-analysis`,
`sophia-citation-graph`, `@sophia/verification`, `@sophia/evaluation`.
**Research orchestration:** `@sophia/agents`, `sophia-research-engine`,
`@sophia/theory`. **Applied/edge:** `@sophia/client`, `@oshun/sophia-client`,
`@sophia/event-publisher`, `@sophia/predictions`, `sophia-trend-curriculum`,
`@sophia/cultural-research`, `@sophia/concordia-knowledge`.

Each project is a `scope:sophia` Nx library (most tagged `layer:domain`) built
with `@nx/js:tsc` and tested with Vitest, as seen in `agents/project.json`. The
package names are deliberately mixed: most use the `@sophia/*` npm scope, a
handful keep bare Nx project names (`sophia-crawling`, `sophia-credibility`,
`sophia-citation-graph`, `sophia-document-parser`, `sophia-knowledge-base`,
`sophia-research-engine`, `sophia-semantic-search`, `sophia-trend-curriculum`),
and two cross-scope deliberately (`@oshun/sophia-client`). The headings below
use each project's exact `project.json` `"name"`.

## How it fits the wider system

The outward face of the area is `@sophia/client` (re-exported as
`@oshun/sophia-client`): a typed SDK with search, ingestion, and knowledge-graph
sub-clients plus fluent query builders, generated from an OpenAPI surface in
`client/src/generated/openapi.ts`. Other Oshun domains consume Sophia two ways:
through that client SDK over HTTP, and through direct in-process imports of the
analysis libraries. `@sophia/concordia-knowledge` is an explicit cross-domain
seam — Concordia (Phase 179.7) calls into Sophia's precedent lookup — and
`semantic-search` ships an `iris-adapter` so the Iris assistant can retrieve
over Sophia's corpus. `@sophia/event-publisher` is how Sophia announces state
changes (documents ingested, indexes rebuilt, entities extracted) onto the
platform event bus for downstream consumers. The honesty boundaries worth noting
up front: `@sophia/database` fails loud until Prisma is generated,
`@sophia/embeddings` throws `NotConfiguredError` rather than fabricate a vector
when the model can't load, and `@sophia/cultural-research`'s AI analysis sits
behind a provider interface with mock providers for local development.

## Entity reference

### @sophia/agents

The advanced research-agent framework (`agents/src`), ported from Metis/Minerva.
It composes a full research pipeline: `QueryDecomposer` (with scientific/simple
variants), `AdvancedResearchAgent` over a pluggable `SearchProvider`
(`InMemorySearchProvider` included), and a verification suite —
`FactCheckingAgent`, `ContradictionDetector`, `TemporalConsistencyChecker`,
`StatisticalValidator`, `MethodologyEvaluator` — all wired together by
`ResearchOrchestrator`. Each agent ships strict/lenient/quick factory variants
(`agents/src/verification/*`, ~3.4K LOC; the whole `@sophia/agents` lib is
~6.6K), and the public surface is the barrel in `agents/src/index.ts`.

### @sophia/citation-analysis

Citation analysis ported from the Metis CitationAnalyzer
(`citation-analysis/src`). Four analyzers: `SourceQualityAnalyzer` (with
STEM/humanities variants, multi-dimensional `QualityDimension` scoring and
`SourceTier` classification), `CitationNetworkAnalyzer` (authority scores,
co-citation, bibliographic coupling, clustering), `CitationLinter` (rule-based
lint with severity levels), and `ScholarlyConsensusAnalyzer` (per-claim stance
aggregation, agreement matrices, contradictions, temporal trends). Real,
config-driven implementations under `source-quality/`, `network/`, `linters/`,
and `consensus/`.

### sophia-citation-graph

A larger, standalone citation-network library (`citation-graph/src`, ~6K LOC of
implementation across five modules). `CitationGraphBuilder` constructs the graph
(DOI parsing, author/title normalization, dedup, field/year filtering);
`NetworkAnalyzer` computes degree/in/out, betweenness, closeness, PageRank
(damping/convergence configurable), clustering coefficient, density, diameter,
reciprocity, and connected components; `CoCitationAnalyzer` builds co-citation
and coupling matrices and clusters by similarity; `TemporalAnalyzer` computes
citation half-life, velocity, growth, bursts, dormant/emerging papers; and
`AnomalyDetector` flags citation rings, spikes, self-citation excess, orphans,
and reciprocal-citation patterns. The exported helpers are real graph
algorithms, not wrappers.

### @sophia/client

The TypeScript client SDK for Sophia's HTTP services (`client/src`). A unified
`SophiaClient` aggregates `SearchClient`, `IngestionClient`, and
`KnowledgeGraphClient`, each with fluent builders (`SearchQueryBuilder`,
`GraphQueryBuilder`) and a shared `HttpClient` with retries/error handling
(`SophiaClientError`). The branded ID types and request/response shapes mirror a
generated OpenAPI surface (`client/src/generated/openapi.ts`, ~1.7K lines).
`createLocalSophiaClient` targets local dev. This is the canonical way other
domains call Sophia over the wire.

### @sophia/concordia-knowledge

A small, focused cross-domain library (`concordia-knowledge/src`,
`precedent-lookup.ts`, ~142 lines) implementing §179.7: Sophia-backed precedent
lookup for Concordia cases. It defines `PrecedentQuerySchema` /
`PrecedentMatchSchema` (Zod, with a fixed `useCaseClass` enum and ISO-country
jurisdiction validation) and a deterministic, documented bounded-heuristic
`scorePrecedent` (a weighted average of jurisdiction match, family overlap, and
keyword overlap, each in `[0,1]`) plus `rankPrecedents`. The determinism is
intentional so "audit reproducibility holds across runs."

### @sophia/corpus

The educational-asset corpus (`corpus/src`): `EducationalImageCorpus`, a
`DiagramTemplateLibrary`, and `CorpusMaintenanceManager` (each ~1.1–1.3K LOC).
It manages curated educational images and diagram templates plus the maintenance
operations over that collection. Substantial real implementations behind the
`types.ts` surface; the barrel re-exports all three modules.

### sophia-crawling

The web-acquisition layer (`crawling/src`). Five real subsystems: `RateLimiter`
(token-bucket with domain tiers, configurable backoff strategies, polite vs
aggressive presets), the crawl frontier (URL normalization, tracking-param
stripping, a hand-rolled Bloom filter with optimal-size calculation, priority
sorting), file detection (magic-byte signatures, format/category
classification), compliance checking (~1.4K LOC: robots.txt directives,
license/ToS, data-usage policy), and URL-pattern classification (including
curriculum-seed source types). Domain-specific crawling logic, not a thin fetch
wrapper.

### sophia-credibility

Source-trust analysis (`credibility/src`). Five analyzers: `source-credibility`
(authority levels, accuracy results, citation metrics, peer-review
verification), `quality-assessment` (~1.2K LOC, multi-dimension quality scoring
with issue severity/category), `claim-extraction` (typed claims from text),
`bias-detection` (bias type/impact indicators), and `contradiction-resolution`
(contradiction typing with resolution strategies and consensus levels). The
`types.ts` surface defines the branded IDs and config types; implementations are
real scoring logic.

### @sophia/cultural-research

Cultural-sensitivity, localization, and accessibility for Sophia content
(`cultural-research/src`). The largest single file in the area
(`consultant/cultural-consultant.ts`, ~4.6K LOC) plus a ~3K-LOC
`LocalizationManager`, a `~1.7K`-LOC `AccessibilityManager`, and a
`CulturalDatabase`. The AI analysis is structured around a `CulturalAIProvider`
interface (sensitivity checks, suggestions, representation analysis) with real
orchestration logic and `Mock*Provider` implementations
(`MockCulturalAIProvider`, etc.) plus `createMock*` factories for local
development — an honest provider seam, not fabricated AI output.

### @sophia/database

The Prisma data-access library (`database/src`). It is a thin, honest wrapper
over a generated Prisma client: `getSophiaClient` is a lazily-initialized
singleton, `createSophiaClient` makes fresh connections, and both **fail loud**
— they throw "Prisma client not generated. Run
`nx run @sophia/database:prisma:generate` first." rather than pretend a client
exists. The generated client under `database/src/generated/` is a build
artifact, so the checked-in source is the client accessors plus model/enum
re-exports.

### sophia-document-parser

Document parsing and normalization (`document-parser/src`). Seven modules:
`format-detection` (extension + content heuristics, `SupportedFormat`),
`text-extraction` (with language detection), `structure-analysis` (headers,
sections), `metadata-extraction` (~885 LOC), `citation-extraction` (~881 LOC:
typed citations, bibliography parsing in multiple styles, validation/issues),
`content-chunking`, and a top-level `document-processor` that ties them into a
`DocumentProcessingResult`. Real parsing logic with branded IDs from `types.ts`.

### @sophia/embeddings

Real, model-backed embeddings (`embeddings/src`). The header is explicit that
this **replaced a former SHA-256 hash "embedding" stub**:
`TransformersTextBackend` runs `sentence-transformers/all-MiniLM-L6-v2`
in-process via onnxruntime through `@huggingface/transformers` (384-d,
mean-pooled, L2-normalized so dot product == cosine), with **no hash fallback**
— it throws `NotConfiguredError` when the model can't load. Also ships an
`image-embedder`, `multimodal-embedder`, a `clip-multimodal-backend`, a
`dense-retriever`, and an `embedding-cache`. This is a genuine fail-loud
inference seam.

### @sophia/evaluation

The quality-gate library (`evaluation/src`). `RetrievalEvaluator`
(faithfulness/relevancy, precision/recall, ~1K LOC), `CitationChecker` (quote
accuracy, entailment, attribution), and `GroundednessScorer` (claim extraction +
evidence matching + hybrid semantic/lexical scoring against a configurable
threshold). Plus benchmark runners — `sophia-release-gate-benchmark` (~1.1K LOC)
and `veritas-explainer-benchmark` — an `answer-regrounding` module, and an
`ingestion-quality` evaluator with remediation/routing. The AI/semantic steps
sit behind `AIEvaluationProvider`/`SemanticSimilarityProvider` interfaces.

### @sophia/event-publisher

Type-safe domain-event publishing for Sophia (`event-publisher/src`). A
`SophiaEventPublisher` (singleton accessor + factory + reset for tests) emits
the full Sophia event taxonomy as typed payloads: document
ingested/updated/deleted, index updated/rebuilt, search performed, entity
extracted, relation discovered, citation created. The payload types in
`types.ts` are the contract other domains subscribe to on the bus.

### @sophia/indexing

The indexing pipeline (`indexing/src`): document chunking → embedding → vector
index. Chunking ships multiple `strategies` over a `base-chunker`; the
`embedding/` module has providers, a cache, a service, and types, while a
parallel `embeddings/` module adds audio/text/multimodal embedders, a backend
abstraction, and a `cutover` path; `graphs/` carries an `entity-resolver` and
`graph-visualization`; and `index/` has the `builder`, `similarity`, and types.
`createIndexBuilder` indexes a document and exposes search. Large, real
implementations (entity-resolver ~621 LOC, graph-visualization ~888 LOC).

### @sophia/ingestion

The six-stage ingestion pipeline (`ingestion/src`, §9.9) with provenance,
idempotency, and tenant isolation. The standout is a rich `adapters/` registry:
peer-reviewed PDF, news-article HTML, RSS/Atom feed, audio- and
video-transcript, IIIF manuscript, LMS SCORM/xAPI, tenant BYOM bundle, and a
`structured-api-base`, all behind a typed `registry` with a `release-gate`. Plus
`chunking/` (policy + strategies), `enrichment/` (classifiers + pipeline), a
`lifecycle/` subsystem (cascade, retirement, scheduler, claim-diff), a
`pipeline/` with events, `provenance/`, and `tenant/isolation`. Real ingestion
machinery, not placeholders.

### sophia-knowledge-base

The educational knowledge base (`knowledge-base/src`). Five large modules:
`entity-extraction` (~1.8K LOC), `relationship-extraction` (~1.1K LOC),
`standards-mapping` (~1.4K LOC: curriculum standard alignment, coverage
analysis), `coherence-analysis` (~1K LOC: prerequisite/progression analysis,
coherence scoring/issues), and a `knowledge-store` (~952 LOC) with query results
and statistics. Branded `KBEntityId`/`KBRelationshipId`/`StandardId` types
anchor the domain in `types.ts`.

### @sophia/knowledge-graph

A directed labeled property-graph engine (`knowledge-graph/src`), documented in
detail in its barrel. `GraphStore` (CRUD, adjacency, filtering, events);
`GraphTraverser` (BFS, DFS, Dijkstra shortest path, all-paths, subgraph,
neighborhood); graph `algorithms` (degree/betweenness/closeness/PageRank/
eigenvector centrality, LPA + Louvain community detection, node similarity);
`ConceptGraph` (prerequisite DAG with topological sort, cycle detection,
critical path, level assignment); a `gap-analyzer`; learning-path generation
(`path-optimizer`); a fluent `query-builder`; graph-aware retrieval; and an NLP
`relationship-mapper`. Genuine algorithm implementations, not stubs.

### @oshun/sophia-client

A one-line namespace re-export (`oshun-client/src/index.ts` is
`export * from '@sophia/client';`). It exists to expose the Sophia client SDK
under the platform-wide `@oshun/*` scope so cross-domain consumers can import it
without reaching into the `@sophia/*` scope. No logic of its own — intentionally
a thin alias, honestly described as such.

### @sophia/predictions

Predictive analytics for streaming platforms (`predictions/src`). Three modules:
revenue forecasting (`forecast/revenue.ts`, ~644 LOC — time-series with
configurable confidence level, minimum data points, seasonality detection, peak
periods, risk factors, recommendations), a schedule `optimizer` (~725 LOC), and
a content-performance `predictor` (~1K LOC). Statistical forecasting methods
over `TimeSeries`/`ForecastResult` types in `types.ts`. Note this is the most
adjacent-to-streaming library in the area rather than core research tooling.

### sophia-research-engine

The end-to-end research orchestration engine (`research-engine/src`, ~7.5K LOC).
Five large modules: `query-planning` (~1.6K LOC: decomposition, strategies),
`knowledge-extraction` (~1.7K LOC: facts, entities, relationships),
`fact-verification` (~1.3K LOC: verification results, consistency,
contradictions), `source-scoring` (~1K LOC), and `research-orchestration` (~1.2K
LOC) tying them into `ResearchSession`/`ResearchResult`. Overlaps conceptually
with `@sophia/agents` but is the lower-level engine; both are real.

### @sophia/schemas

The Sophia domain-schema foundation (`schemas/src`). Zod schemas and branded ID
types for research documents, citations, claims, knowledge packs, entities, and
relations, plus a typed error surface (`errors.ts`) and ID helpers (`ids.ts`).
Each domain object gets its own file (`citation.ts`, `claim.ts`, `entity.ts`,
`relation.ts`, `research-document.ts`, `knowledge-pack.ts`), re-exported from
`index.ts`. This is the shared vocabulary the rest of the area validates
against.

### sophia-semantic-search

The educational semantic-search stack (`semantic-search/src`), the largest
multi-module library here. `BM25` sparse retrieval (configurable k1/b, field
weights, stop-words, highlights), `hierarchical` and `hybrid` retrieval (with
reciprocal-rank fusion), multi-factor `result-ranking`, a two-layer
`search-cache`, `search-config`, a ~1.4K-LOC `search-orchestration`,
`structured` and `tool-mediated` retrieval, `retrieval-labels`
(provenance/source- scope enforcement), `source-set-invalidation`, and an
`iris-adapter` for the Iris assistant. The `educational-search` module alone is
~2K LOC. Real IR implementations.

### @sophia/theory

Research-methodology frameworks and citation integrity (`theory/src`).
`CitationManager` (~2.6K LOC: claim detection, multi-style citation formatting —
chicago-notes and others — verification, and a graph provider) and
`ResearchManager` (~1.8K LOC: projects, hypotheses, methodology). It integrates
the `@kalika/citations` package via `kalika-citation-storage.ts`
(`InMemoryCitationDatabase` + storage provider), so Sophia's citation store is
backed by the shared Kalika citation engine.

### sophia-trend-curriculum

Curriculum-vs-trend analysis (`trend-curriculum/src`). `TrendMapper` (~1.6K LOC,
mapping learning objectives to trend signals, surfacing unmapped trends and
orphaned objectives), a `CurriculumProfiler` (field/Bloom distributions,
coverage gaps, skew analysis), a `RelevanceScorer` (weighted relevance
breakdown), an `objective-trend` analyzer, and a `CurriculumRecommender` (~1.7K
LOC producing recommendation reports). Typed events and branded IDs in
`types.ts`; real scoring and mapping logic.

### @sophia/vectordb

The vector-store abstraction (`vectordb/src`). A common store interface with
four backends — an in-memory `memory-store` (~619 LOC), and real adapters for
`milvus` (~850 LOC), `pinecone` (~819 LOC), and `qdrant` (~842 LOC) — plus
collection `schemas` (e.g. `KNOWLEDGE_CHUNKS_SCHEMA`), `similarity` utilities,
batch helpers, and a `health` module. Supports upsert, vector search with
structured filters, and hybrid search; `createMemoryStore` for dev,
`createMilvusStoreFromEnv` for production. Genuine multi-provider adapters.

### @sophia/verification

The claim-verification pipeline suite (`verification/src`). Independent
pipelines for fact-checking (`fact-checking-pipeline`, plus a `fact-check-loop`
~1.2K LOC), contradiction handling (`contradiction-loop` +
`contradiction-pipeline`), `methodology`, `statistical`, and `temporal`
verification, and an `unsupported-claim-loop` (~1.1K LOC). Orchestration is
real: `job-envelope` (`SophiaGroundingJobEnvelope`) and a `publication-gate`
that runs concrete checks (claim-density, citation-integrity,
contradiction/unsupported backlog, evaluation-thresholds, reviewer-signoff) and
hashes evidence with SHA-256 (`@noble/hashes`) — a genuine block/pass gate, not
a rubber stamp.

### @sophia/training-data

Phase 85–86 flywheel producer for knowledge/research
(`libs/sophia/training-data/src`): `SophiaTrainingDataPipeline` normalizes
thirteen `SophiaTrainingKind` signals — `search-relevance`,
`citation-verification`, `rag-quality`, `chunking-quality`,
`knowledge-graph-extraction`, `source-credibility`, `prediction-calibration`,
`theory-evaluation`, and peers — into `SophiaTrainingRecord`s gated by an
explicit `SophiaGovernanceGrant` before reaching the `SophiaTrainingSink`.