# Sophia — Systems Deep Dive

> The `apps/sophia/` area: four runnable Nx applications that turn raw source
> documents into a searchable, curated research knowledge base — an ingestion
> service, a knowledge-graph service, a hybrid search/RAG API, and the React
> workbench operators use to curate it all.

## What this area is

Sophia is Oshun's research / knowledge-management domain, and `apps/sophia/`
holds its **deployable applications** (as opposed to `libs/sophia/`, which holds
the shared building blocks these apps compose — `@sophia/embeddings`,
`@sophia/vectordb`, `@sophia/indexing`, `@sophia/event-publisher`,
`@sophia/client`). All four projects carry the `scope:sophia` / `type:app` Nx
tags, and each is independently versioned with its own `package.json`,
`tsconfig`, and Vitest config.

The four apps form a pipeline. `@sophia/ingestion-app` is the front door: it
fetches documents from files, URLs, APIs, archives, and academic sources,
extracts and OCRs their text, enriches them with metadata/entities/summaries,
and chunks them for indexing. `@sophia/knowledge-graph-app` holds the
entity/relationship graph that ingestion can feed (entities → nodes, with
resolution and deduplication). `@sophia/search-api` is the retrieval surface — a
hybrid dense+sparse search and RAG service over the indexed corpus.
`@sophia/workbench` is the operator UI on top of all of it: curation, review,
annotation, document management, search, and an operator remediation console.

Three of the four (`ingestion-app`, `knowledge-graph-app`, `search-api`) are
Node services built with `@nx/js:tsc` whose `main.ts` exposes a standalone
`node:http` REST server while their `index.ts` re-exports the same machinery as
an importable library. The fourth (`workbench`) is a Vite-built React SPA
(`platform:web`) with its own Playwright e2e target. The domain that runs
through every node is research/knowledge work — note the knowledge-graph's node
types (`tradition`, `text`, `concept`, `practice`, `figure`, `place`, `event`,
`organization`) and the academic ArXiv/PubMed connectors in ingestion.

## How it fits the wider system

These apps depend downward on the `libs/sophia/` packages and on platform-shared
libs (`@oshun/logging`, and, in the workbench, `@oshun/http-client`,
`@oshun/auth-primitives`, `@oshun/config`). The ingestion pipeline emits
**cross-domain events** through `@sophia/event-publisher`
(`SophiaDocumentIngestedPayload`, `SophiaEntityExtractedPayload`), so other
Oshun domains can react to new documents and extracted entities without coupling
to Sophia's internals. The workbench is a pure client: its `ApiClient`
(`apps/sophia/workbench/src/api/client.ts`) speaks REST to these backend
services (`/api/documents`, `/api/search`, `/api/annotations`, `/api/reviews`,
`/api/operator/workbench/...`) over `fetch` with bearer-token auth, and falls
back to an in-memory local runtime when no backend is wired. The "used by" edges
on each node below show the concrete consumers.

## Entity reference

### @sophia/ingestion-app

The document-ingestion service (`apps/sophia/ingestion`). Its core is
`IngestionPipeline` (`src/pipeline/ingestion-pipeline.ts`), which orchestrates a
four-stage job — fetch → extract → enrich → chunk — emitting typed
`PipelineEvent`s and publishing `document.ingested` / `entity.extracted`
cross-domain events via `@sophia/event-publisher`. It is genuinely feature-rich:
source connectors for file/URL/API/archive plus academic ArXiv and PubMed
(`src/connectors/`), Tesseract.js OCR (`src/ocr/`), rule-based enrichment
(title/author/date/language/entity extraction and extractive summarization,
`src/enrichment/`), five real chunking strategies implemented in full — `fixed`
(with overlap), `paragraph`, `sentence`, Jaccard-keyword `semantic`, and
hierarchical `recursive` — and a Kalika knowledge-graph sync integration
(`src/integrations/`). `src/main.ts` wraps the pipeline in an
`IngestionAPIService` (`node:http`) with routes for ingest/upload (incl.
hand-rolled multipart parsing), jobs, documents, and chunks, backed by
`FileSystemDocumentStorage`/`FileSystemChunkStorage`. Fully implemented, not a
scaffold.

### @sophia/knowledge-graph-app

The knowledge-graph microservice (`apps/sophia/knowledge-graph`). It exposes
four real services from `src/index.ts`: `EntityService` (node CRUD over a
`MemoryGraphStore`), `RelationService` (typed relationships), `GraphService`
(traversal, path-finding, subgraphs), and `EntityResolver`
(`src/resolution/entity-resolver.ts`). The resolver is the substantive piece —
real entity-resolution algorithms: Levenshtein-based `stringSimilarity`, Jaccard
alias/tag overlap, substring `containsSimilarity`, weighted scoring with
confidence levels, duplicate-group discovery (`findDuplicates`), link suggestion
(`suggestLinks`), and a property-merging `merge` that rewires relationships and
supports `prefer_primary`/`prefer_secondary`/`combine` strategies plus per-field
conflict resolution. `src/main.ts` serves it as a REST API (default port 3001)
including a curator-administration surface. Honest caveat: the backing store is
in-memory (`MemoryGraphStore`), so persistence is process-lifetime; the resolver
and graph logic themselves are complete and domain-specific.

### @sophia/search-api

The search and RAG service (`apps/sophia/search-api`). `HybridSearchEngine`
(`src/search/hybrid-search.ts`) combines dense (vector) and sparse (keyword)
retrieval with two fusion methods — Reciprocal Rank Fusion (default, `rrfK=60`)
and weighted linear fusion with min-max normalization — plus dense-only and
sparse-only modes, threshold filtering, and document deduplication. The
retrieval is real, not faked: `src/search/runtime.ts` wires a
`TransformerEmbeddingProvider` backed by `@sophia/embeddings` (MiniLM-L6-v2,
384-d, in-process ONNX) and a hand-written `BM25TextIndex` implementing the
BM25+ scoring formula (`k1`, `b`, and a `delta` lower-bound), over a
`@sophia/vectordb` store selectable as memory/Qdrant/Pinecone from env. Around
the core sit a reranking stack (cross-encoder/feature/listwise pipelines,
`src/ranking/reranker.ts`), citation extraction/verification/graph
(`src/citations/`), and query-expansion / multi-query retrieval
(`src/retrieval/`). `src/main.ts` exposes it as an HTTP service. Fully
implemented.

### @sophia/workbench

The Sophia Research Workbench (`apps/sophia/workbench`) — a React + Vite SPA and
the only `platform:web` node in the area, with its own Playwright e2e suite
(`e2e/`). `src/App.tsx` defines seven routed pages: Dashboard, Curation, Review,
Annotations, Documents, Search, and an Operator workbench. Each page calls a
service in `src/api/services/`, which dispatches to either the REST `ApiClient`
(`src/api/client.ts`, a real `fetch`-based client with JWT bearer auth, timeout
handling, and typed error parsing) or an in-memory **local runtime**
(`src/api/services/local-runtime.ts`, ~1,200 lines of seeded
documents/annotations/reviews/operator state) selected by
`isLocalRuntimeEnabled()`. That local runtime is an honest dev/demo fallback so
the UI runs without live backends — not a fake stand-in for the app under test —
and the production path (`RealOperatorWorkbenchService`, etc.) goes to the real
services. Includes auth context, keyboard navigation, Tailwind styling, and
component/page unit tests. A complete frontend application.