# Content Ingestion Guide

## Overview

This document describes the content ingestion pipeline for adding wisdom texts,
meditation research, philosophy texts, and other source materials to Lilith's
Sophia-backed retrieval stack. The system supports crawling, parsing, chunking,
embedding, and indexing content for semantic search and knowledge graph
construction.

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Source Discovery](#source-discovery)
3. [Document Crawling](#document-crawling)
4. [Document Processing](#document-processing)
5. [Vector Embeddings](#vector-embeddings)
6. [Knowledge Graph Integration](#knowledge-graph-integration)
7. [Citation Metadata](#citation-metadata)
8. [Quality Assurance](#quality-assurance)

---

## Architecture Overview

### Pipeline Flow

```
Source Discovery → Crawling → Parsing → Chunking → Embedding → Storage
                                          ↓
                              Entity Extraction → Graph Storage
```

### Services Involved

| Service                | Role                                               | Location                       |
| ---------------------- | -------------------------------------------------- | ------------------------------ |
| Sophia Ingestion       | Source ingestion, extraction, chunking, enrichment | `apps/sophia/ingestion/`       |
| Sophia Search API      | Search, retrieval, and document read APIs          | `apps/sophia/search-api/`      |
| Sophia Knowledge Graph | Entity and relationship storage/querying           | `apps/sophia/knowledge-graph/` |

### Data Stores

| Store          | Technology  | Purpose                       |
| -------------- | ----------- | ----------------------------- |
| Document Store | PostgreSQL  | Full text storage, metadata   |
| Vector Store   | Milvus      | Semantic embeddings           |
| Graph Store    | Neo4j       | Knowledge graph relationships |
| Time Series    | TimescaleDB | Crawl and access metrics      |

---

## Source Discovery

### Source Service

**Location**: `apps/sophia/ingestion/src/connectors/connectors.ts`

The Source Discovery Service manages registration and discovery of content
sources.

### Source Types

| Type       | Description            | Example                                    |
| ---------- | ---------------------- | ------------------------------------------ |
| `academic` | Peer-reviewed research | JSTOR, PubMed, Google Scholar              |
| `book`     | Published books        | Project Gutenberg, Internet Archive        |
| `article`  | Web articles           | Medium, tradition websites                 |
| `website`  | Full websites          | Dharma repositories, contemplative centers |
| `blog`     | Blog content           | Meditation teachers' blogs                 |

### Source Registration

Sources are registered with quality scoring:

```typescript
interface Source {
  id: string;
  url: string;
  name: string;
  type: SourceType;
  tradition?: string; // e.g., "zen", "stoicism", "vedanta"
  qualityScore: number; // 0-100, affects ranking
  lastCrawled?: Date;
  crawlFrequency: string; // "daily", "weekly", "monthly"
  status: 'active' | 'paused' | 'archived';
}
```

### Quality Scoring Factors

| Factor        | Weight | Description                     |
| ------------- | ------ | ------------------------------- |
| Authority     | 30%    | Academic/established source     |
| Accuracy      | 25%    | Citation quality, fact-checking |
| Completeness  | 20%    | Depth of content                |
| Freshness     | 15%    | Update frequency                |
| Accessibility | 10%    | Open access vs. paywalled       |

---

## Document Crawling

### Crawler Service

**Location**: `apps/sophia/ingestion/src/pipeline/ingestion-pipeline.ts`

The Source Crawler handles fetching content from registered sources.

### Crawl Configuration

```typescript
interface CrawlRequest {
  sourceId: string;
  maxPages?: number; // Limit pages per crawl
  maxDepth?: number; // Link depth limit
  respectRobotsTxt?: boolean; // Default: true
  rateLimitMs?: number; // Delay between requests
  extractionRules?: {
    titleSelector?: string;
    contentSelector?: string;
    dateSelector?: string;
    authorSelector?: string;
  };
}
```

### Crawl Statistics

Each crawl tracks:

```typescript
interface CrawlStats {
  pagesProcessed: number;
  errorsEncountered: number;
  bytesDownloaded: number;
  entitiesExtracted: number;
  duration: number;
}
```

### Rate Limiting

| Source Type | Default Delay | Max Concurrent |
| ----------- | ------------- | -------------- |
| Academic    | 2000ms        | 2              |
| Website     | 1000ms        | 5              |
| Blog        | 500ms         | 10             |

### Deduplication

The crawler maintains a deduplication cache to avoid re-processing:

- URL normalization
- Content fingerprinting
- Last-modified header checking

---

## Document Processing

### Document Processing Service

**Location**: `services/ai/src/document-processing/service.ts`

### Supported Formats

| Format     | Parser       | Extensions         |
| ---------- | ------------ | ------------------ |
| PDF        | pdf-parse    | .pdf               |
| Office     | officeparser | .docx, .doc, .pptx |
| Markdown   | marked       | .md                |
| HTML       | Cheerio      | .html, .htm        |
| Plain Text | Native       | .txt               |
| EPUB       | epub-parser  | .epub              |

### Document Parsing

```typescript
interface ParsedDocument {
  content: string; // Extracted text
  metadata: {
    title?: string;
    authors?: string[];
    date?: Date;
    language?: string;
    pageCount?: number;
  };
  structure: {
    chapters?: ChapterInfo[];
    headings?: HeadingInfo[];
    paragraphs?: ParagraphInfo[];
  };
}
```

### Text Chunking

Documents are chunked for embedding:

| Strategy  | Chunk Size    | Overlap    | Use Case         |
| --------- | ------------- | ---------- | ---------------- |
| Sentence  | 3-5 sentences | 1 sentence | Dense content    |
| Paragraph | 1 paragraph   | None       | Structured texts |
| Fixed     | 512 tokens    | 50 tokens  | General          |
| Semantic  | Variable      | Contextual | Narrative        |

### Chunk Metadata

Each chunk retains:

```typescript
interface ChunkMetadata {
  documentId: string;
  chunkId: string;
  position: number; // Order in document
  content: string; // Chunk text
  language: string;
  tradition?: string;
  genre?: string; // meditation, philosophy, commentary
  title: string;
  authors: string[];
  license?: string;
  confidence: number; // Parsing confidence
}
```

---

## Vector Embeddings

### Search And Vector Storage

**Location**: `apps/sophia/search-api/src/main.ts`

### Collections

| Collection           | Dimensions | Index    | Metric | Purpose             |
| -------------------- | ---------- | -------- | ------ | ------------------- |
| `knowledge_chunks`   | 768        | HNSW     | COSINE | Main content        |
| `semantic_search`    | 768        | IVF_FLAT | IP     | Search optimization |
| `persona_embeddings` | 384        | HNSW     | L2     | Persona matching    |

### Embedding Pipeline

```
Chunk Text → Embedding Model → Vector (768-dim) → Milvus Storage
```

### Index Types

| Type     | Best For         | Memory | Query Speed     |
| -------- | ---------------- | ------ | --------------- |
| FLAT     | Small datasets   | High   | Fastest (exact) |
| IVF_FLAT | Medium datasets  | Medium | Fast            |
| IVF_PQ   | Large datasets   | Low    | Moderate        |
| HNSW     | Real-time search | Medium | Very Fast       |

### Vector Operations

```typescript
// Insert vectors
POST /v1/collections/:name/vectors

// Similarity search
POST /v1/collections/:name/search
{
  "vector": number[],
  "topK": 10,
  "minScore": 0.7,
  "filters": { "tradition": "zen" }
}

// Hybrid search (vector + keyword)
POST /v1/collections/:name/hybrid-search
{
  "vector": number[],
  "keywords": ["meditation", "breath"],
  "vectorWeight": 0.7,
  "keywordWeight": 0.3
}
```

---

## Knowledge Graph Integration

### Entity Extraction

During ingestion, entities are extracted for the knowledge graph:

| Entity Type | Examples                                |
| ----------- | --------------------------------------- |
| `TRADITION` | Zen, Stoicism, Vedanta                  |
| `CONCEPT`   | Mindfulness, Virtue, Dharma             |
| `PRACTICE`  | Zazen, Lectio Divina, Pranayama         |
| `FIGURE`    | Buddha, Seneca, Ramana Maharshi         |
| `TEXT`      | Dhammapada, Meditations, Yoga Sutras    |
| `PLACE`     | Bodh Gaya, Mount Athos, Zen monasteries |

### Relationship Extraction

| Relationship   | Description     | Example                                          |
| -------------- | --------------- | ------------------------------------------------ |
| `TAUGHT`       | Teacher-student | Dogen TAUGHT Ejo                                 |
| `WROTE`        | Authorship      | Seneca WROTE Letters                             |
| `BELONGS_TO`   | Categorization  | Zazen BELONGS_TO Zen                             |
| `DERIVES_FROM` | Influence       | Chan DERIVES_FROM Mahayana                       |
| `ALIGNS_WITH`  | Similarity      | Stoic acceptance ALIGNS_WITH Buddhist equanimity |
| `CONTRADICTS`  | Opposition      | Materialist CONTRADICTS Idealist                 |

### Graph Storage

**Location**: `apps/sophia/knowledge-graph/src/graph/graph-service.ts`

Entities and relationships are stored in Neo4j with:

- Full-text indexing on node properties
- Constraint enforcement on unique IDs
- Connection pooling (default 50 connections)
- Query timeout: 30 seconds

---

## Citation Metadata

### Required Citation Fields

| Field       | Required     | Description             |
| ----------- | ------------ | ----------------------- |
| `title`     | Yes          | Document title          |
| `authors`   | Yes          | Author names            |
| `source`    | Yes          | Publication/source name |
| `date`      | Recommended  | Publication date        |
| `url`       | If available | Original URL            |
| `license`   | Recommended  | Copyright/license info  |
| `tradition` | Recommended  | Associated tradition    |

### Citation Format

```json
{
  "citation": {
    "title": "Zen Mind, Beginner's Mind",
    "authors": ["Shunryu Suzuki"],
    "source": "Weatherhill",
    "date": "1970",
    "isbn": "978-0834800793",
    "license": "copyright",
    "tradition": "zen"
  }
}
```

### License Categories

| License            | Description    | Usage                |
| ------------------ | -------------- | -------------------- |
| `public_domain`    | No copyright   | Full use             |
| `creative_commons` | CC licenses    | Per license terms    |
| `fair_use`         | Limited quotes | Attribution required |
| `copyright`        | Full copyright | Metadata only        |

---

## Quality Assurance

### Ingestion Validation

| Check     | Description               | Action on Fail      |
| --------- | ------------------------- | ------------------- |
| Format    | Valid file format         | Reject              |
| Language  | Supported language        | Flag for review     |
| Length    | Minimum content length    | Skip if too short   |
| Duplicate | Content fingerprint check | Skip if duplicate   |
| Quality   | Readability score         | Flag if low quality |

### Content Quality Metrics

```typescript
interface QualityMetrics {
  readabilityScore: number; // Flesch-Kincaid
  vocabularyRichness: number; // Type-token ratio
  entityDensity: number; // Entities per 1000 words
  citationDensity: number; // Citations per 1000 words
  languageConfidence: number; // Language detection confidence
}
```

### Manual Review Queue

Content flagged for review:

- Low quality scores
- Unknown language
- Missing required metadata
- Potential copyright issues
- Entity extraction uncertainty

---

## Ingestion Workflow

### Step-by-Step Process

1. **Register Source**
   - Add source to discovery service
   - Set crawl frequency and quality score
   - Configure extraction rules

2. **Schedule Crawl**
   - Automatic based on frequency
   - Manual trigger available

3. **Fetch Content**
   - Respect robots.txt
   - Apply rate limiting
   - Track crawl statistics

4. **Parse Documents**
   - Detect format
   - Extract text and metadata
   - Preserve structure

5. **Chunk Content**
   - Apply chunking strategy
   - Maintain context overlap
   - Attach metadata

6. **Generate Embeddings**
   - Process through embedding model
   - Store in Milvus

7. **Extract Entities**
   - Identify traditions, concepts, figures
   - Extract relationships

8. **Update Graph**
   - Create/update Neo4j nodes
   - Add relationships

9. **Index for Search**
   - Full-text indexing (PostgreSQL)
   - Vector indexing (Milvus)

10. **Quality Check**
    - Validate metadata
    - Score quality metrics
    - Route to review if needed

### Monitoring

| Metric             | Description         | Alert Threshold    |
| ------------------ | ------------------- | ------------------ |
| Crawl success rate | % successful crawls | < 90%              |
| Parse success rate | % successful parses | < 95%              |
| Embedding latency  | Time to embed       | > 5s per chunk     |
| Index latency      | Time to index       | > 10s per document |

---

## API Reference

### Ingestion Endpoints

| Method | Endpoint                      | Description         |
| ------ | ----------------------------- | ------------------- |
| POST   | `/v1/sources`                 | Register new source |
| GET    | `/v1/sources`                 | List sources        |
| POST   | `/v1/sources/:id/crawl`       | Trigger crawl       |
| GET    | `/v1/sources/:id/status`      | Crawl status        |
| POST   | `/v1/documents/upload`        | Direct upload       |
| GET    | `/v1/documents/:id`           | Get document        |
| POST   | `/v1/documents/:id/reprocess` | Reprocess document  |

---

## Related Documents

- [Knowledge Graph Documentation](./KNOWLEDGE_GRAPH.md)
- [Content Guidelines](./CONTENT_GUIDELINES.md)
- [Semantic Search Guide](../reference/search-api.md)

---

_Last updated: January 2, 2026_
