# Lilith Algorithms & Technical Approaches

This document describes non-trivial algorithms used throughout the Lilith
platform, including their rationale, implementation details, and performance
characteristics.

## Table of Contents

1. [RAG Pipeline](#1-rag-pipeline)
2. [Multi-Agent Orchestration](#2-multi-agent-orchestration)
3. [Citation Extraction](#3-citation-extraction)
4. [Persona Style Adherence](#4-persona-style-adherence)
5. [Voice Activity Detection](#5-voice-activity-detection)
6. [Conflict Resolution (CRDT)](#6-conflict-resolution-crdt)
7. [Content Moderation Scoring](#7-content-moderation-scoring)
8. [Semantic Chunking](#8-semantic-chunking)
9. [Cross-Device Sync](#9-cross-device-sync)
10. [Brainwave Entrainment](#10-brainwave-entrainment)

---

## 1. RAG Pipeline

### Overview

Retrieval-Augmented Generation (RAG) grounds LLM responses in source material
from the knowledge base, ensuring accuracy and enabling citations.

### Algorithm

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           RAG PIPELINE                                       │
│                                                                             │
│   Query ──> Query Expansion ──> Hybrid Retrieval ──> Re-ranking            │
│                                      │                    │                 │
│                                      ▼                    ▼                 │
│                              ┌─────────────┐      ┌─────────────┐          │
│                              │   BM25      │      │   Dense     │          │
│                              │  (Lexical)  │      │ (Semantic)  │          │
│                              └──────┬──────┘      └──────┬──────┘          │
│                                     │                    │                 │
│                                     └────────┬───────────┘                 │
│                                              ▼                             │
│                                     ┌─────────────┐                        │
│                                     │  Reciprocal │                        │
│                                     │ Rank Fusion │                        │
│                                     └──────┬──────┘                        │
│                                            │                               │
│                                            ▼                               │
│                                     ┌─────────────┐                        │
│                                     │Cross-Encoder│                        │
│                                     │  Re-ranker  │                        │
│                                     └──────┬──────┘                        │
│                                            │                               │
│                                            ▼                               │
│                                      Top-K Chunks ──> LLM Context          │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Implementation Details

**1. Query Expansion**

```python
def expand_query(query: str, tradition: list[str]) -> list[str]:
    """Generate query variants for better recall."""
    expanded = [query]

    # Add tradition-specific synonyms
    for term in extract_concepts(query):
        if term in SPIRITUAL_SYNONYMS:
            for synonym in SPIRITUAL_SYNONYMS[term]:
                expanded.append(query.replace(term, synonym))

    # Add tradition filter queries
    for t in tradition:
        expanded.append(f"{query} in {t} tradition")

    return expanded
```

**2. Hybrid Search with RRF**

```python
def hybrid_search(query: str, k: int = 20) -> list[Chunk]:
    """Combine BM25 and dense retrieval with reciprocal rank fusion."""

    # BM25 lexical search
    bm25_results = bm25_index.search(query, k=k*2)

    # Dense semantic search
    query_embedding = embed(query)
    dense_results = vector_db.search(query_embedding, k=k*2)

    # Reciprocal Rank Fusion
    scores = defaultdict(float)
    RRF_K = 60  # Constant from literature

    for rank, doc in enumerate(bm25_results):
        scores[doc.id] += 1 / (RRF_K + rank + 1)

    for rank, doc in enumerate(dense_results):
        scores[doc.id] += 1 / (RRF_K + rank + 1)

    # Sort by fused score
    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return [get_chunk(doc_id) for doc_id, _ in ranked[:k]]
```

**3. Cross-Encoder Re-ranking**

```python
def rerank(query: str, chunks: list[Chunk], k: int = 5) -> list[Chunk]:
    """Re-rank with cross-encoder for precision."""
    pairs = [(query, chunk.text) for chunk in chunks]
    scores = cross_encoder.predict(pairs)

    ranked = sorted(zip(chunks, scores), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, _ in ranked[:k]]
```

### Rationale

- **Hybrid Search**: Combines lexical matching (exact terms like "anatta") with
  semantic understanding
- **RRF**: Outperforms simple score combination; robust to score scale
  differences
- **Re-ranking**: Cross-encoders are more accurate but slower; applied only to
  top candidates
- **Query Expansion**: Critical for spiritual domain with many synonyms across
  traditions

### Performance

| Metric        | Value |
| ------------- | ----- |
| Recall@10     | 0.92  |
| MRR           | 0.85  |
| Latency (p50) | 150ms |
| Latency (p95) | 350ms |

---

## 2. Multi-Agent Orchestration

### Overview

Multiple specialized AI agents collaborate to produce responses: Librarian
(retrieval), Teacher (instruction), Safety (moderation), Narrator (SSML).

### Algorithm

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                      MULTI-AGENT ORCHESTRATION                              │
│                                                                             │
│  User Query ──> Intent Classification ──> Agent Selection                  │
│                                                                             │
│  Parallel Execution:                                                        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                        │
│  │  Librarian  │  │   Teacher   │  │   Safety    │                        │
│  │  (Sources)  │  │ (Response)  │  │  (Filter)   │                        │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                        │
│         │                │                │                                │
│         └────────────────┼────────────────┘                                │
│                          ▼                                                 │
│                  ┌─────────────┐                                           │
│                  │  Synthesis  │                                           │
│                  │   Agent     │                                           │
│                  └──────┬──────┘                                           │
│                         │                                                  │
│                         ▼                                                  │
│                  ┌─────────────┐                                           │
│                  │  Narrator   │                                           │
│                  │  (SSML)     │                                           │
│                  └─────────────┘                                           │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Implementation

```python
class AgentOrchestrator:
    def __init__(self):
        self.agents = {
            'librarian': LibrarianAgent(),
            'teacher': TeacherAgent(),
            'safety': SafetyAgent(),
            'narrator': NarratorAgent(),
        }

    async def process(self, query: str, persona: str, context: dict) -> Response:
        # Phase 1: Parallel information gathering
        librarian_task = self.agents['librarian'].search(query)
        safety_task = self.agents['safety'].pre_check(query)

        sources, safety_flags = await asyncio.gather(
            librarian_task, safety_task
        )

        if safety_flags.block:
            return self.safety_response(safety_flags)

        # Phase 2: Teacher generates response with sources
        teacher_response = await self.agents['teacher'].respond(
            query=query,
            persona=persona,
            sources=sources,
            context=context
        )

        # Phase 3: Post-moderation
        moderated = await self.agents['safety'].post_check(teacher_response)

        # Phase 4: SSML preparation (if audio requested)
        if context.get('audio_requested'):
            ssml = await self.agents['narrator'].prepare_ssml(moderated)
            return Response(text=moderated.text, ssml=ssml, citations=sources)

        return Response(text=moderated.text, citations=sources)
```

### Rationale

- **Separation of Concerns**: Each agent specializes in one task
- **Parallel Execution**: Reduces latency by running independent agents
  concurrently
- **Safety First**: Pre and post moderation ensures content safety
- **Extensibility**: New agents can be added without changing orchestration
  logic

---

## 3. Citation Extraction

### Overview

Extract and validate citations from LLM-generated content, mapping claims to
source documents.

### Algorithm

```python
def extract_citations(
    response: str,
    sources: list[Chunk],
    min_confidence: float = 0.7
) -> list[Citation]:
    """Extract citations by matching response claims to source chunks."""

    citations = []

    # Split response into claim sentences
    claims = split_into_claims(response)

    for claim in claims:
        claim_embedding = embed(claim)

        # Find best matching source
        best_match = None
        best_score = 0.0

        for source in sources:
            # Compute semantic similarity
            similarity = cosine_similarity(claim_embedding, source.embedding)

            # Boost if lexical overlap exists
            overlap_boost = compute_lexical_overlap(claim, source.text)
            score = similarity * 0.7 + overlap_boost * 0.3

            if score > best_score:
                best_score = score
                best_match = source

        if best_score >= min_confidence and best_match:
            citations.append(Citation(
                claim=claim,
                source=best_match,
                confidence=best_score,
                span=find_span(response, claim)
            ))

    return citations
```

### Claim Splitting

```python
def split_into_claims(text: str) -> list[str]:
    """Split text into individual factual claims."""
    sentences = sent_tokenize(text)
    claims = []

    for sentence in sentences:
        # Skip non-factual sentences (questions, opinions markers)
        if is_question(sentence) or has_opinion_marker(sentence):
            continue

        # Split compound sentences
        if ' and ' in sentence or '; ' in sentence:
            subclaims = split_compound(sentence)
            claims.extend(subclaims)
        else:
            claims.append(sentence)

    return claims
```

### Rationale

- **Claim-level granularity**: More accurate than document-level matching
- **Hybrid scoring**: Combines semantic (meaning) and lexical (exact terms)
  matching
- **Confidence thresholds**: Only high-confidence citations are shown to users

---

## 4. Persona Style Adherence

### Overview

Ensure LLM responses match the expected style, tone, and teaching approach of
each persona.

### Algorithm

```python
class PersonaStyleEnforcer:
    def __init__(self, persona_config: dict):
        self.style_vector = load_style_vector(persona_config['key'])
        self.teaching_style = persona_config['style_config']['teaching_style']
        self.tradition = persona_config['tradition']

    def score_adherence(self, response: str) -> float:
        """Score how well response matches persona style."""

        # Compute response style embedding
        response_style = compute_style_embedding(response)

        # Compare to persona style vector
        style_similarity = cosine_similarity(response_style, self.style_vector)

        # Check teaching style markers
        style_markers = self.get_style_markers()
        marker_score = sum(1 for m in style_markers if m in response.lower()) / len(style_markers)

        # Check tradition authenticity
        tradition_score = self.check_tradition_authenticity(response)

        # Weighted combination
        return (
            style_similarity * 0.4 +
            marker_score * 0.3 +
            tradition_score * 0.3
        )

    def get_style_markers(self) -> list[str]:
        """Get linguistic markers for teaching style."""
        markers = {
            'socratic': ['what do you think', 'have you considered', 'why might'],
            'direct': ['the key is', 'simply', 'directly'],
            'metaphorical': ['like', 'as if', 'imagine'],
            'experiential': ['try this', 'notice', 'feel'],
        }
        return markers.get(self.teaching_style, [])
```

### Style Vectors

128-dimensional embeddings trained on exemplar responses from each persona:

| Dimension | Meaning             |
| --------- | ------------------- |
| 0-31      | Formality level     |
| 32-63     | Teaching directness |
| 64-95     | Use of metaphor     |
| 96-127    | Emotional warmth    |

---

## 5. Voice Activity Detection

### Overview

Detect when users start/stop speaking for voice chat barge-in and turn-taking.

### Algorithm

```python
class VoiceActivityDetector:
    def __init__(
        self,
        sample_rate: int = 16000,
        frame_ms: int = 30,
        threshold_db: float = -35,
        hangover_frames: int = 10
    ):
        self.sample_rate = sample_rate
        self.frame_size = int(sample_rate * frame_ms / 1000)
        self.threshold_db = threshold_db
        self.hangover_frames = hangover_frames

        self.speech_frames = 0
        self.silence_frames = 0
        self.is_speech = False

    def process_frame(self, audio_frame: np.ndarray) -> bool:
        """Process audio frame and return speech/silence state."""

        # Compute frame energy in dB
        energy = 10 * np.log10(np.mean(audio_frame ** 2) + 1e-10)

        if energy > self.threshold_db:
            self.speech_frames += 1
            self.silence_frames = 0

            # Speech onset after 2 consecutive frames
            if self.speech_frames >= 2:
                self.is_speech = True
        else:
            self.silence_frames += 1

            # Speech offset after hangover period
            if self.silence_frames >= self.hangover_frames:
                self.speech_frames = 0
                self.is_speech = False

        return self.is_speech
```

### Barge-In Detection

```python
class BargeInDetector:
    def __init__(self, vad: VoiceActivityDetector):
        self.vad = vad
        self.tts_playing = False
        self.barge_in_threshold_frames = 5
        self.consecutive_speech = 0

    def check_barge_in(self, audio_frame: np.ndarray) -> bool:
        """Check if user is interrupting TTS playback."""
        if not self.tts_playing:
            return False

        is_speech = self.vad.process_frame(audio_frame)

        if is_speech:
            self.consecutive_speech += 1
            if self.consecutive_speech >= self.barge_in_threshold_frames:
                return True
        else:
            self.consecutive_speech = 0

        return False
```

---

## 6. Conflict Resolution (CRDT)

### Overview

Conflict-free resolution for offline edits syncing across devices using CRDTs.

### Algorithm: LWW-Register for Preferences

```python
class LWWRegister:
    """Last-Writer-Wins Register for preference sync."""

    def __init__(self):
        self.value = None
        self.timestamp = 0
        self.node_id = generate_node_id()

    def update(self, new_value, timestamp: int = None):
        ts = timestamp or time.time_ns()
        if ts > self.timestamp:
            self.value = new_value
            self.timestamp = ts

    def merge(self, other: 'LWWRegister'):
        """Merge with another register, keeping latest."""
        if other.timestamp > self.timestamp:
            self.value = other.value
            self.timestamp = other.timestamp
        elif other.timestamp == self.timestamp:
            # Tie-break by node_id for determinism
            if other.node_id > self.node_id:
                self.value = other.value
```

### Algorithm: G-Counter for Progress

```python
class GCounter:
    """Grow-only counter for progress tracking."""

    def __init__(self):
        self.counts = {}  # node_id -> count
        self.node_id = generate_node_id()

    def increment(self, amount: int = 1):
        self.counts[self.node_id] = self.counts.get(self.node_id, 0) + amount

    def value(self) -> int:
        return sum(self.counts.values())

    def merge(self, other: 'GCounter'):
        """Merge by taking max for each node."""
        for node_id, count in other.counts.items():
            self.counts[node_id] = max(self.counts.get(node_id, 0), count)
```

### Rationale

- **CRDTs**: Guarantee eventual consistency without coordination
- **LWW**: Simple and effective for user preferences
- **G-Counter**: Perfect for monotonic values like meditation minutes

---

## 7. Content Moderation Scoring

### Overview

Multi-signal scoring system for content safety with tradition-aware context.

### Algorithm

```python
class ModerationScorer:
    def __init__(self):
        self.toxicity_model = load_model('toxicity')
        self.nsfw_model = load_model('nsfw')
        self.tradition_context = load_tradition_rules()

    def score(self, content: str, tradition: str = None) -> ModerationResult:
        # Base toxicity score
        toxicity = self.toxicity_model.predict(content)

        # NSFW/adult content score
        nsfw = self.nsfw_model.predict(content)

        # Medical/legal disclaimer detection
        requires_disclaimer = self.check_disclaimer_needed(content)

        # Tradition-specific allowances
        tradition_adjustment = 0.0
        if tradition:
            tradition_adjustment = self.tradition_context.get_adjustment(
                content, tradition
            )

        # Compute final scores
        final_toxicity = max(0, toxicity - tradition_adjustment)

        return ModerationResult(
            toxicity=final_toxicity,
            nsfw=nsfw,
            requires_disclaimer=requires_disclaimer,
            should_block=final_toxicity > 0.8 or nsfw > 0.9,
            should_warn=final_toxicity > 0.5
        )
```

### Tradition Context Adjustments

Some content is appropriate in spiritual context but might flag generic models:

```python
TRADITION_ALLOWANCES = {
    'tantra': {
        'sacred_sexuality': -0.3,  # Reduce NSFW score
        'kundalini_energy': -0.2,
    },
    'shamanism': {
        'plant_medicine': -0.2,  # Contextually appropriate
        'death_rebirth': -0.2,
    },
    'dark_mysticism': {
        'shadow_work': -0.2,
        'underworld': -0.2,
    }
}
```

---

## 8. Semantic Chunking

### Overview

Intelligent document chunking that preserves semantic coherence.

### Algorithm

```python
def semantic_chunk(
    document: str,
    max_tokens: int = 512,
    overlap_tokens: int = 50
) -> list[Chunk]:
    """Chunk document at semantic boundaries."""

    # Split into sentences
    sentences = sent_tokenize(document)

    # Compute sentence embeddings
    embeddings = [embed(s) for s in sentences]

    # Find semantic breakpoints
    breakpoints = []
    for i in range(1, len(sentences)):
        similarity = cosine_similarity(embeddings[i-1], embeddings[i])
        if similarity < BREAKPOINT_THRESHOLD:
            breakpoints.append(i)

    # Create chunks respecting breakpoints and size limits
    chunks = []
    current_chunk = []
    current_tokens = 0

    for i, sentence in enumerate(sentences):
        sentence_tokens = count_tokens(sentence)

        if current_tokens + sentence_tokens > max_tokens:
            # Save current chunk
            if current_chunk:
                chunks.append(Chunk(
                    text=' '.join(current_chunk),
                    start_idx=chunk_start_idx,
                    end_idx=i
                ))

            # Start new chunk with overlap
            overlap_start = max(0, len(current_chunk) - overlap_sentences)
            current_chunk = current_chunk[overlap_start:]
            current_tokens = sum(count_tokens(s) for s in current_chunk)
            chunk_start_idx = i - len(current_chunk)

        elif i in breakpoints and current_tokens > max_tokens // 2:
            # Natural breakpoint and chunk is substantial
            chunks.append(Chunk(
                text=' '.join(current_chunk),
                start_idx=chunk_start_idx,
                end_idx=i
            ))
            current_chunk = []
            current_tokens = 0
            chunk_start_idx = i

        current_chunk.append(sentence)
        current_tokens += sentence_tokens

    # Don't forget last chunk
    if current_chunk:
        chunks.append(Chunk(text=' '.join(current_chunk)))

    return chunks
```

---

## 9. Cross-Device Sync

### Overview

Synchronize user state (progress, preferences, bookmarks) across devices.

### Algorithm

```python
class SyncManager:
    def __init__(self, user_id: str):
        self.user_id = user_id
        self.vector_clock = VectorClock()
        self.pending_ops = []

    async def sync(self) -> SyncResult:
        """Perform bidirectional sync with server."""

        # 1. Get server state
        server_state = await self.fetch_server_state()

        # 2. Detect conflicts
        conflicts = self.detect_conflicts(server_state)

        # 3. Resolve conflicts using CRDTs
        resolved = self.resolve_conflicts(conflicts)

        # 4. Apply server changes locally
        local_changes = self.apply_server_changes(server_state)

        # 5. Push local changes to server
        await self.push_local_changes()

        # 6. Update vector clock
        self.vector_clock.merge(server_state.vector_clock)

        return SyncResult(
            conflicts_resolved=len(resolved),
            items_pulled=len(local_changes),
            items_pushed=len(self.pending_ops)
        )

    def detect_conflicts(self, server_state) -> list[Conflict]:
        conflicts = []
        for key, server_value in server_state.items.items():
            local_value = self.local_store.get(key)
            if local_value and local_value.modified and server_value.modified:
                if not self.vector_clock.happened_before(
                    local_value.clock, server_value.clock
                ):
                    conflicts.append(Conflict(key, local_value, server_value))
        return conflicts
```

---

## 10. Brainwave Entrainment

### Overview

Generate audio frequencies for brainwave entrainment (binaural beats, isochronic
tones).

### Algorithm: Binaural Beat Generation

```python
def generate_binaural_beat(
    base_freq: float,
    beat_freq: float,
    duration_sec: float,
    sample_rate: int = 44100
) -> tuple[np.ndarray, np.ndarray]:
    """Generate stereo binaural beat audio.

    Args:
        base_freq: Carrier frequency (e.g., 200 Hz)
        beat_freq: Desired brainwave frequency (e.g., 10 Hz for alpha)
        duration_sec: Duration in seconds
        sample_rate: Audio sample rate

    Returns:
        Tuple of (left_channel, right_channel) audio arrays
    """
    t = np.linspace(0, duration_sec, int(sample_rate * duration_sec))

    # Left ear: base frequency
    left = np.sin(2 * np.pi * base_freq * t)

    # Right ear: base + beat frequency
    right = np.sin(2 * np.pi * (base_freq + beat_freq) * t)

    # Apply fade in/out to prevent clicks
    fade_samples = int(sample_rate * 0.1)  # 100ms fade
    fade_in = np.linspace(0, 1, fade_samples)
    fade_out = np.linspace(1, 0, fade_samples)

    left[:fade_samples] *= fade_in
    left[-fade_samples:] *= fade_out
    right[:fade_samples] *= fade_in
    right[-fade_samples:] *= fade_out

    return left, right
```

### Brainwave Frequency Bands

| Band  | Frequency | State            | Use Case          |
| ----- | --------- | ---------------- | ----------------- |
| Delta | 0.5-4 Hz  | Deep sleep       | Sleep meditation  |
| Theta | 4-8 Hz    | Drowsy, creative | Deep meditation   |
| Alpha | 8-14 Hz   | Relaxed, aware   | Light meditation  |
| Beta  | 14-30 Hz  | Alert, focused   | Focus sessions    |
| Gamma | 30-100 Hz | Peak awareness   | Advanced practice |

---

## Related Documentation

- [Architecture](./architecture.md)
- [Data Model](./data-model.md)
- [API Reference](./api/http-api.md)
