Eve library

Psyche Knowledge Base

Other

Revision 1

On this page

RAG system and document processing for the Psyche AI platform.

Part of the Psyche AI Virtual Assistant Platform.

Overview#

The Knowledge Base provides retrieval-augmented generation (RAG) capabilities with document ingestion, vector search, and quality-evaluated response generation.

Features#

  • Document Ingestion: PDF, DOCX, HTML, Markdown, Excel, CSV, JSON
  • Multiple Chunking Strategies: Fixed-size, semantic, sentence, hierarchical
  • Embedding Providers: Voyage AI, OpenAI, Cohere, BGE
  • Vector Stores: Pinecone, Qdrant, pgvector, in-memory
  • Hybrid Search: Vector + BM25 keyword search
  • Re-ranking: Cross-encoder, RRF, diversity boosting
  • RAG Pipeline: Quality evaluation, hallucination detection
  • Content Management: Versioning, archival, analytics

Architecture#

text
knowledge_base/
├── ingestion/            # Document processing
│   ├── types.py          # Document, TextChunk types
│   ├── manager.py        # IngestionManager
│   ├── parsers.py        # PDF, DOCX, HTML parsers
│   ├── chunking.py       # Chunking strategies
│   └── cleaning.py       # Content preprocessing
├── embedding/            # Vector embeddings
│   ├── types.py          # EmbeddingModel, VectorDatabase
│   ├── generators.py     # Voyage, OpenAI, Cohere
│   ├── manager.py        # EmbeddingManager with caching
│   └── stores.py         # VectorStore implementations
├── retrieval/            # Hybrid search
│   ├── types.py          # QueryType, SearchMode
│   ├── pipeline.py       # RetrievalPipeline
│   ├── query.py          # Query processing/expansion
│   ├── search.py         # HybridSearcher
│   ├── reranking.py      # Cross-encoder reranking
│   └── context.py        # Context assembly
├── rag/                  # Generation
│   ├── types.py          # RAGConfig, RAGResponse
│   ├── pipeline.py       # RAGPipeline
│   ├── generator.py      # LLM response generation
│   ├── prompts.py        # Template management
│   └── evaluator.py      # Quality evaluation
└── management/           # Content lifecycle
    ├── types.py          # ContentStatus, QualityLevel
    ├── lifecycle.py      # Versioning, archival
    ├── quality.py        # Quality monitoring
    ├── updates.py        # Content refresh
    └── analytics.py      # Usage analytics

Document Ingestion#

Supported Formats#

Format Parser Features
PDF PyMuPDF Text, tables, OCR fallback
DOCX python-docx Text, styles, tables
HTML BeautifulSoup Cleaned text, links
Markdown Custom Headers, code blocks
Excel openpyxl Sheets, formulas
CSV pandas Structured data
JSON Built-in Nested structures
PowerPoint python-pptx Slides, notes

Chunking Strategies#

Strategy Description Best For
Fixed-Size Token-based with overlap General documents
Semantic Split on meaning boundaries Long-form content
Sentence Preserve sentence integrity Q&A content
Hierarchical Multi-level chunks Structured docs

Embedding Providers#

Provider Models Dimensions
Voyage AI 3.5-lite, 3 1024
OpenAI Ada, 3-small, 3-large 1536, 3072
Cohere English, Multilingual 1024
BGE M3, Large 1024

Vector Stores#

Store Type Features
Pinecone Cloud Managed, scalable
Qdrant Self-hosted Open source, filtering
pgvector PostgreSQL SQL integration
Memory In-process Development only

Retrieval Pipeline#

Search Modes#

Mode Description
VECTOR_ONLY Pure semantic search
KEYWORD_ONLY BM25 keyword search
HYBRID Weighted combination

Re-ranking Strategies#

Strategy Description
CROSS_ENCODER MS MARCO neural reranker
RRF Reciprocal rank fusion
DIVERSITY Boost unique content
RECENCY Decay older content

RAG Pipeline#

Response Modes#

Mode Description
STANDARD Single-turn RAG
CONVERSATIONAL Multi-turn with history
AGENTIC Tool-augmented
STREAMING Token-by-token

Quality Metrics#

Metric Description
Answer Relevance Response addresses query
Context Relevance Retrieved content is relevant
Groundedness Claims supported by sources
Faithfulness No hallucinated facts
Completeness Query fully answered

Quick Start#

Installation#

bash
# Using Nx
nx install psyche-knowledge-base

# With all vector stores
nx install-all psyche-knowledge-base

# Specific vector store
nx install-pinecone psyche-knowledge-base
nx install-qdrant psyche-knowledge-base
nx install-pgvector psyche-knowledge-base

# Or directly with Poetry
cd apps/psyche/knowledge-base
poetry install
poetry install --extras all

Environment Variables#

bash
# Embedding Providers
VOYAGE_API_KEY=your-voyage-key
OPENAI_API_KEY=your-openai-key
COHERE_API_KEY=your-cohere-key

# Vector Stores
PINECONE_API_KEY=your-pinecone-key
PINECONE_ENVIRONMENT=us-east-1
QDRANT_URL=http://localhost:6333
DATABASE_URL=postgresql://user:pass@localhost:5432/knowledge

# LLM Providers
ANTHROPIC_API_KEY=your-anthropic-key

# Service Config
KNOWLEDGE_BASE_PORT=8008
KNOWLEDGE_BASE_HOST=0.0.0.0
LOG_LEVEL=INFO

Basic Usage#

python
from knowledge_base import (
    IngestionManager,
    EmbeddingManager,
    VectorIndexManager,
    RetrievalPipelineBuilder,
    RAGPipelineBuilder,
    LLMClient,
    EmbeddingConfig,
    EmbeddingModel,
)

# Set up embedding
embed_manager = EmbeddingManager(
    config=EmbeddingConfig(model=EmbeddingModel.VOYAGE_3_5),
    api_keys={"voyage": "your-key"},
)

# Set up vector index
index_manager = VectorIndexManager(embed_manager)

# Ingest documents
ingestion = IngestionManager()
result = await ingestion.ingest_file("/path/to/doc.pdf")
await index_manager.index_chunks(result.chunks, "my-index")

# Set up RAG
llm = LLMClient(provider="anthropic", api_key="your-key")
retrieval = RetrievalPipelineBuilder() \
    .with_embedding_manager(embed_manager) \
    .with_vector_store(index_manager._vector_store) \
    .build()
rag = RAGPipelineBuilder() \
    .with_retrieval_pipeline(retrieval) \
    .with_llm_client(llm) \
    .build()

# Query
result = await rag.query("What is your refund policy?")
print(result.response.answer)
print(f"Confidence: {result.quality.confidence}")

API Endpoints#

Documents#

  • POST /documents/ingest - Ingest document
  • POST /documents/batch - Batch ingest
  • GET /documents/{id} - Get document
  • DELETE /documents/{id} - Delete document
  • GET /documents/{id}/chunks - Get chunks
  • POST /search - Hybrid search
  • POST /search/vector - Vector-only search
  • POST /search/keyword - Keyword-only search

RAG#

  • POST /rag/query - Query with RAG
  • POST /rag/stream - Streaming RAG
  • POST /rag/evaluate - Evaluate response

Health#

  • GET /health - Health check
  • GET /ready - Readiness check
  • GET /metrics - Prometheus metrics

Development#

Running the Server#

bash
# Development mode
nx serve psyche-knowledge-base

# Production mode
nx serve-prod psyche-knowledge-base

Running Tests#

bash
nx test psyche-knowledge-base
nx test-unit psyche-knowledge-base
nx test-integration psyche-knowledge-base
nx test-cov psyche-knowledge-base

Docker#

bash
# Build image
nx docker-build psyche-knowledge-base

# Run container
nx docker-run psyche-knowledge-base

Nx Integration#

bash
# Available targets
nx serve psyche-knowledge-base          # Development server
nx serve-prod psyche-knowledge-base     # Production server
nx build psyche-knowledge-base          # Build package
nx install psyche-knowledge-base        # Install dependencies
nx install-all psyche-knowledge-base    # Install all extras
nx install-pinecone psyche-knowledge-base   # Pinecone only
nx install-qdrant psyche-knowledge-base     # Qdrant only
nx install-pgvector psyche-knowledge-base   # pgvector only
nx lint psyche-knowledge-base           # Run linters
nx format psyche-knowledge-base         # Format code
nx test psyche-knowledge-base           # Run all tests
nx test-unit psyche-knowledge-base      # Unit tests only
nx test-integration psyche-knowledge-base  # Integration tests
nx test-cov psyche-knowledge-base       # Tests with coverage
nx docker-build psyche-knowledge-base   # Build Docker image
nx docker-run psyche-knowledge-base     # Run container

Performance#

Default Configuration#

Setting Default Description
Chunk Size 512 tokens Target chunk size
Chunk Overlap 50 tokens Overlap between chunks
Top-K 10 Retrieved chunks
Token Budget 4000 Max context tokens
Cache TTL 7 days Embedding cache

Latency Targets#

Operation Target
Document ingestion 1-5s per page
Embedding generation 100-500ms
Vector search 10-50ms
RAG response 2-5s

License#

Proprietary - Oshun Platform