RAG system and document processing for the Psyche AI platform.
Part of the Psyche AI Virtual Assistant Platform.
Overview#
The Knowledge Base provides retrieval-augmented generation (RAG) capabilities with document ingestion, vector search, and quality-evaluated response generation.
Features#
- Document Ingestion: PDF, DOCX, HTML, Markdown, Excel, CSV, JSON
- Multiple Chunking Strategies: Fixed-size, semantic, sentence, hierarchical
- Embedding Providers: Voyage AI, OpenAI, Cohere, BGE
- Vector Stores: Pinecone, Qdrant, pgvector, in-memory
- Hybrid Search: Vector + BM25 keyword search
- Re-ranking: Cross-encoder, RRF, diversity boosting
- RAG Pipeline: Quality evaluation, hallucination detection
- Content Management: Versioning, archival, analytics
Architecture#
text
knowledge_base/
├── ingestion/ # Document processing
│ ├── types.py # Document, TextChunk types
│ ├── manager.py # IngestionManager
│ ├── parsers.py # PDF, DOCX, HTML parsers
│ ├── chunking.py # Chunking strategies
│ └── cleaning.py # Content preprocessing
├── embedding/ # Vector embeddings
│ ├── types.py # EmbeddingModel, VectorDatabase
│ ├── generators.py # Voyage, OpenAI, Cohere
│ ├── manager.py # EmbeddingManager with caching
│ └── stores.py # VectorStore implementations
├── retrieval/ # Hybrid search
│ ├── types.py # QueryType, SearchMode
│ ├── pipeline.py # RetrievalPipeline
│ ├── query.py # Query processing/expansion
│ ├── search.py # HybridSearcher
│ ├── reranking.py # Cross-encoder reranking
│ └── context.py # Context assembly
├── rag/ # Generation
│ ├── types.py # RAGConfig, RAGResponse
│ ├── pipeline.py # RAGPipeline
│ ├── generator.py # LLM response generation
│ ├── prompts.py # Template management
│ └── evaluator.py # Quality evaluation
└── management/ # Content lifecycle
├── types.py # ContentStatus, QualityLevel
├── lifecycle.py # Versioning, archival
├── quality.py # Quality monitoring
├── updates.py # Content refresh
└── analytics.py # Usage analytics
Document Ingestion#
Supported Formats#
| Format | Parser | Features |
|---|---|---|
| PyMuPDF | Text, tables, OCR fallback | |
| DOCX | python-docx | Text, styles, tables |
| HTML | BeautifulSoup | Cleaned text, links |
| Markdown | Custom | Headers, code blocks |
| Excel | openpyxl | Sheets, formulas |
| CSV | pandas | Structured data |
| JSON | Built-in | Nested structures |
| PowerPoint | python-pptx | Slides, notes |
Chunking Strategies#
| Strategy | Description | Best For |
|---|---|---|
| Fixed-Size | Token-based with overlap | General documents |
| Semantic | Split on meaning boundaries | Long-form content |
| Sentence | Preserve sentence integrity | Q&A content |
| Hierarchical | Multi-level chunks | Structured docs |
Embedding Providers#
| Provider | Models | Dimensions |
|---|---|---|
| Voyage AI | 3.5-lite, 3 | 1024 |
| OpenAI | Ada, 3-small, 3-large | 1536, 3072 |
| Cohere | English, Multilingual | 1024 |
| BGE | M3, Large | 1024 |
Vector Stores#
| Store | Type | Features |
|---|---|---|
| Pinecone | Cloud | Managed, scalable |
| Qdrant | Self-hosted | Open source, filtering |
| pgvector | PostgreSQL | SQL integration |
| Memory | In-process | Development only |
Retrieval Pipeline#
Search Modes#
| Mode | Description |
|---|---|
| VECTOR_ONLY | Pure semantic search |
| KEYWORD_ONLY | BM25 keyword search |
| HYBRID | Weighted combination |
Re-ranking Strategies#
| Strategy | Description |
|---|---|
| CROSS_ENCODER | MS MARCO neural reranker |
| RRF | Reciprocal rank fusion |
| DIVERSITY | Boost unique content |
| RECENCY | Decay older content |
RAG Pipeline#
Response Modes#
| Mode | Description |
|---|---|
| STANDARD | Single-turn RAG |
| CONVERSATIONAL | Multi-turn with history |
| AGENTIC | Tool-augmented |
| STREAMING | Token-by-token |
Quality Metrics#
| Metric | Description |
|---|---|
| Answer Relevance | Response addresses query |
| Context Relevance | Retrieved content is relevant |
| Groundedness | Claims supported by sources |
| Faithfulness | No hallucinated facts |
| Completeness | Query fully answered |
Quick Start#
Installation#
bash
# Using Nx
nx install psyche-knowledge-base
# With all vector stores
nx install-all psyche-knowledge-base
# Specific vector store
nx install-pinecone psyche-knowledge-base
nx install-qdrant psyche-knowledge-base
nx install-pgvector psyche-knowledge-base
# Or directly with Poetry
cd apps/psyche/knowledge-base
poetry install
poetry install --extras all
Environment Variables#
bash
# Embedding Providers
VOYAGE_API_KEY=your-voyage-key
OPENAI_API_KEY=your-openai-key
COHERE_API_KEY=your-cohere-key
# Vector Stores
PINECONE_API_KEY=your-pinecone-key
PINECONE_ENVIRONMENT=us-east-1
QDRANT_URL=http://localhost:6333
DATABASE_URL=postgresql://user:pass@localhost:5432/knowledge
# LLM Providers
ANTHROPIC_API_KEY=your-anthropic-key
# Service Config
KNOWLEDGE_BASE_PORT=8008
KNOWLEDGE_BASE_HOST=0.0.0.0
LOG_LEVEL=INFO
Basic Usage#
python
from knowledge_base import (
IngestionManager,
EmbeddingManager,
VectorIndexManager,
RetrievalPipelineBuilder,
RAGPipelineBuilder,
LLMClient,
EmbeddingConfig,
EmbeddingModel,
)
# Set up embedding
embed_manager = EmbeddingManager(
config=EmbeddingConfig(model=EmbeddingModel.VOYAGE_3_5),
api_keys={"voyage": "your-key"},
)
# Set up vector index
index_manager = VectorIndexManager(embed_manager)
# Ingest documents
ingestion = IngestionManager()
result = await ingestion.ingest_file("/path/to/doc.pdf")
await index_manager.index_chunks(result.chunks, "my-index")
# Set up RAG
llm = LLMClient(provider="anthropic", api_key="your-key")
retrieval = RetrievalPipelineBuilder() \
.with_embedding_manager(embed_manager) \
.with_vector_store(index_manager._vector_store) \
.build()
rag = RAGPipelineBuilder() \
.with_retrieval_pipeline(retrieval) \
.with_llm_client(llm) \
.build()
# Query
result = await rag.query("What is your refund policy?")
print(result.response.answer)
print(f"Confidence: {result.quality.confidence}")
API Endpoints#
Documents#
POST /documents/ingest- Ingest documentPOST /documents/batch- Batch ingestGET /documents/{id}- Get documentDELETE /documents/{id}- Delete documentGET /documents/{id}/chunks- Get chunks
Search#
POST /search- Hybrid searchPOST /search/vector- Vector-only searchPOST /search/keyword- Keyword-only search
RAG#
POST /rag/query- Query with RAGPOST /rag/stream- Streaming RAGPOST /rag/evaluate- Evaluate response
Health#
GET /health- Health checkGET /ready- Readiness checkGET /metrics- Prometheus metrics
Development#
Running the Server#
bash
# Development mode
nx serve psyche-knowledge-base
# Production mode
nx serve-prod psyche-knowledge-base
Running Tests#
bash
nx test psyche-knowledge-base
nx test-unit psyche-knowledge-base
nx test-integration psyche-knowledge-base
nx test-cov psyche-knowledge-base
Docker#
bash
# Build image
nx docker-build psyche-knowledge-base
# Run container
nx docker-run psyche-knowledge-base
Nx Integration#
bash
# Available targets
nx serve psyche-knowledge-base # Development server
nx serve-prod psyche-knowledge-base # Production server
nx build psyche-knowledge-base # Build package
nx install psyche-knowledge-base # Install dependencies
nx install-all psyche-knowledge-base # Install all extras
nx install-pinecone psyche-knowledge-base # Pinecone only
nx install-qdrant psyche-knowledge-base # Qdrant only
nx install-pgvector psyche-knowledge-base # pgvector only
nx lint psyche-knowledge-base # Run linters
nx format psyche-knowledge-base # Format code
nx test psyche-knowledge-base # Run all tests
nx test-unit psyche-knowledge-base # Unit tests only
nx test-integration psyche-knowledge-base # Integration tests
nx test-cov psyche-knowledge-base # Tests with coverage
nx docker-build psyche-knowledge-base # Build Docker image
nx docker-run psyche-knowledge-base # Run container
Performance#
Default Configuration#
| Setting | Default | Description |
|---|---|---|
| Chunk Size | 512 tokens | Target chunk size |
| Chunk Overlap | 50 tokens | Overlap between chunks |
| Top-K | 10 | Retrieved chunks |
| Token Budget | 4000 | Max context tokens |
| Cache TTL | 7 days | Embedding cache |
Latency Targets#
| Operation | Target |
|---|---|
| Document ingestion | 1-5s per page |
| Embedding generation | 100-500ms |
| Vector search | 10-50ms |
| RAG response | 2-5s |
License#
Proprietary - Oshun Platform