Purpose: Transparent documentation of how Lilith curates, validates, and maintains its knowledge base of wisdom content.
Our Sourcing Philosophy#
Lilith's wisdom AI is only as good as the knowledge it draws from. We are committed to:
- Authentic Sources: Using verified, authoritative sources from established traditions
- Academic Rigor: Applying scholarly standards to content curation
- Cultural Respect: Honoring wisdom traditions without appropriation
- Quality First: Maintaining high standards for accuracy and relevance
- Transparent Process: Making our curation process open and accountable
Knowledge Source Categories#
Traditional Wisdom Texts#
Buddhist Literature#
yaml
buddhist_sources:
primary_texts:
pali_canon:
description: 'Original Buddhist scriptures in Pali language'
time_period: '3rd century BCE - 1st century CE'
authenticity: 'Preserved through oral tradition and early manuscripts'
modern_sources: 'Pali Text Society editions, Wisdom Publications'
usage_approach: 'Respectful excerpting with full context preservation'
mahayana_sutras:
description: 'Mahayana Buddhist scriptures'
key_texts: ['Heart Sutra', 'Diamond Sutra', 'Lotus Sutra']
translation_sources: 'Red Pine, Edward Conze, Burton Watson translations'
usage_approach:
'Traditional interpretations with contemporary application'
zen_texts:
description: 'Zen Buddhist literature and koans'
key_collections:
['Blue Cliff Record', 'Gateless Gate', 'Book of Serenity']
modern_teachers: 'D.T. Suzuki, Alan Watts, Charlotte Joko Beck'
usage_approach:
'Preserving paradoxical nature while providing accessible guidance'
contemporary_buddhist_sources:
mindfulness_research:
description: 'Scientific research on mindfulness and meditation'
key_researchers: ['Jon Kabat-Zinn', 'Tara Brach', 'Jack Kornfield']
methodology: 'Peer-reviewed research and clinical studies'
integration_approach:
'Bridge traditional wisdom with scientific validation'
peer_reviewed_standards: 'All research sources must be peer-reviewed'
buddhist_psychology:
description: 'Integration of Buddhist concepts with Western psychology'
key_authors: ['Mark Epstein', 'Tara Bennett-Goleman', 'Joseph Goldstein']
validation: 'Clinical and therapeutic application evidence'
Stoic Philosophy#
yaml
stoic_sources:
ancient_stoic_texts:
marcus_aurelius:
work: 'Meditations'
time_period: '161-180 CE'
authenticity: 'Personal journal, historically verified'
translations_used: ['Gregory Hays', 'Martin Hammond', 'Robin Hard']
epictetus:
works: ['Discourses', 'Enchiridion']
time_period: '50-135 CE'
preservation: 'Recorded by student Arrian'
translations_used: ['Robin Hard', 'Robert Dobbin']
seneca:
works: ['Letters to Lucilius', 'Essays']
time_period: '4 BCE - 65 CE'
authenticity: 'Well-preserved complete works'
translations_used: ['Robin Campbell', 'John Davie']
modern_stoic_scholarship:
academic_sources:
- 'A Guide to the Good Life - William Irvine'
- 'How to Be a Stoic - Massimo Pigliucci'
- 'The Cambridge Companion to the Stoics'
practical_applications:
- 'Modern Stoicism movement writings'
- 'Cognitive Behavioral Therapy connections'
- 'Contemporary philosophical analysis'
Sufi Literature#
yaml
sufi_sources:
classical_sufi_texts:
rumi:
works: ['Masnavi', 'Diwan-e Shams', 'Fihi Ma Fihi']
time_period: '1207-1273 CE'
language: 'Persian'
translations_used:
['Coleman Barks', 'Jawid Mojaddedi', 'William Chittick']
cultural_context: '13th century Persian mystical tradition'
ibn_arabi:
works: ['Fusus al-Hikam', 'Al-Futuhat al-Makkiyya']
time_period: '1165-1240 CE'
scholarship: 'Leading Islamic mystical philosopher'
translations_used: ['William Chittick', 'Claude Addas']
al_ghazali:
works: ['Revival of the Religious Sciences', 'The Alchemy of Happiness']
time_period: '1058-1111 CE'
significance: 'Bridge between Islamic law and mysticism'
translations_used: ['T.J. Winter', 'Claude Field']
contemporary_sufi_sources:
modern_teachers:
- 'Llewellyn Vaughan-Lee contemporary Sufi teachings'
- 'Kabir Helminski translations and commentary'
- 'Seyyed Hossein Nasr scholarly works'
academic_scholarship:
- 'Carl Ernst - Islamic mysticism research'
- 'Annemarie Schimmel - Sufi poetry and spirituality'
- 'Omid Safi - contemporary Islamic spirituality'
Contemporary Academic Sources#
Mindfulness and Psychology Research#
yaml
academic_research_sources:
mindfulness_research:
clinical_studies:
description: 'Peer-reviewed research on mindfulness interventions'
key_institutions: ['University of Massachusetts', 'Oxford', 'Harvard']
research_areas: ['MBSR', 'MBCT', 'mindfulness-based interventions']
quality_criteria:
'Peer-reviewed, replicated studies with strong methodology'
neuroscience_research:
description:
'Brain imaging studies of meditation and contemplative practices'
key_researchers: ['Richard Davidson', 'Sara Lazar', 'Judson Brewer']
methodologies: ['fMRI', 'EEG', 'longitudinal studies']
integration_approach:
'Connect scientific findings with traditional teachings'
philosophy_journals:
peer_reviewed_sources:
- 'Journal of Contemplative Studies'
- 'Philosophy East and West'
- 'Buddhist Studies Review'
- 'International Journal for Philosophy of Religion'
quality_assurance:
- impact_factor_consideration: 'Prioritize high-impact journals'
- peer_review_verification: 'Confirm peer review process'
- author_credentials: 'Verify academic credentials and expertise'
Content Curation Process#
Source Discovery and Evaluation#
Discovery Methods#
yaml
source_discovery:
systematic_approaches:
academic_databases:
- jstor: 'Academic journal archive'
- philpapers: 'Philosophy research database'
- pubmed: 'Medical and psychological research'
- worldcat: 'Global library catalog'
expert_recommendations:
- tradition_experts: 'Recommendations from recognized scholars'
- practitioner_communities: 'Input from experienced practitioners'
- academic_conferences: 'Current research and scholarship'
community_submissions:
- user_suggestions: 'Community recommendations with expert review'
- scholar_contributions: 'Academic contributors'
- practitioner_insights: 'Experienced practitioner submissions'
evaluation_criteria:
source_quality:
- author_credentials: 'Verified expertise in relevant tradition'
- publisher_reputation: 'Established academic or spiritual publisher'
- peer_review_status: 'Peer-reviewed or equivalent quality assurance'
- citation_network: 'Well-cited within relevant academic community'
content_relevance:
- tradition_authenticity: 'Faithful to source tradition'
- practical_applicability: 'Relevant to contemporary practitioners'
- educational_value: 'Supports learning and understanding'
- cultural_sensitivity: 'Respectful of cultural context'
Quality Validation Process#
Multi-Stage Review#
yaml
quality_validation:
stage_1_automated_screening:
technical_validation:
- format_compatibility: 'Readable and processable format'
- language_detection: 'Accurate language identification'
- ocr_quality: 'High-quality text extraction (>95% accuracy)'
content_analysis:
- relevance_scoring: 'AI-based relevance assessment'
- quality_indicators: 'Academic citation patterns, language quality'
- duplication_detection: 'Identify duplicate or near-duplicate content'
stage_2_expert_review:
tradition_expert_assessment:
- authenticity_verification:
'Confirms faithful representation of tradition'
- accuracy_checking: 'Verifies factual and interpretive accuracy'
- cultural_sensitivity: 'Ensures respectful treatment of tradition'
academic_validation:
- scholarly_standards: 'Meets academic quality standards'
- citation_verification: 'Accurate and complete citations'
- methodology_review: 'Sound research methodology where applicable'
stage_3_final_approval:
comprehensive_review:
- legal_clearance: 'Intellectual property and licensing verification'
- safety_assessment: 'Content safety and appropriateness review'
- integration_planning: 'How content will be integrated and attributed'
Expert Review Network#
Tradition Experts#
yaml
expert_network:
buddhist_studies:
lead_experts:
- name: 'Dr. Sarah Chen'
credentials: 'PhD Buddhist Studies, Harvard University'
specialization: 'Theravada Buddhism, meditation research'
role: 'Primary reviewer for Buddhist content'
- name: 'Dr. Tenzin Norbu'
credentials: 'Geshe degree, Ganden Monastery; PhD Religious Studies'
specialization: 'Tibetan Buddhism, contemplative practice'
role: 'Tibetan Buddhist content specialist'
philosophy:
lead_experts:
- name: 'Dr. Michael Torres'
credentials: 'PhD Philosophy, Oxford University'
specialization: 'Ancient philosophy, Stoicism'
role: 'Ancient philosophy content reviewer'
- name: 'Dr. Rachel Martinez'
credentials: 'PhD Philosophy, Stanford University'
specialization: 'Ethics, contemporary moral philosophy'
role: 'Contemporary philosophy reviewer'
islamic_studies:
lead_experts:
- name: 'Dr. Amara Hassan'
credentials: 'PhD Islamic Studies, Al-Azhar University'
specialization: 'Sufism, Islamic mysticism'
role: 'Sufi content authenticity reviewer'
- name: 'Dr. Omar Qureshi'
credentials: 'PhD Comparative Religion, University of Chicago'
specialization: 'Islamic philosophy, interfaith dialogue'
role: 'Islamic philosophy content reviewer'
review_process:
assignment_criteria:
- expertise_match: 'Expert specialization matches content tradition'
- availability: 'Can provide review within 5 business days'
- conflict_avoidance: 'No conflicts of interest with content sources'
review_standards:
- accuracy_assessment: 'Factual accuracy and interpretive faithfulness'
- authenticity_verification:
'True to tradition without oversimplification'
- cultural_sensitivity: 'Respectful and appropriate representation'
- practical_utility: 'Helpful for contemporary practitioners'
Content Attribution System#
Attribution Standards#
Academic Citation Format#
yaml
academic_attribution:
required_elements:
author: 'Full name(s) as published'
title: 'Complete work title including subtitle'
publication: 'Publisher, journal, or institution'
date: 'Publication date (YYYY-MM-DD format)'
pages: 'Specific page references where applicable'
doi_url: 'DOI or stable URL when available'
license: 'License type and restrictions'
example_citations:
book_citation:
author: 'Marcus Aurelius'
title: 'Meditations'
translator: 'Gregory Hays'
publisher: 'Modern Library'
publication_date: '2006'
isbn: '978-0679642602'
pages: 'Book 2, Section 11'
license: 'Translation under copyright, original text public domain'
journal_article:
author: 'Dr. Sara Lazar'
title:
'Meditation experience is associated with increased cortical thickness'
journal: 'NeuroReport'
volume: '16'
issue: '17'
pages: '1893-1897'
publication_date: '2005-11-28'
doi: '10.1097/01.wnr.0000186598.66243.19'
license: 'Academic fair use'
Traditional Text Attribution#
yaml
traditional_attribution:
ancient_texts:
attribution_format:
tradition: 'Buddhist/Stoic/Sufi/etc.'
text_name: 'Traditional or canonical name'
time_period: 'Historical time period of composition'
language: 'Original language'
translator: 'Modern translator name and date'
source_edition: 'Specific edition or manuscript tradition'
example_traditional_citation:
tradition: 'Buddhist'
text_name: 'Dhammapada'
time_period: '3rd century BCE'
language: 'Pali'
translator: 'Eknath Easwaran (1985)'
source_edition: 'Nilgiri Press edition'
verses: 'Verses 1-2'
context: 'Opening verses on mind and mental cultivation'
license: 'Original text public domain, translation under fair use'
Attribution Implementation#
In-App Citation Display#
typescript
// Citation display in user interface
interface CitationDisplay {
sourceAttribution: {
author: string;
title: string;
publication?: string;
year: string;
license: string;
url?: string;
};
displayFormat: {
inline: boolean; // Show citations inline with response
expandable: boolean; // Allow expansion for full citation
linkable: boolean; // Link to original source when available
};
userControls: {
showCitations: boolean; // User preference for citation display
citationDetail: 'minimal' | 'standard' | 'comprehensive';
attributionPreference: 'academic' | 'simple' | 'legal';
};
}
// Implementation example
const citationComponent = {
render: (citation: Citation) => `
<div class="citation">
<span class="author">${citation.author}</span>
<span class="title">"${citation.title}"</span>
${citation.publication ? `<span class="publication">${citation.publication}</span>` : ''}
<span class="year">(${citation.year})</span>
<span class="license">[${citation.license}]</span>
${citation.url ? `<a href="${citation.url}" target="_blank">View Source</a>` : ''}
</div>
`,
};
API Response Attribution#
json
{
"response": {
"text": "Mindfulness involves bringing your full attention to the present moment...",
"citations": [
{
"source_uri": "https://wisdom-texts.org/heart_sutra.pdf",
"title": "The Heart of Buddhist Meditation",
"author": "Nyanaponika Thera",
"publication": "Buddhist Publication Society",
"year": "1962",
"license": "Buddhist Publication Society License",
"confidence": 0.95,
"span": {
"start": 0,
"end": 45
},
"attribution_text": "Adapted from 'The Heart of Buddhist Meditation' by Nyanaponika Thera (Buddhist Publication Society, 1962)"
}
]
}
}
Quality Assurance#
Source Verification Process#
Authenticity Validation#
yaml
authenticity_validation:
historical_texts:
manuscript_verification:
- 'Cross-reference with established manuscript traditions'
- 'Verify through academic consensus on authenticity'
- 'Consult with relevant scholarly institutions'
translation_validation:
- 'Use recognized, scholarly translations'
- 'Compare multiple translations for accuracy'
- 'Consult with language experts for critical passages'
contemporary_sources:
author_verification:
- 'Verify academic credentials and expertise'
- 'Confirm institutional affiliations'
- 'Review publication history and peer recognition'
publication_verification:
- 'Confirm publisher reputation and standards'
- 'Verify peer review process where applicable'
- 'Check citation patterns and academic reception'
content_validation:
fact_checking:
- 'Cross-reference factual claims with authoritative sources'
- 'Verify historical claims with historical records'
- 'Confirm contemporary research citations'
cultural_accuracy:
- 'Review with tradition experts for cultural accuracy'
- 'Ensure respectful and appropriate representation'
- 'Avoid stereotypes and oversimplifications'
Quality Metrics and Monitoring#
python
# Content quality monitoring system
from lilith.quality_monitoring import SourceQualityMonitor
class SourceQualityMonitor:
def __init__(self):
self.quality_thresholds = self.load_quality_standards()
self.expert_network = self.load_expert_network()
def monitor_source_quality(self, time_period: str = "30d") -> QualityReport:
"""Generate comprehensive quality report for sourced content."""
# Accuracy metrics
accuracy_metrics = self.assess_accuracy_metrics([
"fact_verification_rate",
"citation_accuracy_rate",
"expert_approval_rate",
"user_feedback_accuracy"
])
# Source diversity metrics
diversity_metrics = self.assess_source_diversity([
"tradition_representation",
"geographic_diversity",
"temporal_distribution",
"language_diversity"
])
# Expert review metrics
expert_metrics = self.assess_expert_review_process([
"review_completion_rate",
"expert_consensus_rate",
"review_quality_scores",
"response_time_metrics"
])
# User feedback integration
user_feedback = self.analyze_user_feedback([
"content_helpfulness_scores",
"accuracy_feedback",
"cultural_sensitivity_feedback",
"practical_utility_ratings"
])
return QualityReport(
overall_quality_score=self.calculate_overall_score(
accuracy_metrics, diversity_metrics, expert_metrics, user_feedback
),
detailed_metrics={
"accuracy": accuracy_metrics,
"diversity": diversity_metrics,
"expert_review": expert_metrics,
"user_feedback": user_feedback
},
recommendations=self.generate_improvement_recommendations(),
next_review_date=self.calculate_next_review_date()
)
Transparency Commitments#
Public Reporting#
Source Transparency Report#
yaml
transparency_reporting:
quarterly_source_report:
content_metrics:
- total_sources: 'Number of unique sources in knowledge base'
- source_breakdown: 'Distribution by tradition, time period, type'
- new_additions: 'Recently added sources with justification'
- source_removals: 'Removed sources with reasons'
quality_metrics:
- expert_review_rate:
'Percentage of content reviewed by tradition experts'
- accuracy_score: 'Overall content accuracy assessment'
- user_satisfaction: 'User feedback on content quality and relevance'
diversity_metrics:
- tradition_balance: 'Representation across wisdom traditions'
- cultural_diversity: 'Geographic and cultural source distribution'
- temporal_range: 'Historical span of sources'
- language_diversity: 'Original languages represented'
annual_sourcing_review:
comprehensive_assessment:
- sourcing_strategy_effectiveness: 'How well our approach serves users'
- expert_network_assessment: 'Review of expert reviewer network'
- user_feedback_integration: 'How user feedback improves sourcing'
- future_sourcing_priorities: 'Strategic direction for content expansion'
public_commitments:
- measurable_goals: 'Specific, time-bound commitments for improvement'
- diversity_targets: 'Targets for increasing source diversity'
- quality_improvements: 'Planned enhancements to quality processes'
Community Engagement#
Public Input Mechanisms#
yaml
community_engagement:
source_suggestions:
submission_process:
- 'Public form for suggesting new sources'
- 'Academic and practitioner submission tracks'
- 'Community voting on suggested sources'
review_process:
- 'Expert preliminary assessment'
- 'Community feedback period'
- 'Final determination with public explanation'
content_feedback:
accuracy_reporting:
- 'Easy mechanism for reporting accuracy issues'
- 'Public tracking of reported issues and resolutions'
- 'Recognition for community contributions'
cultural_sensitivity:
- 'Channels for cultural representation concerns'
- 'Regular dialogue with tradition communities'
- 'Responsive process for addressing concerns'
advisory_relationships:
tradition_advisors:
- 'Formal relationships with tradition representatives'
- 'Regular consultation on content direction'
- 'Veto power over inappropriate content use'
academic_advisors:
- 'Scholarly advisory board for academic standards'
- 'Research collaboration opportunities'
- 'Peer review of sourcing methodology'
Source Database#
Public Source Registry#
Searchable Source Database#
typescript
// Public interface for exploring Lilith's sources
interface PublicSourceRegistry {
searchSources(criteria: {
tradition?: string[];
timeperiod?: string;
contentType?: string;
author?: string;
language?: string;
}): Promise<SourceSummary[]>;
getSourceDetails(sourceId: string): Promise<DetailedSource>;
getSourceStatistics(): Promise<{
totalSources: number;
traditionBreakdown: Record<string, number>;
qualityMetrics: QualityMetrics;
recentAdditions: SourceSummary[];
}>;
}
interface SourceSummary {
id: string;
title: string;
author: string;
tradition: string[];
timePeriod: string;
contentType:
| 'traditional_text'
| 'academic_research'
| 'contemporary_teaching';
qualityScore: number;
expertReviewed: boolean;
dateAdded: string;
license: string;
}
interface DetailedSource extends SourceSummary {
fullCitation: string;
abstract: string;
keyTopics: string[];
expertReviewers: string[];
qualityAssessment: QualityAssessment;
usageStatistics: UsageStatistics;
relatedSources: SourceSummary[];
}
Usage Analytics and Reporting#
Public Usage Metrics#
yaml
public_usage_reporting:
monthly_metrics:
source_usage:
- most_cited_sources: 'Top 20 sources by citation frequency'
- tradition_usage_distribution: 'Usage breakdown by wisdom tradition'
- content_type_preferences: 'User preferences for content types'
quality_feedback:
- user_satisfaction_scores: 'User ratings of source quality'
- accuracy_feedback: 'User reports on content accuracy'
- helpfulness_ratings: 'Practical utility assessments'
transparency_dashboard:
real_time_metrics:
- active_sources: 'Number of sources currently in active use'
- recent_additions: 'Sources added in last 30 days'
- expert_review_status: 'Percentage of content expert-reviewed'
public_api:
endpoint: 'https://api.lilith.app/v1/transparency/sources'
authentication: 'Public endpoint, no authentication required'
rate_limits: '100 requests per hour per IP'
data_format: 'JSON with comprehensive source metadata'
Contact and Feedback#
Source-Related Inquiries#
- General Questions: sourcing@lilith.app
- Source Suggestions: sources@lilith.app
- Accuracy Reports: accuracy@lilith.app
- Cultural Concerns: cultural-liaison@lilith.app
Academic and Research#
- Academic Partnerships: academic@lilith.app
- Research Collaboration: research@lilith.app
- Expert Network: experts@lilith.app
Last Updated: 2024-01-15 Review Schedule: Quarterly Next Review: 2024-04-15
Our sourcing practices evolve based on user feedback, expert guidance, and emerging best practices in ethical AI development. We welcome your input in making our knowledge base more accurate, diverse, and respectful.