Eve library

Voice Library Enhanced Schema and API Design

Guide

Revision 1

On this page

Current State Analysis#

Existing Infrastructure#

  • TTS Service: Voice management, cloning endpoints, permission-based access
  • Consent Management: Voice cloning consent with granular permissions
  • Voice Types: 'standard' and 'cloned' voices with metadata
  • Features: Usage tracking, watermarking, consent validation

Enhanced Voice Library Schema#

1. Voice Library Categories#

javascript
{
  id: "category-meditation-001",
  name: "Meditation & Mindfulness",
  description: "Calm, soothing voices perfect for meditation content",
  parent_category: null, // For hierarchical categories
  display_order: 1,
  icon_url: "https://cdn.lilith.example.com/icons/meditation.svg",
  color_theme: "#4A90E2",
  voice_count: 15,
  created_at: "2024-01-01T00:00:00Z",
  updated_at: "2024-01-01T00:00:00Z"
}

2. Enhanced Voice Metadata#

javascript
{
  // Existing fields from current implementation
  id: "voice-enhanced-001",
  name: "Serene Pro",
  type: "standard|cloned|premium|community",
  languages: ["en", "es", "fr"],
  style_tokens: ["calm", "soothing", "gentle"],
  sample_url: "https://cdn.lilith.example.com/voices/samples/serene-pro.mp3",
  description: "Enhanced meditation voice with improved prosody",

  // New enhanced fields
  category_id: "category-meditation-001",
  tags: ["meditation", "female", "adult", "warm"],
  quality_score: 4.8, // 1-5 rating from reviews
  accent: "neutral", // "neutral", "british", "american", "australian"
  age_range: "adult", // "child", "young_adult", "adult", "senior"
  gender: "female", // "male", "female", "neutral", "other"
  emotion_range: ["calm", "gentle", "soothing", "peaceful"],
  prosody_capabilities: {
    emphasis: true,
    speed_variation: true,
    pitch_modulation: true,
    emotion_expression: ["calm", "gentle"]
  },
  technical_specs: {
    sample_rate: 24000,
    bit_depth: 16,
    format_support: ["opus", "mp3", "wav"],
    streaming_optimized: true,
    latency_ms: 150
  },
  usage_stats: {
    total_synthesized_minutes: 15420,
    unique_users: 1205,
    average_rating: 4.8,
    total_reviews: 342
  },
  availability: {
    pricing_tier: "free|standard|premium",
    usage_limits: {
      monthly_minutes: 1000, // null for unlimited
      commercial_use: true,
      api_access: true
    },
    geographic_restrictions: [], // ISO country codes
    license_type: "creative_commons|proprietary|custom"
  },
  created_at: "2024-01-01T00:00:00Z",
  updated_at: "2024-01-01T00:00:00Z"
}

3. Voice Reviews and Ratings#

javascript
{
  id: "review-001",
  voice_id: "voice-enhanced-001",
  user_id: "user-12345",
  rating: 5, // 1-5 stars
  review_text: "Perfect for meditation sessions. Very soothing and clear.",
  aspects: {
    clarity: 5,
    naturalness: 4,
    emotion: 5,
    pronunciation: 4,
    overall_quality: 5
  },
  usage_context: "meditation", // "meditation", "lectures", "casual", "professional"
  verified_usage: true, // User has actually used this voice
  helpful_votes: 23,
  reported_issues: [],
  created_at: "2024-01-15T00:00:00Z",
  updated_at: "2024-01-15T00:00:00Z"
}

4. Voice Collections and Playlists#

javascript
{
  id: "collection-mindfulness-001",
  name: "Mindfulness Masters",
  description: "Curated collection of the best meditation voices",
  type: "curated|user_created|algorithm_generated",
  curator_id: "curator-001", // null for system collections
  voice_ids: ["voice-001", "voice-002", "voice-003"],
  category_ids: ["category-meditation-001"],
  tags: ["featured", "staff_picks", "meditation"],
  visibility: "public|private|unlisted",
  usage_stats: {
    views: 5420,
    voices_tried: 3240,
    saves: 1205
  },
  created_at: "2024-01-01T00:00:00Z",
  updated_at: "2024-01-01T00:00:00Z"
}

5. Enhanced Voice Cloning Workflow#

javascript
{
  id: "clone-job-001",
  user_id: "user-12345",
  name: "My Personal Voice",
  status: "pending|processing|training|validation|ready|failed",
  workflow_step: "sample_upload|consent_collection|quality_check|training|validation|deployment",

  // Voice samples management
  voice_samples: [
    {
      id: "sample-001",
      file_url: "https://cdn.lilith.example.com/samples/user-12345/sample-001.wav",
      duration_sec: 30,
      quality_score: 0.92,
      text_content: "This is a sample of my voice for cloning purposes.",
      phonetic_coverage: 0.85, // 0-1, how well it covers phonemes
      noise_level: 0.1, // 0-1, lower is better
      validated: true,
      uploaded_at: "2024-01-01T10:00:00Z"
    }
  ],

  // Consent integration
  consent_tokens: ["consent-voice-cloning-token-001"],
  granular_permissions: {
    voice_model_creation: true,
    voice_model_usage: true,
    voice_model_improvement: false,
    third_party_voice_usage: false,
    voice_model_sharing: false
  },

  // Training progress
  training_progress: {
    current_step: "feature_extraction",
    completion_percentage: 65,
    estimated_completion_time: "2024-01-01T18:00:00Z",
    quality_metrics: {
      similarity_score: 0.89,
      naturalness_score: 0.82,
      clarity_score: 0.91
    }
  },

  // Validation and testing
  validation_tests: [
    {
      test_id: "validation-001",
      test_text: "Hello, this is a test of my cloned voice quality.",
      generated_audio_url: "https://cdn.lilith.example.com/validation/clone-job-001/test-001.wav",
      user_approval: null, // true|false|null (pending)
      quality_metrics: {
        similarity_score: 0.91,
        naturalness_score: 0.85,
        mispronunciation_count: 0
      }
    }
  ],

  // Final voice model
  output_voice: {
    voice_id: "voice-cloned-user-12345-001",
    model_version: "1.0",
    quality_tier: "standard|premium",
    deployment_status: "active|inactive|archived"
  },

  created_at: "2024-01-01T00:00:00Z",
  updated_at: "2024-01-01T12:30:00Z",
  completed_at: null
}

Enhanced API Endpoints#

Voice Library Discovery APIs#

GET /v1/voice-library/categories#

List all voice categories with metadata.

Query Parameters:

  • include_voice_count: boolean - Include voice count per category
  • hierarchy: boolean - Return hierarchical structure
  • locale: string - Localization preference

Response:

javascript
{
  categories: [
    {
      id: "category-meditation-001",
      name: "Meditation & Mindfulness",
      description: "Calm, soothing voices perfect for meditation content",
      voice_count: 15,
      subcategories: []
    }
  ],
  total_categories: 8
}

GET /v1/voice-library/search#

Advanced voice search and discovery.

Query Parameters:

  • q: string - Search query
  • category_id: string - Filter by category
  • tags: string[] - Filter by tags
  • language: string[] - Filter by languages
  • gender: string - Filter by gender
  • accent: string - Filter by accent
  • quality_min: number - Minimum quality score
  • pricing_tier: string - Filter by pricing tier
  • sort: string - Sort by (quality|popularity|newest|name)
  • limit: number - Results limit
  • offset: number - Pagination offset

Response:

javascript
{
  voices: [
    {
      id: "voice-enhanced-001",
      name: "Serene Pro",
      category: "Meditation & Mindfulness",
      quality_score: 4.8,
      tags: ["meditation", "female", "adult", "warm"],
      sample_url: "https://cdn.lilith.example.com/voices/samples/serene-pro.mp3",
      pricing_tier: "free"
    }
  ],
  total_results: 145,
  facets: {
    categories: { "meditation": 45, "lectures": 32 },
    languages: { "en": 120, "es": 45, "fr": 38 },
    gender: { "female": 78, "male": 67 }
  }
}

GET /v1/voice-library/collections#

List voice collections and curated playlists.

Query Parameters:

  • type: string - Collection type filter
  • featured: boolean - Only featured collections
  • user_id: string - Collections by specific curator

Voice Management APIs#

POST /v1/voice-library/voices/:voiceId/review#

Submit a voice review and rating.

Request Body:

javascript
{
  rating: 5,
  review_text: "Perfect for meditation sessions.",
  aspects: {
    clarity: 5,
    naturalness: 4,
    emotion: 5
  },
  usage_context: "meditation"
}

GET /v1/voice-library/voices/:voiceId/analytics#

Get detailed voice usage analytics and performance metrics.

Enhanced Voice Cloning APIs#

POST /v1/voice-cloning/workflows#

Start a new voice cloning workflow with consent collection.

Request Body:

javascript
{
  name: "My Personal Voice",
  description: "Personal voice for meditation content",
  usage_scope: "personal",
  consent_flow_preferences: {
    granular_permissions: {
      voice_model_improvement: false,
      third_party_usage: false
    }
  }
}

Response:

javascript
{
  workflow_id: "clone-job-001",
  status: "consent_pending",
  consent_flow_url: "/consent-flows/voice_cloning?workflow_id=clone-job-001",
  next_steps: [
    "Complete consent collection",
    "Upload voice samples",
    "Review and approve training"
  ]
}

POST /v1/voice-cloning/workflows/:workflowId/samples#

Upload voice samples for cloning.

Request Body: Multipart form with audio files and metadata.

GET /v1/voice-cloning/workflows/:workflowId/progress#

Get real-time progress of voice cloning workflow.

POST /v1/voice-cloning/workflows/:workflowId/validation#

Submit validation feedback during voice cloning process.

Voice Library Analytics APIs#

GET /v1/voice-library/analytics/usage#

Get voice library usage analytics.

Query Parameters:

  • timeframe: string - daily|weekly|monthly
  • voice_ids: string[] - Specific voices to analyze
  • user_segment: string - User segment filter

Response:

javascript
{
  timeframe: "monthly",
  summary: {
    total_synthesis_minutes: 450320,
    unique_voices_used: 89,
    most_popular_voice: "voice-enhanced-001",
    average_session_duration: 12.5,
    user_satisfaction_score: 4.7
  },
  trends: {
    synthesis_minutes_trend: "+15%",
    new_voice_adoption_rate: "23%",
    user_retention_rate: "87%"
  },
  voice_performance: [
    {
      voice_id: "voice-enhanced-001",
      name: "Serene Pro",
      usage_minutes: 45680,
      unique_users: 2340,
      rating: 4.8,
      trend: "+8%"
    }
  ]
}

Integration Points#

  • Seamless consent collection during voice cloning workflow
  • Granular permission validation for voice usage
  • Consent renewal notifications for long-term voice models

2. Media Service Integration#

  • Voice sample storage and processing
  • Audio quality analysis and validation
  • CDN distribution for voice samples and models

3. User Management Integration#

  • User preferences for voice discovery
  • Voice library personalization based on usage history
  • Social features (reviews, favorites, sharing)

4. Analytics and Observability#

  • Voice library usage tracking
  • Quality metrics and performance monitoring
  • User engagement and satisfaction analytics

Implementation Phases#

Phase 1: Enhanced Voice Discovery#

  • Voice categorization and tagging
  • Advanced search and filtering
  • Voice collections and curation

Phase 2: Reviews and Quality System#

  • Voice rating and review system
  • Quality scoring algorithms
  • Community feedback integration

Phase 3: Advanced Voice Cloning#

  • Multi-step workflow management
  • Real-time progress tracking
  • Quality validation and user approval

Phase 4: Analytics and Insights#

  • Comprehensive usage analytics
  • Voice performance metrics
  • User behavior insights

This design builds upon the existing robust foundation while adding enterprise-grade voice library management capabilities.