# ElevenLabs Integration Reference documentation for the ElevenLabs AI voice provider supporting text-to-speech, streaming, voice cloning, and audio tags. ## Overview The ElevenLabs provider offers: - **Text-to-Speech**: Multiple models (Flash, Turbo, Multilingual, v3) - **Streaming Audio**: Real-time audio generation - **Voice Management**: Listing, preview, cloning - **Audio Tags (v3)**: Performance cues for emotional delivery - **Model Listing**: Available models and capabilities - **Retry Logic**: Exponential backoff with rate limit handling ## Configuration ### Environment Variables | Variable | Description | Required | | -------------------- | ------------------ | -------- | | `ELEVENLABS_API_KEY` | ElevenLabs API key | Yes | ### Provider Initialization ```typescript import { ElevenLabsProvider } from './providers/elevenlabs-provider'; const provider = new ElevenLabsProvider({ apiKey: process.env.ELEVENLABS_API_KEY, baseURL: 'https://api.elevenlabs.io/v1', maxRetries: 3, retryDelayMs: 1000, timeout: 30000, defaultModel: 'eleven_multilingual_v2', defaultVoiceId: '21m00Tcm4TlvDq8ikWAM', defaultOutputFormat: 'mp3_44100_128', }); ``` ### Configuration Options | Option | Type | Default | Description | | --------------------- | ------------ | ------------------------------ | --------------------- | | `apiKey` | string | `ELEVENLABS_API_KEY` | API key | | `baseURL` | string | `https://api.elevenlabs.io/v1` | API base URL | | `maxRetries` | number | 3 | Max retry attempts | | `retryDelayMs` | number | 1000 | Base retry delay (ms) | | `timeout` | number | 30000 | Request timeout (ms) | | `defaultModel` | string | `eleven_multilingual_v2` | Default TTS model | | `defaultVoiceId` | string | - | Default voice ID | | `defaultOutputFormat` | OutputFormat | `mp3_44100_128` | Default audio format | ## Text-to-Speech ### Basic TTS ```typescript const response = await provider.textToSpeech({ text: 'Welcome to your guided meditation.', voiceId: 'pNInz6obpgDQGcFmaJgB', // Adam modelId: 'eleven_multilingual_v2', }); // response.audio is a Buffer containing the audio fs.writeFileSync('output.mp3', response.audio); ``` ### TTS Request Parameters | Parameter | Type | Description | | -------------------------- | ------------- | ----------------------------- | | `text` | string | Text to convert to speech | | `voiceId` | string | Voice ID to use | | `modelId` | string | Model ID (optional) | | `voiceSettings` | VoiceSettings | Voice customization | | `outputFormat` | OutputFormat | Audio format | | `languageCode` | string | Language override | | `seed` | number | For reproducible output | | `previousText` | string | Context from previous segment | | `nextText` | string | Context for next segment | | `optimizeStreamingLatency` | 0-4 | Latency optimization level | ### Voice Settings ```typescript const response = await provider.textToSpeech({ text: 'This is a calm, slow meditation.', voiceId: 'voice_id', voiceSettings: { stability: 0.8, // 0-1, higher = more consistent similarity_boost: 0.75, // 0-1, higher = closer to original style: 0.3, // 0-1, v2+ models only use_speaker_boost: true, // Enhance voice clarity speed: 0.8, // 0.25-4.0, speech rate }, }); ``` ### Output Formats | Format | Description | | ---------------- | ------------------------------- | | `mp3_22050_32` | MP3, 22.05kHz, 32kbps | | `mp3_44100_32` | MP3, 44.1kHz, 32kbps | | `mp3_44100_64` | MP3, 44.1kHz, 64kbps | | `mp3_44100_96` | MP3, 44.1kHz, 96kbps | | `mp3_44100_128` | MP3, 44.1kHz, 128kbps (default) | | `mp3_44100_192` | MP3, 44.1kHz, 192kbps | | `pcm_16000` | PCM, 16kHz | | `pcm_22050` | PCM, 22.05kHz | | `pcm_24000` | PCM, 24kHz | | `pcm_44100` | PCM, 44.1kHz | | `ulaw_8000` | uLaw, 8kHz (telephony) | | `alaw_8000` | aLaw, 8kHz (telephony) | | `opus_48000_64` | Opus, 48kHz, 64kbps | | `opus_48000_128` | Opus, 48kHz, 128kbps | ### TTS Response ```typescript interface TTSResponse { audio: Buffer; contentType: string; requestId?: string; characterCount?: number; historyItemId?: string; } ``` ## Streaming Audio ### Stream TTS ```typescript const stream = await provider.textToSpeechStream({ text: 'This is a long meditation script...', voiceId: 'voice_id', }); for await (const chunk of stream) { // chunk.audio is a Buffer // chunk.isFinal indicates last chunk // chunk.alignment contains timing info socket.write(chunk.audio); if (chunk.alignment) { console.log('Characters:', chunk.alignment.chars); console.log('Timings:', chunk.alignment.charStartTimesMs); } } ``` ### Stream Chunk ```typescript interface TTSStreamChunk { audio: Buffer; isFinal: boolean; alignment?: { chars: string[]; charStartTimesMs: number[]; charDurationsMs: number[]; }; } ``` ### Latency Optimization ```typescript // Level 0: No optimization (highest quality) // Level 1: Slight latency reduction // Level 2: Balanced // Level 3: Optimized for real-time // Level 4: Maximum speed (lowest quality) const response = await provider.textToSpeech({ text: 'Hello!', voiceId: 'voice_id', optimizeStreamingLatency: 3, }); ``` ## Voice Management ### List Voices ```typescript const response = await provider.listVoices({ page_size: 20, search: 'calm', sort: 'name', sort_direction: 'asc', voice_type: 'default', category: 'premade', }); for (const voice of response.voices) { console.log(`${voice.name} (${voice.voice_id})`); console.log(` Category: ${voice.category}`); console.log(` Labels: ${JSON.stringify(voice.labels)}`); console.log(` Preview: ${voice.preview_url}`); } ``` ### Voice Properties ```typescript interface Voice { voice_id: string; name: string; category: 'premade' | 'cloned' | 'generated' | 'professional'; labels?: Record; description?: string; preview_url?: string; settings?: VoiceSettings; samples?: VoiceSample[]; high_quality_base_model_ids?: string[]; } ``` ### Voice Search Parameters | Parameter | Type | Description | | ---------------- | ------ | ------------------------------------------------ | | `page_size` | number | Results per page | | `search` | string | Search query | | `sort` | string | Sort field (`created_at`, `name`) | | `sort_direction` | string | `asc` or `desc` | | `voice_type` | string | `personal`, `community`, `default`, `workspace` | | `category` | string | `premade`, `cloned`, `generated`, `professional` | ### Get Voice Sample ```typescript const sampleUrl = await provider.getVoiceSample('voice_id'); // Returns URL to audio sample ``` ## Voice Cloning Comprehensive voice cloning support with Instant Voice Cloning (IVC), Professional Voice Cloning (PVC), consent management, and quality analysis. ### Supported Audio Formats | Format | MIME Type | Extension | | ------ | -------------------------- | --------- | | MP3 | `audio/mpeg` | `.mp3` | | WAV | `audio/wav`, `audio/x-wav` | `.wav` | | M4A | `audio/mp4`, `audio/x-m4a` | `.m4a` | | OGG | `audio/ogg` | `.ogg` | | FLAC | `audio/flac` | `.flac` | | WebM | `audio/webm` | `.webm` | ### Instant Voice Cloning (IVC) ```typescript const response = await provider.cloneVoice({ name: 'My Custom Voice', files: [ { data: audioBuffer1, filename: 'sample1.mp3' }, { data: audioBuffer2, filename: 'sample2.mp3' }, ], description: 'A calm, meditative voice', labels: { use_case: 'meditation', accent: 'american' }, removeBackgroundNoise: true, }); console.log('Created voice:', response.voice_id); ``` ### Voice Cloning Request ```typescript interface VoiceCloningRequest { name: string; // Voice name (required) files: VoiceCloningSample[]; // Audio samples (required) description?: string; // Voice description labels?: Record; // Categorization labels removeBackgroundNoise?: boolean; // Remove noise from samples } interface VoiceCloningSample { data: Buffer; // Audio data filename: string; // Used for MIME type detection mimeType?: string; // Override MIME type } ``` ### Voice Cloning Response ```typescript interface VoiceCloningResponse { voice_id: string; // Unique voice identifier } ``` ### Edit Voice Metadata ```typescript await provider.editVoice('voice_id', { name: 'Updated Name', description: 'New description', labels: { style: 'calming' }, }); ``` ### Update Voice Settings ```typescript await provider.updateVoiceSettings('voice_id', { stability: 0.8, similarity_boost: 0.7, style: 0.3, use_speaker_boost: true, }); ``` ### Add Samples to Voice ```typescript await provider.addVoiceSamples('voice_id', { files: [{ data: newAudioBuffer, filename: 'new_sample.mp3' }], removeBackgroundNoise: true, }); ``` ### Delete Voice Sample ```typescript await provider.deleteVoiceSample('voice_id', 'sample_id'); ``` ### Delete Voice ```typescript await provider.deleteVoice('voice_id'); ``` ### Professional Voice Cloning (PVC) Higher quality cloning with manual verification (requires 30+ minutes of audio): ```typescript const response = await provider.professionalVoiceClone({ name: 'Professional Voice', files: [{ data: highQualityAudio, filename: 'studio_recording.wav' }], description: 'Professional narrator voice', consentVerificationId: 'consent_id_from_flow', }); // Check status if (response.status === 'processing') { console.log('Estimated completion:', response.estimated_completion); } ``` ### PVC Status Values | Status | Description | | ---------------------- | ----------------------------- | | `pending_verification` | Awaiting consent verification | | `processing` | Voice is being trained | | `completed` | Voice ready for use | | `failed` | Training failed | | `rejected` | Consent rejected | ### Check Cloning Capabilities ```typescript // Check if IVC is available const canUseIVC = await provider.canUseInstantVoiceCloning(); // Check if PVC is available const canUsePVC = await provider.canUseProfessionalVoiceCloning(); // Get detailed limits const limits = await provider.getVoiceCloningLimits(); // { // voiceLimit: 10, // currentVoiceCount: 3, // maxVoiceAddEdits: 100, // voiceAddEditCounter: 15, // canUseIVC: true, // canUsePVC: false // } ``` ## Consent Management Consent flow for ethical voice cloning with verification and tamper detection. ### Generate Consent Statement ```typescript const consent = provider.generateConsentStatement({ voiceOwnerName: 'John Smith', clonedVoiceName: 'Meditation Guide', intendedUse: 'Meditation and wellness applications', expirationDate: new Date('2026-01-01'), }); console.log(consent.consentText); console.log(consent.checksum); // For verification ``` ### Consent Statement Structure ```typescript interface ConsentStatement { version: string; // Statement format version timestamp: string; // When consent was given voiceOwnerName: string; // Voice owner's name clonedVoiceName: string; // Name for cloned voice intendedUse: string; // Description of use expirationDate?: string; // Optional expiration consentText: string; // Full consent agreement checksum?: string; // Tamper verification } ``` ### Verify Consent Statement ```typescript const result = provider.verifyConsentStatement(consent); if (result.valid) { console.log('Consent is valid'); } else { console.log('Errors:', result.errors); if (result.isExpired) { console.log('Consent has expired'); } } ``` ### Consent Verification Result ```typescript interface ConsentVerificationResult { valid: boolean; // Overall validity errors: string[]; // Validation errors isExpired: boolean; // Expiration status } ``` ## Sample Validation Utility functions for validating audio samples before cloning. ### Validate Single Sample ```typescript import { validateVoiceCloningSample } from './providers/elevenlabs-provider'; const sample = { data: audioBuffer, filename: 'sample.mp3' }; const result = validateVoiceCloningSample(sample); if (!result.valid) { console.log('Validation errors:', result.errors); } ``` ### Validation Rules | Check | Requirement | | -------- | ------------------- | | Data | Non-empty buffer | | Filename | Required, non-empty | | Format | Supported MIME type | | Min Size | 10KB minimum | | Max Size | 50MB maximum | ### Analyze Sample Quality ```typescript import { analyzeVoiceCloningSamples } from './providers/elevenlabs-provider'; const samples = [ { data: buffer1, filename: 'sample1.wav' }, { data: buffer2, filename: 'sample2.wav' }, ]; const report = analyzeVoiceCloningSamples(samples); console.log('Quality Score:', report.qualityScore); console.log('Recommendations:', report.recommendations); console.log('Meets Requirements:', report.meetsMinimumRequirements); console.log('Details:', report.details); ``` ### Quality Report Structure ```typescript interface VoiceCloningQualityReport { qualityScore: number; // 0-100 score recommendations: string[]; // Improvement suggestions meetsMinimumRequirements: boolean; // Minimum met details: { totalDuration: number; // Estimated seconds sampleCount: number; // Number of samples averageSampleSize: number; // Bytes formatConsistency: boolean; // Same format }; } ``` ## Best Practices ### Recommended Settings ```typescript import { VOICE_CLONING_BEST_PRACTICES } from './providers/elevenlabs-provider'; // { // minSampleDuration: 60, // 60 seconds minimum // optimalSampleDuration: 300, // 5 minutes optimal // maxSampleDuration: 1800, // 30 minutes max // recommendedSampleCount: 3, // 3 samples // recommendedSampleRate: 44100, // recommendedBitDepth: 16, // tips: [...] // } ``` ### Recording Tips | Tip | Description | | ----------- | ------------------------------------ | | Environment | Quiet room, minimal background noise | | Microphone | Good quality, consistent distance | | Speaking | Natural pace and tone | | Content | Diverse text, varied expressions | | Format | WAV or FLAC for best quality | | Audio | Mono, not stereo | | Volume | Consistent levels across samples | ### MIME Type Detection ```typescript import { getMimeTypeFromFilename } from './providers/elevenlabs-provider'; const mimeType = getMimeTypeFromFilename('sample.wav'); // 'audio/wav' const mimeType2 = getMimeTypeFromFilename('audio.m4a'); // 'audio/mp4' ``` ### Complete Voice Cloning Workflow ```typescript import { validateVoiceCloningSample, analyzeVoiceCloningSamples, VOICE_CLONING_BEST_PRACTICES, } from './providers/elevenlabs-provider'; // 1. Validate individual samples const samples = [ { data: buffer1, filename: 'sample1.wav' }, { data: buffer2, filename: 'sample2.wav' }, ]; for (const sample of samples) { const validation = validateVoiceCloningSample(sample); if (!validation.valid) { throw new Error(`Invalid sample: ${validation.errors.join(', ')}`); } } // 2. Analyze overall quality const report = analyzeVoiceCloningSamples(samples); if (!report.meetsMinimumRequirements) { console.warn('Recommendations:', report.recommendations); } // 3. Generate consent const consent = provider.generateConsentStatement({ voiceOwnerName: 'Jane Doe', clonedVoiceName: 'Wellness Guide', intendedUse: 'Meditation app narration', }); // 4. Verify consent const verification = provider.verifyConsentStatement(consent); if (!verification.valid) { throw new Error(`Invalid consent: ${verification.errors.join(', ')}`); } // 5. Check account capabilities const canClone = await provider.canUseInstantVoiceCloning(); if (!canClone) { throw new Error('Voice cloning not available on this account'); } // 6. Clone the voice const response = await provider.cloneVoice({ name: consent.clonedVoiceName, files: samples, description: 'Created with proper consent', labels: { consent_verified: 'true' }, removeBackgroundNoise: true, }); console.log('Voice created:', response.voice_id); ``` ## Voice Design Create unique AI voices from text descriptions without audio samples. ### Design a Voice ```typescript const response = await provider.designVoice({ voiceDescription: 'A calm, soothing voice ideal for meditation. Warm tone with natural pauses.', text: 'Welcome to your guided meditation session. Take a deep breath in...', guidanceScale: 6, autoGenerateText: false, }); // Returns 3 previews to choose from for (const preview of response.previews) { console.log('Voice ID:', preview.generatedVoiceId); console.log('Duration:', preview.durationSecs, 'seconds'); // preview.audioBase64 contains the audio data } ``` ### Voice Design Request ```typescript interface VoiceDesignRequest { voiceDescription: string; // 20-1000 characters (required) text?: string; // 100-1000 character preview text modelId?: VoiceDesignModelId; // Model to use autoGenerateText?: boolean; // Auto-generate preview text guidanceScale?: number; // 0-10, how closely AI follows prompt loudness?: number; // -1 to 1, volume level seed?: number; // For reproducibility shouldEnhance?: boolean; // AI enhancement of description quality?: number; // Output quality vs. variety referenceAudioBase64?: string; // Reference audio (v3 only) promptStrength?: number; // 0-1, prompt vs. reference balance outputFormat?: OutputFormat; // Audio format } ``` ### Voice Design Model IDs | Model | Description | | ---------------------------- | ------------------------------ | | `eleven_multilingual_ttv_v2` | Multilingual voice design (v2) | | `eleven_ttv_v3` | Latest voice design model (v3) | ### Voice Design Response ```typescript interface VoiceDesignResponse { previews: VoiceDesignPreview[]; // Typically 3 previews text: string; // Text used for previews } interface VoiceDesignPreview { audioBase64: string; // Base64-encoded audio generatedVoiceId: string; // ID for saving mediaType: string; // Audio MIME type durationSecs: number; // Audio duration language?: string; // Detected language } ``` ### Save Designed Voice ```typescript const savedVoice = await provider.saveDesignedVoice({ voiceName: 'Meditation Guide', voiceDescription: 'A calm, soothing voice for meditation', generatedVoiceId: response.previews[0].generatedVoiceId, labels: { category: 'meditation', mood: 'calm' }, playedNotSelectedVoiceIds: [ // Feedback for ElevenLabs response.previews[1].generatedVoiceId, response.previews[2].generatedVoiceId, ], }); console.log('Saved voice:', savedVoice.voice_id); ``` ### Voice Save Request ```typescript interface VoiceSaveRequest { voiceName: string; // Name (required) voiceDescription: string; // Description generatedVoiceId: string; // From preview (required) labels?: Record; // Metadata playedNotSelectedVoiceIds?: string[]; // Feedback } ``` ### Voice Save Response ```typescript interface VoiceSaveResponse { voice_id: string; name: string; category: 'generated' | 'cloned' | 'premade' | 'professional'; description?: string; settings?: VoiceSettings; labels?: Record; created_at_unix?: number; } ``` ### Convenience Methods #### Create Voice from Description Design and save in one call: ```typescript const voice = await provider.createVoiceFromDescription({ voiceDescription: 'A wise, grounded voice for spiritual teachings', voiceName: 'Spiritual Teacher', previewText: 'The journey inward is the most important journey...', guidanceScale: 7, labels: { category: 'spiritual', style: 'wise' }, }); ``` #### Create Voice from Preset Use built-in presets: ```typescript const voice = await provider.createVoiceFromPreset( 'Meditation Guide', 'My Meditation Voice', { labels: { app: 'meditation-app' }, previewText: 'Custom preview text...', } ); ``` ## Voice Design Presets Pre-configured presets for common use cases. ### Available Presets | Preset | Category | Description | | ---------------------- | ------------ | ------------------------------- | | Meditation Guide | meditation | Calm, soothing, nurturing | | Spiritual Teacher | spiritual | Wise, resonant, grounded | | Yoga Instructor | meditation | Balanced, encouraging, peaceful | | Sleep Narrator | meditation | Soft, gentle, dreamlike | | Philosophical Narrator | educational | Thoughtful, articulate | | Mystical Storyteller | storytelling | Enchanting, otherworldly | ### Design from Preset ```typescript const response = await provider.designVoiceFromPreset('Meditation Guide', { text: 'Custom preview text instead of preset sample...', guidanceScale: 7, // Override preset guidance }); ``` ### Preset Structure ```typescript interface VoiceDesignPreset { name: string; description: string; category: | 'meditation' | 'spiritual' | 'educational' | 'storytelling' | 'character' | 'custom'; voiceDescription: string; guidanceScale: number; recommendedSettings: Partial; sampleTexts: string[]; } ``` ### Get Preset by Name ```typescript import { getVoiceDesignPreset } from './providers/elevenlabs-provider'; const preset = getVoiceDesignPreset('Meditation Guide'); console.log(preset?.voiceDescription); console.log(preset?.sampleTexts[0]); ``` ### Get Presets by Category ```typescript import { getVoiceDesignPresetsByCategory } from './providers/elevenlabs-provider'; const meditationPresets = getVoiceDesignPresetsByCategory('meditation'); // ['Meditation Guide', 'Yoga Instructor', 'Sleep Narrator'] ``` ### Create Custom Preset ```typescript import { createVoiceDesignPreset } from './providers/elevenlabs-provider'; const customPreset = createVoiceDesignPreset( 'Zen Master', 'A profound, deeply peaceful voice that speaks with the wisdom of ages...', { category: 'spiritual', description: 'Ancient wisdom voice', guidanceScale: 8, recommendedSettings: { stability: 0.75, similarity_boost: 0.65, style: 0.4, speed: 0.85, }, sampleTexts: [ 'In the silence between thoughts, truth reveals itself.', 'The path to enlightenment begins with a single breath.', ], } ); ``` ### Built-in Preset Details **Meditation Guide**: ```typescript { voiceDescription: 'A calm, serene voice with a gentle, soothing quality...', guidanceScale: 6, recommendedSettings: { stability: 0.7, similarity_boost: 0.6, style: 0.3, speed: 0.85 } } ``` **Spiritual Teacher**: ```typescript { voiceDescription: 'A wise, deeply resonant voice that conveys ancient wisdom...', guidanceScale: 7, recommendedSettings: { stability: 0.65, similarity_boost: 0.7, style: 0.4, speed: 0.9 } } ``` **Sleep Narrator**: ```typescript { voiceDescription: 'An extremely soft, gentle voice with a dreamlike quality...', guidanceScale: 8, recommendedSettings: { stability: 0.8, similarity_boost: 0.5, style: 0.2, speed: 0.75 } } ``` ### Complete Voice Design Workflow ```typescript import { VOICE_DESIGN_PRESETS, getVoiceDesignPreset, getVoiceDesignPresetsByCategory, createVoiceDesignPreset, } from './providers/elevenlabs-provider'; // 1. List available presets console.log( 'Available presets:', VOICE_DESIGN_PRESETS.map((p) => p.name) ); // 2. Filter by category const spiritualPresets = getVoiceDesignPresetsByCategory('spiritual'); // 3. Get a specific preset const preset = getVoiceDesignPreset('Meditation Guide')!; // 4. Generate voice previews const designResponse = await provider.designVoiceFromPreset( 'Meditation Guide', { text: preset.sampleTexts[0], } ); // 5. Review previews (play audio to user) for (const preview of designResponse.previews) { const audioBuffer = Buffer.from(preview.audioBase64, 'base64'); // Play or analyze audio... } // 6. Save the best preview const selectedPreview = designResponse.previews[0]; const voice = await provider.saveDesignedVoice({ voiceName: 'My Meditation Guide', voiceDescription: preset.voiceDescription, generatedVoiceId: selectedPreview.generatedVoiceId, labels: { category: preset.category, source: 'voice_design', }, playedNotSelectedVoiceIds: designResponse.previews .filter((p) => p.generatedVoiceId !== selectedPreview.generatedVoiceId) .map((p) => p.generatedVoiceId), }); // 7. Apply recommended settings await provider.updateVoiceSettings(voice.voice_id, preset.recommendedSettings); // 8. Use the voice for TTS const audio = await provider.textToSpeech({ text: 'Welcome to your meditation...', voiceId: voice.voice_id, }); ``` ## Model Management ### List Models ```typescript const models = await provider.listModels(); for (const model of models) { console.log(`${model.name} (${model.model_id})`); console.log(` TTS: ${model.can_do_text_to_speech}`); console.log(` Voice Conversion: ${model.can_do_voice_conversion}`); console.log(` Style: ${model.can_use_style}`); console.log(` Languages: ${model.languages?.map((l) => l.name).join(', ')}`); } ``` ### Available Models | Model ID | Description | | ------------------------ | ----------------------- | | `eleven_monolingual_v1` | English only, original | | `eleven_multilingual_v1` | Multi-language support | | `eleven_multilingual_v2` | Improved multi-language | | `eleven_turbo_v2` | Fast generation | | `eleven_turbo_v2_5` | Latest turbo model | | `eleven_flash_v2` | Ultra-fast generation | | `eleven_flash_v2_5` | Latest flash model | ## Usage Stats ```typescript const stats = await provider.getUsageStats(); console.log( `Characters used: ${stats.character_count}/${stats.character_limit}` ); console.log(`Voices: ${stats.voice_add_edit_counter}/${stats.voice_limit}`); console.log(`Reset: ${new Date(stats.next_character_count_reset_unix * 1000)}`); ``` ## Audio Tags (v3) Audio tags control emotional delivery in the v3 model. ### Tag Categories | Category | Description | Examples | | -------------- | ------------------ | ----------------------------------- | | `emotion` | Emotional states | `[sad]`, `[happy]`, `[calm]` | | `delivery` | Speech delivery | `[whispers]`, `[shouts]` | | `reaction` | Vocal reactions | `[sighs]`, `[laughs]`, `[gasps]` | | `cognitive` | Thinking patterns | `[pauses]`, `[hesitates]` | | `sound_effect` | Non-speech sounds | `[applause]`, `[thunder]` | | `accent` | Speaking styles | `[French accent]`, `[pirate voice]` | | `pacing` | Speed control | `[slowly]`, `[quickly]` | | `intensity` | Strength modifiers | `[softly]`, `[loudly]` | | `character` | Character voices | `[old man voice]`, `[narrator]` | ### Emotion Tags ```typescript const emotionTags = [ '[sad]', '[happy]', '[angry]', '[excited]', '[nervous]', '[calm]', '[sorrowful]', '[hopeful]', '[fearful]', '[surprised]', '[tender]', '[loving]', '[contemplative]', '[serene]', '[determined]', ]; ``` ### Delivery Tags ```typescript const deliveryTags = [ '[whispers]', '[shouts]', '[murmurs]', '[speaks softly]', '[speaks firmly]', '[monotone]', '[dramatic]', '[matter of fact]', ]; ``` ### Reaction Tags ```typescript const reactionTags = [ '[sighs]', '[laughs]', '[nervous laugh]', '[gasps]', '[gulps]', '[clears throat]', '[coughs]', '[sniffles]', '[yawns]', '[groans]', '[exhales]', '[inhales]', ]; ``` ### Using Audio Tags ```typescript const text = '[calm] Welcome to this meditation. [pauses] ' + 'Take a deep breath. [exhales] ' + 'Let your body relax. [slowly] ' + 'Feel the peace within. [serene]'; const response = await provider.textToSpeech({ text, voiceId: 'voice_id', modelId: 'eleven_multilingual_v2', // v3 model supports tags }); ``` ### Parse Audio Tags ```typescript import { parseAudioTags } from './providers/elevenlabs-audio-tags'; const result = parseAudioTags('[calm] Hello [pauses] there.'); console.log(result.cleanText); // "Hello there." console.log(result.tags); // [ // { tag: '[calm]', innerText: 'calm', category: 'emotion', isValid: true }, // { tag: '[pauses]', innerText: 'pauses', category: 'cognitive', isValid: true } // ] ``` ### Validate Tags ```typescript import { validateTag, validateTaggedText, } from './providers/elevenlabs-audio-tags'; const validation = validateTag('whispers'); // { tag: 'whispers', isValid: true, category: 'delivery' } const result = validateTaggedText('[calm] Hello [unknown tag] world'); // { isValid: true, warnings: ['Tag "[unknown tag]": Custom tag (not in known list)'] } ``` ### Audio Tag Templates Pre-built templates for different content types: ```typescript import { MEDITATION_TEMPLATE, SLEEP_TEMPLATE, ENERGIZING_TEMPLATE, STORYTELLING_TEMPLATE, EDUCATIONAL_TEMPLATE, } from './providers/elevenlabs-audio-tags'; ``` **Meditation Template**: ```typescript { openingTags: ['[calm]', '[serene]', '[gently]'], transitionTags: ['[pauses]', '[softly]', '[deep breath]'], closingTags: ['[tenderly]', '[peacefully]', '[slowly]'], paragraphTags: ['[calmly]', '[serenely]'], emphasisTags: ['[gently]', '[lovingly]'], pauseTags: ['[pauses]', '[long pause]', '[exhales]'] } ``` **Sleep Template**: ```typescript { openingTags: ['[slowly]', '[softly]', '[whispers]'], transitionTags: ['[long pause]', '[exhales]', '[yawns]'], closingTags: ['[whispers]', '[trails off]', '[slowly]'], paragraphTags: ['[softly]', '[murmurs]'], emphasisTags: ['[tenderly]', '[whispers]'], pauseTags: ['[long pause]', '[pauses]', '[exhales deeply]'] } ``` ### Inject Tags Automatically ```typescript import { injectMeditationTags, injectSleepTags, injectEnergizingTags, } from './providers/elevenlabs-audio-tags'; const meditation = injectMeditationTags( 'Welcome to this practice. Close your eyes.', { pauseFrequency: 2 } ); // "[calm] Welcome to this practice. [pauses] Close your eyes. [tenderly]" const sleep = injectSleepTags('Let your body relax. Feel yourself sinking.'); // "[slowly] Let your body relax. [long pause] Feel yourself sinking. [whispers]" ``` ### Custom Tag Injection ```typescript import { injectAudioTags } from './providers/elevenlabs-audio-tags'; const result = injectAudioTags(text, { template: MEDITATION_TEMPLATE, addOpeningTag: true, addClosingTag: true, addTransitions: true, pauseAfterSentences: true, pauseFrequency: 3, addEmphasis: true, emphasisKeywords: ['peace', 'calm', 'relax'], customReplacements: new Map([['breath', '[inhales] breath']]), }); ``` ### Utility Functions ```typescript import { extractTags, removeAudioTags, countAudioTags, hasAudioTags, formatTag, combineTags, getTagsByCategory, getSuggestedTags, } from './providers/elevenlabs-audio-tags'; // Extract tags const tags = extractTags('[calm] Hello [pauses]'); // ['[calm]', '[pauses]'] // Remove tags const clean = removeAudioTags('[calm] Hello [pauses]'); // 'Hello' // Count tags const count = countAudioTags('[calm] Hello [pauses]'); // 2 // Check for tags const hasTags = hasAudioTags('[calm] Hello'); // true // Format tag const formatted = formatTag('whispers'); // '[whispers]' // Combine tags const combined = combineTags('calm', 'slowly'); // '[calm][slowly]' // Get tags by category const emotions = getTagsByCategory('emotion'); // Get suggested tags for content type const suggestions = getSuggestedTags('meditation'); ``` ## Conversational AI (WebRTC/WebSocket) Real-time conversational AI with bidirectional audio for interactive voice agents. ### Connection Types | Type | Description | Use Case | | ----------- | ---------------------- | ------------------------- | | `websocket` | WebSocket connection | Server-side, full control | | `webrtc` | WebRTC peer connection | Client-side, browser | ### Get Conversation Token (WebRTC) ```typescript const tokenResponse = await provider.getConversationToken( 'agent_abc123', 'User Name' // Optional participant name ); console.log('Token:', tokenResponse.token); // Use token to establish WebRTC connection ``` ### Get Signed WebSocket URL ```typescript const urlResponse = await provider.getConversationSignedUrl('agent_abc123'); const ws = new WebSocket(urlResponse.signedUrl); ``` ### Get Public Agent WebSocket URL For public agents that don't require authentication: ```typescript const wsUrl = provider.getPublicAgentWebSocketUrl('agent_abc123'); // 'wss://api.elevenlabs.io/v1/convai/conversation?agent_id=agent_abc123' ``` ### Create Conversation Session ```typescript const session = provider.createConversationSession({ agentId: 'agent_abc123', connectionType: 'websocket', participantName: 'User', initialContext: 'The user is interested in meditation.', audioFormat: 'pcm', sampleRate: 16000, }); console.log('Session ID:', session.conversationId); console.log('Status:', session.status); // 'connecting' console.log('Mode:', session.mode); // 'listening' ``` ### Conversation Configuration ```typescript interface ConversationConfig { agentId: string; // Agent ID (required) connectionType?: 'websocket' | 'webrtc'; // Connection type participantName?: string; // User's name customMetadata?: Record; // Custom data initialContext?: string; // Initial context audioFormat?: 'pcm' | 'mp3' | 'opus'; // Audio format sampleRate?: number; // Sample rate (Hz) } ``` ### Conversation Session ```typescript interface ConversationSession { conversationId: string; agentId: string; connectionType: 'websocket' | 'webrtc'; status: ConversationStatus; mode: ConversationMode; startedAt: Date; participantName?: string; } type ConversationStatus = 'connecting' | 'connected' | 'disconnected' | 'error'; type ConversationMode = 'speaking' | 'listening'; ``` ### Server-to-Client Events | Event Type | Description | | ---------------------------------- | ------------------------- | | `user_transcript` | User speech transcription | | `agent_response` | Agent's text response | | `agent_response_correction` | Corrected agent response | | `audio` | Agent audio chunk | | `interruption` | User interrupted agent | | `ping` | Keep-alive ping | | `conversation_initiation_metadata` | Session metadata | ### Parse Server Events ```typescript const event = provider.parseServerEvent(rawMessage); if (event) { switch (event.type) { case 'user_transcript': console.log('User said:', event.userTranscript); console.log('Final:', event.isFinal); break; case 'agent_response': console.log('Agent:', event.agentResponse); break; case 'audio': const audioBuffer = Buffer.from(event.audioBase64, 'base64'); playAudio(audioBuffer); break; case 'ping': ws.send(JSON.stringify(provider.createPongEvent(event.eventId))); break; case 'interruption': console.log('Interrupted:', event.reason); break; case 'conversation_initiation_metadata': console.log('Conversation ID:', event.conversationId); break; } } ``` ### Server Event Types ```typescript interface UserTranscriptEvent { type: 'user_transcript'; userTranscript: string; isFinal: boolean; } interface AgentResponseEvent { type: 'agent_response'; agentResponse: string; } interface ConversationAudioEvent { type: 'audio'; audioBase64: string; eventId: number; } interface InterruptionEvent { type: 'interruption'; reason: string; } interface PingEvent { type: 'ping'; eventId: number; pingMs?: number; } interface ConversationInitiationMetadata { type: 'conversation_initiation_metadata'; conversationId: string; agentConfig?: Record; } ``` ### Client-to-Server Events #### Send Audio Chunk ```typescript // Audio: PCM 16-bit mono at 16kHz, base64 encoded const audioEvent = provider.createUserAudioChunkEvent(audioBase64); ws.send(JSON.stringify(audioEvent)); ``` #### Send Text Message ```typescript const textEvent = provider.createUserTextMessageEvent('Hello, how are you?'); ws.send(JSON.stringify(textEvent)); ``` #### Send Contextual Update Non-interrupting information for the agent: ```typescript const contextEvent = provider.createContextualUpdateEvent( 'User just started a breathing exercise' ); ws.send(JSON.stringify(contextEvent)); ``` #### Send User Activity Prevent agent interruption during user actions: ```typescript const activityEvent = provider.createUserActivityEvent(); ws.send(JSON.stringify(activityEvent)); ``` #### Send Pong Response Reply to ping events: ```typescript const pongEvent = provider.createPongEvent(pingEvent.eventId); ws.send(JSON.stringify(pongEvent)); ``` ### Client Event Types ```typescript interface ContextualUpdateEvent { type: 'contextual_update'; text: string; } interface UserAudioChunkEvent { userAudioChunk: string; // base64 PCM } interface UserTextMessageEvent { type: 'user_message'; text: string; } interface UserActivityEvent { type: 'user_activity'; } interface PongEvent { type: 'pong'; eventId: number; } ``` ### Conversation Messages Create and track conversation messages: ```typescript const message = provider.createConversationMessage( 'user', // 'user' or 'agent' 'Hello there!', // text true, // isFinal audioBase64 // optional audio ); // { // id: 'msg_1704067200_abc123', // role: 'user', // text: 'Hello there!', // timestamp: Date, // isFinal: true, // audioBase64: '...' // } ``` ### Conversation History ```typescript const history = provider.createConversationHistory( 'conv_123', messages, session.startedAt, new Date() // endedAt ); // { // conversationId: 'conv_123', // messages: [...], // durationSec: 120, // startedAt: Date, // endedAt: Date // } ``` ### Complete WebSocket Example ```typescript async function startConversation(agentId: string) { // 1. Get signed URL const { signedUrl } = await provider.getConversationSignedUrl(agentId); // 2. Create session const session = provider.createConversationSession({ agentId }); const messages: ConversationMessage[] = []; // 3. Connect WebSocket const ws = new WebSocket(signedUrl); ws.onopen = () => { console.log('Connected'); session.status = 'connected'; }; ws.onmessage = (event) => { const parsed = provider.parseServerEvent(event.data); if (!parsed) return; switch (parsed.type) { case 'user_transcript': if (parsed.isFinal) { messages.push( provider.createConversationMessage( 'user', parsed.userTranscript, true ) ); } break; case 'agent_response': messages.push( provider.createConversationMessage( 'agent', parsed.agentResponse, true ) ); break; case 'audio': // Play audio through speakers playAudio(Buffer.from(parsed.audioBase64, 'base64')); break; case 'ping': ws.send(JSON.stringify(provider.createPongEvent(parsed.eventId))); break; } }; ws.onclose = () => { session.status = 'disconnected'; const history = provider.createConversationHistory( session.conversationId, messages, session.startedAt, new Date() ); console.log('Conversation duration:', history.durationSec, 'seconds'); }; // 4. Send user audio (from microphone) startMicrophoneCapture((audioChunk) => { const base64 = audioChunk.toString('base64'); ws.send(JSON.stringify(provider.createUserAudioChunkEvent(base64))); }); // 5. Send context updates as needed ws.send( JSON.stringify( provider.createContextualUpdateEvent( 'User is in a quiet room ready for meditation guidance' ) ) ); } ``` ## Sound Effects Generate AI sound effects from text descriptions for meditation and ambient content. ### Generate Sound Effect ```typescript const response = await provider.generateSoundEffect({ text: 'A Tibetan singing bowl being gently struck', durationSeconds: 8, promptInfluence: 0.5, loop: false, outputFormat: 'mp3_44100_128', }); fs.writeFileSync('singing-bowl.mp3', response.audio); ``` ### Sound Effects Request ```typescript interface SoundEffectsRequest { text: string; // Description (required) modelId?: SoundEffectsModelId; // Model to use durationSeconds?: number; // 0.5-30 seconds promptInfluence?: number; // 0-1 (default 0.3) loop?: boolean; // Seamless loop (v2 only) outputFormat?: OutputFormat; // Audio format } type SoundEffectsModelId = | 'eleven_text_to_sound_v1' | 'eleven_text_to_sound_v2'; ``` ### Sound Effects Response ```typescript interface SoundEffectsResponse { audio: Buffer; // Generated audio contentType: string; // MIME type requestId?: string; // For tracking durationSeconds?: number; isLooped?: boolean; } ``` ### Generate from Preset ```typescript const response = await provider.generateSoundEffectFromPreset( 'Tibetan Singing Bowl', { durationSeconds: 10, // Override preset duration loop: true, // Override preset loop setting } ); ``` ### Batch Generation ```typescript // Multiple custom requests const responses = await provider.generateSoundEffectBatch([ { text: 'Gentle rain', durationSeconds: 20, loop: true }, { text: 'Temple bell', durationSeconds: 6 }, { text: 'Wind chimes', durationSeconds: 10 }, ]); // Multiple presets const presetResults = await provider.generateSoundEffectBatchFromPresets( ['Tibetan Singing Bowl', 'Ocean Waves', 'Forest Birds'], { outputFormat: 'mp3_44100_128' } ); for (const result of presetResults) { if (result.response) { fs.writeFileSync(`${result.preset}.mp3`, result.response.audio); } else { console.log(`Error generating ${result.preset}:`, result.error); } } ``` ## Sound Effects Presets 17 built-in presets for meditation and ambient content. ### Preset Categories | Category | Presets | | -------------- | ----------------------------------------------------------------------------------------- | | `meditation` | Tibetan Singing Bowl, Temple Bell, Meditation Chime, Om Drone | | `nature` | Gentle Rain, Ocean Waves, Forest Birds, Flowing Stream, Wind in Trees, Campfire Crackling | | `ambient` | Ethereal Pad, Crystal Resonance, Deep Space | | `transition` | Breath Transition, Shimmer, Gong Wash | | `notification` | Mindful Bell, Soft Chime | ### Available Presets | Preset | Duration | Loop | Category | | -------------------- | -------- | ---- | ------------ | | Tibetan Singing Bowl | 8s | No | meditation | | Temple Bell | 6s | No | meditation | | Meditation Chime | 4s | No | meditation | | Om Drone | 20s | Yes | meditation | | Gentle Rain | 20s | Yes | nature | | Ocean Waves | 20s | Yes | nature | | Forest Birds | 20s | Yes | nature | | Flowing Stream | 20s | Yes | nature | | Wind in Trees | 20s | Yes | nature | | Campfire Crackling | 20s | Yes | nature | | Ethereal Pad | 20s | Yes | ambient | | Crystal Resonance | 15s | Yes | ambient | | Deep Space | 20s | Yes | ambient | | Breath Transition | 2s | No | transition | | Shimmer | 2s | No | transition | | Gong Wash | 8s | No | transition | | Mindful Bell | 3s | No | notification | | Soft Chime | 2s | No | notification | ### Preset Structure ```typescript interface SoundEffectsPreset { name: string; description: string; category: | 'meditation' | 'nature' | 'ambient' | 'transition' | 'notification' | 'musical' | 'custom'; prompt: string; recommendedDuration: number; shouldLoop: boolean; promptInfluence: number; } ``` ### Preset Utilities ```typescript import { SOUND_EFFECTS_PRESETS, getSoundEffectsPreset, getSoundEffectsPresetsByCategory, createSoundEffectsPreset, } from './providers/elevenlabs-provider'; // Get preset by name const preset = getSoundEffectsPreset('Tibetan Singing Bowl'); // Get presets by category const naturePresets = getSoundEffectsPresetsByCategory('nature'); // Create custom preset const customPreset = createSoundEffectsPreset( 'Zen Garden', 'Water feature trickling in a peaceful zen garden with soft wind chimes', { category: 'ambient', description: 'Peaceful zen garden ambiance', recommendedDuration: 20, shouldLoop: true, promptInfluence: 0.4, } ); ``` ## Music Generation Generate AI music from text descriptions for meditation and wellness content. ### Generate Music ```typescript const response = await provider.generateMusic({ prompt: 'Create slow, ambient meditation music with soft synthesizer pads', durationMs: 180000, // 3 minutes includeVocals: false, }); fs.writeFileSync('meditation-music.mp3', response.audio); ``` ### Music Generation Request ```typescript interface MusicGenerationRequest { prompt: string; // Description (required) durationMs?: number; // 10000-300000ms (10s-5min) compositionPlan?: MusicCompositionPlan; // Structured plan includeVocals?: boolean; // Include vocals vocalsLanguage?: 'en' | 'es' | 'de' | 'ja' | 'fr' | 'pt' | 'it' | 'zh'; } ``` ### Music Generation Response ```typescript interface MusicGenerationResponse { audio: Buffer; // MP3 audio contentType: string; // 'audio/mpeg' requestId?: string; // For tracking durationMs?: number; filename?: string; compositionPlan?: MusicCompositionPlan; // If detailed } ``` ### Generate with Composition Plan ```typescript // First generate a composition plan const plan = await provider.generateCompositionPlan({ prompt: 'Peaceful meditation music with a gentle intro and calming outro', durationMs: 120000, }); // Then generate music with the plan const response = await provider.generateMusic({ prompt: 'Meditation music', durationMs: 120000, compositionPlan: plan, }); ``` ### Composition Plan Structure ```typescript interface MusicCompositionPlan { positiveGlobalStyles: string[]; // Global positive attributes negativeGlobalStyles: string[]; // Styles to avoid sections: MusicSection[]; // Track structure } interface MusicSection { sectionName: string; // 'intro', 'verse', 'outro', etc. positiveLocalStyles: string[]; // Section-specific attributes negativeLocalStyles: string[]; // Section-specific avoids durationMs: number; // Section duration lines?: string[]; // Lyrics (if vocals) } ``` ### Detailed Generation Get both audio and composition plan: ```typescript const response = await provider.generateMusicDetailed({ prompt: 'Ambient yoga music with soft sitar and tabla', durationMs: 180000, }); console.log('Plan:', response.compositionPlan); console.log('Filename:', response.filename); ``` ### Generate from Preset ```typescript const response = await provider.generateMusicFromPreset('Deep Meditation', { durationMs: 300000, // Override to 5 minutes }); ``` ### Batch Generation ```typescript // Multiple custom tracks const tracks = await provider.generateMusicBatch([ { prompt: 'Morning meditation', durationMs: 180000 }, { prompt: 'Evening relaxation', durationMs: 240000 }, ]); // Multiple presets const presetTracks = await provider.generateMusicBatchFromPresets( ['Deep Meditation', 'Yoga Flow', 'Deep Sleep'], { durationMs: 180000 } ); for (const result of presetTracks) { if (result.response) { fs.writeFileSync(`${result.preset}.mp3`, result.response.audio); } } ``` ## Music Generation Presets 14 built-in presets for meditation, yoga, sleep, and spiritual content. ### Preset Categories | Category | Presets | | ------------ | --------------------------------------------------------- | | `meditation` | Deep Meditation, Guided Meditation Background, Zen Garden | | `yoga` | Yoga Flow, Restorative Yoga | | `sleep` | Deep Sleep, Lullaby Dreams | | `focus` | Focus Flow | | `spiritual` | Sacred Om, Temple Atmosphere | | `nature` | Forest Meditation, Ocean Waves Music | | `relaxation` | Spa Relaxation, Evening Unwind | ### Available Presets | Preset | Duration | Vocals | BPM | Category | | ---------------------------- | -------- | ------ | ----- | ---------- | | Deep Meditation | 5min | No | 50-70 | meditation | | Guided Meditation Background | 5min | No | 50-60 | meditation | | Zen Garden | 4min | No | 50-65 | meditation | | Yoga Flow | 5min | No | 75-95 | yoga | | Restorative Yoga | 5min | No | 45-60 | yoga | | Deep Sleep | 5min | No | 40-55 | sleep | | Lullaby Dreams | 4min | No | 50-65 | sleep | | Focus Flow | 5min | No | 65-85 | focus | | Sacred Om | 5min | Yes | 55-70 | spiritual | | Temple Atmosphere | 4min | No | 50-65 | spiritual | | Forest Meditation | 4min | No | 50-65 | nature | | Ocean Waves Music | 4min | No | 50-60 | nature | | Spa Relaxation | 4min | No | 55-70 | relaxation | | Evening Unwind | 4min | No | 50-65 | relaxation | ### Preset Structure ```typescript interface MusicGenerationPreset { name: string; description: string; category: | 'meditation' | 'yoga' | 'sleep' | 'focus' | 'relaxation' | 'spiritual' | 'nature' | 'custom'; prompt: string; recommendedDurationMs: number; includeVocals: boolean; bpmRange?: { min: number; max: number }; musicalElements?: string[]; } ``` ### Preset Utilities ```typescript import { MUSIC_GENERATION_PRESETS, getMusicGenerationPreset, getMusicGenerationPresetsByCategory, createMusicGenerationPreset, } from './providers/elevenlabs-provider'; // Get preset by name const preset = getMusicGenerationPreset('Deep Meditation'); // Get presets by category const sleepPresets = getMusicGenerationPresetsByCategory('sleep'); // Create custom preset const customPreset = createMusicGenerationPreset( 'Morning Awakening', 'Gentle, uplifting music to start the day with soft strings and light percussion', { category: 'meditation', description: 'Energizing morning music', recommendedDurationMs: 180000, includeVocals: false, bpmRange: { min: 65, max: 80 }, musicalElements: ['strings', 'light percussion', 'soft piano'], } ); ``` ## Dubbing Translate and dub audio/video content into multiple languages. ### Create Dubbing Job ```typescript // From file const response = await provider.createDubbing({ file: audioBuffer, filename: 'meditation.mp3', targetLang: 'es', sourceLang: 'en', name: 'Meditation Guide - Spanish', }); // From URL const response = await provider.createDubbing({ sourceUrl: 'https://example.com/video.mp4', targetLang: 'fr', name: 'Tutorial - French', }); console.log('Dubbing ID:', response.dubbingId); console.log('Expected duration:', response.expectedDurationSec, 'seconds'); ``` ### Dubbing Request ```typescript interface DubbingRequest { targetLang: DubbingLanguage | string; // Target language (required) sourceLang?: DubbingLanguage | string; // Source (default: auto-detect) file?: Buffer; // Audio/video file filename?: string; // Filename for file sourceUrl?: string; // Alternative to file name?: string; // Project name targetAccent?: string; // Accent preference numSpeakers?: number; // 0 = auto-detect (max 9) mode?: DubbingMode; // 'automatic' | 'manual' watermark?: boolean; // Apply watermark highestResolution?: boolean; // Use highest resolution dropBackgroundAudio?: boolean; // Remove background useProfanityFilter?: boolean; // Censor profanities disableVoiceCloning?: boolean; // Use voice library dubbingStudio?: boolean; // Enable studio edits startTime?: number; // Clip start (ms) endTime?: number; // Clip end (ms) } ``` ### Dubbing Create Response ```typescript interface DubbingCreateResponse { dubbingId: string; expectedDurationSec: number; } ``` ### Get Dubbing Status ```typescript const metadata = await provider.getDubbingMetadata(dubbingId); console.log('Status:', metadata.status); // 'dubbing' | 'dubbed' | 'failed' console.log('Target Languages:', metadata.targetLanguages); console.log('Editable:', metadata.editable); ``` ### Dubbing Metadata ```typescript interface DubbingMetadata { dubbingId: string; name: string; status: DubbingStatus; targetLanguages: string[]; editable: boolean; createdAt: Date; mediaMetadata?: { contentType: string; duration: number; }; error?: string; } type DubbingStatus = 'dubbing' | 'dubbed' | 'failed' | 'cloning'; ``` ### Wait for Completion ```typescript const metadata = await provider.waitForDubbing(dubbingId, { pollIntervalMs: 5000, // Check every 5 seconds timeoutMs: 600000, // 10 minute timeout }); if (metadata.status === 'dubbed') { console.log('Dubbing complete!'); } else if (metadata.status === 'failed') { console.log('Error:', metadata.error); } ``` ### Get Dubbed Audio ```typescript const audio = await provider.getDubbedAudio(dubbingId, 'es'); fs.writeFileSync('meditation-spanish.mp3', audio.audio); console.log('Content type:', audio.contentType); console.log('Language:', audio.languageCode); ``` ### Dubbed Audio Response ```typescript interface DubbedAudioResponse { audio: Buffer; contentType: string; languageCode: string; } ``` ### Get Transcript ```typescript const transcript = await provider.getDubbingTranscript(dubbingId, 'es'); console.log('Full text:', transcript.text); for (const segment of transcript.segments) { console.log(`[${segment.startMs}-${segment.endMs}ms] ${segment.text}`); if (segment.words) { for (const word of segment.words) { console.log(` ${word.word} (${word.confidence})`); } } } ``` ### Transcript Response ```typescript interface DubbingTranscriptResponse { languageCode: string; text: string; segments: TranscriptSegment[]; } interface TranscriptSegment { text: string; speakerId?: string; startMs: number; endMs: number; words?: TranscriptWord[]; } interface TranscriptWord { word: string; startMs: number; endMs: number; confidence?: number; } ``` ### Delete Dubbing Project ```typescript await provider.deleteDubbing(dubbingId); ``` ## Supported Dubbing Languages 32 languages supported for dubbing. | Code | Language | Code | Language | | ---- | ---------- | ---- | ---------- | | `en` | English | `ko` | Korean | | `es` | Spanish | `hu` | Hungarian | | `fr` | French | `hi` | Hindi | | `de` | German | `sv` | Swedish | | `it` | Italian | `da` | Danish | | `pt` | Portuguese | `fi` | Finnish | | `pl` | Polish | `no` | Norwegian | | `tr` | Turkish | `el` | Greek | | `ru` | Russian | `he` | Hebrew | | `nl` | Dutch | `ro` | Romanian | | `cs` | Czech | `uk` | Ukrainian | | `ar` | Arabic | `id` | Indonesian | | `zh` | Chinese | `ms` | Malay | | `ja` | Japanese | `vi` | Vietnamese | | `th` | Thai | `bg` | Bulgarian | | `sk` | Slovak | `hr` | Croatian | ### Language Utilities ```typescript import { DUBBING_LANGUAGES, getDubbingLanguageName, isDubbingLanguageSupported, } from './providers/elevenlabs-provider'; // Get language name const name = getDubbingLanguageName('ja'); // 'Japanese' // Check if supported if (isDubbingLanguageSupported('es')) { console.log('Spanish is supported'); } // List all languages for (const [code, name] of Object.entries(DUBBING_LANGUAGES)) { console.log(`${code}: ${name}`); } ``` ## Error Handling ### ElevenLabsError ```typescript interface ElevenLabsError extends Error { code: string; status: number; detail?: string; } ``` ### Error Handling Example ```typescript try { const response = await provider.textToSpeech({ text: 'Hello', voiceId: 'invalid_id', }); } catch (error) { if (error.status === 401) { console.log('Invalid API key'); } else if (error.status === 429) { console.log('Rate limited, retry after cooldown'); } else if (error.status === 400) { console.log('Bad request:', error.detail); } } ``` ## Related Documentation - [OpenRouter Integration](./openrouter-integration.md) - LLM provider - [AI Service](./ai-service.md) - Main AI orchestration - [Prompt Management](./prompt-management.md) - Template system