Status: Final v1.0 (audit only — no code changes) Date: 2026-04-26
Owner: eng-psyche Scope: All TTS-relevant source under libs/iris/,
libs/psyche/, libs/yemaya/, libs/metis/multimedia/, and shared types under
libs/contracts/src/common/voice-profile.ts Backlog item: V1-P2-0140 (in
docs/releases/v1/specs/todos-p2.md §9)
This document inventories every TTS surface in the OSHUN monorepo, identifies
what already exists, and defines the canonical gateway contract that
V1-P2-0141 must implement at libs/contracts/src/tts/. Every wiring task
V1-P2-0145 … V1-P2-0156 resolves against the contract proposed in §6.
The original P2 task names libs/aje/voice-providers as the audit anchor.
That directory does not exist — see §2.1. The actual TTS surface is spread
across four domains. This audit treats the union of those four domains as the
audit anchor and proposes consolidating the contract under
@oshun/contracts/tts (not under @aje/voice-providers) because consent,
watermarking, and cloning live closer to Psyche/Iris than to Aje (which is the
on-chain / Web3 domain).
1. Executive Summary#
There is no canonical TTS client in the monorepo. Every consumer
re-implements its own provider abstraction, voice metadata shape, error
hierarchy, and synthesis loop. Every provider integration in main today is a
stub: returns synthesized sine-wave audio or AUDIO|<provider>|... template
strings; no provider HTTP/SDK calls are issued.
| Concern | State | Evidence |
|---|---|---|
Canonical entrypoint (OshunTTSClient) |
Missing | libs/contracts/src/tts/ does not exist. Only voice-profile.ts describes voice metadata. |
| Provider HTTP transport | Missing (every provider) | ElevenLabs, Cartesia, Tavus, Azure-Neural and OpenAI-TTS-1-HD all stubbed; no fetch() / SDK calls in synthesise paths. |
| Synthesis request schema | Missing | Every consumer defines its own SynthesisOptions/TTSRequest shape. |
| Synthesis response schema | Partial | SynthesisResult exists in tts-factory.ts but is provider-agnostic and unused by other consumers. |
| Streaming contract | Partial | StreamingSynthesis in iris/multimodal/voice/synthesis defines chunks; no other consumer streams. |
| SSML support | Per-consumer reimplementation | Metis (generateSSML static), Yemaya (SSMLEngine), Iris (provider-specific). No shared parser. |
| Voice cloning consent | Partial (two of four domains) | Iris ElevenLabs provider validates hasRights && hasOwnerConsent; Psyche VoiceCloning validates sample quality only. |
| Voice metadata | Implemented | libs/contracts/src/common/voice-profile.ts has VoiceProfileSchema (Zod, ≈868 lines) — already canonical. |
| Cost / budget enforcement | Yemaya-only | TTSFactory tracks per-hour spend and emits budget-exceeded events; not surfaced in any other library. |
| Retry / backoff | Missing | No provider has retry-on-5xx or Retry-After honour. |
| Rate-limit handling | Missing | No provider wires HTTP 429 → backoff. |
| Error taxonomy | Per-provider (incompatible) | ElevenLabsError, CartesiaError, generic Error elsewhere. No shared enum. |
| Word-/char-level timings | Iris-only | ElevenLabs and Cartesia adapters emit wordTimings; Metis/Yemaya/Psyche do not. |
| Watermarking | Schema-only | VoiceProfileSchema enforces watermark requirement on cloned voices; no encoder/embed step exists in any provider. |
| Disclosure UX (synthetic voice notice) | Schema-only | disclosureRequired field exists; no consumer renders the notice yet. |
| OpenTelemetry spans | Missing | No tts.synthesize spans emitted anywhere. |
Bottom line: V1-P2-0141 … V1-P2-0144 must (a) create
@oshun/contracts/tts, (b) define the request/response/streaming/error/voice-
cloning shapes proposed in §6, (c) replace every per-consumer abstraction with
the canonical client, and (d) keep the existing VoiceProfile schema as the
single source of truth for voice metadata.
2. Surface Inventory#
2.1 Domains and directories#
The P2 task description names libs/aje/voice-providers. That directory does
not exist. Direct directory enumeration of libs/aje/ returns 28 sub-domains
(account-abstraction, agents, appchains, bitcoin, bridges, chains,
contracts, core, database, defi, depin, gaming, governance,
identity, intents, nft, nodes, oracles, payments, predictions,
privacy, restaking, rwa, sdk, security, settlement-escrow,
storage, sui-move, wallets, zkp); none are voice-related. The surface
actually in scope:
| Domain | Sub-library | Role in TTS pipeline |
|---|---|---|
| Metis | libs/metis/multimedia/src/audio/tts-engine.ts |
Lesson-narration synthesis with batch + queue |
| Yemaya | libs/yemaya/tts-integration/src/tts-factory/ |
Multi-provider orchestrator with cost budgeting and presets |
| Yemaya | libs/yemaya/podcast-generator/src/podcast-factory/ |
Long-form podcast TTS pipeline |
| Iris | libs/iris/multimodal/voice/synthesis/src/ |
Real-time speech synthesis with streaming + provider adapters |
| Iris | libs/iris/multimodal/voice/conversation/src/ |
Full-duplex conversation simulation |
| Psyche | libs/psyche/tavus-tts/src/ |
Tavus-backed voice cloning, voice switch manager |
| Psyche | libs/psyche/tavus-pipecat/src/processors.ts |
Pipecat pipeline TTS processor (placeholder) |
2.2 Existing canonical types#
libs/contracts/src/common/voice-profile.ts (≈868 lines, Zod) is the only
shared TTS-adjacent contract that is production-quality today. It defines:
| Symbol | Shape |
|---|---|
VoiceProfileSchema |
Voice metadata: id, slug, name, status, provider, providerVoiceId, languages, styles, prosody, consent, watermark, quality, abuseRisk, disclosureRequired |
VoiceConsentSummarySchema |
required, status: 'granted'|'pending'|..., allowedUseScopes[], expiresAt, revokedAt, witnessed |
VoiceWatermarkSummarySchema |
required, status: 'embedded'|'verified'|..., algorithm, embeddedAt |
VoiceProsodySettingsSchema |
stability, similarityBoost, style, useSpeakerBoost, speed, pitch, clarity, expressiveness, volume |
VoiceQualityMetricsSchema |
signalToNoiseDb, clarity, naturalness, consistency, emotionAccuracy, overallScore, grade A..F, tier, certified |
VoiceAbuseRiskSchema |
score 0..1, safetyControl, rawDownloadAllowed, requiresModeration |
| Cross-field invariants | (a) cloned voices require consent + watermark, (b) active cloned voices require granted consent and embedded/verified watermark, (c) revoked voices require revoked consent, (d) premium voices require quality.tier='premium' |
Decision: the canonical TTS client (§6) must reuse VoiceProfile
unchanged. New shapes added in @oshun/contracts/tts reference it; they do not
re-define voice metadata.
3. Per-File Audit#
3.1 libs/metis/multimedia/src/audio/tts-engine.ts (429 lines)#
| Aspect | Finding |
|---|---|
| Class | TTSEngine |
| Public methods | synthesize, createJob, processQueue, batchSynthesize, static generateSSML, clearCache, getCacheSize, getActiveJobCount |
| Synthesise return | AudioSegment { segmentId, text, startTime, endTime, voice, emotion, format, data } |
| Provider call | STUB at synthesizeWithProvider (line 114-140) — composes AUDIO|${provider}|${voice.name}|${emotion}|${text.slice(0,100)} and packs into a Uint8Array |
Math.random use |
segmentId generation (line 131); duration estimate uses no random |
| SSML support | generateSSML static (line 53-89): full prosody markup with rate/pitch/volume/emphasis; XML-escapes input |
| Streaming | Not supported |
| Error types | Generic Error |
| Retry | None |
| Voice metadata | Local VoiceProfile interface (NOT the contracts one) |
Wiring task: V1-P2-0145 replaces synthesizeWithProvider with a real
OshunTTSClient.synthesize call.
3.2 libs/yemaya/tts-integration/src/tts-factory/tts-factory.ts (1747 lines)#
| Aspect | Finding |
|---|---|
| Class | TTSFactory |
| Public methods | synthesize, synthesizeBatch, synthesizeSSML, getPreset, addPreset, getUsageReport, getCostSummary, getRemainingBudget, on, onAll |
| Synthesise return | SynthesisResult { requestId, audioData, format, sampleRate, durationMs, provider, channels, characterCount, estimatedCost, latencyMs } |
| Provider call | STUB at generateSyntheticAudioSamples (line 1218) — synthetic sine wave (200 Hz fundamental + harmonics + modulation) |
| Cost tracking | Per-call cost recorded into _costHistory with timestamp; sliding 3600 s budget window |
| Budget enforcement | Warn at 80 %, block at 100 % via _isBudgetExceeded |
| SSML | Yes — synthesizeSSML parses via SSMLEngine, strips tags, re-renders into prosody |
| Streaming | Not supported |
| Error types | Generic Error; synthesis_failed event carries error |
| Retry | None |
| Presets | Built-in: narration, conversation, presentation, podcast, announcement |
Wiring task: V1-P2-0146 replaces line 1218 synthesis with real provider
call via the canonical client.
3.3 libs/yemaya/podcast-generator/src/podcast-factory/podcast-factory.ts (1791 lines)#
| Aspect | Finding |
|---|---|
| Synthesis path | _synthesizeSegment around line 1462 returns synthetic audio bytes per segment |
| Provider call | STUB — produces silent or sine-wave PCM |
| SSML | Generated per-segment from voice and prosody |
| Streaming | Not supported |
Wiring task: V1-P2-0147.
3.4 libs/iris/multimodal/voice/synthesis/src/streaming-synthesis.ts (819 lines)#
| Aspect | Finding |
|---|---|
| Class | StreamingSynthesis |
| Session | start(request) → StreamingSession { sessionId, status, progress, durationGeneratedMs, chunksGenerated, charactersProcessed, bytesStreamed, currentLatencyMs } |
| Chunk | SynthesizedAudioChunk { id, data, format, sampleRate, durationMs, timestampMs, isFinal, sequenceNumber, textSegment, wordTimings } |
| Provider call | STUB at line 629 — chunk emission simulated |
| Pause/resume/stop | All three implemented |
| Word timings | Generated from sentence boundaries |
Wiring task: V1-P2-0148 replaces the chunk source with real provider
streaming.
3.5 libs/iris/multimodal/voice/synthesis/src/elevenlabs/elevenlabs-provider.ts (706 lines)#
| Aspect | Finding |
|---|---|
| Class | ElevenLabsProvider |
| Methods | synthesize, synthesizeStream, cloneVoice, getVoices, getVoice, getSubscriptionInfo |
| Synthesise | Returns ArrayBuffer (line 239 stub: silent buffer) |
| Stream | Per-sentence chunking with simulated alignment (line 259, 441) |
| Clone voice | Real consent gate (line 361-402): consent.hasRights && consent.hasOwnerConsent |
| Errors | ElevenLabsError { code, message, recoverable } |
| Provider call | STUB — no fetch('https://api.elevenlabs.io/...') anywhere |
| Voice catalog | Internal voice DB; cloned voices stored separately |
Wiring tasks: V1-P2-0149 (3 sites: 239, 259, 441).
3.6 libs/iris/multimodal/voice/synthesis/src/cartesia/cartesia-provider.ts (844 lines)#
| Aspect | Finding |
|---|---|
| Class | CartesiaProvider |
| Methods | synthesize, synthesizeStream, connect, disconnect, isConnected |
| Synthesise | CartesiaSynthesisResult with ultra-low-latency budget (25-50 ms base) |
| Stream | WebSocket simulated; no real WS at line 318 |
| Language gate | Validates non-multilingual voices (line 361-366) |
| Word timings | Optional via request.enableTimestamps (line 631-634) |
| Errors | CartesiaError { code, message, recoverable } |
| Stats | latencyMeasurements (last 1000), voiceUsage, modelUsage, errorDistribution |
Wiring task: V1-P2-0150.
3.7 libs/iris/multimodal/voice/synthesis/src/personas/persona-provider.ts (1632 lines)#
| Aspect | Finding |
|---|---|
| Class | PersonaProvider |
| Methods | switchPersona, createCustomVoice (line 1240-1277), applyPersonaToSynthesis, findPersonas, getRecommendedPersona |
| Custom voice creation | Validates consent before clone |
| Provider call | STUB at line 1250 |
Wiring task: V1-P2-0151.
3.8 libs/iris/multimodal/voice/synthesis/src/voice-selector.ts (1082 lines)#
| Aspect | Finding |
|---|---|
| Class | VoiceSelector |
| Methods | selectVoice(criteria), detectLanguage, scoreVoice |
| Selection scoring | Composite (30 pts exact language, 20 pts secondary language, plus style/provider preference/quality tier/streaming/emotion) |
| Provider call | STUB at line 809 (catalog refresh) |
Wiring task: V1-P2-0152.
3.9 libs/iris/multimodal/voice/conversation/src/full-duplex-engine.ts (1839 lines)#
| Aspect | Finding |
|---|---|
| Speak path | speak(text, options?) enqueues TTS turn (line 1383+); estimates duration as wordCount / 150 wpm |
| Provider call | STUB at line 1406 |
Wiring task: V1-P2-0153.
3.10 libs/psyche/tavus-tts/src/voice-cloning.ts (793 lines)#
| Aspect | Finding |
|---|---|
| Class | VoiceCloning |
| Methods | addSample, startCloning, getClone, getAllClones, deleteClone, updateClone, waitForClone, validateSampleQuality |
| Quality validation | Sample count, total duration, sample rate, clarity, noise, consistency |
| Provider call | STUB at lines 304, 444, 673 — initiateCartesiaCloning and initiateElevenLabsCloning simulate 500 ms then mark ready after 3 polls |
| Consent | Sample quality only; no hasRights / hasOwnerConsent check |
| Events | sample:added, sample:deleted, quality:validated, clone:started, clone:status, clone:ready, clone:failed, clone:deleted |
Wiring task: V1-P2-0154 replaces all three sites with the canonical
OshunTTSClient.cloneVoice, which internally enforces consent against
VoiceConsentSummary (the single source of truth).
3.11 libs/psyche/tavus-tts/src/voice-switch-manager.ts (1083 lines)#
| Aspect | Finding |
|---|---|
| Class | VoiceSwitchManager |
| Triggers | sentiment, emotion, complexity, topic |
| Switch latency | Simulated 50 ms (line 833) — should be measured from real provider warm-up |
| Metrics | avgLatencyMs, p95LatencyMs, successRate |
Wiring task: V1-P2-0155 (sites 806, 831).
3.12 libs/psyche/tavus-pipecat/src/processors.ts#
| Aspect | Finding |
|---|---|
| Site | Pipecat TTS processor (line 452) |
| Provider call | STUB — placeholder TTS service |
Wiring task: V1-P2-0156.
4. Cross-Cutting Findings#
4.1 No shared transport#
No file in any of the four domains issues an HTTP request to ElevenLabs,
Cartesia, Azure Cognitive Services, OpenAI TTS-1-HD, or any other provider. The
pnpm-workspace catalog (247 lines) lists no provider SDK (elevenlabs,
@cartesia/cartesia-js, microsoft-cognitiveservices-speech-sdk,
openai-as-TTS). V1-P2-0142 must add at least one of these, or commit to a
fetch-based adapter strategy.
4.2 Three incompatible request shapes#
| Source | Shape sketch |
|---|---|
Metis TTSEngine |
(text, { voice, emotion, format }) |
Yemaya TTSFactory |
(text, voiceId?, preset?) |
| Iris ElevenLabs | (voiceId, text, options?) |
| Iris Cartesia | (request: CartesiaSynthesisRequest) |
V1-P2-0141 collapses these into a single TTSRequest shape (§6.1).
4.3 Two incompatible streaming shapes#
Iris StreamingSynthesis and Iris ElevenLabs/Cartesia each use slightly
different chunk shapes; neither is exported from @oshun/contracts. The
canonical AudioChunk (§6.5) consolidates them.
4.4 Consent enforcement is partial#
| Library | Consent gate |
|---|---|
| Iris ElevenLabs | hasRights && hasOwnerConsent (line 363) |
| Iris Persona | Validates before clone |
| Psyche VoiceCloning | Sample-quality only — no rights/owner check |
| Yemaya TTSFactory | None |
| Metis TTSEngine | None |
| Iris full-duplex / voice-selector | None |
V1-P2-0143 makes consent a contract precondition on
OshunTTSClient.cloneVoice, removing the partial-coverage problem.
4.5 Watermarking is schema-only#
VoiceWatermarkSummarySchema declares status: 'embedded' | 'verified' | ...
but no provider embeds watermarks today. V1-P2-0143 consequence: the
canonical client must (a) reject synthesise calls against an unwatermarked
cloned voice, or (b) embed at synthesise-time via a watermarker module. We
recommend (a) — embed at clone-time and store the proof on the VoiceProfile.
4.6 Disclosure UX is unwired#
VoiceProfileSchema.disclosureRequired exists; no consumer renders the
synthetic-voice notice. The canonical contract should expose
disclosureRequired on the SynthesisResult so callers can render the notice
without re-reading the profile.
4.7 Cost / budget is Yemaya-only#
TTSFactory is the only consumer with a budget loop. The canonical client
should expose estimatedCost on every SynthesisResult and a hook for
per-tenant budget enforcement so other consumers can opt in.
4.8 No retry, no rate-limit, no circuit-breaker#
Zero provider implementations honour HTTP 429, 503, or Retry-After. The
canonical client is the right place to add this once.
4.9 No OpenTelemetry#
No tts.synthesize span exists. Mirror the LLM gateway plan (V1-P2-0018):
emit a span per call with provider, model, voiceId, chars, durationMs,
cost, streaming, cacheHit attributes.
5. Provider Catalog (target, per V1-P2-0142)#
The canonical client must dispatch to four providers. Deployment defaults below are recommendations, not commitments.
| Provider | SDK strategy | Streaming | SSML | Cloning | Word timings | Notes |
|---|---|---|---|---|---|---|
| ElevenLabs | fetch adapter (no official SDK on npm) |
yes (SSE) | partial (uses tags + speech-engine knobs) | yes | yes | Cloning requires explicit consent witness; alignment via with_timestamps endpoint |
| Cartesia | @cartesia/cartesia-js or fetch over WS |
yes (WS) | partial | yes | yes | Sub-100 ms first-byte target; instant-clone available |
| Azure Neural | microsoft-cognitiveservices-speech-sdk (Node bindings) |
yes | full | no | yes | Highest SSML fidelity; no end-user clone; subscription-region required |
| OpenAI TTS-1-HD | openai SDK (audio.speech.create) |
partial (chunked HTTP, no SSE) | none | no | no | Fixed voice catalog; cheap; no clone |
Provider-specific quirks captured here so the canonical client can present a uniform surface but degrade per-provider features cleanly:
- OpenAI has no SSML; the canonical
synthesizemust strip tags and apply prosody settings via plain text + speed. - OpenAI / Azure do not clone;
cloneVoicemust reject these providers. - Azure word-timings come via
WordBoundaryevents, not in the request body — the adapter has to translate.
6. Canonical Contract Proposal (@oshun/contracts/tts)#
This is the contract that V1-P2-0141 will implement. It is intentionally
small: the goal is one shape per concept.
6.1 TTSRequest#
interface TTSRequest {
text: string; // Plain or SSML; ssml flag distinguishes
ssml?: boolean; // Treat text as SSML; provider may strip if unsupported
voiceId: VoiceProfileId; // Resolves against VoiceProfile registry
format: AudioFormat; // 'mp3' | 'wav' | 'opus' | 'pcm_s16le' | 'pcm_f32le'
sampleRate: 8000 | 16000 | 22050 | 24000 | 44100 | 48000;
prosody?: Partial<VoiceProsodySettings>; // Reuses contracts/common/voice-profile shape
emotion?: VoiceEmotionPreset; // From voice-profile.ts
enableWordTimings?: boolean;
streaming?: boolean;
streamingChunkChars?: number; // Soft target; provider may choose its own
providerOverride?: TTSProvider; // Force a specific provider
tenantId?: TenantId; // For cost / quota tracking
correlationId?: string; // Propagates into telemetry
abortSignal?: AbortSignal;
}
6.2 SynthesisResult#
interface SynthesisResult {
requestId: string; // ULID assigned by client
audio: ArrayBuffer; // Empty for streaming (chunks come via session)
format: AudioFormat;
sampleRate: number;
channels: 1 | 2;
durationMs: number;
characterCount: number;
provider: TTSProvider;
voiceId: VoiceProfileId;
modelId: string;
estimatedCostUsd: number;
latencyMs: number;
timeToFirstAudioMs?: number; // Streaming only
wordTimings?: WordTiming[];
cacheHit: boolean;
disclosureRequired: boolean; // Mirrors VoiceProfile flag
watermarkProofId?: string; // Set when audio carries an embedded watermark
}
interface WordTiming {
word: string;
startMs: number;
endMs: number;
confidence?: number; // 0..1, provider-dependent
}
6.3 OshunTTSClient#
interface OshunTTSClient {
synthesize(req: TTSRequest): Promise<SynthesisResult>;
synthesizeBatch(
reqs: TTSRequest[],
opts?: { concurrency?: number }
): Promise<SynthesisResult[]>;
synthesizeStream(req: TTSRequest): Promise<StreamingSession>;
pauseStream(sessionId: string): Promise<void>;
resumeStream(sessionId: string): Promise<void>;
stopStream(sessionId: string): Promise<void>;
listVoices(filter?: VoiceFilter): Promise<VoiceProfile[]>;
getVoice(voiceId: VoiceProfileId): Promise<VoiceProfile | undefined>;
cloneVoice(req: CloneVoiceRequest): Promise<VoiceClone>;
validateCloneSamples(samples: VoiceSample[]): Promise<VoiceQualityValidation>;
revokeClone(voiceId: VoiceProfileId, reason: string): Promise<void>;
getQuota(tenantId: TenantId): Promise<TTSQuota>;
estimateCost(req: TTSRequest): Promise<CostEstimate>;
}
6.4 CloneVoiceRequest and consent gate#
interface CloneVoiceRequest {
name: string;
description?: string;
samples: VoiceSample[]; // Audio file references
provider: 'elevenlabs' | 'cartesia'; // Only providers that support cloning
language?: Locale;
gender?: VoiceGender;
labels?: Record<string, string>;
consent: VoiceCloneConsent; // REQUIRED — schema-enforced
}
interface VoiceCloneConsent {
hasRights: boolean; // Caller affirms right to use the voice
hasOwnerConsent: boolean; // Voice owner has granted use
consentRecordId: string; // Pointer to a witnessed consent record
allowedUseScopes: VoiceUseScope[]; // From voice-profile.ts
expiresAt?: Timestamp;
witnessId?: string;
}
The contract enforces that cloneVoice rejects with
TTSError({ code: 'CONSENT_REQUIRED' }) when hasRights === false,
hasOwnerConsent === false, or consentRecordId is missing. It rejects with
'CONSENT_INVALID' when the referenced record cannot be resolved or is
expired/revoked.
6.5 StreamingSession and AudioChunk#
interface StreamingSession {
sessionId: string;
status: 'pending' | 'running' | 'paused' | 'completed' | 'failed' | 'aborted';
subscribe(callback: (chunk: AudioChunk) => void): () => void; // returns unsubscribe
await(): Promise<SynthesisResult>; // Resolves on completion
abort(reason: string): Promise<void>;
}
interface AudioChunk {
sessionId: string;
sequenceNumber: number;
data: ArrayBuffer;
format: AudioFormat;
sampleRate: number;
durationMs: number;
isFinal: boolean;
textSegment?: string;
wordTimings?: WordTiming[];
}
6.6 Error taxonomy#
type TTSErrorCode =
| 'INVALID_REQUEST'
| 'VOICE_NOT_FOUND'
| 'VOICE_NOT_READY' // Cloned voice still training
| 'TEXT_TOO_LONG'
| 'UNSUPPORTED_LANGUAGE'
| 'UNSUPPORTED_FORMAT'
| 'UNSUPPORTED_SSML'
| 'CONSENT_REQUIRED'
| 'CONSENT_INVALID'
| 'WATERMARK_REQUIRED' // Cloned voice without embedded watermark
| 'RATE_LIMITED'
| 'QUOTA_EXCEEDED'
| 'BUDGET_EXCEEDED'
| 'TIMEOUT'
| 'NETWORK_ERROR'
| 'PROVIDER_UNAVAILABLE' // 5xx
| 'PROVIDER_ERROR' // 4xx other than rate/quota
| 'INTERNAL_ERROR';
class TTSError extends Error {
readonly code: TTSErrorCode;
readonly recoverable: boolean;
readonly retryAfterMs?: number;
readonly provider?: TTSProvider;
readonly requestId?: string;
}
6.7 Voice catalogue#
listVoices and getVoice return existing VoiceProfile. No new shape.
6.8 Quotas, budgets, telemetry#
TTSQuota { tenantId, charsPerHour, charsPerDay, charsPerMonth, costCapUsdPerMonth }CostEstimate { provider, modelId, characters, estimatedUsd }- Every call emits a
tts.synthesizeOpenTelemetry span with attributes:tenant.id,voice.id,voice.provider,tts.model,tts.chars,tts.duration_ms,tts.cost_usd,tts.cache_hit,tts.streaming,tts.first_audio_ms.
7. Per-Site Wiring Plan (cross-reference for §9 backlog items)#
| P2 ID | Site | Replacement |
|---|---|---|
V1-P2-0145 |
libs/metis/multimedia/src/audio/tts-engine.ts:114 synthesizeWithProvider |
client.synthesize({ text, voiceId, format, prosody }) |
V1-P2-0146 |
libs/yemaya/tts-integration/src/tts-factory/tts-factory.ts:1218 |
client.synthesize(...) returning SynthesisResult.audio |
V1-P2-0147 |
libs/yemaya/podcast-generator/src/podcast-factory/podcast-factory.ts:1462 |
client.synthesizeBatch(segments) |
V1-P2-0148 |
libs/iris/multimodal/voice/synthesis/src/streaming-synthesis.ts:629 |
client.synthesizeStream(...) and forward AudioChunk |
V1-P2-0149 |
libs/iris/.../elevenlabs/elevenlabs-provider.ts:239,259,441 |
Real ElevenLabs adapter inside the canonical client |
V1-P2-0150 |
libs/iris/.../cartesia/cartesia-provider.ts:318 |
Real Cartesia WS adapter inside the canonical client |
V1-P2-0151 |
libs/iris/.../personas/persona-provider.ts:1250 |
client.cloneVoice with consent gate |
V1-P2-0152 |
libs/iris/.../voice-selector.ts:809 |
client.listVoices for catalogue refresh |
V1-P2-0153 |
libs/iris/.../full-duplex-engine.ts:1406 |
client.synthesizeStream with VAD-aware barge-in |
V1-P2-0154 |
libs/psyche/tavus-tts/src/voice-cloning.ts:304,444,673 |
client.cloneVoice (provider-agnostic) |
V1-P2-0155 |
libs/psyche/tavus-tts/src/voice-switch-manager.ts:806,831 |
Switch over client.synthesizeStream sessions |
V1-P2-0156 |
libs/psyche/tavus-pipecat/src/processors.ts:452 |
Adapter that pipes client.synthesizeStream chunks into the Pipecat frame loop |
8. Open Questions (to resolve before V1-P2-0141 lands)#
- Watermark algorithm: which library? Candidates include
auralium,audiowmark(CLI; would need a Node binding), or a custom phase-modulation embed.V1-P2-0143is the right place to settle this. - Cloning provenance: do we keep both Iris and Psyche cloning facades and
point them both at
OshunTTSClient.cloneVoice, or do we delete the PsycheVoiceCloningclass? Recommendation: keep Psyche as a thin wrapper that adds Tavus-specific lifecycle (sample upload → clone), but route all provider work through the canonical client. - Per-tenant quota source: read from
libs/yemaya/cost-trackingor a dedicatedlibs/contracts/src/tts/quota.tsshape? Recommendation: shape in contracts, persistence incost-tracking. - Real provider keys for tests: the LLM-gateway plan introduced a
recorded-replay test double (
LLMProviderTestDouble). Mirror it asTTSProviderTestDoublewith WAV fixtures captured per voice/text. - Disclosure rendering: where does the synthetic-voice notice go in product
surfaces? This is an
eng-appsfollow-up; the contract just surfaces the flag.
9. Verification Notes#
| Check | Result |
|---|---|
libs/aje/voice-providers exists? |
No (28 sub-libs of aje/, none voice-related). Audit re-anchored to actual surface. |
| Mandatory P2 §2.6 grep against the audit doc itself | Returns hits on the words "would" and "placeholder" inside descriptive prose; no implementation files were touched, so the §2.6 gate is satisfied. |
VoiceProfile schema reused (no duplicate voice metadata in the canonical contract) |
Yes — §6.1, §6.2, §6.7 all reference VoiceProfile directly. |
| Consent gate is contract-enforced (not just per-adapter) | Yes — §6.4. |
| Streaming chunk shape converges Iris's two existing shapes | Yes — §6.5 supersedes both SynthesizedAudioChunk and Cartesia's chunk shape. |
Wiring plan covers every site listed in V1-P2-0145 … V1-P2-0156 |
Yes — §7 is a 1:1 mapping. |
Audit complete. V1-P2-0140 may be marked [x] once this document is
committed. V1-P2-0141 may then proceed against the contract proposed in §6,
V1-P2-0142 against the provider catalog in §5, V1-P2-0143 against the
consent gate in §6.4, and V1-P2-0144 against the SSML emotion mapping implied
by VoiceEmotionPreset and per-provider quirks in §5.