Esports, Companion & AI Services

A fighting game becomes a sport, and then a show, in three steps that all live outside the match. The match itself is the deterministic, integer-frame simulation described in Rollback Netcode & Tag-Team; everything on this page wraps that simulation without ever being allowed to perturb it. The esports backend turns a verified result into a bracket, a Crown Points standing, and a broadcast; the companion app gives a player a phone-and-watch surface for frame data, replays, store, and coach notes that is deliberately kept off the competitive path; and the AI services — an AI Director that paces single-player encounters and a real-time AI commentary stack that casts a match with synthesized voice, lip-synced faces, ducked music, and live translation — make a V2 broadcast feel produced. The unifying design rule is that V2 builds none of the underlying intelligence: it composes the shared Oshun substrates (Iris, Isis, Psyche, Euterpe, Uzume, Veritas, Shakti) through thin, rollback-aware apps/v2/ bridges, supplies the fighting-game context, and consumes a result it can audit. The section hub is ../V2_ARCHITECTURE.md.

What ships, honestly#

The split here is the same one the sibling Online Backbone & Competitive Integrity page draws, and it is worth stating plainly. The bridges and tooling are real code. apps/v2/ ships the AI-services integration packages — iris-commentary-orchestration, psyche-ai-director-hints, isis-commentary-lipsync-pipeline, psyche-commentary-face-animation, euterpe-commentary-ducking, iris-realtime-translation, and iris-accessibility — each a substantive TypeScript module that composes a shared substrate and enforces a rollback boundary. apps/v2/esports-tools/ ships @v2/esports-tools with three real engines (broadcast, arena LED/projection, fact-checking). apps/oshun/mobile/v2/ ships the companion shell model. The in-engine seams are real Unreal C++: V2/ue/Plugins/V2AICommentary/ and V2/ue/Plugins/V2AdaptiveAI/ carry typed contracts and Blueprint libraries, and the AI Director is a genuine C++ state machine, not a config struct.

The gRPC wire contract for this page's three services now ships: real proto3 contracts live at libs/proto/src/oshun/v2/esports/esports.proto, .../director/ai_director.proto, and .../ai_commentary/commentary.proto (registered in @oshun/proto's loader and service registry), and @v2/grpc-services binds all three — oshun.v2.esports.Esports, oshun.v2.director.Director, oshun.v2.ai_commentary.Commentary — onto one @grpc/grpc-js server backed by the real bridges (@v2/esports-tools + @maat/strategy Crown Points, @v2/psyche-ai-director-hints, @v2/iris-commentary-orchestration), with in-process round-trip tests covering streaming, fallback, and rollback rejection. The other V2 proto packages the architecture names (online, live, progression, community, racing, heist, world-boss) remain the designed contract, not files in the tree. And the generation itself is provider-gated: the bridges orchestrate, validate, budget, and route, but the live commentary text, the synthesized voice, and the generated face animation come from model-backed substrates (and ultimately providers) those bridges call. The orchestration ships; the generation is a capability that fails loud when its provider is absent, exactly the seam the platform uses everywhere.

The esports backend#

V2 treats competitive play as a first-class surface. The backend's job is to take a result the online backbone has already verified against the replay hash, and turn it into a standing and a show — while keeping every one of those operations provably off the deterministic path.

Broadcast, arena, and fact-checking tooling#

@v2/esports-tools is the real operator surface. Its broadcast engine, buildV2EsportsBroadcastPipeline (apps/v2/esports-tools/src/esports-broadcast-pipeline.ts:193), composes seven @uzume/broadcast engines into one plan: a multi-platform streaming engine that upserts per-destination encodings (1080p60 / 8 Mbps default), a stream-health dashboard with adaptive bitrate bands (floor 65%, ceiling 140%, 500 kbps steps), an ATEM-default switcher with CLEAN/PGM/REPLAY aux buses, a director multiviewer layout, three CasparCG graphics templates (scorebug, bracket, next-match countdown), an EVS-LSM replay controller, and SRT contribution feeds. The plan is explicitly inert on the match: it returns offRollback: true, mayInfluenceRollback: false, and deterministicImpact: 'none' (:341). Its reasonCodes builder refuses to let standalone OBS, NDI, or vMix be a primary integration (:542); OBS WebSocket is retained for one purpose only — streamer-mode notification suppression (V2_ESPORTS_BROADCAST_OBS_WEBSOCKET_PURPOSE, :31). Sibling engines in the same package compose @uzume/lumina + @uzume/prism for arena LED walls and projection mapping (presentation/show-control state only, mayInfluenceRollback: false) and @veritas/fact-checking for post-match reporting and bracket-result verification, where a winner-record conflict holds publication without ever changing match authority.

Tournament integrity and Crown Points#

The single load-bearing integrity feature lives one page over but is worth restating: Tournament Mode freezes the balance hotfix at the TO's snapshot and stamps that Tournament Edition into the replay header (see Security, Compliance & Sister-Monorepo Integration), so a recorded set is reproducible against the exact ruleset it was played under. Results are server-authoritative by ingestion shape: V2 pulls results from Start.gg / Challonge / Battlefy via @oshun/inbound-integrations rather than accepting client-pushed scores, and every match is cryptographically verified against its golden-replay hash before it counts. Only then does the Crown Points formula apply — base_points[round] × region_factor × participant_factor × season_decay, with round bases stepping 50 (R32) → 100 (R16) → 200 (QF) → 400 (SF) → 800 (GF) — feeding World Final qualification through the @maat/strategy-owned Pro Circuit calendar. SLOs are concrete: result ingest p99 ≤ 5 s, live bracket fetch p99 ≤ 200 ms, broadcast overlay update p99 ≤ 100 ms.

The results archive and the gRPC seam#

The public-facing spectator surface is real and static: apps/v2/web/esports/results/ ships index.html, results.js, results.css, and a results.json (schema: v2.esports.resultsArchive.v1) that is a searchable tournament archive — searchable by fighter, player, date, and event, with explicit ranking weights (championship 120, major 90, regional 55, local 25, plus win and replay bonuses) and per-event brackets, match scores, and replay URIs. The gRPC service backing live operation — Esports.RegisterEvent(rule_file) → event_id, SubmitResult(event_id, match_result, replay_hash) → ack, GetBracket(event_id), QualifyForFinals(season_id, region) — ships at libs/proto/src/oshun/v2/esports/esports.proto, served by @v2/grpc-services: SubmitResult fact-checks every result through @veritas/fact-checking (winner agreement across bracket, report, and replay records plus the sha256 replay hash) before crediting calculateMaatV2CrownPoints, and a conflict is RESULT_HELD_FOR_REVIEW with zero credit. The integrity claim rides exactly where the backbone page puts it: on the replay-hash verification the engine's determinism already guarantees.

The companion app#

The companion is real and deliberately scoped. apps/oshun/mobile/v2/ (companionAppModel.ts, V2CompanionApp.tsx, V2CompanionLaunchTile.tsx) builds its shell via buildV2CompanionModel() (companionAppModel.ts:394), which composes the shared Oshun client stack — @oshun/shell-core, @oshun/auth-client, @oshun/concordia-integration, @oshun/trust-safety, @iris/agents, and @sophia/client (:31). It is a read surface: move lists and a signature-clip library that are offline-ready, plus replay viewer, profile/match-history, friends-and-party, store-and-wishlist, and Battle Pass surfaces gated behind auth.

Why it sits outside the competitive path#

The companion is, by explicit design, not a participant in a ranked match — an integrity decision as much as a product one, because a phone is not a deterministic match client. Its second-screen mode during a live match is modeled with gameplayAuthority: false, cosmeticOnly: true, optInRequired: true, and a 200 ms latency budget (:265): it receives a low-bandwidth opponent move list and frame-data panels for spectating, and can produce no gameplay effect. The one thing the companion does let you play is a suite of daily reward micro-games (Combo Tap Trial, Drift Steer, Trivia Battle, Daily Puzzle Kombat, Mini Pit Crew), and even those are kept honest: every one carries hmacRequired: true and serverValidated: true (:315), so a reward is signed and server-checked and can never touch a ranked ladder. Push runs over APNS / FCM / web-push with per-category topics; a wearable glance (Apple Watch / Wear OS) surfaces daily stats, friend match starts, and Battle Pass tier progress.

Assistants, RAG, and trust-and-safety#

The companion is where V2 wires the shared assistant plane. buildV2CompanionModel resolves @iris/agents launch readiness (resolveIrisAgentsLaunchReadiness) into a ready/blocked status — a fail-loud seam, not an assumed-available one — and selects a frame-data-query agent and a support/appeal agent, each carrying a requiresHumanReview flag. A local @sophia/client (port 3737) backs the frame-data / codex / wiki-citation RAG surfaces. @oshun/trust-safety classifies assistant-behavior policy hits (the model carries a sample prompt-injection hit) into a severity class with triage/action SLA budgets, and @oshun/concordia-integration registers the appeals nav item, a PII-redacting OTLP telemetry sink, and case-update subscriptions rate-limited to 30/min. The public API the companion and third-party tools share is OAuth 2.1 + PKCE with per-app scopes, an OpenAPI 3.1 document at libs/openapi/v2/companion.yaml, and rate limits of 100/min unauthenticated, 1000/min per registered app, and 30/min on personal data — API p99 ≤ 400 ms, push delivery p99 ≤ 5 s.

AI services: a produced broadcast#

This is where V2's integration ambition shows, and where the rollback boundary is most aggressively defended in code. Two services live here — the AI Director (single-player pacing) and AI commentary (the produced broadcast) — and both are built so that nothing they emit can ever enter the deterministic simulation except as an explicitly serialized, hashed, match-start input.

The AI Director#

@v2/psyche-ai-director-hints is real domain code. It composes @psyche/behavior-prediction — createBehaviorPredictionEngine with a first-order Markov model, a 5-action horizon, and 30 s prediction validity (psyche-ai-director-hints.ts:309) — and maps each predicted action into a fighting-specific tendency vector (pressure, defense, spacing, throwGame, resourceUse) by term-matching the action against move families (deriveTendencyVector, :428). The architecturally critical part is the rollback discipline, and it is enforced, not described: live tendency hints are available only off-rollback (offline/training), and assertValidLiveRequest throws if rollback is enabled (:240). For online rollback-with-CPU, the only legal path is buildV2AdaptiveAIMatchStartSnapshotPlan, which samples Psyche at match load and bakes a serializedTendencyVector plus a sha256 vectorHash into a V2AdaptiveAIDeterministicMatchStartInputVector that v2NetcodeTreatsAsOrdinaryDeterministicInput (:395). Mid-match updates are queued for the next match start; every hint carries mayInfluenceRollback: false. The in-engine V2AdaptiveAI plugin mirrors this: a real C++ state machine EV2AIDirectorState { Calm, Build, Climax, Recovery } (V2AdaptiveAITypes.h:33) whose EvaluateAIDirectorDecision hard-disables the Director — bDirectorActive = false, bUsesStaticRuleOfDay = true — whenever the player opts out, the session is not single-player, or the mode is ranked or tournament (V2AdaptiveAIBlueprintLibrary.cpp:234). Style matchup hints come from @shakti/sota-critical (boxer / striker / grappler / submission-specialist) over an off-rollback shakti.player.style.updated stream. Under the EU AI Act the Director is Limited risk: opt-out switches to a static CPU profile before any classifier use, and a DSR delete cascades to the Director profile, the Psyche tendency vector, and the Shakti classification.

AI commentary and the post-match experience#

@v2/iris-commentary-orchestration composes createMatchCommentaryStream from @iris/conversation-orchestration (v2-ai-commentary.ts:113) into a cue plan whose every field declares the boundary: rollbackPolicy: 'off-rollback-audio-subtitle-only', allowedOutputChannels: ['audio', 'subtitle'], rejectsRollbackFrameOrigin: true. Validation rejects fromRollbackFrame outright (:244) and requires a fallbackSeed, because the real-time budget is a hard contract: the 800 ms KO-trigger budget decomposes into 420 ms LLM round trip + 240 ms persona TTS + 100 ms lipsync + 40 ms safety margin (:23), and on a miss the system falls back to a pre-recorded bank keyed by (cueKind, ruleset, commentatorId, seed). The mirror C++ contract FV2AICommentaryCueContract carries the same budgets and the same off-rollback flags (bOffRollbackBubble, bGameplayInertOutput, bFeedsSimulationState = false, V2AICommentaryTypes.h:53). The post-match path buildV2AIPostMatchExperiencePlan (:445) is real selection logic: it sorts highlight candidates by score and takes the top three to five, requires the canonical moment kinds (longest-combo, counter-hit-punish, power-play, dramatic-comeback, KO), and emits an analysis screen, an auto-captioned clip library, and a compliance block whose disclosure text — "AI commentary is machine-generated presentation content." — is mandatory under EU AI Act cross-ref §80.4. Per deps§67, the generation providers are named (Anthropic Claude Sonnet 4 — the claude-sonnet-4 registry default, with the exact version a configuration choice — for play-by-play text, ElevenLabs Multilingual v2 for TTS, Isis ai-video for the highlight auto-edit), with on-cap routing to @iris/conversation-providers-local; the Calliope commentator roster lives at V2/ue/Content/V2/Audio/Commentary/commentator-personas.json with cleared voice-clone metadata and talent signoff.

The supporting render bridges#

The remaining bridges turn commentary text into a finished broadcast, each a real package composing a real substrate:

• isis-commentary-lipsync-pipeline (@isis/visual-dubbing) extracts phonemes and viseme timing, drives visual dubbing, validates ≤ 45 ms lip-sync drift, and exports UE curves to /Game/V2/Audio/Commentary/Lipsync — a real replacement for Audio2Face.
• psyche-commentary-face-animation reads @iris/voice output metadata and drives @psyche/avatar-lipsync viseme frames plus @psyche/avatar-expressions ARKit expression curves for the caster face.
• euterpe-commentary-ducking does genuine audio-engineering math: computeArrangementDensity from @euterpe/accompany and the @euterpe/master masteringChain.dbToLinear to sidechain-duck V2.DynamicMusic under V2.Commentary to a −18 LUFS commentary target with a −1 dB true-peak ceiling (commentary-ducking.ts:166).
• iris-realtime-translation (@iris/voice) localizes spectator chat, commentary subtitles, broadcast overlays, and the companion second-screen feed; iris-accessibility composes @iris/accessibility + @psyche/caption-streaming.

Every one of these declares mayInfluenceRollback: false and routes only audio/visual/subtitle output back to V2Audio — review-friendly, off the deterministic path, and degrading to silence or the fallback bank rather than fabricating a caster when a provider is absent.

Edge cases and failure modes#

• The rollback envelope is sealed in code, not prose. Psyche, Iris, Shakti, and Calliope can be called freely off-rollback, but inside the rollback loop the only Psyche-derived input is a hashed, match-start-sampled vector; live RPCs are rejected (assertValidLiveRequest throws; the cue validator rejects fromRollbackFrame).
• Latency is a contract with a fallback, not a hope. If the 420/240/100/40 ms commentary budget is exceeded, the cue resolves to the pre-recorded bank keyed by (cueKind, ruleset, commentatorId, seed) — deterministic and provider-free.
• The Director is off in ranked and tournament, by construction. EvaluateAIDirectorDecision falls to the static rule-of-the-day the instant a session is ranked, tournament, multiplayer, or opted-out — no profile is even consulted.
• Results are pulled and hash-verified, never pushed. A bracket integration cannot inject a result; ingestion verifies the golden-replay hash before Crown Points credit, and Veritas holds publication on a winner-record conflict without touching match authority.
• The companion cannot reach the ladder. Second-screen is cosmeticOnly with gameplayAuthority: false; the only playable surface is HMAC-signed, server-validated daily reward micro-games.

Where this connects#

• Sideways: Online Backbone & Competitive Integrity (the backbone the companion reads, the ranked-pro circuit, and the replay-hash determinism the esports backend depends on), Live-Ops, Store, Progression & Community (the @maat/strategy calendar, Crown Points windows, and store the companion surfaces), and Telemetry, Performance, Testing & Release Gates (the v2.match.* stream the AI Director and commentary consume).
• Integrity: Security, Compliance & Sister-Monorepo Integration (Tournament Mode hotfix freeze, ranked-pro MFA, and the EU AI Act conformity export for the Director and commentary).
• Platform: the domain-orchestration layer that owns the Iris / Isis / Psyche / Euterpe / Uzume / Veritas / Shakti substrates these apps/v2/ bridges compose, and the contracts tier that defines the @oshun/event-bus topics they publish and subscribe.
• The section hub: ../V2_ARCHITECTURE.md.