Online Services & Infrastructure

A heist crew forming up in 1947 Urban, a posse queuing for a Frontier showdown, a 5v5 Capital Raid in the Sci-Fi cell, and a lone detective syncing a Mind Palace clue from the companion app on a train all touch the same backbone. They sign in through one account, carry one Bureau XP ledger, route through one skill-rated matchmaker, and land their scores, replays, and cosmetics in one set of domain-isolated stores. That backbone is not a diagram on a slide: it is sixteen deployable services under apps/v5, each a TypeScript/NestJS package with a real Postgres/Redis domain behind it, fronted by a sixteen-endpoint Unreal C++ client catalog in V5/ue/Source/V5OnlineServices that speaks to them over genuine HTTP. This page is the product-and-code tour of that layer: the services, what each really does, the request seam that connects client to cloud, and the backend infrastructure — containers, clusters, regions, health, and graceful degradation — that keeps it standing. The connection models, PvP, ranked, and esports surfaces that ride this backbone live in Modes & Multiplayer; the seasons, calendar, workshop marketplace, and community programs are in Live Service & Community; the engine-side save, persistence, and cross-play machinery is the architecture companion, Persistence, Online Services & Cross-Play. For the full feature scope this slots into, start at the hub: ../V5_features.md.

What ships, honestly#

The split is clean, and stating it up front keeps the rest of the page honest. Two tiers are real code, and a small set of seams are labelled as plan or fixture rather than wired integration.

The client tier is real Unreal C++. V5/ue/Source/V5OnlineServices ships a sixteen-endpoint service catalog (UV5_Online_ServiceCatalog::BuildEndpointCatalog) fronted by a family of UBlueprintFunctionLibrary request builders — one per service (UV5_Online_AuthService, …MatchmakingService, …AntiCheatClient, …WorkshopClient, and the rest) plus a cross-cell-world builder — over a typed V5OnlineServicesTypes.h surface. Crucially the transport underneath is genuine network I/O: V5OnlineServices.Build.cs declares the HTTP and Json modules (the build comment marks this "the first FHttpModule usage in V5 UE"), and FV5OnlineHttp::BuildHttpRequest turns a request value into a real IHttpRequest from FHttpModule::Get(). Authentication is real cryptography: FV5JwtCrypto is a self-contained FIPS 180-4 SHA-256, RFC 2104 HMAC-SHA256, RFC 7519 HS256 JWT, and RFC 7636 PKCE S256 implementation, unit-tested against published vectors.

The service tier is real TypeScript + NestJS. apps/v5 holds sixteen service packages — each with a contract.json, a Dockerfile, and k8s/ manifests — and every one delegates its handle() to handleRealRequest in @v5/service-shared, which resolves live pg.Pool + ioredis clients and dispatches to a domain that is not CRUD: a Glicko-2 rating pipeline, an OAuth refresh-rotation flow with reuse detection, a hash-chained balance ledger, a version-vector CRDT Mind Palace merge. Each carries Vitest cases that assert computed answers, not shapes.

Three honest labels travel with the rest of the page:

• There are two backend code paths, and the live one is real. The pure evaluateServiceSpecificRule in runtime.ts still returns the old jwt.v5.<account>.oauth fixture string for golden contract tests — but the deployed service.handle() calls handleRealRequest → executeRealServiceRequest, the Postgres/Redis-backed path. The fixture path validates contract shape; the real path is what runs in the container.
• Kernel anti-cheat is a planned client tier. The client builds EAC signal requests and platformPolicy names the signed Windows driver (EasyAntiCheat_EOS.sys), but that kernel driver is named, not in-tree — and the server-side strike journal that does ship is real.
• The design's service table is a subset, and "Voice" is design-named. The feature brief lists ten representative rows (including a Login and a Voice service); the shipped backbone is sixteen packages where Login is the auth service and there is no standalone voice package in apps/v5 — in-game voice is routed through party/presence, not a separate in-tree service.

The service catalog — sixteen services on one backbone#

Gameplay code never hand-rolls a URL. BuildEndpointCatalog enumerates all sixteen services, each as an FV5OnlineEndpointDefinition carrying its verb, path, bRequiresJwt, and bSupportsOfflineQueue flags; a UV5_Online_* builder fills an FV5OnlineServiceRequest, and the catalog is the single source of truth for ports and routing. The service catalog (service-catalog.json) declares the same sixteen on the server, with ports assigned as 4210 + service index (auth = 4210 … faction-rep = 4225, identical in FV5OnlineHttp::ServicePort and the catalog).

The HTTP seam and fail-loud transport#

FV5OnlineHttp::BuildHttpRequest constructs the real request: it sets the verb, Content-Type/Accept: application/json, an Authorization: Bearer header when a JWT is present, the routing headers X-V5-Account and X-V5-Region, and the JSON body for non-GET methods. Base-URL resolution is fail-soft and never fabricated — V5_ONLINE_BASE_URL is treated as an API gateway, else V5_ONLINE_HOST plus the service's catalog port.

The latent action node UV5_Online_HttpRequest::Activate is where the fail-loud discipline shows. A request whose endpoint bRequiresJwt but whose token is empty short-circuits to a 401 before touching the network (auth.jwt.required). A genuine transport failure resolves to TransportFailureResponse (service.unreachable, HttpStatus = 0) and sets bQueuedOffline from the endpoint's offline-queue flag. A 503 likewise flips bQueuedOffline and parses the backend's structured errorCode/errorMessage. No path fabricates a success it did not receive — the response booleans reflect the wire, not a hope.

Identity: JWT, PKCE, and the single account#

Login is the gateway every other service trusts, so it is real on both sides of the wire. On the client, UV5_Online_AuthService::IssueOAuthJwt mints a genuine HS256 token: it assembles RFC 7519 registered claims (iss = v5.bureau.auth, sub, aud = v5.client, provider, iat, nbf, exp with a 3,600-second TTL, jti), derives the jti from SHA-256(account|provider|iat) so each issuance is unique yet deterministic, signs with FV5JwtCrypto::IssueHs256, and derives an opaque refresh token as v5rt_ + base64url(HMAC(SHA-256(key.refresh), SHA-256(accessJwt))) — revocable server-side and not user-forgeable. VerifyAccessJwt does the real inverse: a constant-time signature comparison plus exp/nbf enforcement, and VerifyPkceChallenge checks BASE64URL(SHA256(verifier)) == challenge.

Server-side, AuthDomain (domain/auth.ts) is Postgres-backed and stricter still. resolveCanonicalAccount enforces oldest-verified-credential-wins: the first verified (platform, externalSubject) pair binds a canonical v5acct.… id, and later exchanges for the same pair return that id — one human cannot fork into multiple V5 accounts per platform, which is exactly what cross-progression needs to stay coherent across all nine platforms. refreshAccessToken implements RFC 6749 §6 rotation with RFC 6819 reuse detection: presenting an already-rotated token sets every row in the family to revoked = TRUE and returns reuse_detected. Refresh tokens are stored only as SHA-256 hashes, never cleartext. The canonical account this issues is the spine the cross-play policy builds on, and the client mirror BuildPlatformCredentialLinkRequest binds an external subject to it.

Matchmaking and the skill model#

The client builders are deliberately thin — BuildQueueRequest and BuildCrossplayQueueRequest carry mode, region, an integer SkillRating, ping, and (for cross-play) Platform, ModeId, and bCrossplayOptOut to a per-cell queue path. The authority is server-side and genuine. MatchmakingDomain (domain/matchmaking.ts) runs a real Glicko-2 update (Glickman 2013): glicko2Update computes the g(φ) and expected-score terms, then solves the volatility step with an Illinois (regula falsi) root finder — pinned in tests to the canonical worked example (a 1500/200/0.06 player against three opponents converging to ~1464.06 / ~151.52 / ~0.05999). The queue is a Redis ZSET banded by v5:mm:cell:region:mode:skillBand:pingBand (200-wide skill bands, three ping bands), ordered by enqueue time; formMatch uses an atomic ZPOPMIN so two formation passes never claim the same player, and re-enqueues what it took if a rare race leaves it short. The per-cell ranked rules, season ladder, and crossplay opt-out consequences are surfaced in Modes & Multiplayer; this service is the fairness engine they sit on.

Anti-cheat: defense in depth, labelled by tier#

Anti-cheat is defended in depth. The client UV5_Online_AntiCheatClient builds eight distinct, dedicated-endpoint requests: an EAC session signal, aim plausibility (impossible-angle turn rate), sub-tick aimbot snap, line-of-sight wall-hack history, position-delta speed-hack, auto-fire interval pattern, a weighted classifier carrying modelId: V5_AntiCheat_ML, and an appeal routed to the companion-app support path. Each detector's scoring math is real and lives in runtime.ts — the speed-delta detector, for instance, divides observed distance by elapsed time and flags when the ratio exceeds the weapon/character max speed by more than 15%.

Server-side, AntiCheatStore (domain/anti-cheat.ts) makes this stateful: recordSignal journals each flagged detector to Postgres and recomputes the account's running strike count over verdicts in ('strike', 'ban-review'), driving the three-strike ladder (warning → ranked-suspension → ban-review); fileAppeal queues a row on the companion-app path. The governing posture matches the platform's: model output prioritises a review queue — it never silently disciplines. The platformPolicy helper encodes the integration plan honestly: Windows is a signed kernel driver (EasyAntiCheat_EOS.sys, blockLaunchIfUnavailable), Mac/Linux/console/iOS are userland modules — but, per the label above, that kernel driver is named, not shipped in this tree. The strike journal is what ships real.

The ledgers and graphs the social layer rides#

Three domains turn "online" into durable, auditable state:

• BalanceLedgerDomain is a genuine append-only, hash-chained public ledger. Every entry's hash binds the previous entry's hash, its sequence number, and the canonical JSON of its payload; appends serialise through a single head row taken FOR UPDATE for a gap-free monotonic sequence under concurrent writers; and verifyChain re-walks the chain, recomputes every hash, and returns the first brokenAtSeq if a row was edited out from under it. That is what makes a designer's economy tune evidence-linked rather than trust-me.
• MindPalaceCloudSyncDomain is a version-vector CRDT for the deduction graph. mergeNode resolves by causal dominance, falling back to a deterministic (updatedUnix, lastWriter) tiebreak only on genuinely concurrent edits, and accusations are an append-only union (ON CONFLICT DO NOTHING). The merge is commutative and idempotent, so two devices syncing in either order — or the same diff twice — converge to one graph. This is the service that lets a clue annotated on the companion app reach the console save without clobbering it.
• FriendsDomain and PartiesDomain back the cross-platform social graph: friends pairs and presence heartbeats in Postgres + Redis, party create/join/ready-check with cross-cell Bureau-HQ sessions (a 64-player dedicated-server hub where account-scoped profile, cosmetics, Bureau XP, Mind Palace, and codex are shared while per-cell campaign branches stay isolated).

The backend infrastructure#

Every service is the same shape — a contract, a container, a cluster entry — and the shape is what makes sixteen of them operable as one fleet.

One shape per service: contract, container, cluster#

Each service's contract.json is its machine-readable spec — slug, package name, port, primary and additional endpoints (with requiresJwt / supportsOfflineQueue), declared storage (Postgres + Redis, optional analytics), capabilities, contractCases, the environment variable names it reads, and its docker/kubernetes block. The Dockerfile is a real multi-stage Node 24 build: it installs the workspace with pnpm, builds @v5/service-shared then the service, prunes to production deps, and runs node dist/main.js on the service's catalog port. The k8s/ directory carries a Deployment (two replicas, /healthz readiness and liveness probes, CPU/memory requests and limits), a Service, and a per-service MultiRegionFailover custom resource. Databases are domain-isolated — each service owns its store rather than sharing one schema, the database-per-service posture the architecture specifies.

Configuration, health, and graceful degradation#

The most important honesty seam is how a service behaves when its stores are not there. resolveBackingClients hands a service live pg.Pool / ioredis clients only when its catalog-declared V5_POSTGRES_URL and V5_REDIS_URL resolve to non-empty values; otherwise it returns configured: false with null clients, and executeRealServiceRequest serves the 503 outage contract — carrying retryAfterSeconds, the failover mode, and fallback regions, with queuedOffline set from the endpoint flag. buildHealth reports status: not_configured in that case rather than the hardcoded serviceHealthy: true that once made the outage branch dead code. Absence is reported loudly, never faked — and that fail-loud health is what the readiness probe and DNS failover key off.

Region and failover policy is checked-in data. The catalog declares primary regions iad / fra / sin and secondary pdx / dublin / syd, with compliance-dsar, telemetry, and crash-reporting flagged dataResidencyPinned. The shared services/k8s/multi-region-failover.yaml OnlineServiceFailoverPlan sets a 45-second health window, a 0.98 success-rate threshold, a 180 ms regional-latency ceiling, automatic DNS failover, and a per-service mode (global-active-passive for auth/friends, regional-active-active for parties, queue-backed for the offline-tolerant services). The result is the degradation promise the player feels: a service outage drops single-player to offline-deterministic campaign play with local saves that reconcile on reconnect; an unavailable matchmaker shows a clean retry state, never a hang; and telemetry buffers client-side and flushes on recovery — because every layer above is built to read a 503 as "retry or queue," not as a reason to lie.

Where this connects#

• Sideways to the match: Modes & Multiplayer — the connection models (dedicated / listen / LAN / server browser), the PvP and horde modes that queue through this matchmaker, the ranked-season ladder, and the esports surfaces that run on this fleet.
• Sideways to live service: Live Service & Community — the seasonal calendar feeds, persistent world events, workshop marketplace, and faction-rep decay that ride these same services and ledgers.
• Down to the engine: Persistence, Online Services & Cross-Play — the encrypted save substrate, the V5Persistence ledgers and replay determinism, the canonical-account cross-play policy, and the Postgres / Redis / object-storage substrate these services deploy onto.
• The feature hub: ../V5_features.md