# Gateway, Pixel Streaming & the Web Fallback

Egbe's premium surface is a single UE5.5 project — Lumen, Nanite, MetaHuman
agents, an interest-managed crowd of LLM-driven minds. That is wonderful on a
gaming PC and impossible on a Chromebook in a library. The **edge** is the set
of systems that make the world reachable anyway, across the whole device
spectrum, without ever forking the simulation. It has three planes, and they are
deliberately layered so that fidelity can fall while identity, presence, and the
authoritative world never do. The **Egbe Realtime Gateway** is the door: it
terminates whatever transport a client can actually open (QUIC if the network
allows, WebRTC or a plain WebSocket if it does not), pins the session to the
world-server shard holding its instance, and hosts the voice SFU that carries
conversation and squad comms. **Pixel Streaming** is the premium browser path: a
pooled UE worker renders an Orun ground server-side and ships H.264 frames over
WebRTC to a thin browser shell, so a tab gets the same world a native client
does. The **Tier-2 web fallback** is the floor: a three.js/WebGPU client that
renders a reduced Orun _locally_ on constrained devices, speaking the exact same
wire protocol to the exact same world server, capping how many agents it draws
but never which world it joins.

The load-bearing idea is that all three planes converge on one authority. A
native UE client, a Pixel Streaming worker, and a Tier-2 fallback are, from the
world server's point of view, three kinds of the same thing: protocol peers that
receive interest-managed agent state and send actions. The wire protocol encodes
this directly — its `AuthorityKind` enum names `UE_CLIENT`, `TIER2_FALLBACK`,
and `REALTIME_GATEWAY` as distinct, first-class participants. This page is the
deep dive on those three planes and the two browser libraries that implement
them; it is a companion to the orientation map in
[../V6_ARCHITECTURE.md](../V6_ARCHITECTURE.md).

## What ships, honestly

**Real and on-disk.** The wire protocol is genuine and the densest of the three
target libraries: `libs/v6/egbe-protocol` carries a 373-line proto3 schema
(`proto/oshun/v6/egbe/v1/egbe.proto`), generated TypeScript codecs
(`src/generated/...`), a Rust crate (`rust/src/lib.rs`), and UE C++ bindings
(`ue/generated/.../egbe.pb.cc` + `.pb.h`) so the same messages cross the
TypeScript fallback, the Rust services, and the UE client. Its `src/index.ts`
implements interest-managed delta replication for real
(`projectAgentStateSnapshot`, `encodeAgentStateDeltaFromSnapshots`,
`applyAgentStateDelta`), a version/capability handshake
(`negotiateProtocolHandshake`), and a bandwidth budget validator
(`validateReplicationBandwidth` against `GROUND_REPLICATION_MAX_BPS = 256_000`).
The browser Pixel Streaming client (`libs/v6/egbe-web-pxstream`) is a real state
machine wrapping Epic's published frontend SDK
(`@epicgames-ps/lib-pixelstreamingfrontend-ue5.5`, a real dependency in
`package.json`), with a first-frame SLA timer and telemetry. Off-engine, the two
Rust edge services exist and are substantial: the realtime gateway is **3,945
lines** (`apps/v6/egbe-realtime-gateway`, 23 in-crate tests) and the Pixel
Streaming relay **1,290 lines** (`apps/v6/egbe-pxstream-relay`, 9 in-crate
tests).

**Policy-as-data, not a renderer — and labeled as such.** The most important
honest caveat is `libs/v6/egbe-engine-web-fallback`. Despite its name and its
`three` dependency, this library contains **no rendering code**: its `index.ts`
never imports three.js. It is the _budget and density-policy_ layer for the
fallback — it computes deterministic renderer budgets, the Tier-2 agent-density
cap, and a reduced-scene manifest. The actual WebGPU/WebGL2 renderer lives in
the `apps/v6/egbe-web-fallback` application; this library is the honest,
testable contract it renders _against_. That is a real and useful seam, but the
page calls it policy, not pixels.

**The relay's match is a fail-honest env-seam, and simpler than V3's.** V3's
matchmaker was a ~4,800-line affair with POP geo-scoring, HMAC session JWTs, and
SDP construction. V6's relay is **deliberately narrower**:
`match_pxstream_session` resolves a worker, region, signalling URL, and codec by
reading `V6_PXSTREAM_*` environment variables with local-dev defaults (`region`
→ `"local-orun"`, `codec` → `"H264"`) — there is **no POP geo-scoring in
source** (a grep for `select_pop`/`haversine`/`nearest_pop` returns zero hits).
The relay's _real_ logic is admission control: abuse classification,
per-user/per-network concurrency caps, free-tier backpressure, and idle
lifecycle. So "matchmaking" here means "lease a worker and enforce the gates,"
with the geographic fleet picker left as an injectable seam rather than a fake.

**Shared across versions.** The gateway crate is shared infrastructure that has
accreted features from later products: alongside the V6 transport and SFU
capabilities it carries `v7.pheme.*` proximity-voice extensions
(`PHEME_WHISPER_RANGE_M`, occlusion attenuation, parental controls). That is not
fabrication, but it means the crate is broader than "the V6 gateway"; this page
describes the V6-relevant surface and flags the rest as cross-version reuse.

## The shape of the edge

```mermaid
flowchart TB
    subgraph CLIENTS["Client surfaces"]
        direction LR
        native["UE5 native<br/><sub>desktop · console · mobile · VR</sub>"]
        pxBrowser["Browser → Pixel Streaming<br/><sub>@oshun/egbe-web-pxstream</sub>"]
        fbBrowser["Browser fallback<br/><sub>three.js / WebGPU</sub>"]
    end

    bff["<b>V1 Oshun BFF</b> — tier router<br/><sub>native · pxstream · fallback · static</sub>"]
    relay["<b>egbe-pxstream-relay</b> (Rust)<br/><sub>admission · abuse · idle · worker lease</sub>"]
    worker["UE Pixel Streaming worker<br/><sub>headless Orun render</sub>"]
    gateway["<b>Egbe Realtime Gateway</b> (Rust)<br/><sub>WebTransport/QUIC · WebRTC · WS · voice SFU</sub>"]

    subgraph WORLD["Egbe World Server fleet"]
        direction LR
        shard["Shard directory<br/><sub>instance → shard pinning</sub>"]
    end

    CLIENTS -->|"HTTPS: which surface?"| bff
    pxBrowser -->|"POST /api/v6/pxstream/match"| relay
    relay --> worker
    worker -->|"H.264 / WebRTC frames"| pxBrowser
    worker --> gateway
    native --> gateway
    fbBrowser --> gateway
    gateway --> WORLD

    proto["@oshun/egbe-protocol<br/><sub>one wire format · UE C++ + Rust + TS</sub>"]
    native -.->|speaks| proto
    fbBrowser -.->|speaks| proto
    gateway -.->|speaks| proto
    worker -.->|speaks| proto
```

The tier router that picks _which_ of these a visitor gets is a route inside the
V1 BFF, not part of the edge services themselves; see
[../../platform/bff-gateway.html](../../platform/bff-gateway.html) for that
front door. Everything below is what happens after the door.

## The Egbe Realtime Gateway

The gateway (`apps/v6/egbe-realtime-gateway/src/lib.rs`, Rust) is the single
junction every live client reaches. Its `SERVICE_DESCRIPTOR` advertises the V6
capabilities precisely: `webtransport-edge`, `webrtc-fallback`,
`websocket-fallback`, `world-shard-routing`, `presence-transform-20hz`,
`voice-sfu`, and `squad-voice-routing`.

### Transport negotiation and the fallback order

A browser on a corporate network may have UDP/QUIC blocked; a mobile client on a
flaky link may only manage a WebSocket. The gateway models this as a typed
ladder. `RealtimeTransportMode` is an enum of `WebTransportQuic`,
`WebRtcDataChannel`, and `WebSocket`, each with a stable `protocol_label()` and
an `expected_path()` (`/v6/realtime/webtransport`, `/v6/realtime/webrtc`,
`/v6/realtime/websocket`). The canonical preference is
`RealtimeTransportMode::fallback_order()` →
`[WebTransportQuic, WebRtcDataChannel, WebSocket]`, QUIC-primary with two
documented fallbacks.

Clients declare what they can open through `RealtimeClientNetworkProfile`, which
ships three named constructors that read like the real-world cases they
represent: `quic_primary()` (everything available), `quic_blocked_with_webrtc()`
(no QUIC, WebRTC + WS), and `websocket_only()` (the floor).
`route_client_with_best_transport` walks the fallback order and selects the
first mode the profile `supports(...)`; if none is available the gateway fails
loudly with `RealtimeGatewayError::NoAvailableTransport`, carrying the
`client_id` — it does not pretend a connection it could not make. The negotiated
`ClientShardRoutingDecision` also carries the full `fallback_chain` so the
client knows what it can renegotiate to if its link degrades mid-session.

### World-shard routing

Transport is only half the door; the other half is _which shard_. The gateway
holds a `WorldShardDirectory` keyed by instance id (`route_for_instance`,
`upsert_route`). `route_client` first terminates the transport, then looks up
the `WorldShardRoute` for the requested instance, and fails with
`InstanceShardNotFound` if no shard claims it. A successful decision binds
`client_id`, `instance_id`, the terminated transport, and the target shard into
one `ClientShardRoutingDecision`. This is the mechanism behind "session shard
pinning": a reconnecting client returns to the same shard within the retention
window (`GATEWAY_RECONNECT_RETENTION_SECONDS = 60`,
`SESSION_PIN_TTL_SECONDS = 60`), so a dropped frame of network does not scatter
a player across the continuum. The shard model itself — Solo homesteads, Co-op,
and the Commons fleet — is described in
[./world-server-and-shard-continuum.md](./world-server-and-shard-continuum.md);
the gateway's job is only to find the right one and pin to it.

Presence flows over this junction at the protocol's ground cadence:
`presence-transform-20hz`, matching `GROUND_REPLICATION_HZ = 20` from the shared
protocol crate — transforms at ≤ 20 Hz, activity-state changes event-driven.

### The voice SFU

Vac's live conversation and squad comms run as voice, and the gateway is the
selective forwarding unit. A `VoiceSfuPeer` models a participant richly:
`session_id`, `entity_id`, `room_id`, `region`, `regional_rtt_ms`, an optional
`position_mm` for positional audio, plus `squad_ids`,
`muted_speaker_entity_ids`, an accessibility profile, and parental controls.
Voice rides a real codec budget — `OPUS_24KBPS_MONO_*` constants fix 24 kbps
mono in 20 ms frames — against a regional latency target
(`VAC_REGIONAL_VOICE_LATENCY_BUDGET_MS = 80`) and a loss ceiling
(`VAC_VOICE_LOSS_RATE_BUDGET = 0.005`). This is the same SFU whether a squad is
together in Orun or split across an Aye incarnation, which is why the protocol's
squad-comms routing asserts realm parity (below). The proximity-voice ranges and
occlusion model carried here are the `v7.pheme.*` extensions noted above —
present in the shared crate, beyond V6's own scope.

## Pixel Streaming: server-rendered Orun in a browser

Pixel Streaming gives a browser the _premium_ world: a UE worker renders an Orun
ground headless and streams it. Two pieces implement it — the relay that leases
a worker, and the browser library that plays the stream.

### The relay: admission, abuse, idle — and the match seam

`apps/v6/egbe-pxstream-relay/src/lib.rs` exposes `match_pxstream_session`, which
returns a `PixelStreamingMatchResponse`: a `session_id`, `worker_id`, `region`,
`signalling_url`, `streamer_id`, the `first_frame_budget_ms`
(`DEFAULT_FIRST_FRAME_BUDGET_MS = 8_000`), an `expires_at_epoch_seconds`
(`+ DEFAULT_SESSION_TTL_SECONDS = 120`), a `fallback_url`, and `codec: "H264"`.
As flagged in _What ships, honestly_, the worker/region/signalling fields are
resolved from `V6_PXSTREAM_*` env vars with local defaults — a fail-honest seam
where a production fleet picker would plug in, not a simulated one.

The genuine logic is `evaluate_v6_pxstream_admission`, a real gate that runs in
order: first **abuse** — if an `abuse_signal_window` is present it calls
`classify_pxstream_abuse` and, on a non-`Allow` action, returns a review or ban
decision _before any worker is leased_ (the comment is explicit that V6 reuses
the V3 abuse classifier, held to
`MIN_ABUSE_CLASSIFIER_PRECISION_BASIS_POINTS = 9_000`). Then **concurrency**:
`PXSTREAM_MAX_CONCURRENT_SESSIONS_PER_USER = 2` and `..._PER_NETWORK = 8`
(institutional networks get `64`). Then **free-tier backpressure**: a free
entitlement under regional load (`FREE_TIER_BACKPRESSURE_BASIS_POINTS = 8_500`)
is steered to the fallback URL rather than a GPU.
`evaluate_pxstream_idle_lifecycle` reclaims an idle worker — prompt at
`PXSTREAM_IDLE_PROMPT_AFTER_SECONDS = 90`, disconnect and `release-to-pool` at
`150`. Every rejection carries a reason code and a `fallback_url`, so the
relay's answer to "no GPU for you" is always "here is the fallback," never a
dead end.

### The browser client: the first-frame SLA

`libs/v6/egbe-web-pxstream/src/index.ts` is the player.
`EgbePixelStreamingClient` is an observable state machine over the lifecycle
`idle → matching → matched → signalling → streaming`, with `degraded` as the
off-ramp. `start()` POSTs a session request to the matchmaker
(`V6_PIXEL_STREAMING_MATCH_PATH = '/api/v6/pxstream/match'`), validates the
response with `parseEgbePixelStreamingSession` (which _rejects_ a non-`matched`
status or a non-positive budget — the client refuses a malformed lease), then
constructs the real Epic adapter.

`EpicEgbePixelStreamingAdapter` dynamically imports
`@epicgames-ps/lib-pixelstreamingfrontend-ue5.5` and configures it for an Orun
session: the signalling URL and streamer id from the lease, `PreferredCodec`
from the match (`H264` or `AV1`), `WebRTCMaxBitrate` of 12 Mbps, and keyboard /
mouse / touch / gamepad input on with the mic off. It maps Epic's events onto
the client's vocabulary — `webRtcConnecting → signalling`,
`streamLoading → loading`, `videoInitialized → first-frame`,
`webRtcDisconnected → disconnected` with the SDK's `allowClickToReconnect` flag
preserved.

The first-frame SLA is the contract that makes this honest about quality.
`DEFAULT_FIRST_FRAME_BUDGET_MS = 8_000`: when signalling begins the client arms
a timer for the budget; if `first-frame` arrives first it computes
`evaluateFirstFrameTelemetry` and moves to `streaming` only when
`elapsedMs <= budgetMs`, otherwise to `degraded` with the over-budget elapsed
time recorded. If the timer fires first, the session is declared `degraded`
without waiting. The adapter is an injected `adapterFactory`, so the unit spec
drives the whole machine with a fake adapter and asserts the real transitions —
including that an 8,999 ms first frame classifies `within-budget` and a 9,001 ms
one `exceeded-budget`. The state machine, not a mock, is what is under test.

## The Tier-2 web fallback

When Pixel Streaming is unavailable — no GPU near enough, free-tier
backpressure, a relay rejection — the visitor still gets Orun, rendered locally
at reduced fidelity. `libs/v6/egbe-engine-web-fallback` is the policy layer for
that client.

### Budgets and the agent-density cap

`selectFallbackRendererBudget` returns a deterministic budget per backend.
**WebGPU** is the preferred path: `reduced` quality, 45 fps target, a 22 ms
frame cost ceiling, **32** visible agents, agent LOD ≤ 2, procedural foliage on.
**WebGL2** is the floor: `low` quality, 30 fps, a 30 ms ceiling, **18** visible
agents, LOD ≤ 1, no procedural foliage. `selectTier2AgentDensityBudget` then
applies the density rule that makes a constrained device believable rather than
empty: on a `constrained` profile the visible cap drops hard — WebGL2 to **6**,
WebGPU to **10** — but `protectsFocusedAgent` stays `true`, the selection
strategy is `'focus-then-salience-distance'`, and culled agents are not deleted:
`maxNarrativeBackfillAgents = max(visible * 3, 12)` keeps them present as Clio
narrative summary. A small device sees _fewer faces, not a smaller world_. Each
budget emits a `densitySignature` string (e.g.
`tier2-density:webgl2:constrained:visible6:backfill18`) so a parity test can pin
the exact policy. `createOrunFallbackSceneManifest` rounds it out with the
reduced reference scene (`orun-tier2-grove`, district `GroveOfBeginnings`) and
an explicit `omittedTier1Features` list — `lumen-global-illumination`,
`nanite-hero-geometry`, `chaos-cloth-crowds`, `niagara-high-density-particles` —
so the fidelity gap from Tier-1 is documented data, not a surprise.

### One protocol, two renderers

The fallback's whole premise is that it is the _same world_, and that holds
because rendering is the _only_ thing that changes — the wire is identical.
`libs/v6/egbe-protocol` is the shared contract, and its `proto` makes the
fallback a first-class peer: `AuthorityKind` enumerates `UE_CLIENT` _and_
`TIER2_FALLBACK` _and_ `REALTIME_GATEWAY` as distinct authorities, and the
`ClientEnvelope`/`ServerEnvelope` oneofs carry the same `PresencePacket`,
`AgentStateSnapshot`/`AgentStateDelta`, `PerceptionBatch`, `ActionBatch`,
`SquadCommsMessage`, and `WorldEvent` to every client kind. The interest
management that keeps a constrained client cheap is real protocol code:
`projectAgentStateSnapshot` clips a snapshot to an `InterestWindow` (origin,
radius, `GROUND_VISIBLE_AGENT_CAP = 32`, `GROUND_VISIBLE_PLAYER_CAP = 8`),
`encodeAgentStateDeltaFromSnapshots` emits only upserts and removals against a
base sequence, and `validateReplicationBandwidth` proves the result fits
`GROUND_REPLICATION_MAX_BPS = 256_000`. `negotiateProtocolHandshake` lets a
weaker client advertise a smaller capability set and receive a `DEGRADED`
acceptance with the dropped features named — the same mechanism that lets the
fallback ask for fewer perception items or a lower agent LOD ceiling without
falling off the protocol. The UE client renders these messages with Nanite and
MetaHumans; the fallback renders them with three.js; the gateway forwards them;
none of them disagree about what is true in the world.

## How the three planes compose

Read top to bottom, a visitor's path is: the V1 BFF tier router picks a surface
([../../platform/bff-gateway.html](../../platform/bff-gateway.html)); a premium
browser is leased a worker by the relay and plays it through
`@oshun/egbe-web-pxstream` under an 8-second first-frame SLA, degrading to the
fallback URL on any rejection; a constrained browser loads the Tier-2 client
under a focus-protecting density cap; and _all_ of them — plus the native UE
client and the Pixel Streaming worker — open a transport through the Realtime
Gateway, pin to a world shard, and exchange `egbe-protocol` messages with the
authoritative world. Fidelity is the only thing that moves down the ladder.
Identity, presence, the shard, the SFU, and the wire are constant. The client
side of that wire — the UE5 module split, agent embodiment, and the density LOD
pipeline that the protocol's interest windows feed — is detailed in
[./ue5-client-modules-and-embodiment.md](./ue5-client-modules-and-embodiment.md),
and the authoritative world the gateway routes into is in
[./world-server-and-shard-continuum.md](./world-server-and-shard-continuum.md).