Runbook - V6 Production Setup

Use this runbook to stand up a new V6 production-like environment for Egbe: Rust services, data stores, Pixel Streaming relay, web entries, Operator Console coverage, release gates, and manual-QA readiness. It is written so a new environment can be built from an empty host plus the repository, secrets, DNS, and deployment credentials.

Scope#

This procedure covers:

• V6 service stack: egbe-world-server, egbe-moirai-cluster, egbe-realtime-gateway, egbe-pxstream-relay, egbe-ori-service, egbe-clio-service, and egbe-foundry-service.
• Shared data plane: PostgreSQL with pgvector, Redis Streams, MinIO/S3 buckets, V1 audit platform integration, and V1 identity/RBAC scopes.
• Client surfaces: UE5 Tier-1 client, Pixel Streaming web entry, Tier-2 fallback web, Steward App /v6, Egbe Studio, and Egbe Operator Console.
• Release gates: V6 CI workflow verification, security gates, performance gates, browser automation, operator runbooks, and manual-QA signoff.

This is not an incident response runbook. For live capacity pressure use v6-capacity-management.md; for privacy or moderation incidents use the dedicated runbooks in docs/runbooks/.

Required access#

• GitHub access to GreyChimp/oshun and permission to deploy from main.
• Container registry push/pull for V6 images or permission to run the production-like Compose stack.
• Secret-manager access for production/staging values.
• DNS and TLS access for the environment's web, admin, Pixel Streaming, and API hostnames.
• RBAC scopes: admin:*, admin:studio, admin:workspace:egbe, V6 steward/operator scopes, and V1 audit-platform write access.
• Unreal Engine 5.5 access when validating the Tier-1 client. On this host, UnrealEditor-Cmd is expected at /root/workspace/UnrealEngine-5.5/Engine/Binaries/Linux/UnrealEditor-Cmd.

Host prerequisites#

Before starting, record:

free -h
df -h / /tmp
docker version
docker compose version
node --version
pnpm --version
cargo --version

Minimum production-like smoke host:

• 8 vCPU, 32 GiB RAM, 100 GiB free disk for Rust build plus containers.
• Swap enabled if memory is below 32 GiB.
• Docker Compose v2.
• Node and pnpm matching the repo lockfile.
• Rust toolchain compatible with apps/v6/Cargo.toml.

Do not run broad local release gates on a host below 32 GiB RAM without an explicit resource check and approval. Use the targeted commands below while bootstrapping; run broad gates in CI or on a dedicated validation host.

Environment values#

Create an environment file outside git, for example /etc/oshun/v6-production.env. Values below are required by the Compose stack, the services, or the handoff record:

POSTGRES_USER=oshun
POSTGRES_PASSWORD=<secret>
POSTGRES_DB=egbe
REDIS_URL=redis://redis:6379
MINIO_ROOT_USER=<secret>
MINIO_ROOT_PASSWORD=<secret>
MINIO_ENDPOINT=http://minio:9000
V6_WORLD_SERVER_URL=http://egbe-world-server:46101
V6_MOIRAI_CLUSTER_URL=http://egbe-moirai-cluster:46102
V6_REALTIME_GATEWAY_URL=http://egbe-realtime-gateway:46103
V6_PIXELSTREAMING_SIGNALLER_URL=wss://<pixel-signaller-host>
V6_PUBLIC_BASE_URL=https://<v6-web-host>
V6_ADMIN_BASE_URL=https://<admin-host>/egbe
JWT_SECRET=<secret>
ENCRYPTION_KEY=<32-byte-secret>
OPENAI_API_KEY=<secret-if-used>
ANTHROPIC_API_KEY=<secret-if-used>
SENTRY_DSN=<secret-if-used>
POSTHOG_API_KEY=<secret-if-used>

Secret rules:

• Never reuse the local Compose defaults oshun or oshun-local-dev.
• Store production secrets in the secret manager, not in the repo or shell history.
• Record secret versions in the handoff, not secret values.
• Rotate any value pasted into an incident channel.

Build artifacts#

From a clean checkout at the target commit:

pnpm install --frozen-lockfile
pnpm nx run @oshun/v6-egbe-web:build
pnpm nx run @oshun/v6-egbe-web-fallback:build
cargo build --manifest-path apps/v6/Cargo.toml --workspace --locked

For the UE5 Tier-1 client, use the V6 BuildGraph workflow for production artifacts. Local smoke validation can use:

/root/workspace/UnrealEngine-5.5/Engine/Binaries/Linux/UnrealEditor-Cmd \
  V6/ue/V6.uproject \
  -NullRHI \
  -ExecCmds="Automation RunTests V6.Foundation.ProjectDescriptor;Quit" \
  -unattended \
  -nop4

If Unreal refuses to run as root, run the command as ueagent.

Compose bootstrap#

The production-like stack is defined in docker/docker-compose.v6.yml. Validate the Compose file before starting services:

docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env config

Start the data plane first:

docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env up -d postgres redis minio
docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env ps

Initialize object buckets if they do not already exist:

set -a
source /etc/oshun/v6-production.env
set +a
docker run --rm --network oshun-v6 minio/mc \
  alias set v6 http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
docker run --rm --network oshun-v6 minio/mc mb --ignore-existing v6/egbe-ori
docker run --rm --network oshun-v6 minio/mc mb --ignore-existing v6/egbe-clio
docker run --rm --network oshun-v6 minio/mc mb --ignore-existing v6/egbe-foundry

Start the V6 services:

docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env up -d \
  egbe-world-server \
  egbe-moirai-cluster \
  egbe-realtime-gateway \
  egbe-pxstream-relay \
  egbe-ori-service \
  egbe-clio-service \
  egbe-foundry-service

Wait until every service is healthy:

docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env ps
docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env logs --tail=100

If a service is not healthy, inspect the matching V6_SERVICE_PORT in docker/docker-compose.v6.yml and confirm it can reach its declared dependency: PostgreSQL, Redis, MinIO, world server, Moirai cluster, or realtime gateway.

Web and admin surfaces#

Deploy or serve these artifacts after the service stack is healthy:

• Pixel Streaming entry: apps/v6/egbe-web/dist/.
• Tier-2 fallback entry: apps/v6/egbe-web-fallback/dist/.
• Oshun web shell routes: /v6 and /egbe-studio.
• Oshun admin route: /egbe.
• Mobile Steward App deep link: oshun://v6.

Required routing:

TLS must terminate before browser access. Pixel Streaming sessions must pass the V6 relay admission checks before a worker lease is returned.

Verification gates#

Run targeted gates in this order for a new environment:

pnpm run verify:v6 ci-workflows
pnpm run verify:v6 operator-runbooks
pnpm run verify:v6 production-setup-manual-qa
pnpm run verify:v6 browser-automation
pnpm run verify:v6 release-gate
pnpm run verify:v6 latency-budgets
pnpm run verify:v6 commons-scale-load
pnpm run verify:v6 cost-budget
pnpm run verify:v6 cross-platform-certification
pnpm run verify:v6 cross-progression-continuity

Run the full release chain only in CI or on a dedicated validation host:

pnpm run verify:v6 ci-release-gates

For browser signoff, use the exact commands in V6/testing/browser-automation-gates.v6qa.json, including:

node node_modules/playwright/cli.js test --config apps/v6/egbe-web/playwright.config.ts
PORT=3063 node node_modules/playwright/cli.js test --config apps/v6/egbe-web-fallback/playwright.config.ts
E2E_WEB_ONLY=1 node node_modules/@playwright/test/cli.js test --config apps/oshun/web/playwright.config.ts apps/oshun/web/e2e/v6-steward-app-browser.spec.ts apps/oshun/web/e2e/egbe-agent-dossier-editor.spec.ts apps/oshun/web/e2e/egbe-ground-authoring.spec.ts apps/oshun/web/e2e/egbe-scenario-authoring.spec.ts apps/oshun/web/e2e/egbe-provenance-inspector.spec.ts

Manual-QA handoff#

Before handing the environment to QA, attach:

• Git commit SHA and deployment timestamp.
• Environment URLs for web, admin, Pixel Streaming, fallback, and MinIO admin.
• Secret-manager versions for all V6 values.
• Compose config output or deployment manifest digest.
• docker compose ps or orchestrator health output.
• Verification commands and pass/fail results.
• Known disabled features, temporary flags, and expiry owners.

QA signoff then follows docs/testing/v6-manual-qa-recipes.md.

Rollback#

Rollback is allowed only after preserving evidence:

1. Save the deployment manifest, service logs, audit refs, and failed gate output.
2. Disable new Pixel Streaming admissions if user sessions are impacted.
3. Roll service images or Compose deployment back to the previous green digest.
4. Keep PostgreSQL and MinIO data in place unless the privacy or data-loss incident commander explicitly orders restore.
5. Rerun the targeted verification gates and the impacted manual-QA recipe.

Handoff checklist#

• Data plane healthy: PostgreSQL, Redis, MinIO.
• All seven V6 Rust services healthy.
• Web, admin, Pixel Streaming, and fallback routes reachable over TLS.
• V1 audit platform receives V6 operator and Ori-read events.
• Browser automation gate green.
• Security, residency, and cross-progression gates green.
• Manual-QA recipes assigned with owners and evidence folder.
• On-call runbooks linked in the launch channel.
• Rollback digest and previous-green deployment identified.