# Runbook - V6 Production Setup > Use this runbook to stand up a new V6 production-like environment for Egbe: > Rust services, data stores, Pixel Streaming relay, web entries, Operator > Console coverage, release gates, and manual-QA readiness. It is written so a > new environment can be built from an empty host plus the repository, secrets, > DNS, and deployment credentials. ## Scope This procedure covers: - V6 service stack: `egbe-world-server`, `egbe-moirai-cluster`, `egbe-realtime-gateway`, `egbe-pxstream-relay`, `egbe-ori-service`, `egbe-clio-service`, and `egbe-foundry-service`. - Shared data plane: PostgreSQL with pgvector, Redis Streams, MinIO/S3 buckets, V1 audit platform integration, and V1 identity/RBAC scopes. - Client surfaces: UE5 Tier-1 client, Pixel Streaming web entry, Tier-2 fallback web, Steward App `/v6`, Egbe Studio, and Egbe Operator Console. - Release gates: V6 CI workflow verification, security gates, performance gates, browser automation, operator runbooks, and manual-QA signoff. This is not an incident response runbook. For live capacity pressure use `v6-capacity-management.md`; for privacy or moderation incidents use the dedicated runbooks in `docs/runbooks/`. ## Required access - GitHub access to `GreyChimp/oshun` and permission to deploy from `main`. - Container registry push/pull for V6 images or permission to run the production-like Compose stack. - Secret-manager access for production/staging values. - DNS and TLS access for the environment's web, admin, Pixel Streaming, and API hostnames. - RBAC scopes: `admin:*`, `admin:studio`, `admin:workspace:egbe`, V6 steward/operator scopes, and V1 audit-platform write access. - Unreal Engine 5.5 access when validating the Tier-1 client. On this host, `UnrealEditor-Cmd` is expected at `/root/workspace/UnrealEngine-5.5/Engine/Binaries/Linux/UnrealEditor-Cmd`. ## Host prerequisites Before starting, record: ```bash free -h df -h / /tmp docker version docker compose version node --version pnpm --version cargo --version ``` Minimum production-like smoke host: - 8 vCPU, 32 GiB RAM, 100 GiB free disk for Rust build plus containers. - Swap enabled if memory is below 32 GiB. - Docker Compose v2. - Node and pnpm matching the repo lockfile. - Rust toolchain compatible with `apps/v6/Cargo.toml`. Do not run broad local release gates on a host below 32 GiB RAM without an explicit resource check and approval. Use the targeted commands below while bootstrapping; run broad gates in CI or on a dedicated validation host. ## Environment values Create an environment file outside git, for example `/etc/oshun/v6-production.env`. Values below are required by the Compose stack, the services, or the handoff record: ```bash POSTGRES_USER=oshun POSTGRES_PASSWORD= POSTGRES_DB=egbe REDIS_URL=redis://redis:6379 MINIO_ROOT_USER= MINIO_ROOT_PASSWORD= MINIO_ENDPOINT=http://minio:9000 V6_WORLD_SERVER_URL=http://egbe-world-server:46101 V6_MOIRAI_CLUSTER_URL=http://egbe-moirai-cluster:46102 V6_REALTIME_GATEWAY_URL=http://egbe-realtime-gateway:46103 V6_PIXELSTREAMING_SIGNALLER_URL=wss:// V6_PUBLIC_BASE_URL=https:// V6_ADMIN_BASE_URL=https:///egbe JWT_SECRET= ENCRYPTION_KEY=<32-byte-secret> OPENAI_API_KEY= ANTHROPIC_API_KEY= SENTRY_DSN= POSTHOG_API_KEY= ``` Secret rules: - Never reuse the local Compose defaults `oshun` or `oshun-local-dev`. - Store production secrets in the secret manager, not in the repo or shell history. - Record secret versions in the handoff, not secret values. - Rotate any value pasted into an incident channel. ## Build artifacts From a clean checkout at the target commit: ```bash pnpm install --frozen-lockfile pnpm nx run @oshun/v6-egbe-web:build pnpm nx run @oshun/v6-egbe-web-fallback:build cargo build --manifest-path apps/v6/Cargo.toml --workspace --locked ``` For the UE5 Tier-1 client, use the V6 BuildGraph workflow for production artifacts. Local smoke validation can use: ```bash /root/workspace/UnrealEngine-5.5/Engine/Binaries/Linux/UnrealEditor-Cmd \ V6/ue/V6.uproject \ -NullRHI \ -ExecCmds="Automation RunTests V6.Foundation.ProjectDescriptor;Quit" \ -unattended \ -nop4 ``` If Unreal refuses to run as root, run the command as `ueagent`. ## Compose bootstrap The production-like stack is defined in `docker/docker-compose.v6.yml`. Validate the Compose file before starting services: ```bash docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env config ``` Start the data plane first: ```bash docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env up -d postgres redis minio docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env ps ``` Initialize object buckets if they do not already exist: ```bash set -a source /etc/oshun/v6-production.env set +a docker run --rm --network oshun-v6 minio/mc \ alias set v6 http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" docker run --rm --network oshun-v6 minio/mc mb --ignore-existing v6/egbe-ori docker run --rm --network oshun-v6 minio/mc mb --ignore-existing v6/egbe-clio docker run --rm --network oshun-v6 minio/mc mb --ignore-existing v6/egbe-foundry ``` Start the V6 services: ```bash docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env up -d \ egbe-world-server \ egbe-moirai-cluster \ egbe-realtime-gateway \ egbe-pxstream-relay \ egbe-ori-service \ egbe-clio-service \ egbe-foundry-service ``` Wait until every service is healthy: ```bash docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env ps docker compose -f docker/docker-compose.v6.yml --env-file /etc/oshun/v6-production.env logs --tail=100 ``` If a service is not healthy, inspect the matching `V6_SERVICE_PORT` in `docker/docker-compose.v6.yml` and confirm it can reach its declared dependency: PostgreSQL, Redis, MinIO, world server, Moirai cluster, or realtime gateway. ## Web and admin surfaces Deploy or serve these artifacts after the service stack is healthy: - Pixel Streaming entry: `apps/v6/egbe-web/dist/`. - Tier-2 fallback entry: `apps/v6/egbe-web-fallback/dist/`. - Oshun web shell routes: `/v6` and `/egbe-studio`. - Oshun admin route: `/egbe`. - Mobile Steward App deep link: `oshun://v6`. Required routing: | Surface | Route / host | Backing service or artifact | | ---------------- | --------------------- | ------------------------------------------- | | Tier-1 web | `/egbe` Pixel entry | `apps/v6/egbe-web/dist/`, pxstream relay | | Tier-2 fallback | `/egbe/fallback` | `apps/v6/egbe-web-fallback/dist/` | | Steward App web | `/v6` | `apps/oshun/web/src/app/v6/` | | Egbe Studio | `/egbe-studio` | `apps/oshun/web/src/app/egbe-studio/` | | Operator Console | `/egbe` admin route | `apps/oshun/admin/src/app/egbe/` | | MinIO console | restricted admin only | `docker/docker-compose.v6.yml` port `19001` | TLS must terminate before browser access. Pixel Streaming sessions must pass the V6 relay admission checks before a worker lease is returned. ## Verification gates Run targeted gates in this order for a new environment: ```bash pnpm run verify:v6 ci-workflows pnpm run verify:v6 operator-runbooks pnpm run verify:v6 production-setup-manual-qa pnpm run verify:v6 browser-automation pnpm run verify:v6 release-gate pnpm run verify:v6 latency-budgets pnpm run verify:v6 commons-scale-load pnpm run verify:v6 cost-budget pnpm run verify:v6 cross-platform-certification pnpm run verify:v6 cross-progression-continuity ``` Run the full release chain only in CI or on a dedicated validation host: ```bash pnpm run verify:v6 ci-release-gates ``` For browser signoff, use the exact commands in `V6/testing/browser-automation-gates.v6qa.json`, including: ```bash node node_modules/playwright/cli.js test --config apps/v6/egbe-web/playwright.config.ts PORT=3063 node node_modules/playwright/cli.js test --config apps/v6/egbe-web-fallback/playwright.config.ts E2E_WEB_ONLY=1 node node_modules/@playwright/test/cli.js test --config apps/oshun/web/playwright.config.ts apps/oshun/web/e2e/v6-steward-app-browser.spec.ts apps/oshun/web/e2e/egbe-agent-dossier-editor.spec.ts apps/oshun/web/e2e/egbe-ground-authoring.spec.ts apps/oshun/web/e2e/egbe-scenario-authoring.spec.ts apps/oshun/web/e2e/egbe-provenance-inspector.spec.ts ``` ## Manual-QA handoff Before handing the environment to QA, attach: - Git commit SHA and deployment timestamp. - Environment URLs for web, admin, Pixel Streaming, fallback, and MinIO admin. - Secret-manager versions for all V6 values. - Compose `config` output or deployment manifest digest. - `docker compose ps` or orchestrator health output. - Verification commands and pass/fail results. - Known disabled features, temporary flags, and expiry owners. QA signoff then follows `docs/testing/v6-manual-qa-recipes.md`. ## Rollback Rollback is allowed only after preserving evidence: 1. Save the deployment manifest, service logs, audit refs, and failed gate output. 2. Disable new Pixel Streaming admissions if user sessions are impacted. 3. Roll service images or Compose deployment back to the previous green digest. 4. Keep PostgreSQL and MinIO data in place unless the privacy or data-loss incident commander explicitly orders restore. 5. Rerun the targeted verification gates and the impacted manual-QA recipe. ## Handoff checklist - [ ] Data plane healthy: PostgreSQL, Redis, MinIO. - [ ] All seven V6 Rust services healthy. - [ ] Web, admin, Pixel Streaming, and fallback routes reachable over TLS. - [ ] V1 audit platform receives V6 operator and Ori-read events. - [ ] Browser automation gate green. - [ ] Security, residency, and cross-progression gates green. - [ ] Manual-QA recipes assigned with owners and evidence folder. - [ ] On-call runbooks linked in the launch channel. - [ ] Rollback digest and previous-green deployment identified.