This deployment uses a dedicated rootless Docker daemon under
/mnt/volume300/oshun-platform, independent of the host's existing containers.
Each environment has its own Compose project, network, volumes, passwords,
application PostgreSQL, Valkey, and object storage. The Eve task backlog is
intentionally shared; both BFFs receive its read-only PostgreSQL role. Metis
also runs its existing Celery worker (one concurrent job) and scheduler, using
its own Redis databases and six private MinIO buckets. Its signed object URLs
use the environment's public S3 hostname, so file transfers require the final
DNS/TLS activation.
The Metis Python image preserves the repository directory layout. Its shared profiles, contracts, OpenAPI documents, sandbox harness files, and admin route tree are mounted read-only from the same release checkout. Sandboxed code execution and Manim rendering still require their separately provisioned execution workers; the application deployment does not provision those workers. Dev also runs the existing Metis OpenAPI, API contract, command palette, and interoperability tests inside the packaged image with these same mounts. The disposable test container receives no live credentials.
A systemd timer fetches origin/main every two minutes. One lock serializes
builds and deployments. It builds immutable commit tags locally, runs the
repository's document and presentation checks, migrates databases, waits for
service health, and checks the routed applications. Eve’s product graph and full
document-search corpus are regenerated from each release and mounted read-only.
Only a successful run writes a Dev verification receipt. Build failures preserve
the current release; a failed rollout attempts application rollback. Database
migrations are forward-only and must remain compatible with the previous
application version.
No GitLab, GitHub mirroring, hosted runner minutes, or container registry are needed. This uses the existing server's CPU, disk, and bandwidth; the server and object storage still have their normal costs. Only trusted main-branch commits are executed on this deployment host.
The older GitHub Hetzner workflow no longer runs on main pushes, avoiding a duplicate hosted build for Dev. Its manual and production tag/release triggers remain available. Other GitHub workflows are unchanged and can still consume their own hosted-runner minutes and artifact storage.
The legacy production topology suite (v1-topology-record.test.mjs) already has
seven failures at c036653b1a8: its declared inventory predates the monitoring
services and backup metrics mount. These were reproduced from an unmodified
checkout. The deployment-trigger and Prisma migration-path suites pass; this
Dev/UAT overlay has its own isolation and promotion checks and does not claim
that the legacy production topology is certified.
Setup and operation#
Run the Eve PostgreSQL setup in ../eve/README.md first.
Install Docker Engine/Compose/Buildx with its rootless prerequisites (uidmap,
slirp4netns, dbus-user-session), Python 3 with PyYAML/pytest, and the
Node/pnpm versions pinned in the controller and root package manifest.
python3 infra/hetzner/platform/bootstrap-runtime.py
python3 infra/hetzner/platform/configure.py
python3 infra/hetzner/platform/edge.py install
python3 /usr/local/lib/oshun-platform/release.py poll
python3 /usr/local/lib/oshun-platform/release.py status
The isolated BuildKit builder oshun-sequential uses
buildkitd.toml, two CPUs, a 6 GiB memory limit and a 12 GiB
combined memory/swap limit. The controller supervises memory and volume free
space every five seconds. Next builds receive an 8 GiB JavaScript heap ceiling;
their full TypeScript graph exceeds a 6 GiB heap even when compilation passes.
The controller stops its process group and dedicated builder if available RAM
falls below 2 GiB or volume space below 10 GiB. Builds are sequential; broad Nx
sweeps do not run here. Each web app retains its own Next compiler cache inside
BuildKit; source and environment changes still run the build and typecheck
gates.
The consumer web application explicitly enables Next's Webpack build worker and memory optimizations. Its custom Webpack hook otherwise keeps the compiler graph in the parent process while the TypeScript worker starts, exceeding the shared builder's combined memory/swap budget. Compiler and typecheck phases retain their checks and run in separate processes.
The build daemon stops after image creation, including failed builds, to release its retained memory before verification and application rollout. Its disk cache is preserved and the next image build starts it automatically. The daemon also has a 2 GiB Go garbage-collector soft limit, independent of its child build processes. The larger Node typechecks can use swap within the combined 12 GiB limit, leaving physical RAM for both environments and the host's other services. Before a new image build, the controller reduces the physical limit as needed to leave 3 GiB of currently available RAM for the host. It defers builds when less than 5 GiB is available, preserving the running release for the next poll. The combined RAM/swap ceiling remains 12 GiB.
After the first verified deployment and after these files are on origin/main:
python3 infra/hetzner/platform/configure.py --enable
systemctl status oshun-main-deploy.timer
journalctl -u oshun-main-deploy.service
UAT never follows main automatically. Promote a full commit SHA with a successful Dev receipt:
python3 /usr/local/lib/oshun-platform/release.py promote FULL_40_CHARACTER_SHA
Backend artifacts are reused from that tested Dev commit. Browser applications are rebuilt from the same commit with UAT API URLs and pass the UAT rollout checks. The document and presentation artifacts reuse the successful Dev verification for that exact commit. Current and previous releases for each environment are retained; old release checkouts and their application image tags are removed. Persistent volumes are never pruned by the deployment controller.
Eve publication deployment#
The publication controller is opt-in during the content migration. After a full
publication preparation and verification succeeds, operators can activate
ci/publications.json with {"version":1,"enabled":true}. The installed
bootstrap must include publication_deployments.py alongside release.py and
release_storage.py. Keep this configuration absent until that cutover; normal
commit deployment remains available during preparation.
With publication deployment active, the timer freezes current Eve content for
each new main commit and also detects newer Dev publications at the same commit.
An authenticated API snapshot supplies current task state. Preparation builds
and browser-tests both centers, search, and the product graph before sealing
checksummed artifacts under ci/publications/<Dev-publication-id>. Failed
attempts keep diagnostic evidence and cannot issue a tested receipt. Public
routes mount only the generated artifacts and restricted original downloads; the
private preparation tree and code checkout are not served.
The BFF reads the mounted publication graph at startup and rejects a missing or mismatched graph instead of serving its bundled development inventory. Search corpora stream through the actual BFF parser during preparation, with bounded 128 MiB files, 200,000 sections and 1,024-character anchors. Both audience scopes and the graph must load successfully before artifact sealing. The full-corpus capacity rehearsal loaded 155,785 sections with a 1 GiB Node heap and measured 865 MiB peak RSS for the bundled runtime; this is not a live deployment receipt.
Promote the tested publication in Eve's Publications workspace to select UAT.
The timer deploys that explicitly promoted code and content, using the exact
sealed Dev artifacts and ci/tested/publications/<id>.json receipt. A
successful commit receipt for different content cannot authorize promotion.
Missing or changed artifacts stop promotion instead of rebuilding from current
content. Content-only releases have independent deployment paths and rollback
receipts, so rollback at the same commit restores the prior content. Publication
artifacts and their tested receipts remain retained independently of expired
code checkouts. No automatic publication retention policy is enabled yet.
Private .env and settings.json files live under
/mnt/volume300/oshun-platform/environments/{dev,uat} with mode 0600. They
are not Git files. Do not print resolved Compose configuration: it contains
secrets. docker compose config --quiet validates it safely.
Access and DNS#
The internal HTTP routers bind loopback only: Dev 18880, UAT 19880. The
dedicated Docs Center listeners are 18881 / 19881; Presentation Center
listeners are 18882 / 19882. Before DNS, an operator can use an SSH tunnel
and matching Host headers to inspect the private deployment.
DNS is the last step. In Route 53, open the public oshunsynthetics.com
hosted zone and create these four records:
| Name | Type | Alias | Value | TTL | Routing |
|---|---|---|---|---|---|
dev |
A | Off | 46.62.222.247 |
300 | Simple |
*.dev |
A | Off | 46.62.222.247 |
300 | Simple |
uat |
A | Off | 46.62.222.247 |
300 | Simple |
*.uat |
A | Off | 46.62.222.247 |
300 | Simple |
Do not change the apex, nameserver, or email records. A wildcard does not cover
the environment apex, hence both dev and *.dev (and both UAT records). The
routes include api, admin, eve, tenant, tg, docs, presentations,
metis, metis-admin, metis-api, psyche-api, s3, and corporate below
each environment name. The oshun-edge-dns.timer waits for both tested
deployments and all these hostnames to resolve to this IP, then enables the
native Caddy HTTPS service. It does not write DNS. Caddy obtains and renews
certificates automatically; verify issuance with
journalctl -u oshun-edge.service after adding the records. Ports 80 and 443
must be reachable for the HTTP certificate challenge and HTTPS.
Known upstream service blockers#
The user selected deployment of runnable applications while reporting missing servers separately. The controller discovers these missing entrypoints and records them in every deployment receipt; they are excluded from Compose. When main gains an entrypoint, that service rejoins the build and health gates. A successful receipt covers the deployed applications, not these blocked features. Current missing servers:
psyche-orchestrator:apps/psyche/orchestrator/src/orchestrator/main.py.psyche-behavior-engine:apps/psyche/behavior-engine/src/behavior_engine/main.py.psyche-perception-engine:apps/psyche/perception-engine/src/perception_engine/main.py.psyche-computer-use:apps/psyche/computer-use/src/computer_use/main.py.psyche-conferencing:apps/psyche/conferencing/src/video_conferencing/main.py.psyche-knowledge-base:apps/psyche/knowledge-base/src/knowledge_base/main.py.psyche-avatar-engine:apps/psyche/avatar-engine/src/avatar_engine/main.py.psyche-voice-engine:apps/psyche/voice-engine/src/voice_engine/main.py.
Linux hosting also does not build native iOS/watchOS applications or supply a GPU for local neural rendering. Existing feature/release gates remain effective. The content service is also excluded while its source-owned provider posture review explicitly blocks startup. Deployment does not override that review.
Backup scope#
The shared Eve board/catalog has continuous WAL archiving and tested pgBackRest
restores from app-dev-server; see the Eve runbook for retention and recovery.
The application databases additionally receive hourly custom-format PostgreSQL
dumps in app-dev-server/oshun-platform/databases/{dev,uat}/, retained for 30
days. Each dump has a SHA-256 manifest, format validation, and upload-size
verification. Run python3 infra/hetzner/platform/restore-databases.py dev (or
uat) to download the newest manifest, verify every checksum, and restore all
databases into a disposable PostgreSQL container with no network access. These
protect PostgreSQL only, not MinIO objects or other file volumes. Their recovery
exposure is up to one hour plus upload time. Each new snapshot also includes
runtime-keys.json in the private backup prefix: the application-owned
encryption and signing keys needed after a total server loss. Bucket credentials
and third-party API keys are excluded. Restore verification downloads and checks
this file without printing its values or changing the live environment. For
disaster recovery, restore these values into the environment's restricted
settings.json before running configure.py or starting its BFF; preserve the
recorded key refs. Check oshun-database-backup.timer and
/mnt/volume300/oshun-platform/backups/*-latest.json. The Compose stacks also
retain their nightly local PostgreSQL dumps. The original production immutable
off-box recovery integration is not enabled in these Dev/UAT overlays: its
external Object Lock and escrow configuration have not been supplied. A disabled
backup sidecar is not off-site protection. The overlay does not claim production
recovery certification. Account-deletion fan-out remains unavailable until the
independent recovery journal is admitted and configured; deletion requests stay
scheduled and are never falsely marked completed. Fresh server deployments also
leave the six domain HTTP adapters (Tara, Veritas, Nyx, Arete, Nisaba, and
Metis) unavailable: their source-owned provider admission is
blocked-unattested. They use the resolver's closed loopback origin until
admission and real service origins are configured. The independently hosted
Metis applications remain part of the deployment. /healthz verifies the
running BFF process; /readyz continues to report the unavailable domain
adapters rather than claiming full domain readiness. Fresh server checkouts
fetch hosted-application LFS payloads and omit unrelated Unreal assets and
visual-test baselines; an upstream V2 LFS object currently returns 404.