Eve library

Bellona Remote — Production Hetzner Deployment (180.C.25.08)

Other

Revision 1

On this page

Production-ready, single-node Hetzner Cloud deployment of the Bellona Remote Gateway data plane: a TLS edge (Caddy + ACME), the gateway service, a coturn TURN relay, PostgreSQL, MinIO object storage, Prometheus/Grafana monitoring, Loki/Promtail log aggregation, nightly Postgres backups, and a smoke test.

Honesty note. The artifacts in this directory ARE the deliverable. The live docker compose up against a real Hetzner VM is account/credential-gated and is not run in CI or on the build box. Everything here is validated for syntax (docker compose config, bash -n) and internal consistency. The smoke test is the post-deploy validation hook; it fails loud (exit non-zero) when no deployed URL is provided and never fabricates a pass.

Topology#

text
                 internet
                    │ 80/443 (only public app ports)
                    ▼
              ┌───────────┐   ACME / Let's Encrypt
              │   caddy    │── automatic HTTPS for $BELLONA_GATEWAY_DOMAIN
              └─────┬──────┘
       /health /ready /v1/* │ /v1/hosts/connect (WSS)   /grafana  /metrics-internal
                    ▼            ▼
              ┌───────────┐   private "app" network (never published)
              │  gateway  │──┬─ postgres   (persistent store)
              │  :4070    │  ├─ minio      (artifact object store)
              └───────────┘  ├─ prometheus + grafana
                             └─ loki  ◄─ promtail (all container logs)

  coturn :3478/:5349 + relay 49160-49200  (TURN media relay, 180.C.25.04)
  pg-backup  cron → nightly pg_dump + MinIO mirror + checksum + retention

The gateway, Postgres, MinIO, Prometheus, Grafana, and Loki are bound to a private Docker network. Caddy is the only service publishing 80/443; coturn publishes its STUN/TURN/relay ports (host networking) for WebRTC media. Nothing else is reachable from the internet.

Files#

File Purpose
docker-compose.prod.yml The full topology.
.env.example Every required env var (copy → .env, fill secrets).
caddy/Caddyfile TLS edge + reverse-proxy routes.
coturn/turnserver.conf TURN relay (time-limited creds, TLS, hardened).
monitoring/prometheus.yml Scrape config (gateway, coturn, loki, promtail).
monitoring/loki-config.yml Log store (filesystem, 14-day retention).
monitoring/promtail-config.yml Ships com.bellona.*-labelled container logs.
monitoring/grafana/provisioning/ Prometheus + Loki datasources, dashboard provider.
backup/pg-backup.sh pg_dump + SHA-256 + MinIO mirror + retention.
backup/entrypoint.sh Installs the backup cron, runs a baseline dump.

Env wiring#

The gateway env in docker-compose.prod.yml mirrors apps/bellona/remote-gateway/src/config.ts exactly: BELLONA_REMOTE_GATEWAY_HOST/PORT/LOG_LEVEL/STORE_FILE/HOST_TOKEN_SIGNING_SECRET/ HOST_TOKEN_TTL_MS/PAIRING_CODE_TTL_MS/APPROVAL_REQUEST_TTL_MS/ HOST_HEARTBEAT_INTERVAL_MS/HOST_HEARTBEAT_TIMEOUT_MS/SHUTDOWN_GRACE_MS. The data-plane vars (BELLONA_DATABASE_URL, BELLONA_OBJECT_STORE_*, BELLONA_TURN_*) point at the in-stack Postgres/MinIO/coturn.

HOST=0.0.0.0 is safe here because the gateway is only attached to the private Docker networks; only Caddy is published.

Provisioning#

1. Hetzner VM + Cloud Firewall#

  • Create a cx22 (or larger) Ubuntu LTS VM in your closest region.

  • Point your domain's A (and AAAA) records at the VM and set BELLONA_GATEWAY_DOMAIN to it.

  • Hetzner Cloud Firewalls deny inbound by default. Open:

    Direction Protocol Port Source Purpose
    Inbound TCP 22 operator CIDR SSH (remove after commissioning)
    Inbound TCP 80 0.0.0.0/0, ::/0 ACME HTTP-01 + redirect
    Inbound TCP 443 0.0.0.0/0, ::/0 HTTPS/WSS gateway edge
    Inbound UDP 3478 0.0.0.0/0 STUN/TURN
    Inbound TCP 3478 0.0.0.0/0 TURN over TCP
    Inbound TCP 5349 0.0.0.0/0 TURN over TLS
    Inbound UDP 49160-49200 0.0.0.0/0 TURN relay range

    Do not open 4070 (gateway), 5432 (Postgres), 9000/9001 (MinIO), 9090 (Prometheus), 3000 (Grafana), or 3100 (Loki).

2. Install Docker + Compose#

sh
curl -fsSL https://get.docker.com | sh
docker compose version   # confirm the compose plugin is present

3. Configure#

sh
git clone <repo> /opt/oshun && cd /opt/oshun/infra/bellona-remote/hetzner
cp .env.example .env
# Fill EVERY secret. Generate strong values, e.g.:
#   openssl rand -base64 48   # signing secret, db/minio/turn secrets
#   caddy hash-password --plaintext '<pw>'   # metrics basic-auth hash
$EDITOR .env

The compose file marks security-critical vars required (:?), so a missing secret fails docker compose config instead of silently deploying a default.

4. Build + start#

sh
docker compose -f docker-compose.prod.yml --env-file .env build gateway
docker compose -f docker-compose.prod.yml --env-file .env up -d
docker compose -f docker-compose.prod.yml --env-file .env ps

Caddy obtains the certificate on first request to :80. coturn's TLS listener reuses that certificate from the shared caddy_data volume — after the first issuance, confirm the cert/key paths in coturn/turnserver.conf resolve (they follow Caddy's storage layout for acme-v02.api.letsencrypt.org-directory); if your ACME CA differs, adjust the cert=/pkey= paths or symlink the leaf into place, then docker compose restart coturn.

5. Smoke test#

sh
BELLONA_SMOKE_BASE_URL=https://$BELLONA_GATEWAY_DOMAIN ../smoke/smoke-test.sh

Asserts /health → healthy, /ready → ready (with checks), the /v1/pairing/exchange route rejects an invalid body (route + validator live), and /v1/devices returns a device array. Without BELLONA_SMOKE_BASE_URL it exits non-zero — it never reports a fake pass.

Monitoring & logs#

  • Grafana: https://$BELLONA_GATEWAY_DOMAIN/grafana (admin / GRAFANA_ADMIN_PASSWORD). Prometheus + Loki datasources are pre-provisioned.
  • Prometheus: https://$BELLONA_GATEWAY_DOMAIN/metrics-internal/ behind Caddy basic auth.
  • Logs: every Bellona container is labelled com.bellona.service; Promtail ships them to Loki. Query in Grafana → Explore → Loki, e.g. {service="remote-gateway"} | json | level="error".

Operational logs are not the audit record — audit/timeline evidence stays owned by the gateway services and persistent store.

Backups#

pg-backup runs BACKUP_CRON (default 02:30 daily): a custom-format pg_dump --compress=9 written to the backup_data volume with a SHA-256 checksum and JSON metadata, mirrored to the MinIO BELLONA_BACKUP_BUCKET, with local dumps older than BACKUP_RETENTION_DAYS pruned. A baseline dump runs on container start.

Restore:

sh
# Verify integrity first.
sha256sum -c /path/to/bellona_bellona_<ts>.dump.sha256
PGPASSWORD=$POSTGRES_PASSWORD pg_restore --clean --if-exists \
  -h <host> -U $POSTGRES_USER -d $POSTGRES_DB /path/to/bellona_bellona_<ts>.dump

The gateway's first-stage JSON store lives on the gateway_store volume; snapshot it (docker run --rm -v bellona-gateway-store:/v -v "$PWD":/out alpine tar czf /out/gateway-store.tgz -C /v .) before destructive maintenance.

Limitations (honest)#

  • The live docker compose up against real Hetzner is account-gated and not run here; these files are validated for syntax + consistency only.
  • The gateway ships a first-stage JSON store today; Postgres/MinIO are provisioned and wired so the topology is complete and the connection strings are correct, ahead of the gateway's persistence/artifact code paths consuming them.
  • coturn TLS reuses Caddy's ACME cert via a shared volume; the exact cert path depends on your ACME CA and may need a one-line adjustment after first issuance.
  • A live WebRTC media-relay smoke (proving an actual relay session through coturn) requires real peers and is outside this script's scope; the metric classification + summary it feeds are unit-tested in libs/bellona/remote-protocol/src/stream.test.ts.