# Bellona Remote — Production Hetzner Deployment (180.C.25.08)

Production-ready, single-node Hetzner Cloud deployment of the Bellona Remote
Gateway data plane: a TLS edge (Caddy + ACME), the gateway service, a coturn
TURN relay, PostgreSQL, MinIO object storage, Prometheus/Grafana monitoring,
Loki/Promtail log aggregation, nightly Postgres backups, and a smoke test.

> **Honesty note.** The artifacts in this directory ARE the deliverable. The
> live `docker compose up` against a real Hetzner VM is account/credential-gated
> and is **not** run in CI or on the build box. Everything here is validated for
> syntax (`docker compose config`, `bash -n`) and internal consistency. The
> [smoke test](../smoke/smoke-test.sh) is the post-deploy validation hook; it
> **fails loud** (exit non-zero) when no deployed URL is provided and never
> fabricates a pass.

## Topology

```
                 internet
                    │ 80/443 (only public app ports)
                    ▼
              ┌───────────┐   ACME / Let's Encrypt
              │   caddy    │── automatic HTTPS for $BELLONA_GATEWAY_DOMAIN
              └─────┬──────┘
       /health /ready /v1/* │ /v1/hosts/connect (WSS)   /grafana  /metrics-internal
                    ▼            ▼
              ┌───────────┐   private "app" network (never published)
              │  gateway  │──┬─ postgres   (persistent store)
              │  :4070    │  ├─ minio      (artifact object store)
              └───────────┘  ├─ prometheus + grafana
                             └─ loki  ◄─ promtail (all container logs)

  coturn :3478/:5349 + relay 49160-49200  (TURN media relay, 180.C.25.04)
  pg-backup  cron → nightly pg_dump + MinIO mirror + checksum + retention
```

The gateway, Postgres, MinIO, Prometheus, Grafana, and Loki are bound to a
private Docker network. **Caddy is the only service publishing 80/443**; coturn
publishes its STUN/TURN/relay ports (host networking) for WebRTC media. Nothing
else is reachable from the internet.

## Files

| File                               | Purpose                                               |
| ---------------------------------- | ----------------------------------------------------- |
| `docker-compose.prod.yml`          | The full topology.                                    |
| `.env.example`                     | Every required env var (copy → `.env`, fill secrets). |
| `caddy/Caddyfile`                  | TLS edge + reverse-proxy routes.                      |
| `coturn/turnserver.conf`           | TURN relay (time-limited creds, TLS, hardened).       |
| `monitoring/prometheus.yml`        | Scrape config (gateway, coturn, loki, promtail).      |
| `monitoring/loki-config.yml`       | Log store (filesystem, 14-day retention).             |
| `monitoring/promtail-config.yml`   | Ships `com.bellona.*`-labelled container logs.        |
| `monitoring/grafana/provisioning/` | Prometheus + Loki datasources, dashboard provider.    |
| `backup/pg-backup.sh`              | `pg_dump` + SHA-256 + MinIO mirror + retention.       |
| `backup/entrypoint.sh`             | Installs the backup cron, runs a baseline dump.       |

## Env wiring

The gateway env in `docker-compose.prod.yml` mirrors
`apps/bellona/remote-gateway/src/config.ts` exactly:
`BELLONA_REMOTE_GATEWAY_HOST/PORT/LOG_LEVEL/STORE_FILE/HOST_TOKEN_SIGNING_SECRET/ HOST_TOKEN_TTL_MS/PAIRING_CODE_TTL_MS/APPROVAL_REQUEST_TTL_MS/ HOST_HEARTBEAT_INTERVAL_MS/HOST_HEARTBEAT_TIMEOUT_MS/SHUTDOWN_GRACE_MS`.
The data-plane vars (`BELLONA_DATABASE_URL`, `BELLONA_OBJECT_STORE_*`,
`BELLONA_TURN_*`) point at the in-stack Postgres/MinIO/coturn.

`HOST=0.0.0.0` is safe here because the gateway is only attached to the private
Docker networks; only Caddy is published.

## Provisioning

### 1. Hetzner VM + Cloud Firewall

- Create a `cx22` (or larger) Ubuntu LTS VM in your closest region.
- Point your domain's A (and AAAA) records at the VM and set
  `BELLONA_GATEWAY_DOMAIN` to it.
- Hetzner Cloud Firewalls deny inbound by default. Open:

  | Direction | Protocol | Port        | Source          | Purpose                          |
  | --------- | -------- | ----------- | --------------- | -------------------------------- |
  | Inbound   | TCP      | 22          | operator CIDR   | SSH (remove after commissioning) |
  | Inbound   | TCP      | 80          | 0.0.0.0/0, ::/0 | ACME HTTP-01 + redirect          |
  | Inbound   | TCP      | 443         | 0.0.0.0/0, ::/0 | HTTPS/WSS gateway edge           |
  | Inbound   | UDP      | 3478        | 0.0.0.0/0       | STUN/TURN                        |
  | Inbound   | TCP      | 3478        | 0.0.0.0/0       | TURN over TCP                    |
  | Inbound   | TCP      | 5349        | 0.0.0.0/0       | TURN over TLS                    |
  | Inbound   | UDP      | 49160-49200 | 0.0.0.0/0       | TURN relay range                 |

  Do **not** open 4070 (gateway), 5432 (Postgres), 9000/9001 (MinIO), 9090
  (Prometheus), 3000 (Grafana), or 3100 (Loki).

### 2. Install Docker + Compose

```sh
curl -fsSL https://get.docker.com | sh
docker compose version   # confirm the compose plugin is present
```

### 3. Configure

```sh
git clone <repo> /opt/oshun && cd /opt/oshun/infra/bellona-remote/hetzner
cp .env.example .env
# Fill EVERY secret. Generate strong values, e.g.:
#   openssl rand -base64 48   # signing secret, db/minio/turn secrets
#   caddy hash-password --plaintext '<pw>'   # metrics basic-auth hash
$EDITOR .env
```

The compose file marks security-critical vars required (`:?`), so a missing
secret fails `docker compose config` instead of silently deploying a default.

### 4. Build + start

```sh
docker compose -f docker-compose.prod.yml --env-file .env build gateway
docker compose -f docker-compose.prod.yml --env-file .env up -d
docker compose -f docker-compose.prod.yml --env-file .env ps
```

Caddy obtains the certificate on first request to `:80`. coturn's TLS listener
reuses that certificate from the shared `caddy_data` volume — after the first
issuance, confirm the cert/key paths in `coturn/turnserver.conf` resolve (they
follow Caddy's storage layout for `acme-v02.api.letsencrypt.org-directory`); if
your ACME CA differs, adjust the `cert=`/`pkey=` paths or symlink the leaf into
place, then `docker compose restart coturn`.

### 5. Smoke test

```sh
BELLONA_SMOKE_BASE_URL=https://$BELLONA_GATEWAY_DOMAIN ../smoke/smoke-test.sh
```

Asserts `/health` → healthy, `/ready` → ready (with `checks`), the
`/v1/pairing/exchange` route rejects an invalid body (route + validator live),
and `/v1/devices` returns a device array. Without `BELLONA_SMOKE_BASE_URL` it
exits non-zero — it never reports a fake pass.

## Monitoring & logs

- **Grafana**: `https://$BELLONA_GATEWAY_DOMAIN/grafana` (admin /
  `GRAFANA_ADMIN_PASSWORD`). Prometheus + Loki datasources are pre-provisioned.
- **Prometheus**: `https://$BELLONA_GATEWAY_DOMAIN/metrics-internal/` behind
  Caddy basic auth.
- **Logs**: every Bellona container is labelled `com.bellona.service`; Promtail
  ships them to Loki. Query in Grafana → Explore → Loki, e.g.
  `{service="remote-gateway"} | json | level="error"`.

Operational logs are **not** the audit record — audit/timeline evidence stays
owned by the gateway services and persistent store.

## Backups

`pg-backup` runs `BACKUP_CRON` (default 02:30 daily): a custom-format
`pg_dump --compress=9` written to the `backup_data` volume with a SHA-256
checksum and JSON metadata, mirrored to the MinIO `BELLONA_BACKUP_BUCKET`, with
local dumps older than `BACKUP_RETENTION_DAYS` pruned. A baseline dump runs on
container start.

Restore:

```sh
# Verify integrity first.
sha256sum -c /path/to/bellona_bellona_<ts>.dump.sha256
PGPASSWORD=$POSTGRES_PASSWORD pg_restore --clean --if-exists \
  -h <host> -U $POSTGRES_USER -d $POSTGRES_DB /path/to/bellona_bellona_<ts>.dump
```

The gateway's first-stage JSON store lives on the `gateway_store` volume;
snapshot it
(`docker run --rm -v bellona-gateway-store:/v -v "$PWD":/out alpine tar czf /out/gateway-store.tgz -C /v .`)
before destructive maintenance.

## Limitations (honest)

- The live `docker compose up` against real Hetzner is account-gated and **not**
  run here; these files are validated for syntax + consistency only.
- The gateway ships a first-stage JSON store today; Postgres/MinIO are
  provisioned and wired so the topology is complete and the connection strings
  are correct, ahead of the gateway's persistence/artifact code paths consuming
  them.
- coturn TLS reuses Caddy's ACME cert via a shared volume; the exact cert path
  depends on your ACME CA and may need a one-line adjustment after first
  issuance.
- A live WebRTC media-relay smoke (proving an actual `relay` session through
  coturn) requires real peers and is outside this script's scope; the metric
  classification + summary it feeds are unit-tested in
  `libs/bellona/remote-protocol/src/stream.test.ts`.
