# Preview Environments for Pull Requests

Preview environments let reviewers exercise a feature branch in isolation
without waiting for a shared staging deployment. Each pull request receives a
dedicated Kubernetes namespace, a deterministic release of core microservices,
and resource guardrails so we stay within budget.

## Architecture Overview

```
GitHub PR → Preview workflow → manage-preview-env.sh →
  • lilith-pr-### namespace
  • Helm releases (bff, auth, content, conversation, media, notification, moderation, ai)
  • ResourceQuota + LimitRange + TTL annotations
  • Ingress → https://pr-###.preview.lilith.dev
```

Key components:

- **Namespace naming** – `lilith-pr-{number}`. Namespaces carry branch metadata
  labels (`app.kubernetes.io/part-of = lilith-preview`,
  `lilith.ai/pr = {number}`) to simplify cleanup scans.
- **Helm deployment** – Each service is deployed via the `charts/lilith-service`
  umbrella chart and tagged with immutable images built in CI
  (`pr-{number}-{commit}`).
- **Ingress & TLS** – A preview wildcard certificate (`*.preview.lilith.dev`)
  terminates TLS. Routes are created per service behind the BFF; traffic flows
  through the shared Istio ingress gateway with preview-specific routing rules.
- **Cost guardrails** – `ResourceQuota` (CPU, memory, PVC counts) and
  `LimitRange` enforce per-pod ceilings. Namespaces carry a
  `preview.lilith.ai/ttl-hours=24` annotation and are scanned by the cleanup
  job.

## Workflow Summary

1. **Pull request opened / synchronized**
   - GitHub Actions workflow (`.github/workflows/preview-environments.yaml`)
     builds container images, pushes them to the registry with tags
     `pr-{PR_NUMBER}-{SHORT_SHA}`, and runs
     `scripts/manage-preview-env.sh create --pr <PR> --image-tag <tag>`.
   - The script creates the namespace (or updates it if already present),
     installs required Helm releases, applies quotas, and records metadata in a
     ConfigMap.
2. **Pull request updated**
   - Workflow calls `manage-preview-env.sh update` with the new tag. The script
     performs a rolling Helm upgrade, preserving quotas and ingress.
3. **Pull request closed or merged**
   - Workflow executes `manage-preview-env.sh cleanup --pr <PR>`. All Kubernetes
     objects are deleted, and PVCs are removed to avoid orphaned storage.
4. **Nightly cleanup**
   - Scheduled job runs `manage-preview-env.sh cleanup-expired` to delete
     namespaces exceeding the TTL or the maximum environment count.

## Resource Guardrails

- Maximum previews: **10** concurrent namespaces.
- CPU budget: **requests.cpu ≤ 1**, **limits.cpu ≤ 2** per namespace.
- Memory budget: **requests.memory ≤ 2Gi**, **limits.memory ≤ 4Gi**.
- Storage: up to **5 PVCs** and a cumulative **10Gi** ephemeral volume
  allocation.
- TTL: **24 hours**. Namespaces receive `preview.lilith.ai/expires-at`
  annotations that the cleaner inspects.

## Branch-to-Preview Mapping

| Event            | Script Command                                                             | Result                                            |
| ---------------- | -------------------------------------------------------------------------- | ------------------------------------------------- |
| PR opened/sync   | `manage-preview-env.sh create --pr <number> --image-tag pr-<number>-<sha>` | Namespace created, Helm install, ingress, quotas. |
| PR updated       | `manage-preview-env.sh update --pr <number> --image-tag pr-<number>-<sha>` | Helm release upgrade with zero-downtime rollout.  |
| PR closed/merged | `manage-preview-env.sh cleanup --pr <number>`                              | Namespace deletion, ingress removal, PVC cleanup. |
| Nightly cleanup  | `manage-preview-env.sh cleanup-expired`                                    | TTL expirations handled automatically.            |
| Guardrail check  | `manage-preview-env.sh validate-limits`                                    | Ensures environment count stays ≤ 10.             |

## Local Testing

To verify the preview workflow without touching a live cluster:

```bash
# Dry-run creation
scripts/manage-preview-env.sh create --pr 123 --image-tag pr-123-deadbeef --dry-run

# Dry-run update
scripts/manage-preview-env.sh update --pr 123 --image-tag pr-123-feedface --dry-run

# Dry-run cleanup
scripts/manage-preview-env.sh cleanup --pr 123 --dry-run
```

Dry-run mode prints the Kubernetes manifests and guardrails that would be
applied without executing `kubectl` or `helm` commands.

## Observability & Alerts

- Preview namespaces export metrics with label `preview_namespace = true`.
  Grafana dashboards highlight resource usage and impending TTL expiration.
- Orphan detection alerts fire if a namespace exceeds the TTL by >2 hours.
- Cost guardrail breaches (more than 10 namespaces or quota exhaustion) open
  PagerDuty warnings for the platform team.

## Security Considerations

- Secrets are sourced from Vault via the `external-secrets` integration. Preview
  namespaces receive dedicated, short-lived ExternalSecret manifests with
  branch-scoped credentials.
- Service accounts inherit restricted RBAC limiting access to preview resources
  only.
- Ingress routes require authentication via the same OIDC provider as staging,
  ensuring public pull requests do not expose protected endpoints.

## Cleanup Guarantees

- The cleanup command deletes namespaces, persistent volumes, and any leftover
  cluster-scoped resources labeled with `lilith.ai/pr`.
- On failure, the script emits actionable logs and exits non-zero so the
  workflow can notify maintainers.
- The nightly cleanup job ignores namespaces with the annotation
  `preview.lilith.ai/preserve=true` for approved long-lived demos.
