Preview environments let reviewers exercise a feature branch in isolation without waiting for a shared staging deployment. Each pull request receives a dedicated Kubernetes namespace, a deterministic release of core microservices, and resource guardrails so we stay within budget.
Architecture Overview#
text
GitHub PR → Preview workflow → manage-preview-env.sh →
• lilith-pr-### namespace
• Helm releases (bff, auth, content, conversation, media, notification, moderation, ai)
• ResourceQuota + LimitRange + TTL annotations
• Ingress → https://pr-###.preview.lilith.dev
Key components:
- Namespace naming –
lilith-pr-{number}. Namespaces carry branch metadata labels (app.kubernetes.io/part-of = lilith-preview,lilith.ai/pr = {number}) to simplify cleanup scans. - Helm deployment – Each service is deployed via the
charts/lilith-serviceumbrella chart and tagged with immutable images built in CI (pr-{number}-{commit}). - Ingress & TLS – A preview wildcard certificate (
*.preview.lilith.dev) terminates TLS. Routes are created per service behind the BFF; traffic flows through the shared Istio ingress gateway with preview-specific routing rules. - Cost guardrails –
ResourceQuota(CPU, memory, PVC counts) andLimitRangeenforce per-pod ceilings. Namespaces carry apreview.lilith.ai/ttl-hours=24annotation and are scanned by the cleanup job.
Workflow Summary#
- Pull request opened / synchronized
- GitHub Actions workflow (
.github/workflows/preview-environments.yaml) builds container images, pushes them to the registry with tagspr-{PR_NUMBER}-{SHORT_SHA}, and runsscripts/manage-preview-env.sh create --pr <PR> --image-tag <tag>. - The script creates the namespace (or updates it if already present), installs required Helm releases, applies quotas, and records metadata in a ConfigMap.
- GitHub Actions workflow (
- Pull request updated
- Workflow calls
manage-preview-env.sh updatewith the new tag. The script performs a rolling Helm upgrade, preserving quotas and ingress.
- Workflow calls
- Pull request closed or merged
- Workflow executes
manage-preview-env.sh cleanup --pr <PR>. All Kubernetes objects are deleted, and PVCs are removed to avoid orphaned storage.
- Workflow executes
- Nightly cleanup
- Scheduled job runs
manage-preview-env.sh cleanup-expiredto delete namespaces exceeding the TTL or the maximum environment count.
- Scheduled job runs
Resource Guardrails#
- Maximum previews: 10 concurrent namespaces.
- CPU budget: requests.cpu ≤ 1, limits.cpu ≤ 2 per namespace.
- Memory budget: requests.memory ≤ 2Gi, limits.memory ≤ 4Gi.
- Storage: up to 5 PVCs and a cumulative 10Gi ephemeral volume allocation.
- TTL: 24 hours. Namespaces receive
preview.lilith.ai/expires-atannotations that the cleaner inspects.
Branch-to-Preview Mapping#
| Event | Script Command | Result |
|---|---|---|
| PR opened/sync | manage-preview-env.sh create --pr <number> --image-tag pr-<number>-<sha> |
Namespace created, Helm install, ingress, quotas. |
| PR updated | manage-preview-env.sh update --pr <number> --image-tag pr-<number>-<sha> |
Helm release upgrade with zero-downtime rollout. |
| PR closed/merged | manage-preview-env.sh cleanup --pr <number> |
Namespace deletion, ingress removal, PVC cleanup. |
| Nightly cleanup | manage-preview-env.sh cleanup-expired |
TTL expirations handled automatically. |
| Guardrail check | manage-preview-env.sh validate-limits |
Ensures environment count stays ≤ 10. |
Local Testing#
To verify the preview workflow without touching a live cluster:
bash
# Dry-run creation
scripts/manage-preview-env.sh create --pr 123 --image-tag pr-123-deadbeef --dry-run
# Dry-run update
scripts/manage-preview-env.sh update --pr 123 --image-tag pr-123-feedface --dry-run
# Dry-run cleanup
scripts/manage-preview-env.sh cleanup --pr 123 --dry-run
Dry-run mode prints the Kubernetes manifests and guardrails that would be
applied without executing kubectl or helm commands.
Observability & Alerts#
- Preview namespaces export metrics with label
preview_namespace = true. Grafana dashboards highlight resource usage and impending TTL expiration. - Orphan detection alerts fire if a namespace exceeds the TTL by >2 hours.
- Cost guardrail breaches (more than 10 namespaces or quota exhaustion) open PagerDuty warnings for the platform team.
Security Considerations#
- Secrets are sourced from Vault via the
external-secretsintegration. Preview namespaces receive dedicated, short-lived ExternalSecret manifests with branch-scoped credentials. - Service accounts inherit restricted RBAC limiting access to preview resources only.
- Ingress routes require authentication via the same OIDC provider as staging, ensuring public pull requests do not expose protected endpoints.
Cleanup Guarantees#
- The cleanup command deletes namespaces, persistent volumes, and any leftover
cluster-scoped resources labeled with
lilith.ai/pr. - On failure, the script emits actionable logs and exits non-zero so the workflow can notify maintainers.
- The nightly cleanup job ignores namespaces with the annotation
preview.lilith.ai/preserve=truefor approved long-lived demos.