Eve library

Preview Environments for Pull Requests

Guide

Revision 1

On this page

Preview environments let reviewers exercise a feature branch in isolation without waiting for a shared staging deployment. Each pull request receives a dedicated Kubernetes namespace, a deterministic release of core microservices, and resource guardrails so we stay within budget.

Architecture Overview#

text
GitHub PR → Preview workflow → manage-preview-env.sh →
  • lilith-pr-### namespace
  • Helm releases (bff, auth, content, conversation, media, notification, moderation, ai)
  • ResourceQuota + LimitRange + TTL annotations
  • Ingress → https://pr-###.preview.lilith.dev

Key components:

  • Namespace naming – lilith-pr-{number}. Namespaces carry branch metadata labels (app.kubernetes.io/part-of = lilith-preview, lilith.ai/pr = {number}) to simplify cleanup scans.
  • Helm deployment – Each service is deployed via the charts/lilith-service umbrella chart and tagged with immutable images built in CI (pr-{number}-{commit}).
  • Ingress & TLS – A preview wildcard certificate (*.preview.lilith.dev) terminates TLS. Routes are created per service behind the BFF; traffic flows through the shared Istio ingress gateway with preview-specific routing rules.
  • Cost guardrails – ResourceQuota (CPU, memory, PVC counts) and LimitRange enforce per-pod ceilings. Namespaces carry a preview.lilith.ai/ttl-hours=24 annotation and are scanned by the cleanup job.

Workflow Summary#

  1. Pull request opened / synchronized
    • GitHub Actions workflow (.github/workflows/preview-environments.yaml) builds container images, pushes them to the registry with tags pr-{PR_NUMBER}-{SHORT_SHA}, and runs scripts/manage-preview-env.sh create --pr <PR> --image-tag <tag>.
    • The script creates the namespace (or updates it if already present), installs required Helm releases, applies quotas, and records metadata in a ConfigMap.
  2. Pull request updated
    • Workflow calls manage-preview-env.sh update with the new tag. The script performs a rolling Helm upgrade, preserving quotas and ingress.
  3. Pull request closed or merged
    • Workflow executes manage-preview-env.sh cleanup --pr <PR>. All Kubernetes objects are deleted, and PVCs are removed to avoid orphaned storage.
  4. Nightly cleanup
    • Scheduled job runs manage-preview-env.sh cleanup-expired to delete namespaces exceeding the TTL or the maximum environment count.

Resource Guardrails#

  • Maximum previews: 10 concurrent namespaces.
  • CPU budget: requests.cpu ≤ 1, limits.cpu ≤ 2 per namespace.
  • Memory budget: requests.memory ≤ 2Gi, limits.memory ≤ 4Gi.
  • Storage: up to 5 PVCs and a cumulative 10Gi ephemeral volume allocation.
  • TTL: 24 hours. Namespaces receive preview.lilith.ai/expires-at annotations that the cleaner inspects.

Branch-to-Preview Mapping#

Event Script Command Result
PR opened/sync manage-preview-env.sh create --pr <number> --image-tag pr-<number>-<sha> Namespace created, Helm install, ingress, quotas.
PR updated manage-preview-env.sh update --pr <number> --image-tag pr-<number>-<sha> Helm release upgrade with zero-downtime rollout.
PR closed/merged manage-preview-env.sh cleanup --pr <number> Namespace deletion, ingress removal, PVC cleanup.
Nightly cleanup manage-preview-env.sh cleanup-expired TTL expirations handled automatically.
Guardrail check manage-preview-env.sh validate-limits Ensures environment count stays ≤ 10.

Local Testing#

To verify the preview workflow without touching a live cluster:

bash
# Dry-run creation
scripts/manage-preview-env.sh create --pr 123 --image-tag pr-123-deadbeef --dry-run

# Dry-run update
scripts/manage-preview-env.sh update --pr 123 --image-tag pr-123-feedface --dry-run

# Dry-run cleanup
scripts/manage-preview-env.sh cleanup --pr 123 --dry-run

Dry-run mode prints the Kubernetes manifests and guardrails that would be applied without executing kubectl or helm commands.

Observability & Alerts#

  • Preview namespaces export metrics with label preview_namespace = true. Grafana dashboards highlight resource usage and impending TTL expiration.
  • Orphan detection alerts fire if a namespace exceeds the TTL by >2 hours.
  • Cost guardrail breaches (more than 10 namespaces or quota exhaustion) open PagerDuty warnings for the platform team.

Security Considerations#

  • Secrets are sourced from Vault via the external-secrets integration. Preview namespaces receive dedicated, short-lived ExternalSecret manifests with branch-scoped credentials.
  • Service accounts inherit restricted RBAC limiting access to preview resources only.
  • Ingress routes require authentication via the same OIDC provider as staging, ensuring public pull requests do not expose protected endpoints.

Cleanup Guarantees#

  • The cleanup command deletes namespaces, persistent volumes, and any leftover cluster-scoped resources labeled with lilith.ai/pr.
  • On failure, the script emits actionable logs and exits non-zero so the workflow can notify maintainers.
  • The nightly cleanup job ignores namespaces with the annotation preview.lilith.ai/preserve=true for approved long-lived demos.