Eve library

ADR-0002: Production Deployment Strategy (Canary + Auto-Rollback)

ADR

Revision 1

On this page

Status#

Accepted

Context#

We deploy a growing set of Node.js/Fastify microservices to Kubernetes. Previous deployments used blue-green without pre-promotion signal and lacked automated rollback based on live health during rollout. As service count and change velocity increase, we need a safer, incremental promotion path and a post-deploy guardrail to contain risk.

Decision#

We will:

  • Introduce a canary stage (10% traffic) for production releases using dedicated Helm values and canary hostnames.
  • Hold a 5-minute soak window with health sampling across all targeted services before full promotion.
  • Require successful canary completion before the main production rollout job proceeds.
  • Add an automated rollback guard that continuously samples production health post-promotion; any regression triggers Helm rollback to the previous revision and fails the pipeline.
  • Keep blue-green as the underlying deployment primitive for fast cutover while layering canary gating and rollback triggers above it.

Consequences#

  • Safer releases: regressions detected in canary or immediate post-promotion are rolled back automatically.
  • Longer pipeline runtime due to soak/validation steps, but improved reliability for user-facing traffic.
  • Requires maintaining canary-specific ingress/Helm values and ensuring observability endpoints remain fast and stable.
  • Future work: incorporate SLO/error-budget checks from observability backends to replace simple health sampling, and refine traffic weights per service risk profile.