Disciplines · Training

Training — Launch Support

launch-support shift, plus designated launch captains.

10sections3 minread
Markdown source
On this page

Launch-support staff run the canary-monitoring rota, the rollback-trigger recognition shift, the handoff to the steady-state on-call rotation, and the launch-day war room. Active during the canary, GA, and post-GA hyper-care windows.

Audience and prerequisites#

  • Audience: reliability + platform-engineering staff rotated onto the launch-support shift, plus designated launch captains.
  • Prerequisites: reliability foundations, on-call onboarding for the primary substrate, runbook authorship awareness, audit-scope familiarity.
  • Refresh cadence: per-launch; refresh annually for staff retained on the steady-state rota.

Learning objectives#

By certification, launch-support staff can independently:

  1. Operate the canary cohort schedule (1% → 5% → 25% → 50% → 100%) with per-stage monitoring windows.
  2. Read the launch dashboards (dashboards-service-health.ts, dashboards-customer-kpi.ts, dashboards-satisfaction.ts, dashboards-queue-health.ts, dashboards-assistant.ts, dashboards-nisaba.ts) and identify a rollback trigger within the documented SLO.
  3. Trigger the rollback chain: notify, execute, verify, document.
  4. Handoff a healthy GA launch to the steady-state on-call rotation with the audit-grade handoff packet.
  5. Operate the launch-day war room: bridge, situation room, escalation queue, comms cadence.
  6. Coordinate hyper-care: triage backlog, customer-signal escalation, product remediation.

Curriculum modules#

# Module Duration Format Assessment
1 Canary schedule and monitoring windows 90 min hands-on with canary tooling 1 full canary cycle drill (5 stages)
2 Rollback trigger recognition 120 min scenario rehearsal 6-scenario rollback-trigger test (≥ 5/6 detected)
3 Rollback execution and audit 90 min hands-on against rollback hooks 3-rollback drill
4 Handoff to steady-state on-call 60 min guided handoff with packet authoring 1 handoff packet pass rubric
5 War-room operations 90 min scenario rehearsal 1 full war-room drill
6 Hyper-care triage 60 min hands-on with backlog tooling 20-item triage drill
7 Communications cadence with product, T&S, support 60 min scenario rehearsal 3-cadence drill
8 Post-launch postmortem authorship 60 min guided + peer review 1 postmortem draft pass rubric

Canary monitoring protocol#

For each stage (1%, 5%, 25%, 50%, 100%):

  1. Confirm the canary advanced via release.canary.advanced event.
  2. Open the monitoring window per the rollout plan (typically 30–120 min per stage; longer for sensitive stages).
  3. Watch the launch dashboards: error ratio, latency p95/p99, conversion-success, customer-KPI deltas, support-ticket spike, T&S queue spike, assistant-quality deltas.
  4. Watch the rollback triggers list (see below).
  5. At the end of the window, run a structured advance-or-hold decision with the launch captain.

Rollback triggers#

A rollback fires if any of these is observed during a canary stage:

  • Unresolved critical security issue tied to the release.
  • P0 data-loss signal (audit pipeline drop, persistence error class).
  • Cross-tenant leakage suspected.
  • Failed grounded-answer gate (confidence floor breached, retraction served).
  • Elevated crisis-flow defect (crisis routing misses or aged-out).
  • Sustained SLO breach (error ratio, latency, conversion).
  • Customer-KPI regression beyond the documented bound.
  • Cost overshoot beyond the per-stage envelope.
  • T&S queue spike beyond the SLA window.

Handoff protocol#

When the launch advances to GA and hyper-care closes:

  1. Author the handoff packet: scope, residual risks, observed regressions, mitigations in place, owners, ETAs.
  2. Brief the steady-state on-call rotation: walk through the packet, the runbooks invoked, the dashboards still tagged, and the open tickets.
  3. Capture the handoff as an audit event; rotate the launch-support shift down.

Scenario rehearsal#

  1. Canary advance hold: customer-KPI deltas are within bounds but the satisfaction dashboard ticks down; hold the advance and investigate.
  2. Rollback under safety trigger: a model promotion fires a persona-policy-violation alert mid-canary; execute rollback, communicate, follow model-workflow-rollback.md.
  3. War-room cross-domain incident: a shared dependency fails mid-canary; coordinate the war room, route to the right runbook, maintain comms cadence.
  4. Handoff with residual risk: GA hits, but a non-blocking regression remains under remediation; author the handoff packet that scopes the residual risk for steady-state on-call.

Certification criteria#

  • All 8 modules complete with passing assessment.
  • All 4 scenarios passed.
  • Launch-support staff has shadowed 1 prior canary cycle.
  • Reliability lead and release captain sign-off.

Tabletop drills#

  • Per-launch: pre-launch tabletop covering rollback triggers and war- room comms.
  • Post-launch: postmortem drill with product and reliability.

Owner#

Reliability lead and release captain co-own this training. Updates require both sign-offs.