# Canary Analysis Automation — V1 Launch Gate

> Gate owner: SRE Lead. Required for §33.2. Verifies the canary-gate decision
> pipeline is enabled, exercised, and wired to per-substrate rollout-promotion
> automation.

The canary analysis pipeline auto-decides **promote / hold / abort** for every
rollout stage. Decisions follow the metric-comparison logic in
`libs/oshun/search-discovery/src/experiments/canary.ts` and apply the
substrate-specific thresholds in `docs/launch/rollout-plan.md` §3.

## 1. Pipeline architecture

```
[ rollout stage start ]
        │
        ▼
  registerCanaryGate({ experimentId, cohortIds, minDwellSeconds,
                       minSampleSize, metrics, activatedAtUnixSeconds })
        │
        ▼
  every 60 s during dwell window: collectReadings(cohortIds, metrics)
        │
        ▼
  evaluateCanary({ gate, readings, nowUnixSeconds })
        │
        ▼
  ┌─ promote ─→ applyCanaryToExperiment + emit rollout-promotion event
  │             ↳ rollout controller advances substrate to next stage
  ├─ hold (dwell)        ─→ wait, re-collect
  ├─ hold (sample-size)  ─→ wait, re-collect (alert if > 2× dwell window)
  └─ abort ─→ rollback substrate to prior stage
            ↳ freeze further promotions
            ↳ page launch war-room (sev1)
            ↳ commit decision evidence
```

## 2. Code references

| Concern                             | Module                                                             |
| ----------------------------------- | ------------------------------------------------------------------ |
| Canary gate construction + decision | `libs/oshun/search-discovery/src/experiments/canary.ts`            |
| Underlying A/B framework            | `libs/oshun/search-discovery/src/experiments/ab-framework.ts`      |
| Per-domain canary benchmarks        | `libs/isis/3d-quality-gates/src/delivery/canary-benchmark-runs.ts` |
| Rollback orchestration              | `libs/oshun/platform-foundations/src/rollback/`                    |
| Rollback planner                    | `libs/shared/release-management/src/rollback-plan.ts`              |
| Alert routes                        | `libs/oshun/analytics/src/incident-ownership-manifest.ts`          |
| Dashboards                          | `libs/oshun/analytics/src/dashboards-*.ts`                         |

## 3. Metric registration (per substrate)

Each substrate registers its own canary gate with metrics from §3 of
`docs/launch/rollout-plan.md`. Sample registration for the BFF substrate
(illustrative; concrete configuration lives in the rollout controller config):

```ts
import { buildCanaryGate } from '@oshun/search-discovery/experiments/canary';

const bffStage1Gate = buildCanaryGate({
  experimentId: 'rollout:bff:v1:stage-1',
  cohortIds: ['canary'],
  minDwellSeconds: 30 * 60, // 30 minutes
  minSampleSize: 2_000,
  activatedAtUnixSeconds: stageStart,
  metrics: [
    {
      metricId: 'api_p95_latency_ms',
      direction: 'lower-is-better',
      mdeRatio: 0.05,
    },
    { metricId: 'http_5xx_rate', direction: 'lower-is-better', mdeRatio: 0.1 },
    {
      metricId: 'grounded_answer_rate',
      direction: 'higher-is-better',
      mdeRatio: 0.05,
    },
    {
      metricId: 'crisis_safe_route_rate',
      direction: 'higher-is-better',
      mdeRatio: 0.0,
    },
    {
      metricId: 'persona_tone_drift_sigma',
      direction: 'lower-is-better',
      mdeRatio: 0.0,
    },
    // ... see docs/launch/rollout-plan.md §3 for the full list
  ],
});
```

The MDE ratio is the maximum tolerated relative regression vs. control. A
`mdeRatio: 0.00` metric (e.g., crisis-flow safe-route hit rate) means **any**
regression aborts the stage.

## 4. Per-substrate registration

| Substrate              | Gate id pattern                     | Default cohort                                    | Default dwell | Default sample   |
| ---------------------- | ----------------------------------- | ------------------------------------------------- | ------------- | ---------------- |
| Web                    | `rollout:web:v1:stage-<n>`          | `canary` for stage 1; `geo-us-west-2` for stage 3 | 30-60 min     | 5 000 sessions   |
| Mobile iOS             | `rollout:ios:v1:stage-<n>`          | TestFlight internal                               | 24 h          | 500 sessions     |
| Mobile Android         | `rollout:android:v1:stage-<n>`      | Play internal track                               | 24 h          | 500 sessions     |
| BFF                    | `rollout:bff:v1:stage-<n>`          | `canary`                                          | 30 min        | 2 000 requests   |
| Services (per service) | `rollout:svc:<name>:v1:stage-<n>`   | `canary`                                          | 30 min        | 2 000 requests   |
| Workers                | `rollout:wkr:v1:stage-<n>`          | one pool, one AZ                                  | 30 min        | 500 jobs         |
| ML inference           | `rollout:ml:<surface>:v1:stage-<n>` | `shadow` then `canary`                            | 12 h          | 5 000 inferences |

## 5. Auto-promotion + auto-abort

- A `promote` decision emits a rollout-promotion event that the rollout
  controller consumes to advance the substrate to the next stage **only when**
  the stage table in `docs/launch/rollout-plan.md` §1 marks the transition as
  auto-promotable.
- A `hold (dwell)` decision causes the controller to wait and re-poll.
- A `hold (sample-size)` decision causes the controller to wait until 2× the
  dwell window; after that, the SRE lead is paged.
- An `abort` decision:
  1. Reverts the substrate to the prior stage via the per-substrate procedure in
     `docs/launch/rollback-plan.md` §1.
  2. Sets the rollout controller flag `frozen: true` so further auto-promotion
     is blocked until the release captain logs `freeze-lift`.
  3. Pages the launch war-room (sev1 via `incident-ownership-manifest.ts`).
  4. Commits decision evidence to
     `docs/releases/v1/verification/canary-analysis/<substrate>/<stage>/<ts>.md`.

## 6. End-to-end exercise

In the 14 days preceding GA, the SRE lead exercises the pipeline:

- **Shadow run** — fire a synthetic metric regression on a test substrate;
  confirm `abort` decision + rollback executed within 90 s.
- **Promote run** — fire clean metrics; confirm `promote` decision +
  rollout-promotion event observed by controller.
- **Sample-size run** — fire below-minSampleSize readings; confirm `hold` with
  reason `sample-size` and page after 2× dwell window.

Evidence committed at
`docs/releases/v1/verification/canary-analysis/exercise-<date>.md`.

## 7. Test coverage

Unit-level test coverage of the canary-gate evaluation logic lives at
`libs/oshun/search-discovery/src/experiments/canary.test.ts` (asserts `promote`
/ `hold` / `abort` for representative inputs). The dwell window, sample-size,
and per-metric MDE assertions cover the three decision branches.
Integration-level coverage of the rollout-controller wiring is exercised
end-to-end in §6 (shadow / promote / sample-size exercises) with evidence
committed under
`docs/releases/v1/verification/canary-analysis/exercise-<date>.md`.

## 8. Live verification checklist

Required for §33.2 signoff:

- [ ] `canary.ts` test suite green on the release tag.
- [ ] Per-substrate gates registered in the rollout-controller config for every
      substrate in §4.
- [ ] Shadow / promote / sample-size end-to-end exercise (§6) completed and
      evidence committed.
- [ ] Alert wiring verified for `canary_rollback` sev1 route.
- [ ] War-room dashboard renders rollout-progress per-substrate gauge backed by
      the canary-gate decisions.

## 9. Cross-references

- Canary code: `libs/oshun/search-discovery/src/experiments/canary.ts`.
- Rollout plan: `docs/launch/rollout-plan.md`.
- Rollback plan: `docs/launch/rollback-plan.md`.
- Post-deploy monitoring: `docs/launch/post-deploy-monitoring.md`.
- Go/no-go: `docs/launch/go-no-go.md`.