Oshun V1 Rollback Plan (per substrate)

Gate owner: Release Captain + Engineering Lead. Approval: required signatures in docs/releases/v1/verification/rollback-plan/YYYY-MM-DD.md. Required for §33.2. Operational details: docs/reference/rollback-procedures.md.

OSHUN V1 ships across seven substrates. Each substrate has a named rollback procedure with measured RTO and RPO. Every substrate rollback is rehearsed in the 14 days preceding GA. The release captain commits the rehearsal evidence to docs/releases/v1/verification/rollback-plan/<date>/<substrate>.md before the go/no-go meeting opens.

1. Substrates and rollback strategies#

1.1 Customer web (apps/oshun/web)#

• Strategy — fast-forward to the prior commit on the deployment branch; CDN purges the new bundle hashes; service worker forces update by bumping version in the manifest.
• RTO — 5 min for the bundle; 10 min for service-worker forced refresh.
• RPO — 0 (web is stateless).
• Trigger — any §11 trigger in docs/launch/go-no-go.md OR any auto-rollback event from docs/launch/canary-analysis.md.
• Procedure —
  1. gh workflow run rollback-web.yml -F ref=<prior-sha>.
  2. Verify CDN cache invalidation completed.
  3. Force-update PWA manifest version.
  4. Watch dashboards-service-health for 30 min.

1.2 Customer mobile iOS#

• Strategy — halt phased rollout in App Store Connect; remove the in-progress build from sale; pre-staged minimum-version enforcement banner.
• RTO — halt-rollout: 10 min; minimum-version banner: 30 min; full revert (Apple-controlled): 1-3 d.
• RPO — 0 for non-installed users; up to D-1 backup for any installed user blocked by force-update.
• Trigger — same as §1.1.
• Procedure —
  1. App Store Connect → "Pause Release" on the active build.
  2. Push the minimum-version enforcement via remote config.
  3. Pre-staged restore-prior-binary requires Apple expedited review; pre-coordinated with App Review.

1.3 Customer mobile Android#

• Strategy — halt Play Console staged rollout; push minimum- version enforcement; if needed, promote prior internal-track build to production.
• RTO — halt: 5 min; minimum-version: 30 min; full revert: 1-4 h (Play Console push).
• RPO — 0 for non-installed users.
• Procedure —
  1. Play Console → "Halt rollout" on the active build.
  2. Push minimum-version enforcement.
  3. Promote prior production-track build if rollback required.

1.4 BFF (apps/oshun/bff, ECS)#

• Strategy — ECS task-definition revert via the aws ecs update-service flow in docs/reference/rollback-procedures.md §1.
• RTO — 5-10 min (ALB target-group draining).
• RPO — 0 (BFF is stateless).
• Trigger — same as §1.1.
• Procedure —
  1. aws ecs update-service --cluster oshun-production --service oshun-bff --task-definition oshun-bff:<previous>.
  2. Watch service events until desiredCount == runningCount for the previous revision.
  3. Verify dashboards-service-health.

1.5 Domain services (per service)#

Same ECS task-definition flow as BFF (§1.4). Each service maintains its own task-definition history. Rollback procedures named for the service:

RTO/RPO same as BFF.

1.6 Workers (background jobs)#

• Strategy — ECS task-definition revert (same as BFF) plus in-flight job drain.
• RTO — 10-30 min depending on job duration; queue depth must fall back to baseline before declaring "rolled back".
• RPO — depends on the job. Idempotent jobs: 0. Non-idempotent jobs: replay from queue with dedup keys; manual retry for jobs that exceeded the retention window.
• Procedure —
  1. aws ecs update-service to prior task-definition.
  2. Drain in-flight jobs (worker emits worker:draining event; ALB stops dispatching new work).
  3. Verify dashboards-queue-health queue-depth returns to baseline.

1.7 ML inference (libs/iris/* model surfaces)#

• Strategy — model-version pin revert via the model registry; shadow remains on the new model for analysis.
• RTO — 5 min.
• RPO — 0 (model surfaces are stateless).
• Trigger — §11 triggers plus any one of: grounded-answer rate < 0.90 for 15 min, drift KL > 0.10 vs. baseline, persona drift > 2 σ over 1 h.
• Procedure —
  1. pnpm nx run iris-model-registry:pin --model <name> --version <previous>.
  2. Verify dashboards-assistant grounded-answer rate restored.
  3. Open a model-rollback incident (docs/runbooks/model-workflow-rollback.md).

1.8 Database migrations#

• Strategy — every migration MUST ship with a paired down- migration; non-reversible migrations are forbidden unless the release captain explicitly approves with a recovery plan.
• RTO — depends on table size; published with the migration.
• RPO — 0 for additive migrations; up to 15 min for destructive migrations restoring from PITR (point-in-time-recovery).
• Procedure — see docs/reference/rollback-procedures.md §2.

1.9 Persona#

• Strategy — promote prior persona version via the persona registry.
• RTO — 5 min.
• RPO — 0 (persona registry retains every version).
• Trigger — persona drift > 2 σ in any 7-day window, OR safety incident class "persona-drift" opened.
• Procedure — see docs/runbooks/persona-rollback.md.

1.10 Content (Studio publishing)#

• Strategy — revert to prior content version; emit public change note via libs/shared/release-management/src/rollback-plan.ts.
• RTO — 5 min.
• RPO — 0 for additive content; up to 1 h for content edits (revert via the version graph).
• Procedure — see docs/runbooks/oshun-v1.md §rollback.

2. Rehearsal requirements#

Every substrate above is rehearsed in the 14 days preceding GA. Rehearsal evidence:

docs/releases/v1/verification/rollback-plan/<date>/<substrate>.md

- substrate: <web | ios | android | bff | service:<name> | workers
              | ml-inference | db-migration:<name> | persona
              | content>
- rehearsed-at: YYYY-MM-DDTHH:MM TZ
- rehearsed-by: <on-call name>
- procedure-ref: <link to §1 section>
- observed-RTO: <duration>
- observed-RPO: <duration>
- evidence: <link to trace / dashboard screenshot / commit>
- defects: (any deviations from the documented procedure)

Failing a rehearsal blocks GA until the procedure is fixed and the rehearsal repeated.

3. Approval#

Captured at docs/releases/v1/verification/rollback-plan/YYYY-MM-DD.md:

# Rollback Plan Approval
date: YYYY-MM-DD
release tag: vX.Y.Z

per-substrate rehearsal evidence:
- web: <link>
- mobile-ios: <link>
- mobile-android: <link>
- bff: <link>
- services:
  - tara-api: <link>
  - arete-api: <link>
  - ... (every service)
- workers: <link>
- ml-inference: <link>
- db-migrations:
  - <migration name>: <link>
- persona: <link>
- content: <link>

approvers:
- Release Captain: <name> APPROVE @ ts
- Engineering Lead: <name> APPROVE @ ts
- Security Lead (for migrations + ML inference): <name> APPROVE @ ts
- Privacy Lead (for migrations + persona): <name> APPROVE @ ts

4. Cross-references#

• Operational rollback procedures: docs/reference/rollback-procedures.md.
• Persona rollback runbook: docs/runbooks/persona-rollback.md.
• Model/workflow rollback runbook: docs/runbooks/model-workflow-rollback.md.
• Shell-outage runbook: docs/runbooks/shell-outage.md.
• Provider-failover runbook: docs/runbooks/provider-failover.md.
• Release-management rollback planner: libs/shared/release-management/src/rollback-plan.ts.
• Platform rollback orchestration: libs/oshun/platform-foundations/src/rollback/.
• Rollout plan: docs/launch/rollout-plan.md.
• Canary analysis: docs/launch/canary-analysis.md.
• Go/no-go: docs/launch/go-no-go.md.