# OSHUN Studio Notification Center Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Security + Program Operations + OSHUN
Web + OSHUN BFF  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0030-oshun-studio-notification-center.md`
- `docs/releases/v1/specs/studio-notification-center-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-notification-center-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioNotificationCenterWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioNotificationCenterTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio notification-center governance workflows.

This document is the execution artifact for `OST-00144`.

## 1. Ownership Model

| Surface                                | Primary Owner              | Backup Owner       | Dependencies                               |
| -------------------------------------- | -------------------------- | ------------------ | ------------------------------------------ |
| Notification-center route              | OSHUN Web                  | Platform Web       | route + analytics contracts                |
| Notification ingestion + dedupe engine | OSHUN BFF                  | Data Platform      | envelope schema + dedupe pipelines         |
| Triage + assignment operations         | Program Operations         | Domain Integrators | assignment workflow services               |
| Critical escalation workflows          | OSHUN Security             | Incident Response  | on-call routing + escalation channels      |
| Audit/compliance evidence stream       | Data Platform + Compliance | Security           | audit event ingestion + retention controls |

## 2. Core SLOs

| Metric                                | Target             | Alert Threshold               |
| ------------------------------------- | ------------------ | ----------------------------- |
| Notification-center flow success rate | >= 99.0%           | < 98.0% over 15 min           |
| Critical alert escalation timeliness  | <= 5 min           | > 8 min over 10 min           |
| Deduplication correctness             | 100% deterministic | any incorrect merge/collision |
| Notification-center flow p95 latency  | <= 2200ms          | > 3200ms over 15 min          |
| Notification audit event completeness | >= 99.5%           | < 99.0% over 15 min           |

## 3. Operational Playbooks

### 3.1 Feed Triage and Prioritization Failures

Trigger: repeated `NOTIFICATION_FEED_TRIAGE_FAILED` outcomes.

Actions:

1. Validate severity classification and priority scoring contracts.
2. Inspect ordering pipeline and dedupe interaction behavior.
3. Re-run triage flow in preview with tracing enabled.
4. Block notification-policy promotion until deterministic ordering recovers.

### 3.2 Acknowledge and Assign Failures

Trigger: repeated `NOTIFICATION_ACK_ASSIGN_FAILED` outcomes.

Actions:

1. Validate assignment transition schema and assignee identity mapping.
2. Inspect ownership workflow service latency and queue health.
3. Re-run assignment flow with deterministic actor context.
4. Escalate to program operations if assignment backlog exceeds threshold.

### 3.3 Resolve and Close Failures

Trigger: repeated `NOTIFICATION_RESOLVE_CLOSE_FAILED` outcomes.

Actions:

1. Validate closure-evidence requirements and policy-state transitions.
2. Inspect remediation evidence references for stale/invalid payloads.
3. Re-run resolution workflow after evidence integrity checks.
4. Freeze resolution promotion if unresolved critical dependencies remain.

### 3.4 Critical Escalation Failures

Trigger: repeated `NOTIFICATION_CRITICAL_ESCALATION_FAILED` outcomes.

Actions:

1. Validate escalation channel routing and on-call destination health.
2. Inspect policy gates for release-channel and role allowlists.
3. Re-run escalation in controlled mode with security reviewer oversight.
4. Route fallback escalation to incident command if channel delivery fails.

### 3.5 Deduplication Integrity Failures

Trigger: repeated `NOTIFICATION_DEDUPLICATION_REVIEW_FAILED` outcomes.

Actions:

1. Validate dedupe keys and collision-protection contracts.
2. Inspect merged-notification context for incident mismatch.
3. Re-run dedupe review with collision diagnostics enabled.
4. Roll back dedupe rule changes that violate signal integrity.

### 3.6 Policy Denial Spike

Trigger: unexpected increase in `ROLE_POLICY_DENIED` or production-policy
denials.

Actions:

1. Compare flow allowlists with active entitlement policy bundles.
2. Inspect recent role or policy-release changes for drift.
3. Apply policy hotfix or roll back drifted contract version.
4. Confirm deny-rate normalization after mitigation.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed notification-center telemetry/audit events.

Actions:

1. Validate analytics sink health and ingestion queue depth.
2. Enable degraded-observability warning in workspace operations.
3. Replay buffered notification lifecycle events.
4. Confirm event completeness and SLO recovery.

## 4. Release Checklist

- [ ] Validate `/studio/notification-center` renders and executes required
      flows.
- [ ] Validate optimistic, error, and retry behavior for each flow.
- [ ] Validate role-policy allow/deny and production escalation enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Notification Center Incident
Detected: {timestamp}
Impact: {triage | assignment | resolution | escalation | dedupe impact}
Suspected cause: {policy drift | routing issue | dedupe contract regression}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Notification Center Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {feedTriagePrioritization | acknowledgeAssignWorkflow | resolveCloseLoop | criticalEscalationHandling | deduplicationIntegrityReview}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Notification Center Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                         | Primary Channel        | Escalate To             |
| -------------------------------------------- | ---------------------- | ----------------------- |
| Web runtime and workspace                    | `#oshun-web`           | Platform on-call        |
| Notification ingestion + dedupe              | `#oshun-bff`           | BFF lead                |
| Program operations and assignment governance | `#oshun-program-ops`   | Program operations lead |
| Critical escalation + policy governance      | `#oshun-security`      | Security lead           |
| Telemetry and compliance evidence            | `#oshun-data-platform` | Data platform/SRE lead  |

## 7. Acceptance Criteria (OST-00144)

`OST-00144` is complete only when:

1. Ownership and decision authority are documented.
2. Notification-center SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
