Eve library

OSHUN Studio Notification Center Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Security + Program Operations + OSHUN Web + OSHUN BFF
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0030-oshun-studio-notification-center.md
  • docs/releases/v1/specs/studio-notification-center-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-notification-center-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioNotificationCenterWorkspace.tsx
  • apps/oshun/web/src/analytics/studioNotificationCenterTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio notification-center governance workflows.

This document is the execution artifact for OST-00144.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Notification-center route OSHUN Web Platform Web route + analytics contracts
Notification ingestion + dedupe engine OSHUN BFF Data Platform envelope schema + dedupe pipelines
Triage + assignment operations Program Operations Domain Integrators assignment workflow services
Critical escalation workflows OSHUN Security Incident Response on-call routing + escalation channels
Audit/compliance evidence stream Data Platform + Compliance Security audit event ingestion + retention controls

2. Core SLOs#

Metric Target Alert Threshold
Notification-center flow success rate >= 99.0% < 98.0% over 15 min
Critical alert escalation timeliness <= 5 min > 8 min over 10 min
Deduplication correctness 100% deterministic any incorrect merge/collision
Notification-center flow p95 latency <= 2200ms > 3200ms over 15 min
Notification audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Feed Triage and Prioritization Failures#

Trigger: repeated NOTIFICATION_FEED_TRIAGE_FAILED outcomes.

Actions:

  1. Validate severity classification and priority scoring contracts.
  2. Inspect ordering pipeline and dedupe interaction behavior.
  3. Re-run triage flow in preview with tracing enabled.
  4. Block notification-policy promotion until deterministic ordering recovers.

3.2 Acknowledge and Assign Failures#

Trigger: repeated NOTIFICATION_ACK_ASSIGN_FAILED outcomes.

Actions:

  1. Validate assignment transition schema and assignee identity mapping.
  2. Inspect ownership workflow service latency and queue health.
  3. Re-run assignment flow with deterministic actor context.
  4. Escalate to program operations if assignment backlog exceeds threshold.

3.3 Resolve and Close Failures#

Trigger: repeated NOTIFICATION_RESOLVE_CLOSE_FAILED outcomes.

Actions:

  1. Validate closure-evidence requirements and policy-state transitions.
  2. Inspect remediation evidence references for stale/invalid payloads.
  3. Re-run resolution workflow after evidence integrity checks.
  4. Freeze resolution promotion if unresolved critical dependencies remain.

3.4 Critical Escalation Failures#

Trigger: repeated NOTIFICATION_CRITICAL_ESCALATION_FAILED outcomes.

Actions:

  1. Validate escalation channel routing and on-call destination health.
  2. Inspect policy gates for release-channel and role allowlists.
  3. Re-run escalation in controlled mode with security reviewer oversight.
  4. Route fallback escalation to incident command if channel delivery fails.

3.5 Deduplication Integrity Failures#

Trigger: repeated NOTIFICATION_DEDUPLICATION_REVIEW_FAILED outcomes.

Actions:

  1. Validate dedupe keys and collision-protection contracts.
  2. Inspect merged-notification context for incident mismatch.
  3. Re-run dedupe review with collision diagnostics enabled.
  4. Roll back dedupe rule changes that violate signal integrity.

3.6 Policy Denial Spike#

Trigger: unexpected increase in ROLE_POLICY_DENIED or production-policy denials.

Actions:

  1. Compare flow allowlists with active entitlement policy bundles.
  2. Inspect recent role or policy-release changes for drift.
  3. Apply policy hotfix or roll back drifted contract version.
  4. Confirm deny-rate normalization after mitigation.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed notification-center telemetry/audit events.

Actions:

  1. Validate analytics sink health and ingestion queue depth.
  2. Enable degraded-observability warning in workspace operations.
  3. Replay buffered notification lifecycle events.
  4. Confirm event completeness and SLO recovery.

4. Release Checklist#

  • Validate /studio/notification-center renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy allow/deny and production escalation enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Notification Center Incident
Detected: {timestamp}
Impact: {triage | assignment | resolution | escalation | dedupe impact}
Suspected cause: {policy drift | routing issue | dedupe contract regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Notification Center Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {feedTriagePrioritization | acknowledgeAssignWorkflow | resolveCloseLoop | criticalEscalationHandling | deduplicationIntegrityReview}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Notification Center Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and workspace #oshun-web Platform on-call
Notification ingestion + dedupe #oshun-bff BFF lead
Program operations and assignment governance #oshun-program-ops Program operations lead
Critical escalation + policy governance #oshun-security Security lead
Telemetry and compliance evidence #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00144)#

OST-00144 is complete only when:

  1. Ownership and decision authority are documented.
  2. Notification-center SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.