Eve library

OSHUN Studio Activity and Change Feeds Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + Program Operations + OSHUN Security + OSHUN Web + OSHUN BFF
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0031-oshun-studio-activity-and-change-feeds.md
  • docs/releases/v1/specs/studio-activity-and-change-feeds-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-activity-and-change-feeds-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioActivityChangeFeedsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioActivityChangeFeedsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio activity/change-feed governance workflows.

This document is the execution artifact for OST-00152.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Activity/change-feed route OSHUN Web Platform Web route contracts + telemetry hooks
Timeline ingestion and ordering OSHUN BFF Data Platform event envelopes + cursor sequencing
Attribution verification operations OSHUN Security Program Operations identity mapping + policy contracts
Follow-up routing orchestration Program Operations Domain Integrators ownership routing services + queue pipelines
Compliance export packaging and evidence Compliance + Data Platform Security export bundle schema + audit event persistence

2. Core SLOs#

Metric Target Alert Threshold
Activity/change flow success rate >= 99.0% < 98.0% over 15 min
Timeline ordering determinism 100% deterministic any non-deterministic ordering
Attribution verification completeness >= 99.5% < 99.0% over 15 min
Activity/change flow p95 latency <= 2400ms > 3400ms over 15 min
Compliance export package completeness >= 99.5% < 99.0% over 30 min
Activity/change audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Unified Timeline Triage Failures#

Trigger: repeated ACTIVITY_TIMELINE_TRIAGE_FAILED outcomes.

Actions:

  1. Validate cross-domain severity and time-ordering contract inputs.
  2. Inspect cursor sequence integrity and replay-window boundaries.
  3. Re-run triage flow in preview with trace logging enabled.
  4. Freeze promotion of ordering policy changes until determinism is restored.

3.2 Actor Attribution Verification Failures#

Trigger: repeated ACTIVITY_ATTRIBUTION_VERIFICATION_FAILED outcomes.

Actions:

  1. Validate actor identity links and source-domain role mapping.
  2. Inspect attribution evidence records for stale or missing metadata.
  3. Re-run verification after identity reconciliation.
  4. Escalate to security if unresolved attribution drift persists.

3.3 Cursor Replay Reconstruction Failures#

Trigger: repeated ACTIVITY_CURSOR_REPLAY_FAILED outcomes.

Actions:

  1. Validate cursor chain continuity and replay-window semantics.
  2. Inspect replay dependency failures in ordering and storage services.
  3. Re-run cursor reconstruction in controlled mode.
  4. Roll back cursor contract changes if replay determinism degrades.

3.4 Follow-Up Routing Failures#

Trigger: repeated ACTIVITY_FOLLOW_UP_ROUTING_FAILED outcomes.

Actions:

  1. Validate owner-routing mappings and workflow target resolution.
  2. Inspect queue processing latency and assignment backlog.
  3. Re-run follow-up routing with deterministic owner context.
  4. Escalate to program operations if routing backlog exceeds threshold.

3.5 Compliance Export Packaging Failures#

Trigger: repeated ACTIVITY_COMPLIANCE_EXPORT_FAILED outcomes.

Actions:

  1. Validate export package schema and evidence attachment contracts.
  2. Verify attribution prerequisite state for selected change records.
  3. Re-run export packaging with compliance reviewer oversight.
  4. Block production export promotion until package validation stabilizes.

3.6 Policy Denial Spike#

Trigger: unexpected increase in ROLE_POLICY_DENIED, PRODUCTION_EXPORT_POLICY_DENIED, or ATTRIBUTION_VERIFICATION_REQUIRED.

Actions:

  1. Compare flow allowlists with active entitlement/policy bundles.
  2. Inspect recent role and release-policy changes for drift.
  3. Apply policy hotfix or roll back drifted policy revisions.
  4. Confirm denial-rate normalization after mitigation.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed activity/change telemetry or audit events.

Actions:

  1. Validate analytics sink health and event ingestion queue depth.
  2. Enable degraded-observability warning for operators.
  3. Replay buffered events where supported by pipelines.
  4. Confirm SLO panel data freshness and event completeness recovery.

4. Release Checklist#

  • Validate /studio/activity-change-feeds renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy and attribution prerequisite enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Activity/Change Feeds Incident
Detected: {timestamp}
Impact: {triage | attribution | replay | routing | export}
Suspected cause: {ordering drift | identity mapping issue | policy regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Activity/Change Feeds Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {unifiedTimelineTriage | actorAttributionVerification | cursorReplayReconstruction | followUpRoutingExecution | complianceExportPackaging}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Activity/Change Feeds Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and workspace #oshun-web Platform on-call
Activity ingestion and cursor sequencing #oshun-bff BFF lead
Follow-up routing and program operations #oshun-program-ops Program operations lead
Attribution and export policy governance #oshun-security Security lead
Telemetry and compliance evidence #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00152)#

OST-00152 is complete only when:

  1. Ownership and decision authority are documented.
  2. Activity/change-feed SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.