Eve library

OSHUN Studio Presence and Cursor Systems Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0033-oshun-studio-presence-and-cursor-systems.md
  • docs/releases/v1/specs/studio-presence-and-cursor-systems-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-presence-and-cursor-systems-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioPresenceCursorSystemsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioPresenceCursorSystemsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio presence and cursor system workflows.

This document is the execution artifact for OST-00168.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Presence/cursor workspace route OSHUN Web Platform Web route contracts + UI orchestration
Heartbeat and liveness services OSHUN BFF Platform SRE session heartbeat and timeout pipelines
Cursor sequencing and progression BFF + Data Platform Studio Architecture deterministic sequencing and replay semantics
Replay and resume recovery pipeline Reliability + BFF Platform SRE replay cursor services + continuity checks
Presence/cursor governance evidence Security + Compliance Data Platform telemetry ingestion + retention controls

2. Core SLOs#

Metric Target Alert Threshold
Presence/cursor flow success rate >= 99.0% < 98.0% over 15 min
Presence join registration reliability >= 99.5% < 99.0% over 15 min
Presence/cursor flow p95 latency <= 2200ms > 3200ms over 15 min
Liveness transition accuracy >= 99.5% < 99.0% over 15 min
Cursor progression determinism 100% deterministic any non-deterministic progression
Presence/cursor audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Presence Session Join Failures#

Trigger: repeated PRESENCE_SESSION_JOIN_FAILED outcomes.

Actions:

  1. Validate actor/workspace/session context integrity.
  2. Inspect join registration pipeline and policy validation output.
  3. Re-run join flow in preview with diagnostics enabled.
  4. Block production promotion until registration reliability normalizes.

3.2 Liveness Transition Failures#

Trigger: repeated PRESENCE_LIVENESS_TRANSITION_FAILED outcomes.

Actions:

  1. Validate heartbeat cadence and timeout threshold configuration.
  2. Inspect active-idle-offline transition logs for drift.
  3. Re-run transition flow with controlled heartbeat profile.
  4. Escalate to reliability on sustained transition inaccuracy.

3.3 Cursor Progression Failures#

Trigger: repeated CURSOR_PROGRESSION_FAILED outcomes.

Actions:

  1. Validate cursor monotonicity and sequence continuity contracts.
  2. Inspect progression pipeline for skipped or regressed sequence entries.
  3. Re-run progression with deterministic replay safeguards.
  4. Freeze cursor-related rollout when determinism guarantees fail.

3.4 Replay and Resume Recovery Failures#

Trigger: repeated PRESENCE_REPLAY_RESUME_FAILED outcomes.

Actions:

  1. Validate replay cursor eligibility and retention window integrity.
  2. Inspect replay pipeline for ordering or gap detection failures.
  3. Re-run recovery with full snapshot fallback path.
  4. Trigger incident if reconnect divergence persists.

3.5 Governance Visibility Review Failures#

Trigger: repeated PRESENCE_GOVERNANCE_REVIEW_FAILED outcomes.

Actions:

  1. Validate visibility scope and policy redaction contracts.
  2. Inspect governance package completeness and retention tags.
  3. Re-run visibility review with security reviewer oversight.
  4. Block production visibility policy rollout until checks pass.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, HIGH_LATENCY_LIVENESS_POLICY_DENIED, or PRODUCTION_VISIBILITY_POLICY_DENIED.

Actions:

  1. Compare flow allowlists with active RBAC/policy bundles.
  2. Inspect entitlement release changes for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and operator impact resolution.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed presence/cursor telemetry/audit events.

Actions:

  1. Validate sink health and ingestion backlog depth.
  2. Enable degraded-observability warning in workspace.
  3. Replay buffered events where supported.
  4. Confirm SLO and evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/presence-cursor-systems renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy and high-latency governance enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Presence/Cursor Systems Incident
Detected: {timestamp}
Impact: {session-join | liveness-transition | cursor-progression | replay-recovery | governance-visibility}
Suspected cause: {policy drift | sequencing failure | replay continuity regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Presence/Cursor Systems Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {presenceSessionJoin | livenessTransitionOrchestration | cursorProgressionExecution | replayResumeRecovery | governanceVisibilityReview}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Presence/Cursor Systems Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and presence workspace #oshun-web Platform on-call
Presence heartbeat and session services #oshun-bff BFF lead
Cursor sequencing and replay reliability #oshun-platform-sre SRE lead
Policy visibility and security controls #oshun-security Security lead
Audit/compliance evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00168)#

OST-00168 is complete only when:

  1. Ownership and decision authority are documented.
  2. Presence/cursor SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.