Eve library

OSHUN Studio Commenting and Annotation System Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0034-oshun-studio-commenting-and-annotation-system.md
  • docs/releases/v1/specs/studio-commenting-and-annotation-system-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-commenting-and-annotation-system-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioCommentingAnnotationSystemWorkspace.tsx
  • apps/oshun/web/src/analytics/studioCommentingAnnotationSystemTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio commenting and annotation workflows.

This document is the execution artifact for OST-00176.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Commenting/annotation workspace route OSHUN Web Platform Web route contracts + UI orchestration
Annotation/thread persistence services OSHUN BFF Platform SRE thread persistence + lifecycle state transitions
Target binding and domain adapters Domain Integrators Studio Architecture entity/frame/token adapters across domain boundaries
Moderation and policy decisioning Security + Compliance Program Operations policy bundles + moderation workflows
Annotation telemetry and governance export Data Platform + Security Platform SRE analytics ingestion + retention controls

2. Core SLOs#

Metric Target Alert Threshold
Annotation flow success rate >= 99.0% < 98.0% over 15 min
Thread lifecycle transition reliability >= 99.5% < 99.0% over 15 min
Commenting/annotation flow p95 latency <= 2200ms > 3200ms over 15 min
Thread ordering determinism 100% deterministic any ordering divergence
Moderation decision completeness >= 99.5% < 99.0% over 15 min
Annotation audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Create Annotation Failures#

Trigger: repeated ANNOTATION_CREATE_FAILED outcomes.

Actions:

  1. Validate envelope requirements (annotationId, threadId, targetRef, actorId, content, occurredAt) are present and correctly typed.
  2. Inspect target-binding adapter output for missing/invalid target references.
  3. Re-run creation flow in preview with diagnostics enabled.
  4. Block promotion until envelope validation stability normalizes.

3.2 Reply Thread Expansion Failures#

Trigger: repeated ANNOTATION_REPLY_FAILED outcomes.

Actions:

  1. Validate parent-thread continuity and deterministic ordering semantics.
  2. Inspect reply payload for invalid parent linkage or thread mismatch.
  3. Re-run reply flow with ordering diagnostics enabled.
  4. Freeze release when thread continuity invariants fail.

3.3 Resolve Lifecycle Failures#

Trigger: repeated ANNOTATION_RESOLVE_FAILED outcomes.

Actions:

  1. Validate resolution rationale and owner attribution metadata.
  2. Inspect lifecycle transition logs for illegal state transitions.
  3. Re-run resolve flow with lifecycle trace logging enabled.
  4. Escalate to Studio architecture if resolution contract drift persists.

3.4 Reopen Lifecycle Failures#

Trigger: repeated ANNOTATION_REOPEN_FAILED outcomes.

Actions:

  1. Validate resolved-state prerequisites and reopen reason metadata.
  2. Inspect reopen history and closure evidence lineage.
  3. Re-run reopen flow with history-integrity checks enabled.
  4. Lock rollout when reopen integrity checks fail repeatedly.

3.5 Moderation Governance Failures#

Trigger: repeated ANNOTATION_MODERATION_FAILED outcomes.

Actions:

  1. Validate moderation action, policy ID, and rationale metadata.
  2. Inspect moderation decision pipeline and policy evaluation traces.
  3. Re-run moderation flow with security reviewer oversight.
  4. Block production moderation rollout until policy evidence checks pass.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, DOMAIN_VISIBILITY_POLICY_DENIED, REOPEN_ESCALATION_POLICY_DENIED, or PRODUCTION_MODERATION_POLICY_DENIED.

Actions:

  1. Compare flow role allowlists with active RBAC/policy bundles.
  2. Inspect entitlement/policy release diffs for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and operator impact resolution.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed commenting/annotation telemetry/audit events.

Actions:

  1. Validate analytics sink health and ingestion backlog depth.
  2. Enable degraded-observability warning in workspace operations.
  3. Replay buffered events where supported.
  4. Confirm evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/commenting-annotation-system renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy and moderation governance enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Commenting/Annotation Incident
Detected: {timestamp}
Impact: {annotation-create | thread-reply | lifecycle-resolve | lifecycle-reopen | moderation-review}
Suspected cause: {policy drift | thread continuity failure | lifecycle transition regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Commenting/Annotation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {createContextBoundAnnotation | replyThreadExpansion | acknowledgeResolveAnnotation | reopenResolvedAnnotation | moderationGovernanceReview}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Commenting/Annotation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and annotation workspace #oshun-web Platform on-call
Annotation/thread persistence services #oshun-bff BFF lead
Lifecycle reliability and replay #oshun-platform-sre SRE lead
Moderation and policy governance #oshun-security Security lead
Audit/compliance evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00176)#

OST-00176 is complete only when:

  1. Ownership and decision authority are documented.
  2. Commenting/annotation SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.