Eve library

OSHUN Studio Webhooks and External Automation Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Platform SRE + OSHUN Data/Analytics Engineering + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0047-oshun-studio-webhooks-and-external-automation.md
  • docs/releases/v1/specs/studio-webhooks-and-external-automation-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-webhooks-and-external-automation-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioWebhooksExternalAutomationWorkspace.tsx
  • apps/oshun/web/src/analytics/studioWebhooksExternalAutomationTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio webhook and external automation workflows.

This document is the execution artifact for OST-00280.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Webhooks/automation workspace route OSHUN Web Platform Web workflow UX orchestration + route lifecycle
Subscription lifecycle and contract registry OSHUN BFF + Security Studio Architecture schema/version governance + owner attribution
Signed delivery and retry orchestration Platform SRE + BFF Domain operations retry/backoff/idempotency controls + receiver health
Signature verification and secret lifecycle Security Engineering Platform SRE key rotation/revocation + compatibility windows
Sensitive automation policy and approvals Security + Compliance Program Operations role/tier controls + CAB governance
Telemetry and governance audit continuity Data Platform + Security Platform SRE analytics sinks + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Webhook/automation flow success rate >= 99.0% < 98.0% over 15 min
Subscription contract integrity >= 99.5% < 99.0% over 15 min
Dispatch/retry/recovery flow p95 <= 2500ms > 3500ms over 15 min
Signature-verification continuity >= 99.5% < 99.0% over 15 min
Policy-tier/CAB enforcement correctness >= 99.5% < 99.0% over 15 min
Terminal-failure recovery closure determinism >= 99.5% < 99.0% over 15 min
Telemetry/audit event delivery continuity >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Subscription Contract Publish Failures#

Trigger: repeated WEBHOOK_SUBSCRIPTION_CONTRACT_FAILED outcomes.

Actions:

  1. Validate subscription identity (subscriptionId, destinationUrl, sourceDomain, eventTypes, secretVersion).
  2. Inspect schema compatibility and owner-attribution drift.
  3. Re-run contract publish in preview with diagnostics enabled.
  4. Block sensitive automation actions until contract integrity recovers.

3.2 Signed Dispatch Failures#

Trigger: repeated WEBHOOK_SIGNED_DISPATCH_FAILED outcomes.

Actions:

  1. Validate event envelope (eventId, eventType, requestId, idempotencyKey, signature metadata).
  2. Inspect receiver status, response-code patterns, and latency spikes.
  3. Re-run signed dispatch against known-good receiver health.
  4. Escalate to Platform SRE on sustained dispatch error/latency regressions.

3.3 Retry Budget and Terminal Failure Escalation#

Trigger: repeated WEBHOOK_RETRY_RECOVERY_FAILED or exhausted retry budgets.

Actions:

  1. Validate retry preconditions (retryCount, maxRetries, backoff parameters, terminal-state rules).
  2. Inspect idempotency continuity across retry attempts.
  3. Re-run bounded retry under incident commander supervision.
  4. Open/maintain release hold while terminal failures remain unresolved.

3.4 Sensitive Automation Policy Decision Failures#

Trigger: repeated WEBHOOK_AUTOMATION_POLICY_DECISION_FAILED outcomes.

Actions:

  1. Validate role/tier/CAB prerequisites and reason codes.
  2. Inspect policy bundle drift and deny-path anomalies.
  3. Re-run decision with compliance auditor oversight.
  4. Freeze production-triggered automation while policy integrity is degraded.

3.5 Receiver Health Degradation#

Trigger: sustained 4xx/5xx responses, callback timeouts, or signature verification mismatches.

Actions:

  1. Validate receiver endpoint reachability and TLS posture.
  2. Confirm secret-version compatibility and rotation windows.
  3. Route failing subscriptions to controlled pause/degraded mode.
  4. Resume only after verified receiver recovery and successful replay.

3.6 Automation Recovery Closure Failures#

Trigger: repeated WEBHOOK_AUTOMATION_RECOVERY_FAILED outcomes.

Actions:

  1. Validate remediation-action payload and incident-state preconditions.
  2. Verify post-fix dispatch and policy checks before closure.
  3. Re-run recovery flow under incident command oversight.
  4. Prevent incident closure until deterministic revalidation is complete.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed studio_webhooks_external_automation_* telemetry events.

Actions:

  1. Validate analytics sink health and ingestion backlog.
  2. Enable degraded-observability notice in webhooks workspace.
  3. Replay buffered events where replay semantics are supported.
  4. Confirm SLO panel and audit-evidence continuity after recovery.

4. Release Checklist#

  • Validate /studio/webhooks-external-automation renders and executes required flows.
  • Validate optimistic, error, and retry behavior across all webhook and automation flows.
  • Validate permission-tier and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Webhooks/Automation Incident
Detected: {timestamp}
Impact: {subscription-contract | signed-dispatch | retry-recovery | policy-decision | incident-recovery}
Suspected cause: {contract drift | receiver degradation | policy drift | signature mismatch | telemetry degradation}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Webhooks/Automation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | receiver remediation | release hold}
Affected flows: {registerWebhookSubscription | dispatchSignedWebhookEvent | executeRetryRecoveryLifecycle | decideSensitiveAutomationAction | recoverAutomationHealth}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Webhooks/Automation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Webhooks/automation workspace UX #oshun-web Platform on-call
Subscription/delivery orchestration APIs #oshun-bff BFF lead
Receiver health and retry operations #oshun-platform-sre SRE lead
Signature/policy/CAB governance controls #oshun-security Security + Compliance
Telemetry and audit reporting continuity #oshun-data-platform Data platform lead

7. Acceptance Criteria (OST-00280)#

OST-00280 is complete only when:

  1. Ownership and decision authority are documented.
  2. Webhooks/automation SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.