# OSHUN Studio Webhooks and External Automation Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security +
Platform SRE + OSHUN Data/Analytics Engineering + Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0047-oshun-studio-webhooks-and-external-automation.md`
- `docs/releases/v1/specs/studio-webhooks-and-external-automation-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-webhooks-and-external-automation-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioWebhooksExternalAutomationWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioWebhooksExternalAutomationTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio webhook and external automation workflows.

This document is the execution artifact for `OST-00280`.

## 1. Ownership Model

| Surface                                      | Primary Owner            | Backup Owner        | Dependencies                                         |
| -------------------------------------------- | ------------------------ | ------------------- | ---------------------------------------------------- |
| Webhooks/automation workspace route          | OSHUN Web                | Platform Web        | workflow UX orchestration + route lifecycle          |
| Subscription lifecycle and contract registry | OSHUN BFF + Security     | Studio Architecture | schema/version governance + owner attribution        |
| Signed delivery and retry orchestration      | Platform SRE + BFF       | Domain operations   | retry/backoff/idempotency controls + receiver health |
| Signature verification and secret lifecycle  | Security Engineering     | Platform SRE        | key rotation/revocation + compatibility windows      |
| Sensitive automation policy and approvals    | Security + Compliance    | Program Operations  | role/tier controls + CAB governance                  |
| Telemetry and governance audit continuity    | Data Platform + Security | Platform SRE        | analytics sinks + retention/export controls          |

## 2. Core SLOs

| Metric                                        | Target    | Alert Threshold      |
| --------------------------------------------- | --------- | -------------------- |
| Webhook/automation flow success rate          | >= 99.0%  | < 98.0% over 15 min  |
| Subscription contract integrity               | >= 99.5%  | < 99.0% over 15 min  |
| Dispatch/retry/recovery flow p95              | <= 2500ms | > 3500ms over 15 min |
| Signature-verification continuity             | >= 99.5%  | < 99.0% over 15 min  |
| Policy-tier/CAB enforcement correctness       | >= 99.5%  | < 99.0% over 15 min  |
| Terminal-failure recovery closure determinism | >= 99.5%  | < 99.0% over 15 min  |
| Telemetry/audit event delivery continuity     | >= 99.5%  | < 99.0% over 15 min  |

## 3. Operational Playbooks

### 3.1 Subscription Contract Publish Failures

Trigger: repeated `WEBHOOK_SUBSCRIPTION_CONTRACT_FAILED` outcomes.

Actions:

1. Validate subscription identity (`subscriptionId`, `destinationUrl`,
   `sourceDomain`, `eventTypes`, `secretVersion`).
2. Inspect schema compatibility and owner-attribution drift.
3. Re-run contract publish in preview with diagnostics enabled.
4. Block sensitive automation actions until contract integrity recovers.

### 3.2 Signed Dispatch Failures

Trigger: repeated `WEBHOOK_SIGNED_DISPATCH_FAILED` outcomes.

Actions:

1. Validate event envelope (`eventId`, `eventType`, `requestId`,
   `idempotencyKey`, signature metadata).
2. Inspect receiver status, response-code patterns, and latency spikes.
3. Re-run signed dispatch against known-good receiver health.
4. Escalate to Platform SRE on sustained dispatch error/latency regressions.

### 3.3 Retry Budget and Terminal Failure Escalation

Trigger: repeated `WEBHOOK_RETRY_RECOVERY_FAILED` or exhausted retry budgets.

Actions:

1. Validate retry preconditions (`retryCount`, `maxRetries`, backoff parameters,
   terminal-state rules).
2. Inspect idempotency continuity across retry attempts.
3. Re-run bounded retry under incident commander supervision.
4. Open/maintain release hold while terminal failures remain unresolved.

### 3.4 Sensitive Automation Policy Decision Failures

Trigger: repeated `WEBHOOK_AUTOMATION_POLICY_DECISION_FAILED` outcomes.

Actions:

1. Validate role/tier/CAB prerequisites and reason codes.
2. Inspect policy bundle drift and deny-path anomalies.
3. Re-run decision with compliance auditor oversight.
4. Freeze production-triggered automation while policy integrity is degraded.

### 3.5 Receiver Health Degradation

Trigger: sustained 4xx/5xx responses, callback timeouts, or signature
verification mismatches.

Actions:

1. Validate receiver endpoint reachability and TLS posture.
2. Confirm secret-version compatibility and rotation windows.
3. Route failing subscriptions to controlled pause/degraded mode.
4. Resume only after verified receiver recovery and successful replay.

### 3.6 Automation Recovery Closure Failures

Trigger: repeated `WEBHOOK_AUTOMATION_RECOVERY_FAILED` outcomes.

Actions:

1. Validate remediation-action payload and incident-state preconditions.
2. Verify post-fix dispatch and policy checks before closure.
3. Re-run recovery flow under incident command oversight.
4. Prevent incident closure until deterministic revalidation is complete.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed `studio_webhooks_external_automation_*` telemetry
events.

Actions:

1. Validate analytics sink health and ingestion backlog.
2. Enable degraded-observability notice in webhooks workspace.
3. Replay buffered events where replay semantics are supported.
4. Confirm SLO panel and audit-evidence continuity after recovery.

## 4. Release Checklist

- [ ] Validate `/studio/webhooks-external-automation` renders and executes
      required flows.
- [ ] Validate optimistic, error, and retry behavior across all webhook and
      automation flows.
- [ ] Validate permission-tier and production CAB policy enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility/performance evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```text
[P{severity}] Studio Webhooks/Automation Incident
Detected: {timestamp}
Impact: {subscription-contract | signed-dispatch | retry-recovery | policy-decision | incident-recovery}
Suspected cause: {contract drift | receiver degradation | policy drift | signature mismatch | telemetry degradation}
Owner: {on-call}
```

### Incident Update

```text
[P{severity}] Studio Webhooks/Automation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | receiver remediation | release hold}
Affected flows: {registerWebhookSubscription | dispatchSignedWebhookEvent | executeRetryRecoveryLifecycle | decideSensitiveAutomationAction | recoverAutomationHealth}
ETA: {time}
```

### Incident Resolved

```text
[P{severity}] Studio Webhooks/Automation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                     | Primary Channel        | Escalate To           |
| ---------------------------------------- | ---------------------- | --------------------- |
| Webhooks/automation workspace UX         | `#oshun-web`           | Platform on-call      |
| Subscription/delivery orchestration APIs | `#oshun-bff`           | BFF lead              |
| Receiver health and retry operations     | `#oshun-platform-sre`  | SRE lead              |
| Signature/policy/CAB governance controls | `#oshun-security`      | Security + Compliance |
| Telemetry and audit reporting continuity | `#oshun-data-platform` | Data platform lead    |

## 7. Acceptance Criteria (OST-00280)

`OST-00280` is complete only when:

1. Ownership and decision authority are documented.
2. Webhooks/automation SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
