Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security +
Platform SRE + OSHUN Data/Analytics Engineering + Program Operations
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0047-oshun-studio-webhooks-and-external-automation.mddocs/releases/v1/specs/studio-webhooks-and-external-automation-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-webhooks-and-external-automation-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioWebhooksExternalAutomationWorkspace.tsxapps/oshun/web/src/analytics/studioWebhooksExternalAutomationTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio webhook and external automation workflows.
This document is the execution artifact for OST-00280.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Webhooks/automation workspace route | OSHUN Web | Platform Web | workflow UX orchestration + route lifecycle |
| Subscription lifecycle and contract registry | OSHUN BFF + Security | Studio Architecture | schema/version governance + owner attribution |
| Signed delivery and retry orchestration | Platform SRE + BFF | Domain operations | retry/backoff/idempotency controls + receiver health |
| Signature verification and secret lifecycle | Security Engineering | Platform SRE | key rotation/revocation + compatibility windows |
| Sensitive automation policy and approvals | Security + Compliance | Program Operations | role/tier controls + CAB governance |
| Telemetry and governance audit continuity | Data Platform + Security | Platform SRE | analytics sinks + retention/export controls |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Webhook/automation flow success rate | >= 99.0% | < 98.0% over 15 min |
| Subscription contract integrity | >= 99.5% | < 99.0% over 15 min |
| Dispatch/retry/recovery flow p95 | <= 2500ms | > 3500ms over 15 min |
| Signature-verification continuity | >= 99.5% | < 99.0% over 15 min |
| Policy-tier/CAB enforcement correctness | >= 99.5% | < 99.0% over 15 min |
| Terminal-failure recovery closure determinism | >= 99.5% | < 99.0% over 15 min |
| Telemetry/audit event delivery continuity | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Subscription Contract Publish Failures#
Trigger: repeated WEBHOOK_SUBSCRIPTION_CONTRACT_FAILED outcomes.
Actions:
- Validate subscription identity (
subscriptionId,destinationUrl,sourceDomain,eventTypes,secretVersion). - Inspect schema compatibility and owner-attribution drift.
- Re-run contract publish in preview with diagnostics enabled.
- Block sensitive automation actions until contract integrity recovers.
3.2 Signed Dispatch Failures#
Trigger: repeated WEBHOOK_SIGNED_DISPATCH_FAILED outcomes.
Actions:
- Validate event envelope (
eventId,eventType,requestId,idempotencyKey, signature metadata). - Inspect receiver status, response-code patterns, and latency spikes.
- Re-run signed dispatch against known-good receiver health.
- Escalate to Platform SRE on sustained dispatch error/latency regressions.
3.3 Retry Budget and Terminal Failure Escalation#
Trigger: repeated WEBHOOK_RETRY_RECOVERY_FAILED or exhausted retry budgets.
Actions:
- Validate retry preconditions (
retryCount,maxRetries, backoff parameters, terminal-state rules). - Inspect idempotency continuity across retry attempts.
- Re-run bounded retry under incident commander supervision.
- Open/maintain release hold while terminal failures remain unresolved.
3.4 Sensitive Automation Policy Decision Failures#
Trigger: repeated WEBHOOK_AUTOMATION_POLICY_DECISION_FAILED outcomes.
Actions:
- Validate role/tier/CAB prerequisites and reason codes.
- Inspect policy bundle drift and deny-path anomalies.
- Re-run decision with compliance auditor oversight.
- Freeze production-triggered automation while policy integrity is degraded.
3.5 Receiver Health Degradation#
Trigger: sustained 4xx/5xx responses, callback timeouts, or signature verification mismatches.
Actions:
- Validate receiver endpoint reachability and TLS posture.
- Confirm secret-version compatibility and rotation windows.
- Route failing subscriptions to controlled pause/degraded mode.
- Resume only after verified receiver recovery and successful replay.
3.6 Automation Recovery Closure Failures#
Trigger: repeated WEBHOOK_AUTOMATION_RECOVERY_FAILED outcomes.
Actions:
- Validate remediation-action payload and incident-state preconditions.
- Verify post-fix dispatch and policy checks before closure.
- Re-run recovery flow under incident command oversight.
- Prevent incident closure until deterministic revalidation is complete.
3.7 Telemetry and Audit Degradation#
Trigger: missing or delayed studio_webhooks_external_automation_* telemetry
events.
Actions:
- Validate analytics sink health and ingestion backlog.
- Enable degraded-observability notice in webhooks workspace.
- Replay buffered events where replay semantics are supported.
- Confirm SLO panel and audit-evidence continuity after recovery.
4. Release Checklist#
- Validate
/studio/webhooks-external-automationrenders and executes required flows. - Validate optimistic, error, and retry behavior across all webhook and automation flows.
- Validate permission-tier and production CAB policy enforcement.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility/performance evidence artifacts.
5. Incident Communication Templates#
Incident Start#
[P{severity}] Studio Webhooks/Automation Incident
Detected: {timestamp}
Impact: {subscription-contract | signed-dispatch | retry-recovery | policy-decision | incident-recovery}
Suspected cause: {contract drift | receiver degradation | policy drift | signature mismatch | telemetry degradation}
Owner: {on-call}
Incident Update#
[P{severity}] Studio Webhooks/Automation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | receiver remediation | release hold}
Affected flows: {registerWebhookSubscription | dispatchSignedWebhookEvent | executeRetryRecoveryLifecycle | decideSensitiveAutomationAction | recoverAutomationHealth}
ETA: {time}
Incident Resolved#
[P{severity}] Studio Webhooks/Automation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Webhooks/automation workspace UX | #oshun-web |
Platform on-call |
| Subscription/delivery orchestration APIs | #oshun-bff |
BFF lead |
| Receiver health and retry operations | #oshun-platform-sre |
SRE lead |
| Signature/policy/CAB governance controls | #oshun-security |
Security + Compliance |
| Telemetry and audit reporting continuity | #oshun-data-platform |
Data platform lead |
7. Acceptance Criteria (OST-00280)#
OST-00280 is complete only when:
- Ownership and decision authority are documented.
- Webhooks/automation SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.