Eve library

OSHUN Studio Enterprise Tenant Isolation Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Security + Platform Identity + OSHUN Web + OSHUN BFF
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0028-oshun-studio-enterprise-tenant-isolation.md
  • docs/releases/v1/specs/studio-enterprise-tenant-isolation-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-enterprise-tenant-isolation-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioEnterpriseTenantIsolationWorkspace.tsx
  • apps/oshun/web/src/analytics/studioEnterpriseTenantIsolationTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio enterprise tenant isolation governance workflows.

This document is the execution artifact for OST-00128.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Enterprise tenant-isolation route OSHUN Web Platform Web route + analytics contracts
Tenant context and identity contracts Platform Identity + Security BFF tenant registry + identity services
Tenant policy and boundary decisions Security + Program Governance Identity policy engine + boundary rules
Cross-tenant exception controls Security + Compliance Program Ops approval workflows + exception registry
Telemetry and audit evidence Data Platform SRE event sinks + ingestion pipelines

2. Core SLOs#

Metric Target Alert Threshold
Tenant-isolation flow success rate >= 99.0% < 98.0% over 15 min
Tenant boundary violation rate 0% any detected violation
Cross-tenant exception correctness 100% policy-compliant any unauthorized exception allow
Tenant-isolation flow p95 latency <= 2200ms > 3200ms over 15 min
Tenant isolation audit completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Tenant Context Validation Failures#

Trigger: repeated TENANT_CONTEXT_VALIDATION_FAILED outcomes.

Actions:

  1. Validate tenant registry and identity service health.
  2. Inspect tenant context payload for missing or stale fields.
  3. Re-run context validation in preview with diagnostics enabled.
  4. Block tenant-scoped promotions until context consistency recovers.

3.2 Boundary Policy Simulation Failures#

Trigger: repeated TENANT_BOUNDARY_POLICY_SIMULATION_FAILED outcomes.

Actions:

  1. Validate tenant boundary policy version and distribution status.
  2. Inspect read/write partition rule conflicts.
  3. Re-run simulation with policy trace correlation.
  4. Block release promotion until deterministic policy outcomes pass.

3.3 Cross-Tenant Exception Review Failures#

Trigger: repeated TENANT_CROSS_TENANT_EXCEPTION_REVIEW_FAILED outcomes.

Actions:

  1. Validate exception request payload and justification completeness.
  2. Confirm approval chain and expiry windows.
  3. Reconcile mismatched source/target tenant mappings.
  4. Deny unresolved exceptions and escalate for manual review.

3.4 Policy Denial Spike#

Trigger: unexpected increase in ROLE_POLICY_DENIED outcomes.

Actions:

  1. Compare role allowlists with active tenant policy contracts.
  2. Validate recent identity/entitlement payload updates.
  3. Apply policy hotfix or rollback drifted tenant policy bundle.
  4. Confirm deny-rate normalization after mitigation.

3.5 Telemetry and Audit Degradation#

Trigger: missing or delayed tenant-isolation telemetry/audit events.

Actions:

  1. Validate analytics sink health and ingestion queue depth.
  2. Enable degraded-observability warning in workspace.
  3. Reconcile missed events from buffered traces.
  4. Confirm event completeness and SLO recovery.

4. Release Checklist#

  • Validate /studio/enterprise-tenant-isolation renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy allow/deny behavior.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Enterprise Tenant Isolation Incident
Detected: {timestamp}
Impact: {tenant context | boundary policy | cross-tenant exception impact}
Suspected cause: {tenant registry issue | policy drift | exception workflow regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Enterprise Tenant Isolation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {tenantContextValidation | boundaryPolicySimulation | crossTenantExceptionReview}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Enterprise Tenant Isolation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and workspace #oshun-web Platform on-call
Tenant identity and context contracts #oshun-identity Identity architecture lead
Tenant policy and boundaries #oshun-security Security lead
Exception governance and compliance #oshun-compliance Compliance lead
Telemetry pipeline and reliability #oshun-data-platform SRE lead

7. Acceptance Criteria (OST-00128)#

OST-00128 is complete only when:

  1. Ownership and decision authority are documented.
  2. Enterprise tenant-isolation SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.