Workbench operations cover interactive shells, APIs, workers, queues, databases, object storage, search/graph projections, media/model providers, DCC/engine bridges, and external delivery. A green route alone is not workbench health; the supported journey and its recovery path must be observable end to end.
Readiness is a capability state. A shell can remain available while a mutating, provider-backed, or export path is read-only, queued, or explicitly unavailable.
Operational inventory#
For each domain, maintain:
- supported UI/native/CLI entry points and deployment units;
- APIs, event topics, queues, scheduled cycles, workers, remote hosts, and callbacks;
- databases/schemas, blob stores, caches, search/vector/graph indices, audit and signing stores;
- provider/model/tool/DCC dependencies and credentials;
- feature flags, tenant scopes, configuration, secrets, limits, and ownership;
- dashboards, alerts, runbooks, recovery objectives, backups, restore targets, game-day scenarios, and release gates.
The generic estate inventory and domain inventories seed this register. Deployment discovery must reconcile them with what actually runs.
Health and readiness#
Liveness answers whether the process can make progress. Readiness answers whether it can safely accept its supported traffic. Readiness includes critical migrations, stores, queue/outbox, identity/policy configuration, required providers, and incompatible version fences. A noncritical provider may yield a declared degraded-ready state only when capability-specific degradation is safe and visible.
Workers expose lease/fencing health, backlog age/depth, attempts, poison/dead letters, throughput, latency, and result/reclamation state. DCC/remote hosts add host identity, supported versions/capabilities, project scope, heartbeat, current lease, artifact transfer, and command/result receipts.
Service-level indicators#
Measure user and control outcomes by domain and capability:
| Signal | Examples |
|---|---|
| Availability | Authorized project/source/session opens; command acceptance; evidence/history load; job status; export/download; remote-host reachability. |
| Latency | Interactive read/write, search, save acknowledgement, job queue-to-start, processing phase, provider/DCC round trip, projection lag, notification. |
| Correctness | Schema and invariant failures, conflict rate, duplicate effects, stale projections, result validation, lineage completeness, signature verification. |
| Safety and rights | Denials by reason, cross-tenant probes, quarantine, policy/rights blocks, expiry/deletion backlog, export exclusions, provider/model withdrawal. |
| Reliability | Retry success, poison jobs, lease loss, outbox lag, restore consistency, callback/update dedupe, cancellation/compensation, crash loops. |
| Experience | UI error/partial/offline state, save honesty, accessibility regressions, unsupported formats/versions, time to recover or complete a core journey. |
Metrics use bounded dimensions; tenant/resource ids and content do not become high-cardinality labels. Correlation and trace ids link sanitized logs, traces, audit, job, provider, and UI errors without exposing secrets.
Degradation#
Every dependency has a capability plan:
- what reads remain available and whether they are stale;
- what writes are blocked, queued, or safely local;
- whether jobs can start, continue, publish, retry, or cancel;
- what provider/model/tool fallback is compatible and authorized;
- what the user sees and what operators are alerted on;
- how the system detects recovery and reconciles queued/stale state;
- which data or results are withheld rather than guessed.
Fallback quality is verified for the same contract. Silently changing a model, renderer, codec, DCC version, or evaluation method can invalidate comparison and provenance even when a request succeeds.
Capacity and resource safety#
Plan limits for upload/ingest, media duration/resolution, archive expansion, projects/sources/anchors, concurrent sessions, job queue/backlog, GPU/CPU/ memory/disk, database connections/locks, blob bandwidth, search/index growth, provider quotas, exports, and remote hosts.
Admission control distinguishes user, tenant, capability, and global capacity. Backpressure is end to end; accepting work into an unbounded local queue is not protection. Processing sandboxes have time, memory, CPU, output, recursion, and network limits. Capacity failures yield explicit retry/limit guidance and do not corrupt source or history.
Backup, restore, and disaster recovery#
Recovery covers system-of-record relational data, blobs, revision/history, outbox/inbox and job state, audit/signing keys and evidence, plus the sources needed to rebuild search/vector/graph projections. External providers and DCC hosts require reconciliation because a database restore cannot rewind them.
A valid restore proves:
- tenant/resource counts and sampled semantic invariants;
- database-to-blob and source-to-derived reachability;
- history ancestry and no impossible job/lease state;
- outbox/inbox idempotency and external-effect reconciliation;
- projection rebuild and cutover consistency;
- rights, holds, expiry, deletion/tombstone, and revoked-access state;
- signature verification and key availability;
- application journeys, not only store health.
The
Yemaya backup/restore runbook
is the domain's worked example. Shared restore classes and reference edges are
implemented in libs/oshun/workbench-kit/src/restore-consistency.ts.
Rollout and rollback#
Roll out in explicit rings or cohorts with owner, entry criteria, measurement window, abort conditions, and promotion decision. Flags are tenant/capability scoped and observable. Schema, contract, worker, projection, client, and provider versions use a compatibility window; mixed-version behavior is tested.
Rollback is safe only if the previous version can interpret newly written records and external effects. Otherwise use forward-fix, disable new writes, keep compatible readers, or compensate. Never destroy migration or audit evidence to make the old build start.
Each release records source revision, generated artifacts, migrations, configuration/flags, dependency/model/tool versions, test and evaluation evidence, security/privacy review, rollout state, and approver.
Incident handling#
The common sequence is:
- Detect and declare the affected capability, tenants, data, and time window.
- Contain unsafe writes, providers, workers, routes, remote hosts, keys, or exports while retaining supported safe reads where possible.
- Preserve sanitized evidence and audit; do not copy sensitive content into an incident channel.
- Establish source of truth, blast radius, irreversible external effects, and deletion/rights implications.
- Repair, replay/rebuild/reconcile with idempotency and version fences.
- Verify the user journey, tenant isolation, data/provenance, and recovery objective before reopening.
- Communicate impact and residual risk; update runbooks, alerts, gates, and game-day coverage.
Domain-specific operational focus#
- Isis: GPU/provider capacity, model/workflow compatibility, queue and output lineage, safety/policy, consistency/parity, and generated asset reclamation.
- Metis: learning/assessment availability, evaluator/rubric versioning, progress/event ordering, credentials, integrations, and learner-data privacy.
- Yemaya: large media ingest/processing, parsers, analysis providers, projection/search/graph, rights expiry, evidence/export, deletion, and cross-domain adapters.
- Veritas: confidential sources, research/claim state, review/embargo, schedules, publish/delivery/correction/takedown, and social provider limits.
- Euterpe: realtime audio latency/glitches, collaboration consistency, native/browser/plugin compatibility, recording durability, and export.
- Aja: capture/reference ingest, motion jobs/models, skeleton/format compatibility, performer rights, large derived artifacts, and pipeline delivery.
- Bellona: bridge/host connectivity, engine/tool versions, remote command authorization, build/render queues, artifact transfer, approvals, and external-effect reconciliation.
Release evidence checklist#
- Targeted lint, type, unit, contract/schema, migration, and integration gates.
- Core browser/native/CLI journeys with success, denial, error, retry, partial, offline, conflict, cancellation, and recovery states.
- Accessibility and performance budgets for affected compositions.
- Security, tenant, rights, export, deletion, provider, and external-tool tests.
- Current dashboards/alerts/runbooks and a verified restore/rollback path when state or effects changed.
- Inventory and generated artifact freshness.
- Deployed smoke/canary and promotion/abort evidence.
Do not mark a capability complete because one package test passed. Completion means the supported entry point, real effect, enforcement, observability, failure handling, and recovery evidence agree.