Journey: Tenant integrations API keys and webhooks

Journey at a glance

ActorsPlatform integration operatorPartner developerDelivery workerAuditor

Systems exercisedoperator-audit.mdoperator-tenant.mdoperator-tenants.md

Automation2 E2E spec(s) · verdict: partial· 2026-06-25

Journey flow#

Generated from the authored steps below — click a node to jump to that section.

flowchart TD s1["1. Inspect the registry"] s2["2. Create and use an API key"] s3["3. Rotate or revoke at the API boundary"] s4["4. Create a webhook and run the simulator"] s5["5. Dispatch and record partner outcomes"] s6["6. Replay due deliveries"] s7["7. Maintain connector and audit projections"] s1 --> s2 --> s3 --> s4 --> s5 --> s6 --> s7 click s1 href "#1-inspect-the-registry" click s2 href "#2-create-and-use-an-api-key" click s3 href "#3-rotate-or-revoke-at-the-api-boundary" click s4 href "#4-create-a-webhook-and-run-the-simulator" click s5 href "#5-dispatch-and-record-partner-outcomes" click s6 href "#6-replay-due-deliveries" click s7 href "#7-maintain-connector-and-audit-projections" s7 --> v(["verdict: partial"]) click v href "../results/tenant-integrations-api-keys-webhooks.html" style v stroke:#d6a531,color:#d6a531

The platform integrations registry has a strong secret-safe API-key lifecycle and a real signed-webhook state machine. /operator/tenant can inspect the registry, create one key, create one sandbox subscription, and run the simulator. Rotation, revoke, dispatch, receipt, replay, deactivate, and connector controls remain API-only. Most importantly, “dispatch” and the replay worker produce signed request envelopes but never perform the partner HTTP request; delivery receipts must be posted by an external actor.

Personas#

Platform integration operator — holds admin:* or admin:studio and mutates the central registry.
Partner developer — receives a one-time API token or webhook signing secret and operates an external endpoint.
Delivery worker — is expected to send signed requests and report partner outcomes, but is not implemented in this BFF.
Auditor — reads structured admin.integrations.* lifecycle evidence.

Pre-conditions#

Platform-admin authentication is available. Tenant-console workspace scope intentionally receives 403 from every registry mutation/read.
The UI defaults to registered sandbox tenant t-sandbox-alpha; simulator and dispatch reject non-sandbox tenants.
Admin durable snapshots are configured if registry state must survive restart. API-key hashes, subscriptions, deliveries/events, connector state, and display names are persisted.
Webhook raw signing secrets are held only in a process-local map. They are not part of the durable snapshot, so a restored subscription cannot sign until it is recreated.

Steps#

1. Inspect the registry#

GET /v1/admin/integrations/snapshot returns redacted API-key summaries, redacted subscriptions, deliveries, connectors, docs, sandbox tenants, and a seed-data notice. The notice disappears after the first real registry record. The browser panel shows the latest three keys, webhooks, and deliveries.

2. Create and use an API key#

Create accepts tenant id, display name, scopes, optional IP allowlist, rate limit, and expiry. The raw oshun_… token is returned once; the durable record stores its SHA-256 hash and public prefix. Later snapshots never reveal token or hash.

GET /v1/integrations/auth-probe is the only customer-like resource wired to this key authenticator. It checks token state/expiry, tenant, required scope, and IP, then updates lastUsedAt. This proves the auth primitive, not broad adoption across tenant APIs.

3. Rotate or revoke at the API boundary#

Rotation creates a new key linked by rotatedFromKeyId, returns its raw token once, and gives the prior key a grace expiry. Revoke timestamps the key; duplicate revoke returns 409. These operations and their read-backs are covered over real HTTP but have no controls in TenantIntegrationsPanel.

4. Create a webhook and run the simulator#

The browser/API create an active subscription with tenant, display name, topic patterns, HTTPS target, retry parameters, and one-time signing secret. A matching sandbox simulation builds a canonical body and HMAC signature, then synthetically moves the delivery to success. It does not call the configured target URL. Inactive or nonmatching simulations record a failed delivery and return no signed request; a production tenant is rejected.

5. Dispatch and record partner outcomes#

API-only dispatch validates sandbox tenant, active state, topic, and available raw signing secret. It stores an event and an in-flight delivery and returns the signed request envelope. There is no fetch, queue adapter, or network client in the route/store.

An external sender can post a receipt. Success records response metadata; transient failure schedules retry; exhausted failure becomes dead-letter. The receipt endpoint trusts the authenticated platform operator's supplied outcome—it is not a callback authenticated by the partner endpoint.

6. Replay due deliveries#

The replay route and optional interval scan due pending records, rebuild their signed request, and move them back to in-flight. The interval is explicitly single-instance/leader-election work. It also does not send the request; without an external consumer, a replay can remain in-flight indefinitely.

After durable hydration, signing secrets are empty by design. Existing subscriptions/deliveries remain visible, but simulator, dispatch, and replay skip/fail with signing_secret_unavailable until the subscription is recreated.

7. Maintain connector and audit projections#

Connector upgrade enforces known/nondeprecated versions. Health-probe accepts a caller-supplied healthy/failure verdict and response metadata and updates the circuit state; it does not itself contact the connector. API-key, webhook, simulator, dispatch, receipt, replay, activation, upgrade, and probe actions emit structured admin.integrations.* events searchable through both admin search and the audit-log explorer.

Post-conditions#

API-key hashes and lifecycle metadata are durable and a valid key can pass the dedicated scoped auth probe.
Webhook subscriptions/delivery state and audit evidence are durable, while the raw signing material needed for future signatures is not.
The browser can create keys/subscriptions and prove a synthetic signed simulator result without exposing persisted secrets.
Retry/dead-letter state transitions are real when receipts are supplied.
No partner HTTP delivery, partner-authenticated receipt, automatic response observation, or post-restart signing continuity is provided by this path.

Failure modes#

Wrong scope — tenant-console workspace admins cannot read or mutate even a tenant-local subset.
One-time secret loss — leaving the creation result loses the raw API key or signing secret; neither can be recovered.
Webhook restart break — durable subscriptions restore without raw signing secrets, so dispatch/replay cannot sign them.
Synthetic simulator — a success proves canonical signing logic, not that the partner URL is reachable or accepts the payload.
No sender — dispatch/replay returns an envelope and state transition but performs no network request.
Receipt trust boundary — platform-authenticated callers provide status and response excerpts; no delivery-specific partner signature is verified.
In-flight orphan — absent an external sender/receipt, deliveries do not resolve.
Auth-probe island — key enforcement is proven on one probe route, not on general tenant resources.
Reported health — connector probes record the submitted verdict rather than measuring an endpoint themselves.
Fire-and-forget persistence — registry mutations return before async snapshot flush; a crash in that interval can lose the latest delta.

E2E coverage#

apps/oshun/web/e2e/tenant-integrations-api-keys-webhooks-bff.spec.ts covers platform gates, key create/rotate/revoke/auth-probe, redaction, simulator branches, dispatch/receipt/replay/dead-letter, activation, connector changes, and audit/search read-back over the real BFF.
apps/oshun/web/e2e/tenant-integrations-ui.spec.ts covers live snapshot, browser key/subscription creation, one-time secret rendering, redacted refresh, and signed simulator output.
Coverage depth: deep for the state machine, partial for delivery. Tests correctly prove envelope generation and caller-posted receipts; no test can prove external HTTP send/receipt or restored-secret continuity because those joins are absent.

Per-view files touched#

operator/operator-tenant.md — registry, key create, webhook create, and simulator UI.
operator/operator-audit.md — lifecycle evidence.
operator/operator-tenants.md — adjacent platform tenant context.
Rotation/revoke/dispatch/replay/connector operations have no dedicated browser view in this journey.

Cross-references#

tenant-bulk-member-invite-scim.md — adjacent identity integration, not driven by these webhook envelopes.
tenant-oneroster-roster-sync.md — adjacent roster import, not automatically dispatched here.
tenant-audit-log-investigation.md — structured audit follow-up.
Sources: admin-integrations-registry.ts, admin-integrations-registry-store.ts, durable-backed-integrations-registry-store.ts, server.ts, and TenantIntegrationsPanel.tsx under apps/oshun.

Open questions#

Which queue/worker performs the actual HTTPS request, consumes replay envelopes, enforces timeouts, and posts authenticated receipts?
How should webhook signing secrets be encrypted and restored without making them visible in operator snapshots?
Should a delivery-specific capability authenticate receipt writes instead of a general platform-admin bearer?
Which real tenant APIs adopt the integration-key guard beyond the auth probe?
Which controls belong in the operator UI for rotate/revoke/deactivate, delivery inspection/replay, and connector health—and which belong in a tenant-admin-scoped subset?