# Journey result: Tenant integrations API keys and webhooks

- **Walked**: 2026-06-25; updated 2026-07-02 by Codex against local real-BFF dev
  infrastructure
- **Verdict**: **deep pass** - the local BFF API spine now proves platform-admin
  scope gates, API key create/rotate/revoke, secret non-leakage, a downstream
  API-key-authenticated tenant resource probe with tenant/scope/IP and
  revoked-key boundaries, webhook create/toggle/simulator behavior, signed
  sandbox delivery, outbound dispatch handoff, partner receipt retry/backoff,
  replay-due worker tick with regenerated signed request, dead-letter
  exhaustion, success receipt, non-sandbox simulator rejection, connector
  upgrade, connector health probes, malformed payload rejection, audit-search
  read-back, and structured audit payloads. The shipped operator tenant-console
  UI also covers live snapshot, API-key create, webhook create, one-time secret
  surfacing, and signed simulator controls.
- **Primary spec**:
  `apps/oshun/web/e2e/tenant-integrations-api-keys-webhooks-bff.spec.ts`

## Observations

- `GET /v1/admin/integrations/snapshot` rejects missing auth and tenant-console
  workspace-only scope, then returns docs, sandbox tenants, connector records,
  and integration summaries for platform admin scope.
- `POST /v1/admin/integrations/api-keys` returns the one-time token only at
  creation; snapshot read-back exposes only the prefix and redacted metadata.
- `GET /v1/integrations/auth-probe` accepts that raw token through
  `x-oshun-api-key`, rejects missing tokens, tenant mismatch, missing scope,
  non-allowlisted IPs, and revoked keys, and updates the redacted key's
  `lastUsedAt` for snapshot read-back.
- Rotate and revoke flows preserve the previous-key grace window, link the new
  key to `rotatedFromKeyId`, set `revokedAt`, and reject duplicate revoke.
- Webhook simulation signs matching active sandbox deliveries and verifies the
  same signature. Inactive subscriptions fail closed without emitting a signed
  request, and non-sandbox tenants cannot use the simulator.
- Webhook dispatch creates an in-flight signed request for an active matching
  sandbox subscription; partner receipts move the delivery to pending retry with
  backoff metadata, exhausted `dead_letter`, or `success` with response status.
- `POST /v1/admin/integrations/webhooks/replay-due` drives the same pending
  delivery back to `in_flight`, regenerates a signature-verified request from
  persisted event material, reports replay/skip counts, and emits
  `admin.integrations.webhook.delivery_replayed` audit evidence. The deployable
  server can run that tick autonomously when
  `OSHUN_INTEGRATIONS_WEBHOOK_WORKER_INTERVAL_MS` is configured.
- Integration mutations surface through admin audit search using the injected
  audit-events store that real dev infra wires at app startup.

## Out of scope

- External partner endpoint networking beyond the real-BFF dispatch/receipt
  handoff.