Residual Audit — Slice 10: Admin Web / Admin Mobile / Tenant Console / Operator Toolkit

Spec slice: V1/features.md lines 4942–5204. Date: 2026-06-11. Method: static source reading only (no builds/tests/servers); every claim below was traced to current code, following delegation chains end-to-end.

Context honored: B3/C8/C12/C13/D3 closures from the 2026-06-10 audit were re-verified as landed (tenant onboarding store is real; tenant-admin members/sso/audit read the live BFF; admin-mobile action sheet is mounted and posts to POST /v1/admin/review/items/:reviewId/decision which exists at apps/oshun/bff/src/routes/admin.ts:6399; banners/help/status authored→consumed loop is real in apps/oshun/bff/src/routes/customer-communications.ts; OIDC SSO login runtime is real and fail-closed in apps/oshun/bff/src/auth/sso-login-routes.ts). SAML deferral is recorded and not re-raised. The three systemic "management without enforcement" items from the 06-10 audit's pattern 5 had no dedicated backlog task and are confirmed still open — they are findings 1–3 below, characterized precisely.

1. API keys are stored, rotated, revoked — and still never validated on any request#

Severity: P0-STRUCT

Evidence:

• apps/oshun/bff/src/admin/admin-integrations-registry-store.ts:261-277 — authenticateApiKeyFor exists (sha256 hash match → tenant-console authenticateApiKey with expiry/revocation/IP-allowlist checks) and is exported at line 543.
• Call-site grep across apps/oshun/bff/src, apps/oshun/web/src, apps/oshun/admin/src: the ONLY references are the store itself and the durable wrapper's mutator list (apps/oshun/bff/src/admin/durable-backed-integrations-registry-store.ts:59). Zero routes, zero middleware, zero preHandlers call it.
• apps/oshun/bff/src/middleware/authz.ts:99-122 — the only request authentication is bearer dev/tenant./HS256-JWT tokens. No x-api-key or Authorization: oshun_… path exists anywhere in app.ts/server.ts.

Spec promise: features.md:5131-5132 ("API keys with scopes, rotation, expiration, rate limits, IP allowlists, last-used metadata, and revocation"), :5144 ("Tests for key rotation, scope enforcement …").

What the code actually does: full CRUD + rotation-with-grace + revocation + hashing are real, and the route surface (apps/oshun/bff/src/routes/admin-integrations-registry.ts, proxied by apps/oshun/admin/src/app/api/admin/integrations/*) works. But no inbound request anywhere can present an API key, so scopes, rate limits, IP allowlists and expiration are stored-config-never-enforced. Corollary: lastUsedAtUnixSeconds is only ever set by the seed or by authenticateApiKey — so the "Last used" the admin panel renders can never change for any operator-created key (dead UI signal).

Fix sketch: add a createApiKeyAuthPreHandler(store) that accepts Authorization: Bearer oshun_…, calls store.authenticateApiKey, maps key scopes → request.authContext, and mount it on at least one partner-facing route group (the natural first consumer: a public ingest/metrics route matching the seeded ingest:write/metrics:read scopes). Enforce rateLimitPerMinute in the same preHandler via the existing abuse-protection window store.

2. Outbound webhooks: the real delivery executor was built — and orphaned; no real event ever dispatches#

Severity: P0-STRUCT

Evidence:

• libs/oshun/tenant-console/src/integrations/webhook-delivery-execution.ts:1-21 — module docstring says it "closes the gap" found by the audit, builds the canonical signed request and sends via an injected WebhookHttpTransport, noting "the app boundary wires a real fetch-based transport".
• Repo-wide consumer grep for executeWebhookDelivery / WebhookHttpTransport / buildWebhookRequest / classifyDeliveryResponse: only the module itself and libs/oshun/tenant-console/src/integrations/index.ts. No app boundary ever wired it. No fetch-based transport exists in the BFF.
• No event source: nothing in apps/oshun/bff/src constructs a WebhookEvent or calls subscriptionsForEvent outside the simulator. The only delivery ever recorded is fireWebhookSimulator (admin-integrations-registry-store.ts:325-422), which is correctly labeled stub:legitimate as the §20.6 simulator feature and is restricted to sandbox tenants.
• A blocking sub-gap even if someone wires it: raw signing secrets live only in the in-memory webhookSigningSecrets map (admin-integrations-registry-store.ts:158-159) and are excluded from IntegrationsRegistryPersistedState (lines 72-79, 482-490) — after a BFF restart no secret can be resolved for any pre-restart subscription (SIGNING_SECRET_UNAVAILABLE, honest fail-loud, but delivery-dead).
• The two other "webhook" stores are classifiers, not transports: apps/oshun/bff/src/studio/webhook-delivery-store.ts (pure backoff/status classifier over caller-supplied response codes) and apps/oshun/bff/src/aja/webhook-management-store.ts (config validator + status state machine). Neither performs HTTP. @oshun/event-bus ships outbound-delivery.ts but its only non-test consumers are the tenant-console lib comment and a yemaya capture lib.

Spec promise: features.md:5133-5135 ("Outbound webhook management with topic subscriptions, signing keys, retry/backoff policy, dead-letter inspection, replay tooling, and per-event audit"), :5144-5145 ("webhook delivery semantics, signature verification, replay safety").

What the code actually does: subscriptions are authored, stored durably, and listed; deliveries shown in the admin panel can only ever be simulator rows. A partner who registers a webhook will never receive a single real event.

Fix sketch: (1) wire a fetch transport + createWebhookDeliveryExecutor in the BFF composition root; (2) persist signing secrets (encrypted column or env-KMS seam) or store-and-return-once with a durable encrypted copy; (3) publish WebhookEvents from at least the admin audit-events firehose (adminAuditEventsStore.record is the natural single choke point) and run the retry loop on the existing reminder-worker-style interval timer.

3. Role templates and the entire roles engine never feed the live authz check#

Severity: P0-STRUCT

Evidence:

• libs/oshun/tenant-console/src/roles/roles.ts — real engines exist: diffTemplateAgainstCanonical (:60), validateTemplateWithinApprovedBounds (:127), effectiveCapabilities (:189, inheritance-ordered), transitionAssignment (:276, approval/expiry/recertification lifecycle), runDryRun (:326, the spec's role test harness).
• Consumer grep across all apps: the ONLY non-test consumer is the tenant-admin demo page apps/oshun/tenant-admin/src/app/roles/page.tsx (explicitly labeled demo, line 83).
• The live authorization check is raw scope-string matching on the token: apps/oshun/bff/src/middleware/authz.ts:144-145 (scopes.includes('domain:*') || scopes.includes('domain:${id}')) and per-route admin:*/admin:studio/tenant:admin:{id} includes-checks (e.g. routes/tenant-console-reads.ts:24-26, routes/admin-tenant-console.ts:41-44). Nothing ever expands a role template into scopes, and no store or route for tenant role templates exists in the BFF at all (the admin-yemaya-rbac / admin-studio-rbac-policy routes are different domains' own RBAC surfaces).
• Consequently the spec's role-assignment workflow (approval, expiration, recertification) has no backend: RoleAssignment records exist only as values constructed inside the demo page.

Spec promise: features.md:5079-5090 (operator-editable templates, per-tenant customization within bounds, permission diffs with signoff, assignment workflows with approval/expiration/recertification, dry-run harness, tests for isolation/expiry/scope-restriction).

What the code actually does: a complete, well-tested pure-function role engine sits in the library; the running system authorizes purely on literal scope strings minted into tokens by dev tooling; the two never meet.

Fix sketch: add a BFF role-template store (durable, like the other durable-backed-* stores) + /v1/admin/role-templates CRUD; at token issuance (or as an authz-preHandler expansion step keyed on tid + role ids in the token) resolve effectiveCapabilities → scope set, so templates actually gate requests; expose runDryRun on a route so the harness the spec promises is reachable.

4. Bulk-operation "staged commit" commits nothing — committed is a timestamp, not an action#

Severity: P0-HONESTY

Evidence: apps/oshun/bff/src/admin/admin-bulk-operations-store.ts:217-219 — the entire effect of the committed transition is committedAt = stamp;. transition() (139-241) validates rationale + state machine, validated computes real per-row issues, dry_run_complete builds a real per-row plan (buildDryRunPlan → classifyAdminBulkOperationRowAction), and then commit writes... nothing. Grep for consumers of committed operations across the BFF: none. No row is ever applied to tenantInviteStore (users), the OneRoster apply store (rosters), entitlements, taxonomy, or persona assignment stores — all of which exist in the same codebase.

Spec promise: features.md:4977-4979 ("Bulk-operation console for users, rosters, content, taxonomy, persona assignments, and entitlements with dry-run, validation, and staged commit"), :5111-5115 (same for import: "conflict report, and staged commit"), :5124 ("Tests for … bulk-action rollback").

What the code actually does: the operator walks draft → validate → dry-run → stage → commit, the UI reports the operation as committed, and zero records change anywhere. This crosses the result-faking line: the system claims an outcome ("committed") it did not produce.

Fix sketch: on the committing step, dispatch rows by kind to the real stores already in-repo (users → tenantInviteStore.addMember/invite; rosters → the OneRoster apply store; entitlements → customerAuthStateStore.updatePlan), record per-row outcomes on the operation, and fail-loud (failed status + failureReason) for kinds with no executor instead of stamping success.

5. "Bulk export of tenant-owned data" exports whatever JSON the operator pastes — it never reads tenant data#

Severity: P1

Evidence:

• apps/oshun/bff/src/routes/admin-bulk-exports.ts:60-68 — records: parsed.data.records comes from the request body.
• apps/oshun/admin/src/components/AdminBulkExportsPanel.tsx:264-267 — the records are a bulk-export-records-input textarea the operator pastes JSON into (parsed at :100-120).
• The engine itself is real: executeBulkExport formats (JSON/CSV/OneRoster/xAPI/Caliper), residency gating, integrity manifest, and a working verify endpoint (apps/oshun/bff/src/admin/admin-bulk-exports-store.ts:88-148).

Spec promise: features.md:5116-5118 ("Bulk export of tenant-owned content, tenant-owned data, audit bundles, rights/provenance bundles, and Metis academic records…").

What the code actually does: a packaging/manifest service. It is honest (it exports exactly what you give it) but the spec's feature — exporting the tenant's actual records — has no data source: nothing pulls members, audit events, content, or Metis records into an export.

Fix sketch: add server-side record sources per resourceKind (members → tenantInviteStore.listMembers; audit → the tenant-filtered listAcrossOperators scan already written for /v1/tenant-console/audit) and make the pasted-records path an explicit "custom records" mode.

6. Integrations registry boots with unlabeled fixture keys/webhooks/connectors reporting fabricated "healthy" probes#

Severity: P0-HONESTY

Evidence:

• apps/oshun/bff/src/admin/admin-integrations-registry-store.ts:698-806 — seed() fabricates key-prod-1 ("Production ingest", last used 1h ago), hook-1 (→ https://example.partner/api/oshun-webhook), conn-lms-1 ("Acme School LMS"), conn-payment-1, plus one connector of every kind for t-pioneer, each with lastHealthVerdict: 'healthy' and lastHealthCheckUnixSeconds synthesized relative to boot time ("2 minutes ago", "30 seconds ago").
• No health probe exists anywhere: the only mutation of connector health is the manual recordConnectorError route. The "last health check" ages from boot, is never refreshed, and never corresponds to any probe of any endpoint.
• apps/oshun/admin/src/components/AdminIntegrationsRegistryPanel.tsx renders snapshot() rows with no fixture labeling. The honest-labeling primitive exists — apps/oshun/admin/src/components/PlaceholderDataBanner.tsx — but a repo-wide grep shows it is mounted on zero routes (dead component).

Spec promise: features.md:5140-5141 ("Connector health, error budgets, circuit breakers, version pinning, and upgrade pathways"); features.md:4980-4982 (integrations registry as an operator surface).

What the code actually does: an operator opening the registry sees a production-named API key, a partner webhook, and ~10 connectors asserting recent healthy probes — none of which exist. Unlike the tenant-admin demo pages (finding 12), nothing tells the operator this is seed data.

Fix sketch: either drop the seed in favor of honest empty states (the panel already handles creation flows), or mount PlaceholderDataBanner over the seeded rows; replace fabricated lastHealthCheck* with null + "never probed" until a real probe loop exists.

7. SCIM 2.0 has a full engine and no HTTP surface — an IdP cannot provision anyone#

Severity: P1

Evidence: libs/oshun/tenant-console/src/identity/scim.ts implements applyScimSync (create/update/deactivate/suspend, group sync, conflict-resolution policies) with tests. Grep across apps/oshun/bff/src for scim: only a OneRoster route comment (tenant-console/oneroster-route.ts:2) and domain-stub manifests. There is no /scim/v2/Users, /scim/v2/Groups, or any SCIM-shaped endpoint; the only runtime consumer of the engine is the labeled tenant-admin identity demo page.

Spec promise: features.md:5064-5066 ("SCIM 2.0 user and group provisioning with create, update, deactivate, suspend, group membership sync, and conflict-resolution policies"); features.md:5072-5074 (SCIM lifecycle tests).

What the code actually does: orphaned engine. Identity providers have no URL to call; tenant member lifecycle is only mutable via the operator invite routes.

Fix sketch: add a scim-route.ts exposing SCIM 2.0 Users/Groups CRUD that translates resources through applyScimSync onto tenantInviteStore members, authenticated per-tenant (natural first consumer of finding 1's API-key preHandler — SCIM clients authenticate with bearer tokens).

8. Per-tenant authentication policy (MFA / step-up / IP allowlist) is never evaluated on the real login path#

Severity: P1

Evidence: evaluateAuthChallenge (libs/oshun/tenant-console/src/identity/*) is consumed only by the labeled identity demo page (apps/oshun/tenant-admin/src/app/identity/page.tsx:115). The live OIDC callback (apps/oshun/bff/src/auth/sso-login-routes.ts:156-259) mints a session after id_token verification + JIT gate with no MFA requirement, no step-up trigger, no IP allowlist, no device posture, and no session-refresh policy; there is also no per-tenant auth-policy store anywhere in the BFF to hold such a policy.

Spec promise: features.md:5070-5072 ("Per-tenant authentication policy: MFA requirement, step-up triggers, IP allowlists, device posture signals, and session refresh rules"), :5074 ("step-up enforcement" tests).

What the code actually does: the policy engine exists, the login runtime exists, and the login runtime never consults the engine (no policy data exists to consult). Note admin-mobile has its own step-up screen/flow (app/step-up.tsx, src/auth/adminMobileStepUp.ts) — step-up exists as a client concept but not as a tenant-policy-driven server gate.

Fix sketch: add tenantAuthPolicyStore (per-tenant policy CRUD on the operator SSO surface), call evaluateAuthChallenge in the SSO callback and in createAuthPreHandler for step-up-trigger routes, and 403 with a step_up_required reason the clients already know how to render.

9. The tenant cockpit has no production login path#

Severity: P1

Evidence:

• apps/oshun/bff/src/middleware/authz.ts:216-233 — tenant. session tokens are parsed as unsigned dev-format tokens and share the dev-token environment gate; areDevTokensAllowed (:498-503) returns false whenever NODE_ENV/OSHUN_ENV/RUNTIME_ENV === 'production'. The code comment defers to "§28 signed tenant sessions".
• The real OIDC login (finding 8's route) mints a customer session via customerAuthStateStore.provisionFederatedUser (:245-249); nothing ever issues a token carrying tenant:admin:{tenantId} scopes outside dev tooling.
• apps/oshun/tenant-admin/src/app/handoff/page.tsx tells the user "your identity provider will issue one" — no IdP integration does; and no surface in apps/oshun/admin or apps/oshun/web links to the tenant cockpit or its handoff page at all (repo grep).

Spec promise: features.md:5041-5043 ("Tenant admin shell distinct from Oshun operator admin, scoped strictly to the tenant's data…") — implies tenant admins can actually reach it in production.

What the code actually does: in production builds, every tenant-admin page redirects to /unauthorized forever, because the only accepted credential format is environment-disabled. This is acknowledged in code comments (a known seam, not a hidden fake), but it has no recorded backlog item in the 06-10 audit — flagging so it does not fall between "C12 done" and "§28 someday".

Fix sketch: extend createBffAuthToken to mint signed HS256 tenant sessions (tid + tenant:admin:{tid} scopes — the verifier already handles tid), issue them from the SSO callback when the connection's tenant has the member, and set the tenant-admin cookie on a /handoff completion route.

10. Audit tamper-evidence and redaction engines are orphaned; a comment claims a firehose feed that does not exist#

Severity: P2

Evidence:

• libs/oshun/tenant-console/src/audit-explorer/audit-explorer.ts:132 (verifyAuditChain), :165 (applyRedactions), :146 (buildCorrelationThread), :92 (SavedInvestigation) — consumed by no app code (repo grep; the verifyAuditChain in apps/oshun/bff/src/studio/audit-chain-store.ts:64 is a separate studio-domain implementation serving its own route).
• apps/oshun/bff/src/admin/admin-audit-events-store.ts:13-15 — comment: "Cross-user investigations route through the §20.4 tenant-console audit-explorer … which receives a separate firehose feed." No such feed exists; the live explorer (/v1/admin/audit-log/*, apps/oshun/bff/src/routes/admin-audit-log.ts + apps/oshun/web/src/app/operator/audit/page.tsx) serves adminAuditEventsStore records directly, with filters, saved investigations, and Markdown export.
• What is real: investigation/export bundles carry a contentHash (apps/oshun/bff/src/admin/admin-investigation-bundle-store.ts:187), and the tenant-scoped audit read honestly filters by tenant reference (routes/tenant-console-reads.ts:109-142).

Spec promise: features.md:5102-5105 ("Tamper-evident storage with verification tooling, integrity attestation, and policy-controlled redaction for privacy-sensitive fields").

What the code actually does: export-level hashing exists; per-event hash-chain integrity and redaction policy do not run anywhere on the live audit path, and the store's comment overstates the wiring.

Fix sketch: thread recordHash/previousRecordHash through adminAuditEventsStore.record using the lib's chain scheme, expose a /verify endpooint calling verifyAuditChain, and apply applyRedactions in the tenant-scoped read; fix the firehose comment either way.

11. Webhook signing secrets and the simulator do not survive restart#

Severity: P2

Evidence: apps/oshun/bff/src/admin/admin-integrations-registry-store.ts — webhookSigningSecrets (line 159) is deliberately not part of IntegrationsRegistryPersistedState (:72-79); after durable-store hydration (importState, :493-507 clears the map), fireWebhookSimulator throws SIGNING_SECRET_UNAVAILABLE (:351-358) for every pre-restart subscription. Fail-loud and honest — but it means the spec's simulator/replay tooling silently degrades to "recreate your subscription" after any deploy.

Spec promise: features.md:5133-5135 (signing keys, replay tooling), :5144-5145 (signature verification, replay safety).

Fix sketch: persist secrets encrypted-at-rest in the durable state (or an env-keyed AES-GCM wrap), since the verification flow needs the raw secret to re-sign; alternatively store-and-display-once and let replay re-issue a new secret with explicit operator action.

12. The nine labeled demo pages: labeling is honest and consistent; four have backends wireable today#

Severity: P2 (wiring opportunity register; the labeling itself passes)

Verified label text — "The figures on this page exercise the real engines on labeled sample inputs (demo data); this surface is not yet wired to live tenant records." — present, verbatim and visible in the page subtitle, on all nine: agents/page.tsx:74, content/page.tsx:75, roles/page.tsx:83, status/page.tsx:164, integrations/page.tsx:107, data/page.tsx:117, notifications/page.tsx:124, help/page.tsx:118, policy/page.tsx:95. Each page genuinely exercises the real tenant-console engines (e.g. resolvePolicy tighten-only inheritance, dryRunImport/planStagedCommit, buildStatusPageView, evaluateSlo) on inline inputs — no fake engines. The hybrid pages are honest too: identity renders the REAL /v1/tenant-console/sso table first with explicit unavailable/empty states and labels its engine section "Policy engine checks (sample inputs)" (identity/page.tsx:203); members and audit are fully live reads with honest unreachable/empty states.

Cheap-wire ranking (real backend already exists, read-only projection suffices):

1. status — GET /v1/status is already a public BFF route serving the authored components/incidents/maintenance (customer-communications.ts). The page's fabricated "Tenant API degraded / major incident" view (status/page.tsx:26-60) could be replaced by that read today; only the per-tenant SLO/escalation sections lack data.
2. help — GET /v1/communications/help serves published operator-authored articles with locale fallback; the page's targeting demo could ride on it.
3. integrations — adminIntegrationsRegistryStore records are keyed by tenantId; a /v1/tenant-console/integrations read filtering snapshot() to the session tenant is a ~40-line addition in the same style as tenant-console-reads.ts.
4. policy (partial) — tenantOnboardingStore holds the real tenant record (kind, parent, residency, seats: tenant-onboarding-store.ts:26-38); the page currently fabricates a district→school chain (policy/page.tsx:14-48). Identity card is wireable now; policy overlays have no store anywhere.
5. agents (partial) — routes/agentic-runs-lifecycle.ts already resolves tenant-scoped runs (tenantId: authContext.tenantId, :45-53); run inventory is wireable; budgets/approval-gate config per tenant has no store.
6. data (partial) — bulk operations carry targetTenantId (admin-bulk-operations-store.ts:118); a tenant-filtered read is possible, though per-operator bucketing makes the scan awkward, and finding 4 means there is nothing real to show post-commit anyway.

No tenant-scoped backend exists at all for: roles (finding 3), content scopes (only the living-scenes governance slice has a store/route), notifications (templates store is operator-bucketed; channel-preference data lives in customer reminders, not per-tenant; the lib's A/B testing/digest/message-center engines in libs/oshun/tenant-console/src/notifications/ are consumed by this demo page only — note the spec's A/B testing (features.md:5159) and digest modes (:5162) therefore have no live path; the real customer message center is the separate libs/oshun/customer-message-center + BFF store, a benign but duplicate second engine).

13. Secret generator's Math.random() fallback is mislabeled "non-secret"#

Severity: P2

Evidence: apps/oshun/bff/src/admin/admin-integrations-registry-store.ts:151-156 — generateRawToken produces API-key bearer tokens and webhook signing secrets (used at :185, :289). Its fallback branch uses Math.random().toString(36) with the carve-out comment "legitimate non-secret id/token suffix… carved out true secrets" — wrong for this call site; these are exactly the true secrets the V1-P2-08xx audit carved out. Mitigating: the primary branch is randomUUID() (CSPRNG) and typeof randomUUID === 'function' is always true on Node ≥14.17, so the fallback is effectively dead code — but it is a loaded footgun with an incorrect audit annotation.

Fix sketch: replace the fallback with randomBytes(24).toString('base64url') (already imported pattern in sso-login-routes.ts) or simply throw; fix the comment.

14. UX / product cohesion across the three admin surfaces#

Severity: UX

• Two operator front-ends, one BFF. apps/oshun/admin (shell with inbox/review/trust-safety/privacy/policy/tenant-console/... + Next API proxies) and apps/oshun/web /operator/* + /studio/* lane consoles both serve operator workflows against the same /v1/admin/* routes — e.g. the audit explorer exists as apps/oshun/web/src/app/operator/audit/page.tsx (OperatorAuditExplorer) while apps/oshun/admin proxies the same /v1/admin/audit-log/* endpoints for its own surfaces, and tenant member invite/add verbs live in apps/oshun/web/src/components/lilith/TenantMemberActions.tsx, not in the admin app. Operators must know which app owns which verb; nothing cross-links them.
• No journey into the tenant cockpit. Nothing in either operator surface links to apps/oshun/tenant-admin or its /handoff page (repo grep, finding 9); the cockpit is an island reachable only by typing its URL with a hand-minted cookie.
• Tenant admins can look but not touch. The members page is read-only; all member mutations (invite, add) are operator-web components hitting /v1/admin/tenant-console/members/*. The spec (features.md:5048-5050) gives the tenant admin role assignment, group membership, license allocation and seat management — none of which the tenant shell can perform. The sidebar's honest "Locked" treatment partially mitigates.
• Error-state conflation. tenantBffGet returns null for any non-ok response (tenant-bff.ts:44-46), so a 403 scope mismatch renders as "the backend is unreachable — check that the Oshun BFF is running", sending a mis-scoped admin down the wrong debugging path.
• Positive: admin-mobile is genuinely cohesive — login → step-up → operator tabs → workspace queues, all clients hit real /v1/admin/* routes (adminMobileReviewQueueClient → /v1/admin/workspaces/review, urgentQueueClient → /v1/admin/workspaces/inbox, decision client → /v1/admin/review/items/...), with offline cache, honest degraded states, and the C13 action sheet mounted in the queue panel (AdminMobileReviewQueuePanel.tsx:135). The tenant cockpit's shell (scope-gated sidebar, per-page testids, consistent unavailable/empty/demo states) is internally consistent and the most honest surface in the slice.

Severity counts#