V3 GA Security Incident Response Runbooks

runbook-set: v3-ga-security-incident-response.v1

verifier: pnpm verify:v3 security-incident-response (proposed; wire into pnpm verify:v3 operator-runbooks-on-call alongside the availability set)

These runbooks complement V3/runbooks/incident-response-v3-ga.md, which covers availability incidents only. Every security incident below has a named incident commander role, a first-15-minutes checklist, evidence-preservation steps that survive legal review, and an audit event written into the Lilith Operator Console. Severity follows the same SEV-1/2/3 scale as the availability set. Security incidents additionally page the security on-call (oshun-security-oncall) and the Lilith-Privacy lead when personal data may be involved.

Shared rules for all four runbooks:

• The incident commander for the calendar day (per V3/operations/on-call-rota-ga-week.json) owns the bridge; the security commander role below owns the technical response.
• Evidence preservation always precedes remediation that would destroy state: snapshot first, rotate second.
• All notification clocks below start at confirmation of the incident, not detection of the alert — but the confirmation decision itself has a 4-hour SLA from first credible signal, so the clock cannot be parked.
• No public statement, tweet, or status-page update goes out without joint sign-off from the incident commander and the legal owner.

incident: data-breach-personal-data#

• Owner: Security incident commander (oshun-security-oncall), joint with Lilith-Privacy lead
• Severity: SEV-1 if personal data confirmed exfiltrated or exposed; SEV-2 for credible-but-unconfirmed exposure; SEV-3 for contained internal mishandling with no external exposure
• Trigger: confirmed unauthorized access to any datastore holding personal data (Postgres world/persistence DBs, consent ledgers, voice transcripts, DSAR export bundles, V1 account records), anomalous bulk-read alerts, or a credible external report (researcher, user, press).
• First action SLA: acknowledge within 5 minutes; commander assigned and breach-assessment bridge open within 15 minutes; breach-or-not determination within 4 hours of first credible signal.

First 15 minutes#

1. Page oshun-security-oncall, Lilith-Privacy lead, and the daily incident commander; open the incident bridge and a restricted evidence channel.
2. Freeze the suspected access path: revoke the implicated credentials or service tokens, do not yet delete accounts or wipe hosts.
3. Snapshot evidence (see evidence preservation below) before any remediation that mutates state.
4. Start the breach-assessment worksheet: what data classes, what row counts, what regions, what time window.
5. Record operator.security.data_breach.opened in the Operator Console.

Decision tree: is regulator notification required?#

1. Is personal data involved? Personal data in V3 scope: V1 account records, e-mail addresses, payment metadata, voice transcripts (VOICE_TRANSCRIPT_RETENTION.md), recordings and consent-ledger entries (RECORDING_CONSENT_FLOW.md), DSAR exports (DSAR_V1_PIPELINE.md), Pixel Streaming session logs with IP addresses. If no → SEV-3 internal security incident, document and close; no regulator clock.
2. Which residents are affected? Map affected rows to wave-1 regions using the residency zones in V3/PER_REGION_RULES.md (us-west, eu-de, uk, in, br storage zones). A breach of an eu-de replica is a DE/EU matter even if the attacker entry point was elsewhere.
3. Is there a risk to rights and freedoms? Encrypted-at-rest data with keys not compromised may be exempt under GDPR Art. 33(1) / UK GDPR — the legal owner makes this call, never engineering alone.
4. If notification is required, the per-region clocks and owners below apply simultaneously. Run them in parallel, not in sequence.

Regulator notification matrix (wave-1 regions)#

The India CERT-In 6-hour clock is the tightest deadline in the wave-1 set: if any in-resident data may be affected, the CERT-In draft starts in the first hour, before full scoping is complete. Partial notices are amended later; missed clocks are not.

• Steps: contain access path → preserve evidence → scope affected data classes and regions → legal determination → regulator notices per matrix → user notices (per-region template, plain language, what happened / what data / what we did / what you should do) → forced credential rotation for affected accounts via the JWT family-revocation machinery → post-incident review with a written timeline.
• Rollback: none — breach response only moves forward. Re-enable revoked service credentials only after the entry path is closed and verified by an independent reviewer.
• Escalation: legal owner, executive on-call (mandatory for SEV-1), payment provider and cloud TAM if their surfaces are implicated, external forensics retainer if exfiltration is confirmed.
• Evidence preservation: snapshot affected Postgres instances and WAL range (pg_basebackup to the locked evidence bucket s3://oshun-v3-evidence/<incident-id>/, object-lock compliance mode, 7-year retention), export relevant Redis keyspace (redis-cli --rdb), copy access and audit logs for the incident window ±24 h, hash every artifact (SHA-256 manifest signed by two responders), record chain of custody in the Operator Console case.
• Audit event: operator.security.data_breach.handled
• Drill: tabletop required before GA; evidence drill-data-breach-tabletop-ga (owner: Lilith-Privacy lead)

incident: account-takeover-credential-stuffing#

• Owner: Security incident commander, joint with Identity/BFF on-call
• Severity: SEV-2 for an active stuffing wave; SEV-1 if confirmed takeovers include payment actions, minors' accounts, or instructor/operator accounts
• Trigger: any of the detection signals below crossing threshold on the v3-bff-auth-abuse dashboard, or a confirmed user report of an account they did not access.

Detection signals#

First 15 minutes#

1. Page security on-call and Identity/BFF on-call; open the bridge.
2. Confirm the wave on the dashboard; capture the current attacking IP/ASN set and the credential-list fingerprint (sorted username sample hash).
3. Apply rate-limit escalation step 1 (below). Do not block legitimate-login paths wholesale — paid concert windows may be live.
4. Snapshot auth logs for the window to the evidence bucket.
5. Record operator.security.account_takeover.opened.

Rate-limit escalation ladder#

Applied at the BFF auth routes (/api/v1/auth/login, /api/v3/lilith/launch), in order, each step held 15 minutes before the next:

1. Tighten per-IP login limit from 10/min to 3/min; enable CAPTCHA on failure-then-retry from the same IP.
2. Block the top offending ASNs/CIDRs at the WAF (see V3/security/ddos-waf-posture.md rule class auth-protection); require e-mail verification code for logins from never-seen-before device fingerprints.
3. Enable credential-list defense: deny-and-flag any login whose credential pair matches the observed attack corpus, even when correct — route those users to forced reset.
4. (Commander approval required) Temporarily require fresh re-authentication for all sensitive actions platform-wide (payout changes, e-mail changes, DSAR requests).

Forced re-authentication procedure (JWT machinery)#

Per V3/JWT_REFRESH_ROTATION.md, every session has a refresh-token family (familyId) and one-time-use refresh tokens:

1. For each compromised account, revoke all refresh-token families for that user (POST /api/v1/auth/admin/revoke-families {userId}). Access JWTs are short-lived, so full session death occurs within the access-token TTL.
2. Force password reset on next login; invalidate active password if it matched the attack corpus.
3. For accounts with confirmed entry: also revoke device registrations, review payout/e-mail mutations in the incident window, and reverse fraudulent commerce actions via the refund-tax-reversal operator action.
4. Notify each affected user with the account-takeover template (what we saw, what we reset, payment-impact statement).

• Steps: confirm wave → escalate rate limits stepwise → identify compromised accounts (successful logins from attack infrastructure) → revoke families and force reset → reverse fraudulent mutations → user notices → feed attack corpus fingerprints back into the WAF managed rules.
• Rollback: relax rate-limit steps one at a time after the attack traffic stays below 2x baseline for 60 minutes; CAPTCHA-on-login is the last control removed.
• Escalation: Lilith-Commerce lead when payment actions are involved, Lilith-Privacy lead when the takeover set includes minors, then the data-breach runbook above if the attacker read personal data at scale.
• Evidence preservation: auth logs and WAF logs for the window, attacking IP/ASN list, credential-corpus fingerprint, list of revoked familyIds and affected userIds, all hashed and manifested in the evidence bucket.
• Audit event: operator.security.account_takeover.handled
• Drill: stuffing wave replayed against staging auth before GA; evidence drill-credential-stuffing-ga (owner: Identity/BFF on-call lead)

incident: jwt-signing-key-compromise#

• Owner: Security incident commander, joint with Realtime Gateway on-call
• Severity: SEV-1 always — a compromised signing key forges any session
• Trigger: signing key material found outside the KMS boundary (repo, log, paste site), a forged-but-valid token observed (valid signature, no matching issuance record), KMS access-policy violation alert, or insider-risk report.

First 15 minutes#

1. Page security on-call, Realtime Gateway on-call, and the daily incident commander. Treat as SEV-1 immediately; do not wait for confirmation of active exploitation.
2. Confirm which key id (kid) is implicated and which surfaces trust it: BFF-issued access/refresh JWTs, gateway handshake validation, Pixel Streaming session JWTs (PIXEL_STREAMING_SESSION_START_SUCCESS.md).
3. Generate the replacement keypair inside KMS (never on a laptop) and stage it unpublished.
4. Snapshot current JWKS, KMS audit log, and token-issuance logs to the evidence bucket before rotation mutates state.
5. Record operator.security.jwt_key_compromise.opened.

Emergency rotation — exact order of operations#

Deviating from this order either locks out all users (rotating before validators know the new key) or leaves the attacker valid (revoking issuance but not validation). Execute in order, verifying each step:

1. Stage new key: create kid=N+1 in KMS; confirm signing works with a canary token in staging.
2. Publish dual JWKS: add kid=N+1 to the JWKS endpoint while kid=N is still present. Verify all validators (BFF, realtime gateway, pxstream relay) have refreshed their JWKS cache (cache TTL is the gating step — force-refresh via the admin endpoint rather than waiting).
3. Cut issuance over: switch the BFF and gateway token issuers to sign with kid=N+1. From this moment no new tokens carry the compromised key.
4. Hold-down window: wait one access-token TTL so legitimately issued kid=N access tokens age out naturally. During the hold-down, validators log every kid=N token seen, building the blast-radius list.
5. Revoke kid=N validation: remove kid=N from the JWKS. Every remaining kid=N token — legitimate or forged — is now dead.
6. Invalidate refresh state: revoke all refresh-token families issued under kid=N (the rotation machinery in apps/v3/lilith-realtime-gateway/src/lib.rs keys revocation on familyId). Forged refresh tokens have no issuance record and die with the key; legitimate ones die with the family revocation.
7. Verify: synthetic session-start probes on all 7 POPs and a gateway handshake probe must succeed with kid=N+1 and fail with a held-back kid=N token.

Session invalidation blast radius#

Step 5+6 logs out every active user — at steady state ~5,000 concurrent sessions (FLEET_LOAD_VALIDATION.md), of which ~34% are Pixel Streaming sessions on the 7 POPs (TIER_ROUTER_DECISION_DISTRIBUTION.md). Plan for the reconnect stampede: the reconnect path (RECONNECT_VALIDATION.md) requires a valid resume token, which is also dead, so all 5,000 sessions come back through full login. Pre-warm the BFF auth path and enable the surge queue (V3/launch/launch-surge-plan.md) before executing step 5 if concurrency is above 2,500. If a ticketed Saraswati concert is live, coordinate the step-5 moment with the event commander — between songs, not mid-set — unless active exploitation forces immediate execution.

User communications#

• Status page (/status): "We are performing an emergency security maintenance; you may be signed out and asked to sign in again." Published at step 3, before the mass logout, not after.
• No public mention of "key compromise" until the post-incident review and legal sign-off; the honest-but-bounded phrasing is "precautionary credential rotation."
• If forged tokens were used to access personal data, the data-breach runbook above triggers with its notification matrix.
• Rollback: none. A compromised key is never re-trusted. If kid=N+1 proves faulty, roll forward to kid=N+2 via the same procedure.
• Escalation: executive on-call (mandatory), cloud/KMS provider TAM, external forensics if the leak vector is unknown after 24 hours.
• Evidence preservation: KMS audit logs, JWKS change history, the kid=N token-sighting log from the hold-down window (this is the forensic record of attacker activity), issuance-record diffs, all hashed and manifested.
• Audit event: operator.security.jwt_key_compromise.rotated
• Drill: full rotation executed in staging with 1,000 synthetic sessions before GA; evidence drill-jwt-key-rotation-ga (owner: Realtime Gateway lead). The drill must measure JWKS cache-refresh lag on every validator.

incident: voice-recording-privacy-incident#

• Owner: Lilith-Privacy incident commander, joint with Concert capture on-call
• Severity: SEV-1 if a consent-system failure occurred during a live public concert or affected minors; SEV-2 for class/cohort sessions or pre-publication detection
• Trigger: consent-ledger writes failing or disagreeing with the active recording state, privacy-seat or observer-tier attendees appearing in capture output, audience-cam cues firing into privacy sections (blocked path in RECORDING_CONSENT_FLOW.md observed open), minor without guardian confirmation present in a recorded session, or a redaction job silently failing.

First 15 minutes#

1. Page Lilith-Privacy lead and Concert capture on-call; open the bridge.
2. Stop the bleed, keep the show: pause master-capture publication and highlight publication immediately. The performance itself continues — the incident is in the capture/consent plane, not the show plane. If the consent system cannot be trusted at all, stop capture entirely (capture stop, not concert stop).
3. Quarantine all segments from the suspected failure window: mark read-only, block the editorial publication gate, and flag the C2PA signing queue to hold (C2PA_EVERY_EXPORT.md path) so no signed export leaves with tainted content.
4. Snapshot the consent ledger, privacy-seat ledger, and capture-worker logs for the window to the evidence bucket — the ledgers are themselves the evidence of what consent existed when.
5. Record operator.security.recording_privacy.opened.

• Steps: quarantine segments → reconstruct ground truth (cross-check consent ledger vs privacy-seat ledger vs attendance records from the DSAR category set in DSAR_V1_PIPELINE.md) → classify each quarantined segment as clean / redactable / destroy → run redaction re-render from the failure timestamp for redactable segments (the mid-session opt-out machinery in RECORDING_CONSENT_FLOW.md already supports timestamp-anchored redaction) → destroy unredactable segments with a signed destruction record → re-run the publication gate with a second privacy reviewer → notify affected attendees whose privacy election was violated, with what was captured and what was done. If captured data already left the platform (published highlight, LL-HLS crowd band), execute the takedown cascade via the takedown-executor operator action and assess the data-breach runbook trigger — published voice/likeness of a privacy-electing attendee is a personal-data exposure and inherits its notification matrix.
• Rollback: resume capture and publication only after the consent system passes its drill checks (V3/privacy/recording-consent-drills.json scenarios re-run green against production config) and the Privacy lead signs off in the Operator Console.
• Escalation: legal owner, Lilith-Rights lead (performer consent may also be implicated), Saraswati Editorial lead for the affected event, minor-safety reviewer whenever a minor is in scope (mandatory, regardless of severity).
• Evidence preservation: consent-ledger and privacy-seat-ledger snapshots, capture-worker logs, quarantined segment hashes, redaction-job inputs and outputs, destruction records, the C2PA hold queue state — all manifested with chain of custody. Quarantined segments are retained in the locked evidence bucket even when destroyed from production, until legal releases the hold.
• Audit event: operator.security.recording_privacy.handled
• Drill: consent-ledger write failure injected during a staged rehearsal concert before GA; evidence drill-recording-privacy-ga (owner: Lilith-Privacy lead)

Fail-Closed Criteria#

This runbook set is GA-blocking when any of the following holds:

• any of the four incidents lacks a named commander role, first-15-minutes checklist, evidence-preservation procedure, or audit event;
• the data-breach notification matrix does not cover every wave-1 region in V3/REGION_ROLLOUT_COMPLIANCE.md with an authority, deadline, owner, and template path;
• the JWT rotation drill evidence is missing or the staging drill did not measure validator JWKS cache-refresh lag;
• the breach-notice templates referenced in the matrix do not exist under V3/legal/templates/;
• any drill listed above is not recorded green before the GA release-readiness decision.