Disciplines · Runbooks

Concordia Incident Response

Anyone may report an incident.

7sections4 minread
Markdown source
On this page

title: Concordia Incident Response owner: Concordia program lead and Safety Operations version: 1.0.0 last_reviewed: 2026-07-15 next_review: 2026-10-13 applies_to: production and production-pilot Concordia workloads#

This runbook specializes the platform incident-response.md process for Concordia. It covers unsafe recommendations, privacy leakage, biased outcomes, unauthorized acceptance, execution mismatch, reproducibility failure, reviewer misconduct, and security events. Platform containment and notification duties still apply; this document adds mediation-specific actions and evidence rules.

Trigger and immediate authority#

Anyone may report an incident. The on-call operator creates an IncidentReport immediately and must not wait for severity agreement. P0 and P1 classification activates the Concordia kill switch for the affected tenant and capability; a privacy leak also pauses shared summaries, opposing-party prompts, exports, event subscriptions, and training reuse until scoped.

The safety reviewer is incident commander for unsafe, coercive, or biased outputs. The privacy lead commands privacy leakage. The platform security lead commands credential or isolation failures. The Concordia program lead commands unauthorized acceptance, execution mismatch, and reproducibility failure. A commander may expand containment but cannot unilaterally resume a P0/P1 path.

First 15 minutes#

  1. Record the report with tenant and case pseudonyms, detection source, affected artifact references, severity, confidentiality, and immutable timestamp.
  2. Pause the affected case. For a potentially systemic failure, kill the relevant model route, optimizer, domain adapter, acceptance, or execution flag across the tenant.
  3. Preserve the reproducibility manifest, model and policy pins, candidate digest, audit events, encrypted evidence references, and execution receipt. Never copy private statements into the incident summary.
  4. Stop external execution. Freeze escrow or settlement release through the supported pause/challenge mechanism; do not attempt an irreversible reversal without the execution owner and required signatories.
  5. Notify the incident commander, tenant owner, safety reviewer, privacy lead, and legal/compliance reviewer as the incident kind requires.

Severity#

  • P0: confirmed cross-party/private-data exposure at scale, systemic authority bypass, irreversible execution mismatch, active exploitation, or immediate threat to a participant.
  • P1: confirmed single-case sensitive disclosure, unsafe rights-affecting recommendation, acceptance without required approval, material biased outcome, or reproducibility failure affecting an accepted agreement.
  • P2: failure caught before acceptance with bounded impact, or a control degradation with no confirmed exposure.
  • P3: low-risk defect, false-positive investigation, or documentation gap with no participant impact.

When uncertain, use the higher severity until triage establishes otherwise.

Incident-specific containment#

Kind Required containment and evidence
Privacy leak Revoke affected access tokens and exports; quarantine derived embeddings and training rows; diff every shared projection against the private source; preserve access and subscription logs.
Unsafe recommendation Pause the case and related template/model route; preserve prompt, pins, redlines, boundary decision, and reviewer routing; provide human support without repeating unsafe content.
Biased outcome Freeze the candidate and its policy bundle; preserve cohort/fairness inputs and protected-attribute access logs; route an independent fairness review.
Unauthorized acceptance Stop execution; verify identity, delegation, authority, counsel/guardian requirements, and all signoffs; notify every affected party through the approved channel.
Execution mismatch Freeze remaining milestones; compare accepted candidate, signed draft, agreement hash, adapter payload, chain/escrow receipt, and reproducibility manifest.
Reproducibility failure Pin the environment; prohibit model/prompt/policy upgrades; replay from the manifest in isolation and retain the structured diff.
Reviewer misconduct Revoke the reviewer claim and access; preserve queue, conflict-of-interest, recusal, and signoff records; reassign to an independent reviewer.
Security event Invoke the platform security playbook, rotate credentials, preserve forensic logs, and treat possible private-context access as a privacy leak until disproven.

Recovery and closure#

Recovery requires a documented root cause, affected-case inventory, completed participant notification decision, tested remediation, replay or privacy verification, and a post-mortem artifact. P0/P1 closure requires two distinct authorized reviewers in the incident ledger. mitigated is not closed and does not satisfy the pilot gate.

Before resuming a killed capability, rerun the Concordia privacy, negative authorization, refusal, fairness, reproducibility, and domain-adapter gates relevant to the failure. The incident commander and a second qualified reviewer must sign the resume decision. Subsequent acceptance on an affected case retains the incident_followup cooling-off signal.

Pilot evidence rule#

For every §179.10.8 pilot, export an incident snapshot whose coverage spans the entire outcome-observation window. The export must include closed incidents as well as open ones, carry a source artifact identifier and reviewer attestation, and be evaluated through evaluatePilot. Any P0/P1 status other than closed blocks the pilot. Missing coverage is insufficient evidence, never a clean incident result.

Quarterly, the Concordia program lead reviews incident distributions, repeat causes, detection latency, notification decisions, and remediation durability; the signed findings are linked from the governance-review record.