---
artifact_kind: evaluator-card
artifact_name: baseline-benchmark-gate
version: 1.0.0
owner: Nous · agreement-search team
last_reviewed: 2026-04-25
next_review: 2026-07-25
evaluator_kind: benchmark
related_phase_179_tasks: [179.8.1, 179.8.2, 179.10.6]
---

# Evaluator Card — baseline-benchmark-gate v1.0.0

## 1. Identity

- **Evaluator name:** baseline-benchmark-gate
- **What it measures:** whether the Concordia stack outperforms each of four
  reference baselines on at least the configured number of domain suites, using
  paired Welch's t tests on per-case KPI values.
- **Harm families covered (H1–H8):** H4 (fairness — measures whether Concordia's
  outputs are at least as fair as baselines on every domain), H6 (accountability
  — emits significance, effect size, and win count for audit).
- **Consumers:** the §179.10.6 release gate; the §179.10.7 governance review
  (this card); the §179.8.3.5 internal leaderboard.

## 2. Method

- **Input corpus / fixture:** the eight benchmark suites in
  `libs/contracts/concordia/src/__benchmarks__/benchmarks.ts`: `two_party`,
  `multi_party`, `procurement`, `legal_low_stakes`, `dao`, `creative`,
  `restorative`, `agent_to_agent`. Each suite is a deterministic scenario set
  with checked-in party statements and expected outcomes.
- **Procedure:**
  1. For every (suite, baseline, KPI) combination, run Concordia and the
     baseline on the same case set; record paired
     `(concordiaValue, baselineValue)` samples.
  2. Compute the paired difference, run Welch's t test, and emit Cohen's d
     effect size.
  3. KPI direction is encoded per suite — for higher-is-better KPIs (Nash
     product, max-min utility, agreement rate) a positive paired delta at p <
     0.05 is a "win"; for lower-is-better KPIs (latency, cost, dispute rate) a
     negative paired delta at p < 0.05 is a "win".
- **Metrics emitted:** per (suite, baseline) — t statistic, p value, Cohen's d,
  win/loss verdict. Per baseline — total wins across the eight suites. Gate
  verdict — pass / fail per baseline.
- **Aggregation:** the gate passes when, for _every_ baseline, Concordia wins on
  at least `minSuiteWinsPerBaseline` suites (default 6 of 8).

## 3. Baselines

- **Comparison baselines** (`BaselineKindSchema`):
  - `llm_mediator_baseline` — single-LLM mediator with no formal optimization
  - `nash_genetic_baseline` — Mediator.ai-style Nash-product genetic search
    without uncertainty calibration
  - `static_templates_baseline` — rule-based templates with no party-utility
    inference
  - `seed_offer_only_baseline` — human-authored seed offer with no automated
    counterproposals
- **Baseline scores:** materialized in
  `libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.test.ts`
  — every release runs the full matrix and the test asserts the gate pass.

## 4. Thresholds

| Threshold                           | Value                                   |
| ----------------------------------- | --------------------------------------- |
| `minSuiteWinsPerBaseline` (default) | 6 of 8                                  |
| `minSuiteWinsPerBaseline` (range)   | 1..8 (`baseline-benchmark-gate.ts:113`) |
| Significance level                  | p < 0.05                                |
| Per-suite minimum sample size       | enforced by the suite's case            |
|                                     | manifest                                |

**Regression threshold:** loss of one suite for any baseline triggers review.
Loss of more than one suite blocks release.

**Domain-specific thresholds:** tenants may raise `minSuiteWinsPerBaseline` to 7
or 8 for their environment; not lower.

## 5. Known limitations

- Synthetic benchmark scenarios are deterministic and do not capture the full
  distribution of real cases; pilot KPIs are tracked independently in §179.10.8.
- Welch's t test assumes paired samples are independent within a suite; this
  holds for the synthetic scenarios but should be re-validated when production
  traces enter the corpus.
- Cohen's d is reported but not gated; a statistically significant but
  small-effect win still counts as a "win" under the current rule.

## 6. Validation

- **Last validation date:** 2026-04-25.
- **Validation procedure:** every Concordia release re-runs the gate via
  `pnpm vitest run libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.test.ts`.
  The reference Welch's t-statistic, t-table, and gamma-function approximations
  are unit-tested in the same file (the t-table values at the file lines 287–302
  are checked against published references).

## 7. References

- `libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.ts`
- `libs/contracts/concordia/src/__benchmarks__/benchmarks.ts`
- `libs/contracts/concordia/src/__benchmarks__/mediator-ai-scenarios.ts`
- `testing/concordia/benchmarks/benchmarks.ts` — re-export surface
- `docs/research/bibliography.md` §benchmark-gating