---
artifact_kind: evaluator-card
artifact_name: fairness-suite
version: 1.0.0
owner: Nous · preference-inference team
last_reviewed: 2026-04-25
next_review: 2026-07-25
evaluator_kind: fairness
related_phase_179_tasks: [179.3.3.4, 179.3.3.5, 179.4.3, 179.8.2.1]
---

# Evaluator Card — fairness-suite v1.0.0

## 1. Identity

- **Evaluator name:** fairness-suite
- **What it measures:** the cross-party fairness vector for any candidate or
  accepted agreement: Nash product, utilitarian sum, max-min utility,
  egalitarian welfare, Kalai-Smorodinsky proportional gains, envy, regret,
  inequality, burden symmetry, procedural dignity.
- **Harm families covered (H1–H8):** H4 (fairness), H5 (safety — guards against
  laundering redlines into utility), H6 (accountability — emits a structured
  fairness vector for every candidate the workbench surfaces).
- **Consumers:** every search kernel under `@nous/agreement-search`; the
  workbench frontier renderer (§179.4.3.1); the §179.10.6 baseline benchmark
  gate; the spec-gaming audit (`spec-gaming-audit.ts`).

## 2. Method

- **Input corpus / fixture:** any set of `partyUtilities` (per-party utility on
  a common scale) optionally accompanied by `partyBATNAs` and
  `partyAspirations`. Suites are derived from the §179.8.1 benchmarks.
- **Procedure:** for each candidate, compute every `FairnessMetricKind` value
  via the pure functions in
  `libs/nous/preference-inference/src/fairness-metrics.ts` and emit the vector
  along with optional per-party / per-component breakdowns.
- **Metrics emitted:**

  | Metric                       | Definition                                                               |
  | ---------------------------- | ------------------------------------------------------------------------ |
  | `nash_product`               | `Π_i max(0, u_i − BATNA_i)` (Nash 1950)                                  |
  | `utilitarian_sum`            | `Σ_i u_i` (Bentham)                                                      |
  | `max_min_utility`            | `min_i u_i` (Rawls)                                                      |
  | `egalitarian_welfare`        | equalized `u_i − BATNA_i` (Kalai 1977)                                   |
  | `kalai_smorodinsky_distance` | distance from KS solution along `(u*_i − BATNA_i)` ray                   |
  | `envy`                       | sum of pairwise envy (Foley 1967)                                        |
  | `regret`                     | `Σ_i (u*_i − u_i)`                                                       |
  | `inequality`                 | Atkinson-style coefficient of variation                                  |
  | `burden_symmetry`            | mean absolute deviation of normalized concession                         |
  | `procedural_dignity`         | mean of caller-supplied per-party procedural-dignity scalars in `[0, 1]` |

- **Aggregation:** the workbench renders the full vector. The release-gate
  aggregator picks the metric appropriate to the use-case class:
  - `low_stakes_personal`, `creative_collaboration` — Nash product + envy +
    procedural dignity
  - `commerce_negotiation`, `supplier_procurement` — Nash product + burden
    symmetry + KS distance
  - `dao_governance` — egalitarian welfare + max-min utility + envy
  - `community_restorative` — procedural dignity + burden symmetry + KS distance
  - `legal_claim`, `medical_sensitive`, `regulated_financial`,
    `family_sensitive`, `employment_sensitive` — full vector with qualified
    human reviewer in the loop

## 3. Baselines

- **Comparison baselines:** llm-mediator, nash-genetic, static-templates,
  seed-offer-only — same as the
  [baseline-benchmark-gate](./baseline-benchmark-gate.md).
- **Baseline scores:** evaluated per release on the §179.8.1 corpus and rolled
  up by the [baseline-benchmark-gate](./baseline-benchmark-gate.md). The Nash
  baseline reports only Nash product; the static-templates baseline reports a
  degenerate dignity score of 1.0 by construction (templates do not measure
  procedural fairness); these limitations are part of the baseline definition.

## 4. Thresholds

The fairness suite reports values; the release-gate threshold is encoded per
use-case class in `USE_CASE_CLASS_PROFILES`:

| Class                     | Required minima                                           |
| ------------------------- | --------------------------------------------------------- |
| `low_stakes_personal`     | Nash product > 0; envy ≤ 0.10                             |
| `community_restorative`   | procedural dignity ≥ 0.70; burden symmetry ≤ 0.20         |
| `creative_collaboration`  | Nash product > 0; envy ≤ 0.10                             |
| `commerce_negotiation`    | Nash product > 0; KS distance ≤ 0.20                      |
| `supplier_procurement`    | Nash product > 0; burden symmetry ≤ 0.15                  |
| `dao_governance`          | egalitarian welfare > 0; max-min utility > 0; envy ≤ 0.10 |
| `platform_dispute`        | Nash product > 0; envy ≤ 0.15                             |
| `employment_sensitive`    | qualified human reviewer must approve full vector         |
| `family_sensitive`        | qualified human reviewer must approve full vector         |
| `legal_claim`             | counsel must approve full vector                          |
| `regulated_financial`     | counsel + compliance must approve full vector             |
| `medical_sensitive`       | counsel + safety reviewer must approve full vector        |
| `agent_to_agent_contract` | Nash product > 0; KS distance ≤ 0.25                      |
| `simulation_only`         | informational; no gate                                    |

**Regression threshold:** any release where a previously-passing candidate fails
a minimum requires manual review.

## 5. Known limitations

- Procedural-dignity scalars are caller-supplied; their honesty is the workbench
  operator's responsibility. The metric does not detect fabricated dignity
  scores.
- BATNA defaults to 0 when omitted; this can inflate the Nash product for cases
  where the true BATNA is positive. The `batna-plausibility.ts` module guards
  against unsupported BATNA inflation.
- Aspiration defaults to the candidate's own utility for that party when
  omitted, which yields zero regret by construction; meaningful regret requires
  the caller to pass aspirations from the broader candidate set.
- Demographic / cultural fairness profiles (`fairness-profiles.ts`) are inputs
  to weighting; they do not override the no-protected-class-disparate-treatment
  rule.

## 6. Validation

- **Last validation date:** 2026-04-25.
- **Validation procedure:** every metric in `fairness-metrics.ts` has a unit
  test in `fairness-metrics.test.ts` against hand-computed known-correct values.
  Cross-references: Nash product against closed-form `Π (u_i − d_i)`; KS
  distance against the textbook KS solution; Foley envy against pairwise envy
  enumeration.

## 7. References

- `libs/nous/preference-inference/src/fairness-metrics.ts`
- `libs/nous/preference-inference/src/fairness-profiles.ts`
- Nash (1950), "The bargaining problem".
- Kalai, Smorodinsky (1975), "Other solutions to Nash's bargaining problem".
- Foley (1967), "Resource allocation and the public sector".
- `docs/research/bibliography.md` §fairness-metrics