# @nous/autoresearch-evals

Autoresearch benchmarks, observability & experiment ledger for the Phase-178
stack (TODOS/phase-178 §178.9).

## Components

| Module             | Summary                                                                                                                                                                                                                                                                                                                        |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `harness.py`       | An `inspect_ai`-shaped **Task / Sample / Solver / Scorer** harness + `EvalRunner`, a named-suite registry (MLE-Bench, SWE-Bench Verified, RE-Bench, PaperBench, ScienceAgentBench, DiscoveryBench, GAIA, Cybench, CTF, HLE wired by reference), and an `inspect_ai_available()` probe to delegate when installed. (§178.9.1.1) |
| `suites.py`        | **KarpathyLoopBench** (delta reproduction), **AlphaEvolveMathBench** (50-problem deterministic sweep), **OshunDomainBench** (internal multi-domain Q&A). (§178.9.1.2)                                                                                                                                                          |
| `leaderboard.py`   | **Continuous-leaderboard publisher** — per-checkpoint submissions to a Phase-91 hub sink with regression alarms and promotion gating. (§178.9.1.3)                                                                                                                                                                             |
| `observability.py` | **Dual Langfuse + W&B Weave** exporters (real payload mappers) over a fail-loud transport. (§178.9.2.1)                                                                                                                                                                                                                        |
| `experiments.py`   | **Experiments-as-first-class diffing** — per-prompt output/score/cost/latency/token deltas + side-by-side report. (§178.9.2.2)                                                                                                                                                                                                 |
| `ledger.py`        | **Canonical hash-chained + HMAC-signed** experiment ledger; tampering breaks the chain, forging breaks the signature. (§178.9.2.3)                                                                                                                                                                                             |
| `sandbox_pool.py`  | **SWE-ReX-class parallel sandbox pool** — bounded workers, per-job resource envelope (fs/network/cpu/gpu/wall), isolated failures. (§178.9.2.4)                                                                                                                                                                                |
| `flywheel.py`      | **Autoresearch-outcomes collector** — `(prompt, edit, delta, accepted)` tuples → weighted preference training records. (§178.9.2.5)                                                                                                                                                                                            |

External benchmark data (Kaggle competitions, SWE-Bench repos) and the real
Langfuse/Weave/inspect_ai backends are plug-in seams; the harness, suites,
leaderboard gating, payload mapping, diffing, signed ledger, pool and flywheel
logic are fully local and tested.

## Testing

```bash
python3 -m pytest tests/ -q     # 19 tests
python3 -m ruff check .
```