Probing & representation analysis for interpretability (Phase 177.4), in PyTorch,
CPU-runnable. Operates on activation tensors from @nous/interp-core.
Components#
- Linear probing (
linear.py, 177.4.1.1) — multinomial L2-regularised logistic regression trained to convergence with L-BFGS, K-fold cross-validation, percentile bootstrap CIs on accuracy, and per-layer / per-position sweeps to locate where a property becomes linearly decodable. - Representation similarity (
similarity.py, 177.4.1.2) — RSA (RDMs + Spearman correlation of their upper triangles) and CKA (linear + RBF-kernel, HSIC-normalised) for comparing checkpoints, model families, and model↔brain (NSD / Algonauts) representations. - TCAV (
tcav.py, 177.4.1.3) — concept activation vectors (unit normal of a concept-vs-random linear classifier), directional-derivative sensitivity, the TCAV score, statistical TCAV against random directions, and the causal concept effect (injectα·CAV, measure the real output change). - Emergent-capability probes (
emergent.py, 177.4.1.4) — track a probe's cross-validated accuracy across training checkpoints, flag the emergence checkpoint (threshold crossing) and the largest accuracy jump, for scaling-law capability-onset analysis.
Tests#
python3 -m pytest tests/ -q — 16 tests: probe recovers separable classes /
stays at chance on noise under strong L2, bootstrap CI brackets accuracy,
layer/position sweeps prefer the informative layer/position; RSA identity +
ordering, CKA orthogonal/scale invariance and low value for independent reps;
CAV recovers the concept axis, TCAV score high/low/chance by target alignment,
causal effect matches the analytic linear head; and emergence detection at the
decodability transition.