# Eve SOTA dense docs index — `eve.docs-dense-index.v1`

- **Task:** 3.3
- **Evaluated:** 2026-09-05
- **Bound model:** `perplexity/pplx-embed-v1-0.6b` at `perplexity/int8`,
  protocol `plain`, 1024 dims
- **Full index:** 57,008 rows × 1024 dims (222.7 MB), corpus `cc21e105c22d`,
  chunker `79148b760b1f`
- **Member index:** 0 rows — empty by construction, loader returns null
- **Record digest:**
  `0ca5c36ac6688e69151ca17942d94508a3ae646b7ac9a3a0c4bac5c3cbc52f8b`

## Why a file beside the corpus

pgvector on this host cannot compute an L2 or cosine distance even over a
three-dimensional literal, and exact cosine over the real corpus size in-process
costs tens of milliseconds per query — a fraction of the model turn it feeds.
The index therefore lives beside the lexical corpus with the same loader
discipline and no runtime service.

pgvector 0.8.2: L1 works = true; L2 failure = `stack-depth-limit-exceeded`;
cosine failure = `stack-depth-limit-exceeded`. Observed by
docs/audits/eve-sota-environment-capabilities/2026-09-01.json
(vector.pgvector-operators) and a direct psql reproduction on 2026-09-05:
'[1,2,3]'::vector <-> '[1,2,4]'::vector and <=> both raise "stack depth limit
exceeded" at max_stack_depth 2MB; <+> returns 1.

Trigger for an approximate structure: exact search is O(rows × dims) per query;
an order-of-magnitude larger corpus, or a query latency budget below the
measured median, is the point at which an approximate structure earns its
complexity (task 3.8 owns the operational half).

## The exact lexical corpus boundary

| Scope  | Corpus chunks | Pages | Corpus sha256  | Chunker        | Inventory hash | Rows   | Vectors                 | Chunk rows                 |
| ------ | ------------- | ----- | -------------- | -------------- | -------------- | ------ | ----------------------- | -------------------------- |
| member | 0             | 0     | `e3b0c44298fc` | `79148b760b1f` | `3d5f56907f27` | 0      | 0.0 MB `e3b0c44298fc`   | 0 B `e3b0c44298fc`         |
| full   | 57,008        | 3,256 | `cc21e105c22d` | `79148b760b1f` | `3d5f56907f27` | 57,008 | 222.7 MB `28481cc95f0a` | 7,933,623 B `86d02e94ccb7` |

Row i of a scope's vector matrix is chunk i of that scope's corpus file. The
chunker version is derived by the corpus builder from its own chunking source
and constants, so a rechunk shows up as a version change rather than as rows
that silently stopped lining up.

## Disclosure before retrieval

rows are exactly the chunks of this scope's lexical corpus, which the corpus
builder filters by its positive member opt-in (MEMBER_AUDIENCE_PREFIXES); a
member index is additionally refused if any href is not audience=member in
docs-graph-inventory.json

- Member rows: 0; member violations: 0
- Member loader returns null: true
- Audience filtered before retrieval on every scope: true
- The loader never overlays one scope onto another: true (proved by the spec of
  that name)

## Build, tombstones, freshness

| Scope  | Requests | Tokens    | Billed      | Duration | Batch | Pace   | Retried attempts | Tombstones                                                                        |
| ------ | -------- | --------- | ----------- | -------- | ----- | ------ | ---------------- | --------------------------------------------------------------------------------- |
| member | 0        | 0         | $0          | 0 s      | 100   | 300 ms | 0                | 0                                                                                 |
| full   | 571      | 8,836,447 | $0.03534579 | 527 s    | 100   | 300 ms | 0                | not computed — no previous build of this scope is present in the output directory |

### Re-check at generation

- **member:** fresh; loader null
- **full:** fresh; loader loaded; self-query top-1 is its own row: true; exact
  search 42.52 ms median / 61.68 ms max over 7 queries

`pnpm --dir apps/oshun/bff exec tsx scripts/build-docs-dense-index.ts --check`
exits 0 fresh, 1 stale, 2 corrupt. Stale means the corpus or the registry
binding moved; corrupt means the files no longer match their manifest. The
loader refuses both; the lexical path is the degraded mode.

## Persistence

Manifests are committed; vectors and chunk rows are local-only under these
ignore rules:

- `apps/oshun/bff/src/assistant/generated/docs-dense-index.*.vectors.f32`
- `apps/oshun/bff/src/assistant/generated/docs-dense-index.*.chunks.jsonl`

## Honest limits

- The full-scope vectors and chunk rows are local-only (gitignored, ~223 MB). A
  checkout without them records the re-check as not performed, with the reason,
  instead of implying it passed; the committed manifest still binds the corpus
  hash, the file hashes and the embedding identity, so a rebuild elsewhere
  either reproduces the same files or fails --check.
- Nothing serves this index yet. search_docs remains lexical BM25F; task 3.4
  decides hybrid fusion and the fail-loud modes for embedding, index, or
  provider loss. This record proves the index exists, binds what it claims, and
  searches — not that dense retrieval helps.
- Retrieval quality is unmeasured. The self-query proves a row retrieves itself;
  task 3.5 owns Recall@k and nDCG over the relevance set against the recorded
  shortlist, and task 3.7 owns promotion.
- Exact search is a documented approximation of a vector database, not a
  replacement for one: O(rows × dims) per query, all rows resident in memory, no
  incremental update. Tombstones are computed only when the previous build is
  present on the same machine, and the record says when they were not.
- The member index is empty by construction — no page has opted into the member
  audience — so the disclosure boundary is proved on the empty case and on the
  loader refusing to overlay scopes, not on a populated member index. If a page
  ever opts in, the member index must be rebuilt and the audience re-check
  re-run.
- The bound model's snapshot basis is a vendor version, not a dated slug or a
  published checkpoint. A re-pointed version invalidates every stored vector;
  --check catches a registry change but cannot see a silent re-point behind the
  same slug.
- Latency is the in-process scan on this host over unit-vector queries drawn
  from the index itself; it excludes the query embedding call, which is a
  network round trip the pricing record measured at ~200 ms median.