Eve library

Model cost for tests and harnesses — the full record

Guide

Revision 1

On this page

The binding rule lives in CLAUDE.md ("Test & harness model binding"). This file carries the measurements and the two corrections behind it, so the rule cannot be "optimised" without re-reading the evidence.

A model slug on OpenRouter is an auction, not an endpoint — price the ROUTE, not the model. /v1/models returns one pricing object per slug and it is not what you pay: deepseek/deepseek-v4-flash has eighteen providers behind it and OpenRouter picks per request. Measured against the live API on 2026-08-08 (/v1/models/<author>/<slug>/endpoints for the table, usage: { include: true } on a real call for the billed rate):

routing arm in /1M out /1M who served it
deepseek-v4-flash, no preference $0.074 $0.162 3 different providers in 3 calls
deepseek-v4-flash, sort: price $0.068 $0.137 StreamLake ×3 (fp8)
deepseek-v4-flash-0731, no preference $0.127 $0.246 3 different providers in 3 calls
deepseek-v4-flash-0731, sort: price $0.102 $0.180 DeepInfra ×3 (fp4)
claude-opus-5 $5.00 $25.00 —

Against Opus 5 that is still ~74x on input and ~182x on output at the cheap end, which is the whole reason for the rule.

Two corrections, both measured rather than argued (2026-08-08):

  1. An earlier version of the rule claimed the bare deepseek-v4-flash alias cost $0.14/$0.28 against the -0731 snapshot's $0.09/$0.18, and pinned -0731 to "save 1.56x". That was wrong — those were /v1/models summary fields, not routed prices. Priced properly, the alias is the CHEAPER of the two, because it has cheaper third-party endpoints. The pin made the harness ~1.5x dearer.
  2. The dominant lever is not the slug, it is provider routing, and we were not sending any. sort: "price" moves the same slug from $0.074/$0.162 to $0.068/$0.137 and keeps it on one provider instead of three.

Why -0731 stays bound anyway (user decision, 2026-08-08): this repo closes defects on Fisher exact tests over arms run in different sessions, and a slug whose model can change underneath it makes those arms incomparable. Buying reproducibility at 1.5x on a model already ~100x cheaper than the frontier is the trade the rule wants. Do not "optimise" this back to the alias without re-reading this paragraph.

Route re-measured 2026-08-16 (SMX P0 baseline session) — stale a third time, in our favor: /endpoints for -0731 now lists StreamLake fp8 at $0.061/$0.123 and Baidu fp8 at $0.062/$0.123 as the cheap end (DeepInfra fp4 moved to $0.080/$0.180). With sort: price + OPENROUTER_PROVIDER_QUANTIZATIONS=fp8 the SMX k=1 deck smoke was served by StreamLake + Baidu and billed $0.0432 across 121 real assistant turns (~$0.00036/turn, usage: {include: true}). The fp8 measurement pin is currently FREE on -0731 — the cheapest endpoints are already fp8, so pinning no longer costs the $0.130/$0.267 premium the 2026-08-08 note recorded.

Route re-measured 2026-08-30 — price-only was no longer a quality pin. The official endpoint feed listed a new cheapest fp8 endpoint, OpenInference, at $0.030/$0.140 per million. A current seven-case builder slice routed there and fell to 4/7: two required read tools were skipped and a card-gated write's arguments arrived as plain assistant JSON. On the same code, restricting the turn route to the three endpoints with retained successful builder measurements (baidu/fp8, deepinfra/fp8, streamlake/fp8) passed 7/7, served by Baidu and DeepInfra, at $0.0029 total with 5.2 s median latency and 71.4% cache-read. Baidu was the cheapest admitted endpoint at the capture ($0.04494/$0.08988 per million). The conclusion is narrower than “one provider is bad”: an auction can change quality when its price order changes, so the serving pin must include a measured endpoint set as well as slug and quantization. The raw before/after logs and hashes are retained in docs/audits/eve-phase-2/.

Operational notes:

  • OPENROUTER_PROVIDER_SORT=price is read by resolveOpenRouterProviderPreferences (libs/shared/ai), OpenRouter-only, OFF unless set. The assistant registry separately supplies its serving default, and OPENROUTER_PROVIDER_ONLY is the explicit endpoint-set override. Environment fields merge over independent registry safety fields rather than deleting the whole default object. Verified reaching the wire with a positive control: OPENROUTER_PROVIDER_QUANTIZATIONS=int2 comes back 400 provider.quantizations.0: Invalid option…, which OpenRouter could only say if the field arrived.
  • sort: price also buys provider STABILITY: -0731 went to DeepInfra 3/3 instead of three different providers, so quantization stops varying between arms without an explicit OPENROUTER_PROVIDER_QUANTIZATIONS floor (which on this slug would cost $0.130/$0.267 — more than plain -0731).
  • Pin quantization when MEASURING, not just spending. A Fisher arm run against fp8 and its comparison against fp4 are not the same experiment. Set OPENROUTER_PROVIDER_QUANTIZATIONS=fp8 for measurement work — free on the alias, whose cheapest endpoint is already fp8.
  • Price the route before trusting any table, including this one. It has gone stale twice. curl -s "https://openrouter.ai/api/v1/models/deepseek/deepseek-v4-flash/endpoints" -H "Authorization: Bearer $OPENROUTER_API_KEY" lists every provider with its own pricing/context/quantization; adding usage: { include: true } to a chat request returns the exact billed cost.
  • First-party is not cheapest. DeepSeek's own endpoint is $0.140/$0.280 on both slugs; the cheap end is third-party.
  • Start cheaper than you think you need (:free models where rate limits allow), and label findings by model so a weird reply can be re-judged without re-running everything.

Embeddings and reranking are on this route too — and the rules differ (2026-09-05)#

Measured while pricing the retrieval leg (Eve SOTA task 3.2; full table and record: docs/audits/EVE_SOTA_EMBEDDING_PRICING_2026-09.md).

The default model listing is a CHAT catalogue. GET /v1/models returns no embedding and no rerank model at all, so a candidate list assembled by recalling slugs is a list of whatever you happened to remember. Ask for them:

bash
curl -s "https://openrouter.ai/api/v1/models?output_modalities=embeddings" -H "Authorization: Bearer $OPENROUTER_API_KEY"
curl -s "https://openrouter.ai/api/v1/models?output_modalities=rerank"      -H "Authorization: Bearer $OPENROUTER_API_KEY"

Both POST /v1/embeddings and POST /v1/rerank exist, serve, and return a real usage.cost.

The fp8 pin is a CHAT-leg pin. Do not copy it here. On the embeddings route provider: { quantizations: ["fp8"] } returns 404 No endpoints found for the request with quantization: fp8 for most slugs — including openai/text-embedding-3-small and baai/bge-m3, whose endpoints declare unknown and fp32. On the one slug where an fp8 endpoint exists, qwen/qwen3-embedding-8b, the pin forces SiliconFlow and costs 4×: $0.04/1M pinned versus $0.01/1M at Nebius or DeepInfra. A pin copied across routes out of discipline is a disqualification and a cost wearing discipline's name; the registry now carries an explicit quantizationPinWaiver so dropping it has to be argued rather than noticed.

The catalogue price is wrong here in a new way. qwen/qwen3-reranker-8b lists pricing.prompt: "0" and billed a real amount on a real call. Price the route, as ever.

Reranking is not a rounding error beside embedding. Cohere bills per search unit (a flat $0.001 for one query against a document batch), token- priced rerankers bill per token, and either way one rerank costs thousands of times what embedding that same query costs. A reranker needs a quality argument, not a footnote.

Data posture is a routing filter you can measure. provider: { zdr: true } and provider: { data_collection: "deny" } either route — meaning at least one endpoint meets them — or 404 with the reason, which the router can only answer by checking every endpoint. :free routes are refused under both, and the refusal says Free model training. That is the cheapest available evidence about what a route does with submitted text.

An embedding leg has a constraint the chat legs do not. Its output is persisted and compared against vectors produced months earlier, so the endpoint and the query-side calling convention are part of the model's identity. Two endpoints serving one checkpoint at different quantizations return different vectors; several models publish an asymmetric query prefix and are being called wrong without it. Bind the slug, the endpoint set, and the protocol together.

Video generation is on this route too — and the pricing shape is different again (2026-09-09)#

Phase 182 routes video through the same OpenRouter account, and the chat rules above do not transfer. There is no per-token input/output pair, no /endpoints auction to price, and no sort: price lever: GET /api/v1/videos/models is public (no auth needed), returns 28 models, and gives one pricing_skus object per model that is heterogeneous by design. A single per-second rate cannot represent it, so cost estimation needs a normalizer.

The unit rule is derivable from the key name and nothing else: a key beginning cents_per_ is denominated in US cents; every other key is denominated in US dollars. runway/aleph-2's cents_per_second_output: "28" is $0.28/s; alibaba/wan-2.7's duration_seconds: "0.1" is $0.10/s. Mixing these up is a 100x error in either direction, and the catalog carries no unit field to check against.

The six SKU families (every key observed on 2026-09-09)#

Family Key stem Unit Qualifier order Seen on
A. Per output second (USD) duration_seconds USD / s [mode_]stem[_audio][_resolution] Wan, Veo, Kling, Hailuo, Sora, HeyGen, Happyhorse
B. Per output second (cents) cents_per_second_output, cents_per_video_output_second, cents_per_second_video_continuation cents / s stem[_resolution] Flux 3 Video, Runway, Grok Imagine
C. Per megapixel-second cents_per_megapixel_second cents / MP·s stem_qualitymode Flux Video Upscale
D. Per video token video_tokens USD / token stem[_resolution][_audio][_with_video_input] Every Seedance model
E. Per input unit cents_per_image_input, reference_images cents / img, USD / img none Grok Imagine, Hailuo 3
F. Per-generation floor minimum_cents_per_generation cents / generation none Runway Aleph 2

Qualifier vocabulary, in the order it appears in real keys:

  • mode prefix — text_to_video_, image_to_video_ (family A only)
  • audio — _with_audio, _without_audio
  • resolution — _480p, _720p, _768p, _1024p, _1080p, _4k
  • input — _with_video_input (family D only)
  • quality mode — _precise, _creative (family C only)
  • continuation is not a suffix: it is a different stem (cents_per_second_video_continuation_720p)

Note that families A and D put the qualifiers in different orders — duration_seconds_with_audio_4k is audio-then-resolution, while video_tokens_1080p_with_video_input is resolution-then-input. A normalizer that builds keys by concatenating suffixes in one fixed order will silently miss half the catalog, so resolution is by an ordered candidate list per family, most specific first, falling back to the bare stem.

The honest gap: token-priced models cannot be estimated before the call#

Family D prices per video_tokens, and the catalog carries no tokens-per- second, tokens-per-frame, or tokens-per-pixel constant. The token count is only known from the usage figures on the completed generation. Every Seedance model is priced this way, including bytedance/seedance-2.5 and bytedance/seedance-1-5-pro.

So an admission-time estimate for a token-priced model has exactly two honest options, and inventing a rate is not among them:

  1. Refuse to admit the request against a spend ceiling that requires a pre-flight estimate, and say why.
  2. Use a dated, measured tokens-per-second constant for that exact canonical_slug at that resolution and audio setting, recorded as a manual record with an expiry, and widen the estimate by the measured spread.

The second option needs a canary before it exists. Until then a token-priced route is admissible only under a hard per-job ceiling enforced after the fact, never as a "cheap" default.

Prices for the lanes Phase 182 actually routes (fetched 2026-09-09)#

Model canonical_slug Priced as 10 s cost
alibaba/wan-2.7 alibaba/wan-2.7-20260414 flat duration_seconds $0.10/s $1.00
alibaba/wan-2.6 alibaba/wan-2.6-20260327 mode × resolution: t2v 480p $0.04, t2v 720p $0.08, i2v 720p $0.10, t2v 1080p $0.12, i2v 1080p $0.15 $0.80–$1.50 (i2v 1080p $1.50)
bytedance/seedance-1-5-pro bytedance/seedance-1-5-pro-20260320 tokens: $2.4e-6 with audio, $1.2e-6 without not estimable pre-call
bytedance/seedance-2.5 bytedance/seedance-2.5-20260807 tokens: $1.07e-5, $6.4e-6 with video input not estimable pre-call
google/veo-3.1 google/veo-3.1-20260320 $0.40/s with audio, $0.20/s without; 4K $0.60/$0.40 $4.00 with audio (8 s max: $3.20)
google/veo-3.1-fast google/veo-3.1-fast-20260320 $0.12/s with audio, $0.10/s at 720p $1.20
google/veo-3.1-lite google/veo-3.1-lite-20260331 $0.08/s with audio, $0.05/s at 720p with audio $0.80
heygen/avatar-iv heygen/avatar-iv-20260625 flat duration_seconds $0.05/s $0.50
kwaivgi/kling-v3.0-pro kwaivgi/kling-v3.0-pro-20260429 $0.112/s, $0.168/s with audio $1.68 with audio
openai/sora-2-pro openai/sora-2-pro-20260320 $0.30/s at 720p, $0.50/s at 1024p/1080p $3.00–$5.00
runway/aleph-2 runway/aleph-2-20260729 $0.28/s with a $0.56 per-generation floor $2.80
minimax/hailuo-3 minimax/hailuo-03-20260730 $0.13/s plus $0.04 per reference image $1.30 + $0.04·n

The scale that matters for budgeting. One 10-second Wan 2.7 candidate costs $1.00. At the measured chat rate above (~$0.00036 per assistant turn on deepseek-v4-flash-0731), that single video candidate costs about as much as 2,700 harness turns. The chat rule saves ~100x against the frontier; the video rule is about not generating candidates you did not need. Production and hero tiers generate several candidates and may retry, so the per-request estimate is candidates × retries × seconds × rate, and the number that belongs on a dashboard is cost per releasable second, not cost per attempt (182.C.37.06).

Re-fetch before trusting any number above:

text
curl -sS https://openrouter.ai/api/v1/videos/models | jq '.data[] | {id, canonical_slug, pricing_skus}'

The catalog is mutable, the max age for a pricing fact is 24 h, and the revalidation cadence that binds it is in docs/domains/isis/human-video/implementation-tracker.md.