Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris, eng-isis,
eng-shared Backlog item: V1-P2-0064 Audience: every consumer that
needs structured visual reasoning over an image — iris/agents/computer-use
(screen understanding, click-target localisation), the OshunOCRClient Tier 2
escalation path (V1-P2-0060), accessibility audits in
psyche/screen-analysis, and product surfaces that ask Claude to "look at this
screen and tell me what to do."
1. Decision#
OSHUN's vision-LLM access is single-pipe, multi-tenant routing through the
canonical IsisLLMClient (delivered by V1-P2-0002/V1-P2-0003). Every vision
call goes through the gateway so token / pixel / cost accounting, rate limiting,
retries, and per-tenant quotas are uniform. No direct calls to provider SDKs are
permitted.
The vision-capable model fleet is a two-provider primary + on-prem fallback:
- Primary — Anthropic Claude vision.
claude-sonnet-4-6-vision(default for any request whose downstream consumer is product-quality) andclaude-haiku-4-5-vision(default for high-volume agent loops where the sub-second latency floor matters more than peak-quality reasoning). Both share the same Anthropic Messages API surface as text models, soIsisLLMClientexposes a singlemessages.create()call site that acceptsimage_url | image_base64 | image_pathparts in the user message — there is no separate vision-only client. - Secondary — OpenAI vision.
gpt-4.1-vision(default for users on OpenAI-only tenants and for any consumer whose downstream evaluation is already calibrated against OpenAI multi-modal outputs). Routed through the sameIsisLLMClientwith provider hintprovider: 'openai'. - On-prem fallback — Florence-2 / LLaVA-Next via local inference. Used only when (a) the tenant requires data residency that forbids any egress to Anthropic or OpenAI, (b) the gateway has tripped its provider circuit breaker and is in the "vendor outage" branch, or (c) the downstream consumer can tolerate ~3× higher latency for the data-locality guarantee.
Switching tiers is explicit per call via the provider and vision_quality
request fields in IsisLLMRequest; routing is not silent because Anthropic and
OpenAI tokenisers and pricing differ and because consumers with
data_residency: 'on_prem' MUST NOT cascade to Anthropic/OpenAI.
OshunVisionLLMClient (V1-P2-0065) is a thin convenience wrapper over
IsisLLMClient — it does not own any provider knowledge, it owns the
prompt-shape contract for vision tasks (the prompt template, the response
schema, the bounding-box coordinate convention, the per-task verifier).
2. Rationale#
2.1 Why route through IsisLLMClient#
OSHUN already has a backlog (V1-P2-0002 … V1-P2-0027) that delivers a
canonical LLM gateway: provider registry, rate limiter, sliding-window quotas,
microcent cost accounting (@oshun/contracts/llm), retry executor, circuit
breaker, OpenTelemetry instrumentation, and an error taxonomy with stable codes.
Every property on that list is also required for vision calls. Building a
parallel "vision client" that re-implements quotas / retries / cost would be
exactly the duplication the audit at
docs/releases/p2/isis-llm-gateway-audit.md was meant to prevent.
The vision-specific concerns — image MIME inference, multi-image inputs,
pixel-token estimation, output-format coercion (JSON / coordinates) — are real,
but they are input/output adapters around the gateway, not parallel
transport. The OshunVisionLLMClient wrapper owns those adapters.
2.2 Why Claude vision is primary#
Three properties drive the default:
- Best-in-class structured visual reasoning. Claude vision excels at the workhorse query for OSHUN ("which UI element labelled X is the submit button?", "describe what the user is trying to accomplish in this screenshot") because its tool-use mode pairs naturally with the ImageContent block, allowing the model to return JSON-shaped action plans without prompt-engineering acrobatics.
- Tokenisation & cost predictability. Anthropic publishes a deterministic
image-token formula (
tokens ≈ (width * height) / 750) which the existing@isis/llm-providerstoken counter already models pixel-wise; that integration cost is paid. - Same SDK surface as text. The Messages API uses the same
content: [...]block list for text and image inputs, so theIsisLLMRequestschema doesn't need a separate vision-only branch.
2.3 Why OpenAI vision is secondary, not primary#
GPT-4.1 vision matches Claude on most benchmarks and beats it on chart understanding and dense-text-in-image (where cloud OCR Tier 3 would win anyway). It is kept as a routable secondary because:
- some tenants are contractually OpenAI-only (legal/compliance posture);
- accessibility audits in
psyche/screen-analysiswere originally calibrated against GPT-4.1 vision and we don't want to re-baseline beforeV1-P2-0066lands; - the
IsisLLMClientprovider-fallback chain needs a non-Anthropic branch to be a real fallback; otherwise Anthropic outages cascade into full vision outages across OSHUN.
Vision parity is close enough that defaulting to OpenAI for any particular consumer is a configuration toggle, not a re-architecture.
2.4 Why an on-prem fallback at all#
The audit already documents data_residency as a first-class constraint. Any
consumer that flips that flag must NOT cascade to Anthropic or OpenAI even when
the gateway is healthy. Florence-2 (Microsoft, MIT-license) and LLaVA-Next (open
weights) are the two production-grade on-prem candidates today; we pick at
deployment time based on the tenant's GPU profile and let the consumer treat
both as "tier3 vision". They are explicitly not a quality default — only the
data-locality default.
2.5 What we explicitly rejected#
- Google Gemini Vision. Quality is competitive but the tokenisation & pricing surface diverges from both Anthropic and OpenAI; adding it now means three provider-specific cost adapters, three different image token formulas, three different streaming envelopes. Revisit when / if a tenant requires it.
- A standalone "OshunVisionClient" with its own transport. Would reintroduce
the audit's complaint of parallel transport stacks; the shared cross-cutting
concerns (quotas, retries, OTEL) all live in
IsisLLMClient. - Synchronous vision-LLM call inside the agent loop's hot path. Even at
Haiku latency (~400 ms median), making every screenshot OCR call route through
a vision LLM would burn the cost budget. The OCR strategy doc (
V1-P2-0060) keeps Tier 1 Tesseract on the hot path and routes here only on confidence < 0.65 (the documented selective-escalation pattern).
3. Per-Consumer Defaults#
| Consumer | Default | Reason |
|---|---|---|
iris/agents/computer-use/vision/screen-analyzer.ts |
claude-haiku-4-5-vision |
Latency floor matters more than peak quality on the agent inner loop. |
iris/agents/computer-use/vision/index.ts (high-level "find the submit button") |
claude-sonnet-4-6-vision |
Structured visual reasoning is the actual product; quality > latency. |
Tier 2 OCR escalation (OshunOCRClient low-conf fallback) |
claude-haiku-4-5-vision |
Already on a low-frequency escalation path; minimise per-call cost. |
psyche/screen-analysis accessibility audits |
gpt-4.1-vision |
Existing baseline calibrated against OpenAI; switching pre-V1-P2-0066 invalidates A/B comparisons. |
Any tenant with data_residency: 'on_prem' |
florence-2 (or llava-next) |
Compliance constraint; NOT cascadable. |
| Claude-only tenants | claude-sonnet-4-6-vision |
Quality default when one-vendor lock-in is the contract. |
4. Common Request Envelope#
Vision requests piggyback on IsisLLMRequest:
import type {
IsisLLMRequest,
IsisLLMMessageContent,
} from '@oshun/contracts/llm';
// V1-P2-0064 vision input contract (referenced from V1-P2-0065)
export interface VisionImagePart {
readonly type: 'image';
readonly source:
| {
kind: 'base64';
media_type: 'image/png' | 'image/jpeg' | 'image/webp';
data: string;
}
| { kind: 'url'; url: string };
}
// Mixed text + image content. Order is preserved end-to-end so
// captions referring to "the screenshot above" work.
export type IsisLLMVisionContent = ReadonlyArray<
IsisLLMMessageContent | VisionImagePart
>;
export interface VisionExtraOptions {
/** 'low' | 'high' — maps to provider-specific quality flags. */
readonly quality?: 'low' | 'high';
/**
* If set, the model MUST return JSON conforming to this JSON schema.
* Coerced via tool-use on Anthropic, response_format on OpenAI, and
* a constrained-decoding pass on the on-prem fallback.
*/
readonly responseSchema?: Record<string, unknown>;
/**
* Bounding-box coordinate convention. Defaults to 'pixels' (origin
* top-left). 'normalised' produces 0..1 box coords, useful when
* the consumer doesn't know the image dimensions yet.
*/
readonly coordinateSystem?: 'pixels' | 'normalised';
}
Image inputs are normalised by OshunVisionLLMClient before they hit the
gateway — base64 / URL / Buffer / file path → the provider-specific source
shape — so consumers never have to think about the provider's wire format.
5. Output Contract (V1-P2-0065 implements)#
OshunVisionLLMClient.locate(prompt, image) returns:
export interface VisionLocateResult {
/** The model's reasoning text. */
readonly text: string;
/**
* Bounding boxes the model claims contain the requested element.
* Empty when the model couldn't find a match — callers MUST handle
* this, as the model is instructed to return [] rather than
* hallucinate coordinates.
*/
readonly boxes: readonly BoundingBox[];
/** 0..1 — model self-reported confidence. */
readonly confidence: number;
/** Provider model id used for this call. */
readonly model: string;
readonly latencyMs: number;
/** Microcents per OSHUN canonical pricing unit. */
readonly costMicrocents: number;
}
Verification gate V1-P2-0065: a "find the submit button" probe must return
coordinates within the button bounds 9 of 10 trials.
6. Verification Gates#
V1-P2-0064— this strategy doc lives atdocs/releases/p2/vision-llm-strategy.md.V1-P2-0065—OshunVisionLLMClient.locate()returns coords within button bounds ≥ 9/10 trials on a fixed screenshot fixture set.V1-P2-0066— document layout analysis MUST clear mAP ≥ 0.6 on the PubLayNet sample (separate model — LayoutLMv3 ONNX or vision LLM — chosen at that backlog item).V1-P2-0068— every call site inlibs/iris/agents/computer-use/vision/callsOshunVisionLLMClient(not a parallel stub).
7. Future Revisits#
- Multi-image inputs at scale. Anthropic and OpenAI both support multi-image
content arrays today; the gateway currently sees only single-image traffic in
flight. When a consumer requires N-image-per-request reasoning (e.g.
multi-frame video diff), the per-image token estimator and the cost-accounting
code in
@isis/llm-providersneed a multi-image audit. - Vision tool-use loops. Once
iris/agents/computer-usegraduates from "single-screen action" to "multi-step planning over a sequence of screens," we'll need to formalise vision-context retention so the model doesn't re-pay the per-image token cost on every step. Likely shape: avision_sessionopaque token returned by the first call and reusable in subsequent calls. - On-prem model swap. If a license-clean Florence-3 or successor ships with stronger structured-output support, swap the default on-prem fallback. The contract surface in §4–§5 is the abstraction that lets us do that without any consumer changes.