Eve library

Vision LLM Strategy Decision

Guide

Revision 1

On this page

Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris, eng-isis, eng-shared Backlog item: V1-P2-0064 Audience: every consumer that needs structured visual reasoning over an image — iris/agents/computer-use (screen understanding, click-target localisation), the OshunOCRClient Tier 2 escalation path (V1-P2-0060), accessibility audits in psyche/screen-analysis, and product surfaces that ask Claude to "look at this screen and tell me what to do."


1. Decision#

OSHUN's vision-LLM access is single-pipe, multi-tenant routing through the canonical IsisLLMClient (delivered by V1-P2-0002/V1-P2-0003). Every vision call goes through the gateway so token / pixel / cost accounting, rate limiting, retries, and per-tenant quotas are uniform. No direct calls to provider SDKs are permitted.

The vision-capable model fleet is a two-provider primary + on-prem fallback:

  1. Primary — Anthropic Claude vision. claude-sonnet-4-6-vision (default for any request whose downstream consumer is product-quality) and claude-haiku-4-5-vision (default for high-volume agent loops where the sub-second latency floor matters more than peak-quality reasoning). Both share the same Anthropic Messages API surface as text models, so IsisLLMClient exposes a single messages.create() call site that accepts image_url | image_base64 | image_path parts in the user message — there is no separate vision-only client.
  2. Secondary — OpenAI vision. gpt-4.1-vision (default for users on OpenAI-only tenants and for any consumer whose downstream evaluation is already calibrated against OpenAI multi-modal outputs). Routed through the same IsisLLMClient with provider hint provider: 'openai'.
  3. On-prem fallback — Florence-2 / LLaVA-Next via local inference. Used only when (a) the tenant requires data residency that forbids any egress to Anthropic or OpenAI, (b) the gateway has tripped its provider circuit breaker and is in the "vendor outage" branch, or (c) the downstream consumer can tolerate ~3× higher latency for the data-locality guarantee.

Switching tiers is explicit per call via the provider and vision_quality request fields in IsisLLMRequest; routing is not silent because Anthropic and OpenAI tokenisers and pricing differ and because consumers with data_residency: 'on_prem' MUST NOT cascade to Anthropic/OpenAI.

OshunVisionLLMClient (V1-P2-0065) is a thin convenience wrapper over IsisLLMClient — it does not own any provider knowledge, it owns the prompt-shape contract for vision tasks (the prompt template, the response schema, the bounding-box coordinate convention, the per-task verifier).


2. Rationale#

2.1 Why route through IsisLLMClient#

OSHUN already has a backlog (V1-P2-0002 … V1-P2-0027) that delivers a canonical LLM gateway: provider registry, rate limiter, sliding-window quotas, microcent cost accounting (@oshun/contracts/llm), retry executor, circuit breaker, OpenTelemetry instrumentation, and an error taxonomy with stable codes. Every property on that list is also required for vision calls. Building a parallel "vision client" that re-implements quotas / retries / cost would be exactly the duplication the audit at docs/releases/p2/isis-llm-gateway-audit.md was meant to prevent.

The vision-specific concerns — image MIME inference, multi-image inputs, pixel-token estimation, output-format coercion (JSON / coordinates) — are real, but they are input/output adapters around the gateway, not parallel transport. The OshunVisionLLMClient wrapper owns those adapters.

2.2 Why Claude vision is primary#

Three properties drive the default:

  • Best-in-class structured visual reasoning. Claude vision excels at the workhorse query for OSHUN ("which UI element labelled X is the submit button?", "describe what the user is trying to accomplish in this screenshot") because its tool-use mode pairs naturally with the ImageContent block, allowing the model to return JSON-shaped action plans without prompt-engineering acrobatics.
  • Tokenisation & cost predictability. Anthropic publishes a deterministic image-token formula (tokens ≈ (width * height) / 750) which the existing @isis/llm-providers token counter already models pixel-wise; that integration cost is paid.
  • Same SDK surface as text. The Messages API uses the same content: [...] block list for text and image inputs, so the IsisLLMRequest schema doesn't need a separate vision-only branch.

2.3 Why OpenAI vision is secondary, not primary#

GPT-4.1 vision matches Claude on most benchmarks and beats it on chart understanding and dense-text-in-image (where cloud OCR Tier 3 would win anyway). It is kept as a routable secondary because:

  • some tenants are contractually OpenAI-only (legal/compliance posture);
  • accessibility audits in psyche/screen-analysis were originally calibrated against GPT-4.1 vision and we don't want to re-baseline before V1-P2-0066 lands;
  • the IsisLLMClient provider-fallback chain needs a non-Anthropic branch to be a real fallback; otherwise Anthropic outages cascade into full vision outages across OSHUN.

Vision parity is close enough that defaulting to OpenAI for any particular consumer is a configuration toggle, not a re-architecture.

2.4 Why an on-prem fallback at all#

The audit already documents data_residency as a first-class constraint. Any consumer that flips that flag must NOT cascade to Anthropic or OpenAI even when the gateway is healthy. Florence-2 (Microsoft, MIT-license) and LLaVA-Next (open weights) are the two production-grade on-prem candidates today; we pick at deployment time based on the tenant's GPU profile and let the consumer treat both as "tier3 vision". They are explicitly not a quality default — only the data-locality default.

2.5 What we explicitly rejected#

  • Google Gemini Vision. Quality is competitive but the tokenisation & pricing surface diverges from both Anthropic and OpenAI; adding it now means three provider-specific cost adapters, three different image token formulas, three different streaming envelopes. Revisit when / if a tenant requires it.
  • A standalone "OshunVisionClient" with its own transport. Would reintroduce the audit's complaint of parallel transport stacks; the shared cross-cutting concerns (quotas, retries, OTEL) all live in IsisLLMClient.
  • Synchronous vision-LLM call inside the agent loop's hot path. Even at Haiku latency (~400 ms median), making every screenshot OCR call route through a vision LLM would burn the cost budget. The OCR strategy doc (V1-P2-0060) keeps Tier 1 Tesseract on the hot path and routes here only on confidence < 0.65 (the documented selective-escalation pattern).

3. Per-Consumer Defaults#

Consumer Default Reason
iris/agents/computer-use/vision/screen-analyzer.ts claude-haiku-4-5-vision Latency floor matters more than peak quality on the agent inner loop.
iris/agents/computer-use/vision/index.ts (high-level "find the submit button") claude-sonnet-4-6-vision Structured visual reasoning is the actual product; quality > latency.
Tier 2 OCR escalation (OshunOCRClient low-conf fallback) claude-haiku-4-5-vision Already on a low-frequency escalation path; minimise per-call cost.
psyche/screen-analysis accessibility audits gpt-4.1-vision Existing baseline calibrated against OpenAI; switching pre-V1-P2-0066 invalidates A/B comparisons.
Any tenant with data_residency: 'on_prem' florence-2 (or llava-next) Compliance constraint; NOT cascadable.
Claude-only tenants claude-sonnet-4-6-vision Quality default when one-vendor lock-in is the contract.

4. Common Request Envelope#

Vision requests piggyback on IsisLLMRequest:

ts
import type {
  IsisLLMRequest,
  IsisLLMMessageContent,
} from '@oshun/contracts/llm';

// V1-P2-0064 vision input contract (referenced from V1-P2-0065)
export interface VisionImagePart {
  readonly type: 'image';
  readonly source:
    | {
        kind: 'base64';
        media_type: 'image/png' | 'image/jpeg' | 'image/webp';
        data: string;
      }
    | { kind: 'url'; url: string };
}

// Mixed text + image content. Order is preserved end-to-end so
// captions referring to "the screenshot above" work.
export type IsisLLMVisionContent = ReadonlyArray<
  IsisLLMMessageContent | VisionImagePart
>;

export interface VisionExtraOptions {
  /** 'low' | 'high' — maps to provider-specific quality flags. */
  readonly quality?: 'low' | 'high';
  /**
   * If set, the model MUST return JSON conforming to this JSON schema.
   * Coerced via tool-use on Anthropic, response_format on OpenAI, and
   * a constrained-decoding pass on the on-prem fallback.
   */
  readonly responseSchema?: Record<string, unknown>;
  /**
   * Bounding-box coordinate convention. Defaults to 'pixels' (origin
   * top-left). 'normalised' produces 0..1 box coords, useful when
   * the consumer doesn't know the image dimensions yet.
   */
  readonly coordinateSystem?: 'pixels' | 'normalised';
}

Image inputs are normalised by OshunVisionLLMClient before they hit the gateway — base64 / URL / Buffer / file path → the provider-specific source shape — so consumers never have to think about the provider's wire format.


5. Output Contract (V1-P2-0065 implements)#

OshunVisionLLMClient.locate(prompt, image) returns:

ts
export interface VisionLocateResult {
  /** The model's reasoning text. */
  readonly text: string;
  /**
   * Bounding boxes the model claims contain the requested element.
   * Empty when the model couldn't find a match — callers MUST handle
   * this, as the model is instructed to return [] rather than
   * hallucinate coordinates.
   */
  readonly boxes: readonly BoundingBox[];
  /** 0..1 — model self-reported confidence. */
  readonly confidence: number;
  /** Provider model id used for this call. */
  readonly model: string;
  readonly latencyMs: number;
  /** Microcents per OSHUN canonical pricing unit. */
  readonly costMicrocents: number;
}

Verification gate V1-P2-0065: a "find the submit button" probe must return coordinates within the button bounds 9 of 10 trials.


6. Verification Gates#

  • V1-P2-0064 — this strategy doc lives at docs/releases/p2/vision-llm-strategy.md.
  • V1-P2-0065 — OshunVisionLLMClient.locate() returns coords within button bounds ≥ 9/10 trials on a fixed screenshot fixture set.
  • V1-P2-0066 — document layout analysis MUST clear mAP ≥ 0.6 on the PubLayNet sample (separate model — LayoutLMv3 ONNX or vision LLM — chosen at that backlog item).
  • V1-P2-0068 — every call site in libs/iris/agents/computer-use/vision/ calls OshunVisionLLMClient (not a parallel stub).

7. Future Revisits#

  • Multi-image inputs at scale. Anthropic and OpenAI both support multi-image content arrays today; the gateway currently sees only single-image traffic in flight. When a consumer requires N-image-per-request reasoning (e.g. multi-frame video diff), the per-image token estimator and the cost-accounting code in @isis/llm-providers need a multi-image audit.
  • Vision tool-use loops. Once iris/agents/computer-use graduates from "single-screen action" to "multi-step planning over a sequence of screens," we'll need to formalise vision-context retention so the model doesn't re-pay the per-image token cost on every step. Likely shape: a vision_session opaque token returned by the first call and reusable in subsequent calls.
  • On-prem model swap. If a license-clean Florence-3 or successor ships with stronger structured-output support, swap the default on-prem fallback. The contract surface in §4–§5 is the abstraction that lets us do that without any consumer changes.