# Vision LLM Strategy Decision

**Status:** Final v1.0 **Date:** 2026-04-26 **Owners:** eng-iris, eng-isis,
eng-shared **Backlog item:** `V1-P2-0064` **Audience:** every consumer that
needs structured visual reasoning over an image — `iris/agents/computer-use`
(screen understanding, click-target localisation), the `OshunOCRClient` Tier 2
escalation path (`V1-P2-0060`), accessibility audits in
`psyche/screen-analysis`, and product surfaces that ask Claude to "look at this
screen and tell me what to do."

---

## 1. Decision

OSHUN's vision-LLM access is **single-pipe, multi-tenant routing** through the
canonical `IsisLLMClient` (delivered by `V1-P2-0002`/`V1-P2-0003`). Every vision
call goes through the gateway so token / pixel / cost accounting, rate limiting,
retries, and per-tenant quotas are uniform. No direct calls to provider SDKs are
permitted.

The vision-capable model fleet is a **two-provider primary + on-prem fallback**:

1. **Primary — Anthropic Claude vision.** `claude-sonnet-4-6-vision` (default
   for any request whose downstream consumer is product-quality) and
   `claude-haiku-4-5-vision` (default for high-volume agent loops where the
   sub-second latency floor matters more than peak-quality reasoning). Both
   share the same Anthropic Messages API surface as text models, so
   `IsisLLMClient` exposes a single `messages.create()` call site that accepts
   `image_url | image_base64 | image_path` parts in the user message — there is
   no separate vision-only client.
2. **Secondary — OpenAI vision.** `gpt-4.1-vision` (default for users on
   OpenAI-only tenants and for any consumer whose downstream evaluation is
   already calibrated against OpenAI multi-modal outputs). Routed through the
   same `IsisLLMClient` with provider hint `provider: 'openai'`.
3. **On-prem fallback — Florence-2 / LLaVA-Next via local inference.** Used only
   when (a) the tenant requires data residency that forbids any egress to
   Anthropic or OpenAI, (b) the gateway has tripped its provider circuit breaker
   and is in the "vendor outage" branch, or (c) the downstream consumer can
   tolerate ~3× higher latency for the data-locality guarantee.

Switching tiers is **explicit per call** via the `provider` and `vision_quality`
request fields in `IsisLLMRequest`; routing is not silent because Anthropic and
OpenAI tokenisers and pricing differ and because consumers with
`data_residency: 'on_prem'` MUST NOT cascade to Anthropic/OpenAI.

`OshunVisionLLMClient` (V1-P2-0065) is a thin convenience wrapper over
`IsisLLMClient` — it does not own any provider knowledge, it owns the
prompt-shape contract for vision tasks (the prompt template, the response
schema, the bounding-box coordinate convention, the per-task verifier).

---

## 2. Rationale

### 2.1 Why route through `IsisLLMClient`

OSHUN already has a backlog (`V1-P2-0002` … `V1-P2-0027`) that delivers a
canonical LLM gateway: provider registry, rate limiter, sliding-window quotas,
microcent cost accounting (`@oshun/contracts/llm`), retry executor, circuit
breaker, OpenTelemetry instrumentation, and an error taxonomy with stable codes.
Every property on that list is also required for vision calls. Building a
parallel "vision client" that re-implements quotas / retries / cost would be
exactly the duplication the audit at
`docs/releases/p2/isis-llm-gateway-audit.md` was meant to prevent.

The vision-specific concerns — image MIME inference, multi-image inputs,
pixel-token estimation, output-format coercion (JSON / coordinates) — are real,
but they are **input/output adapters around** the gateway, not parallel
transport. The `OshunVisionLLMClient` wrapper owns those adapters.

### 2.2 Why Claude vision is primary

Three properties drive the default:

- **Best-in-class structured visual reasoning.** Claude vision excels at the
  workhorse query for OSHUN ("which UI element labelled X is the submit
  button?", "describe what the user is trying to accomplish in this screenshot")
  because its tool-use mode pairs naturally with the ImageContent block,
  allowing the model to return JSON-shaped action plans without
  prompt-engineering acrobatics.
- **Tokenisation & cost predictability.** Anthropic publishes a deterministic
  image-token formula (`tokens ≈ (width * height) / 750`) which the existing
  `@isis/llm-providers` token counter already models pixel-wise; that
  integration cost is paid.
- **Same SDK surface as text.** The Messages API uses the same `content: [...]`
  block list for text and image inputs, so the `IsisLLMRequest` schema doesn't
  need a separate vision-only branch.

### 2.3 Why OpenAI vision is secondary, not primary

GPT-4.1 vision matches Claude on most benchmarks and beats it on chart
understanding and dense-text-in-image (where cloud OCR Tier 3 would win anyway).
It is kept as a routable secondary because:

- some tenants are contractually OpenAI-only (legal/compliance posture);
- accessibility audits in `psyche/screen-analysis` were originally calibrated
  against GPT-4.1 vision and we don't want to re-baseline before `V1-P2-0066`
  lands;
- the `IsisLLMClient` provider-fallback chain needs a non-Anthropic branch to be
  a real fallback; otherwise Anthropic outages cascade into full vision outages
  across OSHUN.

Vision parity is close enough that defaulting to OpenAI for any particular
consumer is a configuration toggle, not a re-architecture.

### 2.4 Why an on-prem fallback at all

The audit already documents `data_residency` as a first-class constraint. Any
consumer that flips that flag must NOT cascade to Anthropic or OpenAI even when
the gateway is healthy. Florence-2 (Microsoft, MIT-license) and LLaVA-Next (open
weights) are the two production-grade on-prem candidates today; we pick at
deployment time based on the tenant's GPU profile and let the consumer treat
both as "tier3 vision". They are explicitly **not** a quality default — only the
data-locality default.

### 2.5 What we explicitly rejected

- **Google Gemini Vision.** Quality is competitive but the tokenisation &
  pricing surface diverges from both Anthropic and OpenAI; adding it now means
  three provider-specific cost adapters, three different image token formulas,
  three different streaming envelopes. Revisit when / if a tenant requires it.
- **A standalone "OshunVisionClient" with its own transport.** Would reintroduce
  the audit's complaint of parallel transport stacks; the shared cross-cutting
  concerns (quotas, retries, OTEL) all live in `IsisLLMClient`.
- **Synchronous vision-LLM call inside the agent loop's hot path.** Even at
  Haiku latency (~400 ms median), making every screenshot OCR call route through
  a vision LLM would burn the cost budget. The OCR strategy doc (`V1-P2-0060`)
  keeps Tier 1 Tesseract on the hot path and routes here only on confidence <
  0.65 (the documented selective-escalation pattern).

---

## 3. Per-Consumer Defaults

| Consumer                                                                         | Default                        | Reason                                                                                             |
| -------------------------------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------- |
| `iris/agents/computer-use/vision/screen-analyzer.ts`                             | `claude-haiku-4-5-vision`      | Latency floor matters more than peak quality on the agent inner loop.                              |
| `iris/agents/computer-use/vision/index.ts` (high-level "find the submit button") | `claude-sonnet-4-6-vision`     | Structured visual reasoning is the actual product; quality > latency.                              |
| Tier 2 OCR escalation (`OshunOCRClient` low-conf fallback)                       | `claude-haiku-4-5-vision`      | Already on a low-frequency escalation path; minimise per-call cost.                                |
| `psyche/screen-analysis` accessibility audits                                    | `gpt-4.1-vision`               | Existing baseline calibrated against OpenAI; switching pre-V1-P2-0066 invalidates A/B comparisons. |
| Any tenant with `data_residency: 'on_prem'`                                      | `florence-2` (or `llava-next`) | Compliance constraint; NOT cascadable.                                                             |
| Claude-only tenants                                                              | `claude-sonnet-4-6-vision`     | Quality default when one-vendor lock-in is the contract.                                           |

---

## 4. Common Request Envelope

Vision requests piggyback on `IsisLLMRequest`:

```ts
import type {
  IsisLLMRequest,
  IsisLLMMessageContent,
} from '@oshun/contracts/llm';

// V1-P2-0064 vision input contract (referenced from V1-P2-0065)
export interface VisionImagePart {
  readonly type: 'image';
  readonly source:
    | {
        kind: 'base64';
        media_type: 'image/png' | 'image/jpeg' | 'image/webp';
        data: string;
      }
    | { kind: 'url'; url: string };
}

// Mixed text + image content. Order is preserved end-to-end so
// captions referring to "the screenshot above" work.
export type IsisLLMVisionContent = ReadonlyArray<
  IsisLLMMessageContent | VisionImagePart
>;

export interface VisionExtraOptions {
  /** 'low' | 'high' — maps to provider-specific quality flags. */
  readonly quality?: 'low' | 'high';
  /**
   * If set, the model MUST return JSON conforming to this JSON schema.
   * Coerced via tool-use on Anthropic, response_format on OpenAI, and
   * a constrained-decoding pass on the on-prem fallback.
   */
  readonly responseSchema?: Record<string, unknown>;
  /**
   * Bounding-box coordinate convention. Defaults to 'pixels' (origin
   * top-left). 'normalised' produces 0..1 box coords, useful when
   * the consumer doesn't know the image dimensions yet.
   */
  readonly coordinateSystem?: 'pixels' | 'normalised';
}
```

Image inputs are normalised by `OshunVisionLLMClient` before they hit the
gateway — base64 / URL / Buffer / file path → the provider-specific `source`
shape — so consumers never have to think about the provider's wire format.

---

## 5. Output Contract (V1-P2-0065 implements)

`OshunVisionLLMClient.locate(prompt, image)` returns:

```ts
export interface VisionLocateResult {
  /** The model's reasoning text. */
  readonly text: string;
  /**
   * Bounding boxes the model claims contain the requested element.
   * Empty when the model couldn't find a match — callers MUST handle
   * this, as the model is instructed to return [] rather than
   * hallucinate coordinates.
   */
  readonly boxes: readonly BoundingBox[];
  /** 0..1 — model self-reported confidence. */
  readonly confidence: number;
  /** Provider model id used for this call. */
  readonly model: string;
  readonly latencyMs: number;
  /** Microcents per OSHUN canonical pricing unit. */
  readonly costMicrocents: number;
}
```

Verification gate `V1-P2-0065`: a "find the submit button" probe must return
coordinates within the button bounds 9 of 10 trials.

---

## 6. Verification Gates

- `V1-P2-0064` — this strategy doc lives at
  `docs/releases/p2/vision-llm-strategy.md`.
- `V1-P2-0065` — `OshunVisionLLMClient.locate()` returns coords within button
  bounds ≥ 9/10 trials on a fixed screenshot fixture set.
- `V1-P2-0066` — document layout analysis MUST clear mAP ≥ 0.6 on the PubLayNet
  sample (separate model — LayoutLMv3 ONNX or vision LLM — chosen at that
  backlog item).
- `V1-P2-0068` — every call site in `libs/iris/agents/computer-use/vision/`
  calls `OshunVisionLLMClient` (not a parallel stub).

---

## 7. Future Revisits

- **Multi-image inputs at scale.** Anthropic and OpenAI both support multi-image
  content arrays today; the gateway currently sees only single-image traffic in
  flight. When a consumer requires N-image-per-request reasoning (e.g.
  multi-frame video diff), the per-image token estimator and the cost-accounting
  code in `@isis/llm-providers` need a multi-image audit.
- **Vision tool-use loops.** Once `iris/agents/computer-use` graduates from
  "single-screen action" to "multi-step planning over a sequence of screens,"
  we'll need to formalise vision-context retention so the model doesn't re-pay
  the per-image token cost on every step. Likely shape: a `vision_session`
  opaque token returned by the first call and reusable in subsequent calls.
- **On-prem model swap.** If a license-clean Florence-3 or successor ships with
  stronger structured-output support, swap the default on-prem fallback. The
  contract surface in §4–§5 is the abstraction that lets us do that without any
  consumer changes.
