# OCR Strategy Decision

**Status:** Final v1.0 **Date:** 2026-04-26 **Owners:** eng-iris, eng-shared
**Backlog item:** `V1-P2-0060` **Audience:** every consumer that needs to
extract text from an image — `iris/agents/computer-use/vision`,
`yemaya/document-ingest`, `sophia/curriculum-ingest`, screenshot-driven CI
diagnostics.

---

## 1. Decision

OSHUN's OCR stack is a **three-tier cascade** mirroring the embedding stack
shape from `V1-P2-0030`. Tier choice is explicit at the call site — there is no
silent failover.

1. **Tier 1 — Tesseract.js WASM (default, on-device).** `tesseract.js` v5+
   shipped via the OSHUN `pnpm` catalog, language packs cached on disk under
   `~/.cache/oshun/ocr/tessdata/`. Default for: every screenshot OCR call that
   the computer-use vision pipeline issues, every internal admin tool, every CI
   failure-screenshot pipeline, every sub-300-DPI scan.
2. **Tier 2 — vision LLM via `IrisLLMClient`.** Routed through
   `OshunVisionLLMClient` (`V1-P2-0065`) so the call inherits Iris cost
   accounting, retry policy, and provider fallback. Default for: low-confidence
   Tier-1 results (overall confidence < 0.65), handwriting, complex layouts that
   Tesseract misorders, multilingual text where the language pack is not yet
   installed, anything where layout-aware extraction matters more than raw
   character accuracy.
3. **Tier 3 — Cloud OCR (Google Document AI / Azure Document Intelligence).**
   Behind the same `OshunOCRClient` facade, opt-in only. Default for:
   regulated-pipeline jobs (legal e-discovery, HIPAA scans) where audit logs
   from a vendor are a compliance requirement, very-large-batch ingest pipelines
   (>10⁵ pages/day) where the per-call vendor price beats running our own GPUs,
   archival document sets where layout fidelity matters more than latency.

The cascade is **explicit per call** — consumers pick a tier via the `tier`
field in `OshunOCRClient.extractText()`; tiers do not silently fail over because
the result envelopes diverge (Tier 1 returns word-level boxes, Tier 2 returns
LLM-paraphrased text without boxes by default, Tier 3 returns a vendor-shaped
result that we narrow to our common envelope).

---

## 2. Rationale

### 2.1 Why Tesseract.js by default

The largest OCR consumer in OSHUN is the computer-use vision pipeline
(`libs/iris/agents/computer-use/vision/src/screen-analyzer.ts`,
`text-extractor.ts`). Every agent action that reads a screen runs OCR — call
volume is in the millions per day across active sessions and every call is on
the latency-critical path of the agent loop.

Three properties drive the default:

- **Latency.** Tesseract.js runs in-process at ~150 ms for a 1080p screenshot on
  the M-class boxes the agent runtime targets. No network round-trip.
- **Cost.** Tesseract.js is free at the point of use. A vision-LLM call to the
  same screenshot costs ~$0.003 at Tier 2 prices; over the agent call-rate
  budget that's >$10k/day we don't need to spend.
- **Offline guarantees.** Computer-use agents must function when the tenant is
  air-gapped or network-partitioned (deployment environments include on-prem
  Kubernetes clusters that cannot reach `api.openai.com`).

The accuracy gap vs. a vision LLM is real but bounded: Tesseract gets ≥ 95%
character accuracy on machine-rendered UI text at native resolution
(`V1-P2-0061` verification gate) and that is exactly the input distribution the
vision pipeline sees.

### 2.2 Why vision LLM is Tier 2 not Tier 1

Vision LLMs (Claude Sonnet 4.6 vision, GPT-4.x vision) outperform Tesseract on:

- handwriting (Tesseract is effectively unusable here),
- highly-stylised text (game UI, marketing screenshots),
- charts where the text is fused with graphical elements,
- layout reasoning ("which button has this label" — Tesseract returns unordered
  word boxes; the LLM returns structured intent).

But on the workhorse screenshot OCR path they are slower, more expensive, and
require network egress. So they belong as a **selective escalation**: when Tier
1 emits a confidence < 0.65, re-route to Tier 2 with the same image. The cost
calculus is favourable because <5% of screen calls need escalation in observed
traffic.

### 2.3 Why cloud OCR is opt-in

Google Document AI and Azure Document Intelligence beat both Tesseract and
generic vision LLMs on:

- multi-page PDF layout fidelity,
- form / table extraction with structured field outputs,
- handwriting at scale.

They are also subject to per-page vendor pricing ($0.0015–$0.06/page) and a
network egress hop that defeats the on-device guarantee Tier 1 provides. For the
bulk of OSHUN OCR traffic the price-per-page beats the alternatives exactly when
(a) call volume is high enough to amortise the latency hit and (b) layout
fidelity matters more than latency. That is a narrow band — archival ingest,
regulated pipelines — and we want every such caller to opt in explicitly so the
spend is visible.

### 2.4 What we explicitly rejected

- **PaddleOCR via WASM/ONNX.** Better multilingual coverage than Tesseract on
  paper, but the WASM bundle is ~50 MB vs Tesseract's 6 MB language pack and the
  maintenance/security story for the ecosystem is weaker. We can re-evaluate
  when the WASM package matures.
- **Hugging Face inference endpoints.** No durable Iris cost-tracking story yet
  (`isis-cost-tracking-contract.md` does not cover them); risk of silent
  over-spend.
- **Self-hosted Tesseract over HTTP (instead of WASM in-process).** Adds a
  network hop for the high-volume default path with no quality benefit, so any
  latency or autoscaling pressure becomes a problem we now own.

---

## 3. Per-Consumer Defaults

| Consumer                                                   | Default | Reason                                                         |
| ---------------------------------------------------------- | ------- | -------------------------------------------------------------- |
| `iris/agents/computer-use/vision/text-extractor.ts`        | Tier 1  | Latency-critical agent loop; on-device required.               |
| `iris/agents/computer-use/vision/screen-analyzer.ts`       | Tier 1  | Same path; uses Tier 1 → Tier 2 escalation when conf < 0.65.   |
| `yemaya/document-ingest/<pdf-pipeline>`                    | Tier 3  | Multi-page PDFs at scale; layout fidelity beats latency.       |
| `sophia/curriculum-ingest/<scanned-textbook>`              | Tier 3  | Archival workload; vendor table-extraction beats Tesseract.    |
| Screenshot diagnostics in CI (`tools/ci/screenshot-diff/`) | Tier 1  | Offline; no vendor API keys in CI.                             |
| Future "ask Claude what's on this screen" UX               | Tier 2  | Layout reasoning is the actual product; raw OCR is not enough. |

---

## 4. Common Result Envelope

Every tier returns the same shape via `OshunOCRClient.extractText()`
(implemented in `V1-P2-0062`):

```ts
interface OCRResult {
  readonly text: string; // newline-joined whole-image transcription
  readonly words: ReadonlyArray<{
    readonly text: string;
    readonly box: { x: number; y: number; width: number; height: number };
    readonly confidence: number; // 0..1
  }>;
  readonly lines: ReadonlyArray<{
    readonly text: string;
    readonly box: { x: number; y: number; width: number; height: number };
    readonly words: readonly number[]; // indices into `.words`
    readonly confidence: number;
  }>;
  readonly boxes: ReadonlyArray<{
    readonly x: number;
    readonly y: number;
    readonly width: number;
    readonly height: number;
  }>; // alias of words.box, kept for compat with existing callers
  readonly confidence: number; // overall aggregate
  readonly tier: 'tier1_tesseract' | 'tier2_vision_llm' | 'tier3_cloud_ocr';
  readonly model: string; // e.g. 'tesseract-5.0', 'claude-sonnet-4-6-vision'
  readonly latencyMs: number;
  readonly costMicrocents: number; // 0 for Tier 1
}
```

Tier 2 results synthesise word-level boxes from LLM output where the model
returns positional grounding; otherwise `.words` and `.lines` may be empty.
Callers that depend on word-level boxes must check `.words.length > 0` and fall
back to Tier 1 if empty — this is a documented per-tier capability, not a stub.

---

## 5. Verification Gates

Each downstream backlog item carries its own gate; they must collectively pass
before this strategy is considered live:

- `V1-P2-0061` — sample-image character accuracy ≥ 95% on the workhorse
  machine-rendered UI distribution.
- `V1-P2-0062` — typed `OshunOCRClient.extractText()` exposes the envelope in §4
  against all three tiers (Tiers 2/3 may be guarded by env-controlled test
  skips).
- `V1-P2-0063` — language packs `en, es, fr, ar, zh, ja, ko` each extract test
  text with ≥ 90% character accuracy.
- `V1-P2-0067` / `0071` — call sites in
  `iris/agents/computer-use/vision/{text-extractor,screen-analyzer}.ts` call
  `OshunOCRClient.extractText()` instead of returning fabricated text.

---

## 6. Future Revisits

- **Re-evaluate PaddleOCR** when its WASM packaging stabilises (target: Q1
  2027). Multilingual win could move it into Tier 1 displacing Tesseract for
  non-English text.
- **In-process vision LLM via WASM** (e.g. ONNX-quantised Florence-2) would
  compress Tier 2 into Tier 1 on capable boxes; revisit when an end-to-end
  license-clean WASM pipeline exists.
- **Vendor compaction** — if Document AI vs Document Intelligence differential
  narrows, drop one to reduce vendor surface area.
