Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris, eng-shared
Backlog item: V1-P2-0060 Audience: every consumer that needs to
extract text from an image — iris/agents/computer-use/vision,
yemaya/document-ingest, sophia/curriculum-ingest, screenshot-driven CI
diagnostics.
1. Decision#
OSHUN's OCR stack is a three-tier cascade mirroring the embedding stack
shape from V1-P2-0030. Tier choice is explicit at the call site — there is no
silent failover.
- Tier 1 — Tesseract.js WASM (default, on-device).
tesseract.jsv5+ shipped via the OSHUNpnpmcatalog, language packs cached on disk under~/.cache/oshun/ocr/tessdata/. Default for: every screenshot OCR call that the computer-use vision pipeline issues, every internal admin tool, every CI failure-screenshot pipeline, every sub-300-DPI scan. - Tier 2 — vision LLM via
IrisLLMClient. Routed throughOshunVisionLLMClient(V1-P2-0065) so the call inherits Iris cost accounting, retry policy, and provider fallback. Default for: low-confidence Tier-1 results (overall confidence < 0.65), handwriting, complex layouts that Tesseract misorders, multilingual text where the language pack is not yet installed, anything where layout-aware extraction matters more than raw character accuracy. - Tier 3 — Cloud OCR (Google Document AI / Azure Document Intelligence).
Behind the same
OshunOCRClientfacade, opt-in only. Default for: regulated-pipeline jobs (legal e-discovery, HIPAA scans) where audit logs from a vendor are a compliance requirement, very-large-batch ingest pipelines (>10⁵ pages/day) where the per-call vendor price beats running our own GPUs, archival document sets where layout fidelity matters more than latency.
The cascade is explicit per call — consumers pick a tier via the tier
field in OshunOCRClient.extractText(); tiers do not silently fail over because
the result envelopes diverge (Tier 1 returns word-level boxes, Tier 2 returns
LLM-paraphrased text without boxes by default, Tier 3 returns a vendor-shaped
result that we narrow to our common envelope).
2. Rationale#
2.1 Why Tesseract.js by default#
The largest OCR consumer in OSHUN is the computer-use vision pipeline
(libs/iris/agents/computer-use/vision/src/screen-analyzer.ts,
text-extractor.ts). Every agent action that reads a screen runs OCR — call
volume is in the millions per day across active sessions and every call is on
the latency-critical path of the agent loop.
Three properties drive the default:
- Latency. Tesseract.js runs in-process at ~150 ms for a 1080p screenshot on the M-class boxes the agent runtime targets. No network round-trip.
- Cost. Tesseract.js is free at the point of use. A vision-LLM call to the same screenshot costs ~$0.003 at Tier 2 prices; over the agent call-rate budget that's >$10k/day we don't need to spend.
- Offline guarantees. Computer-use agents must function when the tenant is
air-gapped or network-partitioned (deployment environments include on-prem
Kubernetes clusters that cannot reach
api.openai.com).
The accuracy gap vs. a vision LLM is real but bounded: Tesseract gets ≥ 95%
character accuracy on machine-rendered UI text at native resolution
(V1-P2-0061 verification gate) and that is exactly the input distribution the
vision pipeline sees.
2.2 Why vision LLM is Tier 2 not Tier 1#
Vision LLMs (Claude Sonnet 4.6 vision, GPT-4.x vision) outperform Tesseract on:
- handwriting (Tesseract is effectively unusable here),
- highly-stylised text (game UI, marketing screenshots),
- charts where the text is fused with graphical elements,
- layout reasoning ("which button has this label" — Tesseract returns unordered word boxes; the LLM returns structured intent).
But on the workhorse screenshot OCR path they are slower, more expensive, and require network egress. So they belong as a selective escalation: when Tier 1 emits a confidence < 0.65, re-route to Tier 2 with the same image. The cost calculus is favourable because <5% of screen calls need escalation in observed traffic.
2.3 Why cloud OCR is opt-in#
Google Document AI and Azure Document Intelligence beat both Tesseract and generic vision LLMs on:
- multi-page PDF layout fidelity,
- form / table extraction with structured field outputs,
- handwriting at scale.
They are also subject to per-page vendor pricing ($0.0015–$0.06/page) and a network egress hop that defeats the on-device guarantee Tier 1 provides. For the bulk of OSHUN OCR traffic the price-per-page beats the alternatives exactly when (a) call volume is high enough to amortise the latency hit and (b) layout fidelity matters more than latency. That is a narrow band — archival ingest, regulated pipelines — and we want every such caller to opt in explicitly so the spend is visible.
2.4 What we explicitly rejected#
- PaddleOCR via WASM/ONNX. Better multilingual coverage than Tesseract on paper, but the WASM bundle is ~50 MB vs Tesseract's 6 MB language pack and the maintenance/security story for the ecosystem is weaker. We can re-evaluate when the WASM package matures.
- Hugging Face inference endpoints. No durable Iris cost-tracking story yet
(
isis-cost-tracking-contract.mddoes not cover them); risk of silent over-spend. - Self-hosted Tesseract over HTTP (instead of WASM in-process). Adds a network hop for the high-volume default path with no quality benefit, so any latency or autoscaling pressure becomes a problem we now own.
3. Per-Consumer Defaults#
| Consumer | Default | Reason |
|---|---|---|
iris/agents/computer-use/vision/text-extractor.ts |
Tier 1 | Latency-critical agent loop; on-device required. |
iris/agents/computer-use/vision/screen-analyzer.ts |
Tier 1 | Same path; uses Tier 1 → Tier 2 escalation when conf < 0.65. |
yemaya/document-ingest/<pdf-pipeline> |
Tier 3 | Multi-page PDFs at scale; layout fidelity beats latency. |
sophia/curriculum-ingest/<scanned-textbook> |
Tier 3 | Archival workload; vendor table-extraction beats Tesseract. |
Screenshot diagnostics in CI (tools/ci/screenshot-diff/) |
Tier 1 | Offline; no vendor API keys in CI. |
| Future "ask Claude what's on this screen" UX | Tier 2 | Layout reasoning is the actual product; raw OCR is not enough. |
4. Common Result Envelope#
Every tier returns the same shape via OshunOCRClient.extractText()
(implemented in V1-P2-0062):
interface OCRResult {
readonly text: string; // newline-joined whole-image transcription
readonly words: ReadonlyArray<{
readonly text: string;
readonly box: { x: number; y: number; width: number; height: number };
readonly confidence: number; // 0..1
}>;
readonly lines: ReadonlyArray<{
readonly text: string;
readonly box: { x: number; y: number; width: number; height: number };
readonly words: readonly number[]; // indices into `.words`
readonly confidence: number;
}>;
readonly boxes: ReadonlyArray<{
readonly x: number;
readonly y: number;
readonly width: number;
readonly height: number;
}>; // alias of words.box, kept for compat with existing callers
readonly confidence: number; // overall aggregate
readonly tier: 'tier1_tesseract' | 'tier2_vision_llm' | 'tier3_cloud_ocr';
readonly model: string; // e.g. 'tesseract-5.0', 'claude-sonnet-4-6-vision'
readonly latencyMs: number;
readonly costMicrocents: number; // 0 for Tier 1
}
Tier 2 results synthesise word-level boxes from LLM output where the model
returns positional grounding; otherwise .words and .lines may be empty.
Callers that depend on word-level boxes must check .words.length > 0 and fall
back to Tier 1 if empty — this is a documented per-tier capability, not a stub.
5. Verification Gates#
Each downstream backlog item carries its own gate; they must collectively pass before this strategy is considered live:
V1-P2-0061— sample-image character accuracy ≥ 95% on the workhorse machine-rendered UI distribution.V1-P2-0062— typedOshunOCRClient.extractText()exposes the envelope in §4 against all three tiers (Tiers 2/3 may be guarded by env-controlled test skips).V1-P2-0063— language packsen, es, fr, ar, zh, ja, koeach extract test text with ≥ 90% character accuracy.V1-P2-0067/0071— call sites iniris/agents/computer-use/vision/{text-extractor,screen-analyzer}.tscallOshunOCRClient.extractText()instead of returning fabricated text.
6. Future Revisits#
- Re-evaluate PaddleOCR when its WASM packaging stabilises (target: Q1 2027). Multilingual win could move it into Tier 1 displacing Tesseract for non-English text.
- In-process vision LLM via WASM (e.g. ONNX-quantised Florence-2) would compress Tier 2 into Tier 1 on capable boxes; revisit when an end-to-end license-clean WASM pipeline exists.
- Vendor compaction — if Document AI vs Document Intelligence differential narrows, drop one to reduce vendor surface area.