# @psyche/computer-use-core

Native desktop computer-use primitives and an observe/reason/act loop.

ADR-0077 limits this package to the OS-level boundary: native display/window
pixels, accessibility, app/window focus, pointer, keyboard, scrolling,
clipboard, and native file dialogs. It is not a browser driver. A recognized
browser window must be refused and handed to the real Playwright tools in
`@oshun/assistant` unless a preregistered measured exception transfers the
single-owner decision.

## At a glance

- **Path:** `libs/psyche/computer-use-core`
- **Domain:** psyche (active)
- **Owner:** @GreyChimp
- **Docs:** [deep-dive](/docs/domains/psyche/deep-dive) ·
  [systems one-pager](/docs/systems/lib-psyche.md)
- **Nx project:** `psyche-computer-use-core`

## Admission status

This library is not admitted to Eve yet. The remaining gates are explicit:

- Task 6.2 removed the package's direct vendor binding. Planning and vision now
  require separate model-registry bindings backed by the shared `@oshun/ai`
  provider interface. Screenshot bytes cannot reach either provider until an
  injected redaction policy returns a complete decision record.
- Task 6.3 completed the two-pass fabrication audit. Missing native capture or
  input capabilities now raise typed errors; mock capture and action handlers
  are available only from `@psyche/computer-use-core/testing`.
- Task 6.4 proves the native boundary against a real OS-level fixture.
- Task 6.5 adds and proves mandatory per-run authority and execution controls.
- Task 6.6 still owns leased-work admission. Drawer exposure remains none.

See
[ADR-0077](../../../docs/adr/ADR-0077-eve-browser-and-native-computer-use-ownership.md).

## Provider and screenshot contract

`ComputerUseAgent` does not read API keys, construct vendor SDK clients, or
choose model names. Its owning application must provide a
`ComputerUseModelRegistry` that resolves:

- `planning`: text and image input, text output, and tool use;
- `vision`: text and image input and text output (no tool use required).

An absent binding or capability mismatch throws `ComputerUseContractError` with
code `COMPUTER_USE_MODALITY_UNBOUND` before capture or provider use. Missing
registry ownership fails construction with
`COMPUTER_USE_MODEL_REGISTRY_UNBOUND`.

Every image provider call also requires an injected `ScreenshotRedactor`. There
is intentionally no production pass-through implementation. The agent validates
the original and redacted image's declared format/MIME, image signature,
positive dimensions, exact decoded byte count, byte ceiling, and pixel ceiling
before transmission. Empty native-fallback envelopes therefore cannot be sent to
a model. A missing policy throws `COMPUTER_USE_REDACTION_UNBOUND`.

Each accepted image produces a non-pixel `ScreenshotModelInputRecord` with:

- purpose, provider, and registry-owned model;
- format, MIME, width, height, pixel count, and encoded byte size;
- native-binding/provider-hook/data-URL capture source;
- policy id, redaction status, region count, explanation where applicable, and
  timestamps.

Planning records are returned in `AgentExecutionResult.screenshotInputs` and
emitted as `screenshot_prepared` events. Vision records are also attached to
`ScreenAnalysisResult.vision`. Model descriptions are marked untrusted and are
never interpreted as executable instructions.

```ts
import {
  createComputerUseAgent,
  createStaticComputerUseModelRegistry,
  type ComputerUseRunControlHooks,
  type ComputerUseRunPolicy,
  type ScreenshotRedactor,
} from '@psyche/computer-use-core';

const modelRegistry = createStaticComputerUseModelRegistry({
  planning: {
    provider: planningProvider,
    model: planningModel,
    inputModalities: ['text', 'image'],
    outputModalities: ['text'],
    supportsTools: true,
  },
  vision: {
    provider: visionProvider,
    model: visionModel,
    inputModalities: ['text', 'image'],
    outputModalities: ['text'],
    supportsTools: false,
  },
});

const screenshotRedactor: ScreenshotRedactor = operatorOwnedRedactionPolicy;
const runControlHooks: ComputerUseRunControlHooks = owningHostRunControls;
const agent = createComputerUseAgent({
  modelRegistry,
  screenshotRedactor,
  runControlHooks,
});

const policy: ComputerUseRunPolicy = authorityIssuedForThisRun;
const result = await agent.run('Complete the bounded native task', policy);
```

The example deliberately leaves provider construction, credential custody, model
selection, redaction, OS target observation, process confinement, confirmation,
and end-state verification with the owning application. The package rejects an
unbound planning provider, redactor, or run-control host.

## Per-run execution controls

`ComputerUseRunPolicy` is required for every `run` call. It contains exact
application and window tuples, action types, canonical network origins,
canonical absolute task-file read/write entries, and explicit limits for steps,
actions, actions per minute, action interval, duration, total/per-call tokens,
frame age, thinking time, and interrupt acknowledgement. The legacy agent-config
`maxSteps` value is not authority and is no longer used by the run loop;
`policy.budgets.maxSteps` is the enforced ceiling.

Before capture, the host must attest real process-level network and task-file
confinement. `logical-check-only` is never accepted. Run and action previews are
digest-bound and carry monotonic sequence numbers. Interactive runs require an
explicit confirmation, external-side-effect actions require a fresh explicit
confirmation, and destructive actions require step-up.

Every admission returns the exact executable action with automatic retries
disabled. The agent executes that returned primitive, so one preview and one
action-budget unit cannot silently become multiple native attempts.

The agent captures a fresh frame before an active action. Changed or old frames
produce `COMPUTER_USE_STALE_FRAME` and the new governed frame is returned for
replanning without executing the action. Focus is observed before and after
capture and again after confirmation; this package never steals focus. Model
completion is provisional until the host's independent verifier returns named,
passing, digest-backed checks. The returned `controlReceipt` is pixel-free and
contains authority, confirmation, refusal, budget, interrupt, and end-state
facts.

See
[ADR-0080](../../../docs/adr/ADR-0080-eve-native-desktop-execution-controls.md)
for risk rules, the Linux X11 confinement proof, and the remaining admission
boundary.

## Native failure and testing boundary

Production factories fail closed when the native controller is unavailable. They
never report a successful click, key, scroll, drag, clipboard operation, cursor
query, or screenshot unless the corresponding native primitive has completed.
Screenshot results contain real encoded bytes and measured dimensions; an empty
or synthetic fallback frame is not produced.

Display-free tests must opt into the explicit testing subpath:

```ts
import {
  createMockScreenshotProvider,
  createTestActionExecutor,
} from '@psyche/computer-use-core/testing';
```

The package root does not export those doubles. A real OS fixture and native
permission proof remain Task 6.4, so this fail-closed contract is not a native
runtime admission claim.

## Commands

```bash
pnpm nx test psyche-computer-use-core
pnpm nx lint psyche-computer-use-core
pnpm --dir libs/psyche/computer-use-core typecheck
pnpm --dir libs/psyche/computer-use-core build
```

<!-- Generated starting point (audit F-3): owners, replace this note with usage examples and API notes. -->