# Proto Domain — Architecture

> Architectural overview of `@oshun/proto` (`libs/proto/`): why gRPC exists
> alongside REST, how the `.proto` schema surface is organized, how a tiny
> TypeScript runtime layer loads those schemas, how the Buf toolchain and the
> Oya parity gate keep everything honest, and how consuming services wire it all
> together.

---

## What this domain is

`@oshun/proto` is the single source of truth for all gRPC communication inside
the Oshun platform. Every `.proto` file lives here, and every service that calls
another service over gRPC depends on this library for schema definitions, a
runtime loader, and channel helpers. The library owns **no product logic** — it
is purely a schema contract plus a thin loading/configuration layer.

It is a _leaf_ in the monorepo dependency graph. `package.json` lists only three
runtime dependencies — `@grpc/grpc-js`, `@grpc/proto-loader`, and `protobufjs` —
and **no dependency on any other Oshun library**. `project.json` tags it
`scope:shared`, `layer:contracts`. The boundary is deliberate: if `@oshun/proto`
depended on a domain library, a change in that domain could break every service
that speaks gRPC, and import cycles would become possible. Every gRPC-speaking
domain depends on `@oshun/proto`; `@oshun/proto` depends on none of them.

There is **no `services/proto` and no `apps/proto`** — this domain is a library
only. The gRPC _servers_ and _clients_ that implement and consume these
contracts live in their respective domain packages (Isis, Sophia, Hathor, the
engine bridges, …); `@oshun/proto` gives them the schemas, the loader, and the
channel/credential defaults to do so consistently.

### Two roles in one package

`src/index.ts` re-exports exactly two modules' worth of surface, reflecting the
library's dual role:

1. **Schema home.** It owns every `.proto` describing an Oshun gRPC surface —
   currently **30 `.proto` files** under `libs/proto/src/`, one directory per
   service domain. (`git ls-files 'libs/proto/src/**/*.proto'` returns 30, and
   the `PROTO_PATHS` registry has 30 matching entries.)
2. **Runtime helper layer.** A small TypeScript surface — `loader.ts`,
   `services.ts`, `index.ts` — that loads `.proto` files at runtime via
   `@grpc/proto-loader`, exposes typed path/name/metadata registries, and
   centralizes gRPC channel options and credential construction.

The raw schemas are also exposed directly to consumers: the `package.json`
`exports` map carries a `"./protos/*": "./src/*.proto"` subpath, and the Nx
`build` target copies `**/*.proto` from `src` into `protos/` in the build
output.

---

## gRPC vs REST in Oshun

The platform uses both protocols because they serve different audiences. Browser
clients and external API consumers use REST (via `@oshun/openapi`) because
HTTP/JSON is universally understood and easy to introspect. Internal
service-to-service calls use gRPC: Protocol Buffers encode messages in a compact
binary format, gRPC multiplexes calls over a single HTTP/2 connection, and both
request and response types are enforced at schema level. For streaming job
progress to a UI, bidirectionally synchronizing a live document, or coordinating
a render farm, gRPC's native streaming modes are a natural fit that REST cannot
match without polling or SSE workarounds.

| Use Case                          | Protocol                       | Reason                                   |
| --------------------------------- | ------------------------------ | ---------------------------------------- |
| Client-facing public APIs         | REST + OpenAPI                 | Browser compatibility, human-readable    |
| High-frequency service-to-service | gRPC                           | Binary encoding, multiplexing, streaming |
| Job status streaming              | gRPC server streaming          | Efficient, typed, persistent connection  |
| Real-time render coordination     | gRPC bidirectional streaming   | Full-duplex, low overhead                |
| Health checks                     | gRPC (standard health service) | Universal gRPC ecosystem support         |
| Batch operations                  | gRPC                           | Binary framing reduces overhead vs JSON  |

---

## Source layout

```
libs/proto/
├── package.json            # @oshun/proto, 3 grpc deps, "./protos/*" export
├── project.json            # Nx: build, lint, test, proto:gen, proto:lint
├── buf.work.yaml           # buf workspace: directories = [src]
├── buf.gen.yaml            # buf codegen: ts-proto, Go, Go-gRPC, JSON Schema
├── generated/
│   └── buf-image.json      # serialized FileDescriptorSet (breaking-change baseline)
├── scripts/
│   └── generate.ts         # separate pbjs/pbts static-module type generator
├── oya/
│   └── check-proto-parity.mjs   # Oya proto ⇄ zod-contract parity gate (§ below)
└── src/
    ├── index.ts            # public API re-exports (loader + services + grpc types)
    ├── loader.ts           # loadProto/loadProtos/loadAllProtos + PROTO_PATHS
    ├── services.ts         # SERVICE_NAMES, DEFAULT_CHANNEL_OPTIONS, createCredentials
    ├── proto.spec.ts       # Vitest suite for loader + registries + per-proto load
    ├── buf.yaml            # SINGLE buf module config for the whole src/ tree
    │
    ├── common/types.proto              # oshun.common — shared scalar vocabulary
    ├── shared/{common,evidence,memory,persona_policy,generation_control}.proto
    ├── auth/, ai/, agent/, asset/, collaboration/, project/, user/   # core
    ├── isis/, sophia/, hathor/, concordia/   # domain services
    ├── oya/oya.proto                   # oshun.oya.v1 — embodied-hive contracts (NO service)
    ├── generation3d/, rendering/, splatting/, procedural/   # rendering & 3D
    ├── bridge/{blender,godot,unreal}.proto   # DCC/engine bridges
    ├── health/, loadbalancing/, reflection/, pipeline/   # infrastructure
    └── oshun/v2/persistent_economy/economy.proto   # versioned V2 game services
```

> **Exactly one `buf.yaml`**, at `src/buf.yaml`, defines a single Buf module
> (`buf.build/oshun/proto`) covering the entire `src/` subtree. There are **no**
> per-domain `buf.yaml` files. The library root holds `buf.work.yaml` (workspace
> config pointing at `src`) and `buf.gen.yaml` (code-generation config).

---

## The runtime helper layer

### `src/loader.ts` — loading schemas at runtime

`loader.ts` is the bridge between the static `.proto` files and the live gRPC
clients services instantiate at startup. Rather than importing pre-generated
JavaScript stubs, a service calls `loadProto` (or `loadAllProtos` at server
start), gets back a `grpc.GrpcObject`, and reads the service constructor out of
it by its fully-qualified package path. This keeps the `.proto` files the single
source of truth at both build and run time, at the cost of a small async parse
on startup. The module computes its own directory via
`fileURLToPath(import.meta.url)` — the pure-ESM pattern, since the package is
`"type": "module"`.

- **`loadProto(path, options?)`** — async; resolves a relative path against
  `src/`, merges `{ ...DEFAULT_LOADER_OPTIONS, ...options }`, then
  `protoLoader.load` → `grpc.loadPackageDefinition`. Returns a
  `Promise<grpc.GrpcObject>`.
- **`loadProtos(paths[], options?)`** — loads several files and shallow-merges
  them with `Object.assign` keyed on the top-level `oshun` package segment, so
  every file's subpackages accrete onto one shared root.
- **`loadAllProtos(options?)`** — `loadProtos(Object.values(PROTO_PATHS))`; used
  where a server registers every service at once.
- **`getProtoPath(relativePath)`** — pure path helper joining onto `src/` and
  returning the absolute path (it does not load or check existence).
- **`PROTO_PATHS`** — an `as const` registry of 30 stable keys → `.proto` paths
  relative to `src/` (e.g. `PROTO_PATHS.isis` → `isis/isis.proto`,
  `PROTO_PATHS.oya` → `oya/oya.proto`). The exported `ProtoPath` type is the
  union of those path literals.

The most consequential decision lives in `DEFAULT_LOADER_OPTIONS`:

```typescript
export const DEFAULT_LOADER_OPTIONS: protoLoader.Options = {
  keepCase: true, // field names stay snake_case — NOT camelCased
  longs: String, // int64/uint64 surfaced as JS strings
  enums: String, // enum values surfaced as string names
  defaults: true, // default values included in decoded output
  oneofs: true, // virtual oneof discriminator field included
  includeDirs: [__dirname, path.join(__dirname, '..')],
};
```

`keepCase: true` means runtime field names are exactly as declared in the
`.proto` (`queue_name`, `pending_jobs`), which is the opposite of what
`buf generate`'s ts-proto path produces (`snakeToCamel=true`). The two
`includeDirs` entries — `src/` and its parent — are what let a domain file's
`import "common/types.proto"` resolve. `proto.spec.ts` asserts
`keepCase === true`.

### `src/services.ts` — channel names, options, credentials

Where `loader.ts` parses schemas, `services.ts` centralizes everything needed to
construct and configure a channel:

- **`SERVICE_NAMES`** — an `as const` map of registry key → fully-qualified gRPC
  service name (e.g. `SERVICE_NAMES.IsisJob` → `oshun.isis.IsisJobService`).
  `ServiceName` is the union of those values.
- **`DEFAULT_CHANNEL_OPTIONS`** — shared keepalive timing (30 s ping, 10 s ack
  timeout, pings permitted without calls), HTTP/2 ping spacing, and **50 MB**
  inbound/outbound message-size limits. The 50 MB ceiling accommodates the
  largest in-message binaries in the system: captured frame bytes on splatting's
  `UploadFrames`, viewport captures from the engine bridges, and inline document
  content for Sophia ingestion. The 50 MB limit and the keepalive durations are
  asserted by `proto.spec.ts`.
- **`createCredentials(secure, rootCerts?, privateKey?, certChain?)`** — channel
  credential factory. `secure: false` → insecure; `secure: true` with a client
  key + chain → mutual TLS; `secure: true` without → server-authenticated TLS
  (falling back to the system trust store when `rootCerts` is `undefined`). It
  `require`s `@grpc/grpc-js` dynamically into a local binding as an explicit
  ESM/CJS interop accommodation.
- **`getServiceMetadata(name)`** — returns
  `{ name, protoPath, package, methods }` for a registered service, or
  `undefined`.

> **Two honest caveats that a maintainer must know.** First, two `SERVICE_NAMES`
> entries disagree with their `.proto` source and are **not** wire-correct:
> `Procedural` is registered as `…ProceduralGenerationService` but
> `procedural.proto` declares `service ProceduralGenService`; `Reflection` is
> registered as `…ReflectionService` but `reflection.proto` declares
> `ServerReflectionService`. Read those service constructors from the package
> object at their `.proto`-declared names. Second, the `methods` arrays inside
> `getServiceMetadata` are a **hand-maintained summary that has drifted** — they
> are accurate for some services and stale for others (e.g. the `AI` and `Agent`
> entries list RPC names that no longer match the schema). The `.proto` files
> are authoritative for RPC rosters; the `specifications.md` catalog enumerates
> them directly from source.

---

## Schema organization

Every `.proto` is `syntax = "proto3"`, uses an `oshun.<domain>` package,
declares `option go_package = "github.com/oshun/proto/<path>"`, suffixes enum
zero values with `_UNSPECIFIED`, and names fields in `snake_case`. Files import
`google/protobuf/timestamp.proto`, `struct.proto`, or `duration.proto` as
needed, and domain files import `common/types.proto` for shared types.

Three structural tiers sit underneath the per-domain service files:

- **`common/types.proto`** (package `oshun.common`) is a pure type vocabulary
  with **no service** — `UUID`, `PaginationRequest`/`PaginationMeta`,
  `Error`/`FieldError`, `Empty`, `SuccessResponse`, and a health-types pair.
  Defining them once guarantees a `PaginationRequest` means the same thing in an
  auth call as in a rendering call. Most "no payload" RPCs return
  `common.SuccessResponse`; every list RPC pages with the common pagination
  pair.
- **`shared/*.proto`** are **not** a type bag — they are four product-facing
  _substrate service_ contracts (plus `shared/common.proto`, which carries only
  cross-substrate enums and a `SharedContractVersionDescriptor`):
  `OshunEvidenceService` (Sophia grounding), `OshunMemoryService` (Iris
  continuity), `OshunPersonaPolicyService` (Lilith policy), and
  `OshunGenerationControlService` (Isis generation control). These are consumed
  directly by product domains (Tara, Arete, Veritas, Nyx, Nisaba, the Assistant
  shell), expressing cross-cutting concerns no single product domain should own.
- **Domain service files** (`isis`, `sophia`, `hathor`, `concordia`, the engine
  bridges, rendering/3D, pipeline, infrastructure) each declare one or more
  services for one domain's gRPC surface. A single file may declare several
  services (`sophia.proto` declares five, `hathor.proto` seven).
  `concordia.proto` enforces a viewer-role privacy invariant at schema level for
  mediation streams.

### Package-version convention — and its one exception

The historical convention is `oshun.<domain>` with **no `.v1` suffix**; Buf's
`PACKAGE_VERSION_SUFFIX` lint rule is disabled in `src/buf.yaml` precisely to
allow this. **There are now two deliberate exceptions, and they should be read
as the intended direction of travel, not drift:**

- `oshun/v2/persistent_economy/economy.proto` uses package
  `oshun.v2.persistent_economy` for the Section-130 open-world game services
  (`Economy`, `NPCSchedule`, `CrimeRate`).
- **`oya/oya.proto` uses package `oshun.oya.v1`** — the first domain to adopt a
  trailing `.v1`. `proto.spec.ts` confirms its messages resolve under
  `pkg.oshun.oya.v1.*` (not `pkg.oshun.oya.*`). Field numbers in this file are
  treated as the stable wire identity and must never be reused or renumbered.

---

## The Oya embodied-hive contract layer

`oya/oya.proto` is the newest and most distinctive member of the library, and it
behaves differently from every service file above: **it declares no service at
all.** Like `common/types.proto`, it is a pure message/enum vocabulary — but
where `common` is generic platform plumbing, Oya is the wire form of a specific
embodied domain: a hive of drones/robots that share telemetry, allocate tasks,
maintain a spatial world model, and enforce safety and privacy.

Its reason for existing is **tri-directional parity**. The same vocabulary is
spoken in three places, and all three must agree byte-for-byte:

1. the canonical zod contracts in `libs/contracts/src/oya`
   (`@oshun/contracts/oya`);
2. the proto wire messages in `oya/oya.proto`;
3. the Rust engine/hive crates in `libs/oya/engine/crates/` — `oya-types`,
   `oya-math`, `oya-scenegraph`, `oya-fleet`, `oya-safety`, `oya-comms` and
   siblings — that actually run the control loops.

The file's header documents the mirroring rules explicitly (zod `z.number()` →
`double`; `z.number().int()` → `int32`/`int64`; `z.enum` → proto `enum` with a
`_UNSPECIFIED = 0` zero value; `z.array` → `repeated`; `z.record` → `map`;
`z.union`/discriminated → `oneof`). The message set covers 3D math primitives
(`Vec3`, `Quaternion`, `GpsCoordinate`), telemetry/control (`Telemetry`,
`ControlCommand`, the `ControlMode` enum), mission/flight-plan (`Mission`,
`FlightPlan`, `Geofence`, `Waypoint`), CBBA fleet allocation (`FleetState`,
`TaskBid`, `Allocation`), the shared spatial world model (`SceneGraphInstance`,
`Aabb`, `WorldModelQuery`/`WorldModelResult`), sensor observations, capability
manifests, docking/battery-swap, the ISO/TS 15066 `SafetyEnvelope`, and privacy
(`ConsentPolicy`, `PrivacyZone`).

Two domain invariants are encoded _structurally_ in the schema and are worth
calling out because they are the point of the design:

- **Fail-loud spatial memory.** `WorldModelResult` is a three-arm `oneof`
  (`Fresh` / `Stale` / `Unknown`) mirroring the Rust `QueryResult<V>`. A decayed
  memory returns `Stale` (no value) or `Unknown` rather than fabricating a
  confident answer — the schema makes a silent confident-lie _unrepresentable_.
- **Privacy by omission.** `SensorObservation` carries only a
  `CompressedDescriptor` (a `oneof` of an opaque codec `blob` or an explicit
  `FeatureVector`) — there is intentionally **no raw-frame field**. Consent
  (`ConsentPolicy`/`ConsentScope`) is fail-closed: an absent scope means consent
  was _not_ given.

### The parity gate (`oya/check-proto-parity.mjs`)

Tri-directional parity is not a comment — it is enforced by a real fail-loud
gate. `oya/check-proto-parity.mjs` parses `oya.proto` with `protobufjs` and
imports `@oshun/contracts/oya` through the `tsx` ESM loader, then proves three
properties per schema: **coverage** (every zod object/union/enum schema has a
mapped proto message — an unmapped new contract is flagged, not skipped),
**field-set** equality (proto field names match the zod keys after
case-normalization), and **enum-set** equality (members match, allowing exactly
the one extra `_UNSPECIFIED = 0`). Any drift in either direction exits non-zero.

What makes it trustworthy is the **drift-detector self-test** that runs _before_
the real check: the script clones the parsed proto, deliberately drops a field
(`Telemetry.voltage`), renames one (`ControlCommand.armed` → `armed_flag`), and
removes an enum member (`CAPABILITY_GRASP`), then asserts the checker reports
all three. If the checker fails to catch its own injected drift it exits `2` —
"the checker is broken and cannot be trusted to gate anything" — rather than
emitting a false pass.

---

## Component & data flow

```mermaid
flowchart TB
  subgraph authoring["Authoring / CI time"]
    dev["Developer edits a .proto"]
    lint["buf lint (DEFAULT + COMMENTS)"]
    breaking["buf breaking --against buf-image.json (FILE group)"]
    gen["buf generate → gen/ts, gen/go, gen/jsonschema (on demand, not committed)"]
    dev --> lint --> breaking --> gen
    zod["@oshun/contracts/oya (zod)"]
    rust["libs/oya/engine (Rust crates)"]
    parity["oya/check-proto-parity.mjs (self-test → parity)"]
    zod --> parity
    oya["oya/oya.proto (oshun.oya.v1)"]
    oya --> parity
    rust -. mirrors .- oya
  end

  subgraph runtime["Runtime"]
    svcA["Caller service (e.g. Isis worker)"]
    loaderTS["@oshun/proto loader.ts\nloadProto(PROTO_PATHS.x)"]
    ploader["@grpc/proto-loader\n(keepCase, longs:String, enums:String)"]
    gobj["grpc.GrpcObject\n(read ctor at oshun.x.YService)"]
    svcTS["services.ts\nDEFAULT_CHANNEL_OPTIONS + createCredentials"]
    svcB["Callee service\n(gRPC server impl, in its own domain)"]
    svcA --> loaderTS --> ploader --> gobj
    gobj --> svcTS --> svcB
  end

  protos["libs/proto/src/*.proto (30 files)"]
  protos --> oya
  protos --> ploader
```

The left/top column is the contract-integrity pipeline (Buf for every file, plus
the Oya parity gate for the contracts↔proto↔Rust triangle). The bottom column is
the live consumer path: a service loads a schema, reads a constructor by its
fully-qualified name, and connects with the centralized options and credentials.

---

## Consumer pattern

A domain service that makes gRPC calls follows four steps. `PROTO_PATHS` gives
stable keys so callers never hard-code file paths; `createCredentials` and
`DEFAULT_CHANNEL_OPTIONS` keep TLS and keepalive consistent across the fleet.

```typescript
import {
  loadProto,
  PROTO_PATHS,
  createCredentials,
  DEFAULT_CHANNEL_OPTIONS,
} from '@oshun/proto';

const pkg = await loadProto(PROTO_PATHS.isis); // 1. async load
const IsisJobService = (pkg as any).oshun.isis.IsisJobService; // 2. read ctor by .proto name
const client = new IsisJobService( // 3. centralized creds + options
  'isis-service:50051',
  createCredentials(process.env.NODE_ENV === 'production'),
  DEFAULT_CHANNEL_OPTIONS
);
client.getQueueStats({ queue_name: 'default' }, (err, res) => {
  // 4. snake_case fields
  if (err) throw err;
  console.log(res.pending_jobs);
});
```

Because `keepCase: true`, request and response field names are `snake_case`
exactly as declared. For `procedural` and `reflection`, read the constructor at
its `.proto`-declared name (`ProceduralGenService`, `ServerReflectionService`) —
not the drifted `SERVICE_NAMES` string.

---

## Toolchain, generation, and tests

The schema is managed with **Buf**, not raw `protoc`, so there is no
per-developer binary to install — Buf is invoked through pnpm and
lints/generates identically on every machine. `project.json` exposes `proto:gen`
(`buf generate`) and `proto:lint` (`buf lint`).

- **Lint** uses the `DEFAULT` + `COMMENTS` rule groups, with five rules disabled
  (including `PACKAGE_VERSION_SUFFIX` and `SERVICE_SUFFIX`) and
  `enum_zero_value_suffix: _UNSPECIFIED` enforced.
- **Breaking-change detection** uses the `FILE` rule group against the committed
  `generated/buf-image.json` — a serialized, dependency-resolved
  `FileDescriptorSet`. This image is the **one** committed artifact under
  `generated/`.
- **Generation** runs four plugins into directories declared relative to
  `libs/proto/`: ts-proto → `gen/ts` (note `snakeToCamel=true`, the divergence
  from the runtime loader's `keepCase`), `protocolbuffers/go` + `grpc/go` →
  `gen/go`, and `chrusty/protoc-gen-jsonschema` → `gen/jsonschema`. **These
  outputs are produced on demand and are not committed.**
- A separate, **protobufjs-based** generator (`scripts/generate.ts`,
  `pbjs`/`pbts` static-module types) exists independently of the Buf pipeline
  and no-ops gracefully if the protobufjs CLI is absent.
- **`src/proto.spec.ts`** (Vitest) exercises the loader and registries: it
  asserts `PROTO_PATHS` entries, `keepCase`/channel constants, and that each
  proto loads and exposes its expected constructors — including the Oya messages
  under `pkg.oshun.oya.v1.*`.

---

## Invariants, failure modes, and extension points

- **Leaf invariant.** `@oshun/proto` must never import another Oshun library. A
  PR that adds such an import should be rejected; it would reintroduce the cycle
  risk the boundary exists to prevent.
- **`.proto` is authoritative; the TS registries are conveniences.** When the
  `SERVICE_NAMES` string or a `getServiceMetadata` `methods` array disagrees
  with the schema, trust the schema. Two known name mismatches and several stale
  method lists are documented above — treat them as load-bearing footguns, not
  bugs to silently "fix" by renaming the service.
- **Field numbers are immutable** in `oya.proto` (and should be everywhere) —
  they are the wire identity. New fields take the next free number; reuse or
  renumber breaks decoders silently.
- **Loader is async.** `loadProto`/`loadAllProtos` return Promises because
  `@grpc/proto-loader` parses at startup; a server registering all services must
  `await loadAllProtos()` before binding.
- **`Object.assign` merge is shallow.** `loadProtos` merges by the top-level
  `oshun` segment, so two files declaring colliding deep paths would clobber —
  fine in practice because every package is uniquely namespaced.
- **Adding a service:** drop the `.proto` into a new `src/<domain>/` directory,
  add a `PROTO_PATHS` key, add `SERVICE_NAMES` entries (matching the
  `.proto`-declared names), optionally a `getServiceMetadata` block, run
  `buf lint`/`buf breaking`, and add a `proto.spec.ts` load assertion.
- **Adding to Oya:** change the zod contract, the proto message, and the Rust
  type together, then run `node libs/proto/oya/check-proto-parity.mjs` — a
  one-sided change fails the gate by construction.

---

## Status: implemented vs. planned

Everything described above is **implemented and present in source**: all 30
`.proto` files, the loader/registry/credential helpers, the Buf module + gen
config, the committed breaking-change baseline image, the Oya contract layer,
and the Oya parity gate with its self-test. `proto.spec.ts` loads each schema
and asserts the registries, including Oya.

What this library **does not** contain, by design:

- **No gRPC server or client runtime.** `@oshun/proto` provides schemas, a
  loader, and channel/credential defaults only. The actual servers that
  implement these services and the clients that call them live in their owning
  domains, not here. There is no `services/proto` or `apps/proto`.
- **No committed generated code.** `gen/ts`, `gen/go`, and `gen/jsonschema` are
  produced on demand; only `generated/buf-image.json` is committed.
- **Known, documented drift** in two `SERVICE_NAMES` values and several
  `getServiceMetadata` method lists — surfaced here rather than papered over, so
  consumers know to trust the `.proto` source.

The Oya `.v1` package is the explicit signal of where the convention is heading:
new domains may adopt versioned packages even though the platform's older
services deliberately omit the suffix. This document is scoped to
`libs/proto/*`; RPC-by-RPC rosters live in `specifications.md`, and product
behaviour and REST contracts are owned by the relevant domain docs and
`DOMAINS/openapi/`.