Proto Domain — Architecture

Architectural overview of @oshun/proto (libs/proto/): why gRPC exists alongside REST, how the .proto schema surface is organized, how a tiny TypeScript runtime layer loads those schemas, how the Buf toolchain and the Oya parity gate keep everything honest, and how consuming services wire it all together.

What this domain is#

@oshun/proto is the single source of truth for all gRPC communication inside the Oshun platform. Every .proto file lives here, and every service that calls another service over gRPC depends on this library for schema definitions, a runtime loader, and channel helpers. The library owns no product logic — it is purely a schema contract plus a thin loading/configuration layer.

It is a leaf in the monorepo dependency graph. package.json lists only three runtime dependencies — @grpc/grpc-js, @grpc/proto-loader, and protobufjs — and no dependency on any other Oshun library. project.json tags it scope:shared, layer:contracts. The boundary is deliberate: if @oshun/proto depended on a domain library, a change in that domain could break every service that speaks gRPC, and import cycles would become possible. Every gRPC-speaking domain depends on @oshun/proto; @oshun/proto depends on none of them.

There is no services/proto and no apps/proto — this domain is a library only. The gRPC servers and clients that implement and consume these contracts live in their respective domain packages (Isis, Sophia, Hathor, the engine bridges, …); @oshun/proto gives them the schemas, the loader, and the channel/credential defaults to do so consistently.

Two roles in one package#

src/index.ts re-exports exactly two modules' worth of surface, reflecting the library's dual role:

1. Schema home. It owns every .proto describing an Oshun gRPC surface — currently 30 .proto files under libs/proto/src/, one directory per service domain. (git ls-files 'libs/proto/src/**/*.proto' returns 30, and the PROTO_PATHS registry has 30 matching entries.)
2. Runtime helper layer. A small TypeScript surface — loader.ts, services.ts, index.ts — that loads .proto files at runtime via @grpc/proto-loader, exposes typed path/name/metadata registries, and centralizes gRPC channel options and credential construction.

The raw schemas are also exposed directly to consumers: the package.json exports map carries a "./protos/*": "./src/*.proto" subpath, and the Nx build target copies **/*.proto from src into protos/ in the build output.

gRPC vs REST in Oshun#

The platform uses both protocols because they serve different audiences. Browser clients and external API consumers use REST (via @oshun/openapi) because HTTP/JSON is universally understood and easy to introspect. Internal service-to-service calls use gRPC: Protocol Buffers encode messages in a compact binary format, gRPC multiplexes calls over a single HTTP/2 connection, and both request and response types are enforced at schema level. For streaming job progress to a UI, bidirectionally synchronizing a live document, or coordinating a render farm, gRPC's native streaming modes are a natural fit that REST cannot match without polling or SSE workarounds.

Source layout#

libs/proto/
├── package.json            # @oshun/proto, 3 grpc deps, "./protos/*" export
├── project.json            # Nx: build, lint, test, proto:gen, proto:lint
├── buf.work.yaml           # buf workspace: directories = [src]
├── buf.gen.yaml            # buf codegen: ts-proto, Go, Go-gRPC, JSON Schema
├── generated/
│   └── buf-image.json      # serialized FileDescriptorSet (breaking-change baseline)
├── scripts/
│   └── generate.ts         # separate pbjs/pbts static-module type generator
├── oya/
│   └── check-proto-parity.mjs   # Oya proto ⇄ zod-contract parity gate (§ below)
└── src/
    ├── index.ts            # public API re-exports (loader + services + grpc types)
    ├── loader.ts           # loadProto/loadProtos/loadAllProtos + PROTO_PATHS
    ├── services.ts         # SERVICE_NAMES, DEFAULT_CHANNEL_OPTIONS, createCredentials
    ├── proto.spec.ts       # Vitest suite for loader + registries + per-proto load
    ├── buf.yaml            # SINGLE buf module config for the whole src/ tree
    │
    ├── common/types.proto              # oshun.common — shared scalar vocabulary
    ├── shared/{common,evidence,memory,persona_policy,generation_control}.proto
    ├── auth/, ai/, agent/, asset/, collaboration/, project/, user/   # core
    ├── isis/, sophia/, hathor/, concordia/   # domain services
    ├── oya/oya.proto                   # oshun.oya.v1 — embodied-hive contracts (NO service)
    ├── generation3d/, rendering/, splatting/, procedural/   # rendering & 3D
    ├── bridge/{blender,godot,unreal}.proto   # DCC/engine bridges
    ├── health/, loadbalancing/, reflection/, pipeline/   # infrastructure
    └── oshun/v2/persistent_economy/economy.proto   # versioned V2 game services

Exactly one buf.yaml, at src/buf.yaml, defines a single Buf module (buf.build/oshun/proto) covering the entire src/ subtree. There are no per-domain buf.yaml files. The library root holds buf.work.yaml (workspace config pointing at src) and buf.gen.yaml (code-generation config).

The runtime helper layer#

src/loader.ts — loading schemas at runtime#

loader.ts is the bridge between the static .proto files and the live gRPC clients services instantiate at startup. Rather than importing pre-generated JavaScript stubs, a service calls loadProto (or loadAllProtos at server start), gets back a grpc.GrpcObject, and reads the service constructor out of it by its fully-qualified package path. This keeps the .proto files the single source of truth at both build and run time, at the cost of a small async parse on startup. The module computes its own directory via fileURLToPath(import.meta.url) — the pure-ESM pattern, since the package is "type": "module".

• loadProto(path, options?) — async; resolves a relative path against src/, merges { ...DEFAULT_LOADER_OPTIONS, ...options }, then protoLoader.load → grpc.loadPackageDefinition. Returns a Promise<grpc.GrpcObject>.
• loadProtos(paths[], options?) — loads several files and shallow-merges them with Object.assign keyed on the top-level oshun package segment, so every file's subpackages accrete onto one shared root.
• loadAllProtos(options?) — loadProtos(Object.values(PROTO_PATHS)); used where a server registers every service at once.
• getProtoPath(relativePath) — pure path helper joining onto src/ and returning the absolute path (it does not load or check existence).
• PROTO_PATHS — an as const registry of 30 stable keys → .proto paths relative to src/ (e.g. PROTO_PATHS.isis → isis/isis.proto, PROTO_PATHS.oya → oya/oya.proto). The exported ProtoPath type is the union of those path literals.

The most consequential decision lives in DEFAULT_LOADER_OPTIONS:

export const DEFAULT_LOADER_OPTIONS: protoLoader.Options = {
  keepCase: true, // field names stay snake_case — NOT camelCased
  longs: String, // int64/uint64 surfaced as JS strings
  enums: String, // enum values surfaced as string names
  defaults: true, // default values included in decoded output
  oneofs: true, // virtual oneof discriminator field included
  includeDirs: [__dirname, path.join(__dirname, '..')],
};

keepCase: true means runtime field names are exactly as declared in the .proto (queue_name, pending_jobs), which is the opposite of what buf generate's ts-proto path produces (snakeToCamel=true). The two includeDirs entries — src/ and its parent — are what let a domain file's import "common/types.proto" resolve. proto.spec.ts asserts keepCase === true.

src/services.ts — channel names, options, credentials#

Where loader.ts parses schemas, services.ts centralizes everything needed to construct and configure a channel:

• SERVICE_NAMES — an as const map of registry key → fully-qualified gRPC service name (e.g. SERVICE_NAMES.IsisJob → oshun.isis.IsisJobService). ServiceName is the union of those values.
• DEFAULT_CHANNEL_OPTIONS — shared keepalive timing (30 s ping, 10 s ack timeout, pings permitted without calls), HTTP/2 ping spacing, and 50 MB inbound/outbound message-size limits. The 50 MB ceiling accommodates the largest in-message binaries in the system: captured frame bytes on splatting's UploadFrames, viewport captures from the engine bridges, and inline document content for Sophia ingestion. The 50 MB limit and the keepalive durations are asserted by proto.spec.ts.
• createCredentials(secure, rootCerts?, privateKey?, certChain?) — channel credential factory. secure: false → insecure; secure: true with a client key + chain → mutual TLS; secure: true without → server-authenticated TLS (falling back to the system trust store when rootCerts is undefined). It requires @grpc/grpc-js dynamically into a local binding as an explicit ESM/CJS interop accommodation.
• getServiceMetadata(name) — returns { name, protoPath, package, methods } for a registered service, or undefined.

Two honest caveats that a maintainer must know. First, two SERVICE_NAMES entries disagree with their .proto source and are not wire-correct: Procedural is registered as …ProceduralGenerationService but procedural.proto declares service ProceduralGenService; Reflection is registered as …ReflectionService but reflection.proto declares ServerReflectionService. Read those service constructors from the package object at their .proto-declared names. Second, the methods arrays inside getServiceMetadata are a hand-maintained summary that has drifted — they are accurate for some services and stale for others (e.g. the AI and Agent entries list RPC names that no longer match the schema). The .proto files are authoritative for RPC rosters; the specifications.md catalog enumerates them directly from source.

Schema organization#

Every .proto is syntax = "proto3", uses an oshun.<domain> package, declares option go_package = "github.com/oshun/proto/<path>", suffixes enum zero values with _UNSPECIFIED, and names fields in snake_case. Files import google/protobuf/timestamp.proto, struct.proto, or duration.proto as needed, and domain files import common/types.proto for shared types.

Three structural tiers sit underneath the per-domain service files:

• common/types.proto (package oshun.common) is a pure type vocabulary with no service — UUID, PaginationRequest/PaginationMeta, Error/FieldError, Empty, SuccessResponse, and a health-types pair. Defining them once guarantees a PaginationRequest means the same thing in an auth call as in a rendering call. Most "no payload" RPCs return common.SuccessResponse; every list RPC pages with the common pagination pair.
• shared/*.proto are not a type bag — they are four product-facing substrate service contracts (plus shared/common.proto, which carries only cross-substrate enums and a SharedContractVersionDescriptor): OshunEvidenceService (Sophia grounding), OshunMemoryService (Iris continuity), OshunPersonaPolicyService (Lilith policy), and OshunGenerationControlService (Isis generation control). These are consumed directly by product domains (Tara, Arete, Veritas, Nyx, Nisaba, the Assistant shell), expressing cross-cutting concerns no single product domain should own.
• Domain service files (isis, sophia, hathor, concordia, the engine bridges, rendering/3D, pipeline, infrastructure) each declare one or more services for one domain's gRPC surface. A single file may declare several services (sophia.proto declares five, hathor.proto seven). concordia.proto enforces a viewer-role privacy invariant at schema level for mediation streams.

Package-version convention — and its one exception#

The historical convention is oshun.<domain> with no .v1 suffix; Buf's PACKAGE_VERSION_SUFFIX lint rule is disabled in src/buf.yaml precisely to allow this. There are now two deliberate exceptions, and they should be read as the intended direction of travel, not drift:

• oshun/v2/persistent_economy/economy.proto uses package oshun.v2.persistent_economy for the Section-130 open-world game services (Economy, NPCSchedule, CrimeRate).
• oya/oya.proto uses package oshun.oya.v1 — the first domain to adopt a trailing .v1. proto.spec.ts confirms its messages resolve under pkg.oshun.oya.v1.* (not pkg.oshun.oya.*). Field numbers in this file are treated as the stable wire identity and must never be reused or renumbered.

The Oya embodied-hive contract layer#

oya/oya.proto is the newest and most distinctive member of the library, and it behaves differently from every service file above: it declares no service at all. Like common/types.proto, it is a pure message/enum vocabulary — but where common is generic platform plumbing, Oya is the wire form of a specific embodied domain: a hive of drones/robots that share telemetry, allocate tasks, maintain a spatial world model, and enforce safety and privacy.

Its reason for existing is tri-directional parity. The same vocabulary is spoken in three places, and all three must agree byte-for-byte:

1. the canonical zod contracts in libs/contracts/src/oya (@oshun/contracts/oya);
2. the proto wire messages in oya/oya.proto;
3. the Rust engine/hive crates in libs/oya/engine/crates/ — oya-types, oya-math, oya-scenegraph, oya-fleet, oya-safety, oya-comms and siblings — that actually run the control loops.

The file's header documents the mirroring rules explicitly (zod z.number() → double; z.number().int() → int32/int64; z.enum → proto enum with a _UNSPECIFIED = 0 zero value; z.array → repeated; z.record → map; z.union/discriminated → oneof). The message set covers 3D math primitives (Vec3, Quaternion, GpsCoordinate), telemetry/control (Telemetry, ControlCommand, the ControlMode enum), mission/flight-plan (Mission, FlightPlan, Geofence, Waypoint), CBBA fleet allocation (FleetState, TaskBid, Allocation), the shared spatial world model (SceneGraphInstance, Aabb, WorldModelQuery/WorldModelResult), sensor observations, capability manifests, docking/battery-swap, the ISO/TS 15066 SafetyEnvelope, and privacy (ConsentPolicy, PrivacyZone).

Two domain invariants are encoded structurally in the schema and are worth calling out because they are the point of the design:

• Fail-loud spatial memory. WorldModelResult is a three-arm oneof (Fresh / Stale / Unknown) mirroring the Rust QueryResult<V>. A decayed memory returns Stale (no value) or Unknown rather than fabricating a confident answer — the schema makes a silent confident-lie unrepresentable.
• Privacy by omission. SensorObservation carries only a CompressedDescriptor (a oneof of an opaque codec blob or an explicit FeatureVector) — there is intentionally no raw-frame field. Consent (ConsentPolicy/ConsentScope) is fail-closed: an absent scope means consent was not given.

The parity gate (oya/check-proto-parity.mjs)#

Tri-directional parity is not a comment — it is enforced by a real fail-loud gate. oya/check-proto-parity.mjs parses oya.proto with protobufjs and imports @oshun/contracts/oya through the tsx ESM loader, then proves three properties per schema: coverage (every zod object/union/enum schema has a mapped proto message — an unmapped new contract is flagged, not skipped), field-set equality (proto field names match the zod keys after case-normalization), and enum-set equality (members match, allowing exactly the one extra _UNSPECIFIED = 0). Any drift in either direction exits non-zero.

What makes it trustworthy is the drift-detector self-test that runs before the real check: the script clones the parsed proto, deliberately drops a field (Telemetry.voltage), renames one (ControlCommand.armed → armed_flag), and removes an enum member (CAPABILITY_GRASP), then asserts the checker reports all three. If the checker fails to catch its own injected drift it exits 2 — "the checker is broken and cannot be trusted to gate anything" — rather than emitting a false pass.

Component & data flow#

The left/top column is the contract-integrity pipeline (Buf for every file, plus the Oya parity gate for the contracts↔proto↔Rust triangle). The bottom column is the live consumer path: a service loads a schema, reads a constructor by its fully-qualified name, and connects with the centralized options and credentials.

Consumer pattern#

A domain service that makes gRPC calls follows four steps. PROTO_PATHS gives stable keys so callers never hard-code file paths; createCredentials and DEFAULT_CHANNEL_OPTIONS keep TLS and keepalive consistent across the fleet.

import {
  loadProto,
  PROTO_PATHS,
  createCredentials,
  DEFAULT_CHANNEL_OPTIONS,
} from '@oshun/proto';

const pkg = await loadProto(PROTO_PATHS.isis); // 1. async load
const IsisJobService = (pkg as any).oshun.isis.IsisJobService; // 2. read ctor by .proto name
const client = new IsisJobService( // 3. centralized creds + options
  'isis-service:50051',
  createCredentials(process.env.NODE_ENV === 'production'),
  DEFAULT_CHANNEL_OPTIONS
);
client.getQueueStats({ queue_name: 'default' }, (err, res) => {
  // 4. snake_case fields
  if (err) throw err;
  console.log(res.pending_jobs);
});

Because keepCase: true, request and response field names are snake_case exactly as declared. For procedural and reflection, read the constructor at its .proto-declared name (ProceduralGenService, ServerReflectionService) — not the drifted SERVICE_NAMES string.

Toolchain, generation, and tests#

The schema is managed with Buf, not raw protoc, so there is no per-developer binary to install — Buf is invoked through pnpm and lints/generates identically on every machine. project.json exposes proto:gen (buf generate) and proto:lint (buf lint).

• Lint uses the DEFAULT + COMMENTS rule groups, with five rules disabled (including PACKAGE_VERSION_SUFFIX and SERVICE_SUFFIX) and enum_zero_value_suffix: _UNSPECIFIED enforced.
• Breaking-change detection uses the FILE rule group against the committed generated/buf-image.json — a serialized, dependency-resolved FileDescriptorSet. This image is the one committed artifact under generated/.
• Generation runs four plugins into directories declared relative to libs/proto/: ts-proto → gen/ts (note snakeToCamel=true, the divergence from the runtime loader's keepCase), protocolbuffers/go + grpc/go → gen/go, and chrusty/protoc-gen-jsonschema → gen/jsonschema. These outputs are produced on demand and are not committed.
• A separate, protobufjs-based generator (scripts/generate.ts, pbjs/pbts static-module types) exists independently of the Buf pipeline and no-ops gracefully if the protobufjs CLI is absent.
• src/proto.spec.ts (Vitest) exercises the loader and registries: it asserts PROTO_PATHS entries, keepCase/channel constants, and that each proto loads and exposes its expected constructors — including the Oya messages under pkg.oshun.oya.v1.*.

Invariants, failure modes, and extension points#

• Leaf invariant. @oshun/proto must never import another Oshun library. A PR that adds such an import should be rejected; it would reintroduce the cycle risk the boundary exists to prevent.
• .proto is authoritative; the TS registries are conveniences. When the SERVICE_NAMES string or a getServiceMetadata methods array disagrees with the schema, trust the schema. Two known name mismatches and several stale method lists are documented above — treat them as load-bearing footguns, not bugs to silently "fix" by renaming the service.
• Field numbers are immutable in oya.proto (and should be everywhere) — they are the wire identity. New fields take the next free number; reuse or renumber breaks decoders silently.
• Loader is async. loadProto/loadAllProtos return Promises because @grpc/proto-loader parses at startup; a server registering all services must await loadAllProtos() before binding.
• Object.assign merge is shallow. loadProtos merges by the top-level oshun segment, so two files declaring colliding deep paths would clobber — fine in practice because every package is uniquely namespaced.
• Adding a service: drop the .proto into a new src/<domain>/ directory, add a PROTO_PATHS key, add SERVICE_NAMES entries (matching the .proto-declared names), optionally a getServiceMetadata block, run buf lint/buf breaking, and add a proto.spec.ts load assertion.
• Adding to Oya: change the zod contract, the proto message, and the Rust type together, then run node libs/proto/oya/check-proto-parity.mjs — a one-sided change fails the gate by construction.

Status: implemented vs. planned#

Everything described above is implemented and present in source: all 30 .proto files, the loader/registry/credential helpers, the Buf module + gen config, the committed breaking-change baseline image, the Oya contract layer, and the Oya parity gate with its self-test. proto.spec.ts loads each schema and asserts the registries, including Oya.

What this library does not contain, by design:

• No gRPC server or client runtime. @oshun/proto provides schemas, a loader, and channel/credential defaults only. The actual servers that implement these services and the clients that call them live in their owning domains, not here. There is no services/proto or apps/proto.
• No committed generated code. gen/ts, gen/go, and gen/jsonschema are produced on demand; only generated/buf-image.json is committed.
• Known, documented drift in two SERVICE_NAMES values and several getServiceMetadata method lists — surfaced here rather than papered over, so consumers know to trust the .proto source.

The Oya .v1 package is the explicit signal of where the convention is heading: new domains may adopt versioned packages even though the platform's older services deliberately omit the suffix. This document is scoped to libs/proto/*; RPC-by-RPC rosters live in specifications.md, and product behaviour and REST contracts are owned by the relevant domain docs and DOMAINS/openapi/.