ADR-0005: API Contract Approach - OpenAPI and Protocol Buffers

Status: Accepted Date: 2026-01-10 Authors: Development Team Reviewers: API Team, Architecture Team Supersedes: N/A Superseded by: N/A

Context and Problem Statement#

The Oshun monorepo requires a standardized approach to API contracts that enables type-safe communication between services, clients, and external partners. Currently, both primary codebases use different but complementary approaches:

Lilith API Contracts (Current State):

• OpenAPI 3.1.0 specifications (~1MB comprehensive spec)
• REST APIs for public/partner access
• SSE (Server-Sent Events) for streaming
• WebSocket for real-time bidirectional communication
• Single monolithic spec file at /openapi/openapi.yaml

Yemaya API Contracts (Current State):

• Protocol Buffers (17 proto packages)
• gRPC services for internal communication
• Domains: ai, project, bridge (unreal/godot/blender), collaboration, auth, pipeline, loadbalancing, common types, gaussian splatting, user, rendering, health, agent, asset
• TypeScript code generation from protos
• Go package options for cross-language support

We need to decide on a unified API contract strategy:

1. OpenAPI Only: REST-first with generated clients
2. Protocol Buffers Only: gRPC-first for all communication
3. Both (Recommended): OpenAPI for external APIs, Proto for internal services

This decision affects client generation, documentation, developer experience, performance, and cross-language support.

Decision Drivers#

• External Client Support: Web browsers, mobile apps, third-party integrations
• Internal Performance: Low-latency, high-throughput service communication
• Type Safety: Compile-time validation in TypeScript, Python, Go
• Documentation: Auto-generated API documentation for developers
• Code Generation: SDK generation for multiple languages
• Streaming Support: Bidirectional streaming for real-time features
• Tooling Ecosystem: Available tools for validation, mocking, testing
• Existing Investment: Leverage mature implementations from both codebases
• Team Skills: Familiarity with REST/OpenAPI patterns

Considered Options#

Option 1: OpenAPI Only#

Description: Standardize on OpenAPI 3.1 for all API contracts. Use REST/HTTP for all service communication.

Pros:

• ✅ Universal Compatibility: Works with any HTTP client (browsers, curl, etc.)
• ✅ Excellent Documentation: SwaggerUI, Redoc auto-generated docs
• ✅ Mature Tooling: Extensive ecosystem (validators, mocking, testing)
• ✅ Team Familiarity: Most developers know REST patterns
• ✅ Existing Lilith Specs: Large OpenAPI spec already exists

Cons:

• ❌ Performance Overhead: JSON parsing slower than binary protocols
• ❌ Streaming Limitations: HTTP/1.1 limits true bidirectional streaming
• ❌ No Strong Typing: JSON schemas less strict than protobuf
• ❌ Verbose Payloads: JSON larger than binary alternatives
• ❌ Would Discard: Yemaya's 17 proto packages would be abandoned

Option 2: Protocol Buffers Only#

Description: Standardize on Protocol Buffers for all API contracts. Use gRPC for all service communication.

Pros:

• ✅ Superior Performance: Binary serialization, 2-10x smaller payloads
• ✅ Strong Typing: Strict schema enforcement at compile time
• ✅ Bidirectional Streaming: Native gRPC streaming support
• ✅ Code Generation: Generate clients in 10+ languages
• ✅ Existing Yemaya Protos: 17 proto packages already exist

Cons:

• ❌ Browser Limitations: gRPC-Web adds complexity for web clients
• ❌ Learning Curve: Teams less familiar with protobuf/gRPC
• ❌ Debugging Harder: Binary format not human-readable
• ❌ Documentation: Less mature than OpenAPI tooling
• ❌ Would Discard: Lilith's comprehensive OpenAPI spec

Option 3: Both - OpenAPI for External, Proto for Internal (Recommended)#

Description: Use OpenAPI for public/external REST APIs (clients, partners, web browsers) and Protocol Buffers for internal service communication (gRPC, high-performance paths).

Architecture:

                    ┌─────────────────────────────────────────┐
                    │            External Clients              │
                    │  (Browsers, Mobile Apps, Partners)       │
                    └─────────────────┬───────────────────────┘
                                      │
                           REST/HTTP (OpenAPI)
                                      │
                    ┌─────────────────▼───────────────────────┐
                    │           API Gateway / BFF              │
                    │         (REST → gRPC Translation)        │
                    └─────────────────┬───────────────────────┘
                                      │
                            gRPC (Protocol Buffers)
                                      │
         ┌────────────────────────────┼────────────────────────┐
         │                            │                        │
    ┌────▼────┐                 ┌─────▼─────┐            ┌─────▼─────┐
    │  Isis   │                 │  Sophia   │            │  Hathor   │
    │ Service │◄───gRPC────────►│  Service  │◄───gRPC───►│  Service  │
    └─────────┘                 └───────────┘            └───────────┘

Pros:

• ✅ Best of Both Worlds: REST simplicity for external, gRPC performance for internal
• ✅ Preserves Investment: Uses both Lilith OpenAPI and Yemaya protos
• ✅ Browser Compatibility: Standard REST for web clients
• ✅ High Performance: gRPC for internal service mesh
• ✅ Flexible Streaming: SSE/WebSocket for external, gRPC streams for internal
• ✅ Industry Pattern: Used by Google, Netflix, Uber, etc.

Cons:

• ❌ Two Specs to Maintain: Sync between OpenAPI and Proto required
• ❌ Gateway Complexity: Translation layer needed
• ❌ Learning Both: Teams need both REST and gRPC skills

Decision Outcome#

Chosen option: Option 3 - Both OpenAPI and Protocol Buffers

Justification:

A hybrid approach is optimal for Oshun because:

1. Different Needs, Different Tools: External APIs need browser compatibility and developer-friendly documentation (OpenAPI). Internal services need performance and streaming (gRPC).

2. Preserves Investments: Both Lilith's comprehensive OpenAPI specs and Yemaya's 17 proto packages represent significant work that should be unified, not discarded.

3. Industry Best Practice: Major platforms (Google Cloud, AWS, Stripe) expose REST APIs externally while using gRPC internally. This pattern is proven at scale.

4. Performance Where It Matters: Internal communication between isis, sophia, hathor, bellona is high-volume and latency-sensitive. gRPC's binary protocol provides 2-10x performance improvement.

5. Developer Experience: External developers expect REST APIs with OpenAPI documentation. Internal teams benefit from type-safe gRPC with code generation.

Boundary Rules:

• OpenAPI/REST: Public APIs, client SDKs, partner integrations, web browser access
• Proto/gRPC: Internal service communication, GPU worker communication, real-time streaming between services

Implementation Plan:

1. Phase 1: Contract Packages (Week 1)

   • Create @oshun/openapi package in libs/contracts/openapi/
   • Create @oshun/proto package in libs/contracts/proto/
   • Migrate Lilith OpenAPI specs (modularize into domain specs)
   • Migrate Yemaya proto definitions

2. Phase 2: Code Generation (Week 1-2)

   • Set up OpenAPI TypeScript client generation (openapi-typescript)
   • Set up proto TypeScript generation (ts-proto)
   • Set up proto Python generation (betterproto or grpcio)
   • Create generation scripts in tools/

3. Phase 3: Validation & Linting (Week 2)

   • Configure Spectral for OpenAPI linting
   • Configure buf for proto linting and breaking change detection
   • Add CI checks for contract validation

4. Phase 4: Documentation (Week 2-3)

   • Set up Redoc/SwaggerUI for OpenAPI docs
   • Set up proto documentation generation
   • Create developer portal structure

Success Metrics:

• All public APIs have OpenAPI 3.1 specifications
• All internal services use proto-defined gRPC contracts
• Client SDKs generated from contracts compile without errors
• Zero breaking changes detected by buf in proto evolution
• API documentation available at docs.oshun.dev

Review Schedule: 60 days post-implementation

Implementation Details#

Technical Specifications#

OpenAPI Directory Structure:

libs/contracts/openapi/
├── package.json
├── specs/
│   ├── common/
│   │   ├── errors.yaml
│   │   ├── pagination.yaml
│   │   └── auth.yaml
│   ├── lilith/
│   │   ├── lilith-api.yaml          # Main lilith API
│   │   ├── meditation.yaml
│   │   ├── content.yaml
│   │   └── commerce.yaml
│   ├── yemaya/
│   │   ├── yemaya-api.yaml          # Main yemaya API
│   │   ├── projects.yaml
│   │   ├── assets.yaml
│   │   └── collaboration.yaml
│   ├── isis/
│   │   ├── isis-api.yaml            # Generation API
│   │   ├── jobs.yaml
│   │   └── workflows.yaml
│   ├── sophia/
│   │   └── sophia-api.yaml          # Research API
│   ├── hathor/
│   │   └── hathor-api.yaml          # Worldbuilding API
│   └── bellona/
│       └── bellona-api.yaml         # Build/Export API
├── generated/
│   └── typescript/                   # Generated TS clients
└── scripts/
    ├── generate.ts
    ├── validate.ts
    └── bundle.ts

Protocol Buffers Directory Structure:

libs/contracts/proto/
├── package.json
├── buf.yaml
├── buf.gen.yaml
├── src/
│   ├── common/
│   │   ├── types.proto
│   │   ├── errors.proto
│   │   └── pagination.proto
│   ├── isis/
│   │   ├── generation.proto
│   │   ├── workflow.proto
│   │   └── output.proto
│   ├── sophia/
│   │   ├── ingestion.proto
│   │   ├── search.proto
│   │   ├── citation.proto
│   │   └── knowledge_graph.proto
│   ├── hathor/
│   │   ├── world_model.proto
│   │   ├── narrative.proto
│   │   └── simulation.proto
│   ├── bellona/
│   │   ├── bridge.proto
│   │   ├── build.proto
│   │   └── export.proto
│   ├── yemaya/
│   │   ├── project.proto
│   │   ├── asset.proto
│   │   ├── collaboration.proto
│   │   └── agent.proto
│   └── lilith/
│       ├── meditation.proto
│       ├── content.proto
│       └── commerce.proto
├── generated/
│   ├── typescript/
│   ├── python/
│   └── go/
└── scripts/
    └── generate.sh

OpenAPI Common Schemas:

# libs/contracts/openapi/specs/common/errors.yaml
components:
  schemas:
    Error:
      type: object
      required: [code, message]
      properties:
        code:
          type: string
          description: Machine-readable error code
          example: 'VALIDATION_ERROR'
        message:
          type: string
          description: Human-readable error message
        details:
          type: array
          items:
            $ref: '#/components/schemas/FieldError'
        requestId:
          type: string
          format: uuid
          description: Request ID for support reference

    FieldError:
      type: object
      required: [field, message]
      properties:
        field:
          type: string
          description: Field path (e.g., "user.email")
        message:
          type: string
        code:
          type: string

  responses:
    BadRequest:
      description: Invalid request parameters
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
          example:
            code: 'VALIDATION_ERROR'
            message: 'Invalid request parameters'
            details:
              - field: 'email'
                message: 'Invalid email format'
                code: 'INVALID_FORMAT'

    Unauthorized:
      description: Authentication required
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'

    Forbidden:
      description: Insufficient permissions
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'

    NotFound:
      description: Resource not found
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'

    RateLimited:
      description: Rate limit exceeded
      headers:
        X-RateLimit-Limit:
          schema:
            type: integer
        X-RateLimit-Remaining:
          schema:
            type: integer
        X-RateLimit-Reset:
          schema:
            type: integer
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'

Proto Common Types (Enhanced):

// libs/contracts/proto/src/common/types.proto
syntax = "proto3";

package oshun.common;

option go_package = "github.com/oshun/proto/common";

import "google/protobuf/timestamp.proto";
import "google/protobuf/struct.proto";

// Branded UUID for type safety
message UUID {
  string value = 1;
}

// Branded IDs per domain
message UserId {
  string value = 1;
}

message ProjectId {
  string value = 1;
}

message AssetId {
  string value = 1;
}

message JobId {
  string value = 1;
}

// Pagination
message PaginationRequest {
  int32 page = 1;           // 1-indexed
  int32 limit = 2;          // Max 100
  string sort_by = 3;
  SortOrder order = 4;
  string cursor = 5;        // For cursor-based pagination
}

enum SortOrder {
  SORT_ORDER_UNSPECIFIED = 0;
  SORT_ORDER_ASC = 1;
  SORT_ORDER_DESC = 2;
}

message PaginationMeta {
  int32 page = 1;
  int32 limit = 2;
  int64 total = 3;
  int32 total_pages = 4;
  bool has_next = 5;
  bool has_previous = 6;
  string next_cursor = 7;
}

// Error handling
message Error {
  string code = 1;
  string message = 2;
  repeated FieldError details = 3;
  string request_id = 4;
  google.protobuf.Struct metadata = 5;
}

message FieldError {
  string field = 1;
  string message = 2;
  string code = 3;
}

// Service mesh context
message RequestContext {
  string request_id = 1;
  string trace_id = 2;
  string span_id = 3;
  UserId user_id = 4;
  string organization_id = 5;
  google.protobuf.Timestamp timestamp = 6;
  map<string, string> metadata = 7;
}

// Health
enum HealthStatus {
  HEALTH_STATUS_UNSPECIFIED = 0;
  HEALTH_STATUS_HEALTHY = 1;
  HEALTH_STATUS_DEGRADED = 2;
  HEALTH_STATUS_UNHEALTHY = 3;
}

message HealthCheckRequest {
  string service = 1;
}

message HealthCheckResponse {
  HealthStatus status = 1;
  string service = 2;
  string version = 3;
  google.protobuf.Timestamp timestamp = 4;
  int64 uptime_seconds = 5;
  repeated ServiceDependency dependencies = 6;
}

message ServiceDependency {
  string name = 1;
  HealthStatus status = 2;
  int64 latency_ms = 3;
  string error = 4;
}

Buf Configuration:

# libs/contracts/proto/buf.yaml
version: v1
name: buf.build/oshun/proto
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT
    - COMMENTS
  except:
    - PACKAGE_VERSION_SUFFIX
  enum_zero_value_suffix: _UNSPECIFIED
  rpc_allow_same_request_response: false
  rpc_allow_google_protobuf_empty_requests: true
  rpc_allow_google_protobuf_empty_responses: true
  service_suffix: Service

Code Generation Configuration:

# libs/contracts/proto/buf.gen.yaml
version: v1
managed:
  enabled: true
  go_package_prefix:
    default: github.com/oshun/proto
plugins:
  # TypeScript generation
  - plugin: buf.build/community/timostamm-protobuf-ts
    out: generated/typescript
    opt:
      - long_type_string
      - generate_dependencies
      - output_typescript

  # Python generation
  - plugin: buf.build/protocolbuffers/python
    out: generated/python

  # Go generation
  - plugin: buf.build/protocolbuffers/go
    out: generated/go
    opt:
      - paths=source_relative

  # gRPC for Go
  - plugin: buf.build/grpc/go
    out: generated/go
    opt:
      - paths=source_relative

OpenAPI TypeScript Generation:

// libs/contracts/openapi/scripts/generate.ts
import { execSync } from 'child_process';
import { readFileSync, writeFileSync } from 'fs';
import { glob } from 'glob';

async function generateTypeScriptClients() {
  const specs = await glob('specs/**/*.yaml', { cwd: __dirname + '/..' });

  for (const spec of specs) {
    const domain = spec.split('/')[1]; // e.g., "lilith", "yemaya"
    const output = `generated/typescript/${domain}`;

    execSync(`npx openapi-typescript ../specs/${spec} -o ${output}/types.ts`, {
      cwd: __dirname + '/..',
    });

    // Generate fetch client
    execSync(`npx openapi-fetch ../specs/${spec} -o ${output}/client.ts`, {
      cwd: __dirname + '/..',
    });

    console.log(`Generated TypeScript client for ${domain}`);
  }
}

generateTypeScriptClients();

API Design Guidelines#

OpenAPI Guidelines:

1. Use OpenAPI 3.1.0 (JSON Schema compatible)
2. Define all schemas in components/schemas
3. Use $ref for shared schemas
4. Include examples for all schemas
5. Document all error responses
6. Use semantic versioning in paths (/v1/, /v2/)
7. Use kebab-case for paths
8. Use camelCase for JSON properties

Proto Guidelines:

1. Use proto3 syntax
2. Define common types in common/ package
3. Use google.protobuf.Timestamp for times
4. Use wrapper types for optional primitives
5. Add comments for all messages and fields
6. Use streaming for large data transfers
7. Define service methods with clear request/response types
8. Use meaningful enum value names with domain prefix

Migration Strategy#

OpenAPI Migration:

1. Extract Lilith monolithic spec into domain modules
2. Create shared common/ schemas
3. Update service implementations to validate against specs
4. Generate client SDKs from modular specs

Proto Migration:

1. Migrate Yemaya protos to new structure
2. Add missing domains (isis, sophia, hathor, bellona)
3. Unify common types across all protos
4. Update code generation for new structure

Consequences#

Positive Consequences#

• ✅ Optimized for Use Case: REST for external, gRPC for internal
• ✅ Preserves Investment: Both existing specs continue to evolve
• ✅ Type Safety: Generated clients in TypeScript, Python, Go
• ✅ Documentation: Auto-generated from contracts
• ✅ Performance: gRPC for high-throughput internal paths
• ✅ Browser Compatibility: Standard REST for web clients
• ✅ Industry Alignment: Follows proven hybrid pattern

Negative Consequences#

• ❌ Dual Maintenance: Two contract systems to maintain
• ❌ Sync Required: Changes may need updates in both specs
• ❌ Learning Curve: Teams need both REST and gRPC skills
• ❌ Gateway Complexity: Translation layer adds latency

Risks and Mitigation#

Compliance and Security#

Security Implications#

• Input Validation: All inputs validated against schemas
• Authentication: Bearer tokens in OpenAPI, gRPC metadata for internal
• Rate Limiting: Documented in OpenAPI specs
• Audit Logging: Request IDs propagated through both protocols

Compliance Requirements#

• API Versioning: Semantic versions in both OpenAPI and Proto
• Deprecation Policy: 90-day notice before breaking changes
• Documentation: All APIs documented for compliance audits

Monitoring and Observability#

Metrics to Track#

• OpenAPI: Request count, latency, error rates per endpoint
• gRPC: Call count, latency, stream duration, error codes
• Generation: Build time, generated file count, validation errors

Alerting Strategy#

• Critical: API validation failures in production, gRPC unavailable
• Warning: High latency, error rate spike, deprecated endpoint usage
• Info: New API version deployed, contract changes

Related Decisions#

Upstream Dependencies#

• ADR-0001: Git Consolidation (single repo for contract packages)
• ADR-0002: pnpm (workspace dependencies)
• ADR-0004: Eventing (event contracts use similar patterns)

Downstream Impacts#

• Client SDKs: Generated from OpenAPI specs
• Internal Services: Use gRPC contracts
• API Gateway: Translates REST to gRPC
• Documentation Portal: Generated from both specs

References#

External Resources#

• OpenAPI Specification
• Protocol Buffers Language Guide
• gRPC Documentation
• buf Documentation
• openapi-typescript

Internal Resources#

• Lilith OpenAPI spec — lilith/openapi/openapi.yaml (pre-consolidation source repo)
• Yemaya proto package — yemaya/packages/proto/ (pre-consolidation source repo)

Revision History#