# CI/CD Pipelines

## Overview

The Oshun monorepo uses GitHub Actions for continuous integration and
deployment. There are 31 workflow files organized into core pipelines and
domain-specific pipelines.

---

## Core Pipelines

### `ci.yml` - Main CI

**Triggers:** Push to `main`/`develop`, pull requests, nightly schedule

The primary CI pipeline validates all affected projects:

1. **Setup & Analysis** - Detects affected projects using Nx
2. **Dependency Install** - `pnpm install` with caching
3. **Lint** - ESLint across affected projects
4. **Type Check** - TypeScript compilation checks
5. **Build** - Compiles affected projects
6. **Unit Tests** - Vitest across affected projects

**Environment:** Node.js 22, pnpm 10

### `e2e.yml` - End-to-End Tests

**Triggers:** Push to `main`, daily schedule

Runs Playwright E2E tests sharded across 3 parallel jobs. Starts PostgreSQL and
Redis as service containers for realistic integration testing.

### `pr-quality.yml` - Pull Request Quality Gates

**Triggers:** Pull requests

Enforces code quality on PRs:

- Coverage thresholds (80% minimum)
- Lint compliance
- Performance regression checks

### `database.yml` - Database Migrations

**Triggers:** PR (validation), push to `staging`/`production` branches

Validates Prisma migrations on PR. Runs actual migrations against staging and
production databases on merge. Supports per-domain migration:

- Domains: yemaya, lilith, isis, sophia, hathor, bellona

### `release.yml` - Release Management

**Triggers:** Manual dispatch

Handles version bumping, changelog generation, and GitHub release creation.

### `codeql.yml` - Security Scanning

**Triggers:** Push to `main`, weekly schedule

GitHub CodeQL static analysis for JavaScript/TypeScript security
vulnerabilities.

### `accessibility.yml` - Accessibility Audits

**Triggers:** Push to `main`, manual dispatch

WCAG compliance testing using axe-core against web applications.

### `benchmarks.yml` - Performance Benchmarks

**Triggers:** Weekly schedule, manual dispatch

Runs k6 load tests in smoke, load, and stress modes against API endpoints.

---

## Deployment Pipelines

### `deploy-ecs.yml` - AWS ECS Deployment

**Triggers:** Manual dispatch, called by domain pipelines

Orchestrates deployment to AWS ECS:

- **Staging:** Cluster `oshun-staging`
- **Production:** Cluster `oshun-production`
- **Registry:** `${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com`

Supports per-domain and per-service deployment with health check verification.

### `deploy-runpod.yml` - RunPod GPU Deployment

**Triggers:** Manual dispatch

Deploys GPU workloads (generation workers, inference services) to RunPod
infrastructure.

### `runpod-endpoint-image-build.yml` - RunPod Endpoint Image Build

**Triggers:** Push to `main`, pull request, manual dispatch

Builds RunPod endpoint images from manifest-driven build definitions:

- Generates endpoint image manifests from
  `infra/runpod/endpoints/image-build-config.json` +
  `infra/runpod/endpoints/desired-state.json`
- Enforces warm/cold policy alignment from
  `infra/runpod/endpoints/warm-cold-policy.json`
- Generates per-endpoint model/custom-node inventory manifests under
  `infra/runpod/endpoints/inventory`
- Validates manifest parity and desired-state coverage before building
- Builds base + endpoint image matrix using
  `scripts/isis/generate_runpod_endpoint_image_build_matrix.mjs`

### `runpod-endpoint-lifecycle.yml` - RunPod Endpoint Lifecycle

**Triggers:** Manual dispatch

Runs endpoint lifecycle automation (create/update/delete/batch) using
`scripts/isis/runpod_endpoint_lifecycle.py` against RunPod GraphQL APIs. Desired
endpoint configuration is sourced from
`infra/runpod/endpoints/desired-state.json` and validated by
`scripts/isis/validate_runpod_desired_state.py` (including canary endpoint IDs
and canary traffic percentage bounds for staged rollout). The workflow also
enforces warm/cold policy alignment via
`scripts/isis/apply_runpod_warm_cold_policy.mjs --check`.

### `runpod-endpoint-drift-detection.yml` - RunPod Drift Detection

**Triggers:** Scheduled, manual dispatch

Compares live RunPod endpoint configuration against
`infra/runpod/endpoints/desired-state.json` using
`scripts/isis/detect_runpod_endpoint_drift.py` after verifying warm/cold policy
alignment.

### `runpod-endpoint-reconcile.yml` - RunPod Drift Reconcile

**Triggers:** Manual dispatch

Runs `scripts/isis/reconcile_runpod_endpoint_drift.sh` to detect drift, apply
desired state, and verify post-reconcile convergence, with warm/cold policy
alignment checks as a preflight gate.

### RunPod FinOps Attribution Validation (WS09-005)

RunPod pay-per-second attribution behavior is validated in automated tests
covering:

- adapter runtime accounting and dimensional aggregation
  (`workflow`/`project`/`tenant`)
- API metadata propagation for tenant/project context
- web operations surfaces rendering attribution snapshots

Verification commands:

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/services/comfyui.service.spec.ts src/routes/jobs.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
```

### RunPod Endpoint SLO Telemetry Validation (WS09-006)

RunPod endpoint-level SLO metrics are validated with focused tests that cover:

- adapter SLO sampling and p95/rate calculations
- API route payloads exposing endpoint SLO snapshots enriched with endpoint
  topology metadata
- web operations UI rendering objective thresholds, endpoint metrics, and breach
  flags

Verification commands:

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
```

### RunPod Anomaly Detection Validation (WS09-007)

RunPod anomaly telemetry behavior is validated with tests covering:

- adapter-level detection for cost spikes, timeout storms, and model-cache
  thrashing
- API route payload exposure with endpoint topology enrichment
- web operations rendering for anomaly objectives and active alerts

Verification commands:

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
```

### RunPod Endpoint Canary Strategy Validation (WS09-008)

RunPod endpoint-family canary strategy and rollback triggers are validated by:

- route tests asserting canary action derivation from SLO/anomaly telemetry
- web tests rendering strategy action/reason surfaces for operators
- execution-script smoke checks for callback invocation and rollback exit
  semantics

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
FAIL_ON_ROLLBACK=false OPERATIONS_FILE=/tmp/runpod-operations.json scripts/isis/execute_runpod_endpoint_canary_strategy.sh
```

### RunPod Fleet Topology and Economics Validation (WS09-009)

RunPod fleet topology/economics coverage is validated with tests covering:

- adapter endpoint-level cost attribution dimensions (including endpoint keys)
- API route exposure of fleet topology + projected/observed economics
- web operations rendering of fleet source/topology/economics tables

Verification commands:

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
```

### RunPod DR Runbooks and Game-Day Validation (WS09-010)

RunPod disaster-recovery readiness is validated by:

- provider outage drill automation runbook coverage
- endpoint-region failover drill automation runbook coverage
- composite game-day harness that executes both drills and emits a unified
  report

Verification commands:

```bash
bash -n scripts/isis/run_provider_outage_drill.sh scripts/isis/run_endpoint_region_failover_drill.sh scripts/isis/run_runpod_disaster_recovery_gameday.sh
node --test scripts/operations/v1-game-day.test.mjs
# A smoke run of the harness. It injects nothing and executes no scenario, so it
# reports `overall.outcome: "inconclusive"` -- which is the point: until S10.12
# it reported `overall.pass: true` and logged "all game-day checks passed" for
# exactly this invocation.
STRICT=false DRY_RUN=true RUN_PROVIDER_FAILURE_SCENARIO=false RUN_REGION_FAILURE_SCENARIO=false scripts/isis/run_runpod_disaster_recovery_gameday.sh
```

A game day that establishes something needs the fault actually applied and the
instants an operator observed. `FAULT_INJECTED_AT_EPOCH` and `DETECTED_AT_EPOCH`
are what turn the report's recovery time into an interval an actor lived
through; without `DETECTED_AT_EPOCH` the run stays `inconclusive`, because a
recovery time measured from the moment somebody ran the script excludes
detection and decision.

```bash
STRICT=true DRY_RUN=false \
  RUN_PROVIDER_FAILURE_SCENARIO=true RUN_REGION_FAILURE_SCENARIO=true \
  FAULT_INJECTED_AT_EPOCH=$(date -u +%s) \
  DETECTED_AT_EPOCH=<when the alert fired> DECLARED_AT_EPOCH=<when it was declared> \
  ISIS_API_BASE_URL=... ISIS_ADMIN_TOKEN=... \
  scripts/isis/run_runpod_disaster_recovery_gameday.sh
```

### Isis OpenAPI Canonical Generation Validation (WS10-001)

Isis generation-api OpenAPI publication is validated by:

- deterministic route-derived OpenAPI generation
- canonical artifact parity (`apps/isis/generation-api/openapi.yaml` and
  `openapi/isis/openapi.yaml`)
- contract test enforcement for route/spec alignment

Verification commands:

```bash
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
pnpm --filter @isis/generation-api exec vitest run src/openapi.contract.spec.ts --reporter=dot
```

### Workflow Execution Endpoints + RBAC Validation (WS10-002)

Workflow execution parity and RBAC enforcement are validated by:

- route-level tests for `/api/v1/workflows/:workflowId/run` and
  `/api/v1/workflows/:workflowId/executions*`
- OpenAPI contract drift checks to keep documented workflow endpoints aligned
  with source routes

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/workflows.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/routes/workflows.spec.ts src/openapi.contract.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
```

### SDK Route Parity Validation (WS10-003)

SDK-to-backend parity for `WorkflowsResource` is validated by:

- workflow-registry route tests covering import/export, featured templates, and
  stats/star endpoints
- workflow-registry OpenAPI contract parity checks
- SDK parity test asserting every workflow SDK HTTP operation exists in deployed
  OpenAPI specs

Verification commands:

```bash
pnpm --filter @isis/workflow-registry exec vitest run src/routes/workflows.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/workflows.route-parity.spec.ts --reporter=dot
```

### Webhook Subscription Management API Validation (WS10-004)

External orchestration webhook subscription APIs are validated by:

- route-level contract + RBAC tests for webhook subscription CRUD/lifecycle
  endpoints
- generation-api OpenAPI drift checks for `/api/v1/webhook-subscriptions*`
- SDK parity test for typed webhook subscription operations in
  `GenerationResource`

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/webhook-subscriptions.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/generation.webhook-subscriptions.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
```

### Webhook Event Schema Registry Validation (WS10-005)

Webhook lifecycle payload schema registry coverage is validated by:

- route + RBAC tests for `/api/v1/webhook-subscriptions/event-schemas*`
- OpenAPI contract drift checks for webhook schema discovery routes
- SDK parity checks for webhook schema discovery operations in
  `GenerationResource`

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/webhook-subscriptions.spec.ts src/app.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/generation.webhook-subscriptions.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
```

### External Workflow Trigger API Validation (WS10-006)

External workflow trigger APIs for batch/chained orchestration are validated by:

- route-level tests for `/api/v1/workflows/triggers/batch` and
  `/api/v1/workflows/triggers/chained` idempotency contracts
- auth/RBAC coverage for trigger permissions in project-scoped API key flows
- OpenAPI route-drift and SDK route-parity checks for workflow trigger endpoints

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/workflows.spec.ts src/app.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/workflows.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
```

### Webhook Callback Security Validation (WS10-007)

Outbound webhook callback signing and replay-protected verification are
validated by:

- service-level tests for signature generation/verification and replay cache
  behavior
- route-level + RBAC tests for callback security sign/verify endpoints
- OpenAPI route-drift and SDK route-parity checks for callback security
  operations

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/services/webhook-callback-security.service.spec.ts src/routes/webhook-subscriptions.spec.ts src/app.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/generation.webhook-subscriptions.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
```

### External Integration Examples Validation (WS10-008)

Typed external integration examples for game engines and DCC automation are
validated by:

- SDK-level tests for typed example builders and identifier-based lookup
  (`unreal/unity/godot` and `blender/maya/houdini`)
- remediation backlog generation and compliance-gate validation to ensure
  traceability from checklist to evidence artifacts

Verification commands:

```bash
pnpm --filter @isis/client exec vitest run src/examples/external-integrations.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

### Workflow Pack and Preset Bundle API Validation (WS10-009)

Workflow pack and preset bundle import/export APIs are validated by:

- workflow-registry route + RBAC tests covering pack and preset-bundle
  export/import behavior
- workflow-registry OpenAPI route contract validation against route declarations
- SDK route-parity checks for `WorkflowsResource` pack/preset operations

Verification commands:

```bash
pnpm --filter @isis/workflow-registry exec vitest run src/routes/workflows.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/workflows.route-parity.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

### Cross-Service Compatibility Contract Validation (WS10-010)

Cross-service compatibility contracts are validated on every merge by:

- generation/workflow-registry/output-registry OpenAPI route-contract tests
- SDK-level cross-service compatibility spec asserting required end-to-end route
  chains for workflow packs, preset bundles, workflow execution, and output
  provenance/lineage lookup
- CI enforcement via `.github/workflows/ci.yml` in `isis-api-route-contracts`
  job

Verification commands:

```bash
pnpm --filter @isis/generation-api exec vitest run src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/workflow-registry exec vitest run src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/output-registry exec vitest run src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/isis.cross-service-contract.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

### `deploy.yml` - General Deployment

**Triggers:** Manual dispatch

General-purpose deployment orchestration for services not covered by
domain-specific pipelines.

### `container-build.yml` - Docker Image Building

**Triggers:** Called by deployment workflows

Multi-platform Docker image builds with push to ECR.

### `terraform.yml` - Infrastructure as Code

**Triggers:** Push to `main` (infra/ changes), manual dispatch

Runs Terraform plan/apply for AWS infrastructure across 20 modules:

- VPC, ALB, ECS, ECR
- RDS, ElastiCache, S3
- CloudWatch, SNS, SQS
- SSM Parameter Store

### `model-sync.yml` - AI Model Sync

**Triggers:** Manual dispatch

Synchronizes AI model artifacts between registries and deployment environments.

---

## Domain-Specific Pipelines

### Iris Domain (6 workflows)

| Workflow                         | Purpose                        |
| -------------------------------- | ------------------------------ |
| `iris-ci.yml`                    | Domain-specific CI pipeline    |
| `iris-container-build.yml`       | Container image builds         |
| `iris-security.yml`              | Security scanning              |
| `iris-staging-deploy.yml`        | Staging deployment             |
| `iris-production-deploy.yml`     | Production deployment          |
| `iris-iac-modules.yml`           | Terraform module validation    |
| `iris-infra-drift-detection.yml` | Infrastructure drift detection |

### Psyche Domain (4 workflows)

| Workflow                      | Purpose                       |
| ----------------------------- | ----------------------------- |
| `psyche-ci.yml`               | Domain-specific CI            |
| `psyche-container-build.yml`  | Container image builds        |
| `psyche-deploy.yml`           | Deployment orchestration      |
| `psyche-model-management.yml` | ML model lifecycle management |

### Tara Domain (5 workflows)

| Workflow                  | Purpose                     |
| ------------------------- | --------------------------- |
| `tara-ci.yml`             | Domain-specific CI          |
| `tara-api-deploy.yml`     | API deployment to ECS       |
| `tara-web-deploy.yml`     | Web app deployment          |
| `tara-web-preview.yml`    | Preview deployments for PRs |
| `tara-android-deploy.yml` | Android mobile release      |
| `tara-ios-deploy.yml`     | iOS mobile release          |

---

## Workflow Configuration

### Shared Settings

All workflows use:

- **Node.js:** 22 (LTS)
- **pnpm:** 10 (installed via `corepack`)
- **Nx Cloud:** Distributed task caching

### Adding a Domain Pipeline

To add CI/CD for a new domain:

1. Create `.github/workflows/{domain}-ci.yml` based on an existing domain
   pipeline (e.g., `tara-ci.yml`)
2. Update the `paths` filter to trigger on changes in `apps/{domain}/` and
   `libs/{domain}/`
3. Create deployment workflows as needed for the domain's services
4. Add domain-specific secrets to GitHub repository settings

### Running Workflows Locally

Use `act` to test workflows locally:

```bash
# Install act
brew install act  # macOS

# Run a specific workflow
act -W .github/workflows/ci.yml
```

---

## Further Reading

- [Architecture](./architecture.md) - System architecture overview
- [Testing Strategy](./testing.md) - Test infrastructure details
- [Getting Started](./getting-started.md) - Development setup