CI/CD Pipelines

Overview#

The Oshun monorepo uses GitHub Actions for continuous integration and deployment. There are 31 workflow files organized into core pipelines and domain-specific pipelines.

Core Pipelines#

ci.yml - Main CI#

Triggers: Push to main/develop, pull requests, nightly schedule

The primary CI pipeline validates all affected projects:

1. Setup & Analysis - Detects affected projects using Nx
2. Dependency Install - pnpm install with caching
3. Lint - ESLint across affected projects
4. Type Check - TypeScript compilation checks
5. Build - Compiles affected projects
6. Unit Tests - Vitest across affected projects

Environment: Node.js 22, pnpm 10

e2e.yml - End-to-End Tests#

Triggers: Push to main, daily schedule

Runs Playwright E2E tests sharded across 3 parallel jobs. Starts PostgreSQL and Redis as service containers for realistic integration testing.

pr-quality.yml - Pull Request Quality Gates#

Triggers: Pull requests

Enforces code quality on PRs:

• Coverage thresholds (80% minimum)
• Lint compliance
• Performance regression checks

database.yml - Database Migrations#

Triggers: PR (validation), push to staging/production branches

Validates Prisma migrations on PR. Runs actual migrations against staging and production databases on merge. Supports per-domain migration:

• Domains: yemaya, lilith, isis, sophia, hathor, bellona

release.yml - Release Management#

Triggers: Manual dispatch

Handles version bumping, changelog generation, and GitHub release creation.

codeql.yml - Security Scanning#

Triggers: Push to main, weekly schedule

GitHub CodeQL static analysis for JavaScript/TypeScript security vulnerabilities.

accessibility.yml - Accessibility Audits#

Triggers: Push to main, manual dispatch

WCAG compliance testing using axe-core against web applications.

benchmarks.yml - Performance Benchmarks#

Triggers: Weekly schedule, manual dispatch

Runs k6 load tests in smoke, load, and stress modes against API endpoints.

Deployment Pipelines#

deploy-ecs.yml - AWS ECS Deployment#

Triggers: Manual dispatch, called by domain pipelines

Orchestrates deployment to AWS ECS:

• Staging: Cluster oshun-staging
• Production: Cluster oshun-production
• Registry: ${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com

Supports per-domain and per-service deployment with health check verification.

deploy-runpod.yml - RunPod GPU Deployment#

Triggers: Manual dispatch

Deploys GPU workloads (generation workers, inference services) to RunPod infrastructure.

runpod-endpoint-image-build.yml - RunPod Endpoint Image Build#

Triggers: Push to main, pull request, manual dispatch

Builds RunPod endpoint images from manifest-driven build definitions:

• Generates endpoint image manifests from infra/runpod/endpoints/image-build-config.json + infra/runpod/endpoints/desired-state.json
• Enforces warm/cold policy alignment from infra/runpod/endpoints/warm-cold-policy.json
• Generates per-endpoint model/custom-node inventory manifests under infra/runpod/endpoints/inventory
• Validates manifest parity and desired-state coverage before building
• Builds base + endpoint image matrix using scripts/isis/generate_runpod_endpoint_image_build_matrix.mjs

runpod-endpoint-lifecycle.yml - RunPod Endpoint Lifecycle#

Triggers: Manual dispatch

Runs endpoint lifecycle automation (create/update/delete/batch) using scripts/isis/runpod_endpoint_lifecycle.py against RunPod GraphQL APIs. Desired endpoint configuration is sourced from infra/runpod/endpoints/desired-state.json and validated by scripts/isis/validate_runpod_desired_state.py (including canary endpoint IDs and canary traffic percentage bounds for staged rollout). The workflow also enforces warm/cold policy alignment via scripts/isis/apply_runpod_warm_cold_policy.mjs --check.

runpod-endpoint-drift-detection.yml - RunPod Drift Detection#

Triggers: Scheduled, manual dispatch

Compares live RunPod endpoint configuration against infra/runpod/endpoints/desired-state.json using scripts/isis/detect_runpod_endpoint_drift.py after verifying warm/cold policy alignment.

runpod-endpoint-reconcile.yml - RunPod Drift Reconcile#

Triggers: Manual dispatch

Runs scripts/isis/reconcile_runpod_endpoint_drift.sh to detect drift, apply desired state, and verify post-reconcile convergence, with warm/cold policy alignment checks as a preflight gate.

RunPod FinOps Attribution Validation (WS09-005)#

RunPod pay-per-second attribution behavior is validated in automated tests covering:

• adapter runtime accounting and dimensional aggregation (workflow/project/tenant)
• API metadata propagation for tenant/project context
• web operations surfaces rendering attribution snapshots

Verification commands:

pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/services/comfyui.service.spec.ts src/routes/jobs.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot

RunPod Endpoint SLO Telemetry Validation (WS09-006)#

RunPod endpoint-level SLO metrics are validated with focused tests that cover:

• adapter SLO sampling and p95/rate calculations
• API route payloads exposing endpoint SLO snapshots enriched with endpoint topology metadata
• web operations UI rendering objective thresholds, endpoint metrics, and breach flags

Verification commands:

pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot

RunPod Anomaly Detection Validation (WS09-007)#

RunPod anomaly telemetry behavior is validated with tests covering:

• adapter-level detection for cost spikes, timeout storms, and model-cache thrashing
• API route payload exposure with endpoint topology enrichment
• web operations rendering for anomaly objectives and active alerts

Verification commands:

pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot

RunPod Endpoint Canary Strategy Validation (WS09-008)#

RunPod endpoint-family canary strategy and rollback triggers are validated by:

• route tests asserting canary action derivation from SLO/anomaly telemetry
• web tests rendering strategy action/reason surfaces for operators
• execution-script smoke checks for callback invocation and rollback exit semantics

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
FAIL_ON_ROLLBACK=false OPERATIONS_FILE=/tmp/runpod-operations.json scripts/isis/execute_runpod_endpoint_canary_strategy.sh

RunPod Fleet Topology and Economics Validation (WS09-009)#

RunPod fleet topology/economics coverage is validated with tests covering:

• adapter endpoint-level cost attribution dimensions (including endpoint keys)
• API route exposure of fleet topology + projected/observed economics
• web operations rendering of fleet source/topology/economics tables

Verification commands:

pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot

RunPod DR Runbooks and Game-Day Validation (WS09-010)#

RunPod disaster-recovery readiness is validated by:

• provider outage drill automation runbook coverage
• endpoint-region failover drill automation runbook coverage
• composite game-day harness that executes both drills and emits a unified report

Verification commands:

bash -n scripts/isis/run_provider_outage_drill.sh scripts/isis/run_endpoint_region_failover_drill.sh scripts/isis/run_runpod_disaster_recovery_gameday.sh
node --test scripts/operations/v1-game-day.test.mjs
# A smoke run of the harness. It injects nothing and executes no scenario, so it
# reports `overall.outcome: "inconclusive"` -- which is the point: until S10.12
# it reported `overall.pass: true` and logged "all game-day checks passed" for
# exactly this invocation.
STRICT=false DRY_RUN=true RUN_PROVIDER_FAILURE_SCENARIO=false RUN_REGION_FAILURE_SCENARIO=false scripts/isis/run_runpod_disaster_recovery_gameday.sh

A game day that establishes something needs the fault actually applied and the instants an operator observed. FAULT_INJECTED_AT_EPOCH and DETECTED_AT_EPOCH are what turn the report's recovery time into an interval an actor lived through; without DETECTED_AT_EPOCH the run stays inconclusive, because a recovery time measured from the moment somebody ran the script excludes detection and decision.

STRICT=true DRY_RUN=false \
  RUN_PROVIDER_FAILURE_SCENARIO=true RUN_REGION_FAILURE_SCENARIO=true \
  FAULT_INJECTED_AT_EPOCH=$(date -u +%s) \
  DETECTED_AT_EPOCH=<when the alert fired> DECLARED_AT_EPOCH=<when it was declared> \
  ISIS_API_BASE_URL=... ISIS_ADMIN_TOKEN=... \
  scripts/isis/run_runpod_disaster_recovery_gameday.sh

Isis OpenAPI Canonical Generation Validation (WS10-001)#

Isis generation-api OpenAPI publication is validated by:

• deterministic route-derived OpenAPI generation
• canonical artifact parity (apps/isis/generation-api/openapi.yaml and openapi/isis/openapi.yaml)
• contract test enforcement for route/spec alignment

Verification commands:

node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check
pnpm --filter @isis/generation-api exec vitest run src/openapi.contract.spec.ts --reporter=dot

Workflow Execution Endpoints + RBAC Validation (WS10-002)#

Workflow execution parity and RBAC enforcement are validated by:

• route-level tests for /api/v1/workflows/:workflowId/run and /api/v1/workflows/:workflowId/executions*
• OpenAPI contract drift checks to keep documented workflow endpoints aligned with source routes

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/routes/workflows.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/routes/workflows.spec.ts src/openapi.contract.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check

SDK Route Parity Validation (WS10-003)#

SDK-to-backend parity for WorkflowsResource is validated by:

• workflow-registry route tests covering import/export, featured templates, and stats/star endpoints
• workflow-registry OpenAPI contract parity checks
• SDK parity test asserting every workflow SDK HTTP operation exists in deployed OpenAPI specs

Verification commands:

pnpm --filter @isis/workflow-registry exec vitest run src/routes/workflows.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/workflows.route-parity.spec.ts --reporter=dot

Webhook Subscription Management API Validation (WS10-004)#

External orchestration webhook subscription APIs are validated by:

• route-level contract + RBAC tests for webhook subscription CRUD/lifecycle endpoints
• generation-api OpenAPI drift checks for /api/v1/webhook-subscriptions*
• SDK parity test for typed webhook subscription operations in GenerationResource

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/routes/webhook-subscriptions.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/generation.webhook-subscriptions.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check

Webhook Event Schema Registry Validation (WS10-005)#

Webhook lifecycle payload schema registry coverage is validated by:

• route + RBAC tests for /api/v1/webhook-subscriptions/event-schemas*
• OpenAPI contract drift checks for webhook schema discovery routes
• SDK parity checks for webhook schema discovery operations in GenerationResource

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/routes/webhook-subscriptions.spec.ts src/app.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/generation.webhook-subscriptions.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check

External Workflow Trigger API Validation (WS10-006)#

External workflow trigger APIs for batch/chained orchestration are validated by:

• route-level tests for /api/v1/workflows/triggers/batch and /api/v1/workflows/triggers/chained idempotency contracts
• auth/RBAC coverage for trigger permissions in project-scoped API key flows
• OpenAPI route-drift and SDK route-parity checks for workflow trigger endpoints

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/routes/workflows.spec.ts src/app.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/workflows.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check

Webhook Callback Security Validation (WS10-007)#

Outbound webhook callback signing and replay-protected verification are validated by:

• service-level tests for signature generation/verification and replay cache behavior
• route-level + RBAC tests for callback security sign/verify endpoints
• OpenAPI route-drift and SDK route-parity checks for callback security operations

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/services/webhook-callback-security.service.spec.ts src/routes/webhook-subscriptions.spec.ts src/app.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/generation.webhook-subscriptions.route-parity.spec.ts --reporter=dot
node scripts/isis/generate_generation_api_openapi.mjs --check
pnpm --filter @isis/generation-api run openapi:check

External Integration Examples Validation (WS10-008)#

Typed external integration examples for game engines and DCC automation are validated by:

• SDK-level tests for typed example builders and identifier-based lookup (unreal/unity/godot and blender/maya/houdini)
• remediation backlog generation and compliance-gate validation to ensure traceability from checklist to evidence artifacts

Verification commands:

pnpm --filter @isis/client exec vitest run src/examples/external-integrations.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Workflow Pack and Preset Bundle API Validation (WS10-009)#

Workflow pack and preset bundle import/export APIs are validated by:

• workflow-registry route + RBAC tests covering pack and preset-bundle export/import behavior
• workflow-registry OpenAPI route contract validation against route declarations
• SDK route-parity checks for WorkflowsResource pack/preset operations

Verification commands:

pnpm --filter @isis/workflow-registry exec vitest run src/routes/workflows.auth.spec.ts src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/workflows.route-parity.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Cross-Service Compatibility Contract Validation (WS10-010)#

Cross-service compatibility contracts are validated on every merge by:

• generation/workflow-registry/output-registry OpenAPI route-contract tests
• SDK-level cross-service compatibility spec asserting required end-to-end route chains for workflow packs, preset bundles, workflow execution, and output provenance/lineage lookup
• CI enforcement via .github/workflows/ci.yml in isis-api-route-contracts job

Verification commands:

pnpm --filter @isis/generation-api exec vitest run src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/workflow-registry exec vitest run src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/output-registry exec vitest run src/openapi.contract.spec.ts --reporter=dot
pnpm --filter @isis/client exec vitest run src/resources/isis.cross-service-contract.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

deploy.yml - General Deployment#

Triggers: Manual dispatch

General-purpose deployment orchestration for services not covered by domain-specific pipelines.

container-build.yml - Docker Image Building#

Triggers: Called by deployment workflows

Multi-platform Docker image builds with push to ECR.

terraform.yml - Infrastructure as Code#

Triggers: Push to main (infra/ changes), manual dispatch

Runs Terraform plan/apply for AWS infrastructure across 20 modules:

• VPC, ALB, ECS, ECR
• RDS, ElastiCache, S3
• CloudWatch, SNS, SQS
• SSM Parameter Store

model-sync.yml - AI Model Sync#

Triggers: Manual dispatch

Synchronizes AI model artifacts between registries and deployment environments.

Domain-Specific Pipelines#

Iris Domain (6 workflows)#

Psyche Domain (4 workflows)#

Tara Domain (5 workflows)#

Workflow Configuration#

Shared Settings#

All workflows use:

• Node.js: 22 (LTS)
• pnpm: 10 (installed via corepack)
• Nx Cloud: Distributed task caching

Adding a Domain Pipeline#

To add CI/CD for a new domain:

1. Create .github/workflows/{domain}-ci.yml based on an existing domain pipeline (e.g., tara-ci.yml)
2. Update the paths filter to trigger on changes in apps/{domain}/ and libs/{domain}/
3. Create deployment workflows as needed for the domain's services
4. Add domain-specific secrets to GitHub repository settings

Running Workflows Locally#

Use act to test workflows locally:

# Install act
brew install act  # macOS

# Run a specific workflow
act -W .github/workflows/ci.yml

Further Reading#

• Architecture - System architecture overview
• Testing Strategy - Test infrastructure details
• Getting Started - Development setup