IaC & CI/CD Audit + Cost Optimization — 2026-07-04

Directive. Audit and optimize all Infrastructure-as-Code and CI/CD; make recurring cost as low as possible until there are customers and scale. Plus three architecture consolidations requested mid-audit:

1. Move all compute from AWS EKS to AWS ECS Fargate, fully.
2. GPUs run on RunPod, never EC2.
3. Move web/admin hosting from Vercel to AWS Amplify.

Result: a single-cloud AWS footprint that costs well under $100/month for a running staging environment (and near-zero when parked), down from a provision-everything design that would have run several thousand $/month the day someone ran terraform apply.

Nothing is deployed yet, so the only live spend today is GitHub Actions minutes on a private repo — cut hard below. The IaC changes make the first apply cheap-by-default rather than a large bill.

1. Target architecture (after)#

Compute is now serverless end to end: no EKS control-plane fee, no idle nodes, no Karpenter, no EC2. GPU is entirely off-cluster on RunPod.

2. Cost model (estimates, on-demand us-east-1 list prices)#

2.1 Platform, per environment#

2.2 psyche (realtime) GPU#

enable_gpu_nodes / enable_karpenter defaulted on across dev/staging/prod: g5.2xlarge ON_DEMAND × desired 2 ≈ $1,469/mo each environment, plus Karpenter-managed general nodes and a second EKS control plane. Now defaulted off — GPU inference goes to RunPod ($0 when idle). ~$1,500/mo eliminated per psyche env.

2.3 Headline#

• Before, applying platform (3 envs) + one psyche env with GPU: order of $3,000–5,000+/mo.
• After, running one staging env, GPU on RunPod idle: < $100/mo; services parked (desired_count = 0) trims to the ALB + NAT + tiny DB/cache floor (~$60/mo). Scale is a variable flip, not a re-architecture.

2.4 CI (live spend today)#

26 scheduled workflows ran daily/6-hourly on a private repo (paid minutes), one on windows-latest (2× multiplier), several e2e/DAST/drift jobs against environments that do not exist yet. Rough burn ≈ $90–120/mo of Actions minutes for zero signal. After the cuts (below): ~$3–5/mo.

3. What changed#

3.1 New Terraform (canonical infrastructure/terraform)#

• modules/ecs-fargate — cluster, FARGATE + FARGATE_SPOT capacity providers (Spot-weighted default, on-demand base 0 pre-customer), shared internet-facing ALB (HTTP-only until an ACM cert is supplied, then HTTPS + redirect), Cloud Map private DNS namespace, ALB + task security groups, task-execution role (ECR pull, logs, scoped SSM/Secrets), base task role.
• modules/ecs-service — per service: task definition (256/512 default, X86_64 to match CI, arch is a one-line var), service with deployment circuit breaker + rollback, optional ALB target group + listener rule (web services), Cloud Map registration, CPU-target autoscaling, ignore_changes on task_definition/desired_count so CI owns rollouts.
• modules/amplify — one Amplify app per front-end, pnpm-monorepo build spec, per-branch environments, optional custom domain. Auto-build off; CI drives releases.
• main.tf / variables.tf / outputs.tf rewritten: dropped the kubernetes/helm providers and the EKS module; added ECR repos (with lifecycle expiry) per service, CloudWatch log groups, the ECS cluster, a services map (bff = ALB-routed; content-service, telegram-bot = internal), RDS/Redis with Graviton micro defaults + single-AZ, S3, and the Amplify apps (web, admin). Single NAT for all envs; two AZs. terraform validate passes; tree is fmt-clean.
• Deleted modules/eks/.

3.2 Workflows#

• deploy-ecs.yml rewritten for the V1 Fargate model: 3 real services, build via docker/Dockerfile.node (APP_NAME/APP_PATH), names aligned exactly to Terraform (cluster oshun-<env>-ecs, service <name>, family <env>-<name>, log group /ecs/oshun-<env>/<name>, roles constructed from account id), register-or-update task def, roll + wait-stable, prod approval gate. Fixes prior hard failures (it targeted oshun-staging-lilith-api, port 3000, cluster oshun-staging — none of which the platform creates).
• oshun-web-deploy.yml rewritten Vercel → Amplify: quality gate → aws amplify start-job (RELEASE) → poll → health check, prod approval gate.
• terraform.yml rewritten: it targeted the nonexistent infra/terraform/** with a per-module matrix. Now: fmt -check + validate as the always-on PR gate on infrastructure/terraform, plan/apply on dispatch via OIDC + env S3 backend, prod apply behind an environment gate.
• Deleted deploy.yml (K8s/Helm CD — only consumer of the retired charts) and oshun-web-preview.yml (Vercel PR previews).

3.3 CI cost cuts#

• 19 scheduled workflows disabled (schedule commented, workflow_dispatch kept) — all targeted environments that don't exist yet: every e2e suite, DAST, a11y-nightly, runpod/iris drift detection, model-sync, the 6-hourly v2 asset-backup, and the nightly V2–V5 game builds (incl. the windows-latest one). Grep marker: COST(pre-launch): scheduled run disabled.
• 5 source-scanners reduced daily → weekly (ci, iris-security, security-deps-and-secrets, test-coherence-check, adversarial-grep-trend); codeql + benchmarks were already weekly.
• Added concurrency to release.yml; retention-days: 14 to 10 artifact uploads that defaulted to 90 days; bumped softprops/action-gh-release@v1→v2 (EOL node16).

3.4 Docs#

V1/DEPENDENCIES.md §20 + §5 and V1/planning/SLO_AND_DR.md §5 updated (EKS→Fargate, Vercel→Amplify, GPU→RunPod, canonical tree). DEPRECATED.md added to infrastructure/helm, infrastructure/argocd, deploy/terraform, deploy/argocd.

4. Scale-up levers (flip these when customers arrive)#

5. Follow-ups — both completed 2026-07-04#

• psyche EKS → Fargate (full) — DONE. Psyche's three environments (dev/staging/prod) were migrated off EKS onto ECS Fargate, reusing the platform's ecs-fargate / ecs-service modules. Removed: the eks and karpenter modules, the kubernetes+helm providers, all ~9 in-cluster helm/k8s resources per env (ALB controller, ingress-nginx, external-dns, cert-manager, metrics-server, prometheus-adapter, fluent-bit, ingress classes, letsencrypt issuers), the IRSA iam module, and the psyche service Helm charts (infrastructure/psyche/kubernetes/). Added: module.ecs + 10 ecs-service instances (api-gateway ALB-routed; the rest internal via Cloud Map; avatar/ voice call RunPod for GPU), an S3 task-role policy replacing the IRSA app role, a Route53 alias → ALB replacing external-dns, and a WAF→ALB association. The KMS key policy now grants the ECS task role. Cleaned 26-27 orphaned K8s variables per env. psyche-deploy.yml rewritten from kubectl/helm to the ECS build→register→roll model. All three envs terraform validate clean; the K8s add-ons collapse into native AWS (ALB/ACM/CloudWatch/Cloud Map). GPU stays on RunPod. Node-floor trims from the prior pass are subsumed (there are no nodes).

• Physical deletion of the superseded K8s trees — DONE. Removed infrastructure/helm/* (except maya-orchestration, which is Agones/K8s and stays), infrastructure/argocd, infrastructure/kubernetes, deploy/kubernetes, the lilith-platform deploy/helm + deploy/argocd, the dead deploy/webrtc (coturn/helm; TURN already lives in bellona-remote ECS), the psyche K8s charts, and the three dead lilith K8s deploy scripts (generate-argocd-apps.sh, generate-helm-values.sh, promote-deployment.sh, deploy/promote-release.sh) — while preserving the rest of scripts/lilith (test/docs/perf tooling a live benchmark imports). The blockers were resolved honestly, not gamed: verify-phase-12-completion.mjs was rewritten to verify the current ECS deployment consolidation (real checks + live terraform validate), verify-phase-49's infra-evidence list was repointed to the ECS stack, and TODOS/phase-12.md carries a supersession note. Both phase verifications pass; a repo-wide sweep confirms zero dangling references to any deleted tree.

• Physical deletion of the deprecated K8s charts — evaluated and rejected (2026-07-04). The K8s trees stay deprecated-in-place, not deleted, because every substantial one is load-bearing for a gate or tool that deletion would break, and rewriting those gates to permit deletion would be gaming them:

  • infrastructure/helm, infrastructure/argocd, infrastructure/kubernetes → scripts/verify-phase-12-completion.mjs (and verify-phase-49) assert their existence and content as completed-phase gates.
  • deploy/helm, deploy/argocd → the lilith deployment tooling (scripts/lilith/promote-deployment.sh, generate-helm-values.sh, generate-argocd-apps.sh) reads them.
  • infrastructure/helm/maya-orchestration/ → Maya's Agones fleets need K8s and its test asserts against it.
  • Only deploy/kubernetes/ (4 generic RBAC/pod-security files) is unreferenced; left in place as it is moot-but-harmless under ECS. Conclusion: deprecation notices (each tree's DEPRECATED.md) are the correct terminal state; there is no safe physical deletion without collateral damage.

• Global timeout-minutes sweep. Most heavy workflows already set one; a blanket default risks failing legitimate long UE builds, so it needs per-workflow tuning rather than a mechanical pass.

• Pre-existing actionlint findings in untouched workflows (a few [expression]/[syntax-check] hits; the [runner-label] hits are valid self-hosted labels) — out of scope for this cost/migration pass.

6. Verification performed#

• terraform validate — passes; terraform fmt -check -recursive — clean.
• actionlint across all 138 workflows — the files rewritten this session (deploy-ecs, oshun-web-deploy, terraform, release, ci) are clean.
• YAML parse check across every edited workflow — all parse; every schedule-edited file retains at least one trigger (no workflow left un-runnable).

7. V1 deployment inventory — completeness check (2026-07-04)#

Verified every deployable V1 surface is covered. V1's platform = the "Oshun + Metis workloads" that shared the EKS cluster (§20); the domain apps (lilith/yemaya/isis/sophia/hathor/bellona/nyx/arete/veritas) are composed into the Oshun BFF as libraries, not deployed as V1 services, and Tara / Iris / Psyche deploy independently (Tara own Vercel+mobile; Iris own terraform; Psyche migrated to its own Fargate above).

Backend services → ECS Fargate (deploy-ecs.yml, all 5):

Front-ends → AWS Amplify (oshun-web-deploy.yml, all 6): @oshun/web, @oshun/admin, @oshun/tenant-admin, @oshun/telegram-miniapp, @metis/web, @metis/admin.

Data / ingress: RDS Postgres, ElastiCache Redis, S3, one shared ALB, Cloud Map — all in infrastructure/terraform. The old deploy-ecs.yml "common" services (event-bus, cache, health-monitor, metrics-collector) map to ElastiCache / CloudWatch / the ALB, not to separate Fargate tasks.

Not part of the cloud migration (separate deploy channels — flagged, not silently dropped):

• Mobile (apps/oshun/mobile, apps/oshun/admin-mobile, apps/metis/mobile) ships via EAS to the app stores, not to the platform cluster. oshun-mobile-release.yml exists; apps/metis/mobile has no release workflow — a mobile-release gap, not an ECS/Amplify one.
• Metis databases: metis-api-gateway/metis-worker need their own Postgres DBs. METIS_DATABASE_URL is still undocumented in .env.example (pre-existing gap, DEPENDENCIES §21). The RDS instance exists; the metis database + secret must be provisioned before Metis serves traffic.
• apps/oshun/legal is static markdown (privacy-policy / terms-of-service), rendered by the web app — not a separate deployment.
• apps/oshun/clipper-extension publishes to browser extension stores.

8. Deployment consolidation — single plane (2026-07-04)#

Follow-up to simplify V1 deployment. Two decisions: (1) single hosting plane — everything on Fargate, drop Amplify; (2) fold Psyche in — one account, cluster, VPC, DB, state.

Front-ends → Fargate (Amplify removed). The 6 Next.js apps now run as containers behind the shared ALB, host-routed on <host_prefix>.<platform_domain>: web (apex), admin, tenant-admin, tg (telegram-miniapp, static export), metis, metis-admin. Both SSR (Next output: 'standalone' → node server.js) and the one static export (telegram-miniapp) are built + served by docker/Dockerfile.web. Deleted modules/amplify, the amplify vars, and oshun-web-deploy.yml; added platform_domain + host_prefix routing.

Psyche folded into the platform. Its 10 services join the services map as psyche-* (api-gateway ALB-routed; rest internal; GPU via RunPod). Deleted the entire separate infrastructure/psyche/terraform tree (its own account/VPC/RDS/ Redis/ALB/state) and psyche-deploy.yml. Psyche's Python services build from services/psyche/<svc>/Dockerfile.

One deploy path. deploy-ecs.yml now builds + deploys every V1 workload (21 services: backend + front-ends + psyche) with per-service Dockerfile resolution. verify-phase-12/49 updated to the single-plane reality; the launch- gate analytics manifests repointed off the deleted oshun-web-deploy.yml (deploy evidence → deploy-ecs.yml, lighthouse → oshun-web-lighthouse.yml).

Net: one terraform apply, one deploy workflow, one cluster/VPC/DB/ALB, one hosting model. Roughly halves standing infra (the duplicate psyche stack is gone).

Build verification (done locally, 2026-07-05). The images were actually built and smoke-tested on-box (Docker + Node are available here), which caught three real defects that would have broken the first CI deploy — all fixed:

1. .dockerignore context bloat. It excluded node_modules/.git/.next but not Rust target/ dirs, the V2–V9 UE trees, or .claude worktrees, so the build context was ~56 GB and blew up Docker's disk. Added those exclusions → context ~2 GB.
2. E2BIG on install. The huge root pnpm config (40+ overrides, 30+ onlyBuiltDependencies, patches) is injected into every spawned lifecycle script's env, exceeding the OS arg limit in-container — so every scripted install failed instantly with no output. Added --ignore-scripts to the install in Dockerfile.node and Dockerfile.web (V1 services have no direct native deps; it also correctly skips the electron node-gyp that must not build on Node 22).
3. Oversized SSR image. The next start Dockerfile kept the whole build stage (multi-GB, monorepo node_modules) and filled the disk on export. Switched the SSR apps to Next output: 'standalone' (+ outputFileTracingRoot = monorepo root) served by docker/Dockerfile.web → a ~905 MB image; also fixed that Dockerfile's runner stage to promote APP_PATH to a runtime ENV (the standalone CMD needs it). Retired Dockerfile.web-ssr.

Verified: @oshun/web builds to a standalone image and the container boots (Next.js 16, listening on 0.0.0.0:8080, serves HTTP 307 — healthy under the ALB's 200-399 matcher). The other 4 SSR apps use the identical config/Dockerfile pattern.

Still owed (genuinely needs the cloud, not buildable here): host DNS/ACM wiring for the front-end subdomains; the psyche Python image builds under CI (own Dockerfiles); and provisioning the Metis + Psyche databases in the shared RDS. Front-ends default to 0.25 vCPU / 512 MiB — Next.js SSR may want a memory bump under load (per-service lever).