Eve library

ADR-0004: RunPod Endpoint Provisioning Mechanism

ADR

Revision 1

On this page

Status: Accepted
Date: 2026-02-22
Authors: Isis DevOps, Isis Runtime Engineering
Reviewers: @oshun/devops, @oshun/platform, @oshun/ml-team, @oshun/core-team

Context and Problem Statement#

Isis requires desired-state lifecycle control for RunPod serverless endpoints. Current state shows:

  • endpoint credentials/config are managed in Terraform secrets (infra/terraform/secrets/runpod.tf)
  • image deployment and endpoint mutation are already performed through a GitHub Actions workflow (.github/workflows/deploy-runpod.yml)
  • there are no RunPod endpoint creation/update/delete resources in Terraform under infra/terraform

The remediation backlog requires a single provisioning mechanism decision before implementing lifecycle automation (F08-001).

Decision#

Adopt an API-driven CI control plane as the canonical RunPod endpoint provisioning mechanism.

Provisioning, updates, and reconciliation will run via repository-owned CI jobs calling the RunPod GraphQL/API surface, with endpoint desired state committed in-code and validated in CI.

Decision Drivers#

  • Existing operational path already mutates endpoints through CI (deploy-runpod.yml), so this is the lowest-risk production-hardening path.
  • Terraform in this repo currently governs AWS primitives and secret distribution, not RunPod endpoint lifecycle resources.
  • A CI-native approach supports staged rollouts, drift detection, and reconcile loops with explicit release gating.
  • It avoids introducing a second control plane while the RunPod provider surface and organizational workflows are still evolving.

Considered Options#

Option A: Terraform-native RunPod provider/module (Rejected for now)#

Pros:

  • Familiar desired-state workflow.
  • Existing Terraform plan/apply governance.

Cons:

  • No established RunPod endpoint resource usage in this codebase.
  • Would introduce a parallel control plane and migration complexity.

Option B: API-driven CI jobs with declarative endpoint manifests (Chosen)#

Pros:

  • Extends existing, proven deployment workflow.
  • Keeps RunPod operations in one automation surface.
  • Enables progressive rollout and reconcile logic without blocking on provider maturity.

Cons:

  • Requires disciplined script/runtime quality and strong CI safety checks.
  • Drift handling must be explicitly implemented (not inherited from Terraform state).

Option C: External control plane service/operator (Deferred)#

Pros:

  • Strong separation and potentially richer runtime orchestration.

Cons:

  • Highest implementation and operational overhead for current scope.
  • Unnecessary before F08 lifecycle/drift controls are fully delivered.

Consequences#

Positive#

  • Unifies RunPod lifecycle operations under a single, auditable automation path.
  • Preserves Terraform ownership of secrets/credential distribution.
  • Provides clear sequencing for F08-002 to F08-008.

Tradeoffs#

  • CI reliability and script correctness become critical path for endpoint lifecycle.
  • Additional validation and drift checks must be authored explicitly.

Implementation Rules#

  • Terraform remains source-of-truth for RunPod credentials and endpoint secret publication.
  • CI automation becomes source-of-truth executor for endpoint create/update/delete/reconcile.
  • Endpoint desired state must be committed in-repo and validated pre-deploy.
  • Any manual dashboard mutation is treated as drift and must be reconciled by automation.

Rollout Sequence#

  1. Implement endpoint lifecycle automation in CI (F08-002).
  2. Add in-repo desired-state definitions (F08-003).
  3. Add drift detection + reconcile commands (F08-004, F08-005).
  4. Add CI validation and staged rollout controls (F08-006, F08-007).
  5. Publish outage/cost runbook (F08-008).

Ticket Traceability#

  • F08-001
  • F08-002
  • F08-003
  • F08-004
  • F08-005
  • F08-006
  • F08-007
  • F08-008

References#

  • docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md
  • docs/domains/isis/adr/ADR-0003-runpod-serverless-execution-strategy.md
  • .github/workflows/deploy-runpod.yml
  • infra/terraform/secrets/runpod.tf