Eve library

ISIS Cost Backfill Design (MIG-004)

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: MIG-004
Ticket: ISIS-0173

Goal#

Backfill missing historical cost metadata for terminal jobs using provider log reconciliation plus deterministic runtime estimation fallbacks.

Scope#

Eligible records are terminal jobs in jobs:

  • completed
  • failed
  • cancelled

A record is considered missing cost data when no non-negative canonical cost is present in existing job/output metadata aliases (for example metadata.estimatedCostUsd or output.metadata.estimatedCostUsd).

Backfill Strategy#

For each eligible job, the backfill pipeline performs:

  1. Provider log reconciliation:
    • Match by jobId first, then queueJobId.
    • Prefer explicit estimatedCostUsd from provider logs when available.
    • Otherwise derive cost from log executionTimeMs and provider pricing configuration.
  2. Metadata reconciliation:
    • Accept existing cost aliases when present.
  3. Runtime estimation fallback:
    • Use runtime metadata fields (executionTimeMs, worker metrics duration, terminal timestamps) plus configured provider rates.

When a value is resolved, backfill writes:

  • metadata.estimatedCostUsd
  • metadata.costBackfill audit block (source, backfilledAt, optional provider/runtime/endpoint details)
  • output.metadata.estimatedCostUsd (and missing runtime/provider fields when available)

Source Resolution Rules#

Deterministic precedence:

  1. provider_logs.estimated_cost_usd
  2. provider_logs.execution_time_ms
  3. Explicit metadata aliases (output.metadata.*, job.metadata.*)
  4. Runtime-driven estimation from metadata and timestamps

Operational Interfaces#

API (admin)#

  • POST /api/v1/jobs/admin/costs/backfill

Request body:

json
{
  "limit": 2000,
  "dryRun": false,
  "providerLogs": []
}

Access requirements:

  • Bearer auth role: admin, ops, or sre
  • API key permission: jobs:queue:admin

CLI#

  • isis jobs backfill-costs --dry-run --provider-log-file ./runpod-logs.json

Result Contract#

Backfill response includes:

  • scanned
  • missingCostData
  • backfilled
  • unresolved
  • failed
  • providerLogsProcessed
  • providerLogsMatched
  • backfilledJobIds
  • unresolvedJobIds
  • failedJobIds
  • sourceCounts
  • providerCounts
  • checkedAt

Safety Controls#

  • Dry-run mode for impact estimation without writes.
  • Bounded scan with hard max limit (50000) and page batching.
  • Provider logs are optional and validated for identifier + cost/runtime signal.
  • Ambiguous or insufficient data remains unresolved for operator review.