Eve library

ISIS Provenance Backfill Design (MIG-003)

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: MIG-003
Ticket: ISIS-0172

Goal#

Backfill missing provenance records for historical completed outputs so every eligible output has a valid outputs.provenance_id reference and a corresponding row in provenance.

Scope#

Eligible outputs are records in outputs with status in:

  • available
  • archived
  • expired

A record is considered missing provenance when either:

  • outputs.provenance_id is null, or
  • outputs.provenance_id points to a non-existent provenance row.

Backfill Strategy#

For each eligible output, the backfill pipeline performs:

  1. Existing provenance reconciliation:
    • If a provenance row already exists by provenance.output_id, relink outputs.provenance_id to that row.
    • If multiple provenance rows exist for one output, mark unresolved for manual review.
  2. Provenance reconstruction when no row exists:
    • Build RegisterProvenanceRequest from output metadata and canonical fields.
    • Insert provenance and update outputs.provenance_id transactionally.

Source Resolution Rules#

Reconstruction uses deterministic source resolution:

  1. metadata.provenance (highest fidelity)
  2. Output-level workflow/model fields and metadata aliases
  3. Parameter maps (metadata.parameters, inputParameters, generationParameters)
  4. Input reconstruction from metadata.inputs, metadata.inputUrl, and output.parentIds
  5. Fallback defaults from output record (createdAt, jobId, workflowId)

Operational Interfaces#

API (admin)#

  • POST /api/v1/outputs/admin/provenance/backfill

Request body:

json
{
  "limit": 2000,
  "dryRun": false
}

Access requirements:

  • Bearer auth role: admin, ops, or sre
  • API key permission: outputs:admin

CLI#

  • isis outputs backfill-provenance --dry-run --limit 5000

Result Contract#

Backfill response includes:

  • scanned
  • missingProvenance
  • backfilled
  • unresolved
  • failed
  • backfilledOutputIds
  • unresolvedOutputIds
  • failedOutputIds
  • sourceCounts
  • checkedAt

Safety Controls#

  • Dry-run mode for impact estimation without writes.
  • Bounded scan with hard max limit (50000) and page batching.
  • Ambiguous provenance conflicts are not auto-resolved; flagged as unresolved.
  • Audit actor attribution is preserved through actorId from authenticated principal.