Eve library

Isis WS17 Migration Dry-Run Pipeline

Runbook

Revision 1

On this page

Last Updated: 2026-02-24
Scope: Staging rehearsal pipeline for WS17 migration/backfill operations using synthetic high-volume datasets.

Purpose#

Provide a deterministic pre-release dry-run pipeline for WS17 migration tasks:

  1. WS17-002 workflow taxonomy/provenance backfill
  2. WS17-003 historical job taxonomy backfill
  3. WS17-004 output provenance/consistency backfill

The pipeline seeds synthetic data at high volume, executes dry-run migration paths, verifies no-regression snapshot invariants, and emits a machine-readable report for release evidence.

Scripts#

  • scripts/isis/migrate_workflow_taxonomy.sh
  • scripts/isis/migrate_historical_job_taxonomy.sh
  • scripts/isis/migrate_historical_output_provenance.sh
  • scripts/isis/run_ws17_migration_dry_run_pipeline.sh

Prerequisites#

  • psql, curl, jq, bash installed.
  • ISIS_DATABASE_URL (or DATABASE_URL) set to staging database.
  • ISIS_API_KEY set for admin backfill endpoints.
  • Service endpoints reachable:
    • ISIS_WORKFLOW_REGISTRY_URL (default http://localhost:3001)
    • ISIS_GENERATION_API_URL (default http://localhost:3000)
    • ISIS_OUTPUT_REGISTRY_URL (default http://localhost:3002)

Standard Execution#

bash
STRICT=true \
ENABLE_SYNTHETIC_DATASET=true \
CLEANUP_SYNTHETIC_DATASET=true \
ISIS_DATABASE_URL="postgres://..." \
ISIS_API_KEY="..." \
ISIS_WORKFLOW_REGISTRY_URL="https://workflow-registry-staging.example.com" \
ISIS_GENERATION_API_URL="https://generation-api-staging.example.com" \
ISIS_OUTPUT_REGISTRY_URL="https://output-registry-staging.example.com" \
scripts/isis/run_ws17_migration_dry_run_pipeline.sh

High-Volume Controls#

Tune synthetic dataset scale:

bash
SYNTHETIC_WORKFLOW_COUNT=5000 \
SYNTHETIC_JOB_COUNT=25000 \
SYNTHETIC_OUTPUT_COUNT=25000 \
scripts/isis/run_ws17_migration_dry_run_pipeline.sh

Key Environment Variables#

Variable Default Description
STRICT true Exit non-zero when any pipeline check fails.
RUN_ID auto-generated Stable identifier used in seeded record IDs and report artifacts.
ENABLE_SYNTHETIC_DATASET true Enables synthetic high-volume dataset seeding.
CLEANUP_SYNTHETIC_DATASET true Removes synthetic records at end of pipeline.
SYNTHETIC_WORKFLOW_COUNT 3000 Number of synthetic stale workflows/workflow versions to seed.
SYNTHETIC_JOB_COUNT 9000 Number of synthetic historical jobs to seed.
SYNTHETIC_OUTPUT_COUNT 9000 Number of synthetic outputs without provenance to seed.
MIGRATION_LIMIT_OVERRIDE auto-calculated Forces explicit migration dry-run limit for all migration calls.
MIGRATION_LIMIT_BUFFER 200 Added to auto-calculated limit to absorb baseline growth race.
MIN_WORKFLOW_STALE_DETECTIONS 1 Minimum stale workflow records expected in workflow dry-run report.
MIN_JOB_BACKFILLED_DETECTIONS 1 Minimum backfillable jobs expected in historical-job dry-run report.
MIN_OUTPUT_BACKFILLED_DETECTIONS 1 Minimum backfillable outputs expected in provenance dry-run report.

Output Artifacts#

Default output directory: tmp/isis-migration/

Primary artifacts:

  • ws17-migration-dry-run-pipeline-*.json
  • ws17-dry-run-before-<run-id>.json
  • ws17-dry-run-after-<run-id>.json
  • ws17-dry-run-before-after-<run-id>.json
  • ws17-dry-run-workflow-taxonomy-<run-id>.json
  • ws17-dry-run-historical-job-taxonomy-<run-id>.json
  • ws17-dry-run-output-provenance-<run-id>.json
  • ws17-dry-run-synthetic-dataset-<run-id>.json
  • ws17-dry-run-synthetic-cleanup-<run-id>.json

Failure Handling#

If the pipeline fails:

  1. Inspect summary.fail and individual checks[] in the pipeline report.
  2. Review per-step logs in tmp/isis-migration/ for failing migration calls.
  3. If cleanup was skipped or failed, run the pipeline again with the same RUN_ID and CLEANUP_SYNTHETIC_DATASET=true.
  4. Re-run the full pipeline until overallPass=true.