Last Updated: 2026-02-24
Scope: Staging rehearsal pipeline for WS17 migration/backfill operations using
synthetic high-volume datasets.
Purpose#
Provide a deterministic pre-release dry-run pipeline for WS17 migration tasks:
- WS17-002 workflow taxonomy/provenance backfill
- WS17-003 historical job taxonomy backfill
- WS17-004 output provenance/consistency backfill
The pipeline seeds synthetic data at high volume, executes dry-run migration paths, verifies no-regression snapshot invariants, and emits a machine-readable report for release evidence.
Scripts#
scripts/isis/migrate_workflow_taxonomy.shscripts/isis/migrate_historical_job_taxonomy.shscripts/isis/migrate_historical_output_provenance.shscripts/isis/run_ws17_migration_dry_run_pipeline.sh
Prerequisites#
psql,curl,jq,bashinstalled.ISIS_DATABASE_URL(orDATABASE_URL) set to staging database.ISIS_API_KEYset for admin backfill endpoints.- Service endpoints reachable:
ISIS_WORKFLOW_REGISTRY_URL(defaulthttp://localhost:3001)ISIS_GENERATION_API_URL(defaulthttp://localhost:3000)ISIS_OUTPUT_REGISTRY_URL(defaulthttp://localhost:3002)
Standard Execution#
bash
STRICT=true \
ENABLE_SYNTHETIC_DATASET=true \
CLEANUP_SYNTHETIC_DATASET=true \
ISIS_DATABASE_URL="postgres://..." \
ISIS_API_KEY="..." \
ISIS_WORKFLOW_REGISTRY_URL="https://workflow-registry-staging.example.com" \
ISIS_GENERATION_API_URL="https://generation-api-staging.example.com" \
ISIS_OUTPUT_REGISTRY_URL="https://output-registry-staging.example.com" \
scripts/isis/run_ws17_migration_dry_run_pipeline.sh
High-Volume Controls#
Tune synthetic dataset scale:
bash
SYNTHETIC_WORKFLOW_COUNT=5000 \
SYNTHETIC_JOB_COUNT=25000 \
SYNTHETIC_OUTPUT_COUNT=25000 \
scripts/isis/run_ws17_migration_dry_run_pipeline.sh
Key Environment Variables#
| Variable | Default | Description |
|---|---|---|
STRICT |
true |
Exit non-zero when any pipeline check fails. |
RUN_ID |
auto-generated | Stable identifier used in seeded record IDs and report artifacts. |
ENABLE_SYNTHETIC_DATASET |
true |
Enables synthetic high-volume dataset seeding. |
CLEANUP_SYNTHETIC_DATASET |
true |
Removes synthetic records at end of pipeline. |
SYNTHETIC_WORKFLOW_COUNT |
3000 |
Number of synthetic stale workflows/workflow versions to seed. |
SYNTHETIC_JOB_COUNT |
9000 |
Number of synthetic historical jobs to seed. |
SYNTHETIC_OUTPUT_COUNT |
9000 |
Number of synthetic outputs without provenance to seed. |
MIGRATION_LIMIT_OVERRIDE |
auto-calculated | Forces explicit migration dry-run limit for all migration calls. |
MIGRATION_LIMIT_BUFFER |
200 |
Added to auto-calculated limit to absorb baseline growth race. |
MIN_WORKFLOW_STALE_DETECTIONS |
1 |
Minimum stale workflow records expected in workflow dry-run report. |
MIN_JOB_BACKFILLED_DETECTIONS |
1 |
Minimum backfillable jobs expected in historical-job dry-run report. |
MIN_OUTPUT_BACKFILLED_DETECTIONS |
1 |
Minimum backfillable outputs expected in provenance dry-run report. |
Output Artifacts#
Default output directory: tmp/isis-migration/
Primary artifacts:
ws17-migration-dry-run-pipeline-*.jsonws17-dry-run-before-<run-id>.jsonws17-dry-run-after-<run-id>.jsonws17-dry-run-before-after-<run-id>.jsonws17-dry-run-workflow-taxonomy-<run-id>.jsonws17-dry-run-historical-job-taxonomy-<run-id>.jsonws17-dry-run-output-provenance-<run-id>.jsonws17-dry-run-synthetic-dataset-<run-id>.jsonws17-dry-run-synthetic-cleanup-<run-id>.json
Failure Handling#
If the pipeline fails:
- Inspect
summary.failand individualchecks[]in the pipeline report. - Review per-step logs in
tmp/isis-migration/for failing migration calls. - If cleanup was skipped or failed, run the pipeline again with the same
RUN_IDandCLEANUP_SYNTHETIC_DATASET=true. - Re-run the full pipeline until
overallPass=true.