# Isis WS17 Migration Dry-Run Pipeline

Last Updated: 2026-02-24  
Scope: Staging rehearsal pipeline for WS17 migration/backfill operations using
synthetic high-volume datasets.

## Purpose

Provide a deterministic pre-release dry-run pipeline for WS17 migration tasks:

1. WS17-002 workflow taxonomy/provenance backfill
2. WS17-003 historical job taxonomy backfill
3. WS17-004 output provenance/consistency backfill

The pipeline seeds synthetic data at high volume, executes dry-run migration
paths, verifies no-regression snapshot invariants, and emits a machine-readable
report for release evidence.

## Scripts

- `scripts/isis/migrate_workflow_taxonomy.sh`
- `scripts/isis/migrate_historical_job_taxonomy.sh`
- `scripts/isis/migrate_historical_output_provenance.sh`
- `scripts/isis/run_ws17_migration_dry_run_pipeline.sh`

## Prerequisites

- `psql`, `curl`, `jq`, `bash` installed.
- `ISIS_DATABASE_URL` (or `DATABASE_URL`) set to staging database.
- `ISIS_API_KEY` set for admin backfill endpoints.
- Service endpoints reachable:
  - `ISIS_WORKFLOW_REGISTRY_URL` (default `http://localhost:3001`)
  - `ISIS_GENERATION_API_URL` (default `http://localhost:3000`)
  - `ISIS_OUTPUT_REGISTRY_URL` (default `http://localhost:3002`)

## Standard Execution

```bash
STRICT=true \
ENABLE_SYNTHETIC_DATASET=true \
CLEANUP_SYNTHETIC_DATASET=true \
ISIS_DATABASE_URL="postgres://..." \
ISIS_API_KEY="..." \
ISIS_WORKFLOW_REGISTRY_URL="https://workflow-registry-staging.example.com" \
ISIS_GENERATION_API_URL="https://generation-api-staging.example.com" \
ISIS_OUTPUT_REGISTRY_URL="https://output-registry-staging.example.com" \
scripts/isis/run_ws17_migration_dry_run_pipeline.sh
```

## High-Volume Controls

Tune synthetic dataset scale:

```bash
SYNTHETIC_WORKFLOW_COUNT=5000 \
SYNTHETIC_JOB_COUNT=25000 \
SYNTHETIC_OUTPUT_COUNT=25000 \
scripts/isis/run_ws17_migration_dry_run_pipeline.sh
```

## Key Environment Variables

| Variable                           | Default         | Description                                                          |
| ---------------------------------- | --------------- | -------------------------------------------------------------------- |
| `STRICT`                           | `true`          | Exit non-zero when any pipeline check fails.                         |
| `RUN_ID`                           | auto-generated  | Stable identifier used in seeded record IDs and report artifacts.    |
| `ENABLE_SYNTHETIC_DATASET`         | `true`          | Enables synthetic high-volume dataset seeding.                       |
| `CLEANUP_SYNTHETIC_DATASET`        | `true`          | Removes synthetic records at end of pipeline.                        |
| `SYNTHETIC_WORKFLOW_COUNT`         | `3000`          | Number of synthetic stale workflows/workflow versions to seed.       |
| `SYNTHETIC_JOB_COUNT`              | `9000`          | Number of synthetic historical jobs to seed.                         |
| `SYNTHETIC_OUTPUT_COUNT`           | `9000`          | Number of synthetic outputs without provenance to seed.              |
| `MIGRATION_LIMIT_OVERRIDE`         | auto-calculated | Forces explicit migration dry-run limit for all migration calls.     |
| `MIGRATION_LIMIT_BUFFER`           | `200`           | Added to auto-calculated limit to absorb baseline growth race.       |
| `MIN_WORKFLOW_STALE_DETECTIONS`    | `1`             | Minimum stale workflow records expected in workflow dry-run report.  |
| `MIN_JOB_BACKFILLED_DETECTIONS`    | `1`             | Minimum backfillable jobs expected in historical-job dry-run report. |
| `MIN_OUTPUT_BACKFILLED_DETECTIONS` | `1`             | Minimum backfillable outputs expected in provenance dry-run report.  |

## Output Artifacts

Default output directory: `tmp/isis-migration/`

Primary artifacts:

- `ws17-migration-dry-run-pipeline-*.json`
- `ws17-dry-run-before-<run-id>.json`
- `ws17-dry-run-after-<run-id>.json`
- `ws17-dry-run-before-after-<run-id>.json`
- `ws17-dry-run-workflow-taxonomy-<run-id>.json`
- `ws17-dry-run-historical-job-taxonomy-<run-id>.json`
- `ws17-dry-run-output-provenance-<run-id>.json`
- `ws17-dry-run-synthetic-dataset-<run-id>.json`
- `ws17-dry-run-synthetic-cleanup-<run-id>.json`

## Failure Handling

If the pipeline fails:

1. Inspect `summary.fail` and individual `checks[]` in the pipeline report.
2. Review per-step logs in `tmp/isis-migration/` for failing migration calls.
3. If cleanup was skipped or failed, run the pipeline again with the same
   `RUN_ID` and `CLEANUP_SYNTHETIC_DATASET=true`.
4. Re-run the full pipeline until `overallPass=true`.
