# Isis Data Retention and Deletion Compliance Runbook

Last Updated: 2026-02-22  
Scope: Outputs, provenance, and model artifact retention/deletion compliance
checks  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`

## Purpose

Provide a deterministic compliance verification routine for Isis retention and
cleanup controls spanning:

1. Output retention/expiration enforcement.
2. Provenance referential integrity.
3. Model artifact cleanup and deletion audit coverage.

## Verification Automation

Use `scripts/isis/verify_data_retention_compliance.sh` to generate a
machine-readable compliance report.

The script validates:

- required retention tables exist.
- at least one retention policy is configured (optional gate).
- no non-expired output remains past `expires_at`.
- no orphan provenance records and no broken output->provenance links.
- no stale abandoned model uploads beyond configured retention window.
- no inconsistent model soft-deletes (`deleted_at` set while `is_active=true`).
- recent model deletes have corresponding audit records.
- abandoned-upload cleanup deletes backing storage artifacts when expected.

## Prerequisites

- `psql` and `jq` installed.
- `ISIS_DATABASE_URL` (or `DATABASE_URL`) set.
- Read access to Isis shared PostgreSQL schema.

## Execution

```bash
STRICT=true \
ISIS_DATABASE_URL="postgres://..." \
scripts/isis/verify_data_retention_compliance.sh
```

Report output:

- `tmp/isis-retention-compliance/retention-compliance-report-*.json`

## Tunable Thresholds

| Variable                                  | Default | Purpose                                                                 |
| ----------------------------------------- | ------- | ----------------------------------------------------------------------- |
| `REQUIRE_RETENTION_POLICY`                | `true`  | Fail when no retention policy exists.                                   |
| `MODEL_ABANDONED_UPLOAD_RETENTION_HOURS`  | `24`    | Age threshold for stale abandoned model uploads.                        |
| `MODEL_DELETE_AUDIT_LOOKBACK_DAYS`        | `30`    | Lookback window for model delete audit coverage check.                  |
| `MAX_EXPIRED_OUTPUTS_PENDING`             | `0`     | Allowed count for outputs past `expires_at` but not expired/deleted.    |
| `MAX_ORPHAN_PROVENANCE_ROWS`              | `0`     | Allowed count for provenance rows without outputs.                      |
| `MAX_BROKEN_OUTPUT_PROVENANCE_LINKS`      | `0`     | Allowed count for outputs pointing to missing provenance rows.          |
| `MAX_ABANDONED_MODEL_UPLOADS`             | `0`     | Allowed count for stale abandoned model uploads.                        |
| `MAX_INCONSISTENT_MODEL_SOFT_DELETES`     | `0`     | Allowed count for models with `deleted_at` set while active.            |
| `MAX_UNAUDITED_MODEL_DELETES`             | `0`     | Allowed count for recent model deletes without audit rows.              |
| `MAX_ABANDONED_MODEL_STORAGE_NOT_DELETED` | `0`     | Allowed count for cleanup deletes where storage object was not deleted. |

## Remediation Guidance

When a check fails:

1. Run output retention application (`POST /api/v1/policies/apply`) and
   re-evaluate stale output counts.
2. Run output provenance backfill
   (`POST /api/v1/outputs/admin/provenance/backfill`) and re-evaluate provenance
   integrity checks.
3. Verify model abandoned-upload cleanup settings and cleanup job execution:
   - `ISIS_MODEL_ABANDONED_UPLOAD_RETENTION_HOURS`
   - `ISIS_MODEL_ABANDONED_UPLOAD_CLEANUP_INTERVAL_MS`
   - `ISIS_MODEL_ABANDONED_UPLOAD_DELETE_STORAGE`
4. Open incident/remediation ticket if repeat failures persist for two
   consecutive compliance windows.

## Recommended Cadence

- Production: daily automated execution.
- Staging: after each schema migration and before releases touching outputs,
  provenance, or model storage.

## Related Artifacts

- `scripts/isis/verify_data_retention_compliance.sh`
- `apps/isis/output-registry/src/services/output.service.ts`
- `apps/isis/output-registry/src/routes/policies.ts`
- `apps/isis/generation-api/src/services/model.service.ts`
- `docs/domains/isis/database.md`
