Runbook — Provenance Failure

Provenance generation, signing, verification, or ledger commit failed for assets produced or curated by Oshun. Use to block publish/share, regenerate provenance bundles, verify the checksum ledger, and coordinate with the rights and compliance teams.

Scope#

Provenance covers:

• C2PA / content-credentials manifests for generated images, video, audio, and 3D assets produced by Isis.
• Source-binding bundles attached to grounded Sophia answers and the Veritas/Tara/Nisaba/Metis evidence rails.
• Provenance ledger entries for Nisaba editions, manuscripts, and translations.
• Watermark signatures (handled jointly with watermark-verification-failure.md).
• DSAR export packages — each export carries an inner provenance manifest proving export integrity.

Failure modes:

• Manifest generator throws or produces malformed bundles.
• Provenance signature does not validate against the active signing key.
• Checksum mismatch between manifest and asset.
• Ledger commit fails or the ledger consensus rejects the entry.
• Bound source disappeared from the source-of-truth store (orphaned provenance).
• Cross-version drift — manifest references a version that no longer matches the asset bytes.

Severity classification#

Source events: isis.provenance.bundle_generated, isis.provenance.bundle_failed, isis.provenance.ledger_committed, isis.provenance.ledger_rejected, sophia.provenance.source_bound.

Detection signals#

• Alerts: provenance-pipeline panels on dashboards-service-health.ts; the ledger lag and commit-rejection panels.
• Customer signals: spike in evidence.provenance.banner_unavailable_shown, media.provenance.verification_failed_shown.
• Operator signals: rights/compliance review queue flags a provenance defect.
• External signals: a third-party content-credentials verifier reports an invalid manifest publicly.

Initial triage (first 15 minutes)#

1. Acknowledge oshun-isis-oncall (and oshun-compliance-oncall for ledger issues).
2. Block publish/share immediately for the affected asset class — enable the publish-freeze flag scoped to the class. The freeze emits studio.publish.frozen_for_provenance.
3. Determine failure boundary:
   • Generator fault → suspect the most recent Isis workflow promotion (delegate to model-workflow-rollback.md for the workflow rollback).
   • Signing fault → suspect signing-key rotation or HSM availability.
   • Ledger fault → ledger consensus, network, or storage backend.
   • Checksum mismatch → upstream asset corruption or post-sign mutation.
4. Page rights lead if the affected assets are subject to a third-party rights agreement; page compliance lead if a regulatory disclosure is in scope (e.g., DSAR export integrity).

Diagnosis#

1. For generator faults: open the Isis provenance pipeline trace for a failing job — span isis.provenance.generate. The failing sub-step (manifest schema, signing call, ledger commit, asset checksum) tells you which mitigation applies.
2. For signing faults: confirm the signing key fingerprint matches the active key in the key-management record. A rotation drift means the verifier was using a stale key — check rotation logs.
3. For ledger faults: inspect ledger consensus health, peer connectivity, and commit-batch logs. A peer split implies a consensus partition; a single-peer fault implies an availability issue.
4. For checksum mismatches: diff the asset bytes against the bytes recorded in the manifest. A mismatch with a recent post-sign step (e.g., transcoder pass) implicates that step.
5. For orphaned provenance: a bound source disappeared from the source-of-truth store. Inspect the source-store deletion log; a wrongful deletion is the prime cause.

Mitigation#

1. Block publish/share: already done in triage.
2. Regenerate provenance bundles for the affected assets — enqueue isis.provenance.regenerate.requested for the asset cohort. The regeneration uses the certified pipeline; if the certified pipeline is the broken one, the rollback in model-workflow-rollback.md must precede regeneration.
3. Verify checksum ledger: run the ledger reconciliation job over the affected window. The job emits isis.provenance.ledger_reconciled with the per-entry verdict (matched, mismatched, missing). Mismatched entries are quarantined and re-signed against the recovered asset bytes.
4. Signing-key rotation recovery: if a rotation drift caused the failure, re-publish the active key fingerprint to all verifier nodes and re-verify the impacted window. Do not roll back the rotation — forward fixes only for signing keys (rotation rollback creates a non-repudiation hole).
5. Ledger fault recovery: if the ledger backend is partitioned, coordinate with the data-platform team to restore consensus. Pending commits queue durably; publish remains frozen until commit lag is below SLO.
6. Customer-visible degraded mode: while frozen, customer surfaces that depend on provenance render the evidence.provenance.unavailable_shown banner with a "we are verifying" copy block. Source-inspection still works against the source-of-truth store; only the cryptographic provenance attestation is hidden.

Communication cadence#

The customer-visible copy is owned by compliance; provenance failures touch disclosure and trust commitments.

Escalation#

• Isis lead is incident commander for generator/signing/checksum failures.
• Compliance lead is incident commander for ledger/regulator-scope failures and DSAR-export integrity failures.
• For sev1 forged-manifest cases, page security and legal at minute 0.
• Standard 5/5/15/30/60 escalation otherwise.

Recovery verification#

• Provenance generator failure rate < 0.5% for 30 min.
• Ledger commit lag below SLO for 30 min and consensus peers green.
• Manifest verifier returns valid for a 50-asset post-regeneration sample.
• Publish-freeze lifted; smoke-test 20 freshly-published assets through the full verification chain (manifest, signature, checksum, ledger).
• DSAR exports issued during the freeze are reissued where their inner manifest was invalid.

Post-incident#

• Postmortem must include: the failing pipeline step, the affected asset cohort, the manifest-regeneration record, ledger reconciliation report, signing-key history, and any third-party verifier findings.
• Update the provenance evaluation fixtures to cover the failure mode.
• File a rights/compliance notice for any externally-licensed asset whose provenance was customer-visible-broken; the notice route depends on the rights agreement.
• If the ledger backend partitioned, file a data-platform reliability ticket and a ledger-DR drill on the calendar.