Disciplines · Runbooks

Runbook — Watermark Verification Failure

Watermarking is applied to Isis-generated assets (image, video, audio, 3D) and to selected high-trust Sophia evidence captures.

10sections5 minread
Markdown source
On this page

Watermarking signature or recovery against a generated asset failed verification. Use to block asset promotion, run watermark recovery, and route the exception to rights and provenance review.

Scope#

Watermarking is applied to Isis-generated assets (image, video, audio, 3D) and to selected high-trust Sophia evidence captures. The watermark is distinct from the C2PA / content-credentials manifest (covered by provenance-failure.md) — the watermark proves the asset bytes themselves were emitted by Oshun, even when stripped of metadata.

Failure modes:

  • Encoder failed to embed the watermark in a generated asset.
  • Verifier cannot extract a watermark from an asset that should have one.
  • Watermark extracted but does not match the expected payload (signature mismatch, payload tampering).
  • Watermark survives a transformation it should not (false positive) or fails a transformation it should survive (false negative on a known benchmark transform).
  • Recovery pipeline (against scraped/redistributed media) fails to detect a known emitted watermark.

Severity classification#

Sev Condition
sev1 Watermark encoder is silently no-op — assets are emitted unwatermarked while telemetry reports success.
sev1 Verifier accepts a forged or tampered payload as valid (signature compromise).
sev2 Encoder failure rate > 1% sustained; recovery pipeline false-negative rate > 5%; verifier false-positive rate above threshold.
sev3 Single asset rejected; no promotion has occurred.

Source events: isis.watermark.encoded, isis.watermark.encode_failed, isis.watermark.verified, isis.watermark.verification_failed, isis.watermark.recovered.

Detection signals#

  • Alerts: watermark-pipeline panels on dashboards-service-health.ts — encode success rate, verify success rate, recovery hit/miss against the scheduled audit set.
  • Benchmark: scheduled evaluation-manifest.ts watermark scope runs daily against a fixture set covering compression, crop, rotation, color shift, audio re-encode, and video re-encode. A regression past threshold fires a sev2.
  • Customer signals: spike in media.share.watermark_dispute_filed (a third party challenges the asset's origin).
  • Operator signals: rights review queue flags a watermark dispute.

Initial triage (first 15 minutes)#

  1. Acknowledge oshun-isis-oncall (and oshun-compliance-oncall for sev1 signature compromise).
  2. Block asset promotion immediately: enable the promotion-freeze flag for the affected asset class. The freeze emits studio.asset.promotion_frozen. Already-promoted assets are not recalled; new promotions are paused.
  3. Determine failure boundary:
    • Encoder fault → suspect the most recent Isis workflow promotion; delegate to model-workflow-rollback.md.
    • Verifier fault → suspect verifier-version drift or signing-key drift.
    • Recovery fault → suspect detector drift or fixture drift.
    • Forged payload accepted → escalate to security and treat as potential signing-key compromise.
  4. Page rights lead if the affected assets are subject to a rights agreement (e.g., licensed sources).

Diagnosis#

  1. Encoder fault: re-run the encode for 20 sample assets and verify the watermark extract round-trip. If the encoder emits a payload that the verifier rejects, the encoder/verifier pair is misaligned — verify both are on the same certified version.
  2. Verifier fault: pull 50 known-good emitted assets (last 7 days) and run them through the verifier. A drop in extract rate implicates the verifier. Compare verifier configuration (keys, parameters) to the release-taxonomy certified row.
  3. Recovery fault: run the recovery pipeline against the daily fixture set. If the per-transform recovery rate is below threshold, the detector regressed. Compare against the last certified detector version in release-taxonomy.
  4. Forged payload: confirm the payload's claimed signing key versus the active key set. If the signature validates against a key not in the active set, the key is compromised or the rotation history is corrupted — treat as security incident.
  5. Customer dispute: cross-reference the disputed asset's payload against the Oshun emission registry. If no emission record exists, the asset is not Oshun-origin and the dispute is dismissed with a public response template; if a record exists, follow the recovery-and-attribute path.

Mitigation#

  1. Block asset promotion: already done in triage.
  2. Watermark recovery: enqueue isis.watermark.recovery.requested for the affected asset cohort — the recovery pipeline re-extracts payloads from the asset bytes and compares against the emission registry. Mismatches are quarantined.
  3. Encoder rollback: if the encoder is the failure boundary, follow model-workflow-rollback.md for the encoder workflow target. Already-emitted assets remain emitted; their inner watermark is verified against the encoder version that emitted them.
  4. Verifier rollback or forward-fix: verifier regressions should be forward-fixed where possible (an updated verifier sees more transforms, not fewer). If the regression is a misaligned key set, re-publish the certified key set to all verifier nodes; do not rotate keys during the incident.
  5. Signing-key compromise (sev1): do not roll back the rotation — issue a new signing key and publish the revocation of the compromised key. Coordinate with security; treat the compromised window's assets as not-attestable until re-attested under the new key.
  6. Customer-visible mode: while frozen, customer "verify origin" tooling renders media.share.verification_unavailable_shown instead of a hard error. Recovery against externally-redistributed media continues to operate against the recovered emission registry.

Communication cadence#

Phase Template ID Audience Cadence
Internal incident.internal.slack.investigating internal within 5 min
Identified incident.customer.status_page.identified status_page for customer-visible blocks
Rights/partner incident.partner.email.identified partner if external rights affected
Regulator incident.regulator.email.resolved (adapt) regulator only for sev1 signing-key compromise
Resolved incident.customer.status_page.resolved status_page within 60 min of unblock
Postmortem incident.customer.status_page.postmortem status_page/legal within 5 biz day

Customer dispute responses use the rights/legal-co-authored template; do not improvise.

Escalation#

  • Isis lead is incident commander for encoder/verifier/recovery failures.
  • Security lead is incident commander for signing-key compromise.
  • For sev1 forged-payload acceptance, page security, compliance, and legal at minute 0.
  • Standard 5/5/15/30/60 escalation otherwise.

Recovery verification#

  • Encoder success rate ≥ 99.9% for 60 min.
  • Verifier extract rate ≥ 99% over a fresh 50-asset known-good sample.
  • Recovery pipeline passes the daily benchmark set above threshold.
  • Promotion-freeze lifted after a 50-asset smoke check.
  • For signing-key compromise: revocation propagated, new key published, re-attestation queue cleared (or assets explicitly marked not-attestable with public-facing copy approved by legal).
  • No new watermark_dispute_filed for 7 days post-resolution.

Post-incident#

  • Postmortem must include: which side regressed (encoder/verifier/recovery/ key), the affected emission window, the recovery outcomes, the asset cohort with explicit attribution outcomes, and any rights/legal notifications.
  • Update the watermark benchmark fixtures to include the failure transform or attack vector.
  • If a key compromise occurred, file a key-rotation policy review and a HSM access-audit review.
  • Update the customer-facing "verify origin" copy if the incident revealed a copy gap.