Runbook: V1 Signing Service Outage (gd:signing-service-outage)

Owner: delivery pipeline on-call (rota:delivery-pipeline). Last reviewed: 2026-08-14. Version: 1.

Releases cannot be signed and content credentials cannot be attached. Reads and writes are untouched, which is what makes this a service outage rather than a regional one.

What dep:signing-service actually is on this estate. There is no signing service in docker/docker-compose.yml. The signer is in-process in the BFF: apps/oshun/bff/src/generation/audio-c2pa-signer.ts produces a real Ed25519 signature over a canonical manifest that binds the SHA-256 of the exact media bytes, and resolveAudioC2paSigner returns null when OSHUN_C2PA_SIGNING_KEY is absent. So "the signing service is down" and "the BFF is down" are the same event except in the case this runbook is mostly about: the process is healthy and the key is missing or invalid, which is the only signing outage that can happen while everything else keeps serving.

Related: v1-dr-signing-key-compromise.md is the other half — the key is present and no longer trustworthy. That one is not reversible; this one is.

1. Detection and declaration#

Detection signal — a gate block, not an error rate. With no signer, narration-governance.ts, image-governance.ts and music-governance.ts each pass c2paSigner: null and the release gate reports provenanceManifestPresent: false and BLOCKS. Nothing throws. The observable is that provenance-gated output stops being admitted while every liveness check passes.

The one loud failure is a malformed key: resolveAudioC2paSigner throws audio_c2pa_signing_key_invalid rather than falling back to unsigned output.

• Detection authority: delivery pipeline on-call, contact rota:delivery-pipeline, as dep:signing-service is registered in the S10.5 dependency register.
• Declaration authority: production operations duty officer, contact rota:prod-ops.
• Gap, stated. docs/operations/on-call.md describes a weekly primary/secondary rotation and names no person and no rotation config; none of the register's rota: identifiers appears in it or in docs/operations/alert-routing.md. .github/CODEOWNERS names @GreyChimp for every path, which is a review owner and not an incident route. The name above is registered; the route from it to whoever is carrying the pager is not.

2. Communications#

Both travel over dep:notification-channel, which this scenario leaves standing — the fault is confined to dep:signing-service.

3. Dependency assumptions#

The recovery assumes exactly these, and nothing else:

• dep:primary-store is healthy: release rows carry the key version their signature was made under, which is the reference edge S10.11 registers between the two.
• dep:blob-store is healthy: the bytes the manifest binds a hash of have to be readable to re-sign anything.

4. Recovery infrastructure#

In place. The signer runs inside the BFF process; there is no separate host to move to and none is needed, because the fault is a configuration one.

5. Secrets#

One, and it is the whole scenario:

• OSHUN_C2PA_SIGNING_KEY — a PEM or base64-PEM PKCS#8 Ed25519 private key — with OSHUN_C2PA_SIGNER_KEY_ID naming it (default oshun-audio-c2pa-v1).

It is recoverable out of band: the off-box backup captures the configuration-and-secrets state class (infra/hetzner/backup/offbox-backup.sh lists config among its inputs), and that repository is required to be off the V1 host.

Do not re-key to recover an outage. Restoring the SAME key ends this incident. Minting a new one silently starts v1-dr-signing-key-compromise.md's problem without its cause: manifests already issued stay bound to the old key id, and nothing on this estate reconciles two key ids.

6. Restore order#

One dependency, so the order is a single step: bring dep:signing-service back by restoring OSHUN_C2PA_SIGNING_KEY to the BFF environment and restarting the process that reads it.

The order matters against the reference graph only if the relational store is also being restored: dep:primary-store holds release rows that name a key version, so the signer comes back before the rows that reference it. That is the reverse of the capture order, and restoreOrderFaults in @oshun/workbench-kit/game-day checks a proposed order against the same graph.

7. Validation#

verifyAudioC2paManifest in apps/oshun/bff/src/generation/audio-c2pa-signer.ts is a real check and not an assertion. Against a known media file it:

1. recomputes the SHA-256 of the media bytes and compares it to manifest.mediaHashSha256;
2. rebuilds the canonical string and compares its hash to manifest.canonicalHash;
3. verifies the Ed25519 signature.

Sign one known file, verify it, and confirm the release gate reports provenanceManifestPresent: true again.

Gap, stated. Step 3 verifies against manifest.publicKeyPem — the key carried inside the manifest being checked. It establishes that whoever produced the manifest held the matching private key. It does not establish that the key was ours. See §8.

8. Fencing — NOT APPLICABLE, and the reason is not comfort#

Nothing is promoted here, so there is no old primary to fence. But the absence of a trust list is the same defect wearing a different hat: nothing on this estate compares signerKeyId or publicKeyPem against an allowlist, so a manifest signed by any key verifies. During this outage that means unsigned output is refused (good) and a manifest signed by something else is not (bad). dep:trust-store is registered as a core dependency and has no implementation.

9. Failover#

There is no standby signer and none is proposed. The estate has exactly one signing path and a second one would need a second key, which is a trust problem rather than an availability one.

10. Reconciliation#

Output produced while the gate was blocking was refused, not queued. There is no backlog of unsigned artifacts to sign retroactively: the governance layer never emitted a manifest, so nothing carries a placeholder that has to be replaced. Re-run the blocked release; do not attempt to sign old output.

11. Failback#

The estate never left its normal topology, so failback is the confirmation that it is back in it: one signed-and-verified artifact, and the release gate admitting provenance again. Record the instant — recovery time runs from the fault, not from the restart.

12. What this runbook does not cover#

• A compromised key. That is v1-dr-signing-key-compromise.md, and the recovery is rotation rather than restoration.
• The second signer. apps/oshun/bff/src/routes/living-scenes.ts reads OSHUN_LIVING_SCENES_C2PA_KEY_ID, so the estate has more than one signing key id. This runbook covers the generation signer only.
• A total host loss, which takes the BFF with it — see v1-disaster-recovery.md.

13. Machine-checkable statement#

The block below is the input runbookFaults in @oshun/workbench-kit/game-day grades, and scripts/operations/v1-dr-runbook.test.mjs runs it against the scenario register. It is here rather than in a test file so that the statement the checker reads and the statement the operator reads are one object.

{
  "id": "runbook:v1-signing-service-outage",
  "scenarioId": "gd:signing-service-outage",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "delivery pipeline on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:blob-store"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "OSHUN_C2PA_SIGNING_KEY",
      "heldInDependencyId": "dep:signing-service",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:signing-service"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "verifyAudioC2paManifest returns ok for a freshly signed known media file",
    "the release gate reports provenanceManifestPresent true again"
  ],
  "failback": "confirm one signed-and-verified artifact and the gate admitting provenance; the estate never left its normal topology"
}