# Runbook: V1 Signing Service Outage (`gd:signing-service-outage`)

Owner: delivery pipeline on-call (`rota:delivery-pipeline`). Last reviewed:
2026-08-14. Version: 1.

> Releases cannot be signed and content credentials cannot be attached. Reads
> and writes are untouched, which is what makes this a service outage rather
> than a regional one.

**What `dep:signing-service` actually is on this estate.** There is no `signing`
service in `docker/docker-compose.yml`. The signer is **in-process in the BFF**:
`apps/oshun/bff/src/generation/audio-c2pa-signer.ts` produces a real Ed25519
signature over a canonical manifest that binds the SHA-256 of the exact media
bytes, and `resolveAudioC2paSigner` returns `null` when `OSHUN_C2PA_SIGNING_KEY`
is absent. So "the signing service is down" and "the BFF is down" are the same
event **except in the case this runbook is mostly about**: the process is
healthy and the key is missing or invalid, which is the only signing outage that
can happen while everything else keeps serving.

Related: [v1-dr-signing-key-compromise.md](v1-dr-signing-key-compromise.md) is
the other half — the key is present and no longer trustworthy. That one is not
reversible; this one is.

## 1. Detection and declaration

**Detection signal — a gate block, not an error rate.** With no signer,
`narration-governance.ts`, `image-governance.ts` and `music-governance.ts` each
pass `c2paSigner: null` and the release gate reports
`provenanceManifestPresent: false` and BLOCKS. Nothing throws. The observable is
that provenance-gated output stops being admitted while every liveness check
passes.

The one loud failure is a malformed key: `resolveAudioC2paSigner` throws
`audio_c2pa_signing_key_invalid` rather than falling back to unsigned output.

- Detection authority: **delivery pipeline on-call**, contact
  `rota:delivery-pipeline`, as `dep:signing-service` is registered in the S10.5
  dependency register.
- Declaration authority: **production operations duty officer**, contact
  `rota:prod-ops`.
- **Gap, stated.** `docs/operations/on-call.md` describes a weekly
  primary/secondary rotation and names no person and no rotation config; none of
  the register's `rota:` identifiers appears in it or in
  `docs/operations/alert-routing.md`. `.github/CODEOWNERS` names `@GreyChimp`
  for every path, which is a review owner and not an incident route. The name
  above is registered; the route from it to whoever is carrying the pager is
  not.

## 2. Communications

| Audience  | Within | Content                                                                        |
| --------- | ------ | ------------------------------------------------------------------------------ |
| Actors    | 15 min | that signing is unavailable and gated output is blocked, not silently unsigned |
| Operators | 5 min  | that this runbook is running, and by whom                                      |

Both travel over `dep:notification-channel`, which this scenario leaves standing
— the fault is confined to `dep:signing-service`.

## 3. Dependency assumptions

The recovery assumes exactly these, and nothing else:

- `dep:primary-store` is healthy: release rows carry the key version their
  signature was made under, which is the reference edge S10.11 registers between
  the two.
- `dep:blob-store` is healthy: the bytes the manifest binds a hash of have to be
  readable to re-sign anything.

## 4. Recovery infrastructure

**In place.** The signer runs inside the BFF process; there is no separate host
to move to and none is needed, because the fault is a configuration one.

## 5. Secrets

One, and it is the whole scenario:

- `OSHUN_C2PA_SIGNING_KEY` — a PEM or base64-PEM PKCS#8 Ed25519 private key —
  with `OSHUN_C2PA_SIGNER_KEY_ID` naming it (default `oshun-audio-c2pa-v1`).

It is recoverable out of band: the off-box backup captures the
`configuration-and-secrets` state class (`infra/hetzner/backup/offbox-backup.sh`
lists `config` among its inputs), and that repository is required to be off the
V1 host.

> **Do not re-key to recover an outage.** Restoring the SAME key ends this
> incident. Minting a new one silently starts
> [v1-dr-signing-key-compromise.md](v1-dr-signing-key-compromise.md)'s problem
> without its cause: manifests already issued stay bound to the old key id, and
> nothing on this estate reconciles two key ids.

## 6. Restore order

One dependency, so the order is a single step: bring `dep:signing-service` back
by restoring `OSHUN_C2PA_SIGNING_KEY` to the BFF environment and restarting the
process that reads it.

The order matters against the reference graph only if the relational store is
also being restored: `dep:primary-store` holds release rows that name a key
version, so the signer comes back **before** the rows that reference it. That is
the reverse of the capture order, and `restoreOrderFaults` in
`@oshun/workbench-kit/game-day` checks a proposed order against the same graph.

## 7. Validation

`verifyAudioC2paManifest` in
`apps/oshun/bff/src/generation/audio-c2pa-signer.ts` is a real check and not an
assertion. Against a known media file it:

1. recomputes the SHA-256 of the media bytes and compares it to
   `manifest.mediaHashSha256`;
2. rebuilds the canonical string and compares its hash to
   `manifest.canonicalHash`;
3. verifies the Ed25519 signature.

Sign one known file, verify it, and confirm the release gate reports
`provenanceManifestPresent: true` again.

> **Gap, stated.** Step 3 verifies against `manifest.publicKeyPem` — the key
> carried inside the manifest being checked. It establishes that whoever
> produced the manifest held the matching private key. It does **not** establish
> that the key was ours. See §8.

## 8. Fencing — NOT APPLICABLE, and the reason is not comfort

Nothing is promoted here, so there is no old primary to fence. But the absence
of a trust list is the same defect wearing a different hat: nothing on this
estate compares `signerKeyId` or `publicKeyPem` against an allowlist, so a
manifest signed by any key verifies. During this outage that means unsigned
output is refused (good) and a manifest signed by something else is not (bad).
`dep:trust-store` is registered as a core dependency and has no implementation.

## 9. Failover

There is no standby signer and none is proposed. The estate has exactly one
signing path and a second one would need a second key, which is a trust problem
rather than an availability one.

## 10. Reconciliation

Output produced while the gate was blocking was **refused, not queued**. There
is no backlog of unsigned artifacts to sign retroactively: the governance layer
never emitted a manifest, so nothing carries a placeholder that has to be
replaced. Re-run the blocked release; do not attempt to sign old output.

## 11. Failback

The estate never left its normal topology, so failback is the confirmation that
it is back in it: one signed-and-verified artifact, and the release gate
admitting provenance again. Record the instant — recovery time runs from the
fault, not from the restart.

## 12. What this runbook does not cover

- A compromised key. That is
  [v1-dr-signing-key-compromise.md](v1-dr-signing-key-compromise.md), and the
  recovery is rotation rather than restoration.
- The second signer. `apps/oshun/bff/src/routes/living-scenes.ts` reads
  `OSHUN_LIVING_SCENES_C2PA_KEY_ID`, so the estate has more than one signing key
  id. This runbook covers the generation signer only.
- A total host loss, which takes the BFF with it — see
  [v1-disaster-recovery.md](v1-disaster-recovery.md).

## 13. Machine-checkable statement

The block below is the input `runbookFaults` in `@oshun/workbench-kit/game-day`
grades, and `scripts/operations/v1-dr-runbook.test.mjs` runs it against the
scenario register. It is here rather than in a test file so that the statement
the checker reads and the statement the operator reads are one object.

<!-- oshun:runbook -->

```json
{
  "id": "runbook:v1-signing-service-outage",
  "scenarioId": "gd:signing-service-outage",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "delivery pipeline on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:blob-store"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "OSHUN_C2PA_SIGNING_KEY",
      "heldInDependencyId": "dep:signing-service",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:signing-service"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "verifyAudioC2paManifest returns ok for a freshly signed known media file",
    "the release gate reports provenanceManifestPresent true again"
  ],
  "failback": "confirm one signed-and-verified artifact and the gate admitting provenance; the estate never left its normal topology"
}
```