# Phase 92 MLOps, lifecycle, and governance

The Phase 92 implementation is provider-neutral and fail-closed. Durable stores,
MLflow/W&B clients, model routers, queues, evaluators, flag providers, alerting,
Prometheus exporters, artifact storage, and scanners enter through interfaces;
the core does not claim an in-memory adapter is production persistence.

## Capability map

- `mlops-experiments.ts`: unified run SDK, parameters, metrics, artifacts,
  tags/search, lineage DAG, side-by-side/Pareto model comparison, and training
  job dashboard summaries.
- `mlops-lifecycle.ts`: sliding-window output/calibration/latency drift,
  retraining and promotion gates, stable A/B assignment, shadow variants,
  feature-flag routing, staged canaries, transactional rollback with health
  checks, deprecation retention, endpoint SLA evaluation, and model incident
  severity/playbooks aligned to NIST AI 600-1 and MITRE ATLAS.
- `mlops-governance.ts`: Hugging Face model cards, RBAC approval stages, bias
  disparity audits, hash-chained operation logs, license compatibility,
  reproducibility manifests, tiered risk policies, inventory summaries,
  chargeback/budget alerts, GPU FinOps recommendations, five-part adversarial
  gates, and security governance validation.
- `infra/nous/phase-92`: pinned, persistent MLflow + PostgreSQL + MinIO
  deployment with loopback-only exposure and health checks.
- The existing `/models` admin workspace is the operational UI boundary. Its
  model comparison, evaluation and governance panels consume registry evidence;
  Phase 92 stores must project experiment/job, lifecycle/SLA, approval/audit,
  and cost summaries into that authenticated workspace rather than creating a
  second control plane.

## Production invariants

All model-changing operations require authenticated actor identity and append to
both the immutable model audit and deployment audit sinks. Customer-facing and
safety-critical models cannot reach production without human approval, current
adversarial evidence, a rollback plan, and continuous/daily monitoring. Canary
assessment and rollback health failures fail closed. Artifact content is hashed;
reproduction manifests pin data, code, config, seeds, and environment.

Prometheus adapters should export endpoint latency p95/p99, availability,
throughput, errors, drift statistics, rollout stage, GPU utilization, and cost.
Monthly reporters aggregate `evaluateSla`, `allocateMlCosts`, and governance
inventory results; alert sinks own delivery and escalation policy.
