Observability Reference

What the platform’s committed observability stack declares — generated from docker/observability/: the Prometheus alerting and recording rules, the Grafana dashboards, and the scrape configuration. Each alert names the PromQL it fires on, its severity, the owning team, and its runbook — the detect→diagnose contract an on-call engineer reads.

26Alerts

7Recording rules

16Grafana dashboards

22Scrape targets

Scope: the committed docker/observability/ stack (the platform Prometheus + Grafana configuration) — what the observability config declares, not a claim about every metric a running service emits (those are declared per service in code).

Alerts (26)#

`oshun.isis.alerts` (5)#

IsisGPUOOMcriticalfor 5mteam isis#

DCGM_FI_DEV_FB_USED / clamp_min(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE, 1) > 0.95

Isis GPU {{ $labels.Hostname }} gpu{{ $labels.gpu }} near OOM · runbook

IsisGPUOverheatingwarningfor 5mteam isis#

DCGM_FI_DEV_GPU_TEMP > 88

Isis GPU {{ $labels.Hostname }} gpu{{ $labels.gpu }} overheating

IsisInferenceLatencyHighwarningfor 10mteam isis#

histogram_quantile(
  0.95,
  sum by (le, model) (rate(isis_inference_latency_ms_bucket[5m]))
) > 10000

Isis inference latency high for model {{ $labels.model }}

IsisS3UploadSlowwarningfor 10mteam isis#

histogram_quantile(
  0.95,
  sum by (le, bucket) (rate(isis_s3_upload_duration_seconds_bucket[5m]))
) > 30

Isis S3 upload p95 latency high on bucket {{ $labels.bucket }}

IsisValidationErrorRateHighwarningfor 15mteam isis#

100 *
sum(rate(isis_validation_errors_total[5m]))
/
clamp_min(sum(rate(isis_assets_generated_total[5m])), 0.001) > 5

Isis asset validation error rate above 5%

`oshun.lilith.alerts` (5)#

LilithHandlerDLQGrowingcriticalfor 10mteam lilith#

sum by (event) (lilith_dlq_depth) > 50

Lilith event handler DLQ depth above 50 for {{ $labels.event }}

LilithProjectionLagHighwarningfor 10mteam lilith#

max by (projection) (lilith_projection_lag_seconds) > 60

Lilith projection {{ $labels.projection }} lagging

LilithSessionDropcriticalfor 5mteam lilith#

(sum(rate(lilith_sessions_ended_total[1m])) -
 sum(rate(lilith_sessions_started_total[1m])))
/ clamp_min(sum(lilith_sessions_active), 1) > 0.05

Lilith session drop exceeds 5%/min · runbook

LilithTTSLatencyHighwarningfor 10mteam lilith#

histogram_quantile(
  0.95,
  sum by (le) (rate(lilith_tts_latency_ms_bucket[5m]))
) > 600

Lilith TTS p95 latency above 600ms

LilithXRFrameBudgetExceededwarningfor 5mteam lilith#

histogram_quantile(
  0.95,
  sum by (le, device) (rate(lilith_xr_frame_ms_bucket[5m]))
) > 16.7

Lilith XR p95 frame time exceeds 60fps budget on {{ $labels.device }}

`oshun.platform.alerts` (8)#

FilesystemFillingUpwarningfor 15mteam platform#

100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} /
       node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) > 85

Filesystem {{ $labels.mountpoint }} on {{ $labels.instance }} is filling up

HighErrorRatecriticalfor 5mteam platform#

100 * (
  sum by (domain, service) (rate(http_requests_total{status=~"5.."}[5m]))
  /
  clamp_min(sum by (domain, service) (rate(http_requests_total[5m])), 0.001)
) > 1

High HTTP 5xx error rate on {{ $labels.domain }}/{{ $labels.service }} · runbook

HighLatencyP95warningfor 10mteam platform#

histogram_quantile(
  0.95,
  sum by (domain, service, le) (rate(http_request_duration_seconds_bucket[5m]))
) > 0.5

p95 latency above 500ms on {{ $labels.domain }}/{{ $labels.service }} · runbook

NodeDowncriticalfor 3mteam platform#

up{job="node"} == 0

Node {{ $labels.instance }} is down · runbook

NodeHighCPUwarningfor 10mteam platform#

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90

Node {{ $labels.instance }} CPU saturation

NodeHighMemorywarningfor 10mteam platform#

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 95

Node {{ $labels.instance }} memory pressure

PodCrashLoopingcriticalfor 5mteam platform#

increase(kube_pod_container_status_restarts_total[15m]) > 3

Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping · runbook

TargetDownwarningfor 5mteam platform#

up == 0

Prometheus target {{ $labels.job }}/{{ $labels.instance }} is down

`oshun.sophia.alerts` (4)#

SophiaCitationValidationErrorsHighwarningfor 15mteam sophia#

sum(rate(sophia_citation_errors_total[10m])) > 1

Sophia citation validation errors above 1/s

SophiaIngestionStalledcriticalfor 30mteam sophia#

sum(rate(sophia_ingestion_docs_total[30m])) == 0

Sophia ingestion stalled · runbook

SophiaRAGLatencyHighwarningfor 10mteam sophia#

histogram_quantile(
  0.95,
  sum by (le) (rate(sophia_rag_query_duration_seconds_bucket[5m]))
) > 5

Sophia RAG p95 latency high

SophiaVectorRecallLowwarningfor 15mteam sophia#

avg(sophia_vector_search_recall) < 0.8

Sophia vector search recall below 0.8

`oshun.yemaya.alerts` (4)#

YemayaCRDTOperationDropwarningfor 10mteam yemaya#

sum(rate(yemaya_crdt_operations_total[5m])) == 0 and sum(yemaya_collab_rooms_active) > 0

Yemaya CRDT operations halted while rooms are active

YemayaGenerationFailureSpikecriticalfor 10mteam yemaya#

100 *
sum(rate(yemaya_generation_jobs_total{status="failed"}[10m]))
/
clamp_min(sum(rate(yemaya_generation_jobs_total[10m])), 0.001) > 10

Yemaya generation failure rate above 10%

YemayaRenderQueueBacklogwarningfor 10mteam yemaya#

max by (queue) (yemaya_render_queue_depth) > 1000

Yemaya render queue {{ $labels.queue }} backlog · runbook

YemayaWebSocketChurnwarningfor 10mteam yemaya#

sum(rate(yemaya_ws_disconnects_total[5m])) >
3 * clamp_min(sum(rate(yemaya_ws_connects_total[5m])), 0.001)

Yemaya WebSocket disconnect rate exceeds 3x connect rate

Recording rules (7)#

domain:http_requests:rate5moshun.platform.recording#

sum by (domain) (rate(http_requests_total[5m]))

job:http_error_ratio:5moshun.platform.recording#

sum by (domain, service) (rate(http_requests_total{status=~"5.."}[5m]))
  /
clamp_min(sum by (domain, service) (rate(http_requests_total[5m])), 0.001)

job:http_errors:rate5moshun.platform.recording#

sum by (domain, service) (rate(http_requests_total{status=~"5.."}[5m]))

job:http_request_duration_seconds:p95_5moshun.platform.recording#

histogram_quantile(
  0.95,
  sum by (domain, service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

job:http_request_duration_seconds:p99_5moshun.platform.recording#

histogram_quantile(
  0.99,
  sum by (domain, service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

job:http_requests:rate1moshun.platform.recording#

sum by (domain, service) (rate(http_requests_total[1m]))

job:http_requests:rate5moshun.platform.recording#

sum by (domain, service) (rate(http_requests_total[5m]))

Grafana dashboards (16)#

bellona

Bellona Migration Orchestrator12 panels

oshun bellona migration bridge integrity

hathor

Hathor Narrative Lattice12 panels

oshun hathor narrative worldbuilding neo4j

infrastructure

Infrastructure — Elasticsearch10 panels

oshun infrastructure elasticsearch search

Infrastructure — Kafka10 panels

oshun infrastructure kafka streaming

Infrastructure — Node10 panels

oshun infrastructure node node-exporter

Infrastructure — PostgreSQL10 panels

oshun infrastructure postgres database

Infrastructure — Redis10 panels

oshun infrastructure redis cache

isis

Isis Asset Lifecycle7 panels

oshun isis assets s3

Isis Generation Pipeline7 panels

oshun isis gpu inference

lilith

Lilith Event Handlers7 panels

oshun lilith events cqrs

Lilith Experience8 panels

oshun lilith xr experience

oshun

Oshun Infrastructure7 panels

oshun infrastructure kubernetes

Oshun Platform Overview8 panels

oshun platform overview

sophia

Sophia Research9 panels

oshun sophia research rag kg

yemaya

Yemaya Collaboration8 panels

oshun yemaya collaboration realtime

Yemaya Creative Studio7 panels

oshun yemaya creative bullmq

Scrape targets (22)#

prometheusnodeyemaya-api yemayayemaya-web yemayayemaya-worker yemayalilith-api lilithlilith-web lilithisis-api isisisis-generator isissophia-api sophiahathor-api hathorbellona-api bellonapostgresrediselasticsearchkafkaminiotraefiktempolokipromtailjaeger