Danu Server Meshing & Nephthys Persistence

Mawu's promise — a roleplay realm that holds a thousand inhabitants in one seamless world, hosted by a community operator the platform does not trust — has exactly two load-bearing systems underneath it. Danu is the meshing control plane: it makes many simulation nodes behave as one world by holding single authoritative writer per entity, handing authority across partition boundaries atomically, and bending under load instead of breaking. Nephthys is the decoupled source of truth: an event-sourced, snapshot-checkpointed store that brokers read/write views between mesh nodes and rehydrates a replacement node after a crash with nothing lost. The two are designed as a pair — Danu owns who simulates what, right now; Nephthys owns what is true, durably — so a sim node can die, be preempted off a Spot instance, or be split in two without a player ever reconnecting. This is the architecture that lets persistent RP state ride cheap, preemptible compute: the truth is never in the node.

Both follow named industry lessons rather than invention: the Star Citizen replication-layer split (authority decoupled from the durable layer), the SpatialOS failure (never shard tightly-coupled physics across workers), and the EVE Online time-dilation valve (slow the clock, never drop the player). This page is the deep dive behind the "Realm Backbone" group of the orientation hub ../V7_ARCHITECTURE.md. It is the persistence-and-meshing companion to ./moremi-realm-server-and-netcode.md (the authoritative sim loop and the client↔realm wire), draws its character and economy aggregates from ./nana-data-model-and-pheme-voice.md, and feeds the orchestration story in ./hosting-fleet-data-and-compliance.md.

What ships, honestly#

The meshing and replication logic is real, deterministic, and guarded by falsifiable CI gates — but it is library-grade simulation, not a running distributed cluster, and the daemons that wrap it are health endpoints. Four honest layers:

• Real and exercised by tests. Danu's two-phase atomic authority handoff over a Nephthys-modelled ledger, grid interest management with a bounded working set, co-location pinning, density-driven split/merge, and the time-dilation overload valve are all genuine algorithms with named numeric budgets, covered by 22 #[test] cases in apps/v7/danu-mesh-cluster/src/lib.rs. Nephthys's event-sourced store, single-writer replication broker with fencing tokens, deterministic crash rehydrate, and byte-verified persistence API are real, covered by 15 tests in apps/v7/nephthys-replica-service/src/lib.rs. Each eval emits a typed report with a strict passed() predicate that would fail against a regression.
• The eval harnesses are deterministic in-process models, not live multi-node systems. A "node death" is a boolean flip on an in-process struct (source_node.alive = false, lib.rs:1855); the mesh "cluster" is a two-partition BTreeMap. Their value is as golden CI gates that pin exact invariants (zero lost entities, zero duplicate authority, zero reconnects over two million crossings) — not as a deployed control plane.
• The daemons are health-only. Both binaries' run_service() bind a TCP health server that emits the crate's ServiceDescriptor JSON ("status":"ready" plus the capability list, service_contract.rs); each main.rs is one line. There is no accept loop ingesting real sim-node traffic, no live replication socket. The Danu daemon listens on :47202, Nephthys on :47203, and both serve only /health.
• Persistence is an in-memory event-sourced store with a documented Postgres target. NephthysEventSourcedStateStore (lib.rs:1271) is a Vec of events plus a BTreeMap of snapshots — its byte-canonical encoding and deterministic hashing are precisely what make the monolith's Postgres+pgvector durable backing a safe swap, but that backing is not in this crate. The one place V7 touches a reused persistence substrate today is the identity path: libs/v7/substrate-bridge consumes the real V6 Egbe Ori event store, projection materializer, and operator-read audit primitives.

The three-part backbone#

Danu, Nephthys, and the realm wire are three separable concerns that compose into one guarantee. The realm-protocol crate (libs/v7/realm-protocol) carries the bytes; Danu decides which node owns an entity and routes interest; Nephthys is the authority broker and the durable log. The sim nodes themselves are the Moremi realm servers (apps/v7/moremi-realm-server), which depend on the Danu crate directly (DanuMeshCluster, DanuMeshNodeId) and consume Nephthys through a signed-receipt abstraction.

Danu — meshing for a 1000+ population#

Danu's contract (arch§"Danu — Server Meshing") is six invariants. The crate implements each as a data structure plus a deterministic eval that proves it.

Single-writer authority and the atomic two-phase handoff#

Exactly one node holds write authority over an entity; everyone else reads a view sourced from Nephthys. Crossing a partition boundary therefore means moving authority, and the dangerous window is two writers (or zero) for one entity. DanuMeshCluster::move_entity_across_boundary (lib.rs:1822) runs the four-record sequence against a DanuNephthysHandoffLedger: it freezes the entity (frozen = true), records Freeze then Transfer (lib.rs:1850), commits the new owner into the ledger, records Ack, flips the entity's owner_node_id, and finally records Release and unfreezes (lib.rs:1874). The mid-handoff crash case is explicit: when the eval kills the source node between Transfer and Ack, recovery is modelled (alive toggles back, the recovered counter increments) so the handoff still completes against Nephthys.

The gate is run_danu_mesh_handoff_eval (lib.rs:1973): 200 entities cross a boundary 10,000 times each — two million handoffs — with a deliberate mid-handoff node death injected at crossing 5,000. DanuMeshHandoffEvalReport::passed (lib.rs:1086) demands total_handoffs == expected_handoffs, every one of the four phase counters equal to that total, mid_handoff_node_deaths > 0 and fully recovered, connected_clients_before == connected_clients_after, client_reconnects == 0, and empty lost_entities / duplicate_authority_entities / owner_mismatch_entities. The owner-mismatch check cross-references the spatial partition and the ledger, so an entity whose position and ledger owner disagree fails the gate — the single-writer invariant is enforced, not asserted.

Interest management: the bounded working set#

A client never receives the whole realm. run_danu_area_of_interest_eval (lib.rs:2002) populates a 1,000-inhabitant grid (40×25 cells, DANU_AOI_* constants) and, for every inhabitant, computes a working set that is culled three ways: by proximity (radius 2 cells), by line-of-sight, and by channel (Public / Crew / Trade subscriptions must intersect). The report's passed() (lib.rs:1167) requires that the maximum working set stay ≤ 64 entities and the maximum per-frame estimate stay ≤ 4096 bytes, that all three culling paths actually removed entities (each total > 0), and that no working set is empty. Because the budget is a bound on visible density rather than population, per-client bandwidth is independent of realm size — the same property the realm-protocol delta layer enforces with its 32-snapshot ack-relative ring and 64–256 kbit/s budget (REALM_DELTA_SNAPSHOT_RING_SIZE, realm-protocol/src/lib.rs:58), which is the wire Danu's interest sets feed.

Co-location and density-driven split/merge#

The SpatialOS lesson is that tightly-coupled physics must never straddle nodes. run_danu_colocation_eval (lib.rs:2073) builds 128 coupled pairs — PhysicsJoint, VehicleRider, CombatGrapple — and pins both members of each pair to one anchor node; the gate fails on any split_pair_ids, any missing assignment, or any pair whose interaction frequency drops below the 60 Hz co-location threshold. Splitting is the hard part, so Danu proves it static first: run_danu_dynamic_meshing_eval (lib.rs:2121) begins by running the full static handoff eval and refusing to proceed unless it passed, then drives a crowd to gather (density crosses the split threshold of 120) and disperse (below the merge threshold of 40). The smoothness budget caps remaps at 16 entities per tick and per-tick cost at one 60 Hz frame (16,667 µs), and the topology must return to Static with the original partition count — a split that never merges back fails the gate.

The overload valve: time dilation, not authority drop#

When a node cannot simulate a hot region in real time, dropping authority or disconnecting players is the worst outcome. Danu instead slows the region's clock, EVE-style. danu_overload_clock_percent (lib.rs:2500) is the real formula: at or below capacity the clock runs at 100%; above it, the clock is capacity × 100 / load, clamped to a 10% floor. run_danu_overload_eval (lib.rs:2237) drives 256 players through 36 ticks with a load spike to 18,000 units against a 1,000-unit capacity, and the gate (DanuOverloadEvalReport) requires that the clock did dilate, never fell below the floor, dropped zero players, dropped zero authority records, preserved event ordering, and kept per-player event lag at zero — fairness under dilation, not just survival.

Degradation: crash, capacity, control-plane loss#

run_danu_degradation_eval (lib.rs:2345) is the chaos gate, composing three scenarios. Node failure must rehydrate from Nephthys (the RehydratedFromNephthys event, a present fallback, and playable_after). Capacity exhaustion must queue the 64 excess joins rather than reject them (JoinQueued). Control-plane loss must fall back to single-node operation (SingleNodeFallbackEnabled). Across all three, the aggregate gate demands zero client reconnects, zero lost entities, and zero authority drops — the realm degrades, it does not break. This scenario is also where the Danu↔Nephthys seam is most visible: the Spot-reclaim path (handle_spot_preemption, lib.rs:517) shuts down a preempted game server and reallocates a replacement that restores from a DanuNephthysSessionCheckpoint with no player-visible state loss — the literal reason persistent RP realms are safe on preemptible compute. The Agones fleet, autoscaler, and cost-tier machinery that surround this live in the Danu crate too but belong to ./hosting-fleet-data-and-compliance.md.

Nephthys — persistence and the replication layer#

Nephthys is the decoupled truth. Its two jobs are to be the durable event-sourced store and to be the replication broker that enforces the single-writer rule Danu depends on.

Event-sourced aggregates and deterministic rehydrate#

State is an append-only event log with periodic snapshot projections. NephthysEventSourcedStateStore::append_event (lib.rs:1296) assigns a monotonic sequence, hashes the event, and checkpoints the aggregate every third version (NEPHTHYS_CHECKPOINT_INTERVAL = 3). Critically, realm state and character state are separate aggregates (NephthysAggregateKind::Realm vs Character) with independent projections, so a character can travel a federation corridor carrying only its own aggregate. Recovery is snapshot + tail replay: rehydrate_aggregate (lib.rs:1357) loads the latest snapshot and replays only events after its through_sequence. The crash gate run_nephthys_state_store_rehydrate_smoke (lib.rs:1525) builds a realm with two characters, rehydrates a primary and a replacement node, and diffs them with diff_nephthys_rehydrated_nodes (lib.rs:1471); passed() (lib.rs:766) requires that snapshots and tail replay were both used, that the ledger diff is empty, that projection_hash_before == projection_hash_after, and that the replacement reaches the ledger head. A replacement node is bit-identical to the node it replaces — that is the whole point.

The single-writer replication broker (fencing tokens)#

NephthysReplicationBroker (lib.rs:878) is where the single-writer invariant becomes a write protocol. grant_authority (lib.rs:903) issues an incrementing fencing token with a tick-bounded lease; read_only_view (lib.rs:923) gives a non-owner a rehydrated, sequence-stamped view; and submit_write (lib.rs:942) admits a mutation only if the presenting node owns the lease, presents the current fencing token, and writes before the lease expires — otherwise it returns a precise NephthysSingleWriterError (LeaseOwnedByDifferentNode, StaleFencingToken, ExpiredAuthorityLease, ReadOnlyReplicaWrite). run_nephthys_single_writer_eval (lib.rs:1581) exercises two legitimate owners, three hostile writes (a read-only replica, a stale-token grant, an off-lease write), and an authority transfer (a new lease with a fresh token that fences the old owner's stale token). The gate (lib.rs:866) demands ≥ 2 accepted writes, ≥ 3 blocked writes, non-empty read-only views, and — the sharp invariant — that the ledger grew by exactly the accepted count, so a blocked write can never have leaked a side effect.

Byte-verified persistence and hosting-tier migrations#

NephthysPersistenceApi (lib.rs:1061) is the realm-facing surface — bootstrap a realm, mutate world objects, post a double-entry economy line, create a character, grant inventory, set property/vehicle/position. The smoke gate run_nephthys_persistence_api_smoke (lib.rs:1746) writes a full character, captures canonical_bytes(), restarts the writer onto a replacement node, and proves the post-restart bytes are exactly equal (NephthysByteVerification.exact_match) for both the character and realm aggregates — plus that residency tags survive on every property, vehicle, and ledger line. Two further evals prove the hosting-tier story end to end: run_listen_host_migration_eval (lib.rs:2505) migrates a peer-hosted Listen session to a successor host, using a fencing token to reject the stale host's write while byte-verifying zero character-state loss; and run_realm_promotion_eval (lib.rs:2685) promotes a Solo-authored realm to Dedicated then Meshed with a byte-identical composition lock file at every tier, migrating persistence into Nephthys at the Solo→non-Solo boundary (byte-verified). A realm grows from one author to a thousand inhabitants without a rebuild and without losing a save.

The trust seam Danu and Nephthys ride on#

Meshing only matters if a hostile operator cannot forge the writes it routes. Two real seams enforce that. On the wire, libs/v7/realm-protocol binds every cross-trust message to a platform-issued RealmSecurityToken (HMAC-SHA256 over the claims) and a sequence-numbered RealmSignedIntent; its RealmEventTamperEvalReport::passed (realm-protocol/src/lib.rs:1131) proves that only a valid signed client intent mutates authoritative state and that the post-hostile state hash is identical to the baseline — validation strictly precedes mutation. On the identity side, libs/v7/substrate-bridge is the one place V7 wires a reused persistence substrate: V7IdentityFirewall projects a platform principal into an opaque per-realm handle, and the integrated facade writes a Nàná character memory through the real V6 Egbe Ori event store (PartitionedPostgresOriEventStore, OriProjectionMaterializer, OperatorReadAuditStore) and reads it back through the audited operator-read path — proving the realm never sees a platform account id. Moremi itself consumes Nephthys not as a network service but as a signed-receipt abstraction: MoremiNephthysMutationReceipt is HMAC-signed and validate_moremi_nephthys_receipt rejects realm-mismatched, actor-mismatched, rule-mismatched, replayed, or unsigned authority mutations before they touch state.

Where it runs (and doesn't yet)#

The accurate mental model: Danu and Nephthys are a tested decision-and-truth core, not a deployed cluster. The handoff sequencer, the fencing-token broker, the rehydrate path, and the byte-verified API are production-grade Rust with golden gates; the daemons around them are health endpoints, and the evals are deterministic single-process fixtures. To become the runtime the monolith describes, three wires remain: bind the Nephthys store to durable Postgres+pgvector (the byte-canonical design already makes this a swap, not a rewrite), bind the Danu control plane to real sim-node membership and the Agones allocation path, and run the realm-protocol frames over the live V6 Egbe gateway. None of that is faked here — it is honestly absent, with the seams shaped to accept it.

Related#

• ../V7_ARCHITECTURE.md — the orientation hub and the full trust-boundary, determinism, and reuse model
• ./moremi-realm-server-and-netcode.md — the authoritative sim loop, prediction/reconciliation, and the realm wire that Danu's interest sets and Nephthys's writes flow through
• ./nana-data-model-and-pheme-voice.md — the character and economy aggregates Nephthys persists, and the proximity voice budget that gates the same 1000+ population target
• ./hosting-fleet-data-and-compliance.md — the Agones fleet, cost tiering, Spot reclaim, and residency machinery that surround the meshing and persistence cores