Status: Proposal / roadmap (planning artifact, not yet implemented) Date: 2026-06-20 Supersedes scope of: docs/proposals/OYA_DOMAIN_PROPOSAL.md (original vision) Builds on: DOMAINS/oya/architecture.md (current state of the domain) Owner: Oya domain Naming: Oya = the domain. "Oya Companion" = the product (working name; persona/voice configurable per user).

0. How to read this plan#

This is two tracks that share one foundation:

• Track 1 — Rust rewrite (where appropriate). Move Oya's performance-critical, real-time, and numerically-heavy code from TypeScript to Rust, using the repo's existing blessed bridging conventions. "Where appropriate" is the operative phrase — this is a surgical hot-path migration, not a wholesale rewrite. Orchestration, SDKs, services, and glue stay TypeScript.
• Track 2 — Companion product. Expand Oya from a library-only domain into a full product: a home/lab personal-assistant drone that follows you, perches & recharges (wireless charging / battery swap), talks with you, sees what you see, connects to home & cloud servers, runs SOTA LLM + robotics intelligence, acts as a hands-free engineering-lab co-pilot, and serves as a personal filmmaking director (maturing and tying in the director assistant that already exists in yemaya/remote-film-capture).

The foundation both tracks share: the Rust oya-engine crate workspace (Track 1) becomes the on-drone/edge runtime that the product (Track 2) is built on.

Everything below is grounded in what is actually in the repo today (verified file paths included), so the plan matches our real conventions rather than a generic robotics stack.

1. Product vision — the three faces of Oya Companion#

One airframe + one intelligence stack, presenting three "modes" that share perception, voice, navigation, and memory:

All three are the same agent brain with different tool-sets and personas. The differentiator vs. a phone assistant is embodiment: it has a body that moves, perches, and a camera that shares your point of view.

2. Current state (grounded)#

• Oya is library-only today. Six TypeScript packages under libs/oya/: @oya/core (101 source modules + 109 tests, 8,265-line barrel) plus five readiness-evaluator siblings (flight-control, mission-planning, safety, swarm-intelligence, telemetry). No apps/oya/ or services/oya/. No Rust.
• The hard algorithms already exist in TS and are real (stub-audited 2026-05-04, docs/releases/p2/oya-stub-audit.md): ORCA (swarm-collision-avoidance.ts), SLAM/VIO (visual-slam.ts, advanced-slam-vio.ts), sensor fusion (multi-sensor-fusion.ts), MAVLink v2 (mavlink-protocol.ts), trajectory optimization (cinematography-trajectory.ts). These are the prime Rust-rewrite candidates — and porting them is low-risk because we have a working reference + tests to validate against.
• Two known soft spots (honest gaps, not stubs by the repo's definition, but thin):
  • natural-language-control.ts is self-described as "a TypeScript abstraction layer — models NLP/LLM pipelines without actual ML inference." Real intelligence must bind to the actual LLM stack (svc-ai / IsisLLMClient).
  • yemaya-integration.ts's YemayaBridge.connect() is a synchronous state-machine placeholder — the real cross-domain wire (event bus / HTTP) is not implemented.
• A filmmaking director already exists in libs/yemaya/remote-film-capture/ (100+ modules): director-preference-learning.ts, voice-activated-ai-consultation.ts, human-ai-co-direction.ts, human-director-profile-system.ts, directing-style-evolution.ts, direction-effectiveness-scoring.ts, director-session-handoff-protocol.ts. It is mature for home-studio / multi-actor production, but not yet wired to aerial drone cinematography. Bridging it to Oya's cinematography-trajectory.ts / shot-planning.ts / multi-camera-coordination.ts is the "Director" face of the product.

3. Track 1 — Rust rewrite (where appropriate)#

3.1 The triage principle#

Per CLAUDE.md: "Engine kernels / GPU / physics / audio / network / WASM → Rust." Apply one test to each module: does it run in a hard-real-time loop, crunch numbers at high frequency, parse a hot wire protocol, or run on the drone's edge compute? If yes → Rust. If it orchestrates, adapts, exposes an SDK, or talks to a service → stays TypeScript.

3.2 Migration triage of the 101 @oya/core modules#

→ Rust (oya-engine crate workspace):

→ Stay TypeScript (orchestration / glue / services):

• All *-integration.ts adapters (aja-, aphrodite-, bellona-, isis-, lilith-, sophia-, yemaya-, gaussian-splatting-), sdk-core, drone-control-api, swarm-api, telemetry-api.
• regulatory-compliance, faa-bvlos-compliance, enterprise-platform, database-schema, data-access-layer, observability, health-check, flight-logging, post-flight-analysis.
• The five readiness evaluators (@oya/flight-control, etc.) — keep as TS CI gates (see §3.5).
• natural-language-control — rewrite as orchestration over the real LLM stack (not Rust, not abstraction): it becomes a thin TS planner that calls IsisLLMClient + tools and emits typed commands into oya-control.

Hardware SDK adapters (dji-sdk, hardware-platforms-2025, generic-hardware): stay TS where they wrap vendor REST/cloud SDKs; the MAVLink/PX4/ArduPilot paths move to Rust (oya-mavlink).

3.3 Crate workspace layout (mirrors libs/maya/engine-core)#

The repo's canonical pattern is a per-domain Cargo workspace (no root workspace), e.g. libs/maya/engine-core/Cargo.toml with member crates under crates/, wired into Nx via nx:run-commands. Oya follows it exactly:

libs/oya/engine/                      → project "oya-engine"
├── Cargo.toml                        # workspace root (members below)
├── rust-toolchain.toml               # pin toolchain (match neith/maya)
├── package.json                      # cargo:build / cargo:test / cargo:clippy
├── project.json                      # nx:run-commands → cargo, + build:wasm
└── crates/
    ├── oya-types/        oya-math/        oya-estimation/
    ├── oya-control/      oya-navigation/  oya-swarm/
    ├── oya-mavlink/      oya-perception/  oya-cinematography/
    ├── oya-dsp/
    ├── oya-node-bridge/   # #[napi] — pattern: libs/uzume/.../uzume-node-bridge
    ├── oya-wasm/          # #[wasm_bindgen] — pattern: uzume-control-surface-wasm
    └── oya-integration-tests/

Bridge conventions, copied verbatim from existing repo crates:

• Node (services): oya-node-bridge with crate-type = ["cdylib","rlib"], build.rs calling napi_build::setup(), #[napi] annotations, package.json build napi build --platform --dts index.d.ts → .node + index.d.ts. (Pattern: libs/uzume/protocol-engines/crates/uzume-node-bridge.)
• Browser (dashboard previz): oya-wasm built with wasm-pack build crates/oya-wasm --target web --out-dir dist/libs/oya/engine/wasm --release. (Pattern: uzume-control-surface-wasm.)
• On-drone / edge: crates compile native (aarch64 for Jetson/Qualcomm) — no bridge; Rust is the runtime there.
• Nx: each crate's build/test/lint/fmt target = nx:run-commands → cargo … --manifest-path … (pattern: @neith/core project.json).

3.4 Migration method (low-risk, test-anchored)#

For each crate, in order of value (estimation → control → mavlink → swarm → perception → cinematography):

1. Port with parity tests. The existing *.test.ts files encode known-correct values (the repo bans tests that only assert truthiness). Port each module to Rust and re-prove the same numeric assertions in Rust unit tests (cargo test), then add a differential test: feed identical inputs to TS and Rust, assert outputs match within tolerance.
2. Bridge + shadow. Expose via oya-node-bridge; run Rust in shadow mode behind the TS path in SITL, compare outputs before cutover.
3. Cut over, keep TS as oracle. TS implementation stays as the reference oracle in CI (differential tests) even after the Rust path ships.
4. Adversarial pass (per CLAUDE.md): grep the new Rust for stub indicators, read every public + delegated-to private fn, confirm tests would fail on hardcoded returns.

This means no flag day and no capability regression — the TS code is the safety net and the spec.

3.5 Validation: reuse the readiness evaluators as gates#

The five @oya/* readiness evaluators already score whether flight-control / mission / safety / swarm / telemetry configs are production-ready and emit typed issues. Wire them into CI as release gates for the Rust runtime, and add a new rust-premerge entry (pattern: .github/workflows/v2-rust-premerge.yml) covering libs/oya/engine/** with clippy -D warnings, cargo test, and the differential suite.

3.6 Optional further-out: a sovereign runtime#

If/when on-drone determinism demands it, oya-control can run atop a no_std or neith-runtime-style executor rather than tokio. Out of scope for v1; noted so the crate boundaries don't preclude it.

4. Track 2 — The Companion product#

4.1 Three-tier compute architecture#

┌─────────────────────────────────────────────────────────────────────┐
│  ON-DRONE EDGE  (Jetson Orin/Thor or Qualcomm QRB; aarch64)           │
│  oya-engine (native Rust): estimation • control • mavlink • swarm     │
│  perception (ONNX/TensorRT): VIO/SLAM, person+hand+object, depth, VLM-lite│
│  wake-word + VAD; safety watchdog; emergency-land FSM (hard real-time)│
│  Works fully OFFLINE for flight safety & follow-me.                   │
└───────────────▲───────────────────────────────────┬───────────────────┘
   secure link  │ MAVLink2 (signed) + WebRTC media   │ telemetry/events
                │                                     ▼
┌───────────────┴───────────────────────────────────────────────────────┐
│  HOME SERVER  (oya-edge-runtime: Rust binary on a home box/NUC/Pi5)    │
│  Low-latency control hub • video archive • local vector DB / RAG       │
│  Optional local LLM • dock/charging orchestration • offline-first       │
│  This is the "connect to servers in your home" tier.                   │
└───────────────▲───────────────────────────────────┬───────────────────┘
   mTLS / E2E   │                                     │ gRPC + event-bus
                ▼                                     ▼
┌───────────────────────────────────────────────────────────────────────┐
│  CLOUD  (apps/oya/* services in the Oshun monorepo)                     │
│  Heavy reasoning (Claude via IsisLLMClient) • fleet mgmt • model OTA    │
│  Long-term memory • director/yemaya bridge • web/mobile backends        │
└───────────────────────────────────────────────────────────────────────┘

Design rule: safety-critical autonomy is local (flight, collision, land); heavy cognition is cloud/home (LLM reasoning, RAG, director planning), with graceful degradation — losing the cloud must never drop the drone.

4.2 New monorepo structure (follows lilith/yemaya conventions)#

apps/oya/
├── svc-flight-gateway/   # owns the secure drone link; wraps oya-node-bridge
├── svc-perception/       # vision/VLM "see what you see"; pose, hands, objects
├── svc-assistant/        # the AGENT BRAIN: IsisLLMClient + tools + RAG + memory
│                          #   (lab co-pilot + companion personas live here)
├── svc-director/         # cinematography director: yemaya bridge + oya cine
├── svc-mission/          # missions, follow-me policy, energy-aware planning
├── svc-dock/             # perch / wireless-charge / battery-swap orchestration
├── svc-live-stream/      # WebRTC SFU (video + low-latency control/telemetry)
├── bff/                  # Fastify BFF (aggregation, response shaping)
├── web/                  # Next.js dashboard (live map, video, missions, privacy)
└── mobile/               # React Native companion + manual controller

libs/oya/  (TS, mirrors libs/lilith/*)
├── service-lib/ fastify-core/ event-publisher/ event-handlers/ sdk/ common/
└── engine/   →  the Rust crate workspace from §3.3

libs/oya-edge-runtime/    # Rust binary for the HOME SERVER tier (native)

libs/contracts/src/oya/   # zod schemas: telemetry, control, mission, dock, director

Reuse, don't rebuild: @oshun/event-bus (Redis pub/sub), @oshun/websocket (rooms, redis adapter), @oshun/storage (S3/local for video & logs), @oshun/database, @oshun/metrics. Per-domain DB isolation = oya database (pattern: @yemaya/database Prisma or the @lilith/service-lib pg pattern).

4.3 Intelligence stack#

natural-language-control.ts is refactored to be the planner seam: STT text → IsisLLMClient (with Oya tools) → typed commands → behavior layer → oya-control. The current "models pipelines without inference" abstraction is deleted in favor of real calls.

4.4 Follow-me & indoor autonomy#

• Localization: oya-estimation (VIO/SLAM, UWB optional) — GPS-denied indoor, the proposal's stated <15 cm target.
• Person tracking: oya-perception re-ID + subject-tracking.ts math (now Rust) with Kalman/occlusion recovery.
• Social navigation: human-aware-navigation.ts → oya-navigation for polite, non-startling motion; speed caps; keep-out around faces/hands.
• Obstacle avoidance: obstacle-avoidance + depth, 360° coverage.
• Safety FSM: emergency land, return-to-dock, watchdog — all on-drone, offline-capable.

4.5 Perch, wireless charging, battery swap — the SW/FW/HW seam#

This is physical engineering the monorepo can't manufacture. The plan is explicit about the boundary so we build the right software:

Monorepo owns (software/firmware-facing):

• svc-dock + oya-control perch routines: perch-target detection (vision), perch approach trajectory (precision terminal guidance), contact/latch confirmation, controlled spin-down.
• Charging state machine: detect dock, negotiate charging handshake (Qi/resonant or contact pads), monitor battery-state, resume/abort.
• Battery-swap orchestration: fly-to-swap-station, align, signal robotic swap dock, verify new pack, re-arm.
• Energy-aware mission planning: in svc-mission, return-to-dock on battery threshold; schedule perch/charge windows between tasks; pick nearest free dock.
• Protocols/contracts for dock ↔ drone (likely MQTT/gRPC over the home LAN).

Out of repo (physical, reference only): perch mechanism (electro-permanent magnet to a steel wall plate / gecko-adhesive / suction), Qi or resonant wireless-charging coils + UL-compliant charging electronics, a robotic battery-swap dock, ducted/caged props. The plan names candidate approaches and defines the firmware/protocol seam; it does not claim to fabricate hardware. Hardware is tracked as a parallel HW workstream with its own BOM/EE/ME owners.

4.6 Conversational lab / engineering co-pilot#

The hands-free assistant during soldering / IoT assembly. Built on §4.3 plus:

• Procedural task graphs: structured step models for common bench tasks (through-hole vs SMD soldering, reflow profiles, ESP32/STM32 bring-up, I²C/SPI bus debugging). Each step has prerequisites, expected visual state, and hazard notes.
• Egocentric visual grounding: VLM + oya-perception watch your hands and the board to answer "is this joint cold?", "did I bridge pins 3–4?", "is the iron tip oxidized?" — grounded in the live feed, not guessed.
• Datasheet RAG: point at a chip → OCR the part number → retrieve the datasheet (iris/knowledge) → answer pinout/voltage/timing questions hands-free.
• Collaborative problem-solving: the agent (Claude via IsisLLMClient) does multi-step reasoning, proposes next steps, and asks clarifying questions — a true co-thinker, not a command parser.
• Safety alerts: thermal cues (iron left on, hot air), fume/ventilation reminders, ESD warnings, "you're about to power a shorted board."
• Hands-free UX: wake-word + barge-in so you never touch a screen with solder on your fingers; the drone repositions for a better camera angle on request ("look at the underside").

4.7 Filmmaking director — mature it and tie it into Oya#

A coherent director assistant already exists for home-studio production in libs/yemaya/remote-film-capture/. It is not wired to drones. This face of the product closes that gap:

• Reuse as-is: director-preference-learning.ts (learns your directing style → calibrated profile), voice-activated-ai-consultation.ts (the "Iris" between-takes voice consultant), human-ai-co-direction.ts, direction-effectiveness-scoring.ts, directing-style-evolution.ts.
• Build the bridge: a new DroneDirector agent in svc-director that:
  1. takes a director intent (voice or shot-list) and the learned preference profile;
  2. generates a shot plan via shot-planning.ts (composition rules, shot sizes, angles, continuity);
  3. compiles a camera trajectory via cinematography-trajectory.ts (min-snap B-spline/Bezier — now oya-cinematography in Rust);
  4. coordinates multiple drones via multi-camera-coordination.ts (MAPF, coverage, genlock/timecode sync);
  5. emits waypoints + gimbal commands to oya-control.
• Make the integration real: replace yemaya-integration.ts's placeholder YemayaBridge.connect() with actual @oshun/event-bus + HTTP wiring so shot lists flow Yemaya → Oya and footage metadata + telemetry flow back (the module already enumerates exactly these sync surfaces; they just aren't wired).
• New capabilities to add: real-time aerial composition feedback (apply composition rules to the live camera feed and auto-nudge framing), and live director voice commands during flight (today consultation is async between takes).
• Feedback loop: each executed shot → director accept/reject → feeds director-preference-learning → the profile improves over time.

Result: "Oya, give me a slow 180° orbit of the bench at eye level, then push in on the board" becomes a planned, multi-drone-capable, style-aware aerial shot — and Yemaya's post pipeline (dailies-review, conform, timeline) ingests it automatically.

4.8 Connectivity & data#

• Drone ↔ home: MAVLink2 signed for control/telemetry; WebRTC (reuse svc-webrtc SFU pattern) for low-latency video + a command data-channel.
• Home ↔ cloud: gRPC + @oshun/event-bus; mTLS, per-home tenancy, E2E-encrypted media.
• IoT/dock: MQTT for the charging dock / swap station and any home-IoT the assistant controls.
• Offline-first: home server runs a local vector DB + optional local LLM so the assistant keeps working without internet; cloud is for heavy reasoning, fleet, and OTA.

5. Safety, privacy, security, compliance (non-negotiable)#

A flying, always-listening camera in someone's home is a high-trust product. This is a first-class workstream, not an afterthought.

• Flight safety: ducted/caged props mandatory indoors; speed caps (2–5 mph); room-boundary geofencing; human-aware keep-out; emergency-land + watchdog (on-drone, offline); prop-stop on contact. Validated by the @oya/safety readiness evaluator as a release gate.
• Privacy (critical): hardware recording indicator (LED + optional audio chime) that cannot be silently disabled; per-room no-fly / no-record zones; physical camera shutter / privacy perch pose; local-first data with explicit cloud opt-in; on-device face/PII redaction option; guest mode; retention limits; clear "what is recorded and where it goes" UX in the dashboard.
• Security: secure boot + signed firmware/OTA; mTLS everywhere; E2E media; threat-model the drone as a networked camera + actuator (rogue-command, geofence-bypass, exfiltration). Run the repo's /security-review on each service before ship.
• Compliance: indoor flight is largely unregulated by the FAA, but: Remote ID for any outdoor use; UL for batteries + wireless charging; FCC/CE for RF; GDPR/CCPA for the camera/audio data. regulatory-compliance.ts / faa-bvlos-compliance.ts (TS) own this.

6. Hardware reality & the SW/HW boundary#

The monorepo delivers software, firmware-facing protocols, and simulation — not a manufactured drone. To avoid fooling ourselves:

• What we can build & verify here: the full Rust/TS stack, SITL/HITL simulation, perception models, the assistant brain, the director, dashboards, and protocol/firmware seams.
• What needs a parallel HW workstream: airframe + ducted props, perch mechanism, wireless-charging electronics, battery-swap dock, the edge compute module selection (Jetson Orin/Thor vs Qualcomm), and integration/RF/thermal.
• Bring-up path: develop against PX4/ArduPilot SITL + Gazebo/AirSim first (the simulation modules + sitl-integration-framework.ts already exist), then a dev airframe (ModalAI Starling 2 / Crazyflie for swarm), then custom hardware. Note: heavy sim (UE/Isaac) runs on the Linux box per CLAUDE.md, not on the dev Mac.

7. Phased roadmap#

Each phase ends with a gate (the relevant readiness evaluator must pass + the adversarial stub scan + differential tests where applicable). Phases A–B are the shared foundation; C–I are the product.

Cross-cutting throughout: simulation (Gazebo/AirSim/Isaac on the Linux box), HITL, eval harnesses, and the new Rust rust-premerge CI gate.

8. Reuse map (build on what exists)#

Net: a large fraction of Track 2 is integration, not greenfield. The genuinely-new builds are: the Rust crate workspace, wake-word/barge-in, the embodied behavior/safety layer, the DroneDirector bridge, svc-dock, the lab task graphs, and the product surfaces — plus all the physical hardware.

9. Risks & open decisions (need your call)#

These genuinely fork the plan — flagged rather than assumed:

1. Hardware strategy: (a) integrate an existing dev platform first (PX4/ModalAI/Skydio-class) to move fast on software, vs (b) commit to a custom airframe + custom perch/charging from day one. Recommendation: (a) — software on SITL + dev airframe now, custom HW as a parallel workstream.
2. Rust extent: confirm "surgical hot-path migration" (estimation/control/ mavlink/swarm/perception/cinematography → Rust; everything else stays TS) vs a fuller rewrite. Recommendation: surgical, per §3.2.
3. Edge compute target: Jetson Orin/Thor (best perception, heavier/pricier) vs Qualcomm QRB (lighter, 5G) — drives oya-perception runtime choices.
4. On-device LLM: ship a small local model for offline reasoning, or require home-server/cloud for all cognition (flight safety stays local regardless)?
5. Perch mechanism: electro-permanent magnet (needs wall plates) vs gecko/suction (works on more surfaces, less reliable) — affects svc-dock and perch guidance.
6. Privacy posture: local-first default with opt-in cloud (recommended) vs cloud-default — shapes data architecture from the start.

10. Immediate next steps (first concrete slice)#

If approved, the lowest-risk high-value start is Phase A, crate 1–2:

1. Scaffold libs/oya/engine/ Cargo workspace + Nx wiring (copy @neith/core project.json pattern), with oya-types + oya-math.
2. Port math-utils.ts + coordinate-transforms.ts → oya-math, re-proving the existing test vectors in cargo test, plus a differential test harness (TS-vs-Rust) wired into CI.
3. Stand up oya-node-bridge and prove a single function (e.g. Vincenty distance) callable from a TS service, matching the TS result bit-for-bit within tolerance.

That establishes the entire Rust↔TS spine end-to-end on a tiny, verifiable surface before we scale it across the estimation/control crates.

Part II — The Home Robotics Hive (expansion)#

This part expands Oya from "a personal companion drone" into a heterogeneous in-home robotics hive: a fleet of role-specialized robots — floor-care rovers, fetch manipulators, scout micro-drones, hybrid wheel+fly units, companion/patrol units, pet and elder-care units, fixed ambient sensor nodes, and charging docks — that share one persistent semantic home map, one coordinate frame, one energy economy, and one tiered (edge / home-server / cloud) intelligence stack.

It was produced by a multi-agent SOTA research pass (14 parallel domain researchers + 4 architecture synthesizers + an adversarial completeness critic, ~1.65M tokens, web-grounded) and is benchmarked against real industry-leading 2025-2026 systems throughout. Every architecture claim names the deployed system or paper it derives from.

Five invariants carry through every section below (and reconcile with Part I):

1. Roll by default, fly only when the ground path is gone. Hover cost-of-transport is ~10–100× rolling for the same mass; flight is a budgeted, feasibility-gated resource, never a default. (This is the explicit answer to "flight is not always energy efficient.")
2. One shared map, specialized bodies. Every unit reads/writes a single persistent semantic home model rather than carrying a private map that fragments the fleet.
3. The agent decides what and where; deterministic Rust/C++ loops own how and all safety/privacy enforcement. No LLM/VLA sits in a hard real-time loop or holds safety authority. (Same posture as Part I's flight stack.)
4. Fail loud, never fabricate. When confidence is low or the network drops, units fall back to deterministic local control and surface uncertainty to a human — they never claim a result they did not compute. (This is the repo's own zero-stub doctrine, expressed in robot behavior.)
5. Survive offline. Every safety-critical loop (collision, balance, fall-escalation, meds) runs on edge/home-server with the cloud unreachable.

The companion drone of Part I is now one class — the scout micro-drone — within this hive. The Rust oya-engine workspace (Part I §3) is the shared real-time core; the apps/oya/* services (Part I §4.2) are the orchestration tier these new capabilities extend. The unified, authoritative roadmap is §14; Part I's §7 phase table is the flight-track view within it.

The four sections that follow (§11–§14) are the synthesized architecture; §15 converts the adversarial critic's findings into committed requirements.

11. Hive & modular hardware architecture#

Oya is not a single robot with attachments; it is a fleet of role-specialized bodies that share one persistent home model, one coordinate frame, and one energy economy. This section defines the robot-class taxonomy, the modular hardware contract that lets those bodies interoperate and reconfigure, and the locomotion-energy logic that governs the whole hive. Two principles run through every decision below, both grounded in the research corpus:

1. Roll by default, fly only when the ground path is gone. Hover cost-of-transport is roughly 10–100× rolling CoT for the same mass (CapsuleBot, ATMO, Duawlfin, the air-ground actuation work arXiv:2603.26687). Flight is a budgeted, transition-gated resource, never a default.
2. Share perception at the hive, specialize the body at the edge. Every unit reads and writes one persistent semantic home map (the Clio/Hydra + ConceptGraphs/HOV-SG lineage; Astro-style lifelong relocalization) rather than carrying a private map that fragments the fleet.

Robot-class taxonomy#

Each class is defined by the smallest viable body for its job. We deliberately avoid over-humanoiding: a Stretch-class Cartesian fetch arm and a Labrador-style "mobility-as-payload" rover cover the large majority of home tasks at a fraction of the cost, risk, and control burden of a balancing humanoid.

Fixed nodes and docks are first-class hive members, not passive furniture: they anchor drift on each floor (radar/UWB), provide relocalization fiducials, and report charger availability into the fleet scheduler exactly as Open-RMF treats doors and lifts.

Modular & composable hardware: the OyaLink contract#

A heterogeneous fleet only stays maintainable if mechanical, power, and data interfaces are standardized. We define OyaLink, a two-tier docking/payload standard that lets any payload mount on any compatible body and lets units dock to each other and to infrastructure.

Mechanical — two coupling tiers. Light, frequently-swapped payloads (treat launcher, camera head, mop module, scout battery) use a self-aligning electropermanent-magnet (EPM) quick-dock with a chamfered seat and magnet array that pulls out the last millimeters of misalignment (FreeBOT/SMORES-EP lineage; the Edinburgh single-interface EPM connector). EPM holds with zero standing power — critical for parked drones and docked modules that must not idle-drain. Load-bearing or overhead modules (a carry shelf, a manipulator wrist tool) additionally engage a form-fit steel self-locking latch (Schunk/ATI pattern) that stays locked through power loss — a magnet alone must never hold mass over a person or pet. Manipulator wrists expose an ISO 9409-1 bolt pattern so third-party and printed tools interoperate (Stretch quick-change wrist pattern).

Power — dual rail, hot-swappable. A universal USB-C PD 3.1 EPR rail (48 V / 240 W, GaN) carries payload power; a separate 48–58 V traction rail feeds drive. Every hot-swap port has soft-start inrush limiting, ideal-diode ORing, and a per-port e-fuse so a swap can't brown out the bus or arc the contacts. Charging contacts are redundant spring/pogo pins (>100k mate cycles) with magnetic self-centering and charge-confirmation telemetry — single dirty contacts are the classic silent-death failure (the Roomba/docking-failure mode). Docks support bidirectional unit-to-unit power so a docked unit can relay charge.

Data — one bus, hardware-synced. Gigabit Ethernet over the dock plus a hardware PPS/PTP sync line at every dock and node, because UWB TDoA and multi-sensor fusion need sub-millisecond time alignment to associate detections correctly. A CAN/I²C sideband handles hot-plug detect and self-describing payload registration: on dock, a module reports its mass, center-of-mass, geometry, power envelope, capabilities, and driver handle (the Spot GXP / Stretch EndOfArm pattern). The body then updates its dynamics model and the agent's action set live — so a fetch unit that mounts a heavier shelf re-tunes its tip limits before it moves, never after.

Hot-swappable payload modules therefore become the unit of capability: mop ↔ treat-launcher ↔ camera-head ↔ carry-shelf ↔ swap-battery all present the same OyaLink face, and the driver hot-loads with the hardware (a swapped tool swaps its kinematics and policy).

Unfolding and foldable mechanisms (compact stow)#

Stow density and deploy reliability are explicit hardware requirements:

• Foldable rotor arms (scout & hybrid): origami/Sarrus-linkage arms that fold flat into a dock bay and deploy via propeller thrust (actuator-free, ~0.3 s, Peregrine-Falcon/UZH-EPFL lineage), with a positive deployed-state latch and fold-detect sensing that refuses to arm until the airframe is locked open. Folds use dual-stiffness / tensegrity decoupling so an indoor collision folds the frame rather than stripping a servo (EPFL collision-resilient winged drones) — indoors, bumps are guaranteed.
• Unfolding arms (fetch & elder-care): a prismatic-lift + telescoping Cartesian arm retracts into the base footprint when stowed, giving floor-to-shelf vertical range with no IK singularities and intrinsic safety near people (Stretch-3 insight) — preferred over a folding revolute arm for the common fetch case.
• Perching (scout): a lightweight passive claw or gecko/suction pad lets a micro-drone hold an overwatch vantage motors-off on near-zero power, then take off to re-dock — turning "keep an eye on the back door" into a perch-overwatch mission instead of a battery-murdering hover.
• Inter-unit docking: EPM continuous-style self-alignment plus a decentralized assembled-ensemble controller (ModQuad pattern) lets units physically couple — a fetch arm stages an object onto a rover's shelf, or a depleted unit's task is handed off — treated as an auctioned pickup-and-delivery + cooperative-transport problem (Yi 2025; ContactHandover for the final release), with success verified before the giving unit lets go.

All reconfiguration is requested by the agent but executed by a deterministic, gravity-aware, stability-checked planner (Jellyfish-Pump-Algorithm class: structural balance enforced at every intermediate step, not just start and goal). The agent never commands magnet currents or latches directly.

Wheel-vs-fly energy decision logic#

Every locomotion-capable unit runs a deterministic mode-arbitration planner, sitting above the real-time controllers and below the LLM/VLA agent. For each candidate mode (roll · walk · perch · fly) it scores a cost over the persistent home map's per-edge traversability and energy annotations:

cost(mode, edge) = CoT_energy(mode, mass, terrain)   // calibrated power model per unit
                 + risk(mode, occupancy, proximity)   // people/pets → flight penalized hard
                 + noise(mode)                         // quiet hours, elder/pet acceptance
                 + time(mode)                          // urgency from the task
subject to:  battery_SoC − energy(mode,route) ≥ reserve_to_nearest_FREE_dock
             mode ∈ feasible(edge)                     // stairs/gap force fly; clutter forces fold

The planner picks the minimum-cost feasible mode, and the chosen mode becomes a command input to a stabilizing controller — never hand-animated open-loop. For expressive or legged/hybrid motion we use the command-conditioned RL pattern (Disney BD-X): a behavior layer authors intent, an RL policy guarantees balance. Concretely:

• Flight is feasibility-gated, not preference-gated: a hybrid unit flies only when the map shows no traversable ground route (uncrossable stairs, a gap, an elevated target) and the energy/risk budget allows. The multi-floor map is per-floor metric submaps joined by stair/elevator transition edges carrying per-edge energy cost (MuNES/HOV-SG), so the planner pre-decides mode route-by-route.
• Reserve-to-return is a hard constraint: Li-ion delivers near-constant power until it quits, so SoC margin is computed against the nearest free dock (factoring charger contention from the fleet scheduler), with conservative, temperature-compensated SoH estimation — bad SoC estimates are the root cause of mid-task strandings.
• Transitions are the highest-risk events (takeoff, landing, fold/unfold, stand-up). They are gated to known-safe transition pads at docks, with compliant landing and abort-to-hover-and-retry, plus a hard low-SoC ditch-to-nearest-pad behavior.
• Battery is telemetered into every task bid: the hive scheduler (Open-RMF-style auction) prefers the cheapest capable embodiment — a roller before a flyer, a scout drone only when no ground unit can reach — so energy economy is enforced at allocation time, not just locomotion time.

Compute and safety boundary. Real-time control, VIO/SLAM, obstacle avoidance, docking/latch loops, and the mode arbiter run on-robot (Orin/Thor-class) and must work offline; the agent only sets weights, priorities, missions, and the "flight is allowed here" budget. If the home server or cloud is unreachable, units fail safe — return to dock or hold a known coverage/perch routine — rather than stall or fabricate success. This keeps the hive's intelligence shared and its bodies modular, while ensuring no actuation, no fold, and no flight ever depends on a network hop.

12. Embodied intelligence & multi-agent architecture#

This section specifies Oya's intelligence stack: how computation is split across robot, home, and cloud; how the brain reasons and acts; how a persistent home digital twin serves as shared memory; how distributed perception delivers "eyes that come to you"; how a multi-agent fleet coordinates, allocates, and hands off work; and how skills are learned and gated for safety over the system's lifetime. Every layer is grounded in deployed systems and current research. The governing principle, drawn from the convergent design of Figure Helix, NVIDIA GR00T N1, and Physical Intelligence π0.5, is a dual-system split: slow semantic reasoning (System 2) decoupled from fast reactive control (System 1), partitioned across compute tiers so that what to do and how to move never compete for the same latency budget — and so that safety-critical loops never depend on a network hop.

1. The three-tier compute split: edge / home-server / cloud#

Oya's intelligence is partitioned by latency criticality and privacy sensitivity, not by convenience. This mirrors the explicit architecture of Gemini Robotics (an embodied-reasoning model plus an on-device VLA executor) and is the only design that survives the recurring failure mode of cloud-dependent home robots — the Moxie shutdown that bricked a fleet, and the Astro/Aldebaran cautionary tales.

The hard rule, stated across the floor-care, manipulation, fleet, and locomotion research alike: the LLM/VLA agent decides WHAT and WHERE; deterministic, verified controllers decide HOW. SLAM, local planning, motor control, and collision avoidance stay in a deterministic Rust/C++ loop on the edge. A non-deterministic model in the hard real-time loop blows the latency and safety budget — autoregressive VLA decoding is memory-bandwidth-bound and spikes well past control deadlines on Orin-class silicon. When the home server or cloud is unreachable, every robot fails closed to a cached plan and a local safe policy (a known coverage routine for a rover, safe-stop for a manipulator, ditch-to-nearest-pad for a drone) — never stalling, never fabricating success.

To keep cloud latency from poisoning the loop when offload is used, Oya adopts speculative edge-verifier decoding (a lightweight on-edge model filters token blocks before server validation, demonstrated for ~21% speedup on quadruped control) and Real-Time Chunking (RTC's freeze-and-inpaint: actions guaranteed to execute are frozen while the rest are inpainted by the generative model, eliminating chunk-boundary pauses and tolerating extreme inference delay — applicable to any flow/diffusion VLA with no retraining).

2. The brain: VLA + world model + LLM/VLM, in two systems#

Oya's brain is a layered VLA stack, not a monolithic policy.

System 2 — the deliberative reasoner (home server). An embodied-reasoning VLM in the lineage of Gemini Robotics-ER 1.5 and π0.5 ingests voice/natural-language goals, grounds them against the persistent home map and live perception, and decomposes them into language-level subtask sequences. Following π0.5's hierarchical inference, it predicts the next semantic subtask in language (e.g., "locate the mug → choose stand point → navigate → grasp → carry → deliver"), which then conditions the low-level action expert. It performs native digital tool-calling (home APIs, calendar, web) as ER 1.5 does, runs at 5–10 Hz, and is co-trained on hybrid examples — web VQA, object detection, semantic subtask labels, and low-level actions in one mixture — so internet-scale semantics transfer into physical control. π0.5's central empirical lesson governs our data strategy: open-world generalization scaled with the diversity of training homes (3 → 104), not parameter count. A fleet trained in one home collapses elsewhere; per-home fine-tuning from a teleop flywheel is mandatory for the long tail.

System 1 — the reactive action expert (on-robot edge). A flow-matching / diffusion action expert (π0 / GR00T N1 pattern) produces continuous action chunks at ~50 Hz, conditioned on the slow semantic latent from System 2 — exactly Helix's split of a 7–9 Hz onboard VLM driving a 200 Hz reactive visuomotor policy. The iterative denoise is the latency bottleneck (~80% of end-to-end on datacenter GPUs), so we distill toward few-/one-step experts and run RTC for smooth execution under real jitter.

Cross-embodiment base policy. Rather than maintaining N separate stacks for rovers, fetch arms, drones, and hybrids, Oya fine-tunes one base policy with thin embodiment-specific heads — the OpenVLA-OFT adaptation recipe (parallel decoding, continuous action heads, L1 regression), validated at fleet scale by Gemini's "Motion Transfer" (zero-shot control of ALOHA, bi-arm Franka, and Apollo from shared multi-embodiment data) and Skild's omni-bodied control. Improvements pool across the fleet; embodiment-specific action spaces are respected, because blindly sharing a policy across very different morphologies degrades performance.

The world model as planner and data engine. Oya maintains a learned generative world model alongside the geometric map, used three ways: (a) imagined rollouts for MPC-style "predict futures, pick the best action" planning (RFM-1, 1X World Model); (b) synthetic scenario generation of the actual home from a few captured images, cutting new-skill data collection from months to ~36 hours (GR00T-Dreams); and (c) closed-loop policy evaluation before any policy touches a real robot (Genie 3-class interactive environments, GAIA-2-class controllable rollouts for rare/safety-critical cases). Critically — world-model rollouts drift over long horizons (Genie holds minutes; driving world models struggle past short horizons), so their use is bounded and every rollout is validated against the real map. We never treat a generative rollout as ground truth.

For grasping under the VLA, Oya always carries a model-based 6-DoF fallback — AnyGrasp / Contact-GraspNet generating a ranked, geometry-grounded grasp distribution from wrist+head depth, gated by reachability and a graspability confidence score. The VLA alone never localizes a grasp on an unknown object; the model-based grasper gives a verifiable pose, and an eye-in-hand depth camera (RealSense D405-class) closes the loop on final approach so success doesn't hinge on perfect base localization.

3. The persistent semantic home map: the HIVE's shared memory#

The single most important architectural commitment is that the home map is a shared HIVE asset, not a per-robot map. Map fragmentation across a heterogeneous fleet — each robot in its own coordinate frame — is the failure that breaks everything downstream. Oya enforces one shared coordinate frame, consistent semantic instance IDs, and map-locking/merge discipline across rovers, drones, manipulators, fixed nodes, and docks.

The map is built in three layers:

1. Geometric layer (edge-produced). cuVSLAM stereo-visual-inertial odometry (<1% trajectory error) + nvblox GPU TSDF/ESDF voxel reconstruction feeding Nav2 costmaps, with dynamic-object/people masking so transients never poison the persistent layer, fused into an RTAB-Map-style multi-session pose graph with appearance-based loop closure and tiered STM/WM/LTM memory so lifelong operation stays compute-bounded. Floor rovers are the densest map producers (LiDAR-anchored ground truth); scout drones contribute thin top-down monocular-VIO observations.

2. Semantic scene-graph layer (server-canonical). A hierarchical, open-vocabulary 3D scene graph — building → floor → room → object — in the Hydra + ConceptGraphs / HOV-SG lineage, with per-instance CLIP/SigLIP embeddings lifted zero-shot from 2D foundation models (SAM / Grounding-DINO) via multi-view association, no 3D training required. Clio's Information-Bottleneck task-driven compression decides what each agent role remembers versus forgets, bounding fleet-wide memory and query latency over months of operation.

3. Photoreal twin layer (server/cloud only). Optional language-embedded Gaussian-splat reconstruction (LEGS / Spectacular-AI, with "GS on the Move" motion-blur/rolling-shutter compensation) for UX and VLM grounding — never run on rovers, since the embedded-SLAM evidence is clear that NeRF/3DGS are too heavy for real-time edge while semantic-geometric SLAM is viable. This is the Meta Hyperscape split: edge coarse / cloud heavy / streamed render.

The map is change-aware by design (DynaMem's add/remove voxel/instance memory plus LT-mapper version control), which is what lifts real-home mobile-manipulation success from 30% to 70%. Every object instance carries last_seen, observation count, and a confidence/decay so the twin reflects today's reality, not the first scan. Multi-floor structure is per-floor metric submaps joined by explicit stair/elevator transition edges (MuNES), with per-edge energy cost annotated so the planner can make the roll-vs-fly call. Every unit relocalizes into one shared frame on boot via VPS-style persistent anchors; docks and fixed sensor nodes carry AprilTag/UWB fiducials that bound drift and bootstrap relocalization in low-texture/low-light rooms where visual relocalization fails.

The map is the substrate the agent reasons over: the VLA grounds against object instances (IDs + 3D poses), not raw pixels, so "bring me the mug from the kitchen counter" resolves to a specific tracked instance with a known location and freshness. And it enforces the honest fail-loud seam mandated throughout: when relocalization confidence is low or last_seen is stale, the agent says "I last saw the keys on the hallway table 3 days ago, not sure they're still there" rather than fabricating a confident location.

4. "Eyes that come to you": distributed perception#

Perception is tiered and shared, decoupling where sensing happens from where intelligence lives:

• Always-on fixed nodes (cheap, non-imaging by default). 60 GHz FMCW mmWave radar SoCs (TI IWR6843AOP / Infineon BGT60) generating range-Doppler point clouds on-chip (privacy by physics — raw RF never leaves the sensor), PIR-gated to sip power (the trick that made battery mmWave like the Aqara FP300 viable), plus Wi-Fi CSI (802.11bf) presence from every AP and robot radio as a free whole-home layer. These cover the camera blind spots — bathrooms, bedrooms, darkness — and own static structure and calibrated anchors in the shared map.

• On-demand mobile eyes. When a fixed node or the agent decides a higher-confidence look is warranted, the nearest suitable mobile unit — a floor rover, or a scout micro-drone when the target is elevated or the ground path is blocked — navigates to the triggered zone carrying RGB-D + thermal. This is the literal "eyes that come to you" of Astro's go-investigate and Ring's fly-to-viewpoint, but with the intelligence living on the server, not the eye. The drone is a thin observer (Skydio-class VIO + learned obstacle avoidance on edge); the reasoning is elsewhere.

• Cooperative fusion, semantics over the wire. Following the Bonn smart-edge-sensor architecture, nodes transmit compact semantics — skeletons, point clusters, occupancy deltas — not pixels, fused late into the shared world model. Cross-modal fusion (HeCoFuse-style intermediate-feature fusion) lets a camera-only drone and a LiDAR rover jointly detect a person or obstacle, and is uncertainty- and pose-error-aware (COOPERTRIM / V2X-DGPE) so fusion degrades gracefully to onboard-only when an agent drops off the mesh. Crucially, the mobile robot is the noisiest node: its pose is down-weighted and ICP-aligned before integration, never trusted equal to a calibrated static anchor.

All of this rides a Zenoh peer-to-peer mesh over ROS 2, chosen for partition tolerance and lowest trajectory drift on flaky home Wi-Fi, exchanging compressed descriptors rather than raw point clouds to avoid saturating the network. UWB anchors at docks/nodes give ~10 cm localization for "go to where this person is" intents, with BLE Channel Sounding as a dense cheap fallback.

5. The multi-agent architecture: orchestrator, skill agents, specialists#

Oya is a genuine multi-agent system, not one robot with attachments. The architecture has three agent classes:

The fleet orchestrator agent (one, on the home server). This is the System-2 deliberative brain operating at fleet scale, structured on the Open-RMF fleet-adapter pattern: each robot class is a fleet exposing pose/battery/capabilities; the orchestrator bids tasks out and resolves space-time traffic in shared home spaces. It owns a RobotFleet/RoboOS-style shared declarative world state with two-way replanning. Its job is decomposition and allocation: turning "someone spilled in the kitchen" into a tasking for the nearest floor rover, optionally a scout drone to confirm, and a dirt-heatmap update — choosing which embodiment fits each step (drone to scout, rover to traverse, manipulator to grasp, hybrid to clear stairs) and sequencing handoffs. It is a planner/coordinator and skill-router — it orchestrates learned policies and VLA skills, it does not emit motor commands.

Per-robot skill agents (one per unit, on-robot edge). Each robot runs a small edge VLA as its System 1 — fast reactive control, obstacle avoidance, balance, grasp execution — plus wake-word and reflexive safety. This is what keeps the fleet working when the server is down and what guarantees the orchestrator never blocks a real-time loop.

Task-specialist agents (per domain). Cleaning, fetch, patrol, and care each carry domain-specific reasoning layered on the shared substrate:

• Cleaning runs adaptive boustrophedon cellular-decomposition coverage with Matrix-style multi-pass over high-dirt cells, closing the perception→actuation loop with real dirt sensing (acoustic Dirt Detect + dirty-water turbidity/DirtSense), and extends to multi-robot coverage redistribution (resilient-CPP) so two rovers split a house and reabsorb each other's cells on failure.
• Fetch decomposes find→navigate→grasp→deliver with manipulability-aware base placement (solving "navigable but unreachable") and tactile-gated grasping.
• Patrol runs learned-normalcy anomaly detection — fusing RGB + thermal + audio (glass-break, smoke/CO) against each home's routine, with low-false-positive tuning (the Knightscope SOC model) — never fixed motion alarms, which cry wolf and sink consumer products.
• Care runs the proactive engagement engine (ElliQ pattern: goals × availability × success-probability), the multimodal fall pipeline, and the closed-loop medication-adherence ledger.

Coordination, allocation, and handoff are concrete subsystems, not hand-waving:

• Task allocation is hybrid: auction/CBBA-style bidding for fast online assignment (capability- and battery-aware bids that encode hard capability constraints — never ask a drone to grasp or a rover to climb stairs — and coalition-form when no single robot suffices), augmented by a DeepFleet-style learned congestion forecaster once fleet logs exist, to pre-route around home choke points (doorways, halls, stairs).
• Traffic deconfliction uses MAPF (PIBT / LaCAM*) over a shared space-time traffic schedule, because naive independent navigation deadlocks past 2–3 units in a home's tight corridors.
• Coordination policies are trained with CTDE MARL (QMIX / MAPPO) in Isaac/Habitat sim — centralized training, decentralized execution — so the hive keeps coordinating on each robot's local observations when the server is down.
• Handoff is modeled explicitly as an auctioned pickup-and-delivery plus cooperative-manipulation problem (Yi 2025 pairing, decentralized-MPC transport, ContactHandover for the final transfer), with closed-loop success verification — the giving robot does not release until perception/tactile confirms the receiver has the object. Fabricated handoff success is dangerous for elder/pet care; the agent fails loud and escalates to a human when a transfer is uncertain.

A standing guardrail across all three classes: floor arbitration for conversation. An unarbitrated talking fleet talks over itself; a HIVE-level floor manager assigns one responder per human and silences the rest, fusing per-unit mic-array DoA into a shared "who is speaking, from where" estimate so the best-positioned unit orients and answers.

6. The skill library and lifelong learning#

Oya gets more capable over time through a teleop-to-autonomy data flywheel (the 1X NEO "Expert Mode" / Mobile-ALOHA pattern): failed autonomous tasks are completed by a human teleoperator (with operator-side face blurring and owner consent gating — see safety), every session is logged as a demonstration in Open X-Embodiment / LeRobot format, and policies are fine-tuned per-home from 50–100 demos. ACT-style action chunking with temporal ensembling suppresses the compounding error that sinks naive behavior cloning.

A lifelong skill library (LRLL pattern) distills recurring successful trajectories into named, reusable library skills the orchestrator can route to. To add home-specific skills without catastrophic forgetting of base competencies, we use experience replay, retrieval-based local adaptation, and per-morphology adapters rather than full sequential fine-tuning — and we exploit the empirical finding that pretrained VLAs are notably forgetting-resistant. Synthetic data from the home world model (GR00T-Dreams-style) bootstraps new situations; everything is validated in the Habitat 3.0 / HSSD-200 / OVMM sim harness as CI for the perception and policy stack before any hardware rollout. Sim-to-real fidelity hinges on accurate actuator modeling (delays, torque limits, friction) plus domain randomization — the discipline that let RAI's Spot pipeline transfer zero-shot at 5.2 m/s, and whose absence is the classic "works in sim, fails on hardware" trap.

7. The safety-gated learned-policy pattern#

This is the non-negotiable architectural invariant, stated identically across the safety, manipulation, locomotion, and foundation-model research: the learned policy is the high-performance, untrusted controller; an independent, higher-assurance safety supervisor is the verified authority that can veto it. No current foundation model offers certifiable hard safety, and VLM/LLM hallucination of objects or scenes causes real collisions — so we never trust the VLA alone.

Concretely, every actuation command from the agent/VLA passes through a safety governor running as an isolated, high-integrity process (separate from the lower-assurance autonomy/LLM stack, on a dual-channel safety-rated controller, Cat 3/PLd or SIL2):

• Speed-and-Separation Monitoring (R15.08 / ISO 10218-2): safety-rated lidar fields that scale slow/stop zones with velocity, for any moving base around people and pets.
• Power-and-Force-Limiting (ISO/TS 15066, now folded into ISO 10218-2:2025): per-body-region biomechanical force/pressure thresholds cap manipulator momentum on contact, enforced via current/torque sensing.
• Control-barrier-function / model-predictive shielding: forecasts violations and substitutes safe recoveries; CBF-RL constrains the learned policy at the boundary.
• OOD / hallucination filter: catches the VLA acting on a phantom instance, with multi-view confirmation required before any detection is persisted and Bayesian uncertainty-aware semantics so low-confidence detections never become "facts."
• Geofenced no-go AND no-record zones enforced in both the motion planner (refuses entry) and the perception pipeline (sensors disabled/blurred on entry), so a bathroom is simultaneously unreachable and unrecordable even if navigation is compromised.

The expressive/locomotion analog is the Disney BD-X command-conditioned RL pattern: an animation/behavior layer authors character intent, and an RL policy guarantees balance while tracking it — expression rides on top of a stabilizing policy, never replacing it. The agent proposes; the shield disposes. Out-of-bounds commands are routed to a discard-and-log path, not an E-stop spam path (itself a DoS vector flagged in the ROS 2 threat model). And the agent must treat its own perception inputs as an attack surface — a manipulated camera or voice input must never escalate privilege or bypass the supervisor.

The agent's role, finally, is bounded precisely: it is the conversational, reasoning, orchestration, and skill-routing brain — translating intent, grounding it in the shared map, allocating across embodiments, narrating and explaining its choices, and running the human-in-the-loop escalation tier. It is explicitly not a low-level controller, not a safety authority, and not permitted to fabricate a result it did not verify. When confidence is low, the network drops, or interventions aren't working, it does the one thing every layer of this architecture is built to do: fail loud, fall back to deterministic local control, and ask a human — rather than fake a success it never achieved.

13. Capability / product lines#

Oya ships as ten capability lines, each a concrete deliverable mapped onto the shared HIVE substrate (one persistent semantic home map, the three-tier edge/home-server/cloud intelligence split, the energy-aware "roll-by-default" locomotion policy, and the deterministic-safety-supervisor-over-untrusted-agent architecture). Robot classes are reused across lines — the floor rover, the fetch manipulator, the scout micro-drone, the hybrid wheel+fly unit, the companion/patrol unit, the elder-care unit, fixed sensor nodes, and charging docks — so a "line" is a software + skill + hardware bundle, not a separate product. Every line below specifies its robot class(es), the SOTA techniques adopted, the software subsystems and Rust/TS crates, the agent(s) involved, the hardware, and the headline SOTA bar to beat.

A cross-cutting architectural invariant governs all ten lines: the LLM/VLA agent decides what and where; deterministic, verified Rust/C++ control loops own how and all safety/privacy enforcement. No agent sits in a hard real-time loop, and no agent holds safety or privacy authority — a separate safety supervisor (SSM/PFL/E-stop, ISO 10218-2:2025 + ISO/TS 15066 force limits) and a privacy layer (on-edge redaction, hardware recording indicators) can veto it. Routine operation must survive cloud loss; the home server is the center of gravity.

Line 1 — Floor care (mop / sweep / vacuum)#

Line 2 — Fetching small objects#

Line 3 — Home patrol & security#

Line 4 — Companion + presence#

Line 5 — Pet interaction#

Line 6 — Elder-care & health monitoring#

Line 7 — Persistent home mapping#

Line 8 — Energy management & docking#

Line 9 — Ambient / smart-home integration#

Line 10 — Lab co-pilot & filmmaking director#

Shared dependencies across all ten lines. Every line reads and writes the single Line 7 persistent semantic map in one shared coordinate frame; every actuated line routes commands through the deterministic safety supervisor (Lines 1–6, 10) and every perception-bearing line through the privacy layer (on-edge redaction, hardware recording indicators in series with camera power, no-record zones enforced in both planner and perception). Lines 8 and 9 are infrastructure that the other eight consume — energy/docking and ambient sensing are not optional add-ons but the substrate that makes 24/7 autonomy and "eyes that come to you" coherent. The agent tier is consistent throughout: it sets intent, priorities, and missions; it never holds the real-time loop, safety authority, or privacy authority; and it fails loud — surfacing uncertainty and recruiting a human or teammate robot — rather than fabricating a result it did not compute.

14. Monorepo integration & roadmap additions#

This section maps the Oya home-robotics expansion onto the master plan's existing structure. The plan already establishes the oya-engine Rust crate workspace (oya-math, oya-estimation, oya-control, oya-mavlink, oya-swarm, oya-perception, oya-cinematography, plus napi/wasm bridges) and the apps/oya/* service tier (svc-flight-gateway, svc-perception, svc-assistant, svc-director, svc-mission, svc-dock, svc-live-stream, bff, web, mobile), following the lilith/yemaya conventions confirmed in the monorepo (libs/oya, apps/oya, contracts in libs/contracts, proto in libs/proto). The expansion from a cinematography-drone codebase to a heterogeneous in-home HIVE — floor-care rovers, fetch manipulators, scout micro-drones, hybrid wheel+fly units, companion/patrol/care units, fixed ambient nodes, and charging docks — is additive: it reuses the existing real-time Rust core and TypeScript orchestration split, and extends rather than forks the phased roadmap (A–I).

The governing architectural invariant, drawn directly from the SOTA corpus, is the three-tier intelligence split (on-robot edge / home-server / cloud) with a hard separation between the deterministic real-time control loop (Rust) and the LLM/VLA agent tier (TypeScript orchestration + served models). Every crate and service below respects that boundary: SLAM, control allocation, grasp execution, social-navigation, and energy/safety governors are deterministic and verifiable; the agent decides what and where, never how inside a hard-real-time loop. This is the convergent pattern across π0.5 hierarchical inference, Gemini Robotics ER↔VLA, Figure Helix (System-2 at 7–9 Hz conditioning System-1 at 200 Hz), and NVIDIA GR00T N1 — and it is also the master plan's existing posture for the flight stack.

New Rust crates (libs/oya/*)#

These extend the oya-engine workspace. Each ships Cargo.toml, comprehensive types, domain-correctness tests (e.g. boustrophedon-completeness assertions, ISO/TS 15066 force-limit checks against published thresholds, CoT model validation against measured power curves), and exposes a napi-rs binding where a TS service needs synchronous access plus a wasm-pack build where the web/mobile clients need it (matching the existing oya-engine/napi and /wasm convention).

oya-cinematography is retained unchanged; oya-perception is refactored to depend on oya-scenegraph/oya-mapping so the existing drone perception becomes one producer/consumer of the shared map rather than a private pipeline. oya-swarm is generalized by oya-fleet (the swarm-flight trajectory work becomes a strategy within the broader heterogeneous allocator).

New services (apps/oya/*)#

Each follows the lilith/yemaya service convention (project.json, package.json, tsconfig.json, path mapping in tsconfig.base.json, NestJS-style module with contracts from libs/contracts/oya). These are orchestration/IO/agent-tier services in TypeScript; all hot-path math lives in the crates above, reached via napi.

svc-assistant is elevated to the System-2 reasoning brain (Gemini-Robotics-ER-pattern): mission decomposition, embodied spatial reasoning over svc-home-map, tool calls, and delegation to the orchestrator — but it never emits motor commands and is always gated by oya-safety. svc-dock is generalized from a single drone dock to a dock-network controller feeding svc-energy. svc-flight-gateway and svc-live-stream carry over.

New contracts (libs/contracts/oya, libs/proto/oya)#

Typed boundaries are added so heterogeneous units interoperate and the agent's action surface is well-defined: FleetState/TaskBid/Allocation (capability + battery + payload constraints, the hard capability gate that prevents asking a drone to grasp); WorldModelQuery/SceneGraphInstance (object instance with pose, last_seen freshness, uncertainty — the honest fail-loud seam for stale map state); SensorObservation (compressed descriptors/features, never raw frames, with PTP timestamps); CapabilityManifest (self-describing payload/module registration — mass, CoM, power envelope, driver handle — updating dynamics and the agent action set on dock); ConsentPolicy/PrivacyZone (per-person consent, no-go/no-record geofences, retention windows); and SafetyEnvelope (force limits, SSM fields, e-stop state). These follow the existing OpenAPI + proto generation pipeline (libs/openapi, libs/proto).

New readiness-evaluator gates#

The master plan's readiness evaluator is extended with domain-correctness gates that fail loud rather than rubber-stamp — consistent with the project's zero-tolerance-for-stubs and fail-closed posture:

• Coverage completeness — oya-floorcare boustrophedon decomposition provably covers the free-space cell graph; multi-robot partition reabsorbs a failed unit's cells.
• Grasp honesty — svc-manipulation verifies contact via tactile/load feedback before reporting success and escalates below the confidence threshold; no fabricated grasp success (the explicit anti-stub case from the corpus).
• Map freshness — every served SceneGraphInstance carries non-fabricated last_seen/uncertainty; the agent surfaces staleness ("I last saw it 3 days ago") instead of a confident wrong location.
• Energy reserve — no mission dispatched without a verified reserve-to-return-to-a-free-dock budget; SOC estimates temperature-compensated, not voltage-only.
• Safety conformance — manipulator contact forces validated against ISO/TS 15066 per-body-region thresholds; mobile bases against R15.08 safety-field/SSM behavior; the safety governor demonstrably vetoes an out-of-envelope VLA command.
• Privacy enforcement — no-record zones disable/blur sensors in the perception path (not just navigation); recording indicators are hardware-tamper-evident; only derived semantics leave the home tier by default.
• Offline degradation — every safety-critical loop (collision avoidance, balance, fall-escalation, meds) runs on edge/home-server with the cloud unreachable (the Moxie/Cutii anti-orphaning lesson).
• Mode-arbitration economy — oya-locomotion chooses rolling over flight whenever a traversable ground path exists in the map, validated against the per-mode CoT model.

Roadmap extension (phases A–I)#

The expansion threads through the existing phases rather than appending a parallel track; flight (the current A–I content) is treated as one capability within the broader HIVE.

• Phase A (Foundations) — add oya-mapping, oya-scenegraph, oya-comms, oya-energy, oya-safety crates and the libs/contracts/oya typed boundaries; stand up svc-home-map and the shared coordinate frame. This is the substrate everything else reads/writes.
• Phase B (Estimation/Perception) — extend oya-perception onto the shared map; add oya-ambient fixed-node fusion and svc-ambient with Matter/Thread integration.
• Phase C (Control) — add oya-locomotion (mode arbitration + morphing control allocation) and oya-manipulation; wire oya-safety as the mandatory gate on all actuation.
• Phase D (Mission/Director) — generalize oya-swarm→oya-fleet; ship svc-fleet-orchestrator (Open-RMF pattern) and elevate svc-assistant to the System-2 brain.
• Phase E (Dock/Energy) — generalize svc-dock to the dock network; ship svc-energy and the PRCP/opportunistic-charging scheduler.
• Phase F (Domain services) — svc-floorcare, svc-manipulation, svc-pet, svc-eldercare, svc-hri, each behind its readiness gates.
• Phase G (HRI/Live) — full-duplex svc-hri, social navigation, and the supervised-autonomy/teleop console; svc-live-stream carries over.
• Phase H (Learning flywheel) — teleop→autonomy data pipeline, per-home fine-tuning (LoRA adapters), federated learning with differential privacy, and the Habitat/Isaac sim harness as CI for the perception and fleet-fusion stacks.
• Phase I (Hardening) — ETSI EN 303 645 / EU CRA conformance (SBOM, signed A/B OTA, per-device identity, CVD policy), ISO 13482 / UL 3300 certification paths per unit class, and the adversarial safety/privacy verification pass across the whole fleet.

The net effect is a coherent extension: the existing Rust-real-time-core / TypeScript-orchestration division, the three-tier compute split, and the phased delivery model all carry forward, while the new crates and services give the HIVE its shared persistent map, heterogeneous fleet coordination, energy economy, ambient sensing, and the safety/privacy guarantees that in-home deployment makes non-negotiable.

15. Committed requirements from the adversarial critique#

The completeness critic surfaced 19 gaps and 8 high-value additions. None are deferred — each becomes a binding requirement on the crates/services/gates above. Grouped by theme; [C] = critical, [I] = important.

15.1 Multi-floor reach, doors, and the outside boundary#

• [C] Stairs are a first-class locomotion mode, not "just fly." At least the fetch and floor-care classes get a stair-capable mode (Dreame-X50 ProLeap retractable-leg lineage for thresholds/short flights; Unitree-B2-W-class wheeled-legged for full flights) or a per-floor docked sub-fleet with a dedicated stair-transit carrier. The multi-floor map (oya-mapping) encodes which units can cross which transition edge; the mode planner models a real stair-climb cost, never assuming a meds/water-carrying rover can fly.
• [C] Closed doors and articulated objects. Add an articulated-object manipulation skill (door / drawer / cabinet / fridge via VLA + force-compliant arm + handle/affordance detection) to oya-manipulation. For passive passage, the svc-ambient Matter controller can request a smart-lock / door-opener, with doors modeled as shared resources in the svc-fleet-orchestrator traffic schedule. High-consequence actions (unlocking) require a second authorization factor (see 15.4).
• [I] Outdoor / threshold scope is decided explicitly. A weather-capable patrol/scout unit handles porch/yard/garage with GPS↔VIO handoff at the threshold, IP-rated hardware, doorsill/garage-door transition skill, and package detection/retrieval. If a deployment is indoor-only, the front-door and garage are hard map edges — stated, not silently unhandled.

15.2 In-home physical safety (people, children, pets, fluids, payloads, noise)#

• [C] Children are a distinct hazard class (not lumped with pets): treat- launcher/small-part choking interlocks when a child is detected, manipulator/drone lockout or ultra-conservative speed near unsupervised children, tip-over/ride-on resistance, child-height sensor coverage, and UL-3300 "external manipulation" robustness (a grabbed/blocked robot yields safely, never torques through). Detection validated on small/fast child morphologies.
• [I] Water/fluid & spill safety. A shared wet-floor/drying map layer that elder-care and patrol units treat as a soft no-go until dry; mopping gated by occupancy (never wet a path a tracked person/pet is on or approaching); multi-modal carpet/low-pile-rug detection (not height-only); leak detection with fail-closed water cutoff; routing away from detected floor outlets/cords.
• [I] Payload securement & drop/spill safety. Lidded/gimbaled/spill-resistant carriers for liquids and meds; acceleration limits while carrying spillable or hazardous payloads; never carry hot/hazardous payloads over a person or down stairs without securement; continuous payload-retention verification; "dropped payload" is a logged safety event with escalation.
• [I] Pet edge cases harden the social-navigation layer. Never-corner / always-leave-an-escape-route as a hard constraint; chew- resistant, no-exposed-cable hardware with electrical safety if chewed; unknown small/exotic animal → treated as hard-avoid; pet waste/vomit is a fleet-wide hard-avoid zone (not just a cleaning decision); no drone flight near an agitated/predatory pet.
• [I] Noise is a first-class scheduler input. A household acoustic/occupancy schedule with per-zone, per-time-of-day dB caps; loud tasks (vacuum, prop flight) deferred to unoccupied/awake windows; a low-noise night-patrol mode (radar/CSI/thermal, rolling slowly, no vacuum/props) distinct from day cleaning, tied to the elder-care sleep/routine model.

15.3 Resilience (the home server, OTA, and degraded modes)#

• [C] The home server is not a single point of failure. UPS-backed always-on appliance; a hot-standby or edge-elected backup brain (a docked Thor-class unit assumes orchestration + escalation if the server is unreachable); the life-safety path (fall detection → escalation) runs fully on-edge with cellular egress and never routes through the server; brain OTA is forbidden while a resident is unattended. CTDE MARL execution is genuinely server-independent for safety tasks.
• [I] Live-OTA safety in an occupied home. Never update a unit mid-task or while it is the sole unattended safety asset; canary/staged fleet rollout (never all units at once); shadow-mode/sim-harness validation (Habitat/Isaac + readiness gates) before any fine-tuned/federated policy controls a real robot near people; automatic rollback on success-rate regression; a known-good policy kept resident for instant fallback.

15.4 People: authorization, accessibility, consent#

• [I] Per-resident authorization & command arbitration. A role model (admin / adult / child / guest) gates which capabilities each person can invoke; floor-arbitration extends to command-arbitration (priority + conflict resolution for simultaneous conflicting orders); explicit child lockouts on manipulators and drone launch; guest mode with reduced capability and recording defaults — bound to the on-device biometric resident registry.
• [I] Accessibility for the actual care users. Large-button/physical controls and an opt-in pendant as alternatives to app/voice; hearing-impaired modes (visual/haptic alerts, on-screen captioning); dysarthria-robust ASR and slower-paced dialogue; dementia-appropriate interaction (human-in-loop care.coach pattern); and fall/gait/pose models explicitly validated on wheelchair/walker users and frail bodies (a walker is never classified as an obstacle to corner).
• [I] Bystander/visitor consent. On-device face/voice handling with unrecognized-person redaction by default; a visible "recording / remote- operator-active" indicator; guest mode minimizing capture; audio-recording consent gating compliant with two-party-consent jurisdictions; audit logs of who/what was recorded; mandatory operator-side blurring + owner-consent gate for any teleop session.

15.5 Security of the expanded attack surface#

• [C] Threat model extends to the smart-home control plane. A compromised brain must not be able to unlock doors without a separate authorization factor / physical confirmation; UWB/sensor anti-spoofing via cross-modal consistency (a hard oya-comms/oya-ambient requirement); Sidewalk/3rd-party backhaul treated as untrusted egress with explicit consent; and prompt- injection defenses made concrete — manipulated camera/voice inputs can never escalate agent privilege or trigger high-consequence actuation without the deterministic supervisor plus a confirmation gate.

15.6 Honesty, scope, and viability#

• [I] Honest fetch capability boundaries. Real-home OVMM is ~58–70% success; per-object-class hard confidence gating at the task-promise level. For medication specifically, prefer mobility-as-payload + a fixed verified dispenser over autonomous grasp of loose pills; define in-scope vs always-teleop vs out-of-scope object classes; address floor-pickup tip-over and high-shelf reach; specify teleop availability/SLA and its privacy model.
• [I] Filmmaking director gets its own capability spec (not folded into a generic VLA bundle). Cinematographic shot-type library (dolly, orbit, crane, reveal), composition/aesthetic scoring, multi-unit coordinated filming with angle handoff, subject-tracking-for-framing distinct from collision avoidance, gimbal/exposure/focus control, and a director-LLM translating creative intent into shot sequences — reusing oya-cinematography and the existing yemaya/remote-film-capture director (Part I §4.7).
• [I] Map cold-start & onboarding. Guided first-map walkthrough (or autonomous explore), room/zone labeling UX, dock/anchor placement guidance with UWB geometric-dilution feedback, and a "major change detected — re-survey?" flow that distinguishes renovation/move from daily change; graceful first-boot relocalization for a unit with no map yet.
• [I] Consumables & serviceability lifecycle. Per-unit consumable levels (water/detergent/treats/bags) and wear tracking (brushes, props, mop pads, tactile gels, dock contacts) with loud low-consumable/service alerts (never silent failure), design-for-easy-service, and a repairability/right-to-repair stance aligned with EU CRA supported-lifetime obligations.
• [I] Economic & regulatory viability is a readiness gate. A per-line BOM and a phased go-to-market shipping the cheapest viable subset first (floor-care + ambient + companion are nearer-term than humanoid-class fetch); a shared- hardware/shared-compute strategy to avoid Thor-per-unit cost; an explicit product-liability and insurance posture for physical-harm scenarios; and a certification roadmap sequenced by unit class (ISO 13482 / UL 3300 / ISO 10218 / CRA), never gating the whole fleet on the hardest cert. "Will this pencil out at achievable volume?" is itself a gate.

15.7 Adopted additional capabilities (the critic's 8 ideas, accepted)#

• Whole-home hazard monitoring as its own product capability: a queryable household-state model (stove on/off, fridge ajar, faucet running, window/door open at night) fused from ambient nodes + mobile eyes — answers "did I leave the stove on?" and proactively flags gas/flood/open-door hazards. (Arguably the highest-value safety feature; the sensing primitives already exist in oya-ambient.)
• Grid-aware energy + fleet-as-backup-power: schedule charging/heavy tasks in cheap/clean grid windows (Matter 1.5 tariff/carbon signals) and use the docked fleet's UPS-backed batteries as opportunistic home backup power during outages.
• Capability passport + simulation-first onboarding: every new module/unit self-registers a CapabilityManifest and ships a validated sim model so the fleet rehearses integrating it in the home digital twin (Genie/GAIA/Isaac) before it ever moves physically.
• Cross-home federated "home archetype" priors (with differential privacy) so a freshly-installed fleet starts competent and personalizes faster — leveraging the π0.5 home-diversity result and the federated subsystem.
• Anti-bricking dignity guarantee for care/companion units: a documented end-of-life / vendor-survival plan (self-hostable brain, data export, no bricking) and anti-manipulative-engagement guardrails — the Moxie/Cutii/ Aldebaran cautionary tales turned into a marketed trust feature.
• Unified "find my things" across the fleet: change-aware scene-graph last_seen per instance + UWB/BLE tags + on-demand mobile eyes → "where are my keys/glasses/phone/the cat," with honest freshness/uncertainty.
• Proactive fall prevention (not just detection): gait-speed/sit-to-stand deterioration biomarkers + floor trip-hazard detection proactively clear/flag hazards and surface decline trends to caregivers — where the clinical value is.
• Home-wide acoustic-event service: unify the per-line mic arrays + DoA + danger-sound classifiers into one shared sense (glass break, smoke/CO alarm, running water, a call for help, a baby crying, a dog in distress) that triggers the nearest mobile eye.

This document is a plan, not an implementation. No code has been written or modified. Part II (the home-robotics hive) was produced by a web-grounded multi-agent SOTA research workflow and benchmarked against named 2025-2026 industry systems; its architecture is a proposal. Hardware claims are scoped to the software/firmware/protocol boundary; the physical robots, perch/charging, and dock mechanisms require a parallel engineering workstream.