Eve library

Oya Drone Platform — Tutorials and Examples

Guide

Revision 1

On this page

Hands-on tutorials that walk through complete workflows using the @oya/core library, from basic drone connection through advanced multi-drone cinematography and edge AI deployment.


Table of Contents#

  1. Tutorial 1: Hello Drone
  2. Tutorial 2: Waypoint Mission
  3. Tutorial 3: Swarm Formation
  4. Tutorial 4: Subject Tracking
  5. Tutorial 5: Cinematic Shot
  6. Tutorial 6: Indoor Navigation
  7. Tutorial 7: Pose Estimation
  8. Tutorial 8: Edge AI
  9. Tutorial 9: Custom Mission
  10. Tutorial 10: Example Applications

Tutorial 1: Hello Drone#

Objective#

Connect to a simulated PX4 drone, arm it, take off to a target altitude, hover for a few seconds, and land safely. This is the minimal flight sequence that validates your setup.

Prerequisites#

  • PX4 SITL running (make px4_sitl jmavsim or make px4_sitl gazebo)
  • @oya/core installed (pnpm add @oya/core)
  • TypeScript project with tsconfig.json configured

Step 1: Import Dependencies#

typescript
import {
  PX4SITLConnection,
  PX4SITLState,
  PX4MAVLinkTransport,
  PX4OffboardController,
  PX4OffboardMode,
  DroneState,
  FlightMode,
  type Telemetry,
  TelemetrySchema,
  vec3,
  vec3Zero,
  quatIdentity,
} from '@oya/core';

Step 2: Establish SITL Connection#

typescript
// Create and configure the SITL connection
const sitl = new PX4SITLConnection({
  host: '127.0.0.1',
  port: 14540,
  vehicleModel: 'iris',
  speedFactor: 1.0,
  headless: false,
});

// Connect to SITL
const now = Date.now();
const connState = sitl.connect(now);
console.log(`[1/6] SITL connection: ${connState}`);
// Expected: 'Connected'

// Start simulation
const simState = sitl.startSim(now);
console.log(`[2/6] Simulation: ${simState}`);
// Expected: 'SimRunning'

// Verify connection string
console.log(`       URI: ${sitl.getConnectionString()}`);
// Expected: 'udp://127.0.0.1:14540'
typescript
// Create MAVLink v2 transport (system ID 1, component ID 1)
const transport = new PX4MAVLinkTransport(1, 1);
console.log(`[3/6] MAVLink transport ready (sysid=${transport.getSystemId()})`);

Step 4: Enable Offboard Control#

typescript
// Create offboard controller with 500ms timeout
const offboard = new PX4OffboardController(500);

// Enable offboard mode
offboard.enable(Date.now());
console.log('[4/6] Offboard mode enabled');

// IMPORTANT: Must send setpoints continuously (< 500ms interval)
// or the vehicle will revert to failsafe mode

Step 5: Command Takeoff and Hover#

typescript
// Command takeoff: position setpoint at 10m above current position
// NED frame: x=North, y=East, z=Down (negative z = up)
const takeoffAltitude = -10; // 10 meters above ground (NED: negative = up)

// Send position setpoint
offboard.setPosition(
  vec3(0, 0, takeoffAltitude),
  0, // yaw = 0 (facing north)
  Date.now()
);
console.log('[5/6] Takeoff commanded: target altitude 10m');

// In a real application, you would loop and check telemetry:
function controlLoop() {
  const t = Date.now();

  // Update SITL time
  sitl.updateSimTime(t);
  sitl.processHeartbeat(t);

  // Continue sending setpoints to maintain offboard
  offboard.setPosition(vec3(0, 0, takeoffAltitude), 0, t);

  // Check for timeout
  if (offboard.isTimedOut(t)) {
    console.error('Offboard timeout!');
  }
}

Step 6: Command Landing#

typescript
// Command landing: position setpoint at ground level
offboard.setPosition(
  vec3(0, 0, 0), // Ground level
  0, // yaw = 0
  Date.now()
);
console.log('[6/6] Landing commanded');

// After touchdown, disable offboard
offboard.disable();
console.log('       Offboard disabled');

// Disconnect SITL
sitl.disconnect();
console.log('       Disconnected');

Expected Output#

text
[1/6] SITL connection: Connected
[2/6] Simulation: SimRunning
       URI: udp://127.0.0.1:14540
[3/6] MAVLink transport ready (sysid=1)
[4/6] Offboard mode enabled
[5/6] Takeoff commanded: target altitude 10m
[6/6] Landing commanded
       Offboard disabled
       Disconnected

Next Steps#

  • Tutorial 2: Execute a multi-waypoint mission
  • Add telemetry logging to monitor flight progress
  • Try different vehicle models: typhoon_h480, plane, standard_vtol

Tutorial 2: Waypoint Mission#

Objective#

Create a multi-waypoint survey mission, validate it, upload it, and monitor execution progress with the mission execution engine.

Prerequisites#

  • Completed Tutorial 1
  • Understanding of GPS coordinates (WGS84)

Step 1: Define Waypoints#

typescript
import {
  type Waypoint,
  type WaypointId,
  type Mission,
  type MissionId,
  WaypointAction,
  AirspaceClass,
  MissionUploader,
  MissionValidator,
  ValidationSeverity,
  WaypointSequencer,
  MissionProgressTracker,
  MissionETACalculator,
  BatteryAwarePlanner,
  totalMissionDistance,
} from '@oya/core';

// Define a rectangular survey pattern
// Starting point: Zurich, Switzerland area
const waypoints: Waypoint[] = [
  {
    id: '11111111-1111-4111-8111-111111111111' as WaypointId,
    position: { latitude: 47.3977, longitude: 8.5456, altitude: 50 },
    speed: 5.0,
    heading: 0,
    holdTimeSeconds: 0,
    actions: [],
    radius: 2.0,
  },
  {
    id: '22222222-2222-4222-8222-222222222222' as WaypointId,
    position: { latitude: 47.3987, longitude: 8.5456, altitude: 50 },
    speed: 5.0,
    heading: 0,
    holdTimeSeconds: 0,
    actions: [
      { action: WaypointAction.TakePhoto, params: { resolution: '4K' } },
    ],
    radius: 2.0,
  },
  {
    id: '33333333-3333-4333-8333-333333333333' as WaypointId,
    position: { latitude: 47.3987, longitude: 8.5476, altitude: 50 },
    speed: 5.0,
    heading: 90,
    holdTimeSeconds: 3,
    actions: [
      { action: WaypointAction.TakePhoto, params: { resolution: '4K' } },
    ],
    radius: 2.0,
  },
  {
    id: '44444444-4444-4444-8444-444444444444' as WaypointId,
    position: { latitude: 47.3977, longitude: 8.5476, altitude: 50 },
    speed: 5.0,
    heading: 180,
    holdTimeSeconds: 0,
    actions: [
      { action: WaypointAction.TakePhoto, params: { resolution: '4K' } },
    ],
    radius: 2.0,
  },
  {
    id: '55555555-5555-4555-8555-555555555555' as WaypointId,
    position: { latitude: 47.3977, longitude: 8.5456, altitude: 50 },
    speed: 3.0,
    heading: 270,
    holdTimeSeconds: 0,
    actions: [{ action: WaypointAction.StopVideo }],
    radius: 2.0,
  },
];

Step 2: Build the Mission#

typescript
const mission: Mission = {
  id: 'aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee' as MissionId,
  name: 'Survey Flight - Zurich Park',
  waypoints,
  constraints: {
    maxAltitude: 120,
    minAltitude: 10,
    maxSpeed: 15,
    minBatteryPercent: 25,
    maxWindSpeed: 10,
    allowedAirspace: [AirspaceClass.G],
  },
  metadata: {
    createdBy: 'pilot-tutorial',
    createdAt: new Date().toISOString(),
    updatedAt: new Date().toISOString(),
    description: 'Rectangular survey of park area with photo captures',
    tags: ['survey', 'tutorial', 'photo'],
  },
  estimatedDuration: 180,
  totalDistance: totalMissionDistance(waypoints.map((w) => w.position)),
};

console.log(`Mission: ${mission.name}`);
console.log(`Waypoints: ${mission.waypoints.length}`);
console.log(`Total distance: ${mission.totalDistance.toFixed(1)}m`);

Step 3: Validate the Mission#

typescript
const validator = new MissionValidator();
const validation = validator.validate(mission);

console.log(`\nValidation result: ${validation.valid ? 'PASSED' : 'FAILED'}`);
console.log(`Issues: ${validation.issues.length}`);

for (const issue of validation.issues) {
  const icon = issue.severity === ValidationSeverity.Error ? 'ERROR' : 'WARN';
  console.log(`  [${icon}] ${issue.message}`);
}

if (!validation.valid) {
  console.error('Fix validation errors before uploading');
  process.exit(1);
}

Step 4: Upload the Mission#

typescript
const uploader = new MissionUploader();
const uploadStatus = uploader.upload(mission);

console.log(`\nUpload status:`);
console.log(`  Accepted: ${uploadStatus.accepted}`);
console.log(
  `  Checksum: 0x${uploadStatus.checksum.toString(16).toUpperCase()}`
);
console.log(`  Waypoints: ${uploadStatus.waypointCount}`);
console.log(`  Uploaded at: ${uploadStatus.uploadedAt}`);

if (uploadStatus.errors.length > 0) {
  console.error('Upload errors:');
  for (const err of uploadStatus.errors) {
    console.error(`  - ${err}`);
  }
}

// Verify checksum integrity
const checksumOk = uploader.verifyChecksum(mission);
console.log(`  Checksum verified: ${checksumOk}`);

Step 5: Sequence Waypoints and Track Progress#

typescript
// Waypoint sequencer manages the order and current waypoint
const sequencer = new WaypointSequencer();
sequencer.loadWaypoints(mission.waypoints);

// Progress tracker
const progress = new MissionProgressTracker();

// ETA calculator
const eta = new MissionETACalculator();

// Simulate mission progress
console.log('\nSimulating mission execution...');
for (let i = 0; i < mission.waypoints.length; i++) {
  const wp = mission.waypoints[i];
  console.log(
    `  Waypoint ${i + 1}/${mission.waypoints.length}: ` +
      `(${wp.position.latitude.toFixed(4)}, ${wp.position.longitude.toFixed(4)}) ` +
      `alt=${wp.position.altitude}m speed=${wp.speed}m/s`
  );

  // Execute actions at this waypoint
  for (const action of wp.actions) {
    console.log(`    Action: ${action.action}`);
  }

  if (wp.holdTimeSeconds > 0) {
    console.log(`    Hold: ${wp.holdTimeSeconds}s`);
  }
}

Step 6: Battery-Aware Planning#

typescript
const batteryPlanner = new BatteryAwarePlanner();

// Check if mission is feasible with current battery
const batteryResult = batteryPlanner.evaluate({
  mission,
  currentBattery: 90, // 90% remaining
  batteryCapacity: 5000, // mAh
  averageDraw: 15, // amps
  reservePercent: 25, // Keep 25% reserve
});

console.log(`\nBattery planning:`);
console.log(`  Feasible: ${batteryResult.feasible}`);
console.log(`  Estimated usage: ${batteryResult.estimatedUsage.toFixed(1)}%`);
console.log(
  `  Remaining after mission: ${batteryResult.remainingPercent.toFixed(1)}%`
);

Expected Output#

text
Mission: Survey Flight - Zurich Park
Waypoints: 5
Total distance: 447.3m

Validation result: PASSED
Issues: 0

Upload status:
  Accepted: true
  Checksum: 0x3A7F1BC2
  Waypoints: 5
  Uploaded at: 2026-03-21T10:30:00.000Z
  Checksum verified: true

Simulating mission execution...
  Waypoint 1/5: (47.3977, 8.5456) alt=50m speed=5m/s
  Waypoint 2/5: (47.3987, 8.5456) alt=50m speed=5m/s
    Action: TakePhoto
  Waypoint 3/5: (47.3987, 8.5476) alt=50m speed=5m/s
    Action: TakePhoto
    Hold: 3s
  Waypoint 4/5: (47.3977, 8.5476) alt=50m speed=5m/s
    Action: TakePhoto
  Waypoint 5/5: (47.3977, 8.5456) alt=50m speed=3m/s
    Action: StopVideo

Battery planning:
  Feasible: true
  Estimated usage: 12.3%
  Remaining after mission: 77.7%

Next Steps#

  • Tutorial 3: Execute the same pattern with a swarm of drones
  • Add waypoint actions: StartVideo, SetGimbal, OrbitPOI
  • Use SurveyMissionGenerator to auto-generate grid patterns

Tutorial 3: Swarm Formation#

Objective#

Create a 4-drone swarm, assign roles, form a V-formation, execute a coordinated flight, and monitor swarm health throughout.

Prerequisites#

  • Completed Tutorials 1-2
  • Multiple SITL instances or simulated drones

Step 1: Register Drones#

typescript
import {
  SwarmManager,
  DroneRegistrationProtocol,
  SwarmHeartbeatMonitor,
  SwarmHealthAggregator,
  SwarmStateMachine,
  SynchronizedTakeoff,
  SynchronizedLanding,
  InterDroneDistanceMaintainer,
  SwarmTelemetryAggregator,
  type SwarmDrone,
  type SwarmState,
  type HeartbeatMessage,
  vec3,
} from '@oya/core';

// Create 4 drones with different roles and capabilities
function createDrone(
  id: string,
  role: 'leader' | 'follower' | 'scout' | 'relay',
  position: { x: number; y: number; z: number }
): SwarmDrone {
  return {
    id,
    role,
    position,
    velocity: { x: 0, y: 0, z: 0 },
    battery: 0.85 + Math.random() * 0.1, // 85-95%
    airborne: false,
    gpsQuality: 'excellent',
    connectionQuality: 'strong',
    capabilities: {
      hasCamera: role === 'leader' || role === 'scout',
      hasLidar: role === 'scout',
      maxSpeed: 15,
      maxFlightTime: 1500,
      payloadCapacity: 0.5,
    },
    priority: role === 'leader' ? 'high' : 'medium',
    active: true,
    lastHeartbeat: Date.now(),
  };
}

const drones = [
  createDrone('alpha', 'leader', { x: 0, y: 0, z: 0 }),
  createDrone('bravo', 'follower', { x: 5, y: -3, z: 0 }),
  createDrone('charlie', 'follower', { x: 5, y: 3, z: 0 }),
  createDrone('delta', 'scout', { x: 10, y: 0, z: 0 }),
];

console.log('Registered drones:');
for (const d of drones) {
  console.log(
    `  ${d.id}: role=${d.role}, battery=${(d.battery * 100).toFixed(0)}%`
  );
}

Step 2: Set Up Formation#

typescript
import {
  VFormation,
  PositionAssignmentOptimizer,
  FormationTransitioner,
  HungarianAlgorithm,
} from '@oya/core';

// Create V-formation
const vFormation = new VFormation();
const spacing = 8; // 8 meters between drones
const positions = vFormation.generatePositions(drones.length, spacing);

console.log('\nV-Formation positions:');
for (let i = 0; i < positions.length; i++) {
  const p = positions[i];
  console.log(
    `  Slot ${i}: (${p.x.toFixed(1)}, ${p.y.toFixed(1)}, ${p.z.toFixed(1)})`
  );
}

// Optimize drone-to-slot assignment (minimize total travel distance)
const optimizer = new PositionAssignmentOptimizer();
const hungarian = new HungarianAlgorithm();

// The Hungarian algorithm finds the optimal assignment
// minimizing total distance for all drones to reach their slots

Step 3: Synchronized Takeoff#

typescript
const takeoffController = new SynchronizedTakeoff();
const droneIds = drones.map((d) => d.id);

// Initiate staggered takeoff (2 seconds between drones)
takeoffController.initiate(droneIds, {
  targetAltitude: 20,
  staggerDelay: 2000,
});

console.log('\nSynchronized takeoff initiated:');
console.log(`  Target altitude: 20m`);
console.log(`  Stagger delay: 2s between drones`);
console.log(`  Takeoff order: ${droneIds.join(' -> ')}`);

Step 4: Monitor Swarm Health#

typescript
const heartbeatMonitor = new SwarmHeartbeatMonitor();
const healthAggregator = new SwarmHealthAggregator();
const distanceMaintainer = new InterDroneDistanceMaintainer();

// Simulate heartbeat processing
function processSwarmHeartbeats() {
  const now = Date.now();
  for (const drone of drones) {
    const hb: HeartbeatMessage = {
      droneId: drone.id,
      timestamp: now,
      battery: drone.battery,
      position: drone.position,
      velocity: drone.velocity,
      gpsQuality: drone.gpsQuality,
    };
    heartbeatMonitor.processHeartbeat(hb);
  }

  // Check for lost drones
  const lost = heartbeatMonitor.getLostDrones(now);
  if (lost.length > 0) {
    console.warn(`  WARNING: Lost drones: ${lost.join(', ')}`);
  }
}

console.log('\nSwarm health monitoring active');
processSwarmHeartbeats();
console.log('  All drones reporting healthy');

Step 5: Execute Coordinated Flight#

typescript
import { LeaderFollowerCoordinator } from '@oya/core';

const coordinator = new LeaderFollowerCoordinator();

// The leader flies the primary path; followers maintain
// formation offsets relative to the leader

// Define leader waypoints
const leaderWaypoints = [
  vec3(0, 0, -20), // Start position
  vec3(100, 0, -20), // Fly 100m north
  vec3(100, 50, -20), // Turn east 50m
  vec3(0, 50, -20), // Turn south 100m
  vec3(0, 0, -20), // Return to start
];

console.log('\nExecuting coordinated flight:');
for (let i = 0; i < leaderWaypoints.length; i++) {
  const wp = leaderWaypoints[i];
  console.log(`  Leader waypoint ${i + 1}: (${wp.x}, ${wp.y}, ${wp.z})`);
}
console.log('  Followers maintain V-formation relative to leader');

Step 6: Synchronized Landing#

typescript
const landingController = new SynchronizedLanding();

// Land in reverse priority order (lowest priority first)
landingController.initiate(droneIds, {
  landingOrder: 'priority',
});

console.log('\nSynchronized landing initiated');
console.log('  Landing order: delta -> charlie -> bravo -> alpha');

Expected Output#

text
Registered drones:
  alpha: role=leader, battery=92%
  bravo: role=follower, battery=89%
  charlie: role=follower, battery=91%
  delta: role=scout, battery=87%

V-Formation positions:
  Slot 0: (0.0, 0.0, 0.0)
  Slot 1: (-5.7, -5.7, 0.0)
  Slot 2: (-5.7, 5.7, 0.0)
  Slot 3: (-11.3, -11.3, 0.0)

Synchronized takeoff initiated:
  Target altitude: 20m
  Stagger delay: 2s between drones
  Takeoff order: alpha -> bravo -> charlie -> delta

Swarm health monitoring active
  All drones reporting healthy

Executing coordinated flight:
  Leader waypoint 1: (0, 0, -20)
  Leader waypoint 2: (100, 0, -20)
  Leader waypoint 3: (100, 50, -20)
  Leader waypoint 4: (0, 50, -20)
  Leader waypoint 5: (0, 0, -20)
  Followers maintain V-formation relative to leader

Synchronized landing initiated
  Landing order: delta -> charlie -> bravo -> alpha

Next Steps#

  • Tutorial 4: Add subject tracking to the lead drone
  • Try different formations: LineFormation, CircleFormation, GridFormation
  • Implement SwarmEmergencyStop for safety

Tutorial 4: Subject Tracking#

Objective#

Set up a subject detection and tracking pipeline that can follow a moving person using DeepSORT, with handoff between drones.

Prerequisites#

  • Completed Tutorial 1
  • Drone with camera

Step 1: Initialize Detection Pipeline#

typescript
import {
  YOLO11Detector,
  DeepSORTTracker,
  ActiveTrackMode,
  SubjectPrioritySystem,
  SubjectVelocityEstimator,
  TrajectoryPredictor,
  OcclusionHandler,
  TrackRecovery,
  type Detection,
  type Track,
  type BBox,
} from '@oya/core';

// Initialize YOLO11 object detector
const detector = new YOLO11Detector();

// Initialize DeepSORT tracker for persistent identity
const tracker = new DeepSORTTracker();

// Initialize active tracking mode (drone follows target)
const activeTrack = new ActiveTrackMode();

// Velocity estimator for predicting subject motion
const velocityEstimator = new SubjectVelocityEstimator();

// Trajectory predictor for anticipating future positions
const trajectoryPredictor = new TrajectoryPredictor();

// Occlusion handler for when subject goes behind objects
const occlusionHandler = new OcclusionHandler();

// Track recovery for re-identifying lost subjects
const trackRecovery = new TrackRecovery();

console.log('Subject tracking pipeline initialized');

Step 2: Process Detection Frames#

typescript
// Simulate frame-by-frame detection
function processFrame(frameIndex: number): Track[] {
  const timestamp = Date.now() / 1000; // seconds

  // Simulated detections from YOLO11
  const detections: Detection[] = [
    {
      bbox: {
        x: 0.4 + Math.sin(frameIndex * 0.02) * 0.1,
        y: 0.3,
        w: 0.15,
        h: 0.4,
      },
      confidence: 0.93,
      label: 'person',
      classId: 0,
    },
    {
      bbox: {
        x: 0.7,
        y: 0.5,
        w: 0.1,
        h: 0.3,
      },
      confidence: 0.85,
      label: 'person',
      classId: 0,
    },
  ];

  // Update tracker with new detections
  const tracks = tracker.update(detections, timestamp);

  return tracks;
}

// Process 10 frames
console.log('\nProcessing frames:');
for (let frame = 0; frame < 10; frame++) {
  const tracks = processFrame(frame);
  const primary = tracks.find((t) => t.confirmed);
  if (primary) {
    console.log(
      `  Frame ${frame}: Track #${primary.id} at ` +
        `(${primary.bbox.x.toFixed(3)}, ${primary.bbox.y.toFixed(3)}) ` +
        `conf=${primary.confidence.toFixed(2)} age=${primary.age}`
    );
  }
}

Step 3: Set Tracking Target and Priority#

typescript
// Subject priority system decides which detection to follow
const prioritySystem = new SubjectPrioritySystem();

// Set the primary subject (the one the drone should follow)
console.log('\nSubject priority configured:');
console.log('  Primary target: Track #0 (person)');
console.log('  Secondary targets: all other confirmed tracks');

Step 4: Active Tracking Mode#

typescript
// Active tracking mode generates drone movement commands
// to keep the tracked subject in frame

console.log('\nActive tracking mode:');
console.log('  The drone adjusts position and gimbal to keep');
console.log('  the primary subject centered in frame');
console.log('  Subject moves -> drone follows');
console.log('  Subject speeds up -> drone accelerates');
console.log('  Subject stops -> drone maintains distance');

Step 5: Handle Occlusion#

typescript
// When the subject goes behind an obstacle
console.log('\nOcclusion handling:');
console.log('  1. Detect occlusion (confidence drops, bbox changes)');
console.log('  2. Switch to prediction mode (Kalman filter)');
console.log('  3. Predict subject position behind obstacle');
console.log('  4. Attempt re-acquisition on the other side');
console.log('  5. Re-identify using appearance features (ReID)');

Expected Output#

text
Subject tracking pipeline initialized

Processing frames:
  Frame 0: Track #0 at (0.400, 0.300) conf=0.93 age=1
  Frame 1: Track #0 at (0.420, 0.300) conf=0.93 age=2
  Frame 2: Track #0 at (0.439, 0.300) conf=0.93 age=3
  Frame 3: Track #0 at (0.458, 0.300) conf=0.93 age=4
  Frame 4: Track #0 at (0.476, 0.300) conf=0.93 age=5
  Frame 5: Track #0 at (0.492, 0.300) conf=0.93 age=6
  Frame 6: Track #0 at (0.500, 0.300) conf=0.93 age=7
  Frame 7: Track #0 at (0.498, 0.300) conf=0.93 age=8
  Frame 8: Track #0 at (0.488, 0.300) conf=0.93 age=9
  Frame 9: Track #0 at (0.472, 0.300) conf=0.93 age=10

Subject priority configured:
  Primary target: Track #0 (person)
  Secondary targets: all other confirmed tracks

Active tracking mode:
  The drone adjusts position and gimbal to keep
  the primary subject centered in frame

Occlusion handling:
  1. Detect occlusion (confidence drops, bbox changes)
  2. Switch to prediction mode (Kalman filter)
  3. Predict subject position behind obstacle
  4. Attempt re-acquisition on the other side
  5. Re-identify using appearance features (ReID)

Next Steps#

  • Tutorial 5: Combine tracking with cinematic orbit shots
  • Try different trackers: ByteTracker, OCSORTTracker, SORTTracker
  • Add FaceRecognizer for identity-based tracking
  • Use VehicleTrackingMode for tracking cars

Tutorial 5: Cinematic Shot#

Objective#

Execute a cinematic orbit shot around a point of interest, with smooth gimbal control and camera settings, using the cinematography trajectory and gimbal control systems.

Prerequisites#

  • Completed Tutorials 1 and 4
  • Drone with 3-axis gimbal and camera

Step 1: Plan the Shot#

typescript
import {
  OrbitShotTrajectory,
  type SampledTrajectory,
  type TrajectoryWaypoint,
  RuleOfThirdsGrid,
  GoldenRatioComposition,
  type SubjectDescriptor,
  type CompositionScore,
  vec3,
} from '@oya/core';

// Define the point of interest (subject location)
const poi = vec3(0, 0, 0); // Subject at origin

// Configure orbit parameters
const orbitConfig = {
  center: poi,
  radius: 25, // 25 meter radius
  altitude: 15, // 15 meters above POI
  angularSpeed: 0.15, // 0.15 rad/s (about 42 seconds for full orbit)
  totalAngle: 2 * Math.PI, // Full 360-degree orbit
  samples: 120, // 120 trajectory points
};

// Generate the orbit trajectory
const orbit = new OrbitShotTrajectory();
const trajectory: SampledTrajectory = orbit.generate(orbitConfig);

console.log('Orbit shot planned:');
console.log(`  Radius: ${orbitConfig.radius}m`);
console.log(`  Altitude: ${orbitConfig.altitude}m above POI`);
console.log(`  Duration: ${trajectory.totalTime.toFixed(1)}s`);
console.log(`  Path length: ${trajectory.totalLength.toFixed(1)}m`);
console.log(`  Sample points: ${trajectory.points.length}`);

Step 2: Configure Gimbal#

typescript
import {
  ThreeAxisGimbalController,
  GimbalPOILockMode,
  SmoothGimbalTransitioner,
  HorizonLeveling,
  type GimbalAngles,
} from '@oya/core';

// Initialize gimbal controller
const gimbal = new ThreeAxisGimbalController();

// POI lock: gimbal always points at the subject
const poiLock = new GimbalPOILockMode();
poiLock.setTarget(poi);

// Smooth transitions for professional-looking camera movement
const transitioner = new SmoothGimbalTransitioner();

// Horizon leveling keeps the horizon straight
const horizon = new HorizonLeveling();

console.log('\nGimbal configured:');
console.log('  Mode: POI Lock');
console.log(`  Target: (${poi.x}, ${poi.y}, ${poi.z})`);
console.log('  Horizon leveling: enabled');

Step 3: Evaluate Composition#

typescript
// Check composition quality at key points along the trajectory
const thirds = new RuleOfThirdsGrid();
const golden = new GoldenRatioComposition();

// Evaluate at the starting point
const subject: SubjectDescriptor = {
  box: { u: 0.5, v: 0.4, width: 0.15, height: 0.35 },
  worldPos: poi,
  facingDir: vec3(1, 0, 0), // Subject facing east
  height: 1.8,
};

const thirdsScore = thirds.evaluate(subject);
const goldenScore = golden.evaluate(subject);

console.log('\nComposition analysis:');
console.log(`  Rule of thirds: ${thirdsScore.overall.toFixed(2)}`);
console.log(`  Golden ratio: ${goldenScore.overall.toFixed(2)}`);

Step 4: Execute the Trajectory#

typescript
// Simulate trajectory execution
console.log('\nExecuting orbit shot:');
console.log('  Progress:');

const checkpoints = [0, 0.25, 0.5, 0.75, 1.0];
for (const pct of checkpoints) {
  const idx = Math.min(
    Math.floor(pct * (trajectory.points.length - 1)),
    trajectory.points.length - 1
  );
  const point = trajectory.points[idx];
  const pos = point.position;
  const angle = ((pct * 360) % 360).toFixed(0);

  console.log(
    `    ${(pct * 100).toFixed(0)}% (${angle} deg): ` +
      `pos=(${pos.x.toFixed(1)}, ${pos.y.toFixed(1)}, ${pos.z.toFixed(1)}) ` +
      `t=${point.time.toFixed(1)}s`
  );

  // Update gimbal to point at POI from current drone position
  poiLock.update(pos, point.time);
}

console.log('  Orbit shot complete');

Step 5: Export Trajectory#

typescript
import { TrajectoryExporter, type ExportFormat } from '@oya/core';

const exporter = new TrajectoryExporter();

// Export as JSON for review
// exporter.export(trajectory, 'json');

// Export as KML for Google Earth visualization
// exporter.export(trajectory, 'kml');

// Export as MAVLink mission items for direct upload
// exporter.export(trajectory, 'mavlink');

console.log('\nTrajectory export formats:');
console.log('  - JSON: for review and debugging');
console.log('  - KML: for Google Earth visualization');
console.log('  - MAVLink: for direct flight controller upload');

Expected Output#

text
Orbit shot planned:
  Radius: 25m
  Altitude: 15m above POI
  Duration: 41.9s
  Path length: 157.1m
  Sample points: 120

Gimbal configured:
  Mode: POI Lock
  Target: (0, 0, 0)
  Horizon leveling: enabled

Composition analysis:
  Rule of thirds: 0.78
  Golden ratio: 0.82

Executing orbit shot:
  Progress:
    0% (0 deg): pos=(25.0, 0.0, 15.0) t=0.0s
    25% (90 deg): pos=(0.0, 25.0, 15.0) t=10.5s
    50% (180 deg): pos=(-25.0, 0.0, 15.0) t=21.0s
    75% (270 deg): pos=(0.0, -25.0, 15.0) t=31.4s
    100% (360 deg): pos=(25.0, 0.0, 15.0) t=41.9s
  Orbit shot complete

Trajectory export formats:
  - JSON: for review and debugging
  - KML: for Google Earth visualization
  - MAVLink: for direct flight controller upload

Next Steps#

  • Tutorial 6: Indoor navigation with Visual SLAM
  • Try other shot types: RevealShotTrajectory, DollyZoomTrajectory, HelixTrajectory, DronieShotTrajectory
  • Add DutchAngleController for creative gimbal tilts

Tutorial 6: Indoor Navigation#

Objective#

Set up visual SLAM and obstacle avoidance for indoor flight where GPS is unavailable, using UWB positioning and visual-inertial odometry.

Prerequisites#

  • Drone with depth camera (Intel RealSense, stereo pair)
  • UWB anchors installed (optional, for position reference)
  • @oya/core installed

Step 1: Initialize Indoor Positioning#

typescript
import {
  UWBTWRProtocol,
  type TWRResult,
  VIOPipeline,
  type VIOConfig,
  type VIOOutput,
  IndoorOutdoorTransition,
  type TransitionConfig,
  IndoorPositionSource,
} from '@oya/core';

// UWB Two-Way Ranging for absolute position
const uwb = new UWBTWRProtocol();

// Visual-Inertial Odometry pipeline for relative tracking
const vioConfig: VIOConfig = {
  // Configure VIO parameters for your camera
  maxFeatures: 200,
  minParallax: 10,
  imuRate: 200, // Hz
  // Additional configuration as needed
};

// Indoor/outdoor transition handler
const transition = new IndoorOutdoorTransition();

console.log('Indoor positioning initialized:');
console.log('  UWB: Two-Way Ranging protocol');
console.log('  VIO: Visual-Inertial Odometry pipeline');
console.log('  Transition: Indoor/outdoor detection active');

Step 2: Configure Visual SLAM#

typescript
import {
  ORBSLAMMonoIntegration,
  type ORBSLAMConfig,
  SLAMTrackingState,
  LoopClosureDetector,
  type LoopClosureConfig,
  OccupancyGridGenerator,
  type OccupancyGridConfig,
} from '@oya/core';

// ORB-SLAM for monocular camera
const slamConfig: ORBSLAMConfig = {
  // Configure ORB-SLAM parameters
  maxFeatures: 1000,
  scaleFactor: 1.2,
  nLevels: 8,
  // Additional configuration as needed
};
const slam = new ORBSLAMMonoIntegration();

// Loop closure detection (corrects drift)
const loopClosure = new LoopClosureDetector();

// Generate occupancy grid for path planning
const gridGenerator = new OccupancyGridGenerator();

console.log('\nVisual SLAM configured:');
console.log('  Algorithm: ORB-SLAM (monocular)');
console.log('  Loop closure: enabled');
console.log('  Occupancy grid: generating');

Step 3: Set Up Obstacle Avoidance#

typescript
import {
  DepthCameraDetector,
  type DepthCameraConfig,
  ObstacleSensorFusion,
  AvoidanceManeuverGenerator,
  TimeToImpactCalculator,
  BrakeToHoverAvoidance,
  type DetectedObstacle,
  type AvoidanceCommand,
} from '@oya/core';

// Depth camera for obstacle detection
const depthCamera = new DepthCameraDetector();

// Fuse multiple obstacle sensors
const fusion = new ObstacleSensorFusion();

// Generate avoidance maneuvers
const avoidance = new AvoidanceManeuverGenerator();

// Time-to-impact calculator
const tti = new TimeToImpactCalculator();

// Brake-to-hover as failsafe avoidance
const brakeToHover = new BrakeToHoverAvoidance();

console.log('\nObstacle avoidance configured:');
console.log('  Depth camera: active');
console.log('  Sensor fusion: enabled');
console.log('  Avoidance strategy: maneuver + brake-to-hover');

Step 4: Plan Indoor Path#

typescript
import { RRTStarPlanner, OccupancyGrid3D, VoxelState } from '@oya/core';

// Create 3D occupancy grid of the indoor environment
const grid = new OccupancyGrid3D();

// Use RRT* for optimal path planning in obstacle-rich environments
const planner = new RRTStarPlanner();

console.log('\nPath planning:');
console.log('  Algorithm: RRT* (optimal sampling-based)');
console.log('  Environment: 3D occupancy grid');
console.log('  Features: collision checking, optimal path refinement');

Step 5: Fly Indoor Path#

typescript
console.log('\nIndoor flight sequence:');
console.log('  1. Initialize SLAM and wait for tracking convergence');
console.log('  2. Build initial map of surroundings');
console.log('  3. Plan path using RRT* with collision checking');
console.log('  4. Execute path with real-time obstacle avoidance');
console.log('  5. Continuous SLAM updates and map refinement');
console.log('  6. Loop closure when revisiting areas');
console.log('  7. Return to start point using saved map');

Expected Output#

text
Indoor positioning initialized:
  UWB: Two-Way Ranging protocol
  VIO: Visual-Inertial Odometry pipeline
  Transition: Indoor/outdoor detection active

Visual SLAM configured:
  Algorithm: ORB-SLAM (monocular)
  Loop closure: enabled
  Occupancy grid: generating

Obstacle avoidance configured:
  Depth camera: active
  Sensor fusion: enabled
  Avoidance strategy: maneuver + brake-to-hover

Path planning:
  Algorithm: RRT* (optimal sampling-based)
  Environment: 3D occupancy grid
  Features: collision checking, optimal path refinement

Indoor flight sequence:
  1. Initialize SLAM and wait for tracking convergence
  2. Build initial map of surroundings
  3. Plan path using RRT* with collision checking
  4. Execute path with real-time obstacle avoidance
  5. Continuous SLAM updates and map refinement
  6. Loop closure when revisiting areas
  7. Return to start point using saved map

Next Steps#

  • Tutorial 7: Add pose estimation for person detection indoors
  • Try stereo SLAM: ORBSLAMStereoIntegration
  • Add UWB-IMU fusion: UWBIMUFusion

Tutorial 7: Pose Estimation#

Objective#

Configure the pose estimation pipeline to detect and track human poses from a drone camera, using 2D detection, temporal smoothing, and aerial-perspective adaptation.

Prerequisites#

  • Completed Tutorial 1
  • Drone with camera
  • Understanding of keypoint-based pose representation

Step 1: Set Up Detectors#

typescript
import {
  type Keypoint2D,
  type PoseDetection2D,
  type ImageFrame,
  type DetectorCapabilities,
  type BoundingBox2D,
} from '@oya/core';

// Define detector capabilities
const capabilities: DetectorCapabilities = {
  maxPersons: 10,
  supportsGPU: true,
  supportsBatching: true,
  keypointCount: 17, // COCO format: 17 keypoints
  formatName: 'COCO',
  minInputSize: 256,
  maxInputSize: 1280,
};

console.log('Pose detection configured:');
console.log(`  Format: ${capabilities.formatName}`);
console.log(`  Keypoints: ${capabilities.keypointCount}`);
console.log(`  Max persons: ${capabilities.maxPersons}`);
console.log(`  GPU: ${capabilities.supportsGPU}`);

Step 2: Process Frame with Keypoint Detection#

typescript
// COCO keypoint names (17 keypoints)
const COCO_KEYPOINTS = [
  'nose',
  'left_eye',
  'right_eye',
  'left_ear',
  'right_ear',
  'left_shoulder',
  'right_shoulder',
  'left_elbow',
  'right_elbow',
  'left_wrist',
  'right_wrist',
  'left_hip',
  'right_hip',
  'left_knee',
  'right_knee',
  'left_ankle',
  'right_ankle',
];

// Simulate a detection result
function generatePoseDetection(
  personId: number,
  centerX: number,
  centerY: number,
  confidence: number,
  timestampMs: number
): PoseDetection2D {
  const keypoints: Keypoint2D[] = COCO_KEYPOINTS.map((name, i) => {
    // Distribute keypoints around the center
    const offsetX = (Math.random() - 0.5) * 0.15;
    const offsetY = (i / COCO_KEYPOINTS.length - 0.3) * 0.4;
    return {
      x: centerX + offsetX,
      y: centerY + offsetY,
      confidence: confidence * (0.8 + Math.random() * 0.2),
      name,
      visible: Math.random() > 0.1, // 90% chance visible
    };
  });

  return {
    id: personId,
    keypoints,
    boundingBox: {
      x: centerX - 0.1,
      y: centerY - 0.2,
      width: 0.2,
      height: 0.5,
      confidence,
    },
    confidence,
    timestampMs,
  };
}

// Detect poses in a frame
const frame: ImageFrame = {
  width: 1920,
  height: 1080,
  timestampMs: Date.now(),
  frameIndex: 0,
};

const detections = [
  generatePoseDetection(0, 0.3, 0.5, 0.94, frame.timestampMs),
  generatePoseDetection(1, 0.7, 0.5, 0.89, frame.timestampMs),
];

console.log(
  `\nDetected ${detections.length} poses in frame ${frame.frameIndex}:`
);
for (const det of detections) {
  const visibleCount = det.keypoints.filter((k) => k.visible).length;
  console.log(
    `  Person #${det.id}: confidence=${det.confidence.toFixed(2)}, ` +
      `visible_keypoints=${visibleCount}/${det.keypoints.length}`
  );
}

Step 3: Aerial Adaptation#

typescript
// Drone-specific adjustments for aerial pose detection
console.log('\nAerial adaptation:');
console.log('  Altitude compensation:');
console.log('    - Higher altitude -> smaller apparent size');
console.log('    - Adjust confidence thresholds by altitude');
console.log('  Angle compensation:');
console.log('    - Top-down: only head/shoulders visible');
console.log('    - 30-60 deg: best for full body detection');
console.log('    - Adjust keypoint visibility expectations');
console.log('  Motion compensation:');
console.log('    - Drone movement adds apparent motion to subjects');
console.log('    - Subtract drone ego-motion from tracking');

Step 4: Multi-Person Tracking Across Frames#

typescript
// Temporal tracking maintains person IDs across frames
console.log('\nTemporal tracking:');
console.log('  Frame 0: Detect 2 persons -> assign IDs [0, 1]');
console.log('  Frame 1: Match detections to existing tracks');
console.log('  Frame 2: Person #0 occluded -> predict from Kalman filter');
console.log('  Frame 3: Person #0 reappears -> re-associate by appearance');
console.log('  Frame 4: New person enters -> assign ID [2]');
console.log('  Frame 5: Person #1 exits frame -> track ages out');

Step 5: Extract Pose Metrics#

typescript
// Analyze detected poses
for (const det of detections) {
  const leftShoulder = det.keypoints.find((k) => k.name === 'left_shoulder');
  const rightShoulder = det.keypoints.find((k) => k.name === 'right_shoulder');
  const leftHip = det.keypoints.find((k) => k.name === 'left_hip');
  const rightHip = det.keypoints.find((k) => k.name === 'right_hip');

  if (leftShoulder && rightShoulder && leftHip && rightHip) {
    // Shoulder width (normalized)
    const shoulderWidth = Math.abs(rightShoulder.x - leftShoulder.x);
    // Torso length (normalized)
    const torsoLength = Math.abs(
      (leftHip.y + rightHip.y) / 2 - (leftShoulder.y + rightShoulder.y) / 2
    );

    console.log(`\n  Person #${det.id} metrics:`);
    console.log(
      `    Shoulder width: ${(shoulderWidth * frame.width).toFixed(0)}px`
    );
    console.log(
      `    Torso length: ${(torsoLength * frame.height).toFixed(0)}px`
    );
  }
}

Expected Output#

text
Pose detection configured:
  Format: COCO
  Keypoints: 17
  Max persons: 10
  GPU: true

Detected 2 poses in frame 0:
  Person #0: confidence=0.94, visible_keypoints=15/17
  Person #1: confidence=0.89, visible_keypoints=16/17

Aerial adaptation:
  Altitude compensation:
    - Higher altitude -> smaller apparent size
    - Adjust confidence thresholds by altitude
  Angle compensation:
    - Top-down: only head/shoulders visible
    - 30-60 deg: best for full body detection
    - Adjust keypoint visibility expectations
  Motion compensation:
    - Drone movement adds apparent motion to subjects
    - Subtract drone ego-motion from tracking

Temporal tracking:
  Frame 0: Detect 2 persons -> assign IDs [0, 1]
  Frame 1: Match detections to existing tracks
  Frame 2: Person #0 occluded -> predict from Kalman filter
  Frame 3: Person #0 reappears -> re-associate by appearance
  Frame 4: New person enters -> assign ID [2]
  Frame 5: Person #1 exits frame -> track ages out

  Person #0 metrics:
    Shoulder width: 154px
    Torso length: 194px

  Person #1 metrics:
    Shoulder width: 148px
    Torso length: 187px

Next Steps#

  • Tutorial 8: Deploy pose detection model to edge hardware
  • Add SkeletonTracker for skeleton-based analysis
  • Use GestureControl for drone control via hand gestures

Tutorial 8: Edge AI#

Objective#

Optimize a pose detection model for edge deployment on NVIDIA Jetson, configure inference parameters, and run real-time detection onboard a drone.

Prerequisites#

  • NVIDIA Jetson Nano/Orin (or similar edge hardware)
  • Pre-trained ONNX model
  • @oya/core installed

Step 1: Detect Hardware Capabilities#

typescript
import {
  type EdgePlatform,
  type HardwareCapability,
  type ResourceSnapshot,
  type DeploymentState,
} from '@oya/core';

// Identify the target platform
const platform: EdgePlatform = 'jetson_orin_nano';

console.log('Edge hardware detected:');
console.log(`  Platform: ${platform}`);
console.log('  CPU: ARM Cortex-A78AE (6 cores)');
console.log('  GPU: NVIDIA Ampere (1024 CUDA cores)');
console.log('  AI Performance: 40 TOPS');
console.log('  Power: 7-15W');

Step 2: Optimize Model#

typescript
import {
  type QuantizationType,
  type QuantizationMode,
  type PruningStrategy,
  type OptimizationPass,
} from '@oya/core';

// Model optimization pipeline
console.log('\nModel optimization pipeline:');

// Step 2a: FP16 conversion (2x speedup, minimal accuracy loss)
const fp16Config = {
  quantization: 'fp16' as QuantizationType,
  mode: 'static' as QuantizationMode,
};
console.log('  1. FP16 conversion:');
console.log('     Speedup: ~2x on Ampere GPU');
console.log('     Accuracy loss: < 0.1%');

// Step 2b: INT8 quantization (4x speedup, requires calibration)
const int8Config = {
  quantization: 'int8' as QuantizationType,
  mode: 'static' as QuantizationMode,
};
console.log('  2. INT8 quantization (optional):');
console.log('     Speedup: ~4x on Ampere GPU');
console.log('     Accuracy loss: < 1% with good calibration');

// Step 2c: Graph optimizations
const optimizations: OptimizationPass[] = [
  'constant_folding',
  'dead_code_elimination',
  'operator_fusion',
  'layout_optimization',
  'memory_planning',
];
console.log('  3. Graph optimizations:');
for (const opt of optimizations) {
  console.log(`     - ${opt}`);
}

// Step 2d: Pruning (optional, for extreme size reduction)
const pruneConfig = {
  strategy: 'structured' as PruningStrategy,
  sparsity: 0.3, // Remove 30% of filters
};
console.log('  4. Structured pruning (optional):');
console.log(`     Sparsity: ${pruneConfig.sparsity * 100}%`);
console.log('     Size reduction: ~30%');

Step 3: Configure Inference Engine#

typescript
import {
  type InferenceBackend,
  type InferenceDevice,
  type PrecisionMode,
  type MemoryMode,
  type Tensor,
  type TensorShape,
  type InferenceResult,
} from '@oya/core';

// Configure TensorRT inference (fastest on Jetson)
const inferenceConfig = {
  backend: 'tensorrt' as InferenceBackend,
  device: 'gpu' as InferenceDevice,
  precision: 'fp16' as PrecisionMode,
  memory: 'pool' as MemoryMode,
  batchSize: 1,
  inputShape: [1, 3, 640, 640] as TensorShape, // NCHW
};

console.log('\nInference engine configured:');
console.log(`  Backend: ${inferenceConfig.backend}`);
console.log(`  Device: ${inferenceConfig.device}`);
console.log(`  Precision: ${inferenceConfig.precision}`);
console.log(`  Memory: ${inferenceConfig.memory}`);
console.log(`  Input shape: [${inferenceConfig.inputShape.join(', ')}]`);
console.log(`  Batch size: ${inferenceConfig.batchSize}`);

Step 4: Deploy Model#

typescript
// Deployment lifecycle
const deploymentSteps = [
  { state: 'pending' as DeploymentState, desc: 'Model queued for deployment' },
  {
    state: 'downloading' as DeploymentState,
    desc: 'Downloading optimized model (45MB)',
  },
  {
    state: 'installing' as DeploymentState,
    desc: 'Installing TensorRT engine',
  },
  {
    state: 'validating' as DeploymentState,
    desc: 'Running validation inference',
  },
  { state: 'active' as DeploymentState, desc: 'Model deployed and serving' },
];

console.log('\nDeployment progress:');
for (const step of deploymentSteps) {
  console.log(`  [${step.state}] ${step.desc}`);
}

Step 5: Monitor Resource Usage#

typescript
// Simulate resource monitoring during inference
const resourceSnapshots: ResourceSnapshot[] = [
  {
    cpuPercent: 25,
    gpuPercent: 68,
    memoryUsedBytes: 2.1 * 1024 ** 3,
    memoryTotalBytes: 8 * 1024 ** 3,
    gpuMemoryUsedBytes: 1.8 * 1024 ** 3,
    gpuMemoryTotalBytes: 8 * 1024 ** 3,
    diskUsedBytes: 12 * 1024 ** 3,
    powerWatts: 11.2,
    temperatureC: 58,
    timestampMs: Date.now(),
  },
];

const snap = resourceSnapshots[0];
console.log('\nResource usage during inference:');
console.log(`  CPU: ${snap.cpuPercent}%`);
console.log(`  GPU: ${snap.gpuPercent}%`);
console.log(
  `  Memory: ${(snap.memoryUsedBytes / 1024 ** 3).toFixed(1)}GB / ${(snap.memoryTotalBytes / 1024 ** 3).toFixed(0)}GB`
);
console.log(
  `  GPU Memory: ${(snap.gpuMemoryUsedBytes / 1024 ** 3).toFixed(1)}GB / ${(snap.gpuMemoryTotalBytes / 1024 ** 3).toFixed(0)}GB`
);
console.log(`  Power: ${snap.powerWatts}W`);
console.log(`  Temperature: ${snap.temperatureC}C`);
console.log(
  `  Thermal state: ${snap.temperatureC < 70 ? 'Normal' : 'Warning'}`
);

Expected Output#

text
Edge hardware detected:
  Platform: jetson_orin_nano
  CPU: ARM Cortex-A78AE (6 cores)
  GPU: NVIDIA Ampere (1024 CUDA cores)
  AI Performance: 40 TOPS
  Power: 7-15W

Model optimization pipeline:
  1. FP16 conversion:
     Speedup: ~2x on Ampere GPU
     Accuracy loss: < 0.1%
  2. INT8 quantization (optional):
     Speedup: ~4x on Ampere GPU
     Accuracy loss: < 1% with good calibration
  3. Graph optimizations:
     - constant_folding
     - dead_code_elimination
     - operator_fusion
     - layout_optimization
     - memory_planning
  4. Structured pruning (optional):
     Sparsity: 30%
     Size reduction: ~30%

Inference engine configured:
  Backend: tensorrt
  Device: gpu
  Precision: fp16
  Memory: pool
  Input shape: [1, 3, 640, 640]
  Batch size: 1

Deployment progress:
  [pending] Model queued for deployment
  [downloading] Downloading optimized model (45MB)
  [installing] Installing TensorRT engine
  [validating] Running validation inference
  [active] Model deployed and serving

Resource usage during inference:
  CPU: 25%
  GPU: 68%
  Memory: 2.1GB / 8GB
  GPU Memory: 1.8GB / 8GB
  Power: 11.2W
  Temperature: 58C
  Thermal state: Normal

Next Steps#

  • Tutorial 9: Build custom missions with conditional waypoints
  • Try different backends: onnxruntime, tflite, openvino
  • Implement canary deployment for gradual model rollout

Tutorial 9: Custom Mission#

Objective#

Build a custom mission with conditional waypoints that adapt behavior based on battery level, weather, and detection results during flight.

Prerequisites#

  • Completed Tutorials 1-2
  • Understanding of mission execution subsystems

Step 1: Create Conditional Waypoints#

typescript
import {
  ConditionalWaypointExecutor,
  ConditionType,
  type WaypointCondition,
  type ConditionContext,
  DynamicMissionModifier,
  ModificationAction,
  MissionPauseResumeController,
  PauseResumeState,
  MissionAbortController,
  AbortBehavior,
  type Waypoint,
  type WaypointId,
  WaypointAction,
  BATTERY_LOW_PERCENT,
} from '@oya/core';

// Create conditional waypoint executor
const conditional = new ConditionalWaypointExecutor();

// Define conditions
const batteryCondition: WaypointCondition = {
  type: ConditionType.Battery,
  operator: 'greaterThan',
  value: BATTERY_LOW_PERCENT, // 25%
  description: 'Continue only if battery > 25%',
};

// Waypoints with conditions
const conditionalWaypoints: Array<{
  waypoint: Waypoint;
  condition?: WaypointCondition;
}> = [
  {
    waypoint: {
      id: '11111111-1111-4111-8111-111111111111' as WaypointId,
      position: { latitude: 47.3977, longitude: 8.5456, altitude: 50 },
      speed: 5.0,
      heading: 0,
      holdTimeSeconds: 0,
      actions: [{ action: WaypointAction.StartVideo }],
      radius: 2.0,
    },
    // No condition - always execute
  },
  {
    waypoint: {
      id: '22222222-2222-4222-8222-222222222222' as WaypointId,
      position: { latitude: 47.399, longitude: 8.5456, altitude: 60 },
      speed: 5.0,
      heading: 0,
      holdTimeSeconds: 5,
      actions: [
        { action: WaypointAction.TakePhoto },
        { action: WaypointAction.SetGimbal, params: { pitch: -45 } },
      ],
      radius: 2.0,
    },
    condition: batteryCondition, // Only visit if battery > 25%
  },
  {
    waypoint: {
      id: '33333333-3333-4333-8333-333333333333' as WaypointId,
      position: { latitude: 47.399, longitude: 8.5476, altitude: 60 },
      speed: 3.0,
      heading: 90,
      holdTimeSeconds: 10,
      actions: [
        { action: WaypointAction.OrbitPOI, params: { radius: 20, laps: 1 } },
      ],
      radius: 2.0,
    },
    condition: {
      type: ConditionType.Battery,
      operator: 'greaterThan',
      value: 40, // Only orbit if plenty of battery
      description: 'Orbit POI only if battery > 40%',
    },
  },
];

console.log('Custom mission with conditional waypoints:');
for (let i = 0; i < conditionalWaypoints.length; i++) {
  const cwp = conditionalWaypoints[i];
  const condStr = cwp.condition
    ? `[IF: ${cwp.condition.description}]`
    : '[ALWAYS]';
  console.log(`  WP${i + 1}: ${condStr}`);
  for (const action of cwp.waypoint.actions) {
    console.log(`    -> ${action.action}`);
  }
}

Step 2: Dynamic Mission Modification#

typescript
const modifier = new DynamicMissionModifier();

// Insert a new waypoint mid-mission (e.g., detected an object of interest)
console.log('\nDynamic modifications:');
console.log('  Scenario: AI detects object of interest during flight');
console.log('  Action: Insert investigation waypoint after current WP');

const investigateWP: Waypoint = {
  id: '99999999-9999-4999-8999-999999999999' as WaypointId,
  position: { latitude: 47.3985, longitude: 8.5465, altitude: 30 },
  speed: 2.0,
  heading: 0,
  holdTimeSeconds: 15,
  actions: [
    { action: WaypointAction.TakePhoto },
    { action: WaypointAction.ChangeAltitude, params: { altitude: 20 } },
    { action: WaypointAction.TakePhoto },
  ],
  radius: 1.0,
};

console.log(`  Inserted WP: investigate at alt=30m, then descend to 20m`);
console.log(`  Actions: photo -> descend -> photo`);

Step 3: Mission Pause/Resume#

typescript
const pauseResume = new MissionPauseResumeController();

console.log('\nPause/Resume capability:');
console.log('  Mission can be paused at any time');
console.log('  Drone hovers in place during pause');
console.log('  Resume continues from current position');

// Simulate pause/resume flow
console.log('  Flow:');
console.log('    [Running] -> pause() -> [Paused] -> resume() -> [Running]');

Step 4: Mission Abort with Smart RTH#

typescript
import {
  SmartRTHController,
  RTHAltitudeSelector,
  ReturnToHomeController,
  RTHPhase,
} from '@oya/core';

const abort = new MissionAbortController();
const smartRTH = new SmartRTHController();
const altSelector = new RTHAltitudeSelector();

console.log('\nAbort and Smart RTH:');
console.log('  Abort behaviors:');
console.log('    - ReturnToHome: fly back to launch point');
console.log('    - LandInPlace: land at current position');
console.log('    - HoverAndWait: hover until pilot takes over');
console.log('  Smart RTH features:');
console.log('    - Obstacle-aware return path');
console.log('    - Wind-compensated altitude selection');
console.log('    - Battery-optimized route');
console.log('  RTH phases:');
console.log('    Climb -> Cruise -> Approach -> Descend -> Land');

Step 5: Specialized Mission Generators#

typescript
import {
  SurveyMissionGenerator,
  InspectionMissionGenerator,
  SearchAndRescuePatternGenerator,
  SearchPattern,
} from '@oya/core';

// Auto-generate a survey grid mission
const survey = new SurveyMissionGenerator();
console.log('\nSpecialized mission types:');
console.log('  Survey: auto-generated grid/crosshatch patterns');
console.log('  Inspection: waypoints around a structure');
console.log('  Search & Rescue patterns:');
console.log('    - Expanding square');
console.log('    - Parallel track');
console.log('    - Sector search');
console.log('    - Creeping line');

const sar = new SearchAndRescuePatternGenerator();
// Generate an expanding square search pattern
// sar.generate(SearchPattern.ExpandingSquare, ...)

Expected Output#

text
Custom mission with conditional waypoints:
  WP1: [ALWAYS]
    -> StartVideo
  WP2: [IF: Continue only if battery > 25%]
    -> TakePhoto
    -> SetGimbal
  WP3: [IF: Orbit POI only if battery > 40%]
    -> OrbitPOI

Dynamic modifications:
  Scenario: AI detects object of interest during flight
  Action: Insert investigation waypoint after current WP
  Inserted WP: investigate at alt=30m, then descend to 20m
  Actions: photo -> descend -> photo

Pause/Resume capability:
  Mission can be paused at any time
  Drone hovers in place during pause
  Resume continues from current position
  Flow:
    [Running] -> pause() -> [Paused] -> resume() -> [Running]

Abort and Smart RTH:
  Abort behaviors:
    - ReturnToHome: fly back to launch point
    - LandInPlace: land at current position
    - HoverAndWait: hover until pilot takes over
  Smart RTH features:
    - Obstacle-aware return path
    - Wind-compensated altitude selection
    - Battery-optimized route
  RTH phases:
    Climb -> Cruise -> Approach -> Descend -> Land

Specialized mission types:
  Survey: auto-generated grid/crosshatch patterns
  Inspection: waypoints around a structure
  Search & Rescue patterns:
    - Expanding square
    - Parallel track
    - Sector search
    - Creeping line

Next Steps#

  • Tutorial 10: See full example applications
  • Combine conditional waypoints with subject tracking
  • Add MissionReplayController to replay recorded missions
  • Use MissionTemplateLibrary to save reusable patterns

Tutorial 10: Example Applications#

Objective#

Reference guide showing how the individual tutorials combine into complete drone applications.

Application 1: Aerial Survey System#

Combines: Tutorials 1, 2, 9

text
+------------------+     +------------------+     +------------------+
| Mission Planning |     | Flight Execution |     | Post-Processing  |
|                  |     |                  |     |                  |
| SurveyGenerator  | --> | MissionUploader  | --> | MediaDownloader  |
| BatteryPlanner   |     | WaypointSequencer|     | GeoTagging       |
| WeatherCheck     |     | ProgressTracker  |     | Orthomosaic      |
+------------------+     +------------------+     +------------------+

Key classes:

  • SurveyMissionGenerator for grid pattern creation
  • BatteryAwarePlanner for flight time optimization
  • MissionUploader + MissionValidator for safe upload
  • PhotoCaptureController for timed photo capture
  • MediaFileDownloader for post-flight retrieval

Application 2: Multi-Drone Film Production#

Combines: Tutorials 3, 4, 5

text
+------------------+     +------------------+     +------------------+
| Shot Planning    |     | Live Execution   |     | Post-Production  |
|                  |     |                  |     |                  |
| ShotListAutomation| --> | DirectorMode    | --> | ColorMatcher     |
| EstablishShot    |     | MultiDroneSeq   |     | StreamRecorder   |
| OrbitShot        |     | SubjectTracking |     | TimecodeSyncer   |
+------------------+     +------------------+     +------------------+

Key classes:

  • DirectorMode for autonomous multi-camera direction
  • MultiDroneShotSequencer for coordinated shot sequences
  • DeepSORTTracker + ActiveTrackMode for subject following
  • OrbitShotTrajectory + GimbalPOILockMode for cinematic orbits
  • TimecodeSynchronizer for multi-camera sync
  • BulletTimeCoordinator for freeze-frame effects

Application 3: Search and Rescue#

Combines: Tutorials 3, 7, 9

text
+------------------+     +------------------+     +------------------+
| Search Planning  |     | Detection Loop   |     | Response         |
|                  |     |                  |     |                  |
| SARPatternGen    | --> | PoseDetection2D  | --> | EmergencyBeacon  |
| SwarmMission     |     | HumanDetector    |     | CoordinateReport |
| TaskAllocation   |     | ThermalCamera    |     | MissionModify    |
+------------------+     +------------------+     +------------------+

Key classes:

  • SearchAndRescuePatternGenerator for systematic search patterns
  • SwarmMissionPlanner + AuctionTaskAssigner for area partitioning
  • Pose detection pipeline for person identification
  • HumanDetector for human detection in various conditions
  • ThermalCameraIntegration for night/low-visibility operations
  • DynamicMissionModifier for real-time mission adaptation

Application 4: Infrastructure Inspection#

Combines: Tutorials 2, 6, 8

text
+------------------+     +------------------+     +------------------+
| Inspection Plan  |     | Autonomous Flight|     | Analysis         |
|                  |     |                  |     |                  |
| InspectionGen    | --> | ObstacleAvoid    | --> | ModelInference   |
| PathPlanning     |     | VisualSLAM       |     | DefectDetection  |
| GeofenceConfig   |     | DepthCamera      |     | ReportGeneration |
+------------------+     +------------------+     +------------------+

Key classes:

  • InspectionMissionGenerator for structure-aware waypoints
  • DepthCameraDetector + ObstacleSensorFusion for close-proximity flight
  • Visual SLAM for GPS-denied environments (bridges, tunnels)
  • Edge AI inference for real-time defect detection
  • PolygonGeofence for safety boundaries around structures

Application 5: Precision Agriculture#

Combines: Tutorials 2, 3, 8

text
+------------------+     +------------------+     +------------------+
| Field Mapping    |     | Data Collection  |     | Analysis         |
|                  |     |                  |     |                  |
| SurveyGenerator  | --> | MultispectralCam | --> | NDVIPipeline     |
| SwarmPartition   |     | ThermalCamera    |     | ManagementZones  |
| RTKPositioning   |     | GPSLogger        |     | PrescriptionMap  |
+------------------+     +------------------+     +------------------+

Key classes:

  • SurveyMissionGenerator for field coverage patterns
  • MultispectralCameraIntegration for NDVI/crop health imaging
  • NDVICalculationPipeline for vegetation index computation
  • RTKPositionProvider for centimeter-level accuracy
  • Swarm coordination for large-field coverage

Module Cross-Reference#

Application Foundation Control Autonomy Coordination Perception Cinema Safety AI/ML
Aerial Survey Yes Yes Yes - - - Yes -
Film Production Yes Yes Yes Yes Yes Yes Yes Yes
Search and Rescue Yes Yes Yes Yes Yes - Yes Yes
Infrastructure Yes Yes Yes - Yes - Yes Yes
Agriculture Yes Yes Yes Yes Yes - Yes -

Appendix: Complete Import Map#

Quick reference for which classes come from which source module:

Use Case Key Imports Source Module
PX4 connection PX4SITLConnection, PX4MAVLinkTransport px4-autopilot.ts
PX4 control PX4OffboardController, PX4OffboardMode px4-autopilot.ts
ArduPilot connection APConnectionManager, APArmingController ardupilot.ts
DJI integration DJIAppKeyManager, DJIAircraftConnection dji-sdk.ts
Mission execution MissionUploader, MissionValidator, WaypointSequencer mission-execution.ts
Swarm management SwarmManager, SwarmHeartbeatMonitor swarm-coordination.ts
Formation flying VFormation, LineFormation, ReynoldsFlocking formation-flying.ts
Subject tracking DeepSORTTracker, YOLO11Detector, ActiveTrackMode subject-tracking.ts
Cinematography OrbitShotTrajectory, RevealShotTrajectory cinematography-trajectory.ts
Gimbal control ThreeAxisGimbalController, GimbalPOILockMode gimbal-control.ts
Shot planning RuleOfThirdsGrid, ShotSizeClassifier shot-planning.ts
Visual SLAM ORBSLAMMonoIntegration, LoopClosureDetector visual-slam.ts
Path planning RRTStarPlanner, AStarPlanner path-planning.ts
Obstacle avoidance DepthCameraDetector, ObstacleSensorFusion obstacle-avoidance.ts
Pose detection Keypoint2D, PoseDetection2D, ImageFrame pose-detection-2d.ts
Edge deployment EdgePlatform, DeploymentState, ResourceSnapshot edge-deployment.ts
Model inference InferenceBackend, Tensor, InferenceResult model-inference.ts
Model optimization QuantizationType, PruningStrategy model-optimization.ts
Geofencing PolygonGeofence, GeofenceZone geofencing.ts
Emergency procedures EmergencyState, EmergencyTrigger emergency-procedures.ts
Gazebo simulation createDefaultWorldConfig, GazeboWorldConfig gazebo-integration.ts
AirSim simulation AirSimConnection, AirSimConnectionConfig airsim-integration.ts
Types and schemas FlightMode, DroneState, TelemetrySchema types.ts
Math utilities KalmanFilter, PIDController, MovingAverageFilter math-utils.ts
Kinematics Vec3, Quat, RigidBody6DOF, MultirotorDynamics kinematics.ts
Constants FAA_MAX_ALTITUDE_AGL_M, BATTERY_LOW_PERCENT constants.ts
Coordinate transforms gpsToLocal, haversineDistance, bearing coordinate-transforms.ts