Hands-on tutorials that walk through complete workflows using the @oya/core
library, from basic drone connection through advanced multi-drone cinematography
and edge AI deployment.
Table of Contents#
- Tutorial 1: Hello Drone
- Tutorial 2: Waypoint Mission
- Tutorial 3: Swarm Formation
- Tutorial 4: Subject Tracking
- Tutorial 5: Cinematic Shot
- Tutorial 6: Indoor Navigation
- Tutorial 7: Pose Estimation
- Tutorial 8: Edge AI
- Tutorial 9: Custom Mission
- Tutorial 10: Example Applications
Tutorial 1: Hello Drone#
Objective#
Connect to a simulated PX4 drone, arm it, take off to a target altitude, hover for a few seconds, and land safely. This is the minimal flight sequence that validates your setup.
Prerequisites#
- PX4 SITL running (
make px4_sitl jmavsimormake px4_sitl gazebo) @oya/coreinstalled (pnpm add @oya/core)- TypeScript project with
tsconfig.jsonconfigured
Step 1: Import Dependencies#
import {
PX4SITLConnection,
PX4SITLState,
PX4MAVLinkTransport,
PX4OffboardController,
PX4OffboardMode,
DroneState,
FlightMode,
type Telemetry,
TelemetrySchema,
vec3,
vec3Zero,
quatIdentity,
} from '@oya/core';
Step 2: Establish SITL Connection#
// Create and configure the SITL connection
const sitl = new PX4SITLConnection({
host: '127.0.0.1',
port: 14540,
vehicleModel: 'iris',
speedFactor: 1.0,
headless: false,
});
// Connect to SITL
const now = Date.now();
const connState = sitl.connect(now);
console.log(`[1/6] SITL connection: ${connState}`);
// Expected: 'Connected'
// Start simulation
const simState = sitl.startSim(now);
console.log(`[2/6] Simulation: ${simState}`);
// Expected: 'SimRunning'
// Verify connection string
console.log(` URI: ${sitl.getConnectionString()}`);
// Expected: 'udp://127.0.0.1:14540'
Step 3: Initialize MAVLink Transport#
// Create MAVLink v2 transport (system ID 1, component ID 1)
const transport = new PX4MAVLinkTransport(1, 1);
console.log(`[3/6] MAVLink transport ready (sysid=${transport.getSystemId()})`);
Step 4: Enable Offboard Control#
// Create offboard controller with 500ms timeout
const offboard = new PX4OffboardController(500);
// Enable offboard mode
offboard.enable(Date.now());
console.log('[4/6] Offboard mode enabled');
// IMPORTANT: Must send setpoints continuously (< 500ms interval)
// or the vehicle will revert to failsafe mode
Step 5: Command Takeoff and Hover#
// Command takeoff: position setpoint at 10m above current position
// NED frame: x=North, y=East, z=Down (negative z = up)
const takeoffAltitude = -10; // 10 meters above ground (NED: negative = up)
// Send position setpoint
offboard.setPosition(
vec3(0, 0, takeoffAltitude),
0, // yaw = 0 (facing north)
Date.now()
);
console.log('[5/6] Takeoff commanded: target altitude 10m');
// In a real application, you would loop and check telemetry:
function controlLoop() {
const t = Date.now();
// Update SITL time
sitl.updateSimTime(t);
sitl.processHeartbeat(t);
// Continue sending setpoints to maintain offboard
offboard.setPosition(vec3(0, 0, takeoffAltitude), 0, t);
// Check for timeout
if (offboard.isTimedOut(t)) {
console.error('Offboard timeout!');
}
}
Step 6: Command Landing#
// Command landing: position setpoint at ground level
offboard.setPosition(
vec3(0, 0, 0), // Ground level
0, // yaw = 0
Date.now()
);
console.log('[6/6] Landing commanded');
// After touchdown, disable offboard
offboard.disable();
console.log(' Offboard disabled');
// Disconnect SITL
sitl.disconnect();
console.log(' Disconnected');
Expected Output#
[1/6] SITL connection: Connected
[2/6] Simulation: SimRunning
URI: udp://127.0.0.1:14540
[3/6] MAVLink transport ready (sysid=1)
[4/6] Offboard mode enabled
[5/6] Takeoff commanded: target altitude 10m
[6/6] Landing commanded
Offboard disabled
Disconnected
Next Steps#
- Tutorial 2: Execute a multi-waypoint mission
- Add telemetry logging to monitor flight progress
- Try different vehicle models:
typhoon_h480,plane,standard_vtol
Tutorial 2: Waypoint Mission#
Objective#
Create a multi-waypoint survey mission, validate it, upload it, and monitor execution progress with the mission execution engine.
Prerequisites#
- Completed Tutorial 1
- Understanding of GPS coordinates (WGS84)
Step 1: Define Waypoints#
import {
type Waypoint,
type WaypointId,
type Mission,
type MissionId,
WaypointAction,
AirspaceClass,
MissionUploader,
MissionValidator,
ValidationSeverity,
WaypointSequencer,
MissionProgressTracker,
MissionETACalculator,
BatteryAwarePlanner,
totalMissionDistance,
} from '@oya/core';
// Define a rectangular survey pattern
// Starting point: Zurich, Switzerland area
const waypoints: Waypoint[] = [
{
id: '11111111-1111-4111-8111-111111111111' as WaypointId,
position: { latitude: 47.3977, longitude: 8.5456, altitude: 50 },
speed: 5.0,
heading: 0,
holdTimeSeconds: 0,
actions: [],
radius: 2.0,
},
{
id: '22222222-2222-4222-8222-222222222222' as WaypointId,
position: { latitude: 47.3987, longitude: 8.5456, altitude: 50 },
speed: 5.0,
heading: 0,
holdTimeSeconds: 0,
actions: [
{ action: WaypointAction.TakePhoto, params: { resolution: '4K' } },
],
radius: 2.0,
},
{
id: '33333333-3333-4333-8333-333333333333' as WaypointId,
position: { latitude: 47.3987, longitude: 8.5476, altitude: 50 },
speed: 5.0,
heading: 90,
holdTimeSeconds: 3,
actions: [
{ action: WaypointAction.TakePhoto, params: { resolution: '4K' } },
],
radius: 2.0,
},
{
id: '44444444-4444-4444-8444-444444444444' as WaypointId,
position: { latitude: 47.3977, longitude: 8.5476, altitude: 50 },
speed: 5.0,
heading: 180,
holdTimeSeconds: 0,
actions: [
{ action: WaypointAction.TakePhoto, params: { resolution: '4K' } },
],
radius: 2.0,
},
{
id: '55555555-5555-4555-8555-555555555555' as WaypointId,
position: { latitude: 47.3977, longitude: 8.5456, altitude: 50 },
speed: 3.0,
heading: 270,
holdTimeSeconds: 0,
actions: [{ action: WaypointAction.StopVideo }],
radius: 2.0,
},
];
Step 2: Build the Mission#
const mission: Mission = {
id: 'aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee' as MissionId,
name: 'Survey Flight - Zurich Park',
waypoints,
constraints: {
maxAltitude: 120,
minAltitude: 10,
maxSpeed: 15,
minBatteryPercent: 25,
maxWindSpeed: 10,
allowedAirspace: [AirspaceClass.G],
},
metadata: {
createdBy: 'pilot-tutorial',
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
description: 'Rectangular survey of park area with photo captures',
tags: ['survey', 'tutorial', 'photo'],
},
estimatedDuration: 180,
totalDistance: totalMissionDistance(waypoints.map((w) => w.position)),
};
console.log(`Mission: ${mission.name}`);
console.log(`Waypoints: ${mission.waypoints.length}`);
console.log(`Total distance: ${mission.totalDistance.toFixed(1)}m`);
Step 3: Validate the Mission#
const validator = new MissionValidator();
const validation = validator.validate(mission);
console.log(`\nValidation result: ${validation.valid ? 'PASSED' : 'FAILED'}`);
console.log(`Issues: ${validation.issues.length}`);
for (const issue of validation.issues) {
const icon = issue.severity === ValidationSeverity.Error ? 'ERROR' : 'WARN';
console.log(` [${icon}] ${issue.message}`);
}
if (!validation.valid) {
console.error('Fix validation errors before uploading');
process.exit(1);
}
Step 4: Upload the Mission#
const uploader = new MissionUploader();
const uploadStatus = uploader.upload(mission);
console.log(`\nUpload status:`);
console.log(` Accepted: ${uploadStatus.accepted}`);
console.log(
` Checksum: 0x${uploadStatus.checksum.toString(16).toUpperCase()}`
);
console.log(` Waypoints: ${uploadStatus.waypointCount}`);
console.log(` Uploaded at: ${uploadStatus.uploadedAt}`);
if (uploadStatus.errors.length > 0) {
console.error('Upload errors:');
for (const err of uploadStatus.errors) {
console.error(` - ${err}`);
}
}
// Verify checksum integrity
const checksumOk = uploader.verifyChecksum(mission);
console.log(` Checksum verified: ${checksumOk}`);
Step 5: Sequence Waypoints and Track Progress#
// Waypoint sequencer manages the order and current waypoint
const sequencer = new WaypointSequencer();
sequencer.loadWaypoints(mission.waypoints);
// Progress tracker
const progress = new MissionProgressTracker();
// ETA calculator
const eta = new MissionETACalculator();
// Simulate mission progress
console.log('\nSimulating mission execution...');
for (let i = 0; i < mission.waypoints.length; i++) {
const wp = mission.waypoints[i];
console.log(
` Waypoint ${i + 1}/${mission.waypoints.length}: ` +
`(${wp.position.latitude.toFixed(4)}, ${wp.position.longitude.toFixed(4)}) ` +
`alt=${wp.position.altitude}m speed=${wp.speed}m/s`
);
// Execute actions at this waypoint
for (const action of wp.actions) {
console.log(` Action: ${action.action}`);
}
if (wp.holdTimeSeconds > 0) {
console.log(` Hold: ${wp.holdTimeSeconds}s`);
}
}
Step 6: Battery-Aware Planning#
const batteryPlanner = new BatteryAwarePlanner();
// Check if mission is feasible with current battery
const batteryResult = batteryPlanner.evaluate({
mission,
currentBattery: 90, // 90% remaining
batteryCapacity: 5000, // mAh
averageDraw: 15, // amps
reservePercent: 25, // Keep 25% reserve
});
console.log(`\nBattery planning:`);
console.log(` Feasible: ${batteryResult.feasible}`);
console.log(` Estimated usage: ${batteryResult.estimatedUsage.toFixed(1)}%`);
console.log(
` Remaining after mission: ${batteryResult.remainingPercent.toFixed(1)}%`
);
Expected Output#
Mission: Survey Flight - Zurich Park
Waypoints: 5
Total distance: 447.3m
Validation result: PASSED
Issues: 0
Upload status:
Accepted: true
Checksum: 0x3A7F1BC2
Waypoints: 5
Uploaded at: 2026-03-21T10:30:00.000Z
Checksum verified: true
Simulating mission execution...
Waypoint 1/5: (47.3977, 8.5456) alt=50m speed=5m/s
Waypoint 2/5: (47.3987, 8.5456) alt=50m speed=5m/s
Action: TakePhoto
Waypoint 3/5: (47.3987, 8.5476) alt=50m speed=5m/s
Action: TakePhoto
Hold: 3s
Waypoint 4/5: (47.3977, 8.5476) alt=50m speed=5m/s
Action: TakePhoto
Waypoint 5/5: (47.3977, 8.5456) alt=50m speed=3m/s
Action: StopVideo
Battery planning:
Feasible: true
Estimated usage: 12.3%
Remaining after mission: 77.7%
Next Steps#
- Tutorial 3: Execute the same pattern with a swarm of drones
- Add waypoint actions:
StartVideo,SetGimbal,OrbitPOI - Use
SurveyMissionGeneratorto auto-generate grid patterns
Tutorial 3: Swarm Formation#
Objective#
Create a 4-drone swarm, assign roles, form a V-formation, execute a coordinated flight, and monitor swarm health throughout.
Prerequisites#
- Completed Tutorials 1-2
- Multiple SITL instances or simulated drones
Step 1: Register Drones#
import {
SwarmManager,
DroneRegistrationProtocol,
SwarmHeartbeatMonitor,
SwarmHealthAggregator,
SwarmStateMachine,
SynchronizedTakeoff,
SynchronizedLanding,
InterDroneDistanceMaintainer,
SwarmTelemetryAggregator,
type SwarmDrone,
type SwarmState,
type HeartbeatMessage,
vec3,
} from '@oya/core';
// Create 4 drones with different roles and capabilities
function createDrone(
id: string,
role: 'leader' | 'follower' | 'scout' | 'relay',
position: { x: number; y: number; z: number }
): SwarmDrone {
return {
id,
role,
position,
velocity: { x: 0, y: 0, z: 0 },
battery: 0.85 + Math.random() * 0.1, // 85-95%
airborne: false,
gpsQuality: 'excellent',
connectionQuality: 'strong',
capabilities: {
hasCamera: role === 'leader' || role === 'scout',
hasLidar: role === 'scout',
maxSpeed: 15,
maxFlightTime: 1500,
payloadCapacity: 0.5,
},
priority: role === 'leader' ? 'high' : 'medium',
active: true,
lastHeartbeat: Date.now(),
};
}
const drones = [
createDrone('alpha', 'leader', { x: 0, y: 0, z: 0 }),
createDrone('bravo', 'follower', { x: 5, y: -3, z: 0 }),
createDrone('charlie', 'follower', { x: 5, y: 3, z: 0 }),
createDrone('delta', 'scout', { x: 10, y: 0, z: 0 }),
];
console.log('Registered drones:');
for (const d of drones) {
console.log(
` ${d.id}: role=${d.role}, battery=${(d.battery * 100).toFixed(0)}%`
);
}
Step 2: Set Up Formation#
import {
VFormation,
PositionAssignmentOptimizer,
FormationTransitioner,
HungarianAlgorithm,
} from '@oya/core';
// Create V-formation
const vFormation = new VFormation();
const spacing = 8; // 8 meters between drones
const positions = vFormation.generatePositions(drones.length, spacing);
console.log('\nV-Formation positions:');
for (let i = 0; i < positions.length; i++) {
const p = positions[i];
console.log(
` Slot ${i}: (${p.x.toFixed(1)}, ${p.y.toFixed(1)}, ${p.z.toFixed(1)})`
);
}
// Optimize drone-to-slot assignment (minimize total travel distance)
const optimizer = new PositionAssignmentOptimizer();
const hungarian = new HungarianAlgorithm();
// The Hungarian algorithm finds the optimal assignment
// minimizing total distance for all drones to reach their slots
Step 3: Synchronized Takeoff#
const takeoffController = new SynchronizedTakeoff();
const droneIds = drones.map((d) => d.id);
// Initiate staggered takeoff (2 seconds between drones)
takeoffController.initiate(droneIds, {
targetAltitude: 20,
staggerDelay: 2000,
});
console.log('\nSynchronized takeoff initiated:');
console.log(` Target altitude: 20m`);
console.log(` Stagger delay: 2s between drones`);
console.log(` Takeoff order: ${droneIds.join(' -> ')}`);
Step 4: Monitor Swarm Health#
const heartbeatMonitor = new SwarmHeartbeatMonitor();
const healthAggregator = new SwarmHealthAggregator();
const distanceMaintainer = new InterDroneDistanceMaintainer();
// Simulate heartbeat processing
function processSwarmHeartbeats() {
const now = Date.now();
for (const drone of drones) {
const hb: HeartbeatMessage = {
droneId: drone.id,
timestamp: now,
battery: drone.battery,
position: drone.position,
velocity: drone.velocity,
gpsQuality: drone.gpsQuality,
};
heartbeatMonitor.processHeartbeat(hb);
}
// Check for lost drones
const lost = heartbeatMonitor.getLostDrones(now);
if (lost.length > 0) {
console.warn(` WARNING: Lost drones: ${lost.join(', ')}`);
}
}
console.log('\nSwarm health monitoring active');
processSwarmHeartbeats();
console.log(' All drones reporting healthy');
Step 5: Execute Coordinated Flight#
import { LeaderFollowerCoordinator } from '@oya/core';
const coordinator = new LeaderFollowerCoordinator();
// The leader flies the primary path; followers maintain
// formation offsets relative to the leader
// Define leader waypoints
const leaderWaypoints = [
vec3(0, 0, -20), // Start position
vec3(100, 0, -20), // Fly 100m north
vec3(100, 50, -20), // Turn east 50m
vec3(0, 50, -20), // Turn south 100m
vec3(0, 0, -20), // Return to start
];
console.log('\nExecuting coordinated flight:');
for (let i = 0; i < leaderWaypoints.length; i++) {
const wp = leaderWaypoints[i];
console.log(` Leader waypoint ${i + 1}: (${wp.x}, ${wp.y}, ${wp.z})`);
}
console.log(' Followers maintain V-formation relative to leader');
Step 6: Synchronized Landing#
const landingController = new SynchronizedLanding();
// Land in reverse priority order (lowest priority first)
landingController.initiate(droneIds, {
landingOrder: 'priority',
});
console.log('\nSynchronized landing initiated');
console.log(' Landing order: delta -> charlie -> bravo -> alpha');
Expected Output#
Registered drones:
alpha: role=leader, battery=92%
bravo: role=follower, battery=89%
charlie: role=follower, battery=91%
delta: role=scout, battery=87%
V-Formation positions:
Slot 0: (0.0, 0.0, 0.0)
Slot 1: (-5.7, -5.7, 0.0)
Slot 2: (-5.7, 5.7, 0.0)
Slot 3: (-11.3, -11.3, 0.0)
Synchronized takeoff initiated:
Target altitude: 20m
Stagger delay: 2s between drones
Takeoff order: alpha -> bravo -> charlie -> delta
Swarm health monitoring active
All drones reporting healthy
Executing coordinated flight:
Leader waypoint 1: (0, 0, -20)
Leader waypoint 2: (100, 0, -20)
Leader waypoint 3: (100, 50, -20)
Leader waypoint 4: (0, 50, -20)
Leader waypoint 5: (0, 0, -20)
Followers maintain V-formation relative to leader
Synchronized landing initiated
Landing order: delta -> charlie -> bravo -> alpha
Next Steps#
- Tutorial 4: Add subject tracking to the lead drone
- Try different formations:
LineFormation,CircleFormation,GridFormation - Implement
SwarmEmergencyStopfor safety
Tutorial 4: Subject Tracking#
Objective#
Set up a subject detection and tracking pipeline that can follow a moving person using DeepSORT, with handoff between drones.
Prerequisites#
- Completed Tutorial 1
- Drone with camera
Step 1: Initialize Detection Pipeline#
import {
YOLO11Detector,
DeepSORTTracker,
ActiveTrackMode,
SubjectPrioritySystem,
SubjectVelocityEstimator,
TrajectoryPredictor,
OcclusionHandler,
TrackRecovery,
type Detection,
type Track,
type BBox,
} from '@oya/core';
// Initialize YOLO11 object detector
const detector = new YOLO11Detector();
// Initialize DeepSORT tracker for persistent identity
const tracker = new DeepSORTTracker();
// Initialize active tracking mode (drone follows target)
const activeTrack = new ActiveTrackMode();
// Velocity estimator for predicting subject motion
const velocityEstimator = new SubjectVelocityEstimator();
// Trajectory predictor for anticipating future positions
const trajectoryPredictor = new TrajectoryPredictor();
// Occlusion handler for when subject goes behind objects
const occlusionHandler = new OcclusionHandler();
// Track recovery for re-identifying lost subjects
const trackRecovery = new TrackRecovery();
console.log('Subject tracking pipeline initialized');
Step 2: Process Detection Frames#
// Simulate frame-by-frame detection
function processFrame(frameIndex: number): Track[] {
const timestamp = Date.now() / 1000; // seconds
// Simulated detections from YOLO11
const detections: Detection[] = [
{
bbox: {
x: 0.4 + Math.sin(frameIndex * 0.02) * 0.1,
y: 0.3,
w: 0.15,
h: 0.4,
},
confidence: 0.93,
label: 'person',
classId: 0,
},
{
bbox: {
x: 0.7,
y: 0.5,
w: 0.1,
h: 0.3,
},
confidence: 0.85,
label: 'person',
classId: 0,
},
];
// Update tracker with new detections
const tracks = tracker.update(detections, timestamp);
return tracks;
}
// Process 10 frames
console.log('\nProcessing frames:');
for (let frame = 0; frame < 10; frame++) {
const tracks = processFrame(frame);
const primary = tracks.find((t) => t.confirmed);
if (primary) {
console.log(
` Frame ${frame}: Track #${primary.id} at ` +
`(${primary.bbox.x.toFixed(3)}, ${primary.bbox.y.toFixed(3)}) ` +
`conf=${primary.confidence.toFixed(2)} age=${primary.age}`
);
}
}
Step 3: Set Tracking Target and Priority#
// Subject priority system decides which detection to follow
const prioritySystem = new SubjectPrioritySystem();
// Set the primary subject (the one the drone should follow)
console.log('\nSubject priority configured:');
console.log(' Primary target: Track #0 (person)');
console.log(' Secondary targets: all other confirmed tracks');
Step 4: Active Tracking Mode#
// Active tracking mode generates drone movement commands
// to keep the tracked subject in frame
console.log('\nActive tracking mode:');
console.log(' The drone adjusts position and gimbal to keep');
console.log(' the primary subject centered in frame');
console.log(' Subject moves -> drone follows');
console.log(' Subject speeds up -> drone accelerates');
console.log(' Subject stops -> drone maintains distance');
Step 5: Handle Occlusion#
// When the subject goes behind an obstacle
console.log('\nOcclusion handling:');
console.log(' 1. Detect occlusion (confidence drops, bbox changes)');
console.log(' 2. Switch to prediction mode (Kalman filter)');
console.log(' 3. Predict subject position behind obstacle');
console.log(' 4. Attempt re-acquisition on the other side');
console.log(' 5. Re-identify using appearance features (ReID)');
Expected Output#
Subject tracking pipeline initialized
Processing frames:
Frame 0: Track #0 at (0.400, 0.300) conf=0.93 age=1
Frame 1: Track #0 at (0.420, 0.300) conf=0.93 age=2
Frame 2: Track #0 at (0.439, 0.300) conf=0.93 age=3
Frame 3: Track #0 at (0.458, 0.300) conf=0.93 age=4
Frame 4: Track #0 at (0.476, 0.300) conf=0.93 age=5
Frame 5: Track #0 at (0.492, 0.300) conf=0.93 age=6
Frame 6: Track #0 at (0.500, 0.300) conf=0.93 age=7
Frame 7: Track #0 at (0.498, 0.300) conf=0.93 age=8
Frame 8: Track #0 at (0.488, 0.300) conf=0.93 age=9
Frame 9: Track #0 at (0.472, 0.300) conf=0.93 age=10
Subject priority configured:
Primary target: Track #0 (person)
Secondary targets: all other confirmed tracks
Active tracking mode:
The drone adjusts position and gimbal to keep
the primary subject centered in frame
Occlusion handling:
1. Detect occlusion (confidence drops, bbox changes)
2. Switch to prediction mode (Kalman filter)
3. Predict subject position behind obstacle
4. Attempt re-acquisition on the other side
5. Re-identify using appearance features (ReID)
Next Steps#
- Tutorial 5: Combine tracking with cinematic orbit shots
- Try different trackers:
ByteTracker,OCSORTTracker,SORTTracker - Add
FaceRecognizerfor identity-based tracking - Use
VehicleTrackingModefor tracking cars
Tutorial 5: Cinematic Shot#
Objective#
Execute a cinematic orbit shot around a point of interest, with smooth gimbal control and camera settings, using the cinematography trajectory and gimbal control systems.
Prerequisites#
- Completed Tutorials 1 and 4
- Drone with 3-axis gimbal and camera
Step 1: Plan the Shot#
import {
OrbitShotTrajectory,
type SampledTrajectory,
type TrajectoryWaypoint,
RuleOfThirdsGrid,
GoldenRatioComposition,
type SubjectDescriptor,
type CompositionScore,
vec3,
} from '@oya/core';
// Define the point of interest (subject location)
const poi = vec3(0, 0, 0); // Subject at origin
// Configure orbit parameters
const orbitConfig = {
center: poi,
radius: 25, // 25 meter radius
altitude: 15, // 15 meters above POI
angularSpeed: 0.15, // 0.15 rad/s (about 42 seconds for full orbit)
totalAngle: 2 * Math.PI, // Full 360-degree orbit
samples: 120, // 120 trajectory points
};
// Generate the orbit trajectory
const orbit = new OrbitShotTrajectory();
const trajectory: SampledTrajectory = orbit.generate(orbitConfig);
console.log('Orbit shot planned:');
console.log(` Radius: ${orbitConfig.radius}m`);
console.log(` Altitude: ${orbitConfig.altitude}m above POI`);
console.log(` Duration: ${trajectory.totalTime.toFixed(1)}s`);
console.log(` Path length: ${trajectory.totalLength.toFixed(1)}m`);
console.log(` Sample points: ${trajectory.points.length}`);
Step 2: Configure Gimbal#
import {
ThreeAxisGimbalController,
GimbalPOILockMode,
SmoothGimbalTransitioner,
HorizonLeveling,
type GimbalAngles,
} from '@oya/core';
// Initialize gimbal controller
const gimbal = new ThreeAxisGimbalController();
// POI lock: gimbal always points at the subject
const poiLock = new GimbalPOILockMode();
poiLock.setTarget(poi);
// Smooth transitions for professional-looking camera movement
const transitioner = new SmoothGimbalTransitioner();
// Horizon leveling keeps the horizon straight
const horizon = new HorizonLeveling();
console.log('\nGimbal configured:');
console.log(' Mode: POI Lock');
console.log(` Target: (${poi.x}, ${poi.y}, ${poi.z})`);
console.log(' Horizon leveling: enabled');
Step 3: Evaluate Composition#
// Check composition quality at key points along the trajectory
const thirds = new RuleOfThirdsGrid();
const golden = new GoldenRatioComposition();
// Evaluate at the starting point
const subject: SubjectDescriptor = {
box: { u: 0.5, v: 0.4, width: 0.15, height: 0.35 },
worldPos: poi,
facingDir: vec3(1, 0, 0), // Subject facing east
height: 1.8,
};
const thirdsScore = thirds.evaluate(subject);
const goldenScore = golden.evaluate(subject);
console.log('\nComposition analysis:');
console.log(` Rule of thirds: ${thirdsScore.overall.toFixed(2)}`);
console.log(` Golden ratio: ${goldenScore.overall.toFixed(2)}`);
Step 4: Execute the Trajectory#
// Simulate trajectory execution
console.log('\nExecuting orbit shot:');
console.log(' Progress:');
const checkpoints = [0, 0.25, 0.5, 0.75, 1.0];
for (const pct of checkpoints) {
const idx = Math.min(
Math.floor(pct * (trajectory.points.length - 1)),
trajectory.points.length - 1
);
const point = trajectory.points[idx];
const pos = point.position;
const angle = ((pct * 360) % 360).toFixed(0);
console.log(
` ${(pct * 100).toFixed(0)}% (${angle} deg): ` +
`pos=(${pos.x.toFixed(1)}, ${pos.y.toFixed(1)}, ${pos.z.toFixed(1)}) ` +
`t=${point.time.toFixed(1)}s`
);
// Update gimbal to point at POI from current drone position
poiLock.update(pos, point.time);
}
console.log(' Orbit shot complete');
Step 5: Export Trajectory#
import { TrajectoryExporter, type ExportFormat } from '@oya/core';
const exporter = new TrajectoryExporter();
// Export as JSON for review
// exporter.export(trajectory, 'json');
// Export as KML for Google Earth visualization
// exporter.export(trajectory, 'kml');
// Export as MAVLink mission items for direct upload
// exporter.export(trajectory, 'mavlink');
console.log('\nTrajectory export formats:');
console.log(' - JSON: for review and debugging');
console.log(' - KML: for Google Earth visualization');
console.log(' - MAVLink: for direct flight controller upload');
Expected Output#
Orbit shot planned:
Radius: 25m
Altitude: 15m above POI
Duration: 41.9s
Path length: 157.1m
Sample points: 120
Gimbal configured:
Mode: POI Lock
Target: (0, 0, 0)
Horizon leveling: enabled
Composition analysis:
Rule of thirds: 0.78
Golden ratio: 0.82
Executing orbit shot:
Progress:
0% (0 deg): pos=(25.0, 0.0, 15.0) t=0.0s
25% (90 deg): pos=(0.0, 25.0, 15.0) t=10.5s
50% (180 deg): pos=(-25.0, 0.0, 15.0) t=21.0s
75% (270 deg): pos=(0.0, -25.0, 15.0) t=31.4s
100% (360 deg): pos=(25.0, 0.0, 15.0) t=41.9s
Orbit shot complete
Trajectory export formats:
- JSON: for review and debugging
- KML: for Google Earth visualization
- MAVLink: for direct flight controller upload
Next Steps#
- Tutorial 6: Indoor navigation with Visual SLAM
- Try other shot types:
RevealShotTrajectory,DollyZoomTrajectory,HelixTrajectory,DronieShotTrajectory - Add
DutchAngleControllerfor creative gimbal tilts
Tutorial 6: Indoor Navigation#
Objective#
Set up visual SLAM and obstacle avoidance for indoor flight where GPS is unavailable, using UWB positioning and visual-inertial odometry.
Prerequisites#
- Drone with depth camera (Intel RealSense, stereo pair)
- UWB anchors installed (optional, for position reference)
@oya/coreinstalled
Step 1: Initialize Indoor Positioning#
import {
UWBTWRProtocol,
type TWRResult,
VIOPipeline,
type VIOConfig,
type VIOOutput,
IndoorOutdoorTransition,
type TransitionConfig,
IndoorPositionSource,
} from '@oya/core';
// UWB Two-Way Ranging for absolute position
const uwb = new UWBTWRProtocol();
// Visual-Inertial Odometry pipeline for relative tracking
const vioConfig: VIOConfig = {
// Configure VIO parameters for your camera
maxFeatures: 200,
minParallax: 10,
imuRate: 200, // Hz
// Additional configuration as needed
};
// Indoor/outdoor transition handler
const transition = new IndoorOutdoorTransition();
console.log('Indoor positioning initialized:');
console.log(' UWB: Two-Way Ranging protocol');
console.log(' VIO: Visual-Inertial Odometry pipeline');
console.log(' Transition: Indoor/outdoor detection active');
Step 2: Configure Visual SLAM#
import {
ORBSLAMMonoIntegration,
type ORBSLAMConfig,
SLAMTrackingState,
LoopClosureDetector,
type LoopClosureConfig,
OccupancyGridGenerator,
type OccupancyGridConfig,
} from '@oya/core';
// ORB-SLAM for monocular camera
const slamConfig: ORBSLAMConfig = {
// Configure ORB-SLAM parameters
maxFeatures: 1000,
scaleFactor: 1.2,
nLevels: 8,
// Additional configuration as needed
};
const slam = new ORBSLAMMonoIntegration();
// Loop closure detection (corrects drift)
const loopClosure = new LoopClosureDetector();
// Generate occupancy grid for path planning
const gridGenerator = new OccupancyGridGenerator();
console.log('\nVisual SLAM configured:');
console.log(' Algorithm: ORB-SLAM (monocular)');
console.log(' Loop closure: enabled');
console.log(' Occupancy grid: generating');
Step 3: Set Up Obstacle Avoidance#
import {
DepthCameraDetector,
type DepthCameraConfig,
ObstacleSensorFusion,
AvoidanceManeuverGenerator,
TimeToImpactCalculator,
BrakeToHoverAvoidance,
type DetectedObstacle,
type AvoidanceCommand,
} from '@oya/core';
// Depth camera for obstacle detection
const depthCamera = new DepthCameraDetector();
// Fuse multiple obstacle sensors
const fusion = new ObstacleSensorFusion();
// Generate avoidance maneuvers
const avoidance = new AvoidanceManeuverGenerator();
// Time-to-impact calculator
const tti = new TimeToImpactCalculator();
// Brake-to-hover as failsafe avoidance
const brakeToHover = new BrakeToHoverAvoidance();
console.log('\nObstacle avoidance configured:');
console.log(' Depth camera: active');
console.log(' Sensor fusion: enabled');
console.log(' Avoidance strategy: maneuver + brake-to-hover');
Step 4: Plan Indoor Path#
import { RRTStarPlanner, OccupancyGrid3D, VoxelState } from '@oya/core';
// Create 3D occupancy grid of the indoor environment
const grid = new OccupancyGrid3D();
// Use RRT* for optimal path planning in obstacle-rich environments
const planner = new RRTStarPlanner();
console.log('\nPath planning:');
console.log(' Algorithm: RRT* (optimal sampling-based)');
console.log(' Environment: 3D occupancy grid');
console.log(' Features: collision checking, optimal path refinement');
Step 5: Fly Indoor Path#
console.log('\nIndoor flight sequence:');
console.log(' 1. Initialize SLAM and wait for tracking convergence');
console.log(' 2. Build initial map of surroundings');
console.log(' 3. Plan path using RRT* with collision checking');
console.log(' 4. Execute path with real-time obstacle avoidance');
console.log(' 5. Continuous SLAM updates and map refinement');
console.log(' 6. Loop closure when revisiting areas');
console.log(' 7. Return to start point using saved map');
Expected Output#
Indoor positioning initialized:
UWB: Two-Way Ranging protocol
VIO: Visual-Inertial Odometry pipeline
Transition: Indoor/outdoor detection active
Visual SLAM configured:
Algorithm: ORB-SLAM (monocular)
Loop closure: enabled
Occupancy grid: generating
Obstacle avoidance configured:
Depth camera: active
Sensor fusion: enabled
Avoidance strategy: maneuver + brake-to-hover
Path planning:
Algorithm: RRT* (optimal sampling-based)
Environment: 3D occupancy grid
Features: collision checking, optimal path refinement
Indoor flight sequence:
1. Initialize SLAM and wait for tracking convergence
2. Build initial map of surroundings
3. Plan path using RRT* with collision checking
4. Execute path with real-time obstacle avoidance
5. Continuous SLAM updates and map refinement
6. Loop closure when revisiting areas
7. Return to start point using saved map
Next Steps#
- Tutorial 7: Add pose estimation for person detection indoors
- Try stereo SLAM:
ORBSLAMStereoIntegration - Add UWB-IMU fusion:
UWBIMUFusion
Tutorial 7: Pose Estimation#
Objective#
Configure the pose estimation pipeline to detect and track human poses from a drone camera, using 2D detection, temporal smoothing, and aerial-perspective adaptation.
Prerequisites#
- Completed Tutorial 1
- Drone with camera
- Understanding of keypoint-based pose representation
Step 1: Set Up Detectors#
import {
type Keypoint2D,
type PoseDetection2D,
type ImageFrame,
type DetectorCapabilities,
type BoundingBox2D,
} from '@oya/core';
// Define detector capabilities
const capabilities: DetectorCapabilities = {
maxPersons: 10,
supportsGPU: true,
supportsBatching: true,
keypointCount: 17, // COCO format: 17 keypoints
formatName: 'COCO',
minInputSize: 256,
maxInputSize: 1280,
};
console.log('Pose detection configured:');
console.log(` Format: ${capabilities.formatName}`);
console.log(` Keypoints: ${capabilities.keypointCount}`);
console.log(` Max persons: ${capabilities.maxPersons}`);
console.log(` GPU: ${capabilities.supportsGPU}`);
Step 2: Process Frame with Keypoint Detection#
// COCO keypoint names (17 keypoints)
const COCO_KEYPOINTS = [
'nose',
'left_eye',
'right_eye',
'left_ear',
'right_ear',
'left_shoulder',
'right_shoulder',
'left_elbow',
'right_elbow',
'left_wrist',
'right_wrist',
'left_hip',
'right_hip',
'left_knee',
'right_knee',
'left_ankle',
'right_ankle',
];
// Simulate a detection result
function generatePoseDetection(
personId: number,
centerX: number,
centerY: number,
confidence: number,
timestampMs: number
): PoseDetection2D {
const keypoints: Keypoint2D[] = COCO_KEYPOINTS.map((name, i) => {
// Distribute keypoints around the center
const offsetX = (Math.random() - 0.5) * 0.15;
const offsetY = (i / COCO_KEYPOINTS.length - 0.3) * 0.4;
return {
x: centerX + offsetX,
y: centerY + offsetY,
confidence: confidence * (0.8 + Math.random() * 0.2),
name,
visible: Math.random() > 0.1, // 90% chance visible
};
});
return {
id: personId,
keypoints,
boundingBox: {
x: centerX - 0.1,
y: centerY - 0.2,
width: 0.2,
height: 0.5,
confidence,
},
confidence,
timestampMs,
};
}
// Detect poses in a frame
const frame: ImageFrame = {
width: 1920,
height: 1080,
timestampMs: Date.now(),
frameIndex: 0,
};
const detections = [
generatePoseDetection(0, 0.3, 0.5, 0.94, frame.timestampMs),
generatePoseDetection(1, 0.7, 0.5, 0.89, frame.timestampMs),
];
console.log(
`\nDetected ${detections.length} poses in frame ${frame.frameIndex}:`
);
for (const det of detections) {
const visibleCount = det.keypoints.filter((k) => k.visible).length;
console.log(
` Person #${det.id}: confidence=${det.confidence.toFixed(2)}, ` +
`visible_keypoints=${visibleCount}/${det.keypoints.length}`
);
}
Step 3: Aerial Adaptation#
// Drone-specific adjustments for aerial pose detection
console.log('\nAerial adaptation:');
console.log(' Altitude compensation:');
console.log(' - Higher altitude -> smaller apparent size');
console.log(' - Adjust confidence thresholds by altitude');
console.log(' Angle compensation:');
console.log(' - Top-down: only head/shoulders visible');
console.log(' - 30-60 deg: best for full body detection');
console.log(' - Adjust keypoint visibility expectations');
console.log(' Motion compensation:');
console.log(' - Drone movement adds apparent motion to subjects');
console.log(' - Subtract drone ego-motion from tracking');
Step 4: Multi-Person Tracking Across Frames#
// Temporal tracking maintains person IDs across frames
console.log('\nTemporal tracking:');
console.log(' Frame 0: Detect 2 persons -> assign IDs [0, 1]');
console.log(' Frame 1: Match detections to existing tracks');
console.log(' Frame 2: Person #0 occluded -> predict from Kalman filter');
console.log(' Frame 3: Person #0 reappears -> re-associate by appearance');
console.log(' Frame 4: New person enters -> assign ID [2]');
console.log(' Frame 5: Person #1 exits frame -> track ages out');
Step 5: Extract Pose Metrics#
// Analyze detected poses
for (const det of detections) {
const leftShoulder = det.keypoints.find((k) => k.name === 'left_shoulder');
const rightShoulder = det.keypoints.find((k) => k.name === 'right_shoulder');
const leftHip = det.keypoints.find((k) => k.name === 'left_hip');
const rightHip = det.keypoints.find((k) => k.name === 'right_hip');
if (leftShoulder && rightShoulder && leftHip && rightHip) {
// Shoulder width (normalized)
const shoulderWidth = Math.abs(rightShoulder.x - leftShoulder.x);
// Torso length (normalized)
const torsoLength = Math.abs(
(leftHip.y + rightHip.y) / 2 - (leftShoulder.y + rightShoulder.y) / 2
);
console.log(`\n Person #${det.id} metrics:`);
console.log(
` Shoulder width: ${(shoulderWidth * frame.width).toFixed(0)}px`
);
console.log(
` Torso length: ${(torsoLength * frame.height).toFixed(0)}px`
);
}
}
Expected Output#
Pose detection configured:
Format: COCO
Keypoints: 17
Max persons: 10
GPU: true
Detected 2 poses in frame 0:
Person #0: confidence=0.94, visible_keypoints=15/17
Person #1: confidence=0.89, visible_keypoints=16/17
Aerial adaptation:
Altitude compensation:
- Higher altitude -> smaller apparent size
- Adjust confidence thresholds by altitude
Angle compensation:
- Top-down: only head/shoulders visible
- 30-60 deg: best for full body detection
- Adjust keypoint visibility expectations
Motion compensation:
- Drone movement adds apparent motion to subjects
- Subtract drone ego-motion from tracking
Temporal tracking:
Frame 0: Detect 2 persons -> assign IDs [0, 1]
Frame 1: Match detections to existing tracks
Frame 2: Person #0 occluded -> predict from Kalman filter
Frame 3: Person #0 reappears -> re-associate by appearance
Frame 4: New person enters -> assign ID [2]
Frame 5: Person #1 exits frame -> track ages out
Person #0 metrics:
Shoulder width: 154px
Torso length: 194px
Person #1 metrics:
Shoulder width: 148px
Torso length: 187px
Next Steps#
- Tutorial 8: Deploy pose detection model to edge hardware
- Add
SkeletonTrackerfor skeleton-based analysis - Use
GestureControlfor drone control via hand gestures
Tutorial 8: Edge AI#
Objective#
Optimize a pose detection model for edge deployment on NVIDIA Jetson, configure inference parameters, and run real-time detection onboard a drone.
Prerequisites#
- NVIDIA Jetson Nano/Orin (or similar edge hardware)
- Pre-trained ONNX model
@oya/coreinstalled
Step 1: Detect Hardware Capabilities#
import {
type EdgePlatform,
type HardwareCapability,
type ResourceSnapshot,
type DeploymentState,
} from '@oya/core';
// Identify the target platform
const platform: EdgePlatform = 'jetson_orin_nano';
console.log('Edge hardware detected:');
console.log(` Platform: ${platform}`);
console.log(' CPU: ARM Cortex-A78AE (6 cores)');
console.log(' GPU: NVIDIA Ampere (1024 CUDA cores)');
console.log(' AI Performance: 40 TOPS');
console.log(' Power: 7-15W');
Step 2: Optimize Model#
import {
type QuantizationType,
type QuantizationMode,
type PruningStrategy,
type OptimizationPass,
} from '@oya/core';
// Model optimization pipeline
console.log('\nModel optimization pipeline:');
// Step 2a: FP16 conversion (2x speedup, minimal accuracy loss)
const fp16Config = {
quantization: 'fp16' as QuantizationType,
mode: 'static' as QuantizationMode,
};
console.log(' 1. FP16 conversion:');
console.log(' Speedup: ~2x on Ampere GPU');
console.log(' Accuracy loss: < 0.1%');
// Step 2b: INT8 quantization (4x speedup, requires calibration)
const int8Config = {
quantization: 'int8' as QuantizationType,
mode: 'static' as QuantizationMode,
};
console.log(' 2. INT8 quantization (optional):');
console.log(' Speedup: ~4x on Ampere GPU');
console.log(' Accuracy loss: < 1% with good calibration');
// Step 2c: Graph optimizations
const optimizations: OptimizationPass[] = [
'constant_folding',
'dead_code_elimination',
'operator_fusion',
'layout_optimization',
'memory_planning',
];
console.log(' 3. Graph optimizations:');
for (const opt of optimizations) {
console.log(` - ${opt}`);
}
// Step 2d: Pruning (optional, for extreme size reduction)
const pruneConfig = {
strategy: 'structured' as PruningStrategy,
sparsity: 0.3, // Remove 30% of filters
};
console.log(' 4. Structured pruning (optional):');
console.log(` Sparsity: ${pruneConfig.sparsity * 100}%`);
console.log(' Size reduction: ~30%');
Step 3: Configure Inference Engine#
import {
type InferenceBackend,
type InferenceDevice,
type PrecisionMode,
type MemoryMode,
type Tensor,
type TensorShape,
type InferenceResult,
} from '@oya/core';
// Configure TensorRT inference (fastest on Jetson)
const inferenceConfig = {
backend: 'tensorrt' as InferenceBackend,
device: 'gpu' as InferenceDevice,
precision: 'fp16' as PrecisionMode,
memory: 'pool' as MemoryMode,
batchSize: 1,
inputShape: [1, 3, 640, 640] as TensorShape, // NCHW
};
console.log('\nInference engine configured:');
console.log(` Backend: ${inferenceConfig.backend}`);
console.log(` Device: ${inferenceConfig.device}`);
console.log(` Precision: ${inferenceConfig.precision}`);
console.log(` Memory: ${inferenceConfig.memory}`);
console.log(` Input shape: [${inferenceConfig.inputShape.join(', ')}]`);
console.log(` Batch size: ${inferenceConfig.batchSize}`);
Step 4: Deploy Model#
// Deployment lifecycle
const deploymentSteps = [
{ state: 'pending' as DeploymentState, desc: 'Model queued for deployment' },
{
state: 'downloading' as DeploymentState,
desc: 'Downloading optimized model (45MB)',
},
{
state: 'installing' as DeploymentState,
desc: 'Installing TensorRT engine',
},
{
state: 'validating' as DeploymentState,
desc: 'Running validation inference',
},
{ state: 'active' as DeploymentState, desc: 'Model deployed and serving' },
];
console.log('\nDeployment progress:');
for (const step of deploymentSteps) {
console.log(` [${step.state}] ${step.desc}`);
}
Step 5: Monitor Resource Usage#
// Simulate resource monitoring during inference
const resourceSnapshots: ResourceSnapshot[] = [
{
cpuPercent: 25,
gpuPercent: 68,
memoryUsedBytes: 2.1 * 1024 ** 3,
memoryTotalBytes: 8 * 1024 ** 3,
gpuMemoryUsedBytes: 1.8 * 1024 ** 3,
gpuMemoryTotalBytes: 8 * 1024 ** 3,
diskUsedBytes: 12 * 1024 ** 3,
powerWatts: 11.2,
temperatureC: 58,
timestampMs: Date.now(),
},
];
const snap = resourceSnapshots[0];
console.log('\nResource usage during inference:');
console.log(` CPU: ${snap.cpuPercent}%`);
console.log(` GPU: ${snap.gpuPercent}%`);
console.log(
` Memory: ${(snap.memoryUsedBytes / 1024 ** 3).toFixed(1)}GB / ${(snap.memoryTotalBytes / 1024 ** 3).toFixed(0)}GB`
);
console.log(
` GPU Memory: ${(snap.gpuMemoryUsedBytes / 1024 ** 3).toFixed(1)}GB / ${(snap.gpuMemoryTotalBytes / 1024 ** 3).toFixed(0)}GB`
);
console.log(` Power: ${snap.powerWatts}W`);
console.log(` Temperature: ${snap.temperatureC}C`);
console.log(
` Thermal state: ${snap.temperatureC < 70 ? 'Normal' : 'Warning'}`
);
Expected Output#
Edge hardware detected:
Platform: jetson_orin_nano
CPU: ARM Cortex-A78AE (6 cores)
GPU: NVIDIA Ampere (1024 CUDA cores)
AI Performance: 40 TOPS
Power: 7-15W
Model optimization pipeline:
1. FP16 conversion:
Speedup: ~2x on Ampere GPU
Accuracy loss: < 0.1%
2. INT8 quantization (optional):
Speedup: ~4x on Ampere GPU
Accuracy loss: < 1% with good calibration
3. Graph optimizations:
- constant_folding
- dead_code_elimination
- operator_fusion
- layout_optimization
- memory_planning
4. Structured pruning (optional):
Sparsity: 30%
Size reduction: ~30%
Inference engine configured:
Backend: tensorrt
Device: gpu
Precision: fp16
Memory: pool
Input shape: [1, 3, 640, 640]
Batch size: 1
Deployment progress:
[pending] Model queued for deployment
[downloading] Downloading optimized model (45MB)
[installing] Installing TensorRT engine
[validating] Running validation inference
[active] Model deployed and serving
Resource usage during inference:
CPU: 25%
GPU: 68%
Memory: 2.1GB / 8GB
GPU Memory: 1.8GB / 8GB
Power: 11.2W
Temperature: 58C
Thermal state: Normal
Next Steps#
- Tutorial 9: Build custom missions with conditional waypoints
- Try different backends:
onnxruntime,tflite,openvino - Implement canary deployment for gradual model rollout
Tutorial 9: Custom Mission#
Objective#
Build a custom mission with conditional waypoints that adapt behavior based on battery level, weather, and detection results during flight.
Prerequisites#
- Completed Tutorials 1-2
- Understanding of mission execution subsystems
Step 1: Create Conditional Waypoints#
import {
ConditionalWaypointExecutor,
ConditionType,
type WaypointCondition,
type ConditionContext,
DynamicMissionModifier,
ModificationAction,
MissionPauseResumeController,
PauseResumeState,
MissionAbortController,
AbortBehavior,
type Waypoint,
type WaypointId,
WaypointAction,
BATTERY_LOW_PERCENT,
} from '@oya/core';
// Create conditional waypoint executor
const conditional = new ConditionalWaypointExecutor();
// Define conditions
const batteryCondition: WaypointCondition = {
type: ConditionType.Battery,
operator: 'greaterThan',
value: BATTERY_LOW_PERCENT, // 25%
description: 'Continue only if battery > 25%',
};
// Waypoints with conditions
const conditionalWaypoints: Array<{
waypoint: Waypoint;
condition?: WaypointCondition;
}> = [
{
waypoint: {
id: '11111111-1111-4111-8111-111111111111' as WaypointId,
position: { latitude: 47.3977, longitude: 8.5456, altitude: 50 },
speed: 5.0,
heading: 0,
holdTimeSeconds: 0,
actions: [{ action: WaypointAction.StartVideo }],
radius: 2.0,
},
// No condition - always execute
},
{
waypoint: {
id: '22222222-2222-4222-8222-222222222222' as WaypointId,
position: { latitude: 47.399, longitude: 8.5456, altitude: 60 },
speed: 5.0,
heading: 0,
holdTimeSeconds: 5,
actions: [
{ action: WaypointAction.TakePhoto },
{ action: WaypointAction.SetGimbal, params: { pitch: -45 } },
],
radius: 2.0,
},
condition: batteryCondition, // Only visit if battery > 25%
},
{
waypoint: {
id: '33333333-3333-4333-8333-333333333333' as WaypointId,
position: { latitude: 47.399, longitude: 8.5476, altitude: 60 },
speed: 3.0,
heading: 90,
holdTimeSeconds: 10,
actions: [
{ action: WaypointAction.OrbitPOI, params: { radius: 20, laps: 1 } },
],
radius: 2.0,
},
condition: {
type: ConditionType.Battery,
operator: 'greaterThan',
value: 40, // Only orbit if plenty of battery
description: 'Orbit POI only if battery > 40%',
},
},
];
console.log('Custom mission with conditional waypoints:');
for (let i = 0; i < conditionalWaypoints.length; i++) {
const cwp = conditionalWaypoints[i];
const condStr = cwp.condition
? `[IF: ${cwp.condition.description}]`
: '[ALWAYS]';
console.log(` WP${i + 1}: ${condStr}`);
for (const action of cwp.waypoint.actions) {
console.log(` -> ${action.action}`);
}
}
Step 2: Dynamic Mission Modification#
const modifier = new DynamicMissionModifier();
// Insert a new waypoint mid-mission (e.g., detected an object of interest)
console.log('\nDynamic modifications:');
console.log(' Scenario: AI detects object of interest during flight');
console.log(' Action: Insert investigation waypoint after current WP');
const investigateWP: Waypoint = {
id: '99999999-9999-4999-8999-999999999999' as WaypointId,
position: { latitude: 47.3985, longitude: 8.5465, altitude: 30 },
speed: 2.0,
heading: 0,
holdTimeSeconds: 15,
actions: [
{ action: WaypointAction.TakePhoto },
{ action: WaypointAction.ChangeAltitude, params: { altitude: 20 } },
{ action: WaypointAction.TakePhoto },
],
radius: 1.0,
};
console.log(` Inserted WP: investigate at alt=30m, then descend to 20m`);
console.log(` Actions: photo -> descend -> photo`);
Step 3: Mission Pause/Resume#
const pauseResume = new MissionPauseResumeController();
console.log('\nPause/Resume capability:');
console.log(' Mission can be paused at any time');
console.log(' Drone hovers in place during pause');
console.log(' Resume continues from current position');
// Simulate pause/resume flow
console.log(' Flow:');
console.log(' [Running] -> pause() -> [Paused] -> resume() -> [Running]');
Step 4: Mission Abort with Smart RTH#
import {
SmartRTHController,
RTHAltitudeSelector,
ReturnToHomeController,
RTHPhase,
} from '@oya/core';
const abort = new MissionAbortController();
const smartRTH = new SmartRTHController();
const altSelector = new RTHAltitudeSelector();
console.log('\nAbort and Smart RTH:');
console.log(' Abort behaviors:');
console.log(' - ReturnToHome: fly back to launch point');
console.log(' - LandInPlace: land at current position');
console.log(' - HoverAndWait: hover until pilot takes over');
console.log(' Smart RTH features:');
console.log(' - Obstacle-aware return path');
console.log(' - Wind-compensated altitude selection');
console.log(' - Battery-optimized route');
console.log(' RTH phases:');
console.log(' Climb -> Cruise -> Approach -> Descend -> Land');
Step 5: Specialized Mission Generators#
import {
SurveyMissionGenerator,
InspectionMissionGenerator,
SearchAndRescuePatternGenerator,
SearchPattern,
} from '@oya/core';
// Auto-generate a survey grid mission
const survey = new SurveyMissionGenerator();
console.log('\nSpecialized mission types:');
console.log(' Survey: auto-generated grid/crosshatch patterns');
console.log(' Inspection: waypoints around a structure');
console.log(' Search & Rescue patterns:');
console.log(' - Expanding square');
console.log(' - Parallel track');
console.log(' - Sector search');
console.log(' - Creeping line');
const sar = new SearchAndRescuePatternGenerator();
// Generate an expanding square search pattern
// sar.generate(SearchPattern.ExpandingSquare, ...)
Expected Output#
Custom mission with conditional waypoints:
WP1: [ALWAYS]
-> StartVideo
WP2: [IF: Continue only if battery > 25%]
-> TakePhoto
-> SetGimbal
WP3: [IF: Orbit POI only if battery > 40%]
-> OrbitPOI
Dynamic modifications:
Scenario: AI detects object of interest during flight
Action: Insert investigation waypoint after current WP
Inserted WP: investigate at alt=30m, then descend to 20m
Actions: photo -> descend -> photo
Pause/Resume capability:
Mission can be paused at any time
Drone hovers in place during pause
Resume continues from current position
Flow:
[Running] -> pause() -> [Paused] -> resume() -> [Running]
Abort and Smart RTH:
Abort behaviors:
- ReturnToHome: fly back to launch point
- LandInPlace: land at current position
- HoverAndWait: hover until pilot takes over
Smart RTH features:
- Obstacle-aware return path
- Wind-compensated altitude selection
- Battery-optimized route
RTH phases:
Climb -> Cruise -> Approach -> Descend -> Land
Specialized mission types:
Survey: auto-generated grid/crosshatch patterns
Inspection: waypoints around a structure
Search & Rescue patterns:
- Expanding square
- Parallel track
- Sector search
- Creeping line
Next Steps#
- Tutorial 10: See full example applications
- Combine conditional waypoints with subject tracking
- Add
MissionReplayControllerto replay recorded missions - Use
MissionTemplateLibraryto save reusable patterns
Tutorial 10: Example Applications#
Objective#
Reference guide showing how the individual tutorials combine into complete drone applications.
Application 1: Aerial Survey System#
Combines: Tutorials 1, 2, 9
+------------------+ +------------------+ +------------------+
| Mission Planning | | Flight Execution | | Post-Processing |
| | | | | |
| SurveyGenerator | --> | MissionUploader | --> | MediaDownloader |
| BatteryPlanner | | WaypointSequencer| | GeoTagging |
| WeatherCheck | | ProgressTracker | | Orthomosaic |
+------------------+ +------------------+ +------------------+
Key classes:
SurveyMissionGeneratorfor grid pattern creationBatteryAwarePlannerfor flight time optimizationMissionUploader+MissionValidatorfor safe uploadPhotoCaptureControllerfor timed photo captureMediaFileDownloaderfor post-flight retrieval
Application 2: Multi-Drone Film Production#
Combines: Tutorials 3, 4, 5
+------------------+ +------------------+ +------------------+
| Shot Planning | | Live Execution | | Post-Production |
| | | | | |
| ShotListAutomation| --> | DirectorMode | --> | ColorMatcher |
| EstablishShot | | MultiDroneSeq | | StreamRecorder |
| OrbitShot | | SubjectTracking | | TimecodeSyncer |
+------------------+ +------------------+ +------------------+
Key classes:
DirectorModefor autonomous multi-camera directionMultiDroneShotSequencerfor coordinated shot sequencesDeepSORTTracker+ActiveTrackModefor subject followingOrbitShotTrajectory+GimbalPOILockModefor cinematic orbitsTimecodeSynchronizerfor multi-camera syncBulletTimeCoordinatorfor freeze-frame effects
Application 3: Search and Rescue#
Combines: Tutorials 3, 7, 9
+------------------+ +------------------+ +------------------+
| Search Planning | | Detection Loop | | Response |
| | | | | |
| SARPatternGen | --> | PoseDetection2D | --> | EmergencyBeacon |
| SwarmMission | | HumanDetector | | CoordinateReport |
| TaskAllocation | | ThermalCamera | | MissionModify |
+------------------+ +------------------+ +------------------+
Key classes:
SearchAndRescuePatternGeneratorfor systematic search patternsSwarmMissionPlanner+AuctionTaskAssignerfor area partitioning- Pose detection pipeline for person identification
HumanDetectorfor human detection in various conditionsThermalCameraIntegrationfor night/low-visibility operationsDynamicMissionModifierfor real-time mission adaptation
Application 4: Infrastructure Inspection#
Combines: Tutorials 2, 6, 8
+------------------+ +------------------+ +------------------+
| Inspection Plan | | Autonomous Flight| | Analysis |
| | | | | |
| InspectionGen | --> | ObstacleAvoid | --> | ModelInference |
| PathPlanning | | VisualSLAM | | DefectDetection |
| GeofenceConfig | | DepthCamera | | ReportGeneration |
+------------------+ +------------------+ +------------------+
Key classes:
InspectionMissionGeneratorfor structure-aware waypointsDepthCameraDetector+ObstacleSensorFusionfor close-proximity flight- Visual SLAM for GPS-denied environments (bridges, tunnels)
- Edge AI inference for real-time defect detection
PolygonGeofencefor safety boundaries around structures
Application 5: Precision Agriculture#
Combines: Tutorials 2, 3, 8
+------------------+ +------------------+ +------------------+
| Field Mapping | | Data Collection | | Analysis |
| | | | | |
| SurveyGenerator | --> | MultispectralCam | --> | NDVIPipeline |
| SwarmPartition | | ThermalCamera | | ManagementZones |
| RTKPositioning | | GPSLogger | | PrescriptionMap |
+------------------+ +------------------+ +------------------+
Key classes:
SurveyMissionGeneratorfor field coverage patternsMultispectralCameraIntegrationfor NDVI/crop health imagingNDVICalculationPipelinefor vegetation index computationRTKPositionProviderfor centimeter-level accuracy- Swarm coordination for large-field coverage
Module Cross-Reference#
| Application | Foundation | Control | Autonomy | Coordination | Perception | Cinema | Safety | AI/ML |
|---|---|---|---|---|---|---|---|---|
| Aerial Survey | Yes | Yes | Yes | - | - | - | Yes | - |
| Film Production | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Search and Rescue | Yes | Yes | Yes | Yes | Yes | - | Yes | Yes |
| Infrastructure | Yes | Yes | Yes | - | Yes | - | Yes | Yes |
| Agriculture | Yes | Yes | Yes | Yes | Yes | - | Yes | - |
Appendix: Complete Import Map#
Quick reference for which classes come from which source module:
| Use Case | Key Imports | Source Module |
|---|---|---|
| PX4 connection | PX4SITLConnection, PX4MAVLinkTransport |
px4-autopilot.ts |
| PX4 control | PX4OffboardController, PX4OffboardMode |
px4-autopilot.ts |
| ArduPilot connection | APConnectionManager, APArmingController |
ardupilot.ts |
| DJI integration | DJIAppKeyManager, DJIAircraftConnection |
dji-sdk.ts |
| Mission execution | MissionUploader, MissionValidator, WaypointSequencer |
mission-execution.ts |
| Swarm management | SwarmManager, SwarmHeartbeatMonitor |
swarm-coordination.ts |
| Formation flying | VFormation, LineFormation, ReynoldsFlocking |
formation-flying.ts |
| Subject tracking | DeepSORTTracker, YOLO11Detector, ActiveTrackMode |
subject-tracking.ts |
| Cinematography | OrbitShotTrajectory, RevealShotTrajectory |
cinematography-trajectory.ts |
| Gimbal control | ThreeAxisGimbalController, GimbalPOILockMode |
gimbal-control.ts |
| Shot planning | RuleOfThirdsGrid, ShotSizeClassifier |
shot-planning.ts |
| Visual SLAM | ORBSLAMMonoIntegration, LoopClosureDetector |
visual-slam.ts |
| Path planning | RRTStarPlanner, AStarPlanner |
path-planning.ts |
| Obstacle avoidance | DepthCameraDetector, ObstacleSensorFusion |
obstacle-avoidance.ts |
| Pose detection | Keypoint2D, PoseDetection2D, ImageFrame |
pose-detection-2d.ts |
| Edge deployment | EdgePlatform, DeploymentState, ResourceSnapshot |
edge-deployment.ts |
| Model inference | InferenceBackend, Tensor, InferenceResult |
model-inference.ts |
| Model optimization | QuantizationType, PruningStrategy |
model-optimization.ts |
| Geofencing | PolygonGeofence, GeofenceZone |
geofencing.ts |
| Emergency procedures | EmergencyState, EmergencyTrigger |
emergency-procedures.ts |
| Gazebo simulation | createDefaultWorldConfig, GazeboWorldConfig |
gazebo-integration.ts |
| AirSim simulation | AirSimConnection, AirSimConnectionConfig |
airsim-integration.ts |
| Types and schemas | FlightMode, DroneState, TelemetrySchema |
types.ts |
| Math utilities | KalmanFilter, PIDController, MovingAverageFilter |
math-utils.ts |
| Kinematics | Vec3, Quat, RigidBody6DOF, MultirotorDynamics |
kinematics.ts |
| Constants | FAA_MAX_ALTITUDE_AGL_M, BATTERY_LOW_PERCENT |
constants.ts |
| Coordinate transforms | gpsToLocal, haversineDistance, bearing |
coordinate-transforms.ts |