This guide covers best practices for optimizing performance when using the Aja Motion Pipeline in production environments.
Client-Side Optimization#
Connection Pooling#
Reuse client instances instead of creating new ones for each request:
// GOOD: Single client instance
import { createClient } from '@aja/motion-pipeline-sdk';
const client = createClient({
baseUrl: process.env.AJA_API_URL,
auth: { type: 'api-key', apiKey: process.env.AJA_API_KEY },
});
export async function processVideo(videoUrl: string) {
return client.submitJob({
/* ... */
});
}
// BAD: Creating new client for each request
export async function processVideo(videoUrl: string) {
const client = createClient({
/* ... */
}); // Wasteful!
return client.submitJob({
/* ... */
});
}
Request Batching#
For bulk operations, batch requests where possible:
// GOOD: Batch submission
const videos = ['video1.mp4', 'video2.mp4', 'video3.mp4'];
const jobs = await Promise.all(
videos.map((url) =>
client.submitJob({
input: { type: 'url', url, filename: url.split('/').pop()! },
pipelineConfig: configId,
})
)
);
// BAD: Sequential submission with waits
for (const url of videos) {
const result = await client.submitJob({
/* ... */
});
await waitForCompletion(result.data.jobId); // Don't wait sequentially!
}
Caching#
Cache configuration and metadata that doesn't change frequently:
import { LRUCache } from 'lru-cache';
const configCache = new LRUCache<string, PipelineConfiguration>({
max: 100,
ttl: 1000 * 60 * 5, // 5 minute TTL
});
async function getConfig(name: string) {
const cached = configCache.get(name);
if (cached) return cached;
const result = await client.getConfigByName(name);
if (result.success) {
configCache.set(name, result.data);
return result.data;
}
throw new Error(result.error.message);
}
Job Submission Optimization#
Quality Preset Selection#
Choose the appropriate quality preset for your use case:
| Preset | Speed | Use Case |
|---|---|---|
| draft | 5-10x realtime | Prototyping, preview |
| standard | 2-5x realtime | General production |
| high | 0.5-2x realtime | Professional work |
| ultra | 0.2-0.5x realtime | Final delivery |
// For preview/validation - use draft
const preview = await client.submitJob({
input: videoInput,
pipelineConfig: {
qualityPreset: 'draft', // Fast feedback
domain: 'general',
},
});
// For final delivery - use high or ultra
const final = await client.submitJob({
input: videoInput,
pipelineConfig: {
qualityPreset: 'high', // Quality matters
domain: 'dance',
},
});
Video Pre-Processing#
Pre-process videos before submission to improve pipeline efficiency:
import ffmpeg from 'fluent-ffmpeg';
async function optimizeForPipeline(inputPath: string, outputPath: string) {
return new Promise((resolve, reject) => {
ffmpeg(inputPath)
// Resize to 1080p if larger
.size('1920x?')
// Normalize frame rate
.fps(30)
// Use efficient codec
.videoCodec('libx264')
.outputOptions(['-preset fast', '-crf 18', '-profile:v high'])
// Remove audio (not needed for motion)
.noAudio()
.output(outputPath)
.on('end', resolve)
.on('error', reject)
.run();
});
}
Chunking Long Videos#
For videos longer than 5 minutes, consider chunking:
interface VideoChunk {
path: string;
startTime: number;
duration: number;
overlap: number;
}
async function processLongVideo(videoPath: string, chunkDuration = 60) {
const chunks = await splitVideo(videoPath, chunkDuration, 2); // 2 second overlap
// Submit all chunks in parallel
const jobs = await Promise.all(
chunks.map((chunk, index) =>
client.submitJob({
input: {
type: 'storage',
storagePath: chunk.path,
filename: `chunk-${index}.mp4`,
},
pipelineConfig: configId,
options: {
tags: [`parent:${videoPath}`, `chunk:${index}`],
},
})
)
);
// Wait for all to complete
const results = await Promise.all(jobs.map((j) => waitForJob(j.data.jobId)));
// Stitch results together
return stitchMotionData(results, chunks);
}
Polling Optimization#
Exponential Backoff#
Use exponential backoff for status polling:
async function waitForJob(
jobId: JobId,
options: { maxWait?: number; onProgress?: (p: number) => void } = {}
) {
const maxWait = options.maxWait ?? 3600000; // 1 hour default
const startTime = Date.now();
let delay = 1000; // Start with 1 second
const maxDelay = 30000; // Max 30 seconds
while (Date.now() - startTime < maxWait) {
const result = await client.getJob(jobId);
if (!result.success) {
throw new Error(result.error.message);
}
const { status, progress } = result.data;
options.onProgress?.(progress ?? 0);
if (status === 'completed') return result.data;
if (status === 'failed') throw new Error(result.data.error ?? 'Job failed');
if (status === 'cancelled') throw new Error('Job cancelled');
await sleep(delay);
delay = Math.min(delay * 1.5, maxDelay); // Exponential backoff
}
throw new Error('Timeout waiting for job');
}
Use Webhooks in Production#
For production workloads, prefer webhooks over polling:
// GOOD: Webhook-based notification
await client.submitJob({
input: videoInput,
pipelineConfig: configId,
options: {
webhook: {
url: `${process.env.APP_URL}/webhooks/aja`,
events: ['completed', 'failed'],
},
},
});
// Handle webhook asynchronously
app.post('/webhooks/aja', async (req, res) => {
const { event, jobId, data } = req.body;
await processJobResult(jobId, event, data);
res.sendStatus(200);
});
Use Streaming for Real-Time Updates#
For real-time progress updates, use streaming instead of polling:
// GOOD: Streaming for real-time updates
const unsubscribe = client.streamJobProgress(jobId, {
onProgress: ({ data }) => {
updateProgressBar(data.progress);
},
onComplete: ({ data }) => {
handleComplete(data);
unsubscribe();
},
});
// BAD: Polling in tight loop
while (true) {
// Don't do this!
const status = await client.getJob(jobId);
updateProgressBar(status.data.progress);
await sleep(100); // Too frequent!
}
Concurrency Management#
Rate Limiting#
Implement client-side rate limiting:
import Bottleneck from 'bottleneck';
const limiter = new Bottleneck({
maxConcurrent: 10, // Max 10 concurrent requests
minTime: 100, // Min 100ms between requests
reservoir: 100, // 100 requests per minute
reservoirRefreshAmount: 100,
reservoirRefreshInterval: 60000,
});
export const rateLimitedClient = {
submitJob: limiter.wrap(client.submitJob.bind(client)),
getJob: limiter.wrap(client.getJob.bind(client)),
// ... wrap other methods
};
Job Priority Management#
Use priorities effectively to manage queue:
// Critical path jobs get high priority
await client.submitJob({
input: criticalVideoInput,
pipelineConfig: configId,
options: { priority: 'high' },
});
// Background jobs use low priority
await client.submitJob({
input: batchVideoInput,
pipelineConfig: configId,
options: { priority: 'background' },
});
Output Optimization#
Download Only Needed Formats#
Only download the formats you need:
// GOOD: Download specific formats
const formats = ['fbx', 'bvh'];
const downloads = await Promise.all(
formats.map((format) => client.generateDownloadUrl(jobId, { format }))
);
// BAD: Download all formats
const allOutputs = await client.listOutputs(jobId);
for (const output of allOutputs.data.outputs) {
await download(output); // Wasteful if you don't need all formats!
}
Streaming Downloads#
For large files, use streaming downloads:
import { createWriteStream } from 'fs';
import { pipeline } from 'stream/promises';
async function downloadOutput(jobId: JobId, format: string, destPath: string) {
const result = await client.generateDownloadUrl(jobId, { format });
if (!result.success) throw new Error(result.error.message);
const response = await fetch(result.data.url);
if (!response.ok) throw new Error('Download failed');
await pipeline(response.body as any, createWriteStream(destPath));
}
Monitoring and Metrics#
Health Checks#
Implement health checks for the pipeline:
async function checkPipelineHealth(): Promise<HealthStatus> {
try {
const [health, queue] = await Promise.all([
client.health(),
client.queueStats(),
]);
return {
healthy: health.success && health.data.status === 'healthy',
queueDepth: queue.success ? queue.data.waiting : -1,
latency: Date.now() - startTime,
};
} catch (error) {
return { healthy: false, error: error.message };
}
}
// Run periodically
setInterval(async () => {
const health = await checkPipelineHealth();
if (!health.healthy) {
alertOps('Pipeline health check failed', health);
}
}, 60000);
Metrics Collection#
Track key metrics for optimization:
import { Counter, Histogram, Gauge } from 'prom-client';
const jobsSubmitted = new Counter({
name: 'aja_jobs_submitted_total',
help: 'Total jobs submitted',
labelNames: ['preset', 'domain'],
});
const jobDuration = new Histogram({
name: 'aja_job_duration_seconds',
help: 'Job processing duration',
labelNames: ['preset', 'domain'],
buckets: [60, 120, 300, 600, 1800, 3600],
});
const queueDepth = new Gauge({
name: 'aja_queue_depth',
help: 'Current queue depth',
});
// Track job submission
async function submitWithMetrics(config: JobConfig) {
jobsSubmitted.inc({ preset: config.preset, domain: config.domain });
const startTime = Date.now();
const result = await client.submitJob(config);
return {
...result,
onComplete: () => {
jobDuration.observe(
{ preset: config.preset, domain: config.domain },
(Date.now() - startTime) / 1000
);
},
};
}
Best Practices Summary#
- Reuse client instances - Create once, use everywhere
- Choose appropriate quality presets - Don't over-quality for previews
- Use webhooks in production - Avoid polling when possible
- Implement exponential backoff - When polling is necessary
- Batch similar operations - Submit multiple jobs in parallel
- Pre-process videos - Optimize format and resolution
- Chunk long videos - Split videos longer than 5 minutes
- Rate limit requests - Protect against throttling
- Use priorities - Manage queue effectively
- Monitor metrics - Track performance over time