Comprehensive documentation for the shared infrastructure libraries in the Oshun
monorepo. All libraries reside under libs/shared/ and use the @oshun/*
package namespace. They are domain-agnostic and designed to be consumed by any
application or domain library within the monorepo.
Table of Contents#
- Overview
- Quick Reference Table
- Dependency Diagram
- Foundation Layer
- Observability Layer
- Data Layer
- Communication Layer
- Authentication and Security Layer
- AI Layer
- GPU and Compute Layer
- Infrastructure Layer
- Testing
- Configuration Reference
- Building and Development
Overview#
The @oshun/* libraries form a layered architecture of reusable infrastructure
code. Every library is designed with the following principles:
- Zero coupling between domains -- Shared libraries never import from domain-specific code (Lilith, Yemaya, Isis, etc.).
- Layered dependencies -- Libraries depend only on lower layers; there are no circular dependencies.
- Provider-agnostic abstractions -- External services (PostgreSQL, Redis, S3, LLMs) are wrapped behind interfaces that can be swapped at runtime.
- Type safety first -- Branded types, Zod schemas, and strict TypeScript prevent entire categories of bugs at compile time.
- Observable by default -- Logging, metrics, and tracing are baked into every library, not bolted on.
All libraries are workspace packages. Import them directly:
import { createLogger } from '@oshun/logging';
import { OshunError, NotFoundError } from '@oshun/errors';
import { sql, createPostgresClient } from '@oshun/database';
Quick Reference Table#
| Package | Purpose | Key Dependencies | Typical Consumers |
|---|---|---|---|
@oshun/types |
Foundational TypeScript types (zero deps) | None | Every package |
@oshun/errors |
Error classes with HTTP codes and Sentry integration | @oshun/types |
Every service |
@oshun/config |
Env var parsing, Zod validation, feature flags | @oshun/types, zod |
Every service |
@oshun/logging |
Pino-based structured logging with transports | pino |
Every service |
@oshun/metrics |
Prometheus counters, gauges, histograms | prom-client, @oshun/logging |
Services with /metrics |
@oshun/tracing |
OpenTelemetry distributed tracing, AWS X-Ray | @opentelemetry/*, @oshun/logging |
All backend services |
@oshun/health |
Health checks, Kubernetes probes, dependency aggregation | None | All backend services |
@oshun/database |
PostgreSQL client, Redis client, migrations, query builder | pg, ioredis, @oshun/metrics |
Data-access layers |
@oshun/cache |
Redis cache, in-memory cache, distributed locks, pub/sub | ioredis, @oshun/logging, @oshun/metrics |
Services with caching |
@oshun/storage |
S3/MinIO/local file storage, presigned URLs | @aws-sdk/* (peer) |
File upload services |
@oshun/queue |
BullMQ job queues with dead-letter support | bullmq |
Background workers |
@oshun/http-client |
HTTP client with retry, circuit breaker, interceptors | @opentelemetry/*, @oshun/errors |
Service-to-service calls |
@oshun/event-bus |
Redis-backed event pub/sub with patterns and DLQ | ioredis, nanoid |
Cross-domain events |
@oshun/websocket |
WebSocket server with rooms, presence, rate limits | ws, ioredis, @oshun/logging |
Real-time features |
@oshun/service-discovery |
Redis-backed service registry and health monitoring | ioredis, nanoid |
Microservice mesh |
@oshun/traefik-config |
API gateway route/service config, Traefik generation | None | DevOps / deployment |
@oshun/auth-primitives |
JWT (HMAC/RSA), sessions, passwords, API keys | jose |
@oshun/auth |
@oshun/auth |
Full auth service: login, registration, RBAC, OAuth | @oshun/auth-primitives |
API middleware |
@oshun/identity |
Cross-domain JWT, role checking, middleware | None | Route guards |
@oshun/security |
Audit logging, content scanning, secret management | @oshun/logging, @oshun/database, @oshun/cache |
Security-critical paths |
@oshun/rate-limit |
Sliding/fixed/token-bucket rate limiting, Hono middleware | ioredis, @oshun/logging, @oshun/errors |
API endpoints |
@oshun/ai |
Multi-provider LLM client (Anthropic, OpenAI, Google, xAI, Ollama) | SDKs, @oshun/logging, @oshun/errors |
AI features |
@oshun/ai-advanced |
Adapters, benchmarks, model selection, edge AI, research | @oshun/ai, @oshun/logging |
Advanced AI pipelines |
@oshun/runpod-client |
Type-safe RunPod Serverless API client | @opentelemetry/*, @oshun/http-client |
GPU workloads |
@oshun/gpu-dispatcher |
GPU job queue, cost tracking, circuit breaker | @oshun/runpod-client, @oshun/storage |
Image/video generation |
@oshun/infrastructure |
Performance manager, security manager, monitoring manager | @oshun/types, @oshun/logging, @oshun/errors |
Platform services |
@oshun/migration |
Cross-domain data migration runner and scripts | None | Database migrations |
@oshun/testing |
Mocks, fixtures, containers, Vitest config builders | pg, vitest (peer) |
Test suites |
Dependency Diagram#
The diagram below shows how the 28 shared libraries relate to one another. An
arrow A --> B means A depends on B.
+--------------+
| @oshun/types | (zero dependencies, foundation)
+------+-------+
|
+----------------+----------------+
| | |
+------v------+ +-----v------+ +------v------+
| @oshun/errors| |@oshun/config| |@oshun/health|
+------+------+ +-----+------+ +------+------+
| |
| +-----------+
| |
+------v----v--+
|@oshun/logging |
+------+-------+
|
+--------+--------+
| |
+----v------+ +------v-----+
|@oshun/ | |@oshun/ |
|metrics | |tracing |
+----+------+ +------+-----+
| |
| +-------------+
| |
+----v----v---+ +-------------+ +------------------+
|@oshun/ | |@oshun/ | |@oshun/ |
|database | |cache | |http-client |
+----+--------+ +------+------+ +--------+---------+
| | |
| +----------------+ |
| | |
+----v----v---+ +------------+ +-------------v-----------+
|@oshun/ | |@oshun/ | |@oshun/runpod-client |
|security | |queue | +-----------+-------------+
+-------------+ +------+-----+ |
| +-----------v-----------+
+------v------+ |@oshun/gpu-dispatcher |
|@oshun/ | +-----------------------+
|event-bus |
+------+------+
|
+----------+----------+
| |
+----------v-------+ +--------v-----------+
|@oshun/ | |@oshun/ |
|service-discovery | |websocket |
+-------------------+ +--------------------+
+-------------------+ +-------------------+
|@oshun/ | |@oshun/ |
|auth-primitives +---->+auth |
+-------------------+ +-------------------+
+-------------------+ +-------------------+
|@oshun/ai +---->+@oshun/ai-advanced |
+-------------------+ +-------------------+
Standalone / minimal dependencies:
@oshun/identity
@oshun/traefik-config
@oshun/migration
@oshun/infrastructure
@oshun/rate-limit
@oshun/testing
Foundation Layer#
@oshun/types#
Package: @oshun/types v0.1.0 Path: libs/shared/types/
Dependencies: None (zero external dependencies)
The foundational type system for the entire platform. Every other @oshun/*
package can depend on this. It has zero external dependencies by design.
Key Exports#
Branded ID Types -- Compile-time-safe identifiers that prevent accidentally
passing a UserID where a ProjectID is expected:
import { type UserID, type ProjectID, type AssetID } from '@oshun/types';
const userId = 'user-abc' as UserID;
const projectId = 'proj-123' as ProjectID;
// TypeScript error: Type 'UserID' is not assignable to type 'ProjectID'
Provided IDs: UserID, ProjectID, OrganizationID, TeamID, SessionID,
AssetID, ContentID, AgentID, RequestID, CorrelationID.
Result Types -- Functional error handling without exceptions:
import { type Result, type AsyncResult } from '@oshun/types';
function divide(a: number, b: number): Result<number, string> {
if (b === 0) return { success: false, error: 'Division by zero' };
return { success: true, data: a / b };
}
Types: Result<T, E>, AsyncResult<T, E>, ApiResult<T>, Success<T>,
Failure<E>.
Entity Types -- Composable traits for database entities:
import {
type BaseEntity,
type SoftDeletable,
type Versioned,
} from '@oshun/types';
// Compose entity shapes
type MyEntity = BaseEntity & SoftDeletable & Versioned;
Traits: BaseEntity, SoftDeletable, Versioned, Auditable, Orderable,
Taggable, Archivable, FullEntity.
Utility Types -- Advanced TypeScript generics: Nullable<T>,
DeepPartial<T>, DeepReadonly<T>, RequireAtLeastOne<T>,
RequireExactlyOne<T>, PartialExcept<T, K>, ValueOf<T>.
API Types -- Standard request/response contracts: ApiResponse<T>,
PaginatedResponse<T>, CursorPaginatedResponse<T>, ErrorResponse,
HealthStatus, WebhookPayload, RateLimitInfo.
Event Types -- DomainEvent, IntegrationEvent, EventHandler,
EventBus, Job, ActivityLog, AuditLog, Presence, RealtimeMessage.
Configuration Types -- ServiceConfig, DatabaseConfig, CacheConfig,
StorageConfig, AuthConfig, AIConfig, FeatureFlag, AppConfig.
Creative Production Types -- Available via @oshun/types/creative:
Script, Scene, Character, Storyboard, Asset.
Sub-path Exports#
| Import path | Contents |
|---|---|
@oshun/types |
All base, user, API, event, and config types |
@oshun/types/base |
Branded IDs, Result types, entity traits, utility types |
@oshun/types/user |
User roles, permissions, sessions, organizations |
@oshun/types/api |
API responses, pagination, health checks, webhooks |
@oshun/types/creative |
Script, Character, Storyboard, Asset types |
@oshun/errors#
Package: @oshun/errors v0.1.0 Path: libs/shared/errors/
Dependencies: @oshun/types Optional peer: @sentry/node (for
automatic error reporting)
A comprehensive error class hierarchy that maps domain errors to HTTP status codes, provides machine-readable error codes, and integrates with Sentry for production error tracking.
Error Hierarchy#
OshunError (base)
├── BadRequestError (400)
├── UnauthorizedError (401)
├── PaymentRequiredError (402)
├── ForbiddenError (403)
├── NotFoundError (404)
├── MethodNotAllowedError (405)
├── ConflictError (409)
├── GoneError (410)
├── UnprocessableEntityError (422)
├── TooManyRequestsError (429)
├── InternalServerError (500)
├── NotImplementedError (501)
├── BadGatewayError (502)
├── ServiceUnavailableError (503)
├── GatewayTimeoutError (504)
├── ValidationError
├── AuthenticationError / TokenExpiredError / SessionError
├── AuthorizationError
├── ResourceNotFoundError / DuplicateError / VersionConflictError
├── RateLimitError
├── ExternalServiceError / DatabaseError / CacheError / QueueError
├── AIServiceError / GenerationError
├── InvalidStateError / PreconditionError / LimitExceededError
└── AggregateError (wraps multiple errors)
Key Exports#
Error Codes -- Grouped constants for consistent identification:
import { ERROR_CODES, GENERAL_ERRORS, AUTH_ERRORS } from '@oshun/errors';
// ERROR_CODES.NOT_FOUND, ERROR_CODES.UNAUTHORIZED, etc.
// AUTH_ERRORS.TOKEN_EXPIRED, AUTH_ERRORS.INVALID_CREDENTIALS, etc.
Throwing Errors:
import { NotFoundError, ValidationError, ConflictError } from '@oshun/errors';
throw new NotFoundError('User not found');
throw new ValidationError('Invalid input', {
fields: { email: ['Must be a valid email'] },
});
throw new ConflictError('Username already taken');
Type Guards and Utilities:
import {
isOshunError,
isClientError,
isServerError,
wrapError,
ensureError,
catchAsync,
tryCatch,
getErrorChain,
createErrorResponse,
createSafeErrorResponse,
formatErrorForLogging,
} from '@oshun/errors';
// Wrap unknown caught values into OshunError
try {
await riskyOperation();
} catch (err) {
throw wrapError(err, 'Failed during operation');
}
// Async route handler wrapper (catches thrown errors)
app.get(
'/users/:id',
catchAsync(async (req, res) => {
const user = await getUser(req.params.id);
res.json(user);
})
);
// Result-style error handling
const [error, result] = await tryCatch(fetchUser(id));
if (error) {
return createSafeErrorResponse(error); // strips internal details
}
@oshun/config#
Package: @oshun/config v0.1.0 Path: libs/shared/config/
Dependencies: @oshun/types, zod
Type-safe configuration management with Zod validation, environment variable parsing, and feature flag support.
Environment Utilities#
import {
getEnv,
getEnvRequired,
getEnvNumber,
getEnvBool,
getEnvArray,
getEnvJson,
getEnvUrl,
getEnvironment,
isProduction,
isDevelopment,
isTest,
getLogLevel,
getPort,
} from '@oshun/config';
const apiKey = getEnv('API_KEY'); // string | undefined
const dbUrl = getEnvRequired('DATABASE_URL'); // string (throws if missing)
const port = getEnvNumber('PORT', 3000); // number with default
const debug = getEnvBool('DEBUG', false); // boolean with default
const origins = getEnvArray('CORS_ORIGINS', ','); // string[]
const config = getEnvJson<MyConfig>('APP_CONFIG'); // parsed JSON
Zod Configuration Schemas#
Every service configuration has a matching Zod schema for runtime validation:
import {
databaseConfigSchema,
serverConfigSchema,
redisConfigSchema,
aiConfigSchema,
validateConfig,
} from '@oshun/config';
// Validate environment against schema
const dbConfig = validateConfig(databaseConfigSchema, {
url: process.env.DATABASE_URL,
pool: { min: 2, max: 10 },
});
Available schemas: serverConfigSchema, databaseConfigSchema,
redisConfigSchema, storageConfigSchema, authConfigSchema,
loggingConfigSchema, tracingConfigSchema, metricsConfigSchema,
aiConfigSchema, emailConfigSchema, rateLimitConfigSchema,
serviceConfigSchema.
Configuration Loaders#
Load fully validated configs from environment in one call:
import {
loadServiceConfig,
loadDatabaseConfig,
loadRedisConfig,
loadStorageConfig,
loadAuthConfig,
loadAIConfig,
createConfigLoader,
} from '@oshun/config';
const service = loadServiceConfig('my-service');
// Returns: { name, version, port, host, env, logging, database, redis, ... }
// Custom loader for domain-specific config
const loadMyConfig = createConfigLoader(myConfigSchema, (env) => ({
apiKey: env.MY_API_KEY,
maxRetries: parseInt(env.MAX_RETRIES || '3'),
}));
Feature Flags#
import {
isFeatureEnabled,
getFeatureValue,
shouldUseRunPod,
getRunPodConfig,
FeatureFlags,
} from '@oshun/config';
if (isFeatureEnabled('USE_RUNPOD_GPU')) {
const rpConfig = getRunPodConfig();
// rpConfig.apiKey, rpConfig.endpointId, etc.
}
Observability Layer#
@oshun/logging#
Package: @oshun/logging v0.0.1 Path: libs/shared/logging/
Dependencies: pino, pino-pretty Optional peers:
@opentelemetry/api, rotating-file-stream
High-performance structured logging built on Pino with multiple transports, sampling strategies, and framework middleware.
Core Logger#
import { createLogger, log, OshunLogger } from '@oshun/logging';
// Create a service-specific logger
const logger = createLogger({
service: 'api-gateway',
version: '1.2.0',
level: 'info', // 'trace' | 'debug' | 'info' | 'warn' | 'error' | 'fatal'
});
logger.info('Server started', { port: 3000 });
logger.error(new Error('Connection failed'), 'Database error');
logger.warn('Deprecation', { endpoint: '/v1/old' });
// Child logger with persistent context
const reqLogger = logger.child({ requestId: 'req-abc-123' });
reqLogger.info('Processing request'); // includes requestId in every line
// Global logger (initialized once, used everywhere)
import { initializeLogger, log } from '@oshun/logging';
initializeLogger({ service: 'worker', level: 'debug' });
log.info('Worker started');
Sub-path Exports#
Transports (@oshun/logging/transports):
import {
createFileTransport,
createElasticsearchTransport,
createHttpTransport,
createTcpTransport,
createConsoleTransport,
} from '@oshun/logging/transports';
const logger = createLogger({
service: 'api',
transports: [
createFileTransport({ path: './logs/app.log', rotate: true }),
createElasticsearchTransport({
url: 'https://es.example.com',
index: 'app-logs',
apiKey: '...',
}),
],
});
Middleware (@oshun/logging/middleware):
import { expressRequestLogger } from '@oshun/logging/middleware';
app.use(
expressRequestLogger(logger, {
ignorePaths: ['/health', '/metrics'],
logBody: true,
})
);
Sampling (@oshun/logging/sampling):
import {
createFixedRateSampler,
createSamplingTransport,
} from '@oshun/logging/sampling';
const sampler = createFixedRateSampler({
rate: 0.1, // 10% of log entries
alwaysLogLevels: ['error', 'fatal'],
});
Also provides AdaptiveSampler, PrioritySampler, ConsistentHashSampler, and
PrivacyConfig for PII redaction.
@oshun/metrics#
Package: @oshun/metrics v0.1.0 Path: libs/shared/metrics/
Dependencies: prom-client, @oshun/logging Peer: @oshun/types
Prometheus-compatible metrics collection with pre-built helpers for HTTP, database, cache, AI, and queue monitoring.
Registry and Metric Types#
import {
createRegistry,
counter,
gauge,
histogram,
summary,
initializeRegistry,
} from '@oshun/metrics';
// Initialize a global registry
initializeRegistry({ prefix: 'myservice_' });
// Create metrics (auto-registered in global registry)
const requestsTotal = counter({
name: 'http_requests_total',
help: 'Total HTTP requests',
labelNames: ['method', 'path', 'status'],
});
const requestDuration = histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration',
labelNames: ['method', 'path'],
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});
const activeConnections = gauge({
name: 'active_connections',
help: 'Current active connections',
});
Pre-built Metric Helpers#
import {
createHttpMetrics,
recordHttpRequest,
createDbMetrics,
recordDbQuery,
createCacheMetrics,
recordCacheOperation,
createAiMetrics,
recordAiRequest,
createQueueMetrics,
recordJobCompletion,
createTimer,
measureDuration,
} from '@oshun/metrics';
// HTTP metrics (auto-creates counter + histogram)
const httpMetrics = createHttpMetrics({ prefix: 'api_' });
recordHttpRequest(httpMetrics, {
method: 'GET',
path: '/users',
status: 200,
durationMs: 45,
});
// Timer utility
const timer = createTimer();
await doWork();
const elapsed = timer(); // returns milliseconds
Metrics Server#
import { createMetricsServer, startMetricsServer } from '@oshun/metrics';
// Expose /metrics endpoint on port 9090
const server = createMetricsServer({ port: 9090, path: '/metrics' });
await startMetricsServer();
Standard metric name constants: HTTP_METRICS, DB_METRICS, CACHE_METRICS,
AI_METRICS, QUEUE_METRICS. Histogram bucket presets: HISTOGRAM_BUCKETS.
@oshun/tracing#
Package: @oshun/tracing v0.1.0 Path: libs/shared/tracing/
Dependencies: @opentelemetry/api, @opentelemetry/sdk-trace-node,
@opentelemetry/exporter-trace-otlp-http, @opentelemetry/propagator-aws-xray
Peers: @oshun/types, @oshun/logging, hono (optional)
Full OpenTelemetry distributed tracing with W3C Trace Context propagation, framework middleware, and AWS X-Ray support.
Tracer Setup#
import { createTracer, initializeTracer } from '@oshun/tracing';
// Initialize global tracer
initializeTracer({
serviceName: 'api-gateway',
serviceVersion: '1.0.0',
endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
sampler: { type: 'ratio', ratio: 0.1 }, // Sample 10%
});
Function Wrapping and Decorators#
import {
withSpan,
traceServiceCall,
traceDatabaseCall,
traceExternalApiCall,
traceAiCall,
traceJob,
traceBatch,
} from '@oshun/tracing';
// Wrap any async function with a span
const user = await withSpan('get-user', async (span) => {
span.setAttribute('user.id', userId);
return await db.getUser(userId);
});
// Trace a database call
const rows = await traceDatabaseCall(
{
system: 'postgresql',
operation: 'SELECT',
table: 'users',
},
() => db.query('SELECT * FROM users')
);
// Trace an AI request
const response = await traceAiCall(
{
model: 'claude-3-opus',
provider: 'anthropic',
},
() => llm.complete(prompt)
);
Framework Middleware#
import {
createExpressTracingMiddleware,
createFastifyTracingPlugin,
createHonoTracingMiddleware,
createKoaTracingMiddleware,
createTracedFetch,
} from '@oshun/tracing';
// Express
app.use(createExpressTracingMiddleware({ serviceName: 'api' }));
// Hono
app.use('*', createHonoTracingMiddleware({ serviceName: 'api' }));
// Outbound: propagate trace context through fetch calls
const tracedFetch = createTracedFetch(fetch);
const response = await tracedFetch('https://internal-service/api');
Context Propagation#
import {
extractTraceContext,
injectTraceContext,
generateTraceId,
generateSpanId,
generateCorrelationId,
} from '@oshun/tracing';
// Extract from incoming headers
const context = extractTraceContext(request.headers);
// Inject into outgoing headers
const headers = {};
injectTraceContext(headers);
AWS X-Ray Support#
Available via @oshun/tracing/xray:
import {
createXRayTracerConfig,
otelToXRayTraceId,
fetchECSMetadata,
} from '@oshun/tracing/xray';
const config = createXRayTracerConfig({
serviceName: 'api',
region: 'us-east-1',
});
@oshun/health#
Package: @oshun/health v0.0.1 Path: libs/shared/health/
Dependencies: None
Health check framework for microservices with Kubernetes-compatible liveness, readiness, and startup probes.
Health Manager#
import {
createHealthManager,
healthy,
degraded,
unhealthy,
} from '@oshun/health';
const health = createHealthManager({ cacheDuration: 5000 });
// Register health checks
health.register('database', async () => {
try {
await db.query('SELECT 1');
return healthy({ latencyMs: 5 });
} catch (err) {
return unhealthy({ error: err.message });
}
});
health.register('redis', async () => {
const latency = await redis.ping();
return latency < 100
? healthy({ latencyMs: latency })
: degraded({ latencyMs: latency });
});
// Run all checks
const report = await health.check();
// { status: 'healthy', checks: [...], uptime: 3600, ... }
Kubernetes Probes#
import { createProbeHandlers } from '@oshun/health';
const probes = createProbeHandlers(healthManager);
// Attach to your HTTP server
app.get('/healthz', probes.liveness);
app.get('/readyz', probes.readiness);
app.get('/startupz', probes.startup);
Dependency Aggregator#
Pre-built health checks for common dependencies:
import {
createDependencyAggregator,
createDatabaseCheck,
createCacheCheck,
createQueueCheck,
createApiCheck,
} from '@oshun/health';
const aggregator = createDependencyAggregator();
aggregator.add(createDatabaseCheck('postgres', pgClient));
aggregator.add(createCacheCheck('redis', redisClient));
aggregator.add(
createApiCheck('payment-service', 'https://payments.internal/health')
);
Data Layer#
@oshun/database#
Package: @oshun/database v0.0.1 Path: libs/shared/database/
Dependencies: pg, postgres, ioredis, knex, @oshun/metrics
Optional peer: @prisma/client
Comprehensive database utilities for PostgreSQL and Redis with connection pooling, transactions, migrations, and a SQL-safe query builder.
PostgreSQL Client#
import {
createPostgresClient,
createPostgresClientFromUrl,
createPostgresClientFromEnv,
} from '@oshun/database';
const db = createPostgresClient({
host: 'localhost',
port: 5432,
database: 'oshun_dev',
user: 'oshun',
password: 'oshun_dev',
pool: { min: 2, max: 10 },
});
const result = await db.query('SELECT * FROM users WHERE id = $1', [
'user-123',
]);
SQL Tagged Template (Query Builder)#
The sql tagged template provides safe parameterized queries:
import { sql, sqlRaw, sqlJoin, sqlEmpty } from '@oshun/database';
const userId = 'user-123';
const query = sql`SELECT * FROM users WHERE id = ${userId}`;
// { text: 'SELECT * FROM users WHERE id = $1', values: ['user-123'] }
// Raw SQL (no parameterization -- use for trusted identifiers only)
const table = sqlRaw('users');
const q = sql`SELECT * FROM ${table} WHERE active = ${true}`;
// Join multiple conditions
const conditions = [sql`status = ${status}`, sql`role = ${role}`];
const where = sqlJoin(conditions, ' AND ');
const fullQuery = sql`SELECT * FROM users WHERE ${where}`;
Higher-level builders:
import {
buildWhereClause,
buildOrderByClause,
buildPaginationClause,
buildInsertStatement,
buildUpdateStatement,
buildDeleteStatement,
sanitizeIdentifier,
quoteIdentifier,
} from '@oshun/database';
Redis Client#
import {
createRedisClient,
createRedisClientFromUrl,
createRedisClusterClient,
} from '@oshun/database';
const redis = createRedisClient({ host: 'localhost', port: 6379 });
await redis.set('key', 'value');
const val = await redis.get('key');
Transactions#
import {
withTransaction,
withSerializableTransaction,
withAdvisoryLock,
withSavepoint,
} from '@oshun/database';
// Basic transaction
await withTransaction(db, async (tx) => {
await tx.query(sql`INSERT INTO orders (...) VALUES (...)`);
await tx.query(sql`UPDATE inventory SET quantity = quantity - 1 WHERE ...`);
});
// Serializable isolation (strongest consistency)
await withSerializableTransaction(db, async (tx) => {
/* ... */
});
// Advisory locks for distributed mutual exclusion
await withAdvisoryLock(db, 12345, async () => {
// Only one process can hold this lock at a time
});
Migrations#
import {
createMigrationRunner,
createSqlMigration,
MigrationRunner,
} from '@oshun/database';
const runner = createMigrationRunner(db, { schema: 'public' });
runner.addMigration(
createSqlMigration(
'001_create_users',
'CREATE TABLE users (id UUID PRIMARY KEY, email TEXT UNIQUE)',
'DROP TABLE users'
)
);
const summary = await runner.up(); // Apply all pending
await runner.down(1); // Rollback last
Health Checks and Metrics#
import {
checkPostgresHealth,
checkRedisHealth,
createHealthMonitor,
instrumentPostgresClient,
PoolStatsMonitor,
} from '@oshun/database';
// One-shot health check
const pgHealth = await checkPostgresHealth(db);
const redisHealth = await checkRedisHealth(redis);
// Continuous monitoring
const monitor = createHealthMonitor({ postgres: db, redis });
monitor.start(5000); // Check every 5 seconds
monitor.on('unhealthy', (info) => console.error('Database unhealthy:', info));
// Instrument for Prometheus metrics
instrumentPostgresClient(db, metricsRegistry);
Connection String Utilities#
import {
parsePostgresConnectionString,
buildPostgresConnectionString,
maskPostgresConnectionString,
detectDatabaseType,
validateConnectionString,
} from '@oshun/database';
const parsed = parsePostgresConnectionString(
'postgresql://user:pass@host:5432/db'
);
// { host: 'host', port: 5432, database: 'db', user: 'user', password: 'pass' }
const masked = maskPostgresConnectionString(url);
// 'postgresql://user:****@host:5432/db'
@oshun/cache#
Package: @oshun/cache v0.1.0 Path: libs/shared/cache/
Dependencies: ioredis, @oshun/metrics, @oshun/logging Peer:
@oshun/types
Multi-layer caching with Redis, in-memory cache, distributed locks, circuit breaker, pub/sub, and invalidation management.
Redis Client#
import {
createRedisClient,
createRedisClusterClient,
createCacheClientFromEnv,
initializeCacheClient,
} from '@oshun/cache';
const cache = createRedisClient({
host: 'localhost',
port: 6379,
keyPrefix: 'myapp:',
});
await cache.set('user:123', JSON.stringify(user), { ttl: 300 });
const data = await cache.get('user:123');
await cache.del('user:123');
Cache Wrappers (Fetch-Through Pattern)#
import { withRedisCache, withMemoryCache, cachedMethod } from '@oshun/cache';
// Fetch-through cache: returns cached value or calls function and caches result
const user = await withRedisCache(cache, 'user:123', 300, async () => {
return await db.getUser('123');
});
// In-memory cache (no Redis required)
const config = await withMemoryCache('app-config', 60, async () => {
return await loadConfig();
});
// Decorator pattern for class methods
class UserService {
@cachedMethod({ key: (id) => `user:${id}`, ttl: 300 })
async getUser(id: string) {
return await db.findUser(id);
}
}
Key Builder#
Standardized cache key construction to prevent collisions:
import {
cacheKey,
userKey,
sessionKey,
contentKey,
rateLimitKey,
lockKey,
keyPattern,
parseKey,
tempKey,
} from '@oshun/cache';
userKey('123'); // 'user:123'
sessionKey('sess-1'); // 'session:sess-1'
contentKey('article', '456'); // 'content:article:456'
lockKey('process-order', 'ord-789'); // 'lock:process-order:ord-789'
keyPattern('user', '*'); // 'user:*'
tempKey('import'); // 'temp:import:a1b2c3'
In-Memory Cache#
import { createMemoryCache, OshunMemoryCache } from '@oshun/cache';
const memCache = createMemoryCache({ maxSize: 1000, ttl: 60 });
memCache.set('key', value);
const hit = memCache.get('key');
const stats = memCache.stats(); // { hits, misses, size, hitRate }
Distributed Locks#
import { createLockManager } from '@oshun/cache';
const locks = createLockManager(redisClient);
const lock = await locks.acquire('order-processing:ord-123', {
ttl: 30000, // 30 seconds
retries: 5,
retryDelay: 200,
});
try {
await processOrder('ord-123');
} finally {
await lock.release();
}
Circuit Breaker#
import { createCircuitBreaker, withCircuitBreaker } from '@oshun/cache';
const breaker = createCircuitBreaker({
threshold: 5, // Open after 5 failures
resetTimeout: 30000, // Try again after 30 seconds
halfOpenMax: 3, // Allow 3 test requests in half-open
});
const result = await withCircuitBreaker(breaker, async () => {
return await externalService.call();
});
Pub/Sub#
import { createPubSubClient } from '@oshun/cache';
const pubsub = createPubSubClient(redisClient);
await pubsub.subscribe('user:updated', async (message) => {
console.log('User updated:', message);
});
await pubsub.publish('user:updated', { userId: '123', changes: ['email'] });
Invalidation Manager#
import {
createInvalidationManager,
invalidateKey,
invalidatePattern,
invalidateByPrefix,
} from '@oshun/cache';
const invalidation = createInvalidationManager(cache, pubsub);
await invalidateKey(cache, 'user:123');
await invalidatePattern(cache, 'user:*'); // Wildcard invalidation
await invalidateByPrefix(cache, 'session:'); // All sessions
Cache Metrics#
import { instrumentCacheClient, CacheStatsTracker } from '@oshun/cache';
const instrumented = instrumentCacheClient(cache, metricsRegistry);
// Automatically records hit/miss rates, latency, etc.
@oshun/storage#
Package: @oshun/storage v0.0.1 Path: libs/shared/storage/ Optional
peers: @aws-sdk/client-s3, @aws-sdk/s3-request-presigner,
@aws-sdk/lib-storage
Object storage abstraction supporting S3, MinIO, and local filesystem.
S3/MinIO Client#
import { createS3Client, createMinioClient } from '@oshun/storage';
const storage = createS3Client({
bucket: 'oshun-assets',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
});
// For local development with MinIO
const localStorage = createMinioClient({
bucket: 'dev-assets',
endpoint: 'http://localhost:9000',
accessKeyId: 'minioadmin',
secretAccessKey: 'minioadmin',
});
File Operations#
// Upload
const result = await storage.upload({
key: 'uploads/avatar.png',
body: fileBuffer,
contentType: 'image/png',
metadata: { userId: '123' },
});
// Download
const file = await storage.download('uploads/avatar.png');
// List files
const files = await storage.listObjects({
prefix: 'uploads/',
maxKeys: 100,
});
// Copy
await storage.copy({
sourceKey: 'temp/file.pdf',
destinationKey: 'permanent/file.pdf',
});
// Delete
await storage.delete('temp/file.pdf');
Presigned URLs#
import {
generateGpuUploadUrl,
generateGpuDownloadUrl,
generateBatchUploadUrls,
createRunPodUploadInstructions,
} from '@oshun/storage';
// Presigned upload URL (for GPU results)
const { url, key } = await generateGpuUploadUrl(storage, {
jobId: 'job-123',
outputType: 'image/png',
expiresIn: 3600,
});
// Presigned download URL
const downloadUrl = await generateGpuDownloadUrl(storage, {
key: 'results/job-123/output.png',
expiresIn: 7200,
});
Utility Functions#
import {
generateFileKey,
sanitizeFilename,
getMimeType,
isImageMimeType,
formatBytes,
parseBytes,
computeSha256,
createFileManifest,
} from '@oshun/storage';
const key = generateFileKey({ prefix: 'uploads', filename: 'photo.jpg' });
const mime = getMimeType('photo.jpg'); // 'image/jpeg'
const size = formatBytes(1048576); // '1.00 MB'
const hash = computeSha256(buffer);
Local Storage (Testing/Development)#
import { createLocalStorageClient } from '@oshun/storage';
const localStorage = createLocalStorageClient({
basePath: '/tmp/oshun-storage',
baseUrl: 'http://localhost:3000/files',
});
@oshun/queue#
Package: @oshun/queue v0.1.0 Path: libs/shared/queue/
Dependencies: bullmq Peer: @oshun/types
BullMQ-based job queuing with priorities, retries, dead-letter queues, and in-memory implementations for testing.
Queue and Worker#
import { createQueue, createWorker } from '@oshun/queue';
// Create a queue
const queue = createQueue('email-jobs', {
redis: { host: 'localhost', port: 6379 },
});
// Add jobs with priority and retry
await queue.add(
'send-welcome',
{
userId: '123',
template: 'welcome',
},
{
priority: 'high',
retry: { maxRetries: 3, backoff: 'exponential', delay: 1000 },
}
);
// Create a worker
const worker = createWorker(
'email-jobs',
async (job, ctx) => {
await sendEmail(job.data.userId, job.data.template);
ctx.reportProgress(100);
return { sent: true };
},
{
concurrency: 5,
redis: { host: 'localhost', port: 6379 },
}
);
Dead Letter Queue#
import { createDeadLetterQueue } from '@oshun/queue';
const dlq = createDeadLetterQueue('email-jobs', {
maxEntries: 10000,
retentionDays: 30,
});
// List failed jobs
const failed = await dlq.list({ limit: 50 });
// Retry a failed job
await dlq.retry(failed[0].id);
// Get DLQ statistics
const stats = await dlq.stats();
// { total: 42, byReason: { timeout: 12, validation: 30 }, ... }
In-Memory Queue (Testing)#
import {
createMemoryQueue,
createMemoryWorker,
createMemoryDeadLetterQueue,
clearAllMemoryStores,
} from '@oshun/queue';
const queue = createMemoryQueue('test-jobs');
const worker = createMemoryWorker('test-jobs', async (job) => {
return { processed: true };
});
// In test teardown
clearAllMemoryStores();
Constants and Named Queues#
import { QUEUE_NAMES, PRIORITY_VALUES } from '@oshun/queue';
// QUEUE_NAMES: standard queue names used across the platform
// PRIORITY_VALUES: { low: 10, normal: 5, high: 2, critical: 1 }
Communication Layer#
@oshun/http-client#
Package: @oshun/http-client v0.1.0 Path: libs/shared/http-client/
Dependencies: @opentelemetry/api, @opentelemetry/semantic-conventions
Peers: @oshun/errors, @oshun/logging, @oshun/tracing
Production-grade HTTP client with circuit breaker, retry policies, interceptor chains, timeout management, and OpenTelemetry tracing.
Creating Clients#
import {
createHttpClient,
createSimpleHttpClient,
createResilientHttpClient,
} from '@oshun/http-client';
// Full-featured client
const client = createHttpClient({
baseUrl: 'https://api.example.com',
timeout: 30000,
retry: { maxRetries: 3, backoff: 'exponential' },
circuitBreaker: { threshold: 5, resetTimeout: 30000 },
});
// Simple client (no retry, no circuit breaker)
const simple = createSimpleHttpClient({ baseUrl: 'https://api.example.com' });
// Resilient client (aggressive retry + circuit breaker)
const resilient = createResilientHttpClient({
baseUrl: 'https://flaky-service.internal',
});
Making Requests#
const users = await client.get('/users', { params: { page: 1, limit: 20 } });
const created = await client.post('/users', {
name: 'Jane',
email: 'jane@example.com',
});
const updated = await client.put('/users/123', { name: 'Jane Doe' });
await client.delete('/users/123');
Retry Policies#
import {
retry,
createRetryPolicy,
defaultRetryPolicy,
aggressiveRetryPolicy,
conservativeRetryPolicy,
} from '@oshun/http-client';
// Custom retry
const result = await retry(
async () => {
return await unstableService.call();
},
{
maxRetries: 5,
backoff: 'exponential',
delay: 1000,
maxDelay: 30000,
}
);
Circuit Breaker#
import {
createCircuitBreaker,
createCircuitBreakerRegistry,
withCircuitBreaker,
} from '@oshun/http-client';
// Per-service circuit breakers
const registry = createCircuitBreakerRegistry();
const breaker = registry.get('payment-service', {
threshold: 3,
resetTimeout: 60000,
});
const result = await withCircuitBreaker(breaker, async () => {
return await paymentService.charge(amount);
});
Interceptors#
import {
createAuthInterceptor,
createCorrelationIdInterceptor,
createTimingInterceptor,
createLoggingInterceptors,
createStandardInterceptors,
} from '@oshun/http-client';
const client = createHttpClient({
baseUrl: 'https://api.internal',
interceptors: createStandardInterceptors(logger),
// Includes: auth, correlation ID, timing, logging, error handling
});
Timeout Management#
import {
withTimeout,
createDeadline,
getRemainingTime,
combineSignals,
} from '@oshun/http-client';
const result = await withTimeout(5000, async (signal) => {
return await fetch(url, { signal });
});
@oshun/event-bus#
Package: @oshun/event-bus v0.1.0 Path: libs/shared/event-bus/
Dependencies: ioredis, nanoid Peer: @oshun/types
Cross-domain event bus using Redis pub/sub with type-safe events, pattern subscriptions, retry with backoff, dead-letter queues, and event persistence.
Publishing and Subscribing#
import { createEventBus, EventTypes } from '@oshun/event-bus';
const bus = createEventBus({
redisUrl: process.env.REDIS_URL!,
sourceDomain: 'isis',
});
// Publish a typed event
await bus.publish(EventTypes.ISIS_ASSET_GENERATED, {
assetId: 'asset-123',
projectId: 'proj-456',
type: 'image',
url: 'https://cdn.example.com/output.png',
});
// Subscribe with wildcard pattern
await bus.subscribe('isis.asset.*', async (event, ctx) => {
console.log('Event type:', event.type);
console.log('Payload:', event.payload);
// Acknowledge the event
await ctx.ack();
});
// Subscribe to specific event
await bus.subscribe('yemaya.project.created', async (event, ctx) => {
await createProjectAssets(event.payload);
await ctx.ack();
});
Event Envelope#
Every event is wrapped in an envelope with metadata:
interface EventEnvelope {
id: string; // Unique event ID (nanoid)
type: string; // e.g., 'isis.asset.generated'
source: string; // Originating domain
timestamp: string; // ISO 8601
correlationId: string;
causationId?: string; // ID of the event that caused this one
payload: unknown;
}
Retry and Dead Letter#
const bus = createEventBus({
redisUrl: process.env.REDIS_URL!,
sourceDomain: 'isis',
retry: {
maxRetries: 3,
backoff: 'exponential',
initialDelay: 1000,
},
deadLetter: {
enabled: true,
maxEntries: 10000,
},
});
@oshun/websocket#
Package: @oshun/websocket v0.1.0 Path: libs/shared/websocket/
Dependencies: ws, ioredis, zod, nanoid, @oshun/types,
@oshun/logging, @oshun/errors
WebSocket server with room/channel management, JWT authentication, presence tracking, message rate limiting, connection state management, and Redis adapter for horizontal scaling.
Server Setup#
import { createWSServer } from '@oshun/websocket';
import { createServer } from 'http';
import Redis from 'ioredis';
const httpServer = createServer();
const redis = new Redis();
const wsServer = createWSServer(httpServer, redis, {
requireAuth: true,
path: '/ws',
maxPayloadSize: 65536,
});
wsServer.setAuthHandler(async (token) => {
const verified = await jwtService.verifyToken(token);
return verified
? {
userId: verified.payload.sub,
email: verified.payload.email,
role: verified.payload.role,
}
: null;
});
Room/Channel Management#
import { createRoomManager, COMMON_CHANNEL_CONFIGS } from '@oshun/websocket';
const rooms = createRoomManager(redis);
// Subscribe a user to a channel
await rooms.subscribe(clientId, 'project:proj-123');
// Broadcast to channel
await wsServer.broadcastToChannel('project:proj-123', {
type: 'update',
payload: { field: 'title', value: 'New Title' },
});
Connection Limiting and Rate Limiting#
import {
createConnectionLimiter,
createMessageRateLimiter,
MESSAGE_RATE_PRESETS,
} from '@oshun/websocket';
const connLimiter = createConnectionLimiter({
maxConnectionsPerUser: 5,
maxTotalConnections: 10000,
});
const msgLimiter = createMessageRateLimiter(MESSAGE_RATE_PRESETS.standard);
Horizontal Scaling (Redis Adapter)#
import { createRedisAdapter } from '@oshun/websocket';
const adapter = createRedisAdapter(redis, {
serverId: 'ws-server-1',
heartbeatInterval: 5000,
});
// Messages are broadcast across all WS servers via Redis pub/sub
Connection State and Metrics#
import {
createConnectionStateManager,
createWSMetrics,
} from '@oshun/websocket';
const stateManager = createConnectionStateManager();
const metrics = createWSMetrics();
const snapshot = metrics.snapshot();
// { connections, messages, channels, errors, latency }
@oshun/service-discovery#
Package: @oshun/service-discovery v0.1.0 Path:
libs/shared/service-discovery/ Dependencies: ioredis, nanoid Peer:
@oshun/event-bus
Redis-backed service registry for distributed microservices with health monitoring, load balancing, and change watchers.
import { createServiceDiscovery, ServiceNames } from '@oshun/service-discovery';
const discovery = createServiceDiscovery({
redisUrl: process.env.REDIS_URL!,
});
// Register a service instance
const instanceId = await discovery.register({
name: ServiceNames.ISIS_WORKER,
domain: 'isis',
host: 'worker-1.isis.internal',
port: 8080,
protocol: 'http',
version: '1.0.0',
tags: ['gpu', 'a100'],
healthCheck: { path: '/health', interval: 10000 },
});
// Discover a service (with load balancing)
const url = await discovery.getServiceUrl(ServiceNames.SOPHIA_RAG);
// 'http://rag-2.sophia.internal:8080'
// Watch for service changes
const unwatch = discovery.watch(ServiceNames.ISIS_WORKER, (event, instance) => {
if (event === 'removed') {
console.log('Worker went down:', instance.host);
}
});
// Deregister on shutdown
await discovery.deregister(instanceId);
Load balancing strategies: round-robin, least-connections, random,
weighted.
@oshun/traefik-config#
Package: @oshun/traefik-config v0.1.0 Path:
libs/shared/traefik-config/ Dependencies: None
Unified API gateway configuration management with builder DSL and Traefik config generation for all Oshun domains.
Configuration Builders#
import { service, route, gateway } from '@oshun/traefik-config';
const myService = service('isis-api')
.scope('isis')
.url('http://isis-api:4000')
.healthCheck('/health')
.loadBalancing('round-robin')
.build();
const myRoute = route('isis-assets')
.path('/api/v1/assets/*')
.methods(['GET', 'POST', 'PUT', 'DELETE'])
.service('isis-api')
.auth('required')
.rateLimit('standard')
.build();
const config = gateway().addService(myService).addRoute(myRoute).build();
Domain-specific Configurations#
Pre-built service and route definitions for every domain:
import {
YEMAYA_SERVICES,
YEMAYA_ROUTES,
ISIS_SERVICES,
ISIS_ROUTES,
SOPHIA_SERVICES,
SOPHIA_ROUTES,
HATHOR_SERVICES,
HATHOR_ROUTES,
BELLONA_SERVICES,
BELLONA_ROUTES,
LILITH_SERVICES,
LILITH_ROUTES,
SHARED_SERVICES,
SHARED_ROUTES,
getAllServices,
getAllRoutes,
} from '@oshun/traefik-config';
Traefik Config Generation#
import { generateTraefikConfigs } from '@oshun/traefik-config';
const { staticConfig, dynamicConfig } = generateTraefikConfigs(gatewayConfig);
// Write to traefik.yml and dynamic-config.yml
Authentication and Security Layer#
@oshun/auth-primitives#
Package: @oshun/auth-primitives v0.1.0 Path:
libs/shared/auth-primitives/ Dependencies: jose Peers:
@oshun/errors, @oshun/logging
Low-level authentication building blocks: JWT signing/verification, session management, password hashing, and API key management.
JWT Service#
import {
createHmacJwtService,
createRsaJwtService,
parseAuthorizationHeader,
createAuthorizationHeader,
} from '@oshun/auth-primitives';
// HMAC-based JWT (shared secret)
const jwt = createHmacJwtService({
secret: process.env.JWT_SECRET!,
issuer: 'oshun',
audience: ['api'],
accessTokenTtl: 900, // 15 minutes
refreshTokenTtl: 604800, // 7 days
});
const token = await jwt.sign({ sub: 'user-123', role: 'admin' });
const payload = await jwt.verify(token);
// RSA-based JWT (asymmetric keys)
const rsaJwt = createRsaJwtService({
privateKey: process.env.JWT_PRIVATE_KEY!,
publicKey: process.env.JWT_PUBLIC_KEY!,
algorithm: 'RS256',
});
// Extract token from Authorization header
const token = parseAuthorizationHeader('Bearer eyJhbGciOi...');
Password Hashing#
import {
createPasswordHasher,
createPasswordValidator,
generatePassword,
DEFAULT_PASSWORD_POLICY,
} from '@oshun/auth-primitives';
const hasher = createPasswordHasher();
const hash = await hasher.hash('SecureP@ss123');
const valid = await hasher.verify('SecureP@ss123', hash);
const validator = createPasswordValidator(DEFAULT_PASSWORD_POLICY);
const result = validator.validate('weak'); // { valid: false, errors: [...] }
const generated = generatePassword({ length: 20, symbols: true });
Session Management#
import {
createSessionManager,
createInMemorySessionStore,
} from '@oshun/auth-primitives';
const sessions = createSessionManager(createInMemorySessionStore(), {
ttl: 3600,
maxSessions: 5,
});
const session = await sessions.create({
userId: 'user-123',
device: { userAgent: req.headers['user-agent'], ip: req.ip },
});
const active = await sessions.get(session.id);
await sessions.destroy(session.id);
await sessions.destroyAll('user-123'); // Logout everywhere
API Key Management#
import {
createApiKeyManager,
createInMemoryApiKeyStore,
extractApiKey,
maskApiKey,
} from '@oshun/auth-primitives';
const apiKeys = createApiKeyManager(createInMemoryApiKeyStore());
const { key, hashedKey } = await apiKeys.create({
name: 'Production API Key',
scopes: ['read:users', 'write:content'],
expiresAt: new Date(Date.now() + 365 * 24 * 60 * 60 * 1000),
});
const validated = await apiKeys.validate(key);
console.log(maskApiKey(key)); // 'sk_live_abc...xyz'
@oshun/auth#
Package: @oshun/auth v0.1.0 Path: libs/shared/auth/
Dependencies: @oshun/auth-primitives Peers: @oshun/errors,
@oshun/logging
Complete authentication service built on @oshun/auth-primitives with user
registration, login, RBAC, OAuth integration, and framework-agnostic middleware.
Auth Service#
import { AuthService, createAuthService } from '@oshun/auth';
const authService = new AuthService(userRepo, tokenRepo, auditRepo, {
jwt: {
secret: process.env.JWT_SECRET!,
algorithm: 'HS256',
issuer: 'oshun',
accessTokenTtl: 900,
refreshTokenTtl: 604800,
},
lockout: {
maxAttempts: 5,
lockoutDuration: 900000, // 15 minutes
resetAfter: 3600000, // 1 hour
},
emailVerificationRequired: true,
});
// Register
const result = await authService.register({
email: 'user@example.com',
username: 'johndoe',
password: 'SecureP@ss123',
acceptTerms: true,
});
// Login
const tokens = await authService.login({
email: 'user@example.com',
password: 'SecureP@ss123',
});
// Refresh tokens
const newTokens = await authService.refresh(tokens.refreshToken);
RBAC and Permissions#
import {
hasMinimumRole,
hasPermission,
hasAllPermissions,
hasAnyPermission,
ROLE_HIERARCHY,
ROLE_PERMISSIONS,
} from '@oshun/auth';
// Role hierarchy: guest < user < creator < pro < team < enterprise < admin < super-admin
hasMinimumRole('admin', 'creator'); // true (admin >= creator)
// Permission checks
hasPermission('admin', 'write:users'); // true
hasAllPermissions('user', ['read:content', 'write:content']); // depends on role config
Middleware#
import {
createAuthMiddleware,
requireAuth,
requireRole,
requirePermissions,
requireOwnership,
denySelf,
} from '@oshun/auth';
// Authenticate all routes
app.use(createAuthMiddleware({ authService }));
// Route-specific guards
app.get('/admin', requireRole('admin'), adminHandler);
app.post('/articles', requirePermissions(['write:content']), createArticle);
// Ownership check (user can only edit their own resources)
app.put(
'/users/:id',
requireOwnership((req) => req.params.id),
updateUser
);
// Prevent self-action (e.g., admin can't delete themselves)
app.delete('/users/:id', requireRole('admin'), denySelf(), deleteUser);
Account Lockout#
import {
createAccountLockoutManager,
createDistributedLockoutManager,
} from '@oshun/auth';
// In-memory (single instance)
const lockout = createAccountLockoutManager({
maxAttempts: 5,
lockoutDuration: 900000,
});
// Redis-backed (distributed)
const distributedLockout = createDistributedLockoutManager(redisClient, {
maxAttempts: 5,
lockoutDuration: 900000,
});
@oshun/identity#
Package: @oshun/identity v0.1.0 Path: libs/shared/identity/
Dependencies: None (standalone)
Lightweight cross-domain identity library for JWT handling and role/permission checking. Designed for services that need to verify tokens but do not manage authentication flows.
import {
createJwtService,
authenticate,
hasRole,
hasPermissions,
extractBearerToken,
getPermissionsForRole,
} from '@oshun/identity';
const jwtService = createJwtService({
secret: process.env.JWT_SECRET!,
issuer: 'oshun',
audience: ['api'],
});
// Authenticate a request
const result = await authenticate(jwtService, request);
if (result.authenticated) {
console.log('User:', result.user);
}
// Check role
if (hasRole(result.user, 'admin')) {
/* ... */
}
// Check specific permissions
if (hasPermissions(result.user, ['write:content', 'publish:content'])) {
/* ... */
}
Role hierarchy: guest, user, creator, pro, team, enterprise,
admin, super-admin.
Sub-path exports: @oshun/identity/jwt, @oshun/identity/middleware.
@oshun/security#
Package: @oshun/security v0.0.1 Path: libs/shared/security/
Dependencies: @oshun/logging, @oshun/database, @oshun/cache
Security utilities for audit logging, content/file scanning, and secret management with rotation support.
Audit Logging#
import {
createAuditLogger,
createMemoryAuditStore,
createDatabaseAuditStore,
userActor,
systemActor,
auditTarget,
} from '@oshun/security';
const auditLogger = createAuditLogger({
service: 'api-gateway',
version: '1.0.0',
environment: 'production',
store: createDatabaseAuditStore(dbAdapter),
batchingEnabled: true,
batchSize: 100,
batchFlushInterval: 5000,
});
// Log authentication event
await auditLogger.logAuth(
'auth.login',
userActor('user-123', { email: 'user@example.com' }),
'success',
'User logged in successfully'
);
// Log data access
await auditLogger.logDataAccess(
'data.read',
userActor('user-123'),
auditTarget('user', 'user-456'),
'success',
'Viewed user profile'
);
// Query audit logs
const logs = await auditLogger.query({
actorId: 'user-123',
startDate: new Date('2025-01-01'),
limit: 50,
});
Security Scanning#
import { createBuiltinScanner } from '@oshun/security';
const scanner = createBuiltinScanner();
// Scan file for threats
const fileResult = await scanner.scanFile('/path/to/upload.pdf');
if (!fileResult.clean) {
console.log('Threats:', fileResult.threats);
}
// Scan text for sensitive data (API keys, passwords, PII)
const textResult = await scanner.scanText('api_key=sk_live_12345');
// Detects: exposed API keys, passwords, SSN, credit cards, etc.
ClamAV integration is also available via ClamAVScanner.
Secret Management#
import {
createSecretManager,
createMemorySecretStore,
createDatabaseSecretStore,
} from '@oshun/security';
const secretManager = createSecretManager({
store: createDatabaseSecretStore(dbAdapter),
cacheEnabled: true,
cacheTtl: 300,
autoRotationEnabled: true,
rotationCheckInterval: 3600,
});
// Create a secret
await secretManager.createSecret({
name: 'database-password',
value: 'super-secret-password',
type: 'database_credential',
autoRotate: true,
rotationInterval: 30 * 24 * 60 * 60, // 30 days
});
// Retrieve
const password = await secretManager.getSecret('database-password');
// Rotate
const result = await secretManager.rotateSecret('database-password', {
autoGenerate: true,
generateOptions: { length: 32, charset: 'alphanumeric' },
});
@oshun/rate-limit#
Package: @oshun/rate-limit v0.1.0 Path: libs/shared/rate-limit/
Dependencies: ioredis, @oshun/types, @oshun/logging, @oshun/errors
Optional peer: hono
Multiple rate limiting algorithms with Hono middleware, adaptive limiting, exemptions, bypass rules, and monitoring.
Rate Limiting Algorithms#
import {
createSlidingWindowRateLimiter,
createFixedWindowRateLimiter,
createTokenBucketRateLimiter,
createAdaptiveRateLimiter,
createInMemoryRateLimiter,
} from '@oshun/rate-limit';
// Sliding window (most accurate)
const limiter = createSlidingWindowRateLimiter(redis, {
limit: 100,
window: 60, // seconds
});
const result = await limiter.check('user:123');
// { allowed: true, remaining: 99, resetAt: Date, retryAfter?: number }
// Token bucket (allows bursts)
const bucket = createTokenBucketRateLimiter(redis, {
capacity: 100,
refillRate: 10, // 10 tokens per second
});
// Adaptive (adjusts limits based on server load)
const adaptive = createAdaptiveRateLimiter(redis, {
baseLimit: 100,
minLimit: 20,
maxLimit: 500,
loadProvider: createSimpleLoadProvider(),
});
Hono Middleware#
import {
createRateLimitMiddleware,
createUserRateLimitMiddleware,
createApiKeyRateLimitMiddleware,
createEndpointRateLimitMiddleware,
} from '@oshun/rate-limit';
// General rate limit
app.use('/api/*', createRateLimitMiddleware({ limiter }));
// Per-user rate limit
app.use(
'/api/*',
createUserRateLimitMiddleware({
limiter,
keyGenerator: (c) => c.get('userId'),
})
);
// Per-endpoint rate limit
app.use(
'/api/generate',
createEndpointRateLimitMiddleware({
limiter: createSlidingWindowRateLimiter(redis, { limit: 10, window: 60 }),
})
);
Bypass and Exemptions#
import {
createExemptRateLimiter,
createExemptionManager,
COMMON_BYPASS_RULES,
} from '@oshun/rate-limit';
const exemptLimiter = createExemptRateLimiter(limiter, {
exemptionManager: createExemptionManager({
store: createRedisExemptionStore(redis),
}),
});
// Exempt a specific IP
await exemptLimiter.addExemption({
entityType: 'ip',
entityId: '10.0.0.1',
reason: 'internal_service',
expiresAt: new Date(Date.now() + 86400000),
});
Monitoring and Alerts#
import {
createMonitoredRateLimiter,
createAlertManager,
COMMON_ALERT_RULES,
} from '@oshun/rate-limit';
const monitored = createMonitoredRateLimiter(limiter, {
metricsInterval: 60000,
});
const alerts = createAlertManager({
rules: COMMON_ALERT_RULES,
handlers: [(alert) => notifyOps(alert)],
});
AI Layer#
@oshun/ai#
Package: @oshun/ai v0.1.0 Path: libs/shared/ai/ Dependencies:
@anthropic-ai/sdk, openai, @google/generative-ai, zod, @oshun/types,
@oshun/logging, @oshun/errors
Unified multi-provider LLM client with model routing, prompt templates, response caching, usage tracking, batch processing, and local LLM support.
Providers#
import {
createAnthropicProvider,
createOpenAIProvider,
createGoogleProvider,
createXAIProvider,
createOllamaProvider,
} from '@oshun/ai';
const anthropic = createAnthropicProvider({
apiKey: process.env.ANTHROPIC_API_KEY!,
});
const openai = createOpenAIProvider({
apiKey: process.env.OPENAI_API_KEY!,
});
const google = createGoogleProvider({
apiKey: process.env.GOOGLE_AI_API_KEY!,
});
// Local models via Ollama
const ollama = createOllamaProvider({
baseUrl: 'http://localhost:11434',
});
// Make a request
const response = await anthropic.chat({
model: 'claude-3-opus',
messages: [{ role: 'user', content: 'Explain quantum computing' }],
maxTokens: 1000,
});
Model Router#
Route requests to the best provider based on rules or quality scoring:
import {
createModelRouter,
createQualityRouter,
createMLRouter,
} from '@oshun/ai';
// Rule-based routing
const router = createModelRouter({
providers: { anthropic, openai, google },
rules: [
{
taskType: 'creative-writing',
provider: 'anthropic',
model: 'claude-3-opus',
},
{ taskType: 'code', provider: 'openai', model: 'gpt-4' },
{ taskType: 'summarization', provider: 'google', model: 'gemini-pro' },
],
fallback: { provider: 'anthropic', model: 'claude-3-haiku' },
});
// Quality-based routing (evaluates speed, cost, quality)
const qualityRouter = createQualityRouter({
providers: { anthropic, openai },
weights: { quality: 0.5, speed: 0.3, cost: 0.2 },
});
// ML-based routing (learns from historical performance)
const mlRouter = createMLRouter({
providers: { anthropic, openai, google },
config: createCostOptimizedConfig(), // or createQualityOptimizedConfig()
});
Prompt Templates#
import {
createPromptTemplate,
createDefaultTemplateLibrary,
CREATIVE_TEMPLATES,
TECHNICAL_TEMPLATES,
} from '@oshun/ai';
const template = createPromptTemplate({
name: 'character-description',
template:
'Create a detailed character description for {{name}} who is a {{role}} in {{setting}}.',
variables: {
name: { type: 'string', required: true },
role: { type: 'string', required: true },
setting: { type: 'string', required: true },
},
});
const prompt = template.compile({
name: 'Elena',
role: 'detective',
setting: 'cyberpunk Tokyo',
});
Response Caching#
import { createCachedProvider, createResponseCache } from '@oshun/ai';
const cache = createResponseCache({ maxSize: 1000, ttl: 3600 });
const cached = createCachedProvider(anthropic, cache);
// Identical requests return cached responses
const response1 = await cached.chat({ model: 'claude-3-haiku', messages: [...] });
const response2 = await cached.chat({ model: 'claude-3-haiku', messages: [...] }); // from cache
Usage Tracking and Budgets#
import { createUsageTracker } from '@oshun/ai';
const tracker = createUsageTracker({
budget: { monthly: 1000, daily: 50, currency: 'USD' },
});
tracker.on('budget:warning', ({ usage, limit }) => {
console.warn(`AI budget at ${((usage / limit) * 100).toFixed(0)}%`);
});
const summary = tracker.getSummary();
// { totalTokens, totalCost, byProvider: { anthropic: {...}, openai: {...} } }
Advanced Features#
import {
createBatchProcessor,
createPromptCompressor,
createPromptABTester,
createPromptSanitizer,
createPromptDebugger,
} from '@oshun/ai';
// Batch processing (parallel requests with rate limiting)
const batch = createBatchProcessor(anthropic, {
concurrency: 5,
rateLimit: 60,
});
// Prompt compression (reduce token usage)
const compressor = createPromptCompressor();
// A/B testing of prompt variants
const abTester = createPromptABTester();
// Prompt injection prevention
const sanitizer = createPromptSanitizer();
const result = sanitizer.scan(userInput);
if (result.hasInjection) {
console.warn('Injection detected:', result.injectionType);
}
Local LLM Support#
import {
createModelManager,
createInferenceEngine,
createLocalLLMStack,
estimateVRAMRequirements,
getRecommendedQuantization,
} from '@oshun/ai';
// Estimate hardware requirements
const vram = estimateVRAMRequirements('llama-2-7b', 'q4_0');
const quant = getRecommendedQuantization({ vramGb: 8 });
// Full local LLM stack
const stack = createLocalLLMStack({
modelsDir: '/models',
maxLoadedModels: 2,
defaultQuantization: 'q4_0',
});
await stack.modelManager.loadModel('llama-2-7b');
const result = await stack.inferenceEngine.infer({
model: 'llama-2-7b',
prompt: 'Tell me a story',
maxTokens: 500,
});
@oshun/ai-advanced#
Package: @oshun/ai-advanced v0.1.0 Path: libs/shared/ai-advanced/
Dependencies: @oshun/ai, @oshun/types, @oshun/logging, zod
Advanced AI capabilities built on top of @oshun/ai: modular adapter system,
model benchmarking, automatic model selection, on-device/edge AI, and research
tools.
Modular Adapter System#
import {
AdapterManager,
createBuiltInAdapters,
InMemoryAdapterStorage,
} from '@oshun/ai-advanced';
const adapterManager = new AdapterManager(new InMemoryAdapterStorage());
// Register built-in adapters (Anthropic, OpenAI, Google, etc.)
const builtInAdapters = createBuiltInAdapters();
for (const adapter of builtInAdapters) {
await adapterManager.register(adapter);
}
// List available adapters
const adapters = await adapterManager.list();
// Filter by capability
const visionAdapters = await adapterManager.findByCapability('vision');
Model Benchmarking#
import {
BenchmarkManager,
getStandardBenchmarkTasks,
InMemoryBenchmarkStorage,
} from '@oshun/ai-advanced';
const benchmarks = new BenchmarkManager(new InMemoryBenchmarkStorage());
// Run benchmarks against standard tasks
const tasks = getStandardBenchmarkTasks();
const results = await benchmarks.run('claude-3-opus', tasks, adapter);
// { accuracy, latencyP50, latencyP99, tokensPerSecond, costPerToken }
Automatic Model Selection#
import { ModelSelector, InMemoryModelProfileStorage } from '@oshun/ai-advanced';
const selector = new ModelSelector(new InMemoryModelProfileStorage());
// Find the best model for a task
const bestModel = await selector.select({
taskType: 'creative-writing',
maxLatencyMs: 5000,
maxCostPerToken: 0.001,
requiredCapabilities: ['text-generation'],
});
Edge / On-Device AI#
import { EdgeManager, InMemoryEdgeModelStorage } from '@oshun/ai-advanced';
const edgeManager = new EdgeManager(new InMemoryEdgeModelStorage());
// Deploy a model to edge devices
await edgeManager.deploy({
modelId: 'text-classifier-v1',
targetDevice: 'browser',
quantization: 'int8',
maxModelSize: 50 * 1024 * 1024, // 50 MB
});
// Run inference on device
const result = await edgeManager.infer('text-classifier-v1', {
text: 'Classify this input',
});
ONNX runtime integration is supported via the OnnxRuntimeProvider interface.
Research Tools#
import { ResearchManager, InMemoryResearchStorage } from '@oshun/ai-advanced';
const research = new ResearchManager(new InMemoryResearchStorage());
// Search arXiv papers
const papers = await research.searchPapers({
query: 'transformer attention mechanisms',
maxResults: 10,
});
// Search HuggingFace models and datasets
const models = await research.searchModels({ query: 'text-to-image' });
const datasets = await research.searchDatasets({ query: 'sentiment analysis' });
// Track experiments
const experiment = await research.createExperiment({
name: 'prompt-optimization-v2',
description: 'Testing new prompt templates',
});
GPU and Compute Layer#
@oshun/runpod-client#
Package: @oshun/runpod-client v0.1.0 Path:
libs/shared/runpod-client/ Dependencies: @opentelemetry/api,
@opentelemetry/semantic-conventions Peers: @oshun/errors,
@oshun/logging, @oshun/http-client
Type-safe client for the RunPod Serverless API with async and synchronous job execution, status polling, batch operations, and comprehensive error handling.
Client Setup#
import { createRunPodClient } from '@oshun/runpod-client';
const client = createRunPodClient({
apiKey: process.env.RUNPOD_API_KEY!,
retry: { maxRetries: 3, backoff: 'exponential' },
polling: { initialDelay: 1000, maxDelay: 10000 },
});
Job Execution#
// Async job (submit and poll)
const job = await client.run('endpoint-id', {
input: { prompt: 'A photo of a sunset', width: 1024, height: 1024 },
});
// Wait for completion
const result = await client.runAndWait('endpoint-id', {
input: { prompt: 'Generate image' },
maxWaitTime: 120000, // 2 minutes
});
console.log(result.output);
// Synchronous execution (blocks until done, for quick jobs)
const syncResult = await client.runSync('endpoint-id', {
input: { prompt: 'Hello' },
timeout: 30,
});
Batch Operations#
const results = await client.runBatch('endpoint-id', [
{ input: { prompt: 'Image 1' } },
{ input: { prompt: 'Image 2' } },
{ input: { prompt: 'Image 3' } },
]);
Status and Management#
const status = await client.status('endpoint-id', 'job-123');
await client.cancel('endpoint-id', 'job-123');
const health = await client.health('endpoint-id');
await client.purgeQueue('endpoint-id');
Error Types#
Specific error classes: AuthenticationError, EndpointNotFoundError,
JobNotFoundError, RateLimitError, TimeoutError, NetworkError,
JobFailedError, PollingTimeoutError. Utility functions: isRetryableError,
isAuthError, isRateLimitError.
@oshun/gpu-dispatcher#
Package: @oshun/gpu-dispatcher v0.1.0 Path:
libs/shared/gpu-dispatcher/ Peers: @oshun/errors, @oshun/logging,
@oshun/metrics, @oshun/runpod-client, @oshun/storage, @oshun/tracing
High-level GPU job orchestrator built on @oshun/runpod-client with job
queuing, endpoint fallback, circuit breakers, cost tracking, result validation,
and comprehensive metrics.
Dispatcher Setup#
import { createGpuDispatcher, createRunPodClient } from '@oshun/gpu-dispatcher';
const runpod = createRunPodClient({ apiKey: process.env.RUNPOD_API_KEY! });
const dispatcher = createGpuDispatcher(
{
runpodApiKey: process.env.RUNPOD_API_KEY!,
endpoints: [
{ id: 'comfyui-sd', name: 'ComfyUI SD', jobTypes: ['image-generation'] },
{
id: 'comfyui-flux',
name: 'ComfyUI Flux',
jobTypes: ['image-generation'],
},
{ id: 'video-gen', name: 'Video Gen', jobTypes: ['video-generation'] },
],
},
runpod
);
Job Lifecycle#
// Create a job
const job = await dispatcher.createJob({
type: 'image-generation',
input: { prompt: 'A beautiful sunset', width: 1024, height: 1024 },
priority: 'high',
});
// Wait for completion
const result = await dispatcher.waitForJob(job.id);
// Or event-driven approach
dispatcher.on('job:completed', ({ job, result }) => {
console.log(`Job ${job.id} completed:`, result.output);
});
dispatcher.on('job:failed', ({ job, error }) => {
console.error(`Job ${job.id} failed:`, error.message);
});
dispatcher.startProcessing();
Cost Tracking#
import { createCostTracker, DEFAULT_GPU_PRICING } from '@oshun/gpu-dispatcher';
const costs = createCostTracker({
pricing: DEFAULT_GPU_PRICING,
budget: { daily: 100, monthly: 2000, currency: 'USD' },
});
costs.on('budget:exceeded', ({ period, spent, limit }) => {
console.error(`GPU budget exceeded: $${spent}/$${limit} (${period})`);
});
const stats = costs.getStatistics();
// { totalCost, jobCount, avgCostPerJob, costByType, costByEndpoint }
Endpoint Fallback#
import { createFallbackManager } from '@oshun/gpu-dispatcher';
const fallback = createFallbackManager({
strategy: 'priority', // or 'round-robin', 'random', 'least-loaded'
healthCheckInterval: 30000,
});
// Automatically routes to healthy endpoints when primary is down
Result Validation#
import {
createResultValidator,
createImageValidationConfig,
createVideoValidationConfig,
} from '@oshun/gpu-dispatcher';
const validator = createResultValidator(
createImageValidationConfig({
maxFileSize: 10 * 1024 * 1024, // 10 MB
allowedFormats: ['png', 'jpg', 'webp'],
minResolution: { width: 512, height: 512 },
})
);
const validation = await validator.validate(result);
if (!validation.valid) {
console.error('Invalid output:', validation.errors);
}
Timeout Handling#
import {
createGpuTimeoutManager,
IMAGE_GENERATION_TIMEOUT,
VIDEO_GENERATION_TIMEOUT,
} from '@oshun/gpu-dispatcher';
const timeouts = createGpuTimeoutManager({
'image-generation': IMAGE_GENERATION_TIMEOUT,
'video-generation': VIDEO_GENERATION_TIMEOUT,
});
Infrastructure Layer#
@oshun/infrastructure#
Package: @oshun/infrastructure v0.1.0 Path:
libs/shared/infrastructure/ Dependencies: @oshun/types,
@oshun/logging, @oshun/errors, zod
High-level infrastructure managers for performance optimization, security, and monitoring. Provides abstract interfaces with in-memory and mock implementations for testing.
Performance Manager#
import { createPerformanceManager } from '@oshun/infrastructure';
const perf = createPerformanceManager({
assetLoader: myAssetLoader,
assetStorage: new InMemoryAssetStorage(),
lodProvider: myLODProvider,
memoryMonitor: myMemoryMonitor,
taskExecutor: myTaskExecutor,
taskStorage: new InMemoryTaskStorage(),
gpuCompute: myGPUCompute,
});
// Lazy asset loading with Level-of-Detail
await perf.loadAsset('asset-123', { priority: 'high', lod: 'medium' });
// Background task scheduling
await perf.scheduleTask({
type: 'preprocess',
priority: 'low',
payload: { assetId: 'asset-123' },
});
// Memory management
const usage = perf.getMemoryUsage();
if (usage.pressure === 'critical') {
await perf.cleanup({ maxAge: 3600 });
}
Security Manager#
import { createSecurityManager } from '@oshun/infrastructure';
const security = createSecurityManager({
encryption: myEncryptionProvider,
secretStorage: new InMemorySecretStorage(),
apiKeyProvider: myAPIKeyProvider,
apiKeyStorage: new InMemoryAPIKeyStorage(),
vulnerabilityScanner: myScanner,
scanResultStorage: new InMemoryScanResultStorage(),
securityAudit: new InMemorySecurityAudit(),
});
// Encrypt/decrypt data
const encrypted = await security.encrypt(sensitiveData);
const decrypted = await security.decrypt(encrypted);
// Manage API keys
const key = await security.createApiKey({
name: 'Production',
scopes: ['read'],
});
const valid = await security.validateApiKey(key.value);
// Run vulnerability scan
const scanResult = await security.scan({
type: 'dependency',
target: 'package.json',
});
Monitoring Manager#
import { createMonitoringManager } from '@oshun/infrastructure';
const monitoring = createMonitoringManager({
tracing: new InMemoryTracingProvider(),
traceStorage: new InMemoryTraceStorage(),
metrics: new InMemoryMetricsProvider(),
metricsStorage: new InMemoryMetricsStorage(),
alerting: myAlertingProvider,
alertStorage: new InMemoryAlertStorage(),
alertRuleStorage: new InMemoryAlertRuleStorage(),
dashboardStorage: new InMemoryDashboardStorage(),
analytics: new InMemoryAnalyticsProvider(),
});
// Create traces and spans
const trace = await monitoring.createTrace({ name: 'request', service: 'api' });
// Record metrics
await monitoring.recordMetric({
name: 'request_duration',
type: 'histogram',
value: 45.2,
labels: { method: 'GET', path: '/users' },
});
// Manage alert rules
await monitoring.createAlertRule({
name: 'High Error Rate',
condition: { metric: 'error_rate', operator: '>', threshold: 0.05 },
severity: 'critical',
channels: ['pagerduty', 'slack'],
});
// Dashboard management
await monitoring.createDashboard({ name: 'API Overview', panels: [...] });
@oshun/migration#
Package: @oshun/migration v0.1.0 Path: libs/shared/migration/
Dependencies: None
Cross-domain data migration framework with runner, registry, and support for dry-run, progress reporting, and rollback.
import {
MigrationRunner,
MigrationRegistry,
createMigrationRunner,
} from '@oshun/migration';
const registry = new MigrationRegistry();
// Register migrations
registry.add({
id: '001',
name: 'migrate-users-to-new-schema',
description: 'Moves user data from lilith to yemaya schema',
sourceDomain: 'lilith',
targetDomain: 'yemaya',
version: '1.0.0',
reversible: true,
up: async (ctx) => {
const users = await ctx.source.query('SELECT * FROM users');
for (const batch of ctx.progress.batches(users, 100)) {
await ctx.target.query(sql`INSERT INTO users ...`);
ctx.progress.report(batch.processed, batch.total);
}
},
down: async (ctx) => {
await ctx.target.query('DELETE FROM users WHERE migrated = true');
},
});
const runner = createMigrationRunner(registry, { batchSize: 100 });
// Dry run first
const dryResult = await runner.run('001', { dryRun: true, verbose: true });
// Execute
const result = await runner.run('001');
console.log(result.status, result.duration);
// Rollback if needed
await runner.rollback('001');
Domains: lilith, yemaya, isis, sophia, hathor, bellona.
Testing#
@oshun/testing#
Package: @oshun/testing v0.0.1 Path: libs/shared/testing/
Dependencies: pg Peers: vitest, testcontainers (optional)
Comprehensive testing utilities: mock factories, fixture generators, assertion helpers, test containers, and Vitest configuration builders.
Mock Factories#
import {
createMockLogger,
createMockHttpClient,
createMockRedisClient,
createMockDatabaseClient,
createMockEventEmitter,
createMockTimers,
} from '@oshun/testing';
const logger = createMockLogger();
const http = createMockHttpClient();
const redis = createMockRedisClient();
const db = createMockDatabaseClient();
// Mock HTTP responses
http.onGet('/users/123').respond({ id: '123', name: 'Test' });
http.onPost('/users').respond({ id: '456' }, { status: 201 });
Fixture Generators#
import {
createUser,
createUsers,
createContent,
userFactory,
contentFactory,
randomEmail,
randomUUID,
randomString,
createFixtureFactory,
} from '@oshun/testing';
// Quick fixtures
const user = createUser({ role: 'admin' });
const users = createUsers(10); // 10 random users
// Factory pattern
const myFactory = createFixtureFactory({
id: () => randomUUID(),
email: () => randomEmail(),
name: () => randomString(10),
role: 'user',
active: true,
});
const record = myFactory.create({ role: 'admin' }); // Override specific fields
const records = myFactory.createMany(5);
Assertion Helpers#
import {
assertDefined,
assertRejects,
assertAppError,
assertApiSuccess,
assertApiError,
assertResultOk,
assertResultErr,
waitFor,
retryUntil,
measureTime,
assertExecutesWithin,
} from '@oshun/testing';
// Wait for async condition
await waitFor(() => queue.length === 0, { timeout: 5000, interval: 100 });
// Assert execution time
await assertExecutesWithin(500, async () => {
await fastOperation();
});
// Assert error types
await assertAppError(() => service.getUser('nonexistent'), 'NotFoundError');
Test Containers#
import {
createPostgresContainer,
createRedisContainer,
createContainerManager,
} from '@oshun/testing';
// Requires testcontainers peer dependency
const pg = await createPostgresContainer({
image: 'postgres:16',
database: 'test',
});
const redis = await createRedisContainer();
// Manage multiple containers
const manager = createContainerManager();
manager.add('postgres', pg);
manager.add('redis', redis);
await manager.startAll();
// ... run tests ...
await manager.stopAll();
Vitest Configuration Builders#
import {
createVitestConfig,
createServiceVitestConfig,
createIntegrationVitestConfig,
createWorkspaceAliases,
} from '@oshun/testing';
// For a library
export default createVitestConfig({
name: '@oshun/my-lib',
aliases: createWorkspaceAliases({
'@oshun/logging': '../logging/src/index.ts',
'@oshun/errors': '../errors/src/index.ts',
}),
coverage: { threshold: 80 },
});
// For a service with integration tests
export default createServiceVitestConfig({
name: 'my-service',
setupFiles: ['./test/setup.ts'],
});
Sub-path exports: @oshun/testing/mocks, @oshun/testing/fixtures,
@oshun/testing/helpers, @oshun/testing/containers, @oshun/testing/config.
Configuration Reference#
All shared libraries read configuration from environment variables. Here is the complete set of environment variables used across all libraries:
Core#
| Variable | Library | Description | Default |
|---|---|---|---|
NODE_ENV |
@oshun/config |
production, development, test |
development |
LOG_LEVEL |
@oshun/logging |
trace, debug, info, warn, error, fatal |
info |
PORT |
@oshun/config |
HTTP server port | 3000 |
Database#
| Variable | Library | Description |
|---|---|---|
DATABASE_URL |
@oshun/database |
PostgreSQL connection string |
REDIS_URL |
@oshun/database, @oshun/cache |
Redis connection string |
Storage#
| Variable | Library | Description |
|---|---|---|
S3_BUCKET |
@oshun/storage |
S3 bucket name |
S3_REGION |
@oshun/storage |
AWS region |
S3_ENDPOINT |
@oshun/storage |
Custom endpoint (MinIO) |
S3_ACCESS_KEY |
@oshun/storage |
Access key |
S3_SECRET_KEY |
@oshun/storage |
Secret key |
Observability#
| Variable | Library | Description |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
@oshun/tracing |
OpenTelemetry collector URL |
OTEL_SERVICE_NAME |
@oshun/tracing |
Service name for traces |
SENTRY_DSN |
@oshun/errors |
Sentry error tracking DSN |
METRICS_PORT |
@oshun/metrics |
Prometheus scrape port |
Authentication#
| Variable | Library | Description |
|---|---|---|
JWT_SECRET |
@oshun/auth, @oshun/identity |
JWT signing secret |
JWT_PRIVATE_KEY |
@oshun/auth-primitives |
RSA private key (PEM) |
JWT_PUBLIC_KEY |
@oshun/auth-primitives |
RSA public key (PEM) |
AI#
| Variable | Library | Description |
|---|---|---|
ANTHROPIC_API_KEY |
@oshun/ai |
Anthropic API key |
OPENAI_API_KEY |
@oshun/ai |
OpenAI API key |
GOOGLE_AI_API_KEY |
@oshun/ai |
Google AI API key |
XAI_API_KEY |
@oshun/ai |
xAI API key |
GPU#
| Variable | Library | Description |
|---|---|---|
RUNPOD_API_KEY |
@oshun/runpod-client, @oshun/gpu-dispatcher |
RunPod API key |
Building and Development#
Build All Shared Libraries#
# Build everything tagged scope:shared
pnpm nx run-many --target=build --projects=tag:scope:shared
# Build a single library
pnpm nx build @oshun/logging
# Run all shared library tests
pnpm nx run-many --target=test --projects=tag:scope:shared
# Lint
pnpm nx run-many --target=lint --projects=tag:scope:shared
Local Development Infrastructure#
Shared libraries that depend on external services (database, Redis, S3) require the development Docker Compose stack:
# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d
# Verify
docker compose -f docker/docker-compose.dev.yml ps
Services available locally:
| Service | Port | Purpose |
|---|---|---|
| PostgreSQL | 5432 | Primary database (with pgvector) |
| Redis | 6379 | Caching, queues, event bus, service discovery |
| MinIO | 9000/9001 | S3-compatible object storage |
| Mailpit | 1025/8025 | Email testing |
Adding a New Shared Library#
- Create the directory:
libs/shared/my-lib/ - Add
package.jsonwith name@oshun/my-lib - Add
project.jsonwithscope:sharedtag - Add
tsconfig.jsonextending../../../tsconfig.base.json - Add path mapping to root
tsconfig.base.json - Implement
src/index.tswith exports - Write tests alongside source files
- Run
pnpm install, thenpnpm nx build @oshun/my-lib