Kubernetes deployment configuration for Nyx astronomical data pipelines
This directory contains deployment configurations for data ingestion and processing pipelines that keep Nyx's astronomical data fresh and accurate.
Overview#
The Nyx platform requires continuous data updates from multiple external sources:
| Pipeline | Source | Update Frequency | Purpose |
|---|---|---|---|
| TLE Update | Celestrak, Space-Track | Hourly | Satellite orbital elements |
| Catalog Sync | SIMBAD, NED, NASA | Weekly | Star and deep-sky catalogs |
| Ephemeris Precompute | Internal | Daily | Planet/Moon positions |
| Event Calculation | Internal | Weekly | Astronomical events |
Architecture#
┌─────────────────────────────────────────────────────────────────────┐
│ External Data Sources │
├─────────────────┬───────────────────┬───────────────────────────────┤
│ Celestrak │ Space-Track │ SIMBAD/NED/NASA APIs │
└────────┬────────┴─────────┬─────────┴─────────────┬─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ Kubernetes CronJobs │
├─────────────────┬───────────────────┬───────────────────────────────┤
│ tle-updater │ catalog-syncer │ ephemeris-generator │
│ (hourly) │ (weekly) │ (daily) │
└────────┬────────┴─────────┬─────────┴─────────────┬─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ PostgreSQL Database │
│ (nyx schema with pgvector) │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Nyx API │
│ (serves cached and computed data) │
└─────────────────────────────────────────────────────────────────────┘
Pipeline Details#
TLE Update Pipeline (tle-updater)#
Updates Two-Line Element sets for satellite tracking.
Sources:
- Celestrak (primary, free):
https://celestrak.org/NORAD/elements/ - Space-Track (secondary, authenticated):
https://www.space-track.org/
Categories Updated:
- Space stations (ISS, CSS)
- Active satellites
- Starlink constellation
- GPS/GLONASS/Galileo
- Weather satellites
- Amateur radio satellites
Process:
- Fetch TLE data from Celestrak (no auth required)
- Fall back to Space-Track for unavailable categories
- Parse and validate TLE format (checksum verification)
- Calculate derived orbital parameters (apogee, perigee, period)
- Store in database with timestamp
- Mark stale entries (>14 days old) for cleanup
Catalog Sync Pipeline (catalog-syncer)#
Synchronizes astronomical catalogs from authoritative sources.
Catalogs:
- Stars: Hipparcos, Gaia DR3 (subset), Bright Star Catalog
- Deep Sky: Messier, NGC/IC, PGC
- Exoplanets: NASA Exoplanet Archive
- NEOs: Minor Planet Center
Process:
- Query TAP services for updated entries
- Cross-match with existing catalog data
- Apply deduplication and confidence scoring
- Update database with new/modified entries
- Generate catalog statistics
Ephemeris Precompute Pipeline (ephemeris-generator)#
Pre-calculates planetary positions for fast API responses.
Objects:
- Sun, Moon
- Mercury, Venus, Mars, Jupiter, Saturn, Uranus, Neptune
Output:
- 1-minute resolution ephemeris for 7 days ahead
- 1-hour resolution ephemeris for 30 days ahead
- Rise/set/transit times for multiple observer locations
Process:
- Calculate ephemeris using VSOP87/ELP2000 theories
- Apply corrections (precession, nutation, aberration)
- Store in database with interpolation coefficients
- Invalidate API cache for updated ranges
Event Calculation Pipeline (event-calculator)#
Calculates upcoming astronomical events.
Event Types:
- Solar/Lunar eclipses
- Planetary conjunctions
- Meteor showers
- Satellite visible passes
Process:
- Run event detection algorithms
- Calculate visibility for key observer locations
- Store events with geometry details
- Update event feed JSON for CDN
Configuration#
Environment Variables#
| Variable | Description | Default |
|---|---|---|
DATABASE_URL |
PostgreSQL connection string | Required |
REDIS_URL |
Redis for caching and locks | Optional |
CELESTRAK_BASE_URL |
Celestrak API base | https://celestrak.org |
SPACETRACK_USERNAME |
Space-Track login | Required for TLE |
SPACETRACK_PASSWORD |
Space-Track password | Required for TLE |
TLE_UPDATE_CATEGORIES |
Comma-separated categories | stations,active,starlink |
CATALOG_SYNC_SOURCES |
Comma-separated sources | hipparcos,messier,ngc |
EPHEMERIS_DAYS_AHEAD |
Days to precompute | 30 |
LOG_LEVEL |
Logging verbosity | info |
Secrets#
Create the nyx-pipeline-secrets secret with:
kubectl create secret generic nyx-pipeline-secrets \
--from-literal=SPACETRACK_USERNAME=your_username \
--from-literal=SPACETRACK_PASSWORD=your_password \
--from-literal=DATABASE_URL=postgresql://... \
-n nyx
Deployment#
Prerequisites#
- PostgreSQL database with nyx schema initialized
- (Optional) Space-Track account for authenticated TLE access
- Network access to external data sources
Apply Configuration#
# Development
kubectl apply -k infra/nyx/pipelines/overlays/dev
# Staging
kubectl apply -k infra/nyx/pipelines/overlays/staging
# Production
kubectl apply -k infra/nyx/pipelines/overlays/prod
Verify Deployment#
# List cronjobs
kubectl get cronjobs -n nyx
# View recent job runs
kubectl get jobs -n nyx --sort-by=.status.startTime
# Check logs for a specific job
kubectl logs -n nyx job/tle-updater-<timestamp>
Manual Trigger#
Trigger a pipeline manually for testing:
# Trigger TLE update
kubectl create job --from=cronjob/tle-updater manual-tle-$(date +%s) -n nyx
# Trigger catalog sync
kubectl create job --from=cronjob/catalog-syncer manual-catalog-$(date +%s) -n nyx
Monitoring#
Prometheus Metrics#
Each pipeline exports metrics on port 9090:
| Metric | Type | Description |
|---|---|---|
nyx_pipeline_duration_seconds |
Histogram | Pipeline execution time |
nyx_pipeline_records_processed |
Counter | Records processed per run |
nyx_pipeline_errors_total |
Counter | Pipeline error count |
nyx_tle_staleness_seconds |
Gauge | Age of oldest TLE |
nyx_catalog_entries_total |
Gauge | Total catalog entries |
Alerts#
Configure alerts for:
- Pipeline job failures (>2 consecutive failures)
- TLE staleness (>24 hours for active satellites)
- Database write errors
- External API unavailability
Grafana Dashboard#
Import the Nyx Pipelines dashboard from:
infra/nyx/monitoring/dashboards/pipelines.json
Troubleshooting#
TLE Update Failures#
Symptom: TLE update job fails repeatedly
Check:
- Verify Celestrak is accessible:
curl https://celestrak.org/NORAD/elements/gp.php?GROUP=stations - Check Space-Track credentials are valid
- Review job logs for specific errors
Database Connection Issues#
Symptom: Jobs fail with connection errors
Check:
- Verify
DATABASE_URLsecret is correct - Check PostgreSQL pod is running
- Verify network policies allow job→database traffic
Stale Data#
Symptom: API returns old satellite positions
Check:
- Verify CronJob schedule is active
- Check job history for successful completions
- Query database for TLE timestamps
Cost Considerations#
API Rate Limits#
- Celestrak: No authentication, reasonable use expected
- Space-Track: 30 requests/minute, 300/hour for authenticated users
- SIMBAD/VizieR: TAP services have query size limits
Resource Usage#
| Pipeline | CPU Request | Memory Request | Typical Duration |
|---|---|---|---|
| TLE Update | 100m | 256Mi | 2-5 minutes |
| Catalog Sync | 200m | 512Mi | 10-30 minutes |
| Ephemeris | 500m | 1Gi | 5-15 minutes |
| Events | 200m | 512Mi | 5-10 minutes |