Eve library

Nyx Data Pipelines Deployment

Other

Revision 1

On this page

Kubernetes deployment configuration for Nyx astronomical data pipelines

This directory contains deployment configurations for data ingestion and processing pipelines that keep Nyx's astronomical data fresh and accurate.

Overview#

The Nyx platform requires continuous data updates from multiple external sources:

Pipeline Source Update Frequency Purpose
TLE Update Celestrak, Space-Track Hourly Satellite orbital elements
Catalog Sync SIMBAD, NED, NASA Weekly Star and deep-sky catalogs
Ephemeris Precompute Internal Daily Planet/Moon positions
Event Calculation Internal Weekly Astronomical events

Architecture#

text
┌─────────────────────────────────────────────────────────────────────┐
│                      External Data Sources                          │
├─────────────────┬───────────────────┬───────────────────────────────┤
│    Celestrak    │    Space-Track    │   SIMBAD/NED/NASA APIs        │
└────────┬────────┴─────────┬─────────┴─────────────┬─────────────────┘
         │                  │                       │
         ▼                  ▼                       ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Kubernetes CronJobs                              │
├─────────────────┬───────────────────┬───────────────────────────────┤
│  tle-updater    │ catalog-syncer    │   ephemeris-generator         │
│  (hourly)       │ (weekly)          │   (daily)                     │
└────────┬────────┴─────────┬─────────┴─────────────┬─────────────────┘
         │                  │                       │
         ▼                  ▼                       ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        PostgreSQL Database                          │
│                    (nyx schema with pgvector)                       │
└─────────────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Nyx API                                    │
│              (serves cached and computed data)                      │
└─────────────────────────────────────────────────────────────────────┘

Pipeline Details#

TLE Update Pipeline (tle-updater)#

Updates Two-Line Element sets for satellite tracking.

Sources:

  • Celestrak (primary, free): https://celestrak.org/NORAD/elements/
  • Space-Track (secondary, authenticated): https://www.space-track.org/

Categories Updated:

  • Space stations (ISS, CSS)
  • Active satellites
  • Starlink constellation
  • GPS/GLONASS/Galileo
  • Weather satellites
  • Amateur radio satellites

Process:

  1. Fetch TLE data from Celestrak (no auth required)
  2. Fall back to Space-Track for unavailable categories
  3. Parse and validate TLE format (checksum verification)
  4. Calculate derived orbital parameters (apogee, perigee, period)
  5. Store in database with timestamp
  6. Mark stale entries (>14 days old) for cleanup

Catalog Sync Pipeline (catalog-syncer)#

Synchronizes astronomical catalogs from authoritative sources.

Catalogs:

  • Stars: Hipparcos, Gaia DR3 (subset), Bright Star Catalog
  • Deep Sky: Messier, NGC/IC, PGC
  • Exoplanets: NASA Exoplanet Archive
  • NEOs: Minor Planet Center

Process:

  1. Query TAP services for updated entries
  2. Cross-match with existing catalog data
  3. Apply deduplication and confidence scoring
  4. Update database with new/modified entries
  5. Generate catalog statistics

Ephemeris Precompute Pipeline (ephemeris-generator)#

Pre-calculates planetary positions for fast API responses.

Objects:

  • Sun, Moon
  • Mercury, Venus, Mars, Jupiter, Saturn, Uranus, Neptune

Output:

  • 1-minute resolution ephemeris for 7 days ahead
  • 1-hour resolution ephemeris for 30 days ahead
  • Rise/set/transit times for multiple observer locations

Process:

  1. Calculate ephemeris using VSOP87/ELP2000 theories
  2. Apply corrections (precession, nutation, aberration)
  3. Store in database with interpolation coefficients
  4. Invalidate API cache for updated ranges

Event Calculation Pipeline (event-calculator)#

Calculates upcoming astronomical events.

Event Types:

  • Solar/Lunar eclipses
  • Planetary conjunctions
  • Meteor showers
  • Satellite visible passes

Process:

  1. Run event detection algorithms
  2. Calculate visibility for key observer locations
  3. Store events with geometry details
  4. Update event feed JSON for CDN

Configuration#

Environment Variables#

Variable Description Default
DATABASE_URL PostgreSQL connection string Required
REDIS_URL Redis for caching and locks Optional
CELESTRAK_BASE_URL Celestrak API base https://celestrak.org
SPACETRACK_USERNAME Space-Track login Required for TLE
SPACETRACK_PASSWORD Space-Track password Required for TLE
TLE_UPDATE_CATEGORIES Comma-separated categories stations,active,starlink
CATALOG_SYNC_SOURCES Comma-separated sources hipparcos,messier,ngc
EPHEMERIS_DAYS_AHEAD Days to precompute 30
LOG_LEVEL Logging verbosity info

Secrets#

Create the nyx-pipeline-secrets secret with:

bash
kubectl create secret generic nyx-pipeline-secrets \
  --from-literal=SPACETRACK_USERNAME=your_username \
  --from-literal=SPACETRACK_PASSWORD=your_password \
  --from-literal=DATABASE_URL=postgresql://... \
  -n nyx

Deployment#

Prerequisites#

  1. PostgreSQL database with nyx schema initialized
  2. (Optional) Space-Track account for authenticated TLE access
  3. Network access to external data sources

Apply Configuration#

bash
# Development
kubectl apply -k infra/nyx/pipelines/overlays/dev

# Staging
kubectl apply -k infra/nyx/pipelines/overlays/staging

# Production
kubectl apply -k infra/nyx/pipelines/overlays/prod

Verify Deployment#

bash
# List cronjobs
kubectl get cronjobs -n nyx

# View recent job runs
kubectl get jobs -n nyx --sort-by=.status.startTime

# Check logs for a specific job
kubectl logs -n nyx job/tle-updater-<timestamp>

Manual Trigger#

Trigger a pipeline manually for testing:

bash
# Trigger TLE update
kubectl create job --from=cronjob/tle-updater manual-tle-$(date +%s) -n nyx

# Trigger catalog sync
kubectl create job --from=cronjob/catalog-syncer manual-catalog-$(date +%s) -n nyx

Monitoring#

Prometheus Metrics#

Each pipeline exports metrics on port 9090:

Metric Type Description
nyx_pipeline_duration_seconds Histogram Pipeline execution time
nyx_pipeline_records_processed Counter Records processed per run
nyx_pipeline_errors_total Counter Pipeline error count
nyx_tle_staleness_seconds Gauge Age of oldest TLE
nyx_catalog_entries_total Gauge Total catalog entries

Alerts#

Configure alerts for:

  • Pipeline job failures (>2 consecutive failures)
  • TLE staleness (>24 hours for active satellites)
  • Database write errors
  • External API unavailability

Grafana Dashboard#

Import the Nyx Pipelines dashboard from: infra/nyx/monitoring/dashboards/pipelines.json

Troubleshooting#

TLE Update Failures#

Symptom: TLE update job fails repeatedly

Check:

  1. Verify Celestrak is accessible: curl https://celestrak.org/NORAD/elements/gp.php?GROUP=stations
  2. Check Space-Track credentials are valid
  3. Review job logs for specific errors

Database Connection Issues#

Symptom: Jobs fail with connection errors

Check:

  1. Verify DATABASE_URL secret is correct
  2. Check PostgreSQL pod is running
  3. Verify network policies allow job→database traffic

Stale Data#

Symptom: API returns old satellite positions

Check:

  1. Verify CronJob schedule is active
  2. Check job history for successful completions
  3. Query database for TLE timestamps

Cost Considerations#

API Rate Limits#

  • Celestrak: No authentication, reasonable use expected
  • Space-Track: 30 requests/minute, 300/hour for authenticated users
  • SIMBAD/VizieR: TAP services have query size limits

Resource Usage#

Pipeline CPU Request Memory Request Typical Duration
TLE Update 100m 256Mi 2-5 minutes
Catalog Sync 200m 512Mi 10-30 minutes
Ephemeris 500m 1Gi 5-15 minutes
Events 200m 512Mi 5-10 minutes