# Nyx Data Pipelines Deployment

> Kubernetes deployment configuration for Nyx astronomical data pipelines

This directory contains deployment configurations for data ingestion and
processing pipelines that keep Nyx's astronomical data fresh and accurate.

## Overview

The Nyx platform requires continuous data updates from multiple external
sources:

| Pipeline             | Source                 | Update Frequency | Purpose                    |
| -------------------- | ---------------------- | ---------------- | -------------------------- |
| TLE Update           | Celestrak, Space-Track | Hourly           | Satellite orbital elements |
| Catalog Sync         | SIMBAD, NED, NASA      | Weekly           | Star and deep-sky catalogs |
| Ephemeris Precompute | Internal               | Daily            | Planet/Moon positions      |
| Event Calculation    | Internal               | Weekly           | Astronomical events        |

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                      External Data Sources                          │
├─────────────────┬───────────────────┬───────────────────────────────┤
│    Celestrak    │    Space-Track    │   SIMBAD/NED/NASA APIs        │
└────────┬────────┴─────────┬─────────┴─────────────┬─────────────────┘
         │                  │                       │
         ▼                  ▼                       ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Kubernetes CronJobs                              │
├─────────────────┬───────────────────┬───────────────────────────────┤
│  tle-updater    │ catalog-syncer    │   ephemeris-generator         │
│  (hourly)       │ (weekly)          │   (daily)                     │
└────────┬────────┴─────────┬─────────┴─────────────┬─────────────────┘
         │                  │                       │
         ▼                  ▼                       ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        PostgreSQL Database                          │
│                    (nyx schema with pgvector)                       │
└─────────────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Nyx API                                    │
│              (serves cached and computed data)                      │
└─────────────────────────────────────────────────────────────────────┘
```

## Pipeline Details

### TLE Update Pipeline (`tle-updater`)

Updates Two-Line Element sets for satellite tracking.

**Sources:**

- **Celestrak** (primary, free): `https://celestrak.org/NORAD/elements/`
- **Space-Track** (secondary, authenticated): `https://www.space-track.org/`

**Categories Updated:**

- Space stations (ISS, CSS)
- Active satellites
- Starlink constellation
- GPS/GLONASS/Galileo
- Weather satellites
- Amateur radio satellites

**Process:**

1. Fetch TLE data from Celestrak (no auth required)
2. Fall back to Space-Track for unavailable categories
3. Parse and validate TLE format (checksum verification)
4. Calculate derived orbital parameters (apogee, perigee, period)
5. Store in database with timestamp
6. Mark stale entries (>14 days old) for cleanup

### Catalog Sync Pipeline (`catalog-syncer`)

Synchronizes astronomical catalogs from authoritative sources.

**Catalogs:**

- **Stars**: Hipparcos, Gaia DR3 (subset), Bright Star Catalog
- **Deep Sky**: Messier, NGC/IC, PGC
- **Exoplanets**: NASA Exoplanet Archive
- **NEOs**: Minor Planet Center

**Process:**

1. Query TAP services for updated entries
2. Cross-match with existing catalog data
3. Apply deduplication and confidence scoring
4. Update database with new/modified entries
5. Generate catalog statistics

### Ephemeris Precompute Pipeline (`ephemeris-generator`)

Pre-calculates planetary positions for fast API responses.

**Objects:**

- Sun, Moon
- Mercury, Venus, Mars, Jupiter, Saturn, Uranus, Neptune

**Output:**

- 1-minute resolution ephemeris for 7 days ahead
- 1-hour resolution ephemeris for 30 days ahead
- Rise/set/transit times for multiple observer locations

**Process:**

1. Calculate ephemeris using VSOP87/ELP2000 theories
2. Apply corrections (precession, nutation, aberration)
3. Store in database with interpolation coefficients
4. Invalidate API cache for updated ranges

### Event Calculation Pipeline (`event-calculator`)

Calculates upcoming astronomical events.

**Event Types:**

- Solar/Lunar eclipses
- Planetary conjunctions
- Meteor showers
- Satellite visible passes

**Process:**

1. Run event detection algorithms
2. Calculate visibility for key observer locations
3. Store events with geometry details
4. Update event feed JSON for CDN

## Configuration

### Environment Variables

| Variable                | Description                  | Default                    |
| ----------------------- | ---------------------------- | -------------------------- |
| `DATABASE_URL`          | PostgreSQL connection string | Required                   |
| `REDIS_URL`             | Redis for caching and locks  | Optional                   |
| `CELESTRAK_BASE_URL`    | Celestrak API base           | `https://celestrak.org`    |
| `SPACETRACK_USERNAME`   | Space-Track login            | Required for TLE           |
| `SPACETRACK_PASSWORD`   | Space-Track password         | Required for TLE           |
| `TLE_UPDATE_CATEGORIES` | Comma-separated categories   | `stations,active,starlink` |
| `CATALOG_SYNC_SOURCES`  | Comma-separated sources      | `hipparcos,messier,ngc`    |
| `EPHEMERIS_DAYS_AHEAD`  | Days to precompute           | `30`                       |
| `LOG_LEVEL`             | Logging verbosity            | `info`                     |

### Secrets

Create the `nyx-pipeline-secrets` secret with:

```bash
kubectl create secret generic nyx-pipeline-secrets \
  --from-literal=SPACETRACK_USERNAME=your_username \
  --from-literal=SPACETRACK_PASSWORD=your_password \
  --from-literal=DATABASE_URL=postgresql://... \
  -n nyx
```

## Deployment

### Prerequisites

1. PostgreSQL database with nyx schema initialized
2. (Optional) Space-Track account for authenticated TLE access
3. Network access to external data sources

### Apply Configuration

```bash
# Development
kubectl apply -k infra/nyx/pipelines/overlays/dev

# Staging
kubectl apply -k infra/nyx/pipelines/overlays/staging

# Production
kubectl apply -k infra/nyx/pipelines/overlays/prod
```

### Verify Deployment

```bash
# List cronjobs
kubectl get cronjobs -n nyx

# View recent job runs
kubectl get jobs -n nyx --sort-by=.status.startTime

# Check logs for a specific job
kubectl logs -n nyx job/tle-updater-<timestamp>
```

### Manual Trigger

Trigger a pipeline manually for testing:

```bash
# Trigger TLE update
kubectl create job --from=cronjob/tle-updater manual-tle-$(date +%s) -n nyx

# Trigger catalog sync
kubectl create job --from=cronjob/catalog-syncer manual-catalog-$(date +%s) -n nyx
```

## Monitoring

### Prometheus Metrics

Each pipeline exports metrics on port 9090:

| Metric                           | Type      | Description               |
| -------------------------------- | --------- | ------------------------- |
| `nyx_pipeline_duration_seconds`  | Histogram | Pipeline execution time   |
| `nyx_pipeline_records_processed` | Counter   | Records processed per run |
| `nyx_pipeline_errors_total`      | Counter   | Pipeline error count      |
| `nyx_tle_staleness_seconds`      | Gauge     | Age of oldest TLE         |
| `nyx_catalog_entries_total`      | Gauge     | Total catalog entries     |

### Alerts

Configure alerts for:

- Pipeline job failures (>2 consecutive failures)
- TLE staleness (>24 hours for active satellites)
- Database write errors
- External API unavailability

### Grafana Dashboard

Import the Nyx Pipelines dashboard from:
`infra/nyx/monitoring/dashboards/pipelines.json`

## Troubleshooting

### TLE Update Failures

**Symptom:** TLE update job fails repeatedly

**Check:**

1. Verify Celestrak is accessible:
   `curl https://celestrak.org/NORAD/elements/gp.php?GROUP=stations`
2. Check Space-Track credentials are valid
3. Review job logs for specific errors

### Database Connection Issues

**Symptom:** Jobs fail with connection errors

**Check:**

1. Verify `DATABASE_URL` secret is correct
2. Check PostgreSQL pod is running
3. Verify network policies allow job→database traffic

### Stale Data

**Symptom:** API returns old satellite positions

**Check:**

1. Verify CronJob schedule is active
2. Check job history for successful completions
3. Query database for TLE timestamps

## Cost Considerations

### API Rate Limits

- **Celestrak**: No authentication, reasonable use expected
- **Space-Track**: 30 requests/minute, 300/hour for authenticated users
- **SIMBAD/VizieR**: TAP services have query size limits

### Resource Usage

| Pipeline     | CPU Request | Memory Request | Typical Duration |
| ------------ | ----------- | -------------- | ---------------- |
| TLE Update   | 100m        | 256Mi          | 2-5 minutes      |
| Catalog Sync | 200m        | 512Mi          | 10-30 minutes    |
| Ephemeris    | 500m        | 1Gi            | 5-15 minutes     |
| Events       | 200m        | 512Mi          | 5-10 minutes     |

## Related Documentation

- [Nyx API Documentation](../../../docs/domains/nyx/api/)
- [Satellite Tracking Library](../../../libs/nyx/realtime/satellites/)
- [Ephemeris Generator](../../../libs/nyx/ephemeris/)
- [Catalog Libraries](../../../libs/nyx/catalogs/)
