## Phase 87: ML Sovereignty — Dataset Management & Synthetic Data Generation

Centralized infrastructure for managing, versioning, curating, and generating
training datasets across all domains. This is the backbone that connects the
data flywheel (Phases 85-86) to the training infrastructure (Phase 89).

---

### 87.1 Dataset Registry & Versioning

#### 87.1.1 Registry Core

- [x] **87.1.1.1** Design dataset registry schema (DatasetId, name, version,
      domain, modality, format, sample count, size bytes, license, quality
      score, created by, lineage parent, schema version)
- [x] **87.1.1.2** Implement dataset registry PostgreSQL storage with full CRUD
      operations and version history
- [x] **87.1.1.3** Build content-addressable storage backend for dataset
      artifacts (SHA-256 hashed chunks in S3/MinIO, delta-only storage between
      versions)
- [x] **87.1.1.4** Create dataset CLI tool (oshun-dataset
      create/push/pull/list/diff/merge/split/validate/export)
- [x] **87.1.1.5** Build dataset format converters: JSONL, Parquet, HuggingFace
      Datasets, Alpaca, ShareGPT, OpenAI Chat, CSV, Arrow (bidirectional)

#### 87.1.2 Registry Features

- [x] **87.1.2.1** Implement dataset split management (train/val/test with
      configurable ratios, stratified splitting, contamination-preventing
      held-out set isolation)
- [x] **87.1.2.2** Build dataset card auto-generator (HuggingFace format:
      description, intended use, limitations, biases, training data sources,
      license, citation)
- [x] **87.1.2.3** Implement dataset diff tool (compare versions:
      added/removed/modified samples, distribution shift metrics, schema
      changes)
- [x] **87.1.2.4** Build dataset search and discovery (full-text search on
      descriptions + filter by domain/modality/license/quality + similarity
      search by task type)
- [x] **87.1.2.5** Implement dataset access control (RBAC: who can
      read/write/delete datasets, per-domain permissions, audit logging)
- [x] **87.1.2.6** Build dataset statistics dashboard (sample count over time,
      quality distribution, format breakdown, per-domain coverage, storage
      usage)
- [x] **87.1.2.7** Implement dataset garbage collection (auto-expire staging
      datasets after configurable TTL, archive old versions to Glacier)
- [x] **87.1.2.8** Build dataset export pipeline for external sharing
      (anonymization, license bundling, format conversion, validation)
- [x] **87.1.2.9** Implement dataset import from HuggingFace Hub (one-command
      import with automatic format detection and quality scoring)
- [x] **87.1.2.10** Build data mixing recipe system (specify domain ratios,
      curriculum schedules, upsampling/downsampling rules → produce
      training-ready mixed dataset)

### 87.2 Data Quality & Curation

#### 87.2.1 Quality Pipeline

- [x] **87.2.1.1** Build multi-dimensional quality scorer (completeness,
      consistency, length distribution, diversity, toxicity, information density
      — scores 0-1 per dimension)
- [x] **87.2.1.2** Implement exact and near-duplicate detection pipeline
      (MinHash for text, perceptual hashing for images, chromaprint for audio —
      running on ingest and on-demand)
- [x] **87.2.1.3** Build data contamination detector (check if evaluation
      benchmark samples appear in training data — MMLU, HumanEval, etc.)
- [x] **87.2.1.4** Implement PII detection and scrubbing pipeline (Presidio for
      named entities, regex for structured PII, GLiNER for context-dependent PII
      — configurable per domain)
- [x] **87.2.1.5** Build license compliance checker (verify training data
      licenses, flag incompatible combinations, generate license summary for
      model cards)
- [x] **87.2.1.6** Implement toxicity and safety filtering (Llama Guard 4 or
      Granite Guardian for content classification, configurable thresholds per
      domain)
- [x] **87.2.1.7** Build data diversity analyzer (topic distribution, language
      distribution, source distribution, difficulty distribution — flag
      under-represented areas)
- [x] **87.2.1.8** Implement outlier detection and removal (statistical outliers
      by length, quality, embedding distance — configurable sensitivity)
- [x] **87.2.1.9** Build automated data quality reports (weekly summaries per
      domain: new data collected, quality trends, coverage gaps, flagged issues)
- [x] **87.2.1.10** Implement data lineage tracking (which source data → which
      processing steps → which training runs → which models, full DAG
      visualization)

### 87.3 Synthetic Data Generation

#### 87.3.1 Text & Instruction Data Generation

- [x] **87.3.1.1** Build Magpie-style self-instruct pipeline (prompt aligned LLM
      with pre-query template only → generate diverse instruction-response pairs
      → quality filter, per ICLR 2025)
- [x] **87.3.1.2** Build Persona Hub pipeline (curate diverse personas from web
      data → use each persona as seed for domain-specific instruction
      generation, per Tencent 2024)
- [x] **87.3.1.3** Build GLAN-style taxonomy-driven generation (traverse human
      knowledge taxonomy: subject → discipline → field → generate comprehensive
      instruction coverage)
- [x] **87.3.1.4** Build Evol-Instruct pipeline (take seed instructions → evolve
      complexity through deepening, widening, complication, reasoning steps →
      difficulty-graded instruction data)
- [x] **87.3.1.5** Build synthetic preference pair generator (for each prompt:
      generate N responses with different models/temperatures → rank by quality
      → chosen/rejected pairs for DPO/SimPO/KTO)
- [x] **87.3.1.6** Build CoT-Self-Instruct pipeline (chain-of-thought reasoning
      to create high-quality synthetic prompts → self-filter for quality →
      verifiable reasoning and instruction data)
- [x] **87.3.1.7** Build Nemotron-style reward data generator (use
      multidimensional reward model to rate synthetic responses across
      helpfulness/harmlessness/honesty → reward model training data)
- [x] **87.3.1.8** Build comparative knowledge distillation pipeline (generate N
      responses per prompt → create N-squared pairwise comparisons → richer
      training signal without additional API cost)

#### 87.3.2 Domain-Specific Synthetic Data

- [x] **87.3.1.9** Build synthetic NER data generator for Ghanaian entities
      (generate Twi/Ewe/Ga/Hausa text with entity annotations using LLM + Ghana
      NLP knowledge base)
- [x] **87.3.1.10** Build synthetic code generation pipeline (generate
      programming problems → solutions → explanations → test cases for code
      model training)
- [x] **87.3.1.11** Build synthetic dialogue generator for NPC training
      (character-consistent multi-turn dialogues with personality constraints
      from Hathor personality models)
- [x] **87.3.1.12** Build diffusion-based synthetic image dataset generator
      (training images with controlled variation: style, composition, lighting,
      subject → diversity-controlled)
- [x] **87.3.1.13** Build TTS synthetic audio data pipeline (multi-speaker,
      multi-language audio training data using XTTS/Voxtral → with phoneme
      alignment for lip-sync training)
- [x] **87.3.1.14** Build adversarial/red-team data generator (jailbreak
      attempts, prompt injections, harmful instructions per category taxonomy →
      red-team eval sets and safety training data)
- [x] **87.3.1.15** Build multi-lingual synthetic data pipeline for low-resource
      Ghanaian languages (Twi, Ewe, Ga, Hausa, Dagbani using cross-lingual
      transfer + back-translation + Persona Hub)
- [x] **87.3.1.16** Build quality filter for synthetic data (decontamination
      check, deduplication, multi-dimensional quality scoring, reward model
      filtering → retain only top-k percent samples)
- [x] **87.3.1.17** Build TreeSynth-style diversity-controlled generation
      (tree-structured subspace partitioning for systematic coverage of
      topic/difficulty/style space)
- [x] **87.3.1.18** Build Infinity-Instruct-style attribute-guided generation at
      scale (combine selection from existing data + synthesis of new data with
      attribute steering)
- [x] **87.3.1.19** Build SDG Hub integration for composable synthetic data
      pipelines (Red Hat's modular block system: standard Input→Process→Output
      interface, pre-built flows for knowledge tuning, custom pipeline
      composition, data mixing integration with dataset registry)
- [x] **87.3.1.20** Integrate InstructLab taxonomy-driven data generation
      (IBM/Red Hat community model: submit skill/knowledge recipes to taxonomy →
      synthetic data augmentation from curated seeds → domain-specific training
      data without full retraining)
- [x] **87.3.1.21** Build Cosmopedia-style synthetic pre-training corpus
      generator (large-scale synthetic textbooks, articles, stories covering
      hundreds of topics → pre-training-ready corpus for continual pre-training,
      distinct from instruction fine-tuning data)
- [x] **87.3.1.22** Build synthetic VLM training data generator (image-text
      pairs with visual reasoning chains, visual QA pairs, chart/diagram
      understanding pairs, OCR training pairs → vision-language model
      fine-tuning data)
- [x] **87.3.1.23** Build synthetic video understanding data generator (video
      clip descriptions, temporal reasoning QA, action recognition labels,
      video-text alignment pairs → video model training data)
- [x] **87.3.1.24** Build multi-modal interleaved training data generator
      (documents with mixed text+images+tables, interleaved reasoning across
      modalities → unified multimodal model training data)

### 87.4 Human-in-the-Loop Annotation

#### 87.4.1 Annotation Platform & Feedback Pipeline

- [x] **87.4.1.1** Deploy self-hosted Label Studio instance for annotation
      backend (multi-type labeling with standardized output)
- [x] **87.4.1.2** Integrate Argilla for AI-assisted annotation (AI feedback
      suggestions, semantic search, quality filtering integration)
- [x] **87.4.1.3** Build preference comparison UI (side-by-side A/B interface:
      show two model responses, annotator picks preferred → DPO training data)
- [x] **87.4.1.4** Build thumbs-up/down feedback ingestion from all production
      endpoints (Iris, Psyche, Isis → standardized feedback events → preference
      data)
- [x] **87.4.1.5** Build annotation task queue with priority, assignment, and
      deadline management
- [x] **87.4.1.6** Implement inter-annotator agreement calculator (Cohen's kappa
      for 2 annotators, Fleiss' kappa for 3+, Krippendorff's alpha for mixed
      types)
- [x] **87.4.1.7** Build gold standard quality checks (inject known-answer
      questions at configurable rate, flag annotators below accuracy threshold)
- [x] **87.4.1.8** Create annotator calibration workflow (training phase with
      examples + certification test before production annotation)
- [x] **87.4.1.9** Build active learning loop (use model uncertainty to route
      most informative samples to annotators → maximize annotation budget
      efficiency)
- [x] **87.4.1.10** Implement RLHF data pipeline (feedback events → preference
      pair formatting → deduplication → quality filtering → training-ready
      DPO/KTO/SimPO datasets)
- [x] **87.4.1.11** Build red team dataset curation tool (organize adversarial
      prompts by harm category from Kuan Yin taxonomy, track coverage per
      category)
- [x] **87.4.1.12** Build evaluation set management (maintain held-out eval sets
      per domain with contamination monitoring)
- [x] **87.4.1.13** Build annotation analytics dashboard (throughput per
      annotator, quality scores, cost per annotation, task completion rates)
- [x] **87.4.1.14** Implement multi-annotator consensus resolution workflow
      (majority voting, adjudication for disagreements, escalation path)
- [x] **87.4.1.15** Build stepwise process reward annotation tool (annotators
      label each reasoning step as correct/incorrect/unnecessary → PRM training
      data)
- [x] **87.4.1.16** Integrate Distilabel (Argilla's pipeline framework) for
      research-paper-backed synthetic data generation (programmatic pipeline
      composition, LLM batch API cost reduction, step caching, integration with
      Argilla annotation for human validation loop)
- [x] **87.4.1.17** Build Argilla deep curation workflow (ChatField for
      conversational data review, CustomField templates for domain-specific
      annotation, real-time collaborative annotation sessions, synthetic →
      curate → approve → dataset registry pipeline)

#### 87.4.2 Data Flywheel Integration

- [x] **87.4.2.1** Build data flywheel ingestion pipeline (consume events from
      Phase 85/86 data flywheel → auto-format into dataset registry entries →
      apply quality scoring → version and catalog, with configurable domain
      routing and deduplication)
- [x] **87.4.2.2** Build automated training dataset assembly from flywheel data
      (trigger on new flywheel batch → quality filter → mix with existing domain
      datasets per recipe → produce training-ready dataset versions → notify
      training pipeline)

---

## Completion evidence

All 68 locally actionable requirements are implemented and mapped one-to-one in
`PHASE_87_CAPABILITIES`; `auditPhase87Coverage()` fails if an item is missing,
duplicated, or lacks implementation/test evidence.

- Registry, PostgreSQL, S3/MinIO CAS, CLI, converters, splitting, cards, diff,
  search/RBAC/audit, statistics, lifecycle, export/import, and mixing live in
  `libs/nous/training/src/phase-87-dataset-registry.ts` and
  `libs/nous/training/src/oshun-dataset-cli.ts`.
- Quality scoring, multimodal deduplication ports, contamination, PII,
  licensing, guardian filtering, diversity/outliers, reports, and lineage live
  in `libs/nous/training/src/phase-87-curation.ts`.
- All 24 synthetic strategies, structured domain validation, reward pairing,
  media/TTS ports, taxonomy/TreeSynth expansion, SDG composition, InstructLab,
  and top-k quality filtering live in
  `libs/nous/training/src/phase-87-synthetic-data.ts`.
- Label Studio/Argilla clients, annotation queue/quality/agreement/consensus,
  RLHF/PRM/evaluation/red-team/analytics workflows, Distilabel caching, deep
  curation, flywheel ingestion, and training-set assembly live in
  `libs/nous/training/src/phase-87-annotation-flywheel.ts`.
- The self-hosted dependency stack is in `deploy/nous/phase-87`. The preference
  review and dataset/annotation operations surfaces are under
  `apps/oshun/web/src/app/studio/nous`, with component and Playwright coverage.

Verification is provided by the focused `phase-87-*.spec.ts` suites in
`libs/nous/training`, the two Nous web component suites, the two Nous Playwright
specs, targeted typecheck/lint/build gates, Compose validation, and the Phase 87
coverage audit.
