Eve library

phase-87.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 87: ML Sovereignty — Dataset Management & Synthetic Data Generation#

Centralized infrastructure for managing, versioning, curating, and generating training datasets across all domains. This is the backbone that connects the data flywheel (Phases 85-86) to the training infrastructure (Phase 89).


87.1 Dataset Registry & Versioning#

87.1.1 Registry Core#

  • 87.1.1.1 Design dataset registry schema (DatasetId, name, version, domain, modality, format, sample count, size bytes, license, quality score, created by, lineage parent, schema version)
  • 87.1.1.2 Implement dataset registry PostgreSQL storage with full CRUD operations and version history
  • 87.1.1.3 Build content-addressable storage backend for dataset artifacts (SHA-256 hashed chunks in S3/MinIO, delta-only storage between versions)
  • 87.1.1.4 Create dataset CLI tool (oshun-dataset create/push/pull/list/diff/merge/split/validate/export)
  • 87.1.1.5 Build dataset format converters: JSONL, Parquet, HuggingFace Datasets, Alpaca, ShareGPT, OpenAI Chat, CSV, Arrow (bidirectional)

87.1.2 Registry Features#

  • 87.1.2.1 Implement dataset split management (train/val/test with configurable ratios, stratified splitting, contamination-preventing held-out set isolation)
  • 87.1.2.2 Build dataset card auto-generator (HuggingFace format: description, intended use, limitations, biases, training data sources, license, citation)
  • 87.1.2.3 Implement dataset diff tool (compare versions: added/removed/modified samples, distribution shift metrics, schema changes)
  • 87.1.2.4 Build dataset search and discovery (full-text search on descriptions + filter by domain/modality/license/quality + similarity search by task type)
  • 87.1.2.5 Implement dataset access control (RBAC: who can read/write/delete datasets, per-domain permissions, audit logging)
  • 87.1.2.6 Build dataset statistics dashboard (sample count over time, quality distribution, format breakdown, per-domain coverage, storage usage)
  • 87.1.2.7 Implement dataset garbage collection (auto-expire staging datasets after configurable TTL, archive old versions to Glacier)
  • 87.1.2.8 Build dataset export pipeline for external sharing (anonymization, license bundling, format conversion, validation)
  • 87.1.2.9 Implement dataset import from HuggingFace Hub (one-command import with automatic format detection and quality scoring)
  • 87.1.2.10 Build data mixing recipe system (specify domain ratios, curriculum schedules, upsampling/downsampling rules → produce training-ready mixed dataset)

87.2 Data Quality & Curation#

87.2.1 Quality Pipeline#

  • 87.2.1.1 Build multi-dimensional quality scorer (completeness, consistency, length distribution, diversity, toxicity, information density — scores 0-1 per dimension)
  • 87.2.1.2 Implement exact and near-duplicate detection pipeline (MinHash for text, perceptual hashing for images, chromaprint for audio — running on ingest and on-demand)
  • 87.2.1.3 Build data contamination detector (check if evaluation benchmark samples appear in training data — MMLU, HumanEval, etc.)
  • 87.2.1.4 Implement PII detection and scrubbing pipeline (Presidio for named entities, regex for structured PII, GLiNER for context-dependent PII — configurable per domain)
  • 87.2.1.5 Build license compliance checker (verify training data licenses, flag incompatible combinations, generate license summary for model cards)
  • 87.2.1.6 Implement toxicity and safety filtering (Llama Guard 4 or Granite Guardian for content classification, configurable thresholds per domain)
  • 87.2.1.7 Build data diversity analyzer (topic distribution, language distribution, source distribution, difficulty distribution — flag under-represented areas)
  • 87.2.1.8 Implement outlier detection and removal (statistical outliers by length, quality, embedding distance — configurable sensitivity)
  • 87.2.1.9 Build automated data quality reports (weekly summaries per domain: new data collected, quality trends, coverage gaps, flagged issues)
  • 87.2.1.10 Implement data lineage tracking (which source data → which processing steps → which training runs → which models, full DAG visualization)

87.3 Synthetic Data Generation#

87.3.1 Text & Instruction Data Generation#

  • 87.3.1.1 Build Magpie-style self-instruct pipeline (prompt aligned LLM with pre-query template only → generate diverse instruction-response pairs → quality filter, per ICLR 2025)
  • 87.3.1.2 Build Persona Hub pipeline (curate diverse personas from web data → use each persona as seed for domain-specific instruction generation, per Tencent 2024)
  • 87.3.1.3 Build GLAN-style taxonomy-driven generation (traverse human knowledge taxonomy: subject → discipline → field → generate comprehensive instruction coverage)
  • 87.3.1.4 Build Evol-Instruct pipeline (take seed instructions → evolve complexity through deepening, widening, complication, reasoning steps → difficulty-graded instruction data)
  • 87.3.1.5 Build synthetic preference pair generator (for each prompt: generate N responses with different models/temperatures → rank by quality → chosen/rejected pairs for DPO/SimPO/KTO)
  • 87.3.1.6 Build CoT-Self-Instruct pipeline (chain-of-thought reasoning to create high-quality synthetic prompts → self-filter for quality → verifiable reasoning and instruction data)
  • 87.3.1.7 Build Nemotron-style reward data generator (use multidimensional reward model to rate synthetic responses across helpfulness/harmlessness/honesty → reward model training data)
  • 87.3.1.8 Build comparative knowledge distillation pipeline (generate N responses per prompt → create N-squared pairwise comparisons → richer training signal without additional API cost)

87.3.2 Domain-Specific Synthetic Data#

  • 87.3.1.9 Build synthetic NER data generator for Ghanaian entities (generate Twi/Ewe/Ga/Hausa text with entity annotations using LLM + Ghana NLP knowledge base)
  • 87.3.1.10 Build synthetic code generation pipeline (generate programming problems → solutions → explanations → test cases for code model training)
  • 87.3.1.11 Build synthetic dialogue generator for NPC training (character-consistent multi-turn dialogues with personality constraints from Hathor personality models)
  • 87.3.1.12 Build diffusion-based synthetic image dataset generator (training images with controlled variation: style, composition, lighting, subject → diversity-controlled)
  • 87.3.1.13 Build TTS synthetic audio data pipeline (multi-speaker, multi-language audio training data using XTTS/Voxtral → with phoneme alignment for lip-sync training)
  • 87.3.1.14 Build adversarial/red-team data generator (jailbreak attempts, prompt injections, harmful instructions per category taxonomy → red-team eval sets and safety training data)
  • 87.3.1.15 Build multi-lingual synthetic data pipeline for low-resource Ghanaian languages (Twi, Ewe, Ga, Hausa, Dagbani using cross-lingual transfer + back-translation + Persona Hub)
  • 87.3.1.16 Build quality filter for synthetic data (decontamination check, deduplication, multi-dimensional quality scoring, reward model filtering → retain only top-k percent samples)
  • 87.3.1.17 Build TreeSynth-style diversity-controlled generation (tree-structured subspace partitioning for systematic coverage of topic/difficulty/style space)
  • 87.3.1.18 Build Infinity-Instruct-style attribute-guided generation at scale (combine selection from existing data + synthesis of new data with attribute steering)
  • 87.3.1.19 Build SDG Hub integration for composable synthetic data pipelines (Red Hat's modular block system: standard Input→Process→Output interface, pre-built flows for knowledge tuning, custom pipeline composition, data mixing integration with dataset registry)
  • 87.3.1.20 Integrate InstructLab taxonomy-driven data generation (IBM/Red Hat community model: submit skill/knowledge recipes to taxonomy → synthetic data augmentation from curated seeds → domain-specific training data without full retraining)
  • 87.3.1.21 Build Cosmopedia-style synthetic pre-training corpus generator (large-scale synthetic textbooks, articles, stories covering hundreds of topics → pre-training-ready corpus for continual pre-training, distinct from instruction fine-tuning data)
  • 87.3.1.22 Build synthetic VLM training data generator (image-text pairs with visual reasoning chains, visual QA pairs, chart/diagram understanding pairs, OCR training pairs → vision-language model fine-tuning data)
  • 87.3.1.23 Build synthetic video understanding data generator (video clip descriptions, temporal reasoning QA, action recognition labels, video-text alignment pairs → video model training data)
  • 87.3.1.24 Build multi-modal interleaved training data generator (documents with mixed text+images+tables, interleaved reasoning across modalities → unified multimodal model training data)

87.4 Human-in-the-Loop Annotation#

87.4.1 Annotation Platform & Feedback Pipeline#

  • 87.4.1.1 Deploy self-hosted Label Studio instance for annotation backend (multi-type labeling with standardized output)
  • 87.4.1.2 Integrate Argilla for AI-assisted annotation (AI feedback suggestions, semantic search, quality filtering integration)
  • 87.4.1.3 Build preference comparison UI (side-by-side A/B interface: show two model responses, annotator picks preferred → DPO training data)
  • 87.4.1.4 Build thumbs-up/down feedback ingestion from all production endpoints (Iris, Psyche, Isis → standardized feedback events → preference data)
  • 87.4.1.5 Build annotation task queue with priority, assignment, and deadline management
  • 87.4.1.6 Implement inter-annotator agreement calculator (Cohen's kappa for 2 annotators, Fleiss' kappa for 3+, Krippendorff's alpha for mixed types)
  • 87.4.1.7 Build gold standard quality checks (inject known-answer questions at configurable rate, flag annotators below accuracy threshold)
  • 87.4.1.8 Create annotator calibration workflow (training phase with examples + certification test before production annotation)
  • 87.4.1.9 Build active learning loop (use model uncertainty to route most informative samples to annotators → maximize annotation budget efficiency)
  • 87.4.1.10 Implement RLHF data pipeline (feedback events → preference pair formatting → deduplication → quality filtering → training-ready DPO/KTO/SimPO datasets)
  • 87.4.1.11 Build red team dataset curation tool (organize adversarial prompts by harm category from Kuan Yin taxonomy, track coverage per category)
  • 87.4.1.12 Build evaluation set management (maintain held-out eval sets per domain with contamination monitoring)
  • 87.4.1.13 Build annotation analytics dashboard (throughput per annotator, quality scores, cost per annotation, task completion rates)
  • 87.4.1.14 Implement multi-annotator consensus resolution workflow (majority voting, adjudication for disagreements, escalation path)
  • 87.4.1.15 Build stepwise process reward annotation tool (annotators label each reasoning step as correct/incorrect/unnecessary → PRM training data)
  • 87.4.1.16 Integrate Distilabel (Argilla's pipeline framework) for research-paper-backed synthetic data generation (programmatic pipeline composition, LLM batch API cost reduction, step caching, integration with Argilla annotation for human validation loop)
  • 87.4.1.17 Build Argilla deep curation workflow (ChatField for conversational data review, CustomField templates for domain-specific annotation, real-time collaborative annotation sessions, synthetic → curate → approve → dataset registry pipeline)

87.4.2 Data Flywheel Integration#

  • 87.4.2.1 Build data flywheel ingestion pipeline (consume events from Phase 85/86 data flywheel → auto-format into dataset registry entries → apply quality scoring → version and catalog, with configurable domain routing and deduplication)
  • 87.4.2.2 Build automated training dataset assembly from flywheel data (trigger on new flywheel batch → quality filter → mix with existing domain datasets per recipe → produce training-ready dataset versions → notify training pipeline)

Completion evidence#

All 68 locally actionable requirements are implemented and mapped one-to-one in PHASE_87_CAPABILITIES; auditPhase87Coverage() fails if an item is missing, duplicated, or lacks implementation/test evidence.

  • Registry, PostgreSQL, S3/MinIO CAS, CLI, converters, splitting, cards, diff, search/RBAC/audit, statistics, lifecycle, export/import, and mixing live in libs/nous/training/src/phase-87-dataset-registry.ts and libs/nous/training/src/oshun-dataset-cli.ts.
  • Quality scoring, multimodal deduplication ports, contamination, PII, licensing, guardian filtering, diversity/outliers, reports, and lineage live in libs/nous/training/src/phase-87-curation.ts.
  • All 24 synthetic strategies, structured domain validation, reward pairing, media/TTS ports, taxonomy/TreeSynth expansion, SDG composition, InstructLab, and top-k quality filtering live in libs/nous/training/src/phase-87-synthetic-data.ts.
  • Label Studio/Argilla clients, annotation queue/quality/agreement/consensus, RLHF/PRM/evaluation/red-team/analytics workflows, Distilabel caching, deep curation, flywheel ingestion, and training-set assembly live in libs/nous/training/src/phase-87-annotation-flywheel.ts.
  • The self-hosted dependency stack is in deploy/nous/phase-87. The preference review and dataset/annotation operations surfaces are under apps/oshun/web/src/app/studio/nous, with component and Playwright coverage.

Verification is provided by the focused phase-87-*.spec.ts suites in libs/nous/training, the two Nous web component suites, the two Nous Playwright specs, targeted typecheck/lint/build gates, Compose validation, and the Phase 87 coverage audit.