The ingestion service now enforces a deterministic processing pipeline across every connector import. The pipeline is designed to eliminate duplicate payloads, remove low–value boilerplate text, detect language automatically, and persist segmentation metadata for downstream systems.
Pipeline stages#
-
Acquisition & Extraction
- Inline uploads are accepted verbatim; PDF sources are simulated for
example.orgfixtures and otherwise parsed viapdf-parsewith OCR fallback when confidence < 0.6. - The extraction stage preserves source metadata (pages, word counts,
collection hints) in
processing_metadata.
- Inline uploads are accepted verbatim; PDF sources are simulated for
-
Boilerplate Removal
- Headers, horizontal rules, and recurring disclaimers (e.g., Lilith
Foundation notices) are stripped via
cleanIngestedContent. - Duplicate short lines are discarded and whitespace is normalised before segmentation.
- Headers, horizontal rules, and recurring disclaimers (e.g., Lilith
Foundation notices) are stripped via
-
Language Detection
detectLanguagesanalyses ASCII vs. accented character ratios to provide the top candidates with confidence scores; the primary language is promoted to the document record for index filtering.
-
Deduplication
- Every cleaned payload is hashed (
sha256) to create a fingerprint. - The first document for a fingerprint is marked canonical; subsequent
matches reuse the canonical content, mark
deduplication.is_duplicate = true, and exposeduplicate_ofplusreused_from_documentmetadata while keeping the job statuscompletedfor compatibility with existing suites.
- Every cleaned payload is hashed (
-
Segmentation
segmentContentsplits cleaned text into headings, paragraphs, quotes, and list items with absolute character offsets. Segments are stored in-memory for/v1/documents/:docId/segmentsand counted in document metadata.
API surface updates#
GET /v1/documents/:docId/segmentsreturns the canonical segment list for a processed document.- Duplicate documents expose:
document.duplicate_ofdocument.deduplication(including fingerprint and first-seen timestamp)job.processing_metadata.reused_from_document
Test coverage#
tests/ingestion.processing-pipeline.spec.ts- Verifies dedup metadata propagation and ensures boilerplate stripping, language detection, and segmentation output remain intact.
tests/ingestion.pdf-processing.spec.ts- Exercises the simulated PDF fixtures, OCR fallback path, chunking, document-store integration, and entity extraction.
tests/ingestion.document-store-layouts.spec.ts- Confirms checksum integrity alongside the new fingerprint bookkeeping when documents are persisted.
Operational notes#
- The fingerprint index is kept in-memory today; replace it with a durable store before clustering the ingestion service.
- Simulated
example.orgPDFs provide deterministic fixtures for CI. Real-world connectors should still stream and parse raw binaries. - The entity extractor now leverages the cleaned text so deduped documents surface the same concepts as their canonical source.