Eve library

Anonymization & Pseudonymization Pipelines

Guide

Revision 1

On this page

Document Version: 1.0 Last Updated: 2025-10-16 Owners: Privacy Engineering & Data Governance Team

This document describes the anonymization and pseudonymization pipelines, privacy budget tracking, and integration with data governance lifecycle management.

Overview#

The anonymization service provides comprehensive privacy-preserving data transformation using industry-standard algorithms:

  • K-Anonymity: Generalizes quasi-identifiers to ensure at least k indistinguishable records
  • L-Diversity: Ensures at least l distinct sensitive attribute values in each equivalence class
  • Differential Privacy: Adds calibrated noise to protect individual privacy in aggregated queries
  • Pseudonymization: Reversible/irreversible identifier transformation

Architecture#

text
┌──────────────────┐         ┌────────────────────────┐
│ Data Governance  │────────▶│  Anonymization Service │
│   Service        │         │                        │
│                  │         │  - K-Anonymity         │
│  Lifecycle:      │         │  - L-Diversity         │
│  ANONYMIZE action│         │  - Differential Privacy│
│                  │         │  - Pseudonymization    │
└──────────────────┘         │  - Privacy Budgets     │
                             └────────────────────────┘
                                         │
                                         ▼
                             ┌────────────────────────┐
                             │   Anonymized Datasets  │
                             │   Privacy Budget Ledger│
                             └────────────────────────┘

Anonymization Methods#

1. K-Anonymity#

Endpoint: POST /v1/anonymization/k-anonymity

Ensures that each record is indistinguishable from at least k-1 other records based on quasi-identifiers.

Example Request:

json
{
  "dataset": [
    { "age": 25, "city": "New York", "salary": 75000 },
    { "age": 26, "city": "New York", "salary": 78000 },
    { "age": 27, "city": "New York", "salary": 72000 }
  ],
  "quasi_identifiers": ["age", "city"],
  "sensitive_attributes": ["salary"],
  "k": 3
}

Response:

json
{
  "group_id": "k-anon-1",
  "anonymized_dataset": [
    { "age": "20-29", "city": "New***", "salary": 75000 },
    { "age": "20-29", "city": "New***", "salary": 78000 },
    { "age": "20-29", "city": "New***", "salary": 72000 }
  ],
  "k_value": 3,
  "anonymization_method": "k-anonymity",
  "groups_created": 1,
  "privacy_loss": 0.67,
  "applied_at": "2025-10-16T10:00:00Z"
}

2. L-Diversity#

Endpoint: POST /v1/anonymization/l-diversity

Ensures each equivalence class has at least l well-represented sensitive attribute values.

Example Request:

json
{
  "dataset": [
    { "zip": "10001", "age": 25, "disease": "flu" },
    { "zip": "10001", "age": 26, "disease": "covid" },
    { "zip": "10001", "age": 27, "disease": "cold" }
  ],
  "quasi_identifiers": ["zip", "age"],
  "sensitive_attribute": "disease",
  "l": 3
}

3. Differential Privacy#

Endpoint: POST /v1/anonymization/differential-privacy

Adds calibrated Laplace or Gaussian noise to protect individual privacy in aggregated statistics.

Example Request:

json
{
  "query": "SELECT COUNT(*) FROM users WHERE age > 25",
  "result": 1523,
  "sensitivity": 1,
  "epsilon": 1.0,
  "noise_type": "laplace"
}

Response:

json
{
  "query_id": "dp-query-1",
  "query": "SELECT COUNT(*) FROM users WHERE age > 25",
  "original_result": 1523,
  "noisy_result": 1528,
  "epsilon": 1.0,
  "sensitivity": 1,
  "noise_added": 5,
  "privacy_guarantee": "(1.0, 0)-differential privacy",
  "noise_type": "laplace",
  "applied_at": "2025-10-16T10:00:00Z"
}

4. Pseudonymization#

Endpoint: POST /v1/anonymization/pseudonymize

Creates reversible pseudonyms or irreversible anonymous identifiers.

Reversible Pseudonymization:

json
{
  "identifiers": ["user-12345", "user-67890"],
  "context": "analytics",
  "reversible": true
}

Response:

json
{
  "mappings": [
    {
      "original_id": "user-12345",
      "pseudonym": "pseudo_analytics_a3f8e9b..._1a2b3c",
      "context": "analytics",
      "reversible": true,
      "algorithm": "aes-256-cbc",
      "created_at": "2025-10-16T10:00:00Z"
    }
  ],
  "mapping_type": "pseudonymization"
}

Irreversible Anonymization:

json
{
  "identifiers": ["user-12345"],
  "context": "research",
  "reversible": false
}

Privacy Budget Management#

Privacy budgets track the cumulative privacy loss from repeated queries, essential for differential privacy guarantees.

Creating a Privacy Budget#

Endpoint: POST /v1/privacy-budgets

json
{
  "user_id": "user-12345",
  "total_budget": 100,
  "period_days": 30,
  "auto_reset": true,
  "budget_type": "differential_privacy"
}

Response:

json
{
  "id": "budget-1",
  "user_id": "user-12345",
  "total_budget": 100,
  "consumed_budget": 0,
  "remaining_budget": 100,
  "period_days": 30,
  "auto_reset": true,
  "period_start": "2025-10-16T00:00:00Z",
  "period_end": "2025-11-15T00:00:00Z",
  "consumption_log": []
}

Consuming Budget#

Endpoint: POST /v1/privacy-budgets/:user_id/consume

json
{
  "operation": "differential_privacy_query",
  "cost": 5,
  "metadata": {
    "query_id": "dp-query-1",
    "epsilon": 1.0
  }
}

Budget Exceeded (429):

json
{
  "code": "anonymization.budget_exceeded",
  "message": "Insufficient privacy budget",
  "remaining_budget": 2,
  "required_budget": 5,
  "period_end": "2025-11-15T00:00:00Z"
}

Privacy Budget for Memories#

User memories and personalization vectors consume privacy budget when:

  1. Memory retrieval: Queries accessing user memories cost budget proportional to epsilon
  2. Personalization inference: RAG queries over user data consume budget
  3. Analytics aggregation: Aggregated statistics about user behavior

Budget Allocation:

  • Standard tier: 100 epsilon per 30 days
  • Premium tier: 200 epsilon per 30 days
  • Research tier: 50 epsilon per 30 days (stricter privacy)

Budget Costs:

  • Simple memory retrieval: 1 epsilon
  • Complex RAG query: 2-5 epsilon
  • Aggregated analytics: 5-10 epsilon

Transformation Pipelines#

Pipelines orchestrate multiple anonymization steps in sequence.

Creating a Pipeline#

Endpoint: POST /v1/transformation-pipelines

json
{
  "pipeline_name": "User Data Anonymization",
  "source_service": "conversation",
  "transformation_steps": [
    {
      "type": "k-anonymity",
      "parameters": {
        "k": 5,
        "quasi_identifiers": ["age", "location"],
        "sensitive_attributes": ["message_content"]
      }
    },
    {
      "type": "pseudonymization",
      "parameters": {
        "context": "analytics",
        "reversible": false
      }
    }
  ],
  "schedule": "0 2 * * *",
  "enabled": true,
  "target_anonymization_level": "high"
}

Executing a Pipeline#

Endpoint: POST /v1/transformation-pipelines/:id/execute

json
{
  "dry_run": false
}

Integration with Data Governance#

The anonymization service integrates with data governance lifecycle management:

Lifecycle Action: ANONYMIZE#

When data-governance executes a lifecycle rule with the ANONYMIZE action:

  1. Data-governance service identifies records requiring anonymization
  2. Creates an anonymization job via POST /v1/anonymization/jobs
  3. Anonymization service executes the job using configured method (k-anonymity, l-diversity, etc.)
  4. Anonymized data is stored in anonymized_datasets
  5. Original data lifecycle_stage updated to ANONYMIZING then ANONYMIZED
  6. Audit trail created in data-governance service

Example Lifecycle Rule:

json
{
  "name": "Anonymize old PII after 2 years",
  "trigger_conditions": {
    "age_days": 730,
    "access_count_threshold": 0
  },
  "actions": ["ANONYMIZE", "DELETE"],
  "data_classifications": ["PII_HIGH", "PII_MEDIUM"]
}

Integration Flow#

text
Data Governance Service                    Anonymization Service
        │                                           │
        │  1. Lifecycle rule triggers               │
        │     ANONYMIZE action                      │
        │                                           │
        │  2. POST /v1/anonymization/jobs          │
        ├──────────────────────────────────────────▶│
        │                                           │
        │                                           │  3. Create job
        │                                           │     (PENDING)
        │                                           │
        │  4. POST /v1/anonymization/jobs/:id/execute│
        ├──────────────────────────────────────────▶│
        │                                           │
        │                                           │  5. Execute K-anonymity/
        │                                           │     L-diversity/DP
        │                                           │
        │  6. Return anonymized result             │
        │◀────────────────────────────────────────┤
        │                                           │
        │  7. Update lifecycle_stage              │
        │     ANONYMIZED                          │
        │                                           │
        │  8. Create audit trail                  │
        │                                           │

Anonymization Policies#

Define organization-wide anonymization requirements.

Endpoint: POST /v1/anonymization/policies

json
{
  "policy_name": "User Memory Anonymization Policy",
  "description": "Anonymize user memories after 1 year of inactivity",
  "data_types": ["personalization_vectors", "agent_memory", "chat_history"],
  "anonymization_requirements": {
    "method": "k-anonymity",
    "k": 5,
    "quasi_identifiers": ["user_demographic", "usage_pattern"]
  },
  "retention_after_anonymization": "3 years",
  "compliance_frameworks": ["GDPR", "CCPA"],
  "enabled": true
}

Privacy Budget Best Practices#

1. Budget Allocation Strategy#

  • Conservative: Start with low budgets (50-100 epsilon/month)
  • Monitor consumption: Track usage patterns via GET /v1/privacy-budgets/:user_id
  • Alert on depletion: Notify users at 80% consumption

2. Query Optimization#

  • Batch queries: Combine multiple queries to reduce epsilon cost
  • Caching: Cache noisy results to avoid re-querying
  • Higher epsilon for less sensitive data: Adjust epsilon based on data sensitivity

3. Budget Reset Policies#

  • Auto-reset: Enable for continuous access (suitable for analytics)
  • Manual reset: Require approval for sensitive research applications

API Reference#

Anonymization Jobs#

Endpoint Method Description
/v1/anonymization/jobs POST Create anonymization job
/v1/anonymization/jobs/:id GET Get job status
/v1/anonymization/jobs/:id/execute POST Execute job

Privacy Budgets#

Endpoint Method Description
/v1/privacy-budgets POST Create privacy budget
/v1/privacy-budgets/:user_id GET Get current budget
/v1/privacy-budgets/:user_id/consume POST Consume budget

Transformation Pipelines#

Endpoint Method Description
/v1/transformation-pipelines POST Create pipeline
/v1/transformation-pipelines/:id/execute POST Execute pipeline

Anonymization Policies#

Endpoint Method Description
/v1/anonymization/policies POST Create policy
/v1/anonymization/policies GET List policies

Testing#

The anonymization service has comprehensive test coverage (87/87 tests passing, 86% coverage):

  • Unit tests: Individual algorithm correctness
  • Integration tests: End-to-end anonymization workflows
  • Budget tests: Privacy budget consumption and enforcement
  • Pipeline tests: Multi-step transformation validation

Test Locations:

  • /services/anonymization/__tests__/
  • Test coverage: cd services/anonymization && npm test

Monitoring & Metrics#

Key metrics to monitor:

  1. Budget consumption rate: Track epsilon consumption per user
  2. Anonymization job success rate: Monitor failed jobs
  3. Privacy loss metrics: Track information loss from k-anonymity
  4. Pipeline execution time: Optimize transformation performance

Compliance#

This implementation supports:

  • GDPR Article 25: Privacy by Design and Default
  • CCPA Section 1798.100: Right to anonymized data
  • NIST Privacy Framework: Data processing principles
  • ISO/IEC 29100: Privacy framework standards

References#

  • Implementation: /services/anonymization/src/app.ts
  • Tests: /services/anonymization/__tests__/
  • Data Governance Integration: /docs/security/DATA_GOVERNANCE.md
  • Privacy Budget Algorithms: Dwork & Roth, "The Algorithmic Foundations of Differential Privacy"