Document Version: 1.0 Last Updated: 2025-10-16 Owners: Privacy Engineering & Data Governance Team
This document describes the anonymization and pseudonymization pipelines, privacy budget tracking, and integration with data governance lifecycle management.
Overview#
The anonymization service provides comprehensive privacy-preserving data transformation using industry-standard algorithms:
- K-Anonymity: Generalizes quasi-identifiers to ensure at least k indistinguishable records
- L-Diversity: Ensures at least l distinct sensitive attribute values in each equivalence class
- Differential Privacy: Adds calibrated noise to protect individual privacy in aggregated queries
- Pseudonymization: Reversible/irreversible identifier transformation
Architecture#
┌──────────────────┐ ┌────────────────────────┐
│ Data Governance │────────▶│ Anonymization Service │
│ Service │ │ │
│ │ │ - K-Anonymity │
│ Lifecycle: │ │ - L-Diversity │
│ ANONYMIZE action│ │ - Differential Privacy│
│ │ │ - Pseudonymization │
└──────────────────┘ │ - Privacy Budgets │
└────────────────────────┘
│
▼
┌────────────────────────┐
│ Anonymized Datasets │
│ Privacy Budget Ledger│
└────────────────────────┘
Anonymization Methods#
1. K-Anonymity#
Endpoint: POST /v1/anonymization/k-anonymity
Ensures that each record is indistinguishable from at least k-1 other records based on quasi-identifiers.
Example Request:
{
"dataset": [
{ "age": 25, "city": "New York", "salary": 75000 },
{ "age": 26, "city": "New York", "salary": 78000 },
{ "age": 27, "city": "New York", "salary": 72000 }
],
"quasi_identifiers": ["age", "city"],
"sensitive_attributes": ["salary"],
"k": 3
}
Response:
{
"group_id": "k-anon-1",
"anonymized_dataset": [
{ "age": "20-29", "city": "New***", "salary": 75000 },
{ "age": "20-29", "city": "New***", "salary": 78000 },
{ "age": "20-29", "city": "New***", "salary": 72000 }
],
"k_value": 3,
"anonymization_method": "k-anonymity",
"groups_created": 1,
"privacy_loss": 0.67,
"applied_at": "2025-10-16T10:00:00Z"
}
2. L-Diversity#
Endpoint: POST /v1/anonymization/l-diversity
Ensures each equivalence class has at least l well-represented sensitive attribute values.
Example Request:
{
"dataset": [
{ "zip": "10001", "age": 25, "disease": "flu" },
{ "zip": "10001", "age": 26, "disease": "covid" },
{ "zip": "10001", "age": 27, "disease": "cold" }
],
"quasi_identifiers": ["zip", "age"],
"sensitive_attribute": "disease",
"l": 3
}
3. Differential Privacy#
Endpoint: POST /v1/anonymization/differential-privacy
Adds calibrated Laplace or Gaussian noise to protect individual privacy in aggregated statistics.
Example Request:
{
"query": "SELECT COUNT(*) FROM users WHERE age > 25",
"result": 1523,
"sensitivity": 1,
"epsilon": 1.0,
"noise_type": "laplace"
}
Response:
{
"query_id": "dp-query-1",
"query": "SELECT COUNT(*) FROM users WHERE age > 25",
"original_result": 1523,
"noisy_result": 1528,
"epsilon": 1.0,
"sensitivity": 1,
"noise_added": 5,
"privacy_guarantee": "(1.0, 0)-differential privacy",
"noise_type": "laplace",
"applied_at": "2025-10-16T10:00:00Z"
}
4. Pseudonymization#
Endpoint: POST /v1/anonymization/pseudonymize
Creates reversible pseudonyms or irreversible anonymous identifiers.
Reversible Pseudonymization:
{
"identifiers": ["user-12345", "user-67890"],
"context": "analytics",
"reversible": true
}
Response:
{
"mappings": [
{
"original_id": "user-12345",
"pseudonym": "pseudo_analytics_a3f8e9b..._1a2b3c",
"context": "analytics",
"reversible": true,
"algorithm": "aes-256-cbc",
"created_at": "2025-10-16T10:00:00Z"
}
],
"mapping_type": "pseudonymization"
}
Irreversible Anonymization:
{
"identifiers": ["user-12345"],
"context": "research",
"reversible": false
}
Privacy Budget Management#
Privacy budgets track the cumulative privacy loss from repeated queries, essential for differential privacy guarantees.
Creating a Privacy Budget#
Endpoint: POST /v1/privacy-budgets
{
"user_id": "user-12345",
"total_budget": 100,
"period_days": 30,
"auto_reset": true,
"budget_type": "differential_privacy"
}
Response:
{
"id": "budget-1",
"user_id": "user-12345",
"total_budget": 100,
"consumed_budget": 0,
"remaining_budget": 100,
"period_days": 30,
"auto_reset": true,
"period_start": "2025-10-16T00:00:00Z",
"period_end": "2025-11-15T00:00:00Z",
"consumption_log": []
}
Consuming Budget#
Endpoint: POST /v1/privacy-budgets/:user_id/consume
{
"operation": "differential_privacy_query",
"cost": 5,
"metadata": {
"query_id": "dp-query-1",
"epsilon": 1.0
}
}
Budget Exceeded (429):
{
"code": "anonymization.budget_exceeded",
"message": "Insufficient privacy budget",
"remaining_budget": 2,
"required_budget": 5,
"period_end": "2025-11-15T00:00:00Z"
}
Privacy Budget for Memories#
User memories and personalization vectors consume privacy budget when:
- Memory retrieval: Queries accessing user memories cost budget proportional to epsilon
- Personalization inference: RAG queries over user data consume budget
- Analytics aggregation: Aggregated statistics about user behavior
Budget Allocation:
- Standard tier: 100 epsilon per 30 days
- Premium tier: 200 epsilon per 30 days
- Research tier: 50 epsilon per 30 days (stricter privacy)
Budget Costs:
- Simple memory retrieval: 1 epsilon
- Complex RAG query: 2-5 epsilon
- Aggregated analytics: 5-10 epsilon
Transformation Pipelines#
Pipelines orchestrate multiple anonymization steps in sequence.
Creating a Pipeline#
Endpoint: POST /v1/transformation-pipelines
{
"pipeline_name": "User Data Anonymization",
"source_service": "conversation",
"transformation_steps": [
{
"type": "k-anonymity",
"parameters": {
"k": 5,
"quasi_identifiers": ["age", "location"],
"sensitive_attributes": ["message_content"]
}
},
{
"type": "pseudonymization",
"parameters": {
"context": "analytics",
"reversible": false
}
}
],
"schedule": "0 2 * * *",
"enabled": true,
"target_anonymization_level": "high"
}
Executing a Pipeline#
Endpoint: POST /v1/transformation-pipelines/:id/execute
{
"dry_run": false
}
Integration with Data Governance#
The anonymization service integrates with data governance lifecycle management:
Lifecycle Action: ANONYMIZE#
When data-governance executes a lifecycle rule with the ANONYMIZE action:
- Data-governance service identifies records requiring anonymization
- Creates an anonymization job via
POST /v1/anonymization/jobs - Anonymization service executes the job using configured method (k-anonymity, l-diversity, etc.)
- Anonymized data is stored in anonymized_datasets
- Original data lifecycle_stage updated to
ANONYMIZINGthenANONYMIZED - Audit trail created in data-governance service
Example Lifecycle Rule:
{
"name": "Anonymize old PII after 2 years",
"trigger_conditions": {
"age_days": 730,
"access_count_threshold": 0
},
"actions": ["ANONYMIZE", "DELETE"],
"data_classifications": ["PII_HIGH", "PII_MEDIUM"]
}
Integration Flow#
Data Governance Service Anonymization Service
│ │
│ 1. Lifecycle rule triggers │
│ ANONYMIZE action │
│ │
│ 2. POST /v1/anonymization/jobs │
├──────────────────────────────────────────▶│
│ │
│ │ 3. Create job
│ │ (PENDING)
│ │
│ 4. POST /v1/anonymization/jobs/:id/execute│
├──────────────────────────────────────────▶│
│ │
│ │ 5. Execute K-anonymity/
│ │ L-diversity/DP
│ │
│ 6. Return anonymized result │
│◀────────────────────────────────────────┤
│ │
│ 7. Update lifecycle_stage │
│ ANONYMIZED │
│ │
│ 8. Create audit trail │
│ │
Anonymization Policies#
Define organization-wide anonymization requirements.
Endpoint: POST /v1/anonymization/policies
{
"policy_name": "User Memory Anonymization Policy",
"description": "Anonymize user memories after 1 year of inactivity",
"data_types": ["personalization_vectors", "agent_memory", "chat_history"],
"anonymization_requirements": {
"method": "k-anonymity",
"k": 5,
"quasi_identifiers": ["user_demographic", "usage_pattern"]
},
"retention_after_anonymization": "3 years",
"compliance_frameworks": ["GDPR", "CCPA"],
"enabled": true
}
Privacy Budget Best Practices#
1. Budget Allocation Strategy#
- Conservative: Start with low budgets (50-100 epsilon/month)
- Monitor consumption: Track usage patterns via
GET /v1/privacy-budgets/:user_id - Alert on depletion: Notify users at 80% consumption
2. Query Optimization#
- Batch queries: Combine multiple queries to reduce epsilon cost
- Caching: Cache noisy results to avoid re-querying
- Higher epsilon for less sensitive data: Adjust epsilon based on data sensitivity
3. Budget Reset Policies#
- Auto-reset: Enable for continuous access (suitable for analytics)
- Manual reset: Require approval for sensitive research applications
API Reference#
Anonymization Jobs#
| Endpoint | Method | Description |
|---|---|---|
/v1/anonymization/jobs |
POST | Create anonymization job |
/v1/anonymization/jobs/:id |
GET | Get job status |
/v1/anonymization/jobs/:id/execute |
POST | Execute job |
Privacy Budgets#
| Endpoint | Method | Description |
|---|---|---|
/v1/privacy-budgets |
POST | Create privacy budget |
/v1/privacy-budgets/:user_id |
GET | Get current budget |
/v1/privacy-budgets/:user_id/consume |
POST | Consume budget |
Transformation Pipelines#
| Endpoint | Method | Description |
|---|---|---|
/v1/transformation-pipelines |
POST | Create pipeline |
/v1/transformation-pipelines/:id/execute |
POST | Execute pipeline |
Anonymization Policies#
| Endpoint | Method | Description |
|---|---|---|
/v1/anonymization/policies |
POST | Create policy |
/v1/anonymization/policies |
GET | List policies |
Testing#
The anonymization service has comprehensive test coverage (87/87 tests passing, 86% coverage):
- Unit tests: Individual algorithm correctness
- Integration tests: End-to-end anonymization workflows
- Budget tests: Privacy budget consumption and enforcement
- Pipeline tests: Multi-step transformation validation
Test Locations:
/services/anonymization/__tests__/- Test coverage:
cd services/anonymization && npm test
Monitoring & Metrics#
Key metrics to monitor:
- Budget consumption rate: Track epsilon consumption per user
- Anonymization job success rate: Monitor failed jobs
- Privacy loss metrics: Track information loss from k-anonymity
- Pipeline execution time: Optimize transformation performance
Compliance#
This implementation supports:
- GDPR Article 25: Privacy by Design and Default
- CCPA Section 1798.100: Right to anonymized data
- NIST Privacy Framework: Data processing principles
- ISO/IEC 29100: Privacy framework standards
References#
- Implementation:
/services/anonymization/src/app.ts - Tests:
/services/anonymization/__tests__/ - Data Governance Integration:
/docs/security/DATA_GOVERNANCE.md - Privacy Budget Algorithms: Dwork & Roth, "The Algorithmic Foundations of Differential Privacy"