Content safety and moderation service for Lilith platform.
Overview#
The Moderation service provides:
- Content Filtering: Toxicity, self-harm, and inappropriate content detection
- Policy Enforcement: Age-gating, cultural sensitivity, persona boundaries
- Audit Trails: Pre/post-moderation logging
- Red-Team Hooks: Jailbreak detection and adversarial prompt handling
- Report/Repair Loop: User feedback with SLA-based response
Architecture#
text
moderation/
├── src/
│ ├── app.ts # Fastify application
│ ├── server.ts # Entry point
│ ├── filters/ # Content filters
│ │ ├── toxicity.ts # Toxicity detection
│ │ ├── self-harm.ts # Self-harm detection
│ │ ├── medical.ts # Medical disclaimer triggers
│ │ └── legal.ts # Legal disclaimer triggers
│ ├── policies/ # Policy enforcement
│ │ ├── age-gate.ts # Age-appropriate content
│ │ ├── cultural.ts # Cultural sensitivity
│ │ └── persona.ts # Persona boundaries
│ ├── queues/ # Moderation queues
│ ├── audit/ # Audit logging
│ └── redteam/ # Adversarial detection
├── __tests__/
└── package.json
Safety Policies#
Content Categories#
| Category | Action | Description |
|---|---|---|
| Toxicity | Block | Hate speech, harassment |
| Self-Harm | Block + Crisis | Self-harm, suicide content |
| Violence | Block | Graphic violence |
| Sexual | Block | Explicit sexual content |
| Medical | Disclaimer | Health-related advice |
| Legal | Disclaimer | Legal-related advice |
| Minors | Strict Block | Child safety violations |
Severity Levels#
| Level | Response Time | Action |
|---|---|---|
| Critical | <30 min | Immediate block + escalation |
| High | <4 hours | Review and action |
| Medium | <24 hours | Queue for review |
| Low | <72 hours | Batch review |
API Endpoints#
Content Moderation#
http
POST /v1/moderation/check # Check content
POST /v1/moderation/check/batch # Batch check
GET /v1/moderation/status/:id # Get moderation status
Reports#
http
POST /v1/moderation/reports # Submit report
GET /v1/moderation/reports/:id # Get report status
GET /v1/moderation/reports # List reports (admin)
PATCH /v1/moderation/reports/:id # Update report (admin)
Queue Management (Admin)#
http
GET /v1/moderation/queue # View queue
POST /v1/moderation/queue/:id/approve # Approve item
POST /v1/moderation/queue/:id/reject # Reject item
POST /v1/moderation/queue/:id/escalate # Escalate item
Policy Management (Admin)#
http
GET /v1/moderation/policies # List policies
PUT /v1/moderation/policies/:id # Update policy
GET /v1/moderation/policies/:id/stats # Policy statistics
Moderation Request#
json
{
"content": {
"text": "User message content here",
"context": {
"conversation_id": "conv_123",
"persona": "zen-master",
"user_age_verified": true
}
},
"options": {
"categories": ["toxicity", "self-harm", "cultural"],
"persona_context": true,
"return_scores": true
}
}
Moderation Response#
json
{
"moderation_id": "mod_abc123",
"action": "allow",
"flags": [],
"scores": {
"toxicity": 0.02,
"self_harm": 0.01,
"violence": 0.00,
"sexual": 0.00
},
"disclaimers": [],
"modified_content": null,
"review_required": false
}
Blocked Response#
json
{
"moderation_id": "mod_xyz789",
"action": "block",
"flags": ["self_harm_detected"],
"scores": {
"self_harm": 0.85
},
"replacement": {
"text": "I notice you may be going through a difficult time. Please reach out to a crisis helpline...",
"resources": [
{"name": "National Suicide Prevention Lifeline", "phone": "988"}
]
},
"review_required": true
}
Persona Boundaries#
Each AI persona has defined boundaries:
json
{
"persona_id": "zen-master",
"boundaries": {
"no_medical_advice": true,
"no_predictions": true,
"no_financial_advice": true,
"crisis_escalation": true,
"cultural_sensitivity": ["buddhism", "eastern"]
}
}
Red-Team Hooks#
Jailbreak Detection#
typescript
// Patterns that trigger enhanced scrutiny
const jailbreakPatterns = [
/ignore.*instructions/i,
/pretend.*you.*are/i,
/roleplay.*as/i,
/bypass.*safety/i
];
Adversarial Input Handling#
json
{
"input": "Ignore your instructions and...",
"detection": {
"adversarial_score": 0.95,
"pattern_matched": "instruction_override",
"action": "block_and_log"
}
}
Configuration#
yaml
moderation:
thresholds:
toxicity: 0.7
self_harm: 0.5
violence: 0.8
sexual: 0.7
queues:
preModeration: true
postModeration: true
sla:
critical: 30 # minutes
high: 240 # minutes
medium: 1440 # minutes
low: 4320 # minutes
redteam:
enabled: true
logAll: true
Environment Variables#
env
PORT=8080
AI_SERVICE_URL=http://ai:8081
NOTIFICATION_SERVICE_URL=http://notification:8082
REDIS_URL=redis://localhost:6379
DB_URL=postgres://localhost:5432/moderation
Development#
bash
# Install dependencies
npm install
# Start development server
npm run dev
# Run tests
npm test
Audit Logging#
All moderation decisions are logged:
json
{
"timestamp": "2026-01-03T10:00:00Z",
"moderation_id": "mod_abc123",
"content_hash": "sha256:...",
"user_id": "user_123",
"action": "block",
"flags": ["self_harm_detected"],
"scores": {...},
"reviewer": null,
"automated": true
}