Eve library

Moderation Service

Other

Revision 1

On this page

Content safety and moderation service for Lilith platform.

Overview#

The Moderation service provides:

  • Content Filtering: Toxicity, self-harm, and inappropriate content detection
  • Policy Enforcement: Age-gating, cultural sensitivity, persona boundaries
  • Audit Trails: Pre/post-moderation logging
  • Red-Team Hooks: Jailbreak detection and adversarial prompt handling
  • Report/Repair Loop: User feedback with SLA-based response

Architecture#

text
moderation/
├── src/
│   ├── app.ts              # Fastify application
│   ├── server.ts           # Entry point
│   ├── filters/            # Content filters
│   │   ├── toxicity.ts     # Toxicity detection
│   │   ├── self-harm.ts    # Self-harm detection
│   │   ├── medical.ts      # Medical disclaimer triggers
│   │   └── legal.ts        # Legal disclaimer triggers
│   ├── policies/           # Policy enforcement
│   │   ├── age-gate.ts     # Age-appropriate content
│   │   ├── cultural.ts     # Cultural sensitivity
│   │   └── persona.ts      # Persona boundaries
│   ├── queues/             # Moderation queues
│   ├── audit/              # Audit logging
│   └── redteam/            # Adversarial detection
├── __tests__/
└── package.json

Safety Policies#

Content Categories#

Category Action Description
Toxicity Block Hate speech, harassment
Self-Harm Block + Crisis Self-harm, suicide content
Violence Block Graphic violence
Sexual Block Explicit sexual content
Medical Disclaimer Health-related advice
Legal Disclaimer Legal-related advice
Minors Strict Block Child safety violations

Severity Levels#

Level Response Time Action
Critical <30 min Immediate block + escalation
High <4 hours Review and action
Medium <24 hours Queue for review
Low <72 hours Batch review

API Endpoints#

Content Moderation#

http
POST /v1/moderation/check             # Check content
POST /v1/moderation/check/batch       # Batch check
GET  /v1/moderation/status/:id        # Get moderation status

Reports#

http
POST /v1/moderation/reports           # Submit report
GET  /v1/moderation/reports/:id       # Get report status
GET  /v1/moderation/reports           # List reports (admin)
PATCH /v1/moderation/reports/:id      # Update report (admin)

Queue Management (Admin)#

http
GET  /v1/moderation/queue             # View queue
POST /v1/moderation/queue/:id/approve # Approve item
POST /v1/moderation/queue/:id/reject  # Reject item
POST /v1/moderation/queue/:id/escalate # Escalate item

Policy Management (Admin)#

http
GET  /v1/moderation/policies          # List policies
PUT  /v1/moderation/policies/:id      # Update policy
GET  /v1/moderation/policies/:id/stats # Policy statistics

Moderation Request#

json
{
  "content": {
    "text": "User message content here",
    "context": {
      "conversation_id": "conv_123",
      "persona": "zen-master",
      "user_age_verified": true
    }
  },
  "options": {
    "categories": ["toxicity", "self-harm", "cultural"],
    "persona_context": true,
    "return_scores": true
  }
}

Moderation Response#

json
{
  "moderation_id": "mod_abc123",
  "action": "allow",
  "flags": [],
  "scores": {
    "toxicity": 0.02,
    "self_harm": 0.01,
    "violence": 0.00,
    "sexual": 0.00
  },
  "disclaimers": [],
  "modified_content": null,
  "review_required": false
}

Blocked Response#

json
{
  "moderation_id": "mod_xyz789",
  "action": "block",
  "flags": ["self_harm_detected"],
  "scores": {
    "self_harm": 0.85
  },
  "replacement": {
    "text": "I notice you may be going through a difficult time. Please reach out to a crisis helpline...",
    "resources": [
      {"name": "National Suicide Prevention Lifeline", "phone": "988"}
    ]
  },
  "review_required": true
}

Persona Boundaries#

Each AI persona has defined boundaries:

json
{
  "persona_id": "zen-master",
  "boundaries": {
    "no_medical_advice": true,
    "no_predictions": true,
    "no_financial_advice": true,
    "crisis_escalation": true,
    "cultural_sensitivity": ["buddhism", "eastern"]
  }
}

Red-Team Hooks#

Jailbreak Detection#

typescript
// Patterns that trigger enhanced scrutiny
const jailbreakPatterns = [
  /ignore.*instructions/i,
  /pretend.*you.*are/i,
  /roleplay.*as/i,
  /bypass.*safety/i
];

Adversarial Input Handling#

json
{
  "input": "Ignore your instructions and...",
  "detection": {
    "adversarial_score": 0.95,
    "pattern_matched": "instruction_override",
    "action": "block_and_log"
  }
}

Configuration#

yaml
moderation:
  thresholds:
    toxicity: 0.7
    self_harm: 0.5
    violence: 0.8
    sexual: 0.7
  queues:
    preModeration: true
    postModeration: true
  sla:
    critical: 30    # minutes
    high: 240       # minutes
    medium: 1440    # minutes
    low: 4320       # minutes
  redteam:
    enabled: true
    logAll: true

Environment Variables#

env
PORT=8080
AI_SERVICE_URL=http://ai:8081
NOTIFICATION_SERVICE_URL=http://notification:8082
REDIS_URL=redis://localhost:6379
DB_URL=postgres://localhost:5432/moderation

Development#

bash
# Install dependencies
npm install

# Start development server
npm run dev

# Run tests
npm test

Audit Logging#

All moderation decisions are logged:

json
{
  "timestamp": "2026-01-03T10:00:00Z",
  "moderation_id": "mod_abc123",
  "content_hash": "sha256:...",
  "user_id": "user_123",
  "action": "block",
  "flags": ["self_harm_detected"],
  "scores": {...},
  "reviewer": null,
  "automated": true
}