Eve library

Moderation Policies

Guide

Revision 1

On this page

Reference documentation for the Moderation Service content policies, filter configurations, and automated detection systems.

Overview#

The Moderation Service provides multi-layered content moderation through:

  • Toxicity Detection: 12 content categories with configurable thresholds
  • External APIs: OpenAI Moderation API (13 categories) and Perspective API (10 attributes)
  • Semantic Safety: Embedding-based contextual analysis
  • Pattern Detection: Jailbreak, prompt injection, PII detection
  • Persona Policies: Context-aware boundary enforcement

Content Filter Categories#

Toxicity Categories#

Category Description Default Action
hate_speech Content targeting protected groups block
harassment Personal attacks and bullying block
threats Violent threats and intimidation block
self_harm Content promoting self-harm block
sexual_content Inappropriate sexual material block
violence Graphic violence descriptions block
profanity Offensive language block
spam Repetitive or promotional content block
misinformation False or misleading claims block
pii_exposure Personal information leaks block
medical Unqualified medical advice block
legal Unqualified legal advice block

Default Filter Configuration#

All filters are initialized with:

typescript
{
  enabled: true,
  threshold: 0.7,  // CONFIDENCE_THRESHOLDS.VERY_LOW
  action: 'block'
}

Moderation Actions#

Action Description
allow Content passes all checks
block Content is rejected
redact PII or sensitive data is removed
allow_with_warning Content passes with user warning

External API Integrations#

OpenAI Moderation API#

Integrated via openai-moderation.ts for comprehensive content safety checking.

Categories (13 total):

OpenAI Category Internal Mapping
sexual sexual_content
sexual/minors sexual_content
harassment harassment
harassment/threatening harassment, threats
hate hate_speech
hate/threatening hate_speech, threats
illicit misinformation
illicit/violent violence
self-harm self_harm
self-harm/intent self_harm
self-harm/instructions self_harm
violence violence
violence/graphic violence

Configuration:

bash
OPENAI_API_KEY=sk-...  # Required to enable

Usage:

typescript
// Include 'openai' in check_types to enable
const response = await fetch('/v1/moderate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    content: 'Text to moderate',
    check_types: ['toxicity', 'safety', 'openai'],
  }),
});

Response includes:

json
{
  "openai_moderation": {
    "id": "modr-...",
    "model": "omni-moderation-latest",
    "flagged": true,
    "categories": { "harassment": true },
    "scores": { "harassment": 0.95 }
  }
}

Perspective API#

Integrated via perspective-api.ts for toxicity scoring using Google's ML models.

Attributes (10 total):

Attribute Threshold Internal Mapping
TOXICITY 0.7 toxicity
SEVERE_TOXICITY 0.5 toxicity
IDENTITY_ATTACK 0.7 hate_speech, harassment
INSULT 0.7 harassment
PROFANITY 0.8 profanity
THREAT 0.5 threats
SEXUALLY_EXPLICIT 0.7 sexual_content
FLIRTATION 0.8 sexual_content
ATTACK_ON_AUTHOR 0.7 harassment
ATTACK_ON_COMMENTER 0.7 harassment

Configuration:

bash
PERSPECTIVE_API_KEY=...  # Required to enable

Usage:

typescript
const response = await fetch('/v1/moderate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    content: 'Text to moderate',
    check_types: ['toxicity', 'safety', 'perspective'],
  }),
});

Response includes:

json
{
  "perspective_analysis": {
    "flagged": true,
    "scores": {
      "TOXICITY": 0.85,
      "SEVERE_TOXICITY": 0.72
    },
    "flagged_attributes": ["TOXICITY", "SEVERE_TOXICITY"],
    "languages": ["en"]
  }
}

Semantic Safety Analyzer#

Embedding-based detection that goes beyond keyword matching to understand context and meaning.

Reference Patterns#

The system compares input against pre-defined unsafe patterns across categories:

Category Pattern Examples Severity
self_harm Self-injury expressions, suicidal ideation critical
violence Plans to harm others, revenge fantasies critical
threats Death threats, stalking language critical
hate_speech Dehumanization, discrimination critical
harassment Targeted abuse, cyberbullying high
sexual_content CSAM references, non-consensual content critical
misinformation Health misinformation, election denial medium

Similarity Thresholds#

Severity Threshold Action
critical 0.75 block
high 0.70 block or review
medium 0.65 warn
low 0.60 warn

API Endpoints#

Get Status:

http
GET /v1/semantic-safety/status
json
{
  "enabled": true,
  "provider": "openai-embeddings",
  "initialized": true,
  "pattern_counts": {
    "self_harm": 4,
    "violence": 4,
    "threats": 3
  },
  "thresholds": {
    "critical": 0.75,
    "high": 0.7
  },
  "cache_size": 150
}

Update Thresholds:

http
POST /v1/semantic-safety/thresholds
Content-Type: application/json

{
  "thresholds": {
    "critical": 0.80,
    "high": 0.75
  }
}

Clear Cache:

http
POST /v1/semantic-safety/clear-cache

Jailbreak Detection#

Comprehensive detection for 10 jailbreak categories with 60+ patterns.

Categories#

Category Description Weight
role_play Role assignment attempts ("pretend to be...") 0.7-0.9
instruction_override Ignore/bypass instructions 0.9-1.0
context_manipulation Fictional/educational framing 0.5-0.7
encoding_obfuscation Base64, ROT13, steganography hints 0.5-0.7
multi_turn_manipulation False memory/agreement injection 0.5-0.7
hypothetical_exploitation "Hypothetically, what if..." 0.6
dan_variant DAN, DUDA, STAN jailbreak variants 0.9-1.0
developer_mode Admin/root/debug mode requests 0.5-1.0
system_prompt_leak Attempts to reveal system prompt 0.6-0.9
authority_claim False admin/developer claims 0.8-0.9

Example Patterns#

Instruction Override:

regex
/ignore\s+(all\s+)?(previous|prior)\s+instructions/i
/disregard\s+(all\s+)?(your)?\s*rules/i
/bypass\s+safety/i

DAN Variants:

regex
/\bdan\b.*do\s+anything\s+now/i
/jailbreak(ed)?\s+(mode|prompt)/i
/enable\s+(jailbreak|uncensored)\s+mode/i

System Prompt Leak:

regex
/show\s+(me\s+)?(your|the)\s+(system\s+)?prompt/i
/reveal\s+(your|the)\s+(system|initial)\s+prompt/i

Risk Assessment#

Risk Score Level Action
0.8+ critical Block
0.6-0.8 high Block
0.4-0.6 medium Review
0.2-0.4 low Review
< 0.2 none Allow

Blocked Pattern Management#

Default Patterns#

Jailbreak Attempts:

javascript
blockedPatterns.set('jailbreak_attempts', [
  /ignore.*previous.*instructions/i,
  /disregard.*rules/i,
  /pretend.*you.*are/i,
  /act.*as.*if/i,
  /bypass.*safety/i,
]);

Prompt Injection:

javascript
blockedPatterns.set('prompt_injection', [
  /\[system\]/i,
  /<\|.*\|>/,
  /###.*instruction/i,
  /you.*must.*obey/i,
]);

Add Custom Patterns#

http
POST /v1/patterns/blocked
Content-Type: application/json

{
  "category": "custom_abuse",
  "pattern": "specific-pattern-to-block",
  "flags": "i"
}

Persona Policies#

Context-aware moderation for AI personas with defined boundaries.

Available Personas#

Zen Master:

  • Allowed: Meditation, mindfulness, working with emotions
  • Forbidden: Medical advice, financial counsel, relationship counseling
  • Abstain/Refer: Crisis → 988 Lifeline, Medical → healthcare professional

Stoic Guide:

  • Allowed: Emotional resilience, decision-making, self-discipline
  • Forbidden: Financial investment, political advocacy, psychiatric treatment
  • Abstain/Refer: Financial → financial advisor, Medical → healthcare professional

Sufi Guide:

  • Allowed: Spiritual purification, divine love, practices of remembrance
  • Forbidden: Sectarian disputes, Islamic jurisprudence, medical treatment
  • Abstain/Refer: Legal Islamic → mufti/faqih, Crisis → crisis services

Persona Validation Request#

http
POST /v1/moderate
Content-Type: application/json

{
  "content": "How do I treat my anxiety disorder?",
  "persona": "zen",
  "user_id": "user123"
}

Response with policy violation:

json
{
  "moderation_id": "mod_...",
  "flagged": false,
  "allowed": true,
  "flags": ["persona_boundary", "requires_abstain_refer"],
  "policy_violations": [
    {
      "persona": "zen",
      "topic_type": "medical_concerns",
      "action_required": "abstain_and_refer",
      "severity": "high"
    }
  ],
  "queued_for_review": true
}

Filter Management API#

List All Filters#

http
GET /v1/filters
json
{
  "filters": {
    "hate_speech": {
      "enabled": true,
      "threshold": 0.7,
      "action": "block"
    },
    "harassment": {
      "enabled": true,
      "threshold": 0.7,
      "action": "block"
    }
  },
  "total": 12
}

Update Filter Configuration#

http
PUT /v1/filters/harassment
Content-Type: application/json

{
  "enabled": true,
  "threshold": 0.8,
  "action": "review"
}
json
{
  "category": "harassment",
  "enabled": true,
  "threshold": 0.8,
  "action": "review",
  "updated_at": "2025-01-15T10:30:00.000Z"
}

Confidence Thresholds#

Standard threshold levels used across the system:

Level Value Use Case
HIGH 0.95 High-confidence detection
MEDIUM 0.9 Standard detection
LOW 0.8 Sensitive detection
VERY_LOW 0.7 Default filter threshold
MINIMUM 0.6 Edge case detection
THRESHOLD 0.5 Binary decision point
WEAK 0.4 Low-confidence signals
VERY_WEAK 0.3 Minimal indicators

Environment Variables#

Variable Description Required
OPENAI_API_KEY OpenAI API key for moderation No
PERSPECTIVE_API_KEY Google Perspective API key No
SERVICE_NAME Service identifier (default: moderation) No
SERVICE_VERSION Service version (default: 1.0.0) No
ENVIRONMENT Deployment environment (default: dev) No
APM_ENABLED Enable APM telemetry (default: true) No

Integration Example#

typescript
import { ModerationClient } from '@lilith/sdk';

const client = new ModerationClient({
  baseUrl: 'http://moderation-service:8080',
});

// Full moderation check
const result = await client.moderate({
  content: userInput,
  user_id: 'user123',
  context: 'chat',
  check_types: ['toxicity', 'safety', 'pii', 'openai', 'perspective'],
});

if (result.flagged) {
  console.log('Content blocked:', result.reasons);
  // Handle violation
} else if (result.queued_for_review) {
  console.log('Content requires review');
  // Allow with monitoring
} else {
  console.log('Content approved');
  // Process content
}