Reference documentation for the Moderation Service content policies, filter configurations, and automated detection systems.
Overview#
The Moderation Service provides multi-layered content moderation through:
- Toxicity Detection: 12 content categories with configurable thresholds
- External APIs: OpenAI Moderation API (13 categories) and Perspective API (10 attributes)
- Semantic Safety: Embedding-based contextual analysis
- Pattern Detection: Jailbreak, prompt injection, PII detection
- Persona Policies: Context-aware boundary enforcement
Content Filter Categories#
Toxicity Categories#
| Category | Description | Default Action |
|---|---|---|
hate_speech |
Content targeting protected groups | block |
harassment |
Personal attacks and bullying | block |
threats |
Violent threats and intimidation | block |
self_harm |
Content promoting self-harm | block |
sexual_content |
Inappropriate sexual material | block |
violence |
Graphic violence descriptions | block |
profanity |
Offensive language | block |
spam |
Repetitive or promotional content | block |
misinformation |
False or misleading claims | block |
pii_exposure |
Personal information leaks | block |
medical |
Unqualified medical advice | block |
legal |
Unqualified legal advice | block |
Default Filter Configuration#
All filters are initialized with:
{
enabled: true,
threshold: 0.7, // CONFIDENCE_THRESHOLDS.VERY_LOW
action: 'block'
}
Moderation Actions#
| Action | Description |
|---|---|
allow |
Content passes all checks |
block |
Content is rejected |
redact |
PII or sensitive data is removed |
allow_with_warning |
Content passes with user warning |
External API Integrations#
OpenAI Moderation API#
Integrated via openai-moderation.ts for comprehensive content safety checking.
Categories (13 total):
| OpenAI Category | Internal Mapping |
|---|---|
sexual |
sexual_content |
sexual/minors |
sexual_content |
harassment |
harassment |
harassment/threatening |
harassment, threats |
hate |
hate_speech |
hate/threatening |
hate_speech, threats |
illicit |
misinformation |
illicit/violent |
violence |
self-harm |
self_harm |
self-harm/intent |
self_harm |
self-harm/instructions |
self_harm |
violence |
violence |
violence/graphic |
violence |
Configuration:
OPENAI_API_KEY=sk-... # Required to enable
Usage:
// Include 'openai' in check_types to enable
const response = await fetch('/v1/moderate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
content: 'Text to moderate',
check_types: ['toxicity', 'safety', 'openai'],
}),
});
Response includes:
{
"openai_moderation": {
"id": "modr-...",
"model": "omni-moderation-latest",
"flagged": true,
"categories": { "harassment": true },
"scores": { "harassment": 0.95 }
}
}
Perspective API#
Integrated via perspective-api.ts for toxicity scoring using Google's ML
models.
Attributes (10 total):
| Attribute | Threshold | Internal Mapping |
|---|---|---|
TOXICITY |
0.7 | toxicity |
SEVERE_TOXICITY |
0.5 | toxicity |
IDENTITY_ATTACK |
0.7 | hate_speech, harassment |
INSULT |
0.7 | harassment |
PROFANITY |
0.8 | profanity |
THREAT |
0.5 | threats |
SEXUALLY_EXPLICIT |
0.7 | sexual_content |
FLIRTATION |
0.8 | sexual_content |
ATTACK_ON_AUTHOR |
0.7 | harassment |
ATTACK_ON_COMMENTER |
0.7 | harassment |
Configuration:
PERSPECTIVE_API_KEY=... # Required to enable
Usage:
const response = await fetch('/v1/moderate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
content: 'Text to moderate',
check_types: ['toxicity', 'safety', 'perspective'],
}),
});
Response includes:
{
"perspective_analysis": {
"flagged": true,
"scores": {
"TOXICITY": 0.85,
"SEVERE_TOXICITY": 0.72
},
"flagged_attributes": ["TOXICITY", "SEVERE_TOXICITY"],
"languages": ["en"]
}
}
Semantic Safety Analyzer#
Embedding-based detection that goes beyond keyword matching to understand context and meaning.
Reference Patterns#
The system compares input against pre-defined unsafe patterns across categories:
| Category | Pattern Examples | Severity |
|---|---|---|
self_harm |
Self-injury expressions, suicidal ideation | critical |
violence |
Plans to harm others, revenge fantasies | critical |
threats |
Death threats, stalking language | critical |
hate_speech |
Dehumanization, discrimination | critical |
harassment |
Targeted abuse, cyberbullying | high |
sexual_content |
CSAM references, non-consensual content | critical |
misinformation |
Health misinformation, election denial | medium |
Similarity Thresholds#
| Severity | Threshold | Action |
|---|---|---|
critical |
0.75 | block |
high |
0.70 | block or review |
medium |
0.65 | warn |
low |
0.60 | warn |
API Endpoints#
Get Status:
GET /v1/semantic-safety/status
{
"enabled": true,
"provider": "openai-embeddings",
"initialized": true,
"pattern_counts": {
"self_harm": 4,
"violence": 4,
"threats": 3
},
"thresholds": {
"critical": 0.75,
"high": 0.7
},
"cache_size": 150
}
Update Thresholds:
POST /v1/semantic-safety/thresholds
Content-Type: application/json
{
"thresholds": {
"critical": 0.80,
"high": 0.75
}
}
Clear Cache:
POST /v1/semantic-safety/clear-cache
Jailbreak Detection#
Comprehensive detection for 10 jailbreak categories with 60+ patterns.
Categories#
| Category | Description | Weight |
|---|---|---|
role_play |
Role assignment attempts ("pretend to be...") | 0.7-0.9 |
instruction_override |
Ignore/bypass instructions | 0.9-1.0 |
context_manipulation |
Fictional/educational framing | 0.5-0.7 |
encoding_obfuscation |
Base64, ROT13, steganography hints | 0.5-0.7 |
multi_turn_manipulation |
False memory/agreement injection | 0.5-0.7 |
hypothetical_exploitation |
"Hypothetically, what if..." | 0.6 |
dan_variant |
DAN, DUDA, STAN jailbreak variants | 0.9-1.0 |
developer_mode |
Admin/root/debug mode requests | 0.5-1.0 |
system_prompt_leak |
Attempts to reveal system prompt | 0.6-0.9 |
authority_claim |
False admin/developer claims | 0.8-0.9 |
Example Patterns#
Instruction Override:
/ignore\s+(all\s+)?(previous|prior)\s+instructions/i
/disregard\s+(all\s+)?(your)?\s*rules/i
/bypass\s+safety/i
DAN Variants:
/\bdan\b.*do\s+anything\s+now/i
/jailbreak(ed)?\s+(mode|prompt)/i
/enable\s+(jailbreak|uncensored)\s+mode/i
System Prompt Leak:
/show\s+(me\s+)?(your|the)\s+(system\s+)?prompt/i
/reveal\s+(your|the)\s+(system|initial)\s+prompt/i
Risk Assessment#
| Risk Score | Level | Action |
|---|---|---|
| 0.8+ | critical |
Block |
| 0.6-0.8 | high |
Block |
| 0.4-0.6 | medium |
Review |
| 0.2-0.4 | low |
Review |
| < 0.2 | none |
Allow |
Blocked Pattern Management#
Default Patterns#
Jailbreak Attempts:
blockedPatterns.set('jailbreak_attempts', [
/ignore.*previous.*instructions/i,
/disregard.*rules/i,
/pretend.*you.*are/i,
/act.*as.*if/i,
/bypass.*safety/i,
]);
Prompt Injection:
blockedPatterns.set('prompt_injection', [
/\[system\]/i,
/<\|.*\|>/,
/###.*instruction/i,
/you.*must.*obey/i,
]);
Add Custom Patterns#
POST /v1/patterns/blocked
Content-Type: application/json
{
"category": "custom_abuse",
"pattern": "specific-pattern-to-block",
"flags": "i"
}
Persona Policies#
Context-aware moderation for AI personas with defined boundaries.
Available Personas#
Zen Master:
- Allowed: Meditation, mindfulness, working with emotions
- Forbidden: Medical advice, financial counsel, relationship counseling
- Abstain/Refer: Crisis → 988 Lifeline, Medical → healthcare professional
Stoic Guide:
- Allowed: Emotional resilience, decision-making, self-discipline
- Forbidden: Financial investment, political advocacy, psychiatric treatment
- Abstain/Refer: Financial → financial advisor, Medical → healthcare professional
Sufi Guide:
- Allowed: Spiritual purification, divine love, practices of remembrance
- Forbidden: Sectarian disputes, Islamic jurisprudence, medical treatment
- Abstain/Refer: Legal Islamic → mufti/faqih, Crisis → crisis services
Persona Validation Request#
POST /v1/moderate
Content-Type: application/json
{
"content": "How do I treat my anxiety disorder?",
"persona": "zen",
"user_id": "user123"
}
Response with policy violation:
{
"moderation_id": "mod_...",
"flagged": false,
"allowed": true,
"flags": ["persona_boundary", "requires_abstain_refer"],
"policy_violations": [
{
"persona": "zen",
"topic_type": "medical_concerns",
"action_required": "abstain_and_refer",
"severity": "high"
}
],
"queued_for_review": true
}
Filter Management API#
List All Filters#
GET /v1/filters
{
"filters": {
"hate_speech": {
"enabled": true,
"threshold": 0.7,
"action": "block"
},
"harassment": {
"enabled": true,
"threshold": 0.7,
"action": "block"
}
},
"total": 12
}
Update Filter Configuration#
PUT /v1/filters/harassment
Content-Type: application/json
{
"enabled": true,
"threshold": 0.8,
"action": "review"
}
{
"category": "harassment",
"enabled": true,
"threshold": 0.8,
"action": "review",
"updated_at": "2025-01-15T10:30:00.000Z"
}
Confidence Thresholds#
Standard threshold levels used across the system:
| Level | Value | Use Case |
|---|---|---|
HIGH |
0.95 | High-confidence detection |
MEDIUM |
0.9 | Standard detection |
LOW |
0.8 | Sensitive detection |
VERY_LOW |
0.7 | Default filter threshold |
MINIMUM |
0.6 | Edge case detection |
THRESHOLD |
0.5 | Binary decision point |
WEAK |
0.4 | Low-confidence signals |
VERY_WEAK |
0.3 | Minimal indicators |
Environment Variables#
| Variable | Description | Required |
|---|---|---|
OPENAI_API_KEY |
OpenAI API key for moderation | No |
PERSPECTIVE_API_KEY |
Google Perspective API key | No |
SERVICE_NAME |
Service identifier (default: moderation) |
No |
SERVICE_VERSION |
Service version (default: 1.0.0) |
No |
ENVIRONMENT |
Deployment environment (default: dev) |
No |
APM_ENABLED |
Enable APM telemetry (default: true) |
No |
Integration Example#
import { ModerationClient } from '@lilith/sdk';
const client = new ModerationClient({
baseUrl: 'http://moderation-service:8080',
});
// Full moderation check
const result = await client.moderate({
content: userInput,
user_id: 'user123',
context: 'chat',
check_types: ['toxicity', 'safety', 'pii', 'openai', 'perspective'],
});
if (result.flagged) {
console.log('Content blocked:', result.reasons);
// Handle violation
} else if (result.queued_for_review) {
console.log('Content requires review');
// Allow with monitoring
} else {
console.log('Content approved');
// Process content
}
Related Documentation#
- Safety Measures - Detailed safety guardrails
- Reporting System - User reporting and appeals
- Human Review Workflow - Moderator queue management