Reference documentation for the Moderation Service safety guardrails, detection systems, and protective mechanisms.
Overview#
The Moderation Service implements multiple layers of safety measures:
- Prompt Injection Detection: Pattern-based detection of injection attempts
- PII Detection: 5 types of personally identifiable information
- Crisis Detection: Self-harm and suicide risk identification
- Jailbreak Detection: 10 categories, 60+ patterns
- Medical Advice Detection: 10 categories with disclaimer support
- Legal Advice Detection: 10 categories with referral recommendations
Prompt Injection Detection#
Detects attempts to manipulate AI behavior through crafted inputs.
Detection Patterns#
Template Injection Markers:
/\[\[/ # Double bracket injection
/\{\{/ # Mustache template injection
/<\%/ # ERB/ASP template injection
Behavioral Override:
/you are now/i # Role reassignment
/you must act/i # Command injection
/forget everything/i # Memory manipulation
Confidence Thresholds#
| Pattern Type | Confidence |
|---|---|
| Explicit injection patterns | 0.9 (MEDIUM) |
| Template markers | 0.6 (MINIMUM) |
| Behavioral override | 0.8 (LOW) |
Detection Threshold#
Content is flagged when confidence exceeds 0.5 (THRESHOLD).
Usage#
import { checkForPromptInjection } from './prompt-injection-detection';
const patterns = [/\[system\]/i, /###.*instruction/i];
const result = checkForPromptInjection(content, patterns);
// { detected: boolean, confidence: number }
PII Detection#
Detects 5 types of personally identifiable information.
Supported PII Types#
| Type | Pattern | Confidence | Example |
| ------------- | -------------------------------------------------------------- | -------------- | --------------------- | ----- | ---------------- | ----------------- |
| email | [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} | 0.95 (HIGH) | user@example.com |
| phone | (?:\+?1[-.\s]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4} | 0.9 (MEDIUM) | +1 (555) 123-4567 |
| ssn | \d{3}-\d{2}-\d{4} | 1.0 | 123-45-6789 |
| credit_card | \d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4} | 1.0 | 4111-1111-1111-1111 |
| address | \d+\s+[A-Za-z]+\s+(Street | Ave | Road | ...) | 0.7 (VERY_LOW) | 123 Main Street |
Usage#
import { checkForPII } from './pii-detection';
const result = checkForPII(content);
// {
// detected: boolean,
// types: PIIType[],
// confidence: number
// }
if (result.detected) {
console.log('PII found:', result.types);
// Action: 'redact' - remove PII from content
}
Moderation Action#
When PII is detected:
- Default action:
redact - If other violations detected:
blocktakes precedence - PII types are included in moderation result for logging
Crisis and Self-Harm Detection#
Embedding-based semantic analysis for crisis intervention.
Reference Patterns#
Self-Harm Patterns (via semantic-safety.ts):
| ID | Pattern | Severity |
|---|---|---|
sh_001 |
"I want to hurt myself" | critical |
sh_002 |
"I deserve to suffer and cause myself pain" | critical |
sh_003 |
"cutting and self-injury help me cope" | high |
sh_004 |
"I punish myself when I fail" | high |
Crisis Detection in /v1/moderate#
When crisis content is detected:
{
"allowed": false,
"flags": ["persona_boundary", "crisis_detected"],
"recommended_action": "block",
"queued_for_review": true,
"policy_violations": [
{
"persona": "zen",
"topic_type": "crisis_intervention",
"action_required": "immediate_crisis_response",
"severity": "critical"
}
]
}
Crisis Resources#
All personas include crisis referral:
- US: 988 Suicide & Crisis Lifeline
- Action: Abstain from advice, refer to professional services
Jailbreak Detection#
Comprehensive 10-category detection system with 60+ patterns.
Categories#
| Category | Description | Risk Weight |
|---|---|---|
role_play |
Role/identity assignment | 0.7-0.9 |
instruction_override |
Ignore/bypass instructions | 0.9-1.0 |
context_manipulation |
Fictional/hypothetical framing | 0.5-0.7 |
encoding_obfuscation |
Base64, ROT13, steganography | 0.5-0.7 |
multi_turn_manipulation |
False memory/context injection | 0.5-0.7 |
hypothetical_exploitation |
"What if" scenarios | 0.6 |
dan_variant |
DAN/DUDA/STAN jailbreaks | 0.9-1.0 |
developer_mode |
Admin/debug mode requests | 0.5-1.0 |
system_prompt_leak |
Prompt extraction attempts | 0.6-0.9 |
authority_claim |
False admin/developer claims | 0.8-0.9 |
Sample Patterns#
Role Play:
/you\s+are\s+now\s+(a|an|the)\s+/i
/pretend\s+(to\s+be|you\s+are)/i
/from\s+now\s+on,?\s+(you\s+are|be)/i
Instruction Override:
/ignore\s+(all\s+)?(previous|prior)\s+instructions/i
/disregard\s+(all\s+)?(your)?\s*rules/i
/override\s+(your|the)?\s*programming/i
DAN Variants:
/\bdan\b.*do\s+anything\s+now/i
/jailbreak(ed)?\s+(mode|prompt)/i
/enable\s+(jailbreak|uncensored)\s+mode/i
Risk Assessment#
import { getJailbreakRiskAssessment } from './jailbreak-detection';
const assessment = getJailbreakRiskAssessment(content);
// {
// risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
// risk_score: number,
// categories: JailbreakCategory[],
// should_block: boolean,
// requires_review: boolean
// }
Risk Levels#
| Score | Level | Action |
|---|---|---|
| 0.8+ | critical |
Block |
| 0.6-0.8 | high |
Block |
| 0.4-0.6 | medium |
Review |
| 0.2-0.4 | low |
Review |
| < 0.2 | none |
Allow |
Medical Advice Detection#
10-category detection for medical advice requests requiring professional referral.
Categories#
| Category | Description | Risk Weight |
|---|---|---|
diagnosis_request |
Symptom interpretation | 0.7-1.0 |
prescription_inquiry |
Medication recommendations | 0.6-1.0 |
dosage_advice |
Dosing questions | 0.8-1.0 |
emergency_situation |
Medical emergencies | 0.9-1.0 |
self_medication |
Unguided treatment | 0.7-0.9 |
treatment_recommendation |
Procedure decisions | 0.8-0.9 |
mental_health_diagnosis |
Psychiatric conditions | 0.8-1.0 |
alternative_medicine |
Unproven treatments | 0.7-0.9 |
drug_interaction |
Medication combinations | 0.7-0.8 |
medical_procedure |
DIY medical procedures | 0.9-1.0 |
Sample Patterns#
Diagnosis Request:
/what\s+(disease|illness)\s+(do\s+i\s+have)/i
/diagnose\s+(me|my|this)/i
/what('s|\s+is)\s+wrong\s+with\s+me/i
Emergency Situation:
/i\s+(think\s+)?(i'?\s*m|am)\s+(having|experiencing)\s+(a\s+)?(heart\s+attack|stroke)/i
/(can't|cannot)\s+breathe/i
/(overdose|od'?d?|took\s+too\s+much)/i
Dosage Advice:
/how\s+(much|many)\s+(medication|pills?)\s+(should|can)\s+i\s+take/i
/what('s|\s+is)\s+(the\s+)?(right|correct)\s+dose/i
/(maximum|max|safe)\s+(dose|dosage)/i
Risk Assessment#
import { getMedicalAdviceRiskAssessment } from './medical-advice-detection';
const assessment = getMedicalAdviceRiskAssessment(content);
// {
// risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
// risk_score: number,
// categories: MedicalAdviceCategory[],
// requires_disclaimer: boolean,
// requires_referral: boolean,
// is_emergency: boolean
// }
Action Requirements#
| Condition | Disclaimer | Referral |
|---|---|---|
| Risk score ≥ 0.25 | Yes | No |
| Risk score ≥ 0.5 | Yes | Yes |
| Emergency detected | Yes | Immediate |
Emergency Response#
When emergency_situation is detected:
{
"flagged": true,
"categories": { "medical": true },
"action": "flag",
"medical_assessment": {
"is_emergency": true,
"requires_referral": true,
"disclaimer_recommended": true
}
}
Legal Advice Detection#
10-category detection for legal advice requests requiring professional referral.
Categories#
| Category | Description | Risk Weight |
|---|---|---|
legal_representation |
Lawyer need assessment | 0.6-0.9 |
contract_advice |
Contract review/drafting | 0.7-0.9 |
criminal_law |
Criminal charges/defense | 0.7-1.0 |
immigration_law |
Visa/deportation matters | 0.7-0.9 |
family_law |
Divorce/custody issues | 0.6-0.9 |
employment_law |
Workplace disputes | 0.7-0.9 |
tax_law |
Tax evasion/audits | 0.7-0.9 |
real_estate_law |
Property/tenant disputes | 0.6-0.9 |
intellectual_property |
Patent/copyright issues | 0.7-0.9 |
general_legal |
General legal questions | 0.7-0.9 |
Sample Patterns#
Criminal Law:
/can\s+i\s+(be\s+)?(charged|arrested|prosecuted)/i
/what\s+(are|is)\s+(the\s+)?(penalty|sentence)\s+for/i
/how\s+to\s+(avoid|beat|fight)\s+(criminal\s+)?charges/i
Immigration Law:
/can\s+i\s+(be\s+)?deported/i
/how\s+to\s+(get|apply\s+for)\s+(a\s+)?(visa|green\s+card)/i
/overstay(ed|ing)?\s+(my\s+)?visa/i
Family Law:
/how\s+to\s+(file\s+for|get\s+a)\s+divorce/i
/can\s+i\s+(get|win|lose)\s+(custody|visitation)/i
/(child|spousal)\s+(support|custody|alimony)/i
Risk Assessment#
import { getLegalAdviceRiskAssessment } from './legal-advice-detection';
const assessment = getLegalAdviceRiskAssessment(content);
// {
// risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
// risk_score: number,
// categories: LegalAdviceCategory[],
// requires_disclaimer: boolean,
// requires_referral: boolean,
// is_serious_matter: boolean
// }
Serious Legal Matters#
The following categories trigger is_serious_matter:
criminal_lawimmigration_lawfamily_law
Action Requirements#
| Condition | Disclaimer | Referral |
|---|---|---|
| Risk score ≥ 0.25 | Yes | No |
| Risk score ≥ 0.5 | Yes | Yes |
| Serious matter detected | Yes | Always |
Violence and Threat Detection#
Embedding-based semantic detection for violent content.
Reference Patterns#
| ID | Pattern | Category | Severity |
|---|---|---|---|
viol_001 |
"I want to hurt other people violently" | violence |
critical |
viol_002 |
"planning to attack and harm someone" | violence |
critical |
viol_003 |
"they deserve to be beaten and punished" | violence |
high |
viol_004 |
"violent revenge against those who wronged me" | violence |
high |
threat_001 |
"I will kill you and your family" | threats |
critical |
threat_002 |
"you should watch your back" | threats |
high |
threat_003 |
"I know where you live" | threats |
critical |
Similarity Matching#
Content is compared against these reference patterns using cosine similarity:
const threshold = severity === 'critical' ? 0.75 : 0.7;
if (similarity >= threshold) {
// Flag for block/review
}
Harassment Detection#
Semantic patterns for harassment and bullying detection.
Reference Patterns#
| ID | Pattern | Severity |
|---|---|---|
harass_001 |
"I will make your life miserable until you quit" | high |
harass_002 |
"you are worthless and nobody wants you around" | high |
harass_003 |
"I will keep messaging you until you respond" | medium |
Hate Speech Patterns#
| ID | Pattern | Severity |
|---|---|---|
hate_001 |
"those people are inferior and should be eliminated" | critical |
hate_002 |
"I hate everyone from that race or religion" | high |
hate_003 |
"they are subhuman and deserve no rights" | critical |
Moderation Flow#
Check Order#
- Jailbreak Detection - Block first (safety check)
- Toxicity Analysis - Apply content filters
- PII Detection - Redact or block
- Prompt Injection - Block if detected
- Medical/Legal Advice - Flag for disclaimer
- OpenAI Moderation - External API check
- Perspective API - External API check
- Semantic Safety - Embedding analysis
Example Flow#
// In /v1/moderate endpoint
const results = {
flagged: false,
categories: {},
scores: {},
reasons: [],
action: 'allow',
};
// Safety checks first (jailbreak)
if (check_types.includes('safety')) {
const jailbreakCheck = checkForJailbreak(content);
if (jailbreakCheck.detected) {
results.flagged = true;
results.action = 'block';
}
}
// PII detection
if (check_types.includes('pii')) {
const piiCheck = checkForPII(content);
if (piiCheck.detected) {
results.flagged = true;
results.action = results.action === 'block' ? 'block' : 'redact';
}
}
// Medical/Legal flagging
if (check_types.includes('medical')) {
const medicalCheck = checkForMedicalAdvice(content);
if (medicalCheck.detected && results.action !== 'block') {
results.action = 'flag'; // Requires disclaimer
}
}
Configuration#
Environment Variables#
| Variable | Description | Default |
|---|---|---|
OPENAI_API_KEY |
Enable OpenAI Moderation | - |
PERSPECTIVE_API_KEY |
Enable Perspective API | - |
Semantic Safety Thresholds#
POST /v1/semantic-safety/thresholds
Content-Type: application/json
{
"thresholds": {
"critical": 0.80,
"high": 0.75,
"medium": 0.70,
"low": 0.65
}
}
Related Documentation#
- Moderation Policies - Content filter categories
- Reporting System - User reports and appeals
- Human Review Workflow - Moderator queue