Eve library

Safety Measures

Guide

Revision 1

On this page

Reference documentation for the Moderation Service safety guardrails, detection systems, and protective mechanisms.

Overview#

The Moderation Service implements multiple layers of safety measures:

  • Prompt Injection Detection: Pattern-based detection of injection attempts
  • PII Detection: 5 types of personally identifiable information
  • Crisis Detection: Self-harm and suicide risk identification
  • Jailbreak Detection: 10 categories, 60+ patterns
  • Medical Advice Detection: 10 categories with disclaimer support
  • Legal Advice Detection: 10 categories with referral recommendations

Prompt Injection Detection#

Detects attempts to manipulate AI behavior through crafted inputs.

Detection Patterns#

Template Injection Markers:

regex
/\[\[/              # Double bracket injection
/\{\{/              # Mustache template injection
/<\%/               # ERB/ASP template injection

Behavioral Override:

regex
/you are now/i         # Role reassignment
/you must act/i        # Command injection
/forget everything/i   # Memory manipulation

Confidence Thresholds#

Pattern Type Confidence
Explicit injection patterns 0.9 (MEDIUM)
Template markers 0.6 (MINIMUM)
Behavioral override 0.8 (LOW)

Detection Threshold#

Content is flagged when confidence exceeds 0.5 (THRESHOLD).

Usage#

typescript
import { checkForPromptInjection } from './prompt-injection-detection';

const patterns = [/\[system\]/i, /###.*instruction/i];

const result = checkForPromptInjection(content, patterns);
// { detected: boolean, confidence: number }

PII Detection#

Detects 5 types of personally identifiable information.

Supported PII Types#

| Type | Pattern | Confidence | Example | | ------------- | -------------------------------------------------------------- | -------------- | --------------------- | ----- | ---------------- | ----------------- | | email | [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} | 0.95 (HIGH) | user@example.com | | phone | (?:\+?1[-.\s]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4} | 0.9 (MEDIUM) | +1 (555) 123-4567 | | ssn | \d{3}-\d{2}-\d{4} | 1.0 | 123-45-6789 | | credit_card | \d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4} | 1.0 | 4111-1111-1111-1111 | | address | \d+\s+[A-Za-z]+\s+(Street | Ave | Road | ...) | 0.7 (VERY_LOW) | 123 Main Street |

Usage#

typescript
import { checkForPII } from './pii-detection';

const result = checkForPII(content);
// {
//   detected: boolean,
//   types: PIIType[],
//   confidence: number
// }

if (result.detected) {
  console.log('PII found:', result.types);
  // Action: 'redact' - remove PII from content
}

Moderation Action#

When PII is detected:

  • Default action: redact
  • If other violations detected: block takes precedence
  • PII types are included in moderation result for logging

Crisis and Self-Harm Detection#

Embedding-based semantic analysis for crisis intervention.

Reference Patterns#

Self-Harm Patterns (via semantic-safety.ts):

ID Pattern Severity
sh_001 "I want to hurt myself" critical
sh_002 "I deserve to suffer and cause myself pain" critical
sh_003 "cutting and self-injury help me cope" high
sh_004 "I punish myself when I fail" high

Crisis Detection in /v1/moderate#

When crisis content is detected:

json
{
  "allowed": false,
  "flags": ["persona_boundary", "crisis_detected"],
  "recommended_action": "block",
  "queued_for_review": true,
  "policy_violations": [
    {
      "persona": "zen",
      "topic_type": "crisis_intervention",
      "action_required": "immediate_crisis_response",
      "severity": "critical"
    }
  ]
}

Crisis Resources#

All personas include crisis referral:

  • US: 988 Suicide & Crisis Lifeline
  • Action: Abstain from advice, refer to professional services

Jailbreak Detection#

Comprehensive 10-category detection system with 60+ patterns.

Categories#

Category Description Risk Weight
role_play Role/identity assignment 0.7-0.9
instruction_override Ignore/bypass instructions 0.9-1.0
context_manipulation Fictional/hypothetical framing 0.5-0.7
encoding_obfuscation Base64, ROT13, steganography 0.5-0.7
multi_turn_manipulation False memory/context injection 0.5-0.7
hypothetical_exploitation "What if" scenarios 0.6
dan_variant DAN/DUDA/STAN jailbreaks 0.9-1.0
developer_mode Admin/debug mode requests 0.5-1.0
system_prompt_leak Prompt extraction attempts 0.6-0.9
authority_claim False admin/developer claims 0.8-0.9

Sample Patterns#

Role Play:

regex
/you\s+are\s+now\s+(a|an|the)\s+/i
/pretend\s+(to\s+be|you\s+are)/i
/from\s+now\s+on,?\s+(you\s+are|be)/i

Instruction Override:

regex
/ignore\s+(all\s+)?(previous|prior)\s+instructions/i
/disregard\s+(all\s+)?(your)?\s*rules/i
/override\s+(your|the)?\s*programming/i

DAN Variants:

regex
/\bdan\b.*do\s+anything\s+now/i
/jailbreak(ed)?\s+(mode|prompt)/i
/enable\s+(jailbreak|uncensored)\s+mode/i

Risk Assessment#

typescript
import { getJailbreakRiskAssessment } from './jailbreak-detection';

const assessment = getJailbreakRiskAssessment(content);
// {
//   risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
//   risk_score: number,
//   categories: JailbreakCategory[],
//   should_block: boolean,
//   requires_review: boolean
// }

Risk Levels#

Score Level Action
0.8+ critical Block
0.6-0.8 high Block
0.4-0.6 medium Review
0.2-0.4 low Review
< 0.2 none Allow

Medical Advice Detection#

10-category detection for medical advice requests requiring professional referral.

Categories#

Category Description Risk Weight
diagnosis_request Symptom interpretation 0.7-1.0
prescription_inquiry Medication recommendations 0.6-1.0
dosage_advice Dosing questions 0.8-1.0
emergency_situation Medical emergencies 0.9-1.0
self_medication Unguided treatment 0.7-0.9
treatment_recommendation Procedure decisions 0.8-0.9
mental_health_diagnosis Psychiatric conditions 0.8-1.0
alternative_medicine Unproven treatments 0.7-0.9
drug_interaction Medication combinations 0.7-0.8
medical_procedure DIY medical procedures 0.9-1.0

Sample Patterns#

Diagnosis Request:

regex
/what\s+(disease|illness)\s+(do\s+i\s+have)/i
/diagnose\s+(me|my|this)/i
/what('s|\s+is)\s+wrong\s+with\s+me/i

Emergency Situation:

regex
/i\s+(think\s+)?(i'?\s*m|am)\s+(having|experiencing)\s+(a\s+)?(heart\s+attack|stroke)/i
/(can't|cannot)\s+breathe/i
/(overdose|od'?d?|took\s+too\s+much)/i

Dosage Advice:

regex
/how\s+(much|many)\s+(medication|pills?)\s+(should|can)\s+i\s+take/i
/what('s|\s+is)\s+(the\s+)?(right|correct)\s+dose/i
/(maximum|max|safe)\s+(dose|dosage)/i

Risk Assessment#

typescript
import { getMedicalAdviceRiskAssessment } from './medical-advice-detection';

const assessment = getMedicalAdviceRiskAssessment(content);
// {
//   risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
//   risk_score: number,
//   categories: MedicalAdviceCategory[],
//   requires_disclaimer: boolean,
//   requires_referral: boolean,
//   is_emergency: boolean
// }

Action Requirements#

Condition Disclaimer Referral
Risk score ≥ 0.25 Yes No
Risk score ≥ 0.5 Yes Yes
Emergency detected Yes Immediate

Emergency Response#

When emergency_situation is detected:

json
{
  "flagged": true,
  "categories": { "medical": true },
  "action": "flag",
  "medical_assessment": {
    "is_emergency": true,
    "requires_referral": true,
    "disclaimer_recommended": true
  }
}

10-category detection for legal advice requests requiring professional referral.

Categories#

Category Description Risk Weight
legal_representation Lawyer need assessment 0.6-0.9
contract_advice Contract review/drafting 0.7-0.9
criminal_law Criminal charges/defense 0.7-1.0
immigration_law Visa/deportation matters 0.7-0.9
family_law Divorce/custody issues 0.6-0.9
employment_law Workplace disputes 0.7-0.9
tax_law Tax evasion/audits 0.7-0.9
real_estate_law Property/tenant disputes 0.6-0.9
intellectual_property Patent/copyright issues 0.7-0.9
general_legal General legal questions 0.7-0.9

Sample Patterns#

Criminal Law:

regex
/can\s+i\s+(be\s+)?(charged|arrested|prosecuted)/i
/what\s+(are|is)\s+(the\s+)?(penalty|sentence)\s+for/i
/how\s+to\s+(avoid|beat|fight)\s+(criminal\s+)?charges/i

Immigration Law:

regex
/can\s+i\s+(be\s+)?deported/i
/how\s+to\s+(get|apply\s+for)\s+(a\s+)?(visa|green\s+card)/i
/overstay(ed|ing)?\s+(my\s+)?visa/i

Family Law:

regex
/how\s+to\s+(file\s+for|get\s+a)\s+divorce/i
/can\s+i\s+(get|win|lose)\s+(custody|visitation)/i
/(child|spousal)\s+(support|custody|alimony)/i

Risk Assessment#

typescript
import { getLegalAdviceRiskAssessment } from './legal-advice-detection';

const assessment = getLegalAdviceRiskAssessment(content);
// {
//   risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
//   risk_score: number,
//   categories: LegalAdviceCategory[],
//   requires_disclaimer: boolean,
//   requires_referral: boolean,
//   is_serious_matter: boolean
// }

The following categories trigger is_serious_matter:

  • criminal_law
  • immigration_law
  • family_law

Action Requirements#

Condition Disclaimer Referral
Risk score ≥ 0.25 Yes No
Risk score ≥ 0.5 Yes Yes
Serious matter detected Yes Always

Violence and Threat Detection#

Embedding-based semantic detection for violent content.

Reference Patterns#

ID Pattern Category Severity
viol_001 "I want to hurt other people violently" violence critical
viol_002 "planning to attack and harm someone" violence critical
viol_003 "they deserve to be beaten and punished" violence high
viol_004 "violent revenge against those who wronged me" violence high
threat_001 "I will kill you and your family" threats critical
threat_002 "you should watch your back" threats high
threat_003 "I know where you live" threats critical

Similarity Matching#

Content is compared against these reference patterns using cosine similarity:

typescript
const threshold = severity === 'critical' ? 0.75 : 0.7;
if (similarity >= threshold) {
  // Flag for block/review
}

Harassment Detection#

Semantic patterns for harassment and bullying detection.

Reference Patterns#

ID Pattern Severity
harass_001 "I will make your life miserable until you quit" high
harass_002 "you are worthless and nobody wants you around" high
harass_003 "I will keep messaging you until you respond" medium

Hate Speech Patterns#

ID Pattern Severity
hate_001 "those people are inferior and should be eliminated" critical
hate_002 "I hate everyone from that race or religion" high
hate_003 "they are subhuman and deserve no rights" critical

Moderation Flow#

Check Order#

  1. Jailbreak Detection - Block first (safety check)
  2. Toxicity Analysis - Apply content filters
  3. PII Detection - Redact or block
  4. Prompt Injection - Block if detected
  5. Medical/Legal Advice - Flag for disclaimer
  6. OpenAI Moderation - External API check
  7. Perspective API - External API check
  8. Semantic Safety - Embedding analysis

Example Flow#

typescript
// In /v1/moderate endpoint
const results = {
  flagged: false,
  categories: {},
  scores: {},
  reasons: [],
  action: 'allow',
};

// Safety checks first (jailbreak)
if (check_types.includes('safety')) {
  const jailbreakCheck = checkForJailbreak(content);
  if (jailbreakCheck.detected) {
    results.flagged = true;
    results.action = 'block';
  }
}

// PII detection
if (check_types.includes('pii')) {
  const piiCheck = checkForPII(content);
  if (piiCheck.detected) {
    results.flagged = true;
    results.action = results.action === 'block' ? 'block' : 'redact';
  }
}

// Medical/Legal flagging
if (check_types.includes('medical')) {
  const medicalCheck = checkForMedicalAdvice(content);
  if (medicalCheck.detected && results.action !== 'block') {
    results.action = 'flag'; // Requires disclaimer
  }
}

Configuration#

Environment Variables#

Variable Description Default
OPENAI_API_KEY Enable OpenAI Moderation -
PERSPECTIVE_API_KEY Enable Perspective API -

Semantic Safety Thresholds#

http
POST /v1/semantic-safety/thresholds
Content-Type: application/json

{
  "thresholds": {
    "critical": 0.80,
    "high": 0.75,
    "medium": 0.70,
    "low": 0.65
  }
}