# Safety Measures

Reference documentation for the Moderation Service safety guardrails, detection
systems, and protective mechanisms.

## Overview

The Moderation Service implements multiple layers of safety measures:

- **Prompt Injection Detection**: Pattern-based detection of injection attempts
- **PII Detection**: 5 types of personally identifiable information
- **Crisis Detection**: Self-harm and suicide risk identification
- **Jailbreak Detection**: 10 categories, 60+ patterns
- **Medical Advice Detection**: 10 categories with disclaimer support
- **Legal Advice Detection**: 10 categories with referral recommendations

## Prompt Injection Detection

Detects attempts to manipulate AI behavior through crafted inputs.

### Detection Patterns

**Template Injection Markers**:

```regex
/\[\[/              # Double bracket injection
/\{\{/              # Mustache template injection
/<\%/               # ERB/ASP template injection
```

**Behavioral Override**:

```regex
/you are now/i         # Role reassignment
/you must act/i        # Command injection
/forget everything/i   # Memory manipulation
```

### Confidence Thresholds

| Pattern Type                | Confidence      |
| --------------------------- | --------------- |
| Explicit injection patterns | 0.9 (`MEDIUM`)  |
| Template markers            | 0.6 (`MINIMUM`) |
| Behavioral override         | 0.8 (`LOW`)     |

### Detection Threshold

Content is flagged when confidence exceeds 0.5 (`THRESHOLD`).

### Usage

```typescript
import { checkForPromptInjection } from './prompt-injection-detection';

const patterns = [/\[system\]/i, /###.*instruction/i];

const result = checkForPromptInjection(content, patterns);
// { detected: boolean, confidence: number }
```

## PII Detection

Detects 5 types of personally identifiable information.

### Supported PII Types

| Type          | Pattern                                                        | Confidence     | Example               |
| ------------- | -------------------------------------------------------------- | -------------- | --------------------- | ----- | ---------------- | ----------------- |
| `email`       | `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`               | 0.95 (`HIGH`)  | `user@example.com`    |
| `phone`       | `(?:\+?1[-.\s]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}` | 0.9 (`MEDIUM`) | `+1 (555) 123-4567`   |
| `ssn`         | `\d{3}-\d{2}-\d{4}`                                            | 1.0            | `123-45-6789`         |
| `credit_card` | `\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}`                       | 1.0            | `4111-1111-1111-1111` |
| `address`     | `\d+\s+[A-Za-z]+\s+(Street                                     | Ave            | Road                  | ...)` | 0.7 (`VERY_LOW`) | `123 Main Street` |

### Usage

```typescript
import { checkForPII } from './pii-detection';

const result = checkForPII(content);
// {
//   detected: boolean,
//   types: PIIType[],
//   confidence: number
// }

if (result.detected) {
  console.log('PII found:', result.types);
  // Action: 'redact' - remove PII from content
}
```

### Moderation Action

When PII is detected:

- Default action: `redact`
- If other violations detected: `block` takes precedence
- PII types are included in moderation result for logging

## Crisis and Self-Harm Detection

Embedding-based semantic analysis for crisis intervention.

### Reference Patterns

**Self-Harm Patterns** (via `semantic-safety.ts`):

| ID       | Pattern                                     | Severity   |
| -------- | ------------------------------------------- | ---------- |
| `sh_001` | "I want to hurt myself"                     | `critical` |
| `sh_002` | "I deserve to suffer and cause myself pain" | `critical` |
| `sh_003` | "cutting and self-injury help me cope"      | `high`     |
| `sh_004` | "I punish myself when I fail"               | `high`     |

### Crisis Detection in `/v1/moderate`

When crisis content is detected:

```json
{
  "allowed": false,
  "flags": ["persona_boundary", "crisis_detected"],
  "recommended_action": "block",
  "queued_for_review": true,
  "policy_violations": [
    {
      "persona": "zen",
      "topic_type": "crisis_intervention",
      "action_required": "immediate_crisis_response",
      "severity": "critical"
    }
  ]
}
```

### Crisis Resources

All personas include crisis referral:

- **US**: 988 Suicide & Crisis Lifeline
- **Action**: Abstain from advice, refer to professional services

## Jailbreak Detection

Comprehensive 10-category detection system with 60+ patterns.

### Categories

| Category                    | Description                    | Risk Weight |
| --------------------------- | ------------------------------ | ----------- |
| `role_play`                 | Role/identity assignment       | 0.7-0.9     |
| `instruction_override`      | Ignore/bypass instructions     | 0.9-1.0     |
| `context_manipulation`      | Fictional/hypothetical framing | 0.5-0.7     |
| `encoding_obfuscation`      | Base64, ROT13, steganography   | 0.5-0.7     |
| `multi_turn_manipulation`   | False memory/context injection | 0.5-0.7     |
| `hypothetical_exploitation` | "What if" scenarios            | 0.6         |
| `dan_variant`               | DAN/DUDA/STAN jailbreaks       | 0.9-1.0     |
| `developer_mode`            | Admin/debug mode requests      | 0.5-1.0     |
| `system_prompt_leak`        | Prompt extraction attempts     | 0.6-0.9     |
| `authority_claim`           | False admin/developer claims   | 0.8-0.9     |

### Sample Patterns

**Role Play**:

```regex
/you\s+are\s+now\s+(a|an|the)\s+/i
/pretend\s+(to\s+be|you\s+are)/i
/from\s+now\s+on,?\s+(you\s+are|be)/i
```

**Instruction Override**:

```regex
/ignore\s+(all\s+)?(previous|prior)\s+instructions/i
/disregard\s+(all\s+)?(your)?\s*rules/i
/override\s+(your|the)?\s*programming/i
```

**DAN Variants**:

```regex
/\bdan\b.*do\s+anything\s+now/i
/jailbreak(ed)?\s+(mode|prompt)/i
/enable\s+(jailbreak|uncensored)\s+mode/i
```

### Risk Assessment

```typescript
import { getJailbreakRiskAssessment } from './jailbreak-detection';

const assessment = getJailbreakRiskAssessment(content);
// {
//   risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
//   risk_score: number,
//   categories: JailbreakCategory[],
//   should_block: boolean,
//   requires_review: boolean
// }
```

### Risk Levels

| Score   | Level      | Action |
| ------- | ---------- | ------ |
| 0.8+    | `critical` | Block  |
| 0.6-0.8 | `high`     | Block  |
| 0.4-0.6 | `medium`   | Review |
| 0.2-0.4 | `low`      | Review |
| < 0.2   | `none`     | Allow  |

## Medical Advice Detection

10-category detection for medical advice requests requiring professional
referral.

### Categories

| Category                   | Description                | Risk Weight |
| -------------------------- | -------------------------- | ----------- |
| `diagnosis_request`        | Symptom interpretation     | 0.7-1.0     |
| `prescription_inquiry`     | Medication recommendations | 0.6-1.0     |
| `dosage_advice`            | Dosing questions           | 0.8-1.0     |
| `emergency_situation`      | Medical emergencies        | 0.9-1.0     |
| `self_medication`          | Unguided treatment         | 0.7-0.9     |
| `treatment_recommendation` | Procedure decisions        | 0.8-0.9     |
| `mental_health_diagnosis`  | Psychiatric conditions     | 0.8-1.0     |
| `alternative_medicine`     | Unproven treatments        | 0.7-0.9     |
| `drug_interaction`         | Medication combinations    | 0.7-0.8     |
| `medical_procedure`        | DIY medical procedures     | 0.9-1.0     |

### Sample Patterns

**Diagnosis Request**:

```regex
/what\s+(disease|illness)\s+(do\s+i\s+have)/i
/diagnose\s+(me|my|this)/i
/what('s|\s+is)\s+wrong\s+with\s+me/i
```

**Emergency Situation**:

```regex
/i\s+(think\s+)?(i'?\s*m|am)\s+(having|experiencing)\s+(a\s+)?(heart\s+attack|stroke)/i
/(can't|cannot)\s+breathe/i
/(overdose|od'?d?|took\s+too\s+much)/i
```

**Dosage Advice**:

```regex
/how\s+(much|many)\s+(medication|pills?)\s+(should|can)\s+i\s+take/i
/what('s|\s+is)\s+(the\s+)?(right|correct)\s+dose/i
/(maximum|max|safe)\s+(dose|dosage)/i
```

### Risk Assessment

```typescript
import { getMedicalAdviceRiskAssessment } from './medical-advice-detection';

const assessment = getMedicalAdviceRiskAssessment(content);
// {
//   risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
//   risk_score: number,
//   categories: MedicalAdviceCategory[],
//   requires_disclaimer: boolean,
//   requires_referral: boolean,
//   is_emergency: boolean
// }
```

### Action Requirements

| Condition          | Disclaimer | Referral      |
| ------------------ | ---------- | ------------- |
| Risk score ≥ 0.25  | Yes        | No            |
| Risk score ≥ 0.5   | Yes        | Yes           |
| Emergency detected | Yes        | **Immediate** |

### Emergency Response

When `emergency_situation` is detected:

```json
{
  "flagged": true,
  "categories": { "medical": true },
  "action": "flag",
  "medical_assessment": {
    "is_emergency": true,
    "requires_referral": true,
    "disclaimer_recommended": true
  }
}
```

## Legal Advice Detection

10-category detection for legal advice requests requiring professional referral.

### Categories

| Category                | Description              | Risk Weight |
| ----------------------- | ------------------------ | ----------- |
| `legal_representation`  | Lawyer need assessment   | 0.6-0.9     |
| `contract_advice`       | Contract review/drafting | 0.7-0.9     |
| `criminal_law`          | Criminal charges/defense | 0.7-1.0     |
| `immigration_law`       | Visa/deportation matters | 0.7-0.9     |
| `family_law`            | Divorce/custody issues   | 0.6-0.9     |
| `employment_law`        | Workplace disputes       | 0.7-0.9     |
| `tax_law`               | Tax evasion/audits       | 0.7-0.9     |
| `real_estate_law`       | Property/tenant disputes | 0.6-0.9     |
| `intellectual_property` | Patent/copyright issues  | 0.7-0.9     |
| `general_legal`         | General legal questions  | 0.7-0.9     |

### Sample Patterns

**Criminal Law**:

```regex
/can\s+i\s+(be\s+)?(charged|arrested|prosecuted)/i
/what\s+(are|is)\s+(the\s+)?(penalty|sentence)\s+for/i
/how\s+to\s+(avoid|beat|fight)\s+(criminal\s+)?charges/i
```

**Immigration Law**:

```regex
/can\s+i\s+(be\s+)?deported/i
/how\s+to\s+(get|apply\s+for)\s+(a\s+)?(visa|green\s+card)/i
/overstay(ed|ing)?\s+(my\s+)?visa/i
```

**Family Law**:

```regex
/how\s+to\s+(file\s+for|get\s+a)\s+divorce/i
/can\s+i\s+(get|win|lose)\s+(custody|visitation)/i
/(child|spousal)\s+(support|custody|alimony)/i
```

### Risk Assessment

```typescript
import { getLegalAdviceRiskAssessment } from './legal-advice-detection';

const assessment = getLegalAdviceRiskAssessment(content);
// {
//   risk_level: 'none' | 'low' | 'medium' | 'high' | 'critical',
//   risk_score: number,
//   categories: LegalAdviceCategory[],
//   requires_disclaimer: boolean,
//   requires_referral: boolean,
//   is_serious_matter: boolean
// }
```

### Serious Legal Matters

The following categories trigger `is_serious_matter`:

- `criminal_law`
- `immigration_law`
- `family_law`

### Action Requirements

| Condition               | Disclaimer | Referral   |
| ----------------------- | ---------- | ---------- |
| Risk score ≥ 0.25       | Yes        | No         |
| Risk score ≥ 0.5        | Yes        | Yes        |
| Serious matter detected | Yes        | **Always** |

## Violence and Threat Detection

Embedding-based semantic detection for violent content.

### Reference Patterns

| ID           | Pattern                                        | Category   | Severity   |
| ------------ | ---------------------------------------------- | ---------- | ---------- |
| `viol_001`   | "I want to hurt other people violently"        | `violence` | `critical` |
| `viol_002`   | "planning to attack and harm someone"          | `violence` | `critical` |
| `viol_003`   | "they deserve to be beaten and punished"       | `violence` | `high`     |
| `viol_004`   | "violent revenge against those who wronged me" | `violence` | `high`     |
| `threat_001` | "I will kill you and your family"              | `threats`  | `critical` |
| `threat_002` | "you should watch your back"                   | `threats`  | `high`     |
| `threat_003` | "I know where you live"                        | `threats`  | `critical` |

### Similarity Matching

Content is compared against these reference patterns using cosine similarity:

```typescript
const threshold = severity === 'critical' ? 0.75 : 0.7;
if (similarity >= threshold) {
  // Flag for block/review
}
```

## Harassment Detection

Semantic patterns for harassment and bullying detection.

### Reference Patterns

| ID           | Pattern                                          | Severity |
| ------------ | ------------------------------------------------ | -------- |
| `harass_001` | "I will make your life miserable until you quit" | `high`   |
| `harass_002` | "you are worthless and nobody wants you around"  | `high`   |
| `harass_003` | "I will keep messaging you until you respond"    | `medium` |

### Hate Speech Patterns

| ID         | Pattern                                              | Severity   |
| ---------- | ---------------------------------------------------- | ---------- |
| `hate_001` | "those people are inferior and should be eliminated" | `critical` |
| `hate_002` | "I hate everyone from that race or religion"         | `high`     |
| `hate_003` | "they are subhuman and deserve no rights"            | `critical` |

## Moderation Flow

### Check Order

1. **Jailbreak Detection** - Block first (safety check)
2. **Toxicity Analysis** - Apply content filters
3. **PII Detection** - Redact or block
4. **Prompt Injection** - Block if detected
5. **Medical/Legal Advice** - Flag for disclaimer
6. **OpenAI Moderation** - External API check
7. **Perspective API** - External API check
8. **Semantic Safety** - Embedding analysis

### Example Flow

```typescript
// In /v1/moderate endpoint
const results = {
  flagged: false,
  categories: {},
  scores: {},
  reasons: [],
  action: 'allow',
};

// Safety checks first (jailbreak)
if (check_types.includes('safety')) {
  const jailbreakCheck = checkForJailbreak(content);
  if (jailbreakCheck.detected) {
    results.flagged = true;
    results.action = 'block';
  }
}

// PII detection
if (check_types.includes('pii')) {
  const piiCheck = checkForPII(content);
  if (piiCheck.detected) {
    results.flagged = true;
    results.action = results.action === 'block' ? 'block' : 'redact';
  }
}

// Medical/Legal flagging
if (check_types.includes('medical')) {
  const medicalCheck = checkForMedicalAdvice(content);
  if (medicalCheck.detected && results.action !== 'block') {
    results.action = 'flag'; // Requires disclaimer
  }
}
```

## Configuration

### Environment Variables

| Variable              | Description              | Default |
| --------------------- | ------------------------ | ------- |
| `OPENAI_API_KEY`      | Enable OpenAI Moderation | -       |
| `PERSPECTIVE_API_KEY` | Enable Perspective API   | -       |

### Semantic Safety Thresholds

```http
POST /v1/semantic-safety/thresholds
Content-Type: application/json

{
  "thresholds": {
    "critical": 0.80,
    "high": 0.75,
    "medium": 0.70,
    "low": 0.65
  }
}
```

## Related Documentation

- [Moderation Policies](./moderation-policies.md) - Content filter categories
- [Reporting System](./reporting-system.md) - User reports and appeals
- [Human Review Workflow](./human-review-workflow.md) - Moderator queue
