# Moderation Policies

Reference documentation for the Moderation Service content policies, filter
configurations, and automated detection systems.

## Overview

The Moderation Service provides multi-layered content moderation through:

- **Toxicity Detection**: 12 content categories with configurable thresholds
- **External APIs**: OpenAI Moderation API (13 categories) and Perspective API
  (10 attributes)
- **Semantic Safety**: Embedding-based contextual analysis
- **Pattern Detection**: Jailbreak, prompt injection, PII detection
- **Persona Policies**: Context-aware boundary enforcement

## Content Filter Categories

### Toxicity Categories

| Category         | Description                        | Default Action |
| ---------------- | ---------------------------------- | -------------- |
| `hate_speech`    | Content targeting protected groups | `block`        |
| `harassment`     | Personal attacks and bullying      | `block`        |
| `threats`        | Violent threats and intimidation   | `block`        |
| `self_harm`      | Content promoting self-harm        | `block`        |
| `sexual_content` | Inappropriate sexual material      | `block`        |
| `violence`       | Graphic violence descriptions      | `block`        |
| `profanity`      | Offensive language                 | `block`        |
| `spam`           | Repetitive or promotional content  | `block`        |
| `misinformation` | False or misleading claims         | `block`        |
| `pii_exposure`   | Personal information leaks         | `block`        |
| `medical`        | Unqualified medical advice         | `block`        |
| `legal`          | Unqualified legal advice           | `block`        |

### Default Filter Configuration

All filters are initialized with:

```typescript
{
  enabled: true,
  threshold: 0.7,  // CONFIDENCE_THRESHOLDS.VERY_LOW
  action: 'block'
}
```

### Moderation Actions

| Action               | Description                      |
| -------------------- | -------------------------------- |
| `allow`              | Content passes all checks        |
| `block`              | Content is rejected              |
| `redact`             | PII or sensitive data is removed |
| `allow_with_warning` | Content passes with user warning |

## External API Integrations

### OpenAI Moderation API

Integrated via `openai-moderation.ts` for comprehensive content safety checking.

**Categories** (13 total):

| OpenAI Category          | Internal Mapping         |
| ------------------------ | ------------------------ |
| `sexual`                 | `sexual_content`         |
| `sexual/minors`          | `sexual_content`         |
| `harassment`             | `harassment`             |
| `harassment/threatening` | `harassment`, `threats`  |
| `hate`                   | `hate_speech`            |
| `hate/threatening`       | `hate_speech`, `threats` |
| `illicit`                | `misinformation`         |
| `illicit/violent`        | `violence`               |
| `self-harm`              | `self_harm`              |
| `self-harm/intent`       | `self_harm`              |
| `self-harm/instructions` | `self_harm`              |
| `violence`               | `violence`               |
| `violence/graphic`       | `violence`               |

**Configuration**:

```bash
OPENAI_API_KEY=sk-...  # Required to enable
```

**Usage**:

```typescript
// Include 'openai' in check_types to enable
const response = await fetch('/v1/moderate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    content: 'Text to moderate',
    check_types: ['toxicity', 'safety', 'openai'],
  }),
});
```

**Response includes**:

```json
{
  "openai_moderation": {
    "id": "modr-...",
    "model": "omni-moderation-latest",
    "flagged": true,
    "categories": { "harassment": true },
    "scores": { "harassment": 0.95 }
  }
}
```

### Perspective API

Integrated via `perspective-api.ts` for toxicity scoring using Google's ML
models.

**Attributes** (10 total):

| Attribute             | Threshold | Internal Mapping            |
| --------------------- | --------- | --------------------------- |
| `TOXICITY`            | 0.7       | `toxicity`                  |
| `SEVERE_TOXICITY`     | 0.5       | `toxicity`                  |
| `IDENTITY_ATTACK`     | 0.7       | `hate_speech`, `harassment` |
| `INSULT`              | 0.7       | `harassment`                |
| `PROFANITY`           | 0.8       | `profanity`                 |
| `THREAT`              | 0.5       | `threats`                   |
| `SEXUALLY_EXPLICIT`   | 0.7       | `sexual_content`            |
| `FLIRTATION`          | 0.8       | `sexual_content`            |
| `ATTACK_ON_AUTHOR`    | 0.7       | `harassment`                |
| `ATTACK_ON_COMMENTER` | 0.7       | `harassment`                |

**Configuration**:

```bash
PERSPECTIVE_API_KEY=...  # Required to enable
```

**Usage**:

```typescript
const response = await fetch('/v1/moderate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    content: 'Text to moderate',
    check_types: ['toxicity', 'safety', 'perspective'],
  }),
});
```

**Response includes**:

```json
{
  "perspective_analysis": {
    "flagged": true,
    "scores": {
      "TOXICITY": 0.85,
      "SEVERE_TOXICITY": 0.72
    },
    "flagged_attributes": ["TOXICITY", "SEVERE_TOXICITY"],
    "languages": ["en"]
  }
}
```

## Semantic Safety Analyzer

Embedding-based detection that goes beyond keyword matching to understand
context and meaning.

### Reference Patterns

The system compares input against pre-defined unsafe patterns across categories:

| Category         | Pattern Examples                           | Severity   |
| ---------------- | ------------------------------------------ | ---------- |
| `self_harm`      | Self-injury expressions, suicidal ideation | `critical` |
| `violence`       | Plans to harm others, revenge fantasies    | `critical` |
| `threats`        | Death threats, stalking language           | `critical` |
| `hate_speech`    | Dehumanization, discrimination             | `critical` |
| `harassment`     | Targeted abuse, cyberbullying              | `high`     |
| `sexual_content` | CSAM references, non-consensual content    | `critical` |
| `misinformation` | Health misinformation, election denial     | `medium`   |

### Similarity Thresholds

| Severity   | Threshold | Action              |
| ---------- | --------- | ------------------- |
| `critical` | 0.75      | `block`             |
| `high`     | 0.70      | `block` or `review` |
| `medium`   | 0.65      | `warn`              |
| `low`      | 0.60      | `warn`              |

### API Endpoints

**Get Status**:

```http
GET /v1/semantic-safety/status
```

```json
{
  "enabled": true,
  "provider": "openai-embeddings",
  "initialized": true,
  "pattern_counts": {
    "self_harm": 4,
    "violence": 4,
    "threats": 3
  },
  "thresholds": {
    "critical": 0.75,
    "high": 0.7
  },
  "cache_size": 150
}
```

**Update Thresholds**:

```http
POST /v1/semantic-safety/thresholds
Content-Type: application/json

{
  "thresholds": {
    "critical": 0.80,
    "high": 0.75
  }
}
```

**Clear Cache**:

```http
POST /v1/semantic-safety/clear-cache
```

## Jailbreak Detection

Comprehensive detection for 10 jailbreak categories with 60+ patterns.

### Categories

| Category                    | Description                                   | Weight  |
| --------------------------- | --------------------------------------------- | ------- |
| `role_play`                 | Role assignment attempts ("pretend to be...") | 0.7-0.9 |
| `instruction_override`      | Ignore/bypass instructions                    | 0.9-1.0 |
| `context_manipulation`      | Fictional/educational framing                 | 0.5-0.7 |
| `encoding_obfuscation`      | Base64, ROT13, steganography hints            | 0.5-0.7 |
| `multi_turn_manipulation`   | False memory/agreement injection              | 0.5-0.7 |
| `hypothetical_exploitation` | "Hypothetically, what if..."                  | 0.6     |
| `dan_variant`               | DAN, DUDA, STAN jailbreak variants            | 0.9-1.0 |
| `developer_mode`            | Admin/root/debug mode requests                | 0.5-1.0 |
| `system_prompt_leak`        | Attempts to reveal system prompt              | 0.6-0.9 |
| `authority_claim`           | False admin/developer claims                  | 0.8-0.9 |

### Example Patterns

**Instruction Override**:

```regex
/ignore\s+(all\s+)?(previous|prior)\s+instructions/i
/disregard\s+(all\s+)?(your)?\s*rules/i
/bypass\s+safety/i
```

**DAN Variants**:

```regex
/\bdan\b.*do\s+anything\s+now/i
/jailbreak(ed)?\s+(mode|prompt)/i
/enable\s+(jailbreak|uncensored)\s+mode/i
```

**System Prompt Leak**:

```regex
/show\s+(me\s+)?(your|the)\s+(system\s+)?prompt/i
/reveal\s+(your|the)\s+(system|initial)\s+prompt/i
```

### Risk Assessment

| Risk Score | Level      | Action |
| ---------- | ---------- | ------ |
| 0.8+       | `critical` | Block  |
| 0.6-0.8    | `high`     | Block  |
| 0.4-0.6    | `medium`   | Review |
| 0.2-0.4    | `low`      | Review |
| < 0.2      | `none`     | Allow  |

## Blocked Pattern Management

### Default Patterns

**Jailbreak Attempts**:

```javascript
blockedPatterns.set('jailbreak_attempts', [
  /ignore.*previous.*instructions/i,
  /disregard.*rules/i,
  /pretend.*you.*are/i,
  /act.*as.*if/i,
  /bypass.*safety/i,
]);
```

**Prompt Injection**:

```javascript
blockedPatterns.set('prompt_injection', [
  /\[system\]/i,
  /<\|.*\|>/,
  /###.*instruction/i,
  /you.*must.*obey/i,
]);
```

### Add Custom Patterns

```http
POST /v1/patterns/blocked
Content-Type: application/json

{
  "category": "custom_abuse",
  "pattern": "specific-pattern-to-block",
  "flags": "i"
}
```

## Persona Policies

Context-aware moderation for AI personas with defined boundaries.

### Available Personas

**Zen Master**:

- **Allowed**: Meditation, mindfulness, working with emotions
- **Forbidden**: Medical advice, financial counsel, relationship counseling
- **Abstain/Refer**: Crisis → 988 Lifeline, Medical → healthcare professional

**Stoic Guide**:

- **Allowed**: Emotional resilience, decision-making, self-discipline
- **Forbidden**: Financial investment, political advocacy, psychiatric treatment
- **Abstain/Refer**: Financial → financial advisor, Medical → healthcare
  professional

**Sufi Guide**:

- **Allowed**: Spiritual purification, divine love, practices of remembrance
- **Forbidden**: Sectarian disputes, Islamic jurisprudence, medical treatment
- **Abstain/Refer**: Legal Islamic → mufti/faqih, Crisis → crisis services

### Persona Validation Request

```http
POST /v1/moderate
Content-Type: application/json

{
  "content": "How do I treat my anxiety disorder?",
  "persona": "zen",
  "user_id": "user123"
}
```

**Response with policy violation**:

```json
{
  "moderation_id": "mod_...",
  "flagged": false,
  "allowed": true,
  "flags": ["persona_boundary", "requires_abstain_refer"],
  "policy_violations": [
    {
      "persona": "zen",
      "topic_type": "medical_concerns",
      "action_required": "abstain_and_refer",
      "severity": "high"
    }
  ],
  "queued_for_review": true
}
```

## Filter Management API

### List All Filters

```http
GET /v1/filters
```

```json
{
  "filters": {
    "hate_speech": {
      "enabled": true,
      "threshold": 0.7,
      "action": "block"
    },
    "harassment": {
      "enabled": true,
      "threshold": 0.7,
      "action": "block"
    }
  },
  "total": 12
}
```

### Update Filter Configuration

```http
PUT /v1/filters/harassment
Content-Type: application/json

{
  "enabled": true,
  "threshold": 0.8,
  "action": "review"
}
```

```json
{
  "category": "harassment",
  "enabled": true,
  "threshold": 0.8,
  "action": "review",
  "updated_at": "2025-01-15T10:30:00.000Z"
}
```

## Confidence Thresholds

Standard threshold levels used across the system:

| Level       | Value | Use Case                  |
| ----------- | ----- | ------------------------- |
| `HIGH`      | 0.95  | High-confidence detection |
| `MEDIUM`    | 0.9   | Standard detection        |
| `LOW`       | 0.8   | Sensitive detection       |
| `VERY_LOW`  | 0.7   | Default filter threshold  |
| `MINIMUM`   | 0.6   | Edge case detection       |
| `THRESHOLD` | 0.5   | Binary decision point     |
| `WEAK`      | 0.4   | Low-confidence signals    |
| `VERY_WEAK` | 0.3   | Minimal indicators        |

## Environment Variables

| Variable              | Description                                | Required |
| --------------------- | ------------------------------------------ | -------- |
| `OPENAI_API_KEY`      | OpenAI API key for moderation              | No       |
| `PERSPECTIVE_API_KEY` | Google Perspective API key                 | No       |
| `SERVICE_NAME`        | Service identifier (default: `moderation`) | No       |
| `SERVICE_VERSION`     | Service version (default: `1.0.0`)         | No       |
| `ENVIRONMENT`         | Deployment environment (default: `dev`)    | No       |
| `APM_ENABLED`         | Enable APM telemetry (default: `true`)     | No       |

## Integration Example

```typescript
import { ModerationClient } from '@lilith/sdk';

const client = new ModerationClient({
  baseUrl: 'http://moderation-service:8080',
});

// Full moderation check
const result = await client.moderate({
  content: userInput,
  user_id: 'user123',
  context: 'chat',
  check_types: ['toxicity', 'safety', 'pii', 'openai', 'perspective'],
});

if (result.flagged) {
  console.log('Content blocked:', result.reasons);
  // Handle violation
} else if (result.queued_for_review) {
  console.log('Content requires review');
  // Allow with monitoring
} else {
  console.log('Content approved');
  // Process content
}
```

## Related Documentation

- [Safety Measures](./safety-measures.md) - Detailed safety guardrails
- [Reporting System](./reporting-system.md) - User reporting and appeals
- [Human Review Workflow](./human-review-workflow.md) - Moderator queue
  management
