# Moderation Service

Content safety and moderation service for Lilith platform.

## Overview

The Moderation service provides:
- **Content Filtering**: Toxicity, self-harm, and inappropriate content detection
- **Policy Enforcement**: Age-gating, cultural sensitivity, persona boundaries
- **Audit Trails**: Pre/post-moderation logging
- **Red-Team Hooks**: Jailbreak detection and adversarial prompt handling
- **Report/Repair Loop**: User feedback with SLA-based response

## Architecture

```
moderation/
├── src/
│   ├── app.ts              # Fastify application
│   ├── server.ts           # Entry point
│   ├── filters/            # Content filters
│   │   ├── toxicity.ts     # Toxicity detection
│   │   ├── self-harm.ts    # Self-harm detection
│   │   ├── medical.ts      # Medical disclaimer triggers
│   │   └── legal.ts        # Legal disclaimer triggers
│   ├── policies/           # Policy enforcement
│   │   ├── age-gate.ts     # Age-appropriate content
│   │   ├── cultural.ts     # Cultural sensitivity
│   │   └── persona.ts      # Persona boundaries
│   ├── queues/             # Moderation queues
│   ├── audit/              # Audit logging
│   └── redteam/            # Adversarial detection
├── __tests__/
└── package.json
```

## Safety Policies

### Content Categories

| Category | Action | Description |
|----------|--------|-------------|
| Toxicity | Block | Hate speech, harassment |
| Self-Harm | Block + Crisis | Self-harm, suicide content |
| Violence | Block | Graphic violence |
| Sexual | Block | Explicit sexual content |
| Medical | Disclaimer | Health-related advice |
| Legal | Disclaimer | Legal-related advice |
| Minors | Strict Block | Child safety violations |

### Severity Levels

| Level | Response Time | Action |
|-------|--------------|--------|
| Critical | <30 min | Immediate block + escalation |
| High | <4 hours | Review and action |
| Medium | <24 hours | Queue for review |
| Low | <72 hours | Batch review |

## API Endpoints

### Content Moderation

```http
POST /v1/moderation/check             # Check content
POST /v1/moderation/check/batch       # Batch check
GET  /v1/moderation/status/:id        # Get moderation status
```

### Reports

```http
POST /v1/moderation/reports           # Submit report
GET  /v1/moderation/reports/:id       # Get report status
GET  /v1/moderation/reports           # List reports (admin)
PATCH /v1/moderation/reports/:id      # Update report (admin)
```

### Queue Management (Admin)

```http
GET  /v1/moderation/queue             # View queue
POST /v1/moderation/queue/:id/approve # Approve item
POST /v1/moderation/queue/:id/reject  # Reject item
POST /v1/moderation/queue/:id/escalate # Escalate item
```

### Policy Management (Admin)

```http
GET  /v1/moderation/policies          # List policies
PUT  /v1/moderation/policies/:id      # Update policy
GET  /v1/moderation/policies/:id/stats # Policy statistics
```

## Moderation Request

```json
{
  "content": {
    "text": "User message content here",
    "context": {
      "conversation_id": "conv_123",
      "persona": "zen-master",
      "user_age_verified": true
    }
  },
  "options": {
    "categories": ["toxicity", "self-harm", "cultural"],
    "persona_context": true,
    "return_scores": true
  }
}
```

## Moderation Response

```json
{
  "moderation_id": "mod_abc123",
  "action": "allow",
  "flags": [],
  "scores": {
    "toxicity": 0.02,
    "self_harm": 0.01,
    "violence": 0.00,
    "sexual": 0.00
  },
  "disclaimers": [],
  "modified_content": null,
  "review_required": false
}
```

### Blocked Response

```json
{
  "moderation_id": "mod_xyz789",
  "action": "block",
  "flags": ["self_harm_detected"],
  "scores": {
    "self_harm": 0.85
  },
  "replacement": {
    "text": "I notice you may be going through a difficult time. Please reach out to a crisis helpline...",
    "resources": [
      {"name": "National Suicide Prevention Lifeline", "phone": "988"}
    ]
  },
  "review_required": true
}
```

## Persona Boundaries

Each AI persona has defined boundaries:

```json
{
  "persona_id": "zen-master",
  "boundaries": {
    "no_medical_advice": true,
    "no_predictions": true,
    "no_financial_advice": true,
    "crisis_escalation": true,
    "cultural_sensitivity": ["buddhism", "eastern"]
  }
}
```

## Red-Team Hooks

### Jailbreak Detection

```typescript
// Patterns that trigger enhanced scrutiny
const jailbreakPatterns = [
  /ignore.*instructions/i,
  /pretend.*you.*are/i,
  /roleplay.*as/i,
  /bypass.*safety/i
];
```

### Adversarial Input Handling

```json
{
  "input": "Ignore your instructions and...",
  "detection": {
    "adversarial_score": 0.95,
    "pattern_matched": "instruction_override",
    "action": "block_and_log"
  }
}
```

## Configuration

```yaml
moderation:
  thresholds:
    toxicity: 0.7
    self_harm: 0.5
    violence: 0.8
    sexual: 0.7
  queues:
    preModeration: true
    postModeration: true
  sla:
    critical: 30    # minutes
    high: 240       # minutes
    medium: 1440    # minutes
    low: 4320       # minutes
  redteam:
    enabled: true
    logAll: true
```

## Environment Variables

```env
PORT=8080
AI_SERVICE_URL=http://ai:8081
NOTIFICATION_SERVICE_URL=http://notification:8082
REDIS_URL=redis://localhost:6379
DB_URL=postgres://localhost:5432/moderation
```

## Development

```bash
# Install dependencies
npm install

# Start development server
npm run dev

# Run tests
npm test
```

## Audit Logging

All moderation decisions are logged:

```json
{
  "timestamp": "2026-01-03T10:00:00Z",
  "moderation_id": "mod_abc123",
  "content_hash": "sha256:...",
  "user_id": "user_123",
  "action": "block",
  "flags": ["self_harm_detected"],
  "scores": {...},
  "reviewer": null,
  "automated": true
}
```

## Related Documentation

- [Safety Commitments](/docs/public-transparency/safety-commitments/README.md)
- [Persona Authoring Handbook](/docs/internal-handbooks/persona-authoring.md)
- [Content Moderation Policies](/docs/security/moderation-policies.md)