Reference documentation for user reporting, appeals, reputation management, and human review workflows.
Overview#
The Moderation Service reporting system provides:
- Appeal Process: Challenge moderation decisions
- User History: Track moderation events per user
- Reputation System: Trust levels based on behavior
- Human Review Workflow: Queue-based moderator workflow
- Reviewer Management: Capacity tracking and performance metrics
Appeals Process#
Users can appeal moderation decisions through the appeals API.
Submit Appeal#
POST /v1/appeals
Content-Type: application/json
{
"moderation_id": "mod_1704067200_abc123",
"user_id": "user123",
"reason": "This was educational content about history",
"additional_context": "Discussing historical events in an academic context"
}
Required Fields:
| Field | Type | Description |
|---|---|---|
moderation_id |
string | Original moderation decision ID |
user_id |
string | User submitting the appeal |
reason |
string | Justification for appeal |
Optional Fields:
| Field | Type | Description |
|---|---|---|
additional_context |
string | Supporting information |
Appeal Response#
{
"appeal_id": "appeal_1704067200_xyz789",
"original_decision": {
"moderation_id": "mod_1704067200_abc123",
"flagged": true,
"action": "block",
"categories": { "hate_speech": true }
},
"appeal_reason": "This was educational content about history",
"additional_context": "Discussing historical events in an academic context",
"status": "approved",
"new_action": "allow_with_warning",
"created_at": "2025-01-15T10:30:00.000Z"
}
Appeal Status Values#
| Status | Description |
|---|---|
pending_review |
Awaiting moderator review |
approved |
Appeal granted, action revised |
rejected |
Appeal denied, original decision stands |
Automatic Approval#
Appeals containing educational or research justification keywords may be automatically approved:
// Automatic approval keywords
if (reason.includes('educational') || reason.includes('research')) {
reevaluation.status = 'approved';
reevaluation.new_action = 'allow_with_warning';
}
User History#
Track all moderation events for a specific user.
Get User History#
GET /v1/users/:userId/history?limit=50&offset=0
Query Parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
string | "100" | Max records to return |
offset |
string | "0" | Skip N records |
Response:
{
"user_id": "user123",
"total": 47,
"flagged_count": 3,
"history": [
{
"moderation_id": "mod_1704067200_abc123",
"timestamp": "2025-01-15T10:30:00.000Z",
"content_length": 256,
"flagged": false,
"categories": {},
"scores": {},
"action": "allow"
}
]
}
History Sorting#
Results are sorted by timestamp in descending order (newest first).
Reputation System#
User reputation is calculated based on moderation history.
Get User Reputation#
GET /v1/users/:userId/reputation
Response:
{
"total_content": 150,
"flagged_content": 3,
"violations": {
"profanity": 2,
"harassment": 1
},
"reputation_score": 98,
"trust_level": "trusted",
"last_flagged": "2025-01-10T15:20:00.000Z"
}
Reputation Score Calculation#
reputation_score = max(0, 100 - (flagged_content / total_content) * 100);
Trust Levels#
| Trust Level | Score Threshold | Description |
|---|---|---|
trusted |
≥ 90 | Excellent history, priority moderation |
standard |
≥ 70 | Normal user, standard moderation |
limited |
≥ 50 | Some violations, stricter moderation |
restricted |
< 50 | Multiple violations, most restrictive |
Reputation Thresholds#
const REPUTATION_THRESHOLDS = {
HIGH: 90, // trusted
MEDIUM: 70, // standard
LOW: 50, // limited threshold
};
User Flags Update#
Flags are automatically updated on each moderation event:
function updateUserFlags(userFlags, userId, moderationResult) {
const flags = userFlags.get(userId) || initialFlags;
flags.total_content++;
if (moderationResult.flagged) {
flags.flagged_content++;
flags.last_flagged = new Date().toISOString();
// Track violations by category
Object.keys(moderationResult.categories).forEach((category) => {
flags.violations[category] = (flags.violations[category] || 0) + 1;
});
}
userFlags.set(userId, flags);
}
Human Review Workflow#
Comprehensive queue-based system for human moderators.
Review Types#
| Type | Description | Est. Review Time |
|---|---|---|
content_moderation |
Standard content review | 5 min |
appeal_review |
User appeal review | 15 min |
policy_violation |
Policy violation assessment | 10 min |
edge_case |
Ambiguous/complex cases | 20 min |
Submit for Review#
POST /v1/reviews
Content-Type: application/json
{
"type": "content_moderation",
"content_id": "content_abc123",
"content_type": "comment",
"reason": "Automated moderation flagged potential harassment",
"severity": "medium",
"priority": "high",
"submitted_by": "system",
"context": {
"channel": "forum",
"thread_id": "thread_xyz"
},
"auto_moderation_result": {
"flagged": true,
"categories": { "harassment": true },
"scores": { "harassment": 0.78 }
}
}
Response:
{
"review_id": "review_1704067200_def456",
"status": "pending",
"queue_position": 3,
"estimated_review_time": 5
}
Review Priority#
| Priority | SLA (minutes) | Queue Order |
|---|---|---|
critical |
30 | First |
high |
120 | Second |
medium |
480 | Third |
low |
1440 | Last |
Review Severity#
| Severity | Description |
|---|---|
critical |
Immediate safety concern |
high |
Significant violation |
medium |
Moderate violation |
low |
Minor issue |
Get Review Queues#
GET /v1/review-queues
Response:
{
"queues": {
"content_moderation": {
"type": "content_moderation",
"pending_items_count": 15,
"in_review_items_count": 3,
"completed_items_count": 127,
"escalated_items_count": 5,
"total_capacity": 100,
"active_reviewers": 4,
"average_review_time": 5.2,
"sla_breach_count": 2
}
},
"total_pending": 25,
"total_in_review": 5,
"total_active_reviewers": 6
}
Get Available Reviews#
GET /v1/reviews/available?reviewer_id=reviewer_abc&types=content_moderation,appeal_review&limit=10
Response:
{
"available_reviews": [
{
"id": "review_1704067200_def456",
"type": "content_moderation",
"content_id": "content_abc123",
"reason": "Automated moderation flagged content",
"severity": "medium",
"priority": "high",
"status": "pending",
"queue_position": 1,
"created_at": "2025-01-15T10:30:00.000Z"
}
],
"total_available": 15,
"reviewer_capacity": 7,
"queue_types": ["content_moderation", "appeal_review"]
}
Assign Review#
POST /v1/reviews/:reviewId/assign
Content-Type: application/json
{
"reviewer_id": "reviewer_abc123"
}
Response:
{
"assignment_id": "assign_1704067200_xyz",
"review_id": "review_1704067200_def456",
"assigned_to": "reviewer_abc123",
"assigned_at": "2025-01-15T10:35:00.000Z",
"sla_deadline": "2025-01-15T12:35:00.000Z"
}
Assignment Validation#
Assignment is rejected if:
- Review is not in
pendingstatus - Reviewer is not active
- Reviewer has reached daily capacity
- Reviewer cannot review this type
Unassign Review#
POST /v1/reviews/:reviewId/unassign
Content-Type: application/json
{
"actor_id": "supervisor_abc"
}
Submit Decision#
POST /v1/reviews/:reviewId/decision
Content-Type: application/json
{
"action": "reject",
"reasoning": "Content violates community guidelines for harassment",
"reviewer_id": "reviewer_abc123",
"confidence": 0.95,
"tags": ["harassment", "targeted"],
"notes": "Repeated pattern from this user",
"escalation_requested": false
}
Response:
{
"decision_id": "decision_1704067200_abc",
"review_id": "review_1704067200_def456",
"action": "reject",
"status": "completed",
"completed_at": "2025-01-15T10:45:00.000Z"
}
Review Actions#
| Action | Description |
|---|---|
approve |
Content is acceptable |
reject |
Content violates policies |
escalate |
Requires senior review |
request_changes |
Ask for content modification |
remove_content |
Delete content entirely |
Get Review Details#
GET /v1/reviews/:reviewId
Get Review History#
GET /v1/reviews/:reviewId/history
Response:
{
"review_id": "review_1704067200_def456",
"events": [
{
"event_id": "evt_1704067200_001",
"event_type": "submitted",
"timestamp": "2025-01-15T10:30:00.000Z",
"actor_id": "system",
"details": { "reason": "Auto-flagged", "priority": "high" }
},
{
"event_id": "evt_1704067200_002",
"event_type": "assigned",
"timestamp": "2025-01-15T10:35:00.000Z",
"actor_id": "reviewer_abc123",
"details": { "sla_deadline": "2025-01-15T12:35:00.000Z" }
},
{
"event_id": "evt_1704067200_003",
"event_type": "decision_made",
"timestamp": "2025-01-15T10:45:00.000Z",
"actor_id": "reviewer_abc123",
"details": { "action": "reject", "confidence": 0.95 }
},
{
"event_id": "evt_1704067200_004",
"event_type": "completed",
"timestamp": "2025-01-15T10:45:00.000Z",
"actor_id": "reviewer_abc123"
}
]
}
Event Types#
| Event | Description |
|---|---|
submitted |
Review created |
assigned |
Reviewer assigned |
unassigned |
Reviewer removed |
decision_made |
Decision submitted |
escalated |
Escalated to senior |
completed |
Review finished |
Reviewer Management#
Create Reviewer Profile#
POST /v1/reviewers
Content-Type: application/json
{
"user_id": "user_abc123",
"display_name": "Jane Smith",
"email": "jane@example.com",
"can_review_types": ["content_moderation", "appeal_review"],
"daily_capacity": 15,
"experience_level": "intermediate",
"specialties": ["harassment", "hate_speech"]
}
Response:
{
"reviewer_id": "reviewer_1704067200_xyz",
"status": "created",
"profile": {
"reviewer_id": "reviewer_1704067200_xyz",
"user_id": "user_abc123",
"display_name": "Jane Smith",
"email": "jane@example.com",
"can_review_types": ["content_moderation", "appeal_review"],
"daily_capacity": 15,
"current_assigned": 0,
"total_completed": 0,
"average_review_time": 0,
"accuracy_score": 1.0,
"experience_level": "intermediate",
"specialties": ["harassment", "hate_speech"],
"active": true,
"created_at": "2025-01-15T10:30:00.000Z"
}
}
Experience Levels#
| Level | Description |
|---|---|
beginner |
New reviewer, limited types |
intermediate |
Standard reviewer |
advanced |
Senior reviewer |
expert |
Senior + edge cases |
Get Reviewer Profile#
GET /v1/reviewers/:reviewerId
Response includes:
{
"reviewer_id": "reviewer_abc123",
"display_name": "Jane Smith",
"can_review_types": ["content_moderation", "appeal_review"],
"daily_capacity": 15,
"current_assigned": 5,
"total_completed": 127,
"average_review_time": 4.5,
"accuracy_score": 0.98,
"experience_level": "intermediate",
"active": true,
"current_assignments": 5,
"utilization_rate": 0.333
}
Update Reviewer Profile#
PUT /v1/reviewers/:reviewerId
Content-Type: application/json
{
"daily_capacity": 20,
"experience_level": "advanced",
"can_review_types": ["content_moderation", "appeal_review", "policy_violation"]
}
Metrics and Analytics#
Get Review Metrics#
GET /v1/review/analytics/metrics?period=24h&type=content_moderation
Query Parameters:
| Parameter | Format | Example | Description |
|---|---|---|---|
period |
{n}{unit} |
24h, 7d, 4w |
Time period |
type |
ReviewType | content_moderation |
Filter by type |
reviewer_id |
string | reviewer_abc |
Filter by reviewer |
Response:
{
"period": "24h",
"total_reviews": 47,
"completed_reviews": 42,
"escalated_reviews": 5,
"average_review_time": 5.3,
"sla_compliance_rate": 0.94,
"reviewer_performance": {
"reviewer_abc": 15,
"reviewer_xyz": 12,
"reviewer_def": 20
},
"type_breakdown": {
"content_moderation": 30,
"appeal_review": 10,
"policy_violation": 5,
"edge_case": 2
},
"priority_breakdown": {
"critical": 2,
"high": 15,
"medium": 25,
"low": 5
},
"severity_breakdown": {
"critical": 3,
"high": 12,
"medium": 27,
"low": 5
}
}
Period Units#
| Unit | Description |
|---|---|
h |
Hours |
d |
Days |
w |
Weeks |
m |
Months |
SLA Configuration#
Default SLA Times#
| Priority | SLA (minutes) |
|---|---|
critical |
30 |
high |
120 |
medium |
480 |
low |
1440 |
SLA Tracking#
SLA deadline is calculated at assignment:
const slaDeadline = new Date(
now.getTime() + SLA_TIMES[review.priority] * 60000
);
SLA Compliance#
Compliance rate is tracked in metrics:
sla_compliance_rate = sla_compliant / total_reviews;
Integration Example#
// Submit content for moderation
const moderationResult = await fetch('/v1/moderate', {
method: 'POST',
body: JSON.stringify({ content: userContent, user_id: userId }),
}).then((r) => r.json());
// If flagged and requires review
if (moderationResult.queued_for_review) {
// Content was automatically submitted to review queue
console.log('Content submitted for human review');
}
// User appeals the decision
if (moderationResult.flagged) {
const appeal = await fetch('/v1/appeals', {
method: 'POST',
body: JSON.stringify({
moderation_id: moderationResult.moderation_id,
user_id: userId,
reason: 'This is educational content',
additional_context: 'Teaching about historical events',
}),
}).then((r) => r.json());
if (appeal.status === 'approved') {
console.log('Appeal approved:', appeal.new_action);
}
}
// Check user reputation
const reputation = await fetch(`/v1/users/${userId}/reputation`).then((r) =>
r.json()
);
if (reputation.trust_level === 'restricted') {
console.log('User has restricted access due to violations');
}
Related Documentation#
- Moderation Policies - Content filter categories
- Safety Measures - Safety guardrails
- API Documentation - Full API reference