Historical task source. The Eve task board holds current task state.
This document outlines the monitoring dashboards, metrics, and alerting configuration for Tara's production environment.
Table of Contents#
- Monitoring Overview
- Key Metrics
- Dashboard Configuration
- Alert Configuration
- Log Management
- Distributed Tracing
- Real User Monitoring
- Status Page
Monitoring Overview#
Monitoring Stack#
| Component | Tool | Purpose |
|---|---|---|
| APM | DataDog | Performance monitoring |
| Error Tracking | Sentry | Error aggregation |
| Logging | CloudWatch/DD | Centralized logs |
| Alerting | PagerDuty | Incident alerting |
| Status Page | Statuspage.io | External status |
| Uptime | Pingdom | Availability monitoring |
| RUM | DataDog RUM | Real user metrics |
| Mobile Analytics | Firebase/Mixpanel | Mobile performance |
Monitoring Philosophy#
- Observe, don't guess - Instrument everything measurable
- User-centric - Focus on metrics that impact user experience
- Actionable alerts - Every alert should require action
- Low noise - Reduce alert fatigue through tuning
- Correlated data - Link metrics, logs, and traces
Key Metrics#
Service Level Indicators (SLIs)#
| SLI | Target | Measurement |
|---|---|---|
| Availability | 99.9% | Successful requests / Total |
| Latency (p50) | < 200ms | 50th percentile response time |
| Latency (p99) | < 1000ms | 99th percentile response time |
| Error Rate | < 0.1% | 5xx responses / Total requests |
| Successful Playback | > 99% | Completed plays / Started plays |
| Auth Success Rate | > 99.5% | Successful logins / Login attempts |
Service Level Objectives (SLOs)#
yaml
slos:
api_availability:
target: 99.9%
window: 30d
burn_rate_alert:
fast: 14.4 # 2% budget in 1h
slow: 6 # 5% budget in 6h
api_latency:
target: 99%
threshold: 500ms
window: 30d
meditation_playback:
target: 99%
window: 7d
Business Metrics#
| Metric | Description |
|---|---|
| Daily Active Users (DAU) | Unique users with activity per day |
| Monthly Active Users (MAU) | Unique users with activity per month |
| Meditation Sessions | Total meditation sessions started |
| Meditation Minutes | Total minutes of meditation played |
| Completion Rate | Sessions completed / Sessions started |
| Trial Starts | New trial subscriptions |
| Conversions | Trial to paid conversions |
| Churn Rate | Subscription cancellations |
Infrastructure Metrics#
| Category | Metrics |
|---|---|
| Compute | CPU %, Memory %, Container count |
| Database | Connections, Query time, IOPS, Replication |
| Cache | Hit rate, Memory, Evictions |
| Queue | Depth, Processing time, Failures |
| Network | Requests/sec, Bandwidth, Errors |
| Storage | Space used, IOPS, Latency |
Dashboard Configuration#
Executive Dashboard#
Purpose: High-level health overview for leadership
Refresh: Every 5 minutes
yaml
executive_dashboard:
row_1: # Health Summary
- panel: Service Status
type: status_map
services: [api, web, mobile]
- panel: Error Budget
type: slo_burn
slos: [availability, latency]
row_2: # User Activity
- panel: Active Users (24h)
type: big_number
query: count(distinct user_id) where timestamp > now() - 24h
- panel: Meditation Minutes (24h)
type: big_number
query: sum(duration_minutes) where timestamp > now() - 24h
- panel: New Subscriptions (24h)
type: big_number
query: count(*) from subscriptions where created_at > now() - 24h
row_3: # Trends
- panel: DAU Trend (30d)
type: timeseries
query: count(distinct user_id) by day
- panel: Revenue Trend (30d)
type: timeseries
query: sum(amount) by day from transactions
Engineering Dashboard#
Purpose: Technical health and performance details
Refresh: Every 1 minute
yaml
engineering_dashboard:
row_1: # Real-Time Health
- panel: Request Rate
type: timeseries
query: rate(http_requests_total[5m])
- panel: Error Rate
type: timeseries
query: rate(http_errors_total[5m]) / rate(http_requests_total[5m])
- panel: Latency Distribution
type: heatmap
query: histogram(http_request_duration_seconds)
row_2: # API Performance
- panel: Endpoint Latencies
type: table
query: p50, p95, p99 by endpoint
- panel: Slow Endpoints
type: table
query: endpoints where p99 > 1s
row_3: # Infrastructure
- panel: Container CPU
type: timeseries
query: avg(cpu_usage) by container
- panel: Container Memory
type: timeseries
query: avg(memory_usage) by container
- panel: Database Connections
type: gauge
query: active_connections / max_connections
row_4: # Errors
- panel: Error Distribution
type: pie
query: count by error_type
- panel: Recent Errors
type: log_stream
query: level:error | latest 20
Database Dashboard#
Purpose: PostgreSQL performance monitoring
yaml
database_dashboard:
row_1: # Health
- panel: Connection Pool
type: gauge
query: active / max connections
- panel: Replication Lag
type: timeseries
query: replica_lag_seconds
- panel: Disk Usage
type: gauge
query: disk_used / disk_total
row_2: # Performance
- panel: Query Duration (p95)
type: timeseries
query: percentile(query_duration, 95)
- panel: Queries per Second
type: timeseries
query: rate(queries_total[1m])
- panel: Cache Hit Rate
type: gauge
query: cache_hits / (cache_hits + cache_misses)
row_3: # Slow Queries
- panel: Slow Queries (>1s)
type: log_stream
query: query_duration > 1000ms | latest 10
Mobile Dashboard#
Purpose: Mobile app health and performance
yaml
mobile_dashboard:
row_1: # Crashes
- panel: Crash Rate (iOS)
type: timeseries
query: crashes / sessions (iOS)
- panel: Crash Rate (Android)
type: timeseries
query: crashes / sessions (Android)
- panel: Crash-Free Users
type: big_number
query: (users - crashed_users) / users * 100
row_2: # Performance
- panel: App Launch Time
type: histogram
query: cold_start_duration
- panel: Screen Load Times
type: table
query: avg(screen_load_time) by screen
- panel: Network Errors
type: timeseries
query: rate(network_errors[5m])
row_3: # Media
- panel: Playback Start Time
type: histogram
query: playback_start_latency
- panel: Buffering Events
type: timeseries
query: rate(buffering_events[5m])
- panel: Playback Completion
type: gauge
query: completed_plays / started_plays
Launch Day Dashboard#
Purpose: Critical metrics during launch
yaml
launch_dashboard:
row_1: # Critical Health
- panel: System Status
type: status
color_rules:
green: error_rate < 0.1%
yellow: error_rate < 1%
red: error_rate >= 1%
- panel: Active Users (Real-Time)
type: big_number_live
query: count(distinct user_id) where timestamp > now() - 5m
- panel: Error Rate
type: gauge
thresholds: [0.1%, 0.5%, 1%]
row_2: # Funnel
- panel: Sign-ups (Today)
type: counter
query: count from users where created_at > today()
- panel: First Meditation (Today)
type: counter
query: count from sessions where user.first_session = true
- panel: Trial Starts (Today)
type: counter
query: count from subscriptions where trial_start > today()
row_3: # Real-Time Activity
- panel: Live Request Map
type: geo_map
query: requests by location
- panel: Request Stream
type: live_stream
query: latest 50 requests
row_4: # Deployment Status
- panel: Current Versions
type: table
data: [api_version, web_version, ios_version, android_version]
- panel: Recent Deployments
type: timeline
query: deployments | last 24h
Alert Configuration#
Alert Tiers#
| Tier | Response | Notification | Example |
|---|---|---|---|
| P0 | Immediate | Page + Slack + SMS | Service down |
| P1 | 15 min | Page + Slack | High error rate |
| P2 | 1 hour | Slack | Elevated latency |
| P3 | Business | Slack/Ticket | Warning threshold |
Critical Alerts (P0)#
yaml
alerts:
- name: service_down
condition: uptime_check == 0
duration: 2m
severity: P0
notify: [pagerduty, slack_incidents]
message: 'CRITICAL: {{service}} is DOWN'
- name: database_unavailable
condition: database_connection_errors > 10
duration: 1m
severity: P0
notify: [pagerduty, slack_incidents]
message: 'CRITICAL: Database connection failures'
- name: high_error_rate
condition: error_rate > 5%
duration: 5m
severity: P0
notify: [pagerduty, slack_incidents]
message: 'CRITICAL: Error rate at {{value}}%'
- name: security_breach
condition: suspicious_activity_detected
severity: P0
notify: [pagerduty, slack_security, sms_security_team]
message: 'SECURITY: Suspicious activity detected'
Major Alerts (P1)#
yaml
alerts:
- name: elevated_error_rate
condition: error_rate > 1%
duration: 5m
severity: P1
notify: [pagerduty, slack_incidents]
message: 'HIGH: Error rate elevated to {{value}}%'
- name: latency_degradation
condition: p99_latency > 2000ms
duration: 10m
severity: P1
notify: [pagerduty, slack_incidents]
message: 'HIGH: API latency p99 at {{value}}ms'
- name: auth_failures_spike
condition: auth_failure_rate > 5%
duration: 5m
severity: P1
notify: [pagerduty, slack_incidents]
message: 'HIGH: Authentication failures at {{value}}%'
- name: payment_failures
condition: payment_failure_rate > 10%
duration: 5m
severity: P1
notify: [pagerduty, slack_incidents]
message: 'HIGH: Payment failures at {{value}}%'
- name: playback_failures
condition: playback_failure_rate > 5%
duration: 10m
severity: P1
notify: [pagerduty, slack_incidents]
message: 'HIGH: Media playback failures at {{value}}%'
Warning Alerts (P2/P3)#
yaml
alerts:
- name: elevated_latency
condition: p95_latency > 500ms
duration: 15m
severity: P2
notify: [slack_alerts]
message: 'WARNING: API latency p95 at {{value}}ms'
- name: database_connections_high
condition: db_connections > 80% of max
duration: 10m
severity: P2
notify: [slack_alerts]
message: 'WARNING: DB connections at {{value}}%'
- name: cache_hit_rate_low
condition: cache_hit_rate < 90%
duration: 30m
severity: P2
notify: [slack_alerts]
message: 'WARNING: Cache hit rate at {{value}}%'
- name: disk_space_warning
condition: disk_usage > 80%
duration: 0m
severity: P3
notify: [slack_alerts]
message: 'INFO: Disk usage at {{value}}%'
- name: certificate_expiring
condition: cert_days_remaining < 30
duration: 0m
severity: P3
notify: [slack_alerts]
message: 'INFO: SSL certificate expires in {{value}} days'
Alert Suppression#
yaml
suppression_rules:
- name: maintenance_window
schedule: 'SAT 02:00-06:00 UTC'
suppress: [P2, P3]
- name: deployment_window
condition: deployment_in_progress == true
duration: 15m
suppress: [P2, P3]
- name: dependent_service
condition: parent_service_down == true
suppress: [child_service_alerts]
Log Management#
Log Levels#
| Level | Usage | Retention |
|---|---|---|
| ERROR | Errors requiring attention | 90 days |
| WARN | Potential issues | 30 days |
| INFO | Important events | 14 days |
| DEBUG | Detailed debugging (disabled prod) | 7 days |
Log Format#
json
{
"timestamp": "2026-01-23T10:30:00.123Z",
"level": "INFO",
"service": "tara-api",
"trace_id": "abc123xyz",
"span_id": "def456",
"user_id": "usr_12345",
"message": "Meditation session started",
"metadata": {
"meditation_id": "med_98765",
"duration": 600,
"platform": "ios"
}
}
Log Queries#
bash
# Recent errors
level:error | last 1h
# Errors by endpoint
level:error | stats count by endpoint | sort -count
# Slow requests
request.duration_ms:>1000 | last 1h
# User activity
user_id:"usr_12345" | last 24h
# Failed authentications
endpoint:/auth/login status:401 | last 1h
# Payment failures
service:payments level:error | last 1h
Log-Based Alerts#
yaml
log_alerts:
- name: repeated_errors
query: 'level:error'
threshold: 100 / 5m
notify: slack_alerts
- name: security_events
query: 'tag:security level:warn'
threshold: 10 / 1m
notify: [slack_security, pagerduty]
- name: unusual_activity
query: 'tag:suspicious'
threshold: 1
notify: [slack_security]
Distributed Tracing#
Instrumented Services#
| Service | Framework | Auto-Instrumented |
|---|---|---|
| API | Hono | HTTP, DB, Redis, HTTP clients |
| Web | Next.js | SSR, API routes, fetch |
| Workers | Node.js | Queue processing, DB |
Trace Context#
typescript
// Example trace context propagation
const span = tracer.startSpan('meditation.play', {
attributes: {
'meditation.id': meditationId,
'user.id': userId,
platform: platform,
},
});
// Pass context to downstream services
const headers = {
traceparent: span.spanContext().traceId,
};
await fetch(audioUrl, { headers });
span.end();
Key Traces to Monitor#
| Operation | Expected Latency | Alert Threshold |
|---|---|---|
| Login flow | < 500ms | > 2000ms |
| Meditation list | < 200ms | > 1000ms |
| Meditation play start | < 1000ms | > 3000ms |
| Subscription check | < 100ms | > 500ms |
| Download initiation | < 500ms | > 2000ms |
Real User Monitoring#
Web RUM Configuration#
javascript
// DataDog RUM setup
datadogRum.init({
applicationId: 'tara-web',
clientToken: process.env.DD_CLIENT_TOKEN,
site: 'datadoghq.com',
service: 'tara-web',
env: 'production',
version: process.env.APP_VERSION,
sessionSampleRate: 100,
sessionReplaySampleRate: 20,
trackUserInteractions: true,
trackResources: true,
trackLongTasks: true,
});
Mobile RUM Metrics#
typescript
// Track custom events
RUM.addAction('meditation_started', {
meditation_id: meditation.id,
duration: meditation.duration,
category: meditation.category,
});
// Track timing
RUM.addTiming('playback_start', Date.now() - startTime);
// Track errors
RUM.addError(error, {
context: 'meditation_playback',
meditation_id: meditationId,
});
Core Web Vitals Tracking#
| Metric | Target | Warning | Critical |
|---|---|---|---|
| LCP | < 2.5s | < 4.0s | >= 4.0s |
| FID | < 100ms | < 300ms | >= 300ms |
| CLS | < 0.1 | < 0.25 | >= 0.25 |
| TTFB | < 800ms | < 1.8s | >= 1.8s |
User Journey Tracking#
yaml
user_journeys:
onboarding:
steps:
- app_open
- signup_started
- signup_completed
- first_meditation_started
- first_meditation_completed
target_completion: 50%
subscription:
steps:
- paywall_viewed
- plan_selected
- payment_initiated
- payment_completed
- premium_content_accessed
target_completion: 5%
daily_meditation:
steps:
- app_open
- meditation_selected
- playback_started
- playback_completed
target_completion: 80%
Status Page#
Status Page Configuration#
URL: status.tara.app
Components#
| Component | Description | Affects |
|---|---|---|
| Tara API | Core backend services | All features |
| Web Application | tara.app website | Web users |
| iOS App | iOS application | iOS users |
| Android App | Android application | Android users |
| Authentication | Login and account services | All users |
| Meditation Streaming | Audio content delivery | Playback |
| Payments | Subscription and billing | Purchases |
| Push Notifications | Notification delivery | Reminders |
Status Levels#
| Level | Icon | Description |
|---|---|---|
| Operational | ✓ | All systems functioning normally |
| Degraded | ⚠ | Partial functionality, slower than normal |
| Partial Outage | ⚡ | Some features unavailable |
| Major Outage | ✗ | Service significantly impaired |
Automated Status Updates#
yaml
status_automation:
rules:
- condition: api_error_rate > 5%
status: Major Outage
component: Tara API
message: 'We are experiencing elevated error rates'
- condition: api_error_rate > 1%
status: Degraded
component: Tara API
message: 'Some users may experience errors'
- condition: api_latency_p99 > 3000ms
status: Degraded
component: Tara API
message: 'Services are slower than normal'
- condition: cdn_availability < 99%
status: Partial Outage
component: Meditation Streaming
message: 'Some meditation content may be unavailable'
Incident Communication Templates#
See Incident Response - Communication Protocols
Quick Reference#
Dashboard URLs#
| Dashboard | URL |
|---|---|
| Executive | [Internal DataDog URL] |
| Engineering | [Internal DataDog URL] |
| Database | [Internal DataDog URL] |
| Mobile | [Internal DataDog URL] |
| Launch Day | [Internal DataDog URL] |
| Errors | [Internal Sentry URL] |
| Status Page | status.tara.app/admin |
Alert Channels#
| Channel | Purpose |
|---|---|
| #alerts | All automated alerts |
| #incidents | Active incident discussion |
| #security | Security alerts |
| PagerDuty | On-call paging |
On-Call Information#
- Schedule: [PagerDuty URL]
- Escalation: See Incident Response
- Runbooks: [Confluence/Notion URL]
Last Updated: January 2026
For questions about monitoring, contact ops@tara.app