Disciplines · Reference

Cost Management Runbook

Fargate Spot provides up to 70% discount for interruptible workloads.

7sections2 minread
Markdown source
On this page

This runbook provides guidance for monitoring, optimizing, and managing cloud costs for the Oshun infrastructure.

Cost Overview#

Monthly Cost Breakdown (Estimated)#

Service Staging Production % of Total
ECS (Fargate) $200-400 $800-1500 25-30%
RDS (PostgreSQL) $100-200 $400-800 15-20%
RunPod (GPU) $100-300 $500-2000 20-40%
S3 Storage $20-50 $50-150 2-5%
CloudWatch $50-100 $100-300 5-8%
ALB $20-40 $40-100 2-4%
Data Transfer $50-100 $200-500 5-10%
Other $50-100 $100-300 5-10%
Total $600-1300 $2200-5600

Monitoring Costs#

AWS Cost Explorer#

Monthly cost by service:

bash
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of this month' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.BlendedCost.Amount}' \
  --output table

Daily cost trend:

bash
aws ce get-cost-and-usage \
  --time-period Start=$(date -d '7 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity DAILY \
  --metrics BlendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.BlendedCost.Amount}' \
  --output table

Cost by tag (environment):

bash
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of this month' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=TAG,Key=Environment \
  --query 'ResultsByTime[0].Groups[*].{Environment:Keys[0],Cost:Metrics.BlendedCost.Amount}'

AWS Budgets#

Create monthly budget alert:

bash
aws budgets create-budget \
  --account-id $(aws sts get-caller-identity --query Account --output text) \
  --budget '{
    "BudgetName": "oshun-monthly",
    "BudgetLimit": {"Amount": "5000", "Unit": "USD"},
    "BudgetType": "COST",
    "TimeUnit": "MONTHLY"
  }' \
  --notifications-with-subscribers '[
    {
      "Notification": {
        "NotificationType": "ACTUAL",
        "ComparisonOperator": "GREATER_THAN",
        "Threshold": 80,
        "ThresholdType": "PERCENTAGE"
      },
      "Subscribers": [
        {"SubscriptionType": "EMAIL", "Address": "ops@oshun.ai"}
      ]
    }
  ]'

RunPod Costs#

Check RunPod spending:

bash
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { spending { totalSpend currentMonthSpend } } }"}'

Cost Optimization Strategies#

ECS Optimization#

Use Fargate Spot (Staging)#

Fargate Spot provides up to 70% discount for interruptible workloads.

hcl
# In Terraform
capacity_provider_strategy {
  capacity_provider = "FARGATE_SPOT"
  weight            = 4
  base              = 0
}

capacity_provider_strategy {
  capacity_provider = "FARGATE"
  weight            = 1
  base              = 1  # At least 1 on-demand task
}

Right-Size Tasks#

Analyze CPU/memory utilization:

bash
# Average CPU utilization over 7 days
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --dimensions Name=ClusterName,Value=oshun-production Name=ServiceName,Value=oshun-api \
  --start-time $(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 86400 \
  --statistics Average Maximum p99

# If average CPU < 30%, consider reducing task size
# If max CPU > 80%, consider increasing

Task sizing recommendations:

Avg CPU Avg Memory Recommendation
< 20% < 40% Reduce by 50%
20-40% 40-60% May reduce by 25%
40-60% 60-80% Well-sized
> 70% > 80% Increase or scale out

Scheduled Scaling#

Reduce costs during off-peak hours:

bash
# Scale down at night (10 PM UTC)
aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --scheduled-action-name night-scale-down \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --schedule "cron(0 22 ? * * *)" \
  --scalable-target-action MinCapacity=1,MaxCapacity=4

# Scale up in morning (8 AM UTC)
aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --scheduled-action-name morning-scale-up \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --schedule "cron(0 8 ? * MON-FRI *)" \
  --scalable-target-action MinCapacity=2,MaxCapacity=10

RDS Optimization#

Reserved Instances#

For predictable database workloads, RIs save 30-60%:

bash
# Check current on-demand usage
aws ce get-reservation-coverage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Relational Database Service"]}}'

# Purchase RI via console or CLI
aws rds purchase-reserved-db-instances-offering \
  --reserved-db-instances-offering-id <offering-id> \
  --reserved-db-instance-id oshun-production-ri

Storage Optimization#

Enable storage autoscaling to avoid over-provisioning:

bash
aws rds modify-db-instance \
  --db-instance-identifier oshun-production \
  --max-allocated-storage 500  # GB

Delete old snapshots:

bash
# List snapshots older than 30 days
aws rds describe-db-snapshots \
  --db-instance-identifier oshun-production \
  --query "DBSnapshots[?SnapshotCreateTime<='$(date -d '30 days ago' +%Y-%m-%d)'].DBSnapshotIdentifier"

# Delete old manual snapshots (automated retained per policy)
aws rds delete-db-snapshot --db-snapshot-identifier <snapshot-id>

S3 Optimization#

Lifecycle Policies#

Move infrequently accessed data to cheaper storage:

bash
aws s3api put-bucket-lifecycle-configuration \
  --bucket oshun-production-assets \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "MoveToIA",
        "Status": "Enabled",
        "Filter": {"Prefix": "exports/"},
        "Transitions": [
          {"Days": 30, "StorageClass": "STANDARD_IA"},
          {"Days": 90, "StorageClass": "GLACIER"}
        ]
      },
      {
        "ID": "DeleteTemp",
        "Status": "Enabled",
        "Filter": {"Prefix": "temp/"},
        "Expiration": {"Days": 7}
      }
    ]
  }'

Intelligent-Tiering#

For unpredictable access patterns:

bash
# Enable Intelligent-Tiering by default
aws s3api put-bucket-intelligent-tiering-configuration \
  --bucket oshun-production-assets \
  --id entire-bucket \
  --intelligent-tiering-configuration '{
    "Id": "entire-bucket",
    "Status": "Enabled",
    "Tierings": [
      {"Days": 90, "AccessTier": "ARCHIVE_ACCESS"},
      {"Days": 180, "AccessTier": "DEEP_ARCHIVE_ACCESS"}
    ]
  }'

RunPod Optimization#

Idle Timeout Tuning#

Traffic Pattern Recommended Timeout Impact
Continuous 120-300s Low cold starts
Bursty 60-120s Balance
Infrequent 30-60s Lower cost
Very Infrequent 0-30s Lowest cost
bash
# Reduce idle timeout for low-traffic endpoint
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{
    "query": "mutation { updateEndpoint(input: { id: \"ENDPOINT_ID\", idleTimeout: 30 }) { id idleTimeout } }"
  }'

GPU Selection#

Use Case Recommended GPU Price/Hour Notes
SD1.5 RTX 4090 $0.50 Fast, cheap
SDXL A40 $0.86 Good balance
Flux A100 40GB $1.58 Required for Flux
Training/Fine-tune A100 80GB $2.00 Max VRAM

CloudWatch Optimization#

Reduce Log Retention#

bash
# Set 14-day retention for staging
aws logs put-retention-policy \
  --log-group-name /ecs/oshun-staging/api \
  --retention-in-days 14

# Set 30-day retention for less critical production logs
aws logs put-retention-policy \
  --log-group-name /ecs/oshun-production/api \
  --retention-in-days 30

Optimize Metrics Collection#

Disable high-cardinality metrics if not needed:

bash
# Review custom metrics
aws cloudwatch list-metrics --namespace Oshun/Custom

# Delete unused custom metrics (stop publishing)

Data Transfer Optimization#

Use VPC endpoints to avoid NAT costs:

bash
# S3 Gateway Endpoint (free)
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-xxx

# ECR, CloudWatch Interface Endpoints
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-xxx

Cost Alerts#

Setting Up Alerts#

80% budget threshold:

bash
aws cloudwatch put-metric-alarm \
  --alarm-name oshun-cost-warning \
  --alarm-description "AWS costs exceeding 80% of budget" \
  --metric-name EstimatedCharges \
  --namespace AWS/Billing \
  --statistic Maximum \
  --period 21600 \
  --threshold 4000 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=Currency,Value=USD \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789:billing-alerts

Anomaly Detection#

bash
aws cloudwatch put-anomaly-detector \
  --namespace AWS/Billing \
  --metric-name EstimatedCharges \
  --stat Maximum \
  --dimensions Name=Currency,Value=USD

Monthly Cost Review#

Review Checklist#

  • Review AWS Cost Explorer for unexpected spikes
  • Check RunPod spending dashboard
  • Review underutilized resources
  • Check for orphaned resources (unused EBS, old snapshots)
  • Verify Reserved Instance coverage
  • Review data transfer costs
  • Update budget forecasts if needed

Cost Reporting#

Generate monthly report:

bash
#!/bin/bash
# Monthly cost report script

echo "=== Oshun Monthly Cost Report ==="
echo "Period: $(date -d 'first day of last month' +%B\ %Y)"
echo ""

echo "=== AWS Costs by Service ==="
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.BlendedCost.Amount}' \
  --output table

echo ""
echo "=== Costs by Environment ==="
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=TAG,Key=Environment \
  --query 'ResultsByTime[0].Groups[*].{Environment:Keys[0],Cost:Metrics.BlendedCost.Amount}' \
  --output table

echo ""
echo "=== Total ==="
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --query 'ResultsByTime[0].Total.BlendedCost.Amount'

Emergency Cost Reduction#

If costs are spiking unexpectedly:

1. Immediate Actions#

bash
# Scale down non-critical services
aws ecs update-service --cluster oshun-staging --service oshun-api --desired-count 0

# Reduce RunPod max workers
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "mutation { updateEndpoint(input: { id: \"ENDPOINT_ID\", workersMax: 1 }) { id } }"}'

2. Identify Cost Source#

bash
# Check for cost spikes by service
aws ce get-cost-and-usage \
  --time-period Start=$(date -d '3 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity DAILY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --output table

3. Common Causes#

Symptom Likely Cause Fix
ECS costs doubled Auto-scaling spike Check scaling policies
Data transfer spike Large file uploads/downloads Review CloudFront/CDN
RunPod costs high Workers staying alive Reduce idle timeout
S3 costs increased Lifecycle not applied Check lifecycle policies
RDS costs increased Storage autoscale Review storage usage