This Terraform module creates SNS topics for event fan-out across all Oshun domains, with subscriptions to SQS queues, Lambda functions, and HTTP endpoints.
Features#
- Standard Topics: High-throughput topics for event distribution
- FIFO Topics: Ordered message delivery with deduplication
- Multiple Subscriptions: SQS, Lambda, HTTP/HTTPS endpoints
- Filter Policies: Route messages based on attributes
- Encryption: Optional KMS encryption for topics
- CloudWatch Alarms: Failed delivery monitoring
- CloudWatch Dashboard: Visual monitoring for all topics
Usage#
hcl
module "sns" {
source = "../sns"
environment = "production"
region = "us-east-1"
name_prefix = "oshun"
# Optional: KMS encryption
# kms_key_id = module.kms.sns_key_id
# Add SQS subscriptions
sqs_subscriptions = {
generation-to-output = {
topic_key = "isis-generation-completed"
queue_arn = module.sqs.queue_arns["isis-output"]
raw_message_delivery = true
}
}
# Add Lambda subscriptions
lambda_subscriptions = {
notify-on-build = {
topic_key = "bellona-build-completed"
function_arn = module.lambda.notify_function_arn
}
}
}
Default Topics#
Isis Domain#
isis-generation-completed: Generation job completedisis-generation-failed: Generation job failedisis-workflow-state-changed: Workflow state change
Sophia Domain#
sophia-document-ingested: Document ingestedsophia-embedding-completed: Embeddings generated
Hathor Domain#
hathor-world-updated: World modificationhathor-simulation-tick: Simulation tick event
Bellona Domain#
bellona-build-completed: Build completedbellona-build-failed: Build failedbellona-render-completed: Rendering completed
Yemaya Domain#
yemaya-project-saved: Project savedyemaya-export-completed: Export completedyemaya-collaboration-event: Collaboration events
Lilith Domain#
lilith-user-activity: User activity eventslilith-content-updated: Content updateslilith-meditation-completed: Meditation session completedlilith-notification-request: Notification requests
Topic Configuration Options#
hcl
topics = {
my-topic = {
# Required
domain = "isis"
description = "My custom topic"
# Optional (with defaults)
enabled = true
fifo_topic = false
content_based_deduplication = false # For FIFO only
# Encryption
kms_key_id = "alias/my-key"
# Tags
tags = {
Custom = "value"
}
}
}
Subscription Types#
SQS Subscriptions#
hcl
sqs_subscriptions = {
my-subscription = {
topic_key = "isis-generation-completed"
queue_arn = "arn:aws:sqs:us-east-1:123456789:my-queue"
raw_message_delivery = true # Deliver message body directly
# Optional: Filter messages
filter_policy = jsonencode({
status = ["success"]
})
}
}
Lambda Subscriptions#
hcl
lambda_subscriptions = {
my-handler = {
topic_key = "bellona-build-completed"
function_arn = "arn:aws:lambda:us-east-1:123456789:function:my-handler"
# Optional: Filter messages
filter_policy = jsonencode({
buildType = ["production"]
})
}
}
HTTP/HTTPS Subscriptions#
hcl
http_subscriptions = {
webhook = {
topic_key = "yemaya-export-completed"
endpoint = "https://api.example.com/webhooks/export"
raw_message_delivery = false
}
}
Filter Policies#
Route messages to specific subscribers based on message attributes:
hcl
sqs_subscriptions = {
high-priority = {
topic_key = "isis-generation-completed"
queue_arn = module.sqs.queue_arns["priority-queue"]
filter_policy = jsonencode({
priority = ["high", "critical"]
})
}
low-priority = {
topic_key = "isis-generation-completed"
queue_arn = module.sqs.queue_arns["standard-queue"]
filter_policy = jsonencode({
priority = ["low", "normal"]
})
}
}
Outputs#
| Name | Description |
|---|---|
| topic_arns | Map of topic names to ARNs |
| topic_names | Map of topic keys to AWS names |
| topics_by_domain | Topics grouped by domain |
| ecs_topic_environment | Environment variables for ECS |
| publisher_policy_json | IAM policy for publishing |
| domain_publisher_policies | Per-domain IAM policies |
| sqs_subscription_arns | SQS subscription ARNs |
| lambda_subscription_arns | Lambda subscription ARNs |
| topics_summary | Summary of deployed topics |
Using with ECS Services#
The module outputs environment variables formatted for ECS:
hcl
module "ecs_services" {
# ...
services = {
isis-generation-api = {
environment = [
for k, v in module.sns.ecs_topic_environment : {
name = v.name
value = v.value
} if startswith(k, "ISIS")
]
}
}
}
IAM Policies#
Attach the publisher policy to ECS task roles:
hcl
resource "aws_iam_role_policy" "service_sns" {
name = "sns-publish"
role = aws_iam_role.service_task.id
policy = module.sns.domain_publisher_policies["isis"]
}
Event-Driven Architecture#
Example fan-out pattern:
text
┌─────────────────┐
│ ISIS API │
│ (Publisher) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ SNS Topic │
│ generation- │
│ completed │
└────────┬────────┘
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ SQS Queue │ │ Lambda │ │ HTTP │
│ (Output │ │ (Analytics │ │ (Webhook) │
│ Processing)│ │ Handler) │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
Best Practices#
- Topic Per Event Type: Create separate topics for different event types
- Filter Policies: Use filters to route to appropriate subscribers
- Raw Delivery: Enable for SQS to avoid double-encoding
- Dead Letter Queues: Configure on SQS subscribers for failed messages
- Idempotency: Design subscribers to handle duplicate messages
- Message Attributes: Add attributes for filtering and routing