Network infrastructure design and configuration for the Lilith platform.
Table of Contents#
- Overview
- VPC Architecture
- Subnet Design
- Security Groups
- Network Policies
- DNS Configuration
- Load Balancing
- VPN & Direct Connect
Overview#
The Lilith platform uses a multi-tier VPC architecture designed for:
- Security: Network isolation between tiers
- High Availability: Multi-AZ deployment
- Scalability: Flexible CIDR allocation
- Observability: VPC Flow Logs for all traffic
Network Diagram#
text
Internet
│
┌────────▼────────┐
│ Internet GW │
└────────┬────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ Public Sub │ │ Public Sub │ │ Public Sub │
│ 10.0.0.0/24 │ │ 10.0.1.0/24 │ │ 10.0.2.0/24 │
│ (AZ-a) │ │ (AZ-b) │ │ (AZ-c) │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ NAT GW-a │ │ NAT GW-b │ │ NAT GW-c │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ Private Sub │ │ Private Sub │ │ Private Sub │
│ 10.0.10.0/24 │ │ 10.0.11.0/24 │ │ 10.0.12.0/24 │
│ (EKS Nodes) │ │ (EKS Nodes) │ │ (EKS Nodes) │
└───────────────┘ └───────────────┘ └───────────────┘
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ Database Sub │ │ Database Sub │ │ Database Sub │
│ 10.0.20.0/24 │ │ 10.0.21.0/24 │ │ 10.0.22.0/24 │
│ (RDS/DB) │ │ (RDS/DB) │ │ (RDS/DB) │
└───────────────┘ └───────────────┘ └───────────────┘
VPC Architecture#
CIDR Planning#
| Environment | VPC CIDR | Region |
|---|---|---|
| Development | 10.0.0.0/16 | us-east-1 |
| Staging | 10.1.0.0/16 | us-east-1 |
| Production (Primary) | 10.2.0.0/16 | us-east-1 |
| Production (DR) | 10.3.0.0/16 | eu-west-1 |
| Production (APAC) | 10.4.0.0/16 | ap-southeast-1 |
Terraform Configuration#
hcl
# modules/networking/main.tf
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "${var.environment}-lilith-vpc"
Environment = var.environment
Project = "lilith"
}
}
# Internet Gateway
resource "aws_internet_gateway" "main" {
vpc_id = aws_vpc.main.id
tags = {
Name = "${var.environment}-lilith-igw"
Environment = var.environment
}
}
VPC Features Enabled#
| Feature | Status | Purpose |
|---|---|---|
| DNS Hostnames | Enabled | Service discovery |
| DNS Support | Enabled | Internal DNS resolution |
| Flow Logs | Enabled | Network monitoring |
| IPv6 | Disabled | Not currently required |
Subnet Design#
Subnet Allocation#
| Subnet Type | CIDR Offset | Purpose | Internet Access |
|---|---|---|---|
| Public | /24 (0-2) | Load balancers, NAT GW | Direct (IGW) |
| Private | /24 (10-12) | EKS nodes, application pods | Via NAT GW |
| Database | /24 (20-22) | RDS, ElastiCache | None |
Terraform Configuration#
hcl
# Public Subnets
resource "aws_subnet" "public" {
count = length(var.availability_zones)
vpc_id = aws_vpc.main.id
cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index)
availability_zone = var.availability_zones[count.index]
map_public_ip_on_launch = true
tags = {
Name = "${var.environment}-lilith-public-${count.index + 1}"
Environment = var.environment
Type = "public"
"kubernetes.io/role/elb" = "1" # For AWS LB Controller
}
}
# Private Subnets
resource "aws_subnet" "private" {
count = length(var.availability_zones)
vpc_id = aws_vpc.main.id
cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index + 10)
availability_zone = var.availability_zones[count.index]
tags = {
Name = "${var.environment}-lilith-private-${count.index + 1}"
Environment = var.environment
Type = "private"
"kubernetes.io/role/internal-elb" = "1" # For internal LBs
}
}
# Database Subnets
resource "aws_subnet" "database" {
count = length(var.availability_zones)
vpc_id = aws_vpc.main.id
cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index + 20)
availability_zone = var.availability_zones[count.index]
tags = {
Name = "${var.environment}-lilith-database-${count.index + 1}"
Environment = var.environment
Type = "database"
}
}
NAT Gateway Configuration#
hcl
# Elastic IPs for NAT Gateways
resource "aws_eip" "nat" {
count = var.single_nat_gateway ? 1 : length(var.availability_zones)
domain = "vpc"
tags = {
Name = "${var.environment}-lilith-nat-eip-${count.index + 1}"
Environment = var.environment
}
depends_on = [aws_internet_gateway.main]
}
# NAT Gateways
resource "aws_nat_gateway" "main" {
count = var.single_nat_gateway ? 1 : length(var.availability_zones)
allocation_id = aws_eip.nat[count.index].id
subnet_id = aws_subnet.public[count.index].id
tags = {
Name = "${var.environment}-lilith-nat-${count.index + 1}"
Environment = var.environment
}
depends_on = [aws_internet_gateway.main]
}
Route Tables#
hcl
# Public Route Table
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.main.id
}
tags = {
Name = "${var.environment}-lilith-public-rt"
Environment = var.environment
}
}
# Private Route Tables (one per AZ for HA NAT)
resource "aws_route_table" "private" {
count = length(var.availability_zones)
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = var.single_nat_gateway ? aws_nat_gateway.main[0].id : aws_nat_gateway.main[count.index].id
}
tags = {
Name = "${var.environment}-lilith-private-rt-${count.index + 1}"
Environment = var.environment
}
}
Security Groups#
EKS Cluster Security Group#
hcl
resource "aws_security_group" "eks_cluster" {
name_prefix = "${var.environment}-lilith-eks-cluster"
vpc_id = aws_vpc.main.id
# Allow inbound from VPC for API server
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = [var.vpc_cidr]
description = "Kubernetes API server"
}
# Allow all outbound
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "${var.environment}-lilith-eks-cluster-sg"
Environment = var.environment
}
}
EKS Node Security Group#
hcl
resource "aws_security_group" "eks_nodes" {
name_prefix = "${var.environment}-lilith-eks-nodes"
vpc_id = aws_vpc.main.id
# Node-to-node communication
ingress {
from_port = 0
to_port = 65535
protocol = "tcp"
self = true
description = "Node to node"
}
# Control plane to nodes
ingress {
from_port = 1025
to_port = 65535
protocol = "tcp"
security_groups = [aws_security_group.eks_cluster.id]
description = "Control plane to nodes"
}
# Kubelet API
ingress {
from_port = 10250
to_port = 10250
protocol = "tcp"
security_groups = [aws_security_group.eks_cluster.id]
description = "Kubelet API"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "${var.environment}-lilith-eks-nodes-sg"
Environment = var.environment
}
}
Database Security Group#
hcl
resource "aws_security_group" "database" {
name_prefix = "${var.environment}-lilith-database"
vpc_id = aws_vpc.main.id
# PostgreSQL from EKS nodes only
ingress {
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [aws_security_group.eks_nodes.id]
description = "PostgreSQL from EKS"
}
# Redis from EKS nodes only
ingress {
from_port = 6379
to_port = 6379
protocol = "tcp"
security_groups = [aws_security_group.eks_nodes.id]
description = "Redis from EKS"
}
tags = {
Name = "${var.environment}-lilith-database-sg"
Environment = var.environment
}
}
Network Policies#
Kubernetes Network Policies#
yaml
# Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: lilith
spec:
podSelector: {}
policyTypes:
- Ingress
---
# Allow ingress from same namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
namespace: lilith
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {}
---
# Allow ingress from ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-controller
namespace: lilith
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress
Service-Specific Policies#
yaml
# BFF can access all backend services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: bff-egress
namespace: lilith
spec:
podSelector:
matchLabels:
app: bff
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
tier: backend
---
# AI service can access external APIs
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ai-egress
namespace: lilith
spec:
podSelector:
matchLabels:
app: ai
policyTypes:
- Egress
egress:
- to: [] # Allow all egress for external AI APIs
DNS Configuration#
Route53 Configuration#
hcl
# Public hosted zone
resource "aws_route53_zone" "public" {
name = var.domain_name
tags = {
Environment = var.environment
Project = "lilith"
}
}
# Private hosted zone for internal services
resource "aws_route53_zone" "private" {
name = "internal.${var.domain_name}"
vpc {
vpc_id = module.networking.vpc_id
}
tags = {
Environment = var.environment
Project = "lilith"
}
}
External DNS Configuration#
yaml
# External DNS deployment for automatic DNS management
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/lilith-external-dns-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: registry.k8s.io/external-dns/external-dns:v0.14.0
args:
- --source=service
- --source=ingress
- --domain-filter=lilith.app
- --provider=aws
- --policy=sync
- --aws-zone-type=public
- --registry=txt
- --txt-owner-id=lilith-eks
Load Balancing#
AWS Load Balancer Controller#
yaml
# Ingress with ALB
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lilith-ingress
namespace: lilith
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT_ID:certificate/CERT_ID
alb.ingress.kubernetes.io/ssl-redirect: '443'
alb.ingress.kubernetes.io/healthcheck-path: /health
spec:
rules:
- host: api.lilith.app
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: bff
port:
number: 3000
Internal Load Balancer#
yaml
# Internal NLB for service-to-service
apiVersion: v1
kind: Service
metadata:
name: internal-api
namespace: lilith
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: 'true'
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
spec:
type: LoadBalancer
ports:
- port: 443
targetPort: 3000
selector:
app: bff
VPN & Direct Connect#
Site-to-Site VPN#
hcl
# Customer Gateway
resource "aws_customer_gateway" "main" {
bgp_asn = 65000
ip_address = var.vpn_peer_ip
type = "ipsec.1"
tags = {
Name = "${var.environment}-lilith-cgw"
Environment = var.environment
}
}
# VPN Gateway
resource "aws_vpn_gateway" "main" {
vpc_id = aws_vpc.main.id
tags = {
Name = "${var.environment}-lilith-vgw"
Environment = var.environment
}
}
# VPN Connection
resource "aws_vpn_connection" "main" {
vpn_gateway_id = aws_vpn_gateway.main.id
customer_gateway_id = aws_customer_gateway.main.id
type = "ipsec.1"
static_routes_only = true
tags = {
Name = "${var.environment}-lilith-vpn"
Environment = var.environment
}
}
VPC Flow Logs#
hcl
resource "aws_flow_log" "main" {
vpc_id = aws_vpc.main.id
traffic_type = "ALL"
iam_role_arn = aws_iam_role.flow_log.arn
log_destination = aws_cloudwatch_log_group.flow_log.arn
tags = {
Name = "${var.environment}-lilith-flow-logs"
Environment = var.environment
}
}
resource "aws_cloudwatch_log_group" "flow_log" {
name = "/aws/vpc/flow-logs/${var.environment}-lilith"
retention_in_days = 30
}
Monitoring & Troubleshooting#
VPC Reachability Analyzer#
bash
# Check connectivity between resources
aws ec2 create-network-insights-path \
--source i-source-instance-id \
--destination i-destination-instance-id \
--protocol TCP \
--destination-port 5432
aws ec2 start-network-insights-analysis \
--network-insights-path-id nip-path-id
Common Issues#
| Issue | Symptom | Solution |
|---|---|---|
| NAT Gateway limit | Connection timeouts | Scale to multiple NAT GWs |
| Security Group | Connection refused | Check ingress/egress rules |
| Route table | No route to host | Verify route associations |
| DNS resolution | Unknown host | Check DNS settings, VPC DHCP |