Comprehensive guide for containerization and container registry management in the Lilith platform.
Table of Contents#
- Overview
- Base Images
- Dockerfile Standards
- Multi-Stage Builds
- Image Registry (ECR)
- Image Scanning
- Tagging Strategy
- Build Pipeline
- Best Practices
Overview#
The Lilith platform uses containerized microservices deployed to Amazon EKS. All services follow consistent containerization patterns for security, efficiency, and maintainability.
Container Architecture#
┌─────────────────────────────────────────────────────────────────────┐
│ Container Registry (ECR) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ lilith/auth │ │ lilith/bff │ │ lilith/ai │ ... │
│ │ :v1.2.3 │ │ :v1.2.3 │ │ :v1.2.3 │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
┌─────────▼─────────┐
│ EKS Cluster │
│ ┌─────┐ ┌─────┐ │
│ │ Pod │ │ Pod │ │
│ └─────┘ └─────┘ │
└───────────────────┘
Key Principles#
| Principle | Implementation |
|---|---|
| Security | Non-root user, minimal base image, vulnerability scanning |
| Efficiency | Multi-stage builds, layer caching, small image size |
| Reliability | Health checks, proper signal handling, graceful shutdown |
| Consistency | Standardized Dockerfile template across all services |
Base Images#
Standard Base Image#
All Node.js services use node:20-alpine as the base image:
FROM node:20-alpine
Why Alpine?#
| Feature | Alpine | Debian-based |
|---|---|---|
| Base size | ~5MB | ~120MB |
| Final image | ~150MB | ~400MB+ |
| Attack surface | Minimal | Larger |
| Package manager | apk | apt |
Required System Packages#
# Runtime dependencies
RUN apk add --no-cache dumb-init
# Build dependencies (builder stage only)
RUN apk add --no-cache python3 make g++
Dockerfile Standards#
Standard Template#
All services use this standardized Dockerfile structure:
# Multi-stage build for production-ready service
FROM node:20-alpine AS builder
# Install build dependencies
RUN apk add --no-cache python3 make g++
# Set working directory
WORKDIR /app
# Copy package files for layer caching
COPY services/SERVICE_NAME/package*.json ./services/SERVICE_NAME/
COPY services/lib/package*.json ./services/lib/
COPY services/common/package*.json ./services/common/
# Install dependencies
WORKDIR /app/services/SERVICE_NAME
RUN npm ci --only=production || npm install --only=production
# Production stage
FROM node:20-alpine
# Install dumb-init for signal handling
RUN apk add --no-cache dumb-init
# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
adduser -S nodejs -u 1001
WORKDIR /app
# Copy from builder
COPY --from=builder --chown=nodejs:nodejs /app/services/SERVICE_NAME/node_modules ./services/SERVICE_NAME/node_modules
# Copy source
COPY --chown=nodejs:nodejs services/SERVICE_NAME ./services/SERVICE_NAME
COPY --chown=nodejs:nodejs services/lib ./services/lib
COPY --chown=nodejs:nodejs services/common ./services/common
# Switch to non-root
USER nodejs
# Environment
ENV NODE_ENV=production
ENV SERVICE_NAME=SERVICE_NAME
ENV PORT=8080
EXPOSE 8080
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD node -e "require('http').get('http://localhost:8080/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"
WORKDIR /app/services/SERVICE_NAME
# Use dumb-init for proper signal handling
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "src/server.js"]
Required Elements#
| Element | Purpose | Required |
|---|---|---|
| Multi-stage build | Reduce image size | Yes |
| Non-root user | Security | Yes |
| dumb-init | Signal handling | Yes |
| HEALTHCHECK | Container health | Yes |
| Layer caching | Build efficiency | Yes |
Multi-Stage Builds#
Build Stage#
The builder stage includes all build tools and dependencies:
FROM node:20-alpine AS builder
# Build dependencies needed for native modules
RUN apk add --no-cache python3 make g++
# Install all dependencies including devDependencies if needed
RUN npm ci
Production Stage#
The production stage contains only runtime requirements:
FROM node:20-alpine
# Only runtime dependencies
RUN apk add --no-cache dumb-init
# Copy only production node_modules
COPY --from=builder /app/node_modules ./node_modules
# Copy application code
COPY --chown=nodejs:nodejs . .
Image Size Comparison#
| Build Type | Image Size |
|---|---|
| Single-stage (dev deps) | ~800MB |
| Multi-stage (prod only) | ~150MB |
| With build caching | ~150MB (faster builds) |
Image Registry (ECR)#
Repository Structure#
AWS ECR
├── lilith/auth
├── lilith/bff
├── lilith/content
├── lilith/conversation
├── lilith/media
├── lilith/ai
├── lilith/notification
├── lilith/moderation
└── lilith/base (shared base images)
Terraform Configuration#
resource "aws_ecr_repository" "services" {
for_each = toset([
"auth", "bff", "content", "conversation",
"media", "ai", "notification", "moderation"
])
name = "lilith/${each.key}"
image_tag_mutability = "IMMUTABLE"
image_scanning_configuration {
scan_on_push = true
}
encryption_configuration {
encryption_type = "KMS"
kms_key = aws_kms_key.ecr.arn
}
tags = {
Service = each.key
Environment = var.environment
Project = "lilith"
}
}
# Lifecycle policy to manage image retention
resource "aws_ecr_lifecycle_policy" "services" {
for_each = aws_ecr_repository.services
repository = each.value.name
policy = jsonencode({
rules = [
{
rulePriority = 1
description = "Keep last 30 production images"
selection = {
tagStatus = "tagged"
tagPrefixList = ["v"]
countType = "imageCountMoreThan"
countNumber = 30
}
action = {
type = "expire"
}
},
{
rulePriority = 2
description = "Keep last 10 dev images"
selection = {
tagStatus = "tagged"
tagPrefixList = ["dev-", "pr-"]
countType = "imageCountMoreThan"
countNumber = 10
}
action = {
type = "expire"
}
},
{
rulePriority = 3
description = "Delete untagged images after 7 days"
selection = {
tagStatus = "untagged"
countType = "sinceImagePushed"
countUnit = "days"
countNumber = 7
}
action = {
type = "expire"
}
}
]
})
}
Cross-Account Access#
For multi-account setups:
resource "aws_ecr_repository_policy" "cross_account" {
repository = aws_ecr_repository.services["auth"].name
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "AllowPull"
Effect = "Allow"
Principal = {
AWS = [
"arn:aws:iam::${var.staging_account_id}:root",
"arn:aws:iam::${var.prod_account_id}:root"
]
}
Action = [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}
]
})
}
Image Scanning#
Automated Scanning#
ECR automatically scans images on push:
image_scanning_configuration {
scan_on_push = true
}
Scanning Results#
Check scan results via CLI:
# Get scan findings
aws ecr describe-image-scan-findings \
--repository-name lilith/auth \
--image-id imageTag=v1.2.3
# List images with vulnerabilities
aws ecr describe-images \
--repository-name lilith/auth \
--query 'imageDetails[?imageScanStatus.status==`COMPLETE`].{Tag:imageTags[0],Critical:imageScanFindingsSummary.findingSeverityCounts.CRITICAL,High:imageScanFindingsSummary.findingSeverityCounts.HIGH}'
CI/CD Integration#
# .github/workflows/build.yml
- name: Scan image
run: |
# Wait for scan to complete
aws ecr wait image-scan-complete \
--repository-name lilith/${{ matrix.service }} \
--image-id imageTag=${{ github.sha }}
# Get findings
CRITICAL=$(aws ecr describe-image-scan-findings \
--repository-name lilith/${{ matrix.service }} \
--image-id imageTag=${{ github.sha }} \
--query 'imageScanFindings.findingSeverityCounts.CRITICAL' \
--output text)
# Fail if critical vulnerabilities
if [ "$CRITICAL" != "None" ] && [ "$CRITICAL" -gt 0 ]; then
echo "Critical vulnerabilities found!"
exit 1
fi
Trivy Integration#
For additional scanning:
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref:
'${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}'
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
exit-code: '1'
Tagging Strategy#
Tag Format#
| Environment | Tag Format | Example |
|---|---|---|
| Production | v{major}.{minor}.{patch} |
v1.2.3 |
| Staging | staging-{sha} |
staging-abc1234 |
| Development | dev-{sha} |
dev-abc1234 |
| PR | pr-{number} |
pr-123 |
| Latest | latest |
latest |
Semantic Versioning#
# Tag format: v{MAJOR}.{MINOR}.{PATCH}
# MAJOR: Breaking changes
# MINOR: New features (backward compatible)
# PATCH: Bug fixes
# Examples
v1.0.0 # Initial release
v1.1.0 # New feature added
v1.1.1 # Bug fix
v2.0.0 # Breaking change
Git-Based Tags#
# Automatic tagging based on git
- name: Generate tags
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}
tags: |
type=semver,pattern=v{{version}}
type=semver,pattern=v{{major}}.{{minor}}
type=sha,prefix=
type=ref,event=branch
type=ref,event=pr
Build Pipeline#
GitHub Actions Workflow#
name: Build and Push
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
env:
AWS_REGION: us-east-1
ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
service:
[
auth,
bff,
content,
conversation,
media,
ai,
notification,
moderation,
]
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume:
arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
aws-region: ${{ env.AWS_REGION }}
- name: Login to ECR
uses: aws-actions/amazon-ecr-login@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
file: services/${{ matrix.service }}/Dockerfile
push: true
tags: |
${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}
${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:latest
cache-from: type=gha
cache-to: type=gha,mode=max
platforms: linux/amd64
Build Caching#
# GitHub Actions cache
cache-from: type=gha
cache-to: type=gha,mode=max
# ECR cache (for self-hosted runners)
cache-from: type=registry,ref=${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:buildcache
cache-to: type=registry,ref=${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:buildcache,mode=max
Local Build#
# Build single service
docker build -t lilith/auth:local -f services/auth/Dockerfile .
# Build with BuildKit (faster)
DOCKER_BUILDKIT=1 docker build -t lilith/auth:local -f services/auth/Dockerfile .
# Build all services
for service in auth bff content conversation media ai notification moderation; do
docker build -t lilith/${service}:local -f services/${service}/Dockerfile .
done
Best Practices#
Security#
-
Non-root user: Always run as non-root
dockerfileRUN adduser -S nodejs -u 1001 USER nodejs -
Read-only filesystem: Use read-only root filesystem in Kubernetes
yamlsecurityContext: readOnlyRootFilesystem: true -
No secrets in images: Never bake secrets into images
dockerfile# BAD - Don't do this ENV API_KEY=secret123 # GOOD - Use runtime environment ENV API_KEY="" -
Pin versions: Pin base image versions
dockerfileFROM node:20.10.0-alpine3.18
Efficiency#
-
Layer ordering: Put frequently changing layers last
dockerfileCOPY package*.json ./ # Changes rarely RUN npm ci # Cached when package.json unchanged COPY . . # Changes frequently -
Minimize layers: Combine RUN commands
dockerfileRUN apk add --no-cache dumb-init && \ addgroup -g 1001 -S nodejs && \ adduser -S nodejs -u 1001 -
Use .dockerignore: Exclude unnecessary files
text# .dockerignore node_modules .git *.md tests coverage .env*
Reliability#
-
Health checks: Always include health checks
dockerfileHEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost:8080/health || exit 1 -
Signal handling: Use dumb-init or tini
dockerfileENTRYPOINT ["dumb-init", "--"] CMD ["node", "src/server.js"] -
Graceful shutdown: Handle SIGTERM in application
javascriptprocess.on('SIGTERM', async () => { await server.close(); process.exit(0); });
Troubleshooting#
Common Issues#
| Issue | Cause | Solution |
|---|---|---|
| Large image size | Dev dependencies included | Use multi-stage build |
| Build failures | Missing native deps | Add build tools in builder stage |
| Container crashes | Signal handling | Use dumb-init |
| Permission denied | Running as root | Add non-root user |
Debug Commands#
# Inspect image layers
docker history lilith/auth:latest
# Check image size
docker images lilith/auth --format "{{.Size}}"
# Run container interactively
docker run -it --rm lilith/auth:latest sh
# View container logs
docker logs -f <container-id>
# Inspect running container
docker exec -it <container-id> sh
Related Documentation#
- AWS Setup - ECR configuration
- EKS Setup - Kubernetes deployment
- CI/CD Pipeline - Build automation
- Deployment Process - Deployment procedures