Eve library

Container Strategy

Guide

Revision 1

On this page

Comprehensive guide for containerization and container registry management in the Lilith platform.

Table of Contents#

Overview#

The Lilith platform uses containerized microservices deployed to Amazon EKS. All services follow consistent containerization patterns for security, efficiency, and maintainability.

Container Architecture#

text
┌─────────────────────────────────────────────────────────────────────┐
│                        Container Registry (ECR)                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                 │
│  │ lilith/auth │  │ lilith/bff  │  │ lilith/ai   │  ...            │
│  │  :v1.2.3    │  │  :v1.2.3    │  │  :v1.2.3    │                 │
│  └─────────────┘  └─────────────┘  └─────────────┘                 │
└─────────────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │   EKS Cluster     │
                    │  ┌─────┐ ┌─────┐  │
                    │  │ Pod │ │ Pod │  │
                    │  └─────┘ └─────┘  │
                    └───────────────────┘

Key Principles#

Principle Implementation
Security Non-root user, minimal base image, vulnerability scanning
Efficiency Multi-stage builds, layer caching, small image size
Reliability Health checks, proper signal handling, graceful shutdown
Consistency Standardized Dockerfile template across all services

Base Images#

Standard Base Image#

All Node.js services use node:20-alpine as the base image:

dockerfile
FROM node:20-alpine

Why Alpine?#

Feature Alpine Debian-based
Base size ~5MB ~120MB
Final image ~150MB ~400MB+
Attack surface Minimal Larger
Package manager apk apt

Required System Packages#

dockerfile
# Runtime dependencies
RUN apk add --no-cache dumb-init

# Build dependencies (builder stage only)
RUN apk add --no-cache python3 make g++

Dockerfile Standards#

Standard Template#

All services use this standardized Dockerfile structure:

dockerfile
# Multi-stage build for production-ready service
FROM node:20-alpine AS builder

# Install build dependencies
RUN apk add --no-cache python3 make g++

# Set working directory
WORKDIR /app

# Copy package files for layer caching
COPY services/SERVICE_NAME/package*.json ./services/SERVICE_NAME/
COPY services/lib/package*.json ./services/lib/
COPY services/common/package*.json ./services/common/

# Install dependencies
WORKDIR /app/services/SERVICE_NAME
RUN npm ci --only=production || npm install --only=production

# Production stage
FROM node:20-alpine

# Install dumb-init for signal handling
RUN apk add --no-cache dumb-init

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

# Copy from builder
COPY --from=builder --chown=nodejs:nodejs /app/services/SERVICE_NAME/node_modules ./services/SERVICE_NAME/node_modules

# Copy source
COPY --chown=nodejs:nodejs services/SERVICE_NAME ./services/SERVICE_NAME
COPY --chown=nodejs:nodejs services/lib ./services/lib
COPY --chown=nodejs:nodejs services/common ./services/common

# Switch to non-root
USER nodejs

# Environment
ENV NODE_ENV=production
ENV SERVICE_NAME=SERVICE_NAME
ENV PORT=8080

EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
    CMD node -e "require('http').get('http://localhost:8080/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"

WORKDIR /app/services/SERVICE_NAME

# Use dumb-init for proper signal handling
ENTRYPOINT ["dumb-init", "--"]

CMD ["node", "src/server.js"]

Required Elements#

Element Purpose Required
Multi-stage build Reduce image size Yes
Non-root user Security Yes
dumb-init Signal handling Yes
HEALTHCHECK Container health Yes
Layer caching Build efficiency Yes

Multi-Stage Builds#

Build Stage#

The builder stage includes all build tools and dependencies:

dockerfile
FROM node:20-alpine AS builder

# Build dependencies needed for native modules
RUN apk add --no-cache python3 make g++

# Install all dependencies including devDependencies if needed
RUN npm ci

Production Stage#

The production stage contains only runtime requirements:

dockerfile
FROM node:20-alpine

# Only runtime dependencies
RUN apk add --no-cache dumb-init

# Copy only production node_modules
COPY --from=builder /app/node_modules ./node_modules

# Copy application code
COPY --chown=nodejs:nodejs . .

Image Size Comparison#

Build Type Image Size
Single-stage (dev deps) ~800MB
Multi-stage (prod only) ~150MB
With build caching ~150MB (faster builds)

Image Registry (ECR)#

Repository Structure#

text
AWS ECR
├── lilith/auth
├── lilith/bff
├── lilith/content
├── lilith/conversation
├── lilith/media
├── lilith/ai
├── lilith/notification
├── lilith/moderation
└── lilith/base (shared base images)

Terraform Configuration#

hcl
resource "aws_ecr_repository" "services" {
  for_each = toset([
    "auth", "bff", "content", "conversation",
    "media", "ai", "notification", "moderation"
  ])

  name                 = "lilith/${each.key}"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  encryption_configuration {
    encryption_type = "KMS"
    kms_key         = aws_kms_key.ecr.arn
  }

  tags = {
    Service     = each.key
    Environment = var.environment
    Project     = "lilith"
  }
}

# Lifecycle policy to manage image retention
resource "aws_ecr_lifecycle_policy" "services" {
  for_each   = aws_ecr_repository.services
  repository = each.value.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Keep last 30 production images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["v"]
          countType     = "imageCountMoreThan"
          countNumber   = 30
        }
        action = {
          type = "expire"
        }
      },
      {
        rulePriority = 2
        description  = "Keep last 10 dev images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["dev-", "pr-"]
          countType     = "imageCountMoreThan"
          countNumber   = 10
        }
        action = {
          type = "expire"
        }
      },
      {
        rulePriority = 3
        description  = "Delete untagged images after 7 days"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 7
        }
        action = {
          type = "expire"
        }
      }
    ]
  })
}

Cross-Account Access#

For multi-account setups:

hcl
resource "aws_ecr_repository_policy" "cross_account" {
  repository = aws_ecr_repository.services["auth"].name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowPull"
        Effect = "Allow"
        Principal = {
          AWS = [
            "arn:aws:iam::${var.staging_account_id}:root",
            "arn:aws:iam::${var.prod_account_id}:root"
          ]
        }
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
      }
    ]
  })
}

Image Scanning#

Automated Scanning#

ECR automatically scans images on push:

hcl
image_scanning_configuration {
  scan_on_push = true
}

Scanning Results#

Check scan results via CLI:

bash
# Get scan findings
aws ecr describe-image-scan-findings \
  --repository-name lilith/auth \
  --image-id imageTag=v1.2.3

# List images with vulnerabilities
aws ecr describe-images \
  --repository-name lilith/auth \
  --query 'imageDetails[?imageScanStatus.status==`COMPLETE`].{Tag:imageTags[0],Critical:imageScanFindingsSummary.findingSeverityCounts.CRITICAL,High:imageScanFindingsSummary.findingSeverityCounts.HIGH}'

CI/CD Integration#

yaml
# .github/workflows/build.yml
- name: Scan image
  run: |
    # Wait for scan to complete
    aws ecr wait image-scan-complete \
      --repository-name lilith/${{ matrix.service }} \
      --image-id imageTag=${{ github.sha }}

    # Get findings
    CRITICAL=$(aws ecr describe-image-scan-findings \
      --repository-name lilith/${{ matrix.service }} \
      --image-id imageTag=${{ github.sha }} \
      --query 'imageScanFindings.findingSeverityCounts.CRITICAL' \
      --output text)

    # Fail if critical vulnerabilities
    if [ "$CRITICAL" != "None" ] && [ "$CRITICAL" -gt 0 ]; then
      echo "Critical vulnerabilities found!"
      exit 1
    fi

Trivy Integration#

For additional scanning:

yaml
- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref:
      '${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'

Tagging Strategy#

Tag Format#

Environment Tag Format Example
Production v{major}.{minor}.{patch} v1.2.3
Staging staging-{sha} staging-abc1234
Development dev-{sha} dev-abc1234
PR pr-{number} pr-123
Latest latest latest

Semantic Versioning#

bash
# Tag format: v{MAJOR}.{MINOR}.{PATCH}
# MAJOR: Breaking changes
# MINOR: New features (backward compatible)
# PATCH: Bug fixes

# Examples
v1.0.0  # Initial release
v1.1.0  # New feature added
v1.1.1  # Bug fix
v2.0.0  # Breaking change

Git-Based Tags#

yaml
# Automatic tagging based on git
- name: Generate tags
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}
    tags: |
      type=semver,pattern=v{{version}}
      type=semver,pattern=v{{major}}.{{minor}}
      type=sha,prefix=
      type=ref,event=branch
      type=ref,event=pr

Build Pipeline#

GitHub Actions Workflow#

yaml
name: Build and Push

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  AWS_REGION: us-east-1
  ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service:
          [
            auth,
            bff,
            content,
            conversation,
            media,
            ai,
            notification,
            moderation,
          ]

    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume:
            arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: services/${{ matrix.service }}/Dockerfile
          push: true
          tags: |
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
          platforms: linux/amd64

Build Caching#

yaml
# GitHub Actions cache
cache-from: type=gha
cache-to: type=gha,mode=max

# ECR cache (for self-hosted runners)
cache-from: type=registry,ref=${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:buildcache
cache-to: type=registry,ref=${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:buildcache,mode=max

Local Build#

bash
# Build single service
docker build -t lilith/auth:local -f services/auth/Dockerfile .

# Build with BuildKit (faster)
DOCKER_BUILDKIT=1 docker build -t lilith/auth:local -f services/auth/Dockerfile .

# Build all services
for service in auth bff content conversation media ai notification moderation; do
  docker build -t lilith/${service}:local -f services/${service}/Dockerfile .
done

Best Practices#

Security#

  1. Non-root user: Always run as non-root

    dockerfile
    RUN adduser -S nodejs -u 1001
    USER nodejs
    
  2. Read-only filesystem: Use read-only root filesystem in Kubernetes

    yaml
    securityContext:
      readOnlyRootFilesystem: true
    
  3. No secrets in images: Never bake secrets into images

    dockerfile
    # BAD - Don't do this
    ENV API_KEY=secret123
    
    # GOOD - Use runtime environment
    ENV API_KEY=""
    
  4. Pin versions: Pin base image versions

    dockerfile
    FROM node:20.10.0-alpine3.18
    

Efficiency#

  1. Layer ordering: Put frequently changing layers last

    dockerfile
    COPY package*.json ./    # Changes rarely
    RUN npm ci               # Cached when package.json unchanged
    COPY . .                 # Changes frequently
    
  2. Minimize layers: Combine RUN commands

    dockerfile
    RUN apk add --no-cache dumb-init && \
        addgroup -g 1001 -S nodejs && \
        adduser -S nodejs -u 1001
    
  3. Use .dockerignore: Exclude unnecessary files

    text
    # .dockerignore
    node_modules
    .git
    *.md
    tests
    coverage
    .env*
    

Reliability#

  1. Health checks: Always include health checks

    dockerfile
    HEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost:8080/health || exit 1
    
  2. Signal handling: Use dumb-init or tini

    dockerfile
    ENTRYPOINT ["dumb-init", "--"]
    CMD ["node", "src/server.js"]
    
  3. Graceful shutdown: Handle SIGTERM in application

    javascript
    process.on('SIGTERM', async () => {
      await server.close();
      process.exit(0);
    });
    

Troubleshooting#

Common Issues#

Issue Cause Solution
Large image size Dev dependencies included Use multi-stage build
Build failures Missing native deps Add build tools in builder stage
Container crashes Signal handling Use dumb-init
Permission denied Running as root Add non-root user

Debug Commands#

bash
# Inspect image layers
docker history lilith/auth:latest

# Check image size
docker images lilith/auth --format "{{.Size}}"

# Run container interactively
docker run -it --rm lilith/auth:latest sh

# View container logs
docker logs -f <container-id>

# Inspect running container
docker exec -it <container-id> sh