Comprehensive guide for the Continuous Integration and Continuous Deployment pipeline for the Lilith platform.
Table of Contents#
- Overview
- Pipeline Architecture
- Continuous Integration
- Continuous Deployment
- Environment Strategy
- GitOps with ArgoCD
- Secrets Management
- Rollback Procedures
- Monitoring & Alerting
Overview#
Pipeline Flow#
text
┌─────────────────────────────────────────────────────────────────────────┐
│ CI/CD Pipeline │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Push │──▶│ Lint │──▶│ Test │──▶│ Build │──▶│ Scan │ │
│ │ │ │ Check │ │ Suite │ │ Image │ │ Image │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Push │──▶│ ArgoCD │──▶│ Staging │──▶│ Prod │ │
│ │ ECR │ │ Sync │ │ Deploy │ │ Deploy │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │ │ │
│ ┌──────▼─────┐ ┌─────▼──────┐ │
│ │ Smoke │ │ Approval │ │
│ │ Tests │ │ Required │ │
│ └────────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Key Principles#
| Principle | Implementation |
|---|---|
| Fast Feedback | Parallel jobs, early failure, caching |
| Consistency | Docker builds, immutable artifacts |
| Security | Image scanning, secret management, OIDC |
| Observability | Detailed logs, metrics, notifications |
Pipeline Architecture#
GitHub Actions Workflows#
| Workflow | Trigger | Purpose |
|---|---|---|
ci.yml |
Push, PR | Lint, test, quality checks |
deploy.yml |
Push to main, tags | Build, push, deploy |
deploy-promote.yaml |
Manual | Promote between environments |
maintenance.yml |
Schedule | Dependency updates, cleanup |
Job Dependencies#
yaml
# Typical job flow
prelim-checks ──▶ quality-checks ──▶ unit-tests ──▶ build-images
│ │ ▼ ▼
security-scan integration-tests │ │
└────────────┬────────────────────┘ ▼ deploy-staging │ ▼ smoke-tests │ ▼
deploy-production
Continuous Integration#
CI Workflow Configuration#
yaml
# .github/workflows/ci.yml
name: Continuous Integration
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: '0 2 * * *' # Nightly builds
env:
NODE_VERSION: '18'
CACHE_VERSION: 'v1'
jobs:
prelim-checks:
name: Preliminary Checks
runs-on: ubuntu-latest
outputs:
affected-services: ${{ steps.detect-changes.outputs.services }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- name: Cache dependencies
uses: actions/cache@v4
with:
path: |
node_modules
services/*/node_modules
key:
${{ env.CACHE_VERSION }}-deps-${{ hashFiles('**/package-lock.json')
}}
- name: Install dependencies
run: npm ci
- name: Detect changed services
id: detect-changes
run: |
CHANGED=$(git diff --name-only ${{ github.event.pull_request.base.sha }}..${{ github.sha }} | \
grep -E '^services/[^/]+/' | cut -d'/' -f2 | sort -u | tr '\n' ',')
echo "services=${CHANGED%,}" >> $GITHUB_OUTPUT
Quality Checks#
yaml
quality-checks:
name: Quality & Security
needs: prelim-checks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run ESLint
run: npm run lint
- name: Check TypeScript
run: npm run typecheck
- name: Check formatting
run: npm run format:check
- name: Run security audit
run: npm audit --audit-level=high
- name: Check for secrets
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: ${{ github.event.repository.default_branch }}
Unit Tests#
yaml
unit-tests:
name: Unit Tests
needs: prelim-checks
runs-on: ubuntu-latest
strategy:
matrix:
service: [auth, bff, content, conversation, media, ai]
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Run tests
run: |
cd services/${{ matrix.service }}
npm ci
npm test -- --coverage
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: services/${{ matrix.service }}/coverage/lcov.info
flags: ${{ matrix.service }}
Integration Tests#
yaml
integration-tests:
name: Integration Tests
needs: [unit-tests, build-images]
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: test
options: >-
--health-cmd pg_isready --health-interval 10s
redis:
image: redis:7
options: >-
--health-cmd "redis-cli ping"
steps:
- uses: actions/checkout@v4
- name: Run integration tests
run: npm run test:integration
env:
DATABASE_URL: postgresql://postgres:test@localhost/test
REDIS_URL: redis://localhost:6379
Continuous Deployment#
Deployment Workflow#
yaml
# .github/workflows/deploy.yml
name: Continuous Deployment
on:
push:
branches: [main]
tags: ['v*.*.*']
workflow_dispatch:
inputs:
environment:
description: 'Target environment'
required: true
type: choice
options: [staging, production]
service:
description: 'Service to deploy (empty for all)'
required: false
env:
AWS_REGION: us-east-1
ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com
jobs:
build-and-push:
name: Build & Push
runs-on: ubuntu-latest
strategy:
matrix:
service:
[
auth,
bff,
content,
conversation,
media,
ai,
notification,
moderation,
]
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume:
arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
aws-region: ${{ env.AWS_REGION }}
- name: Login to ECR
uses: aws-actions/amazon-ecr-login@v2
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
file: services/${{ matrix.service }}/Dockerfile
push: true
tags: |
${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}
${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Scan image
run: |
aws ecr wait image-scan-complete \
--repository-name lilith/${{ matrix.service }} \
--image-id imageTag=${{ github.sha }}
Deployment Strategies#
| Strategy | Use Case | Rollback Time |
|---|---|---|
| Rolling | Standard deployments | 2-5 minutes |
| Blue-Green | Zero-downtime critical | Instant |
| Canary | High-risk changes | Gradual |
Rolling Deployment#
yaml
deploy-staging:
name: Deploy to Staging
needs: build-and-push
runs-on: ubuntu-latest
environment:
name: staging
url: https://api.staging.lilith.app
steps:
- name: Update ArgoCD application
run: |
argocd app set lilith-staging \
--helm-set image.tag=${{ github.sha }}
argocd app sync lilith-staging --prune
argocd app wait lilith-staging --health
Production Deployment#
yaml
deploy-production:
name: Deploy to Production
needs: [deploy-staging, smoke-tests]
runs-on: ubuntu-latest
environment:
name: production
url: https://api.lilith.app
steps:
- name: Deploy to production
run: |
argocd app set lilith-production \
--helm-set image.tag=${{ github.sha }}
argocd app sync lilith-production --prune
argocd app wait lilith-production --health
- name: Notify on success
uses: slackapi/slack-github-action@v1
with:
channel-id: ${{ secrets.SLACK_DEPLOY_CHANNEL }}
payload: |
{
"text": "✅ Production deployment successful",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Deployment Complete*\nVersion: `${{ github.sha }}`\nEnvironment: Production"
}
}
]
}
Environment Strategy#
Environment Configuration#
| Environment | Branch | Auto-Deploy | Approval |
|---|---|---|---|
| Development | develop | Yes | No |
| Staging | main | Yes | No |
| Production | tags (v*) | No | Required |
Environment Protection#
yaml
# GitHub Environment settings
environments:
staging:
deployment_branch_policy:
protected_branches: true
reviewers: []
production:
deployment_branch_policy:
protected_branches: true
reviewers:
- platform-team
wait_timer: 5 # 5 minutes
Environment Variables#
yaml
# Per-environment configuration
env:
staging:
REPLICAS: 2
LOG_LEVEL: debug
FEATURE_FLAGS: experimental
production:
REPLICAS: 5
LOG_LEVEL: info
FEATURE_FLAGS: stable
GitOps with ArgoCD#
Application Definition#
yaml
# deploy/argocd/apps/lilith-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: lilith-staging
namespace: argocd
spec:
project: lilith
source:
repoURL: https://github.com/org/lilith
targetRevision: HEAD
path: deploy/helm/lilith-service
helm:
valueFiles:
- values.yaml
- values-staging.yaml
destination:
server: https://kubernetes.default.svc
namespace: lilith-staging
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
ApplicationSet for Multiple Services#
yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: lilith-services
namespace: argocd
spec:
generators:
- list:
elements:
- service: auth
- service: bff
- service: content
- service: ai
template:
metadata:
name: 'lilith-{{service}}-staging'
spec:
project: lilith
source:
repoURL: https://github.com/org/lilith
path: 'deploy/helm/lilith-service'
helm:
values: |
service:
name: {{service}}
image:
repository: ECR_REGISTRY/lilith/{{service}}
destination:
server: https://kubernetes.default.svc
namespace: lilith-staging
Sync Waves#
yaml
# Control deployment order
metadata:
annotations:
argocd.argoproj.io/sync-wave: "1" # Deploy first
# Dependencies
metadata:
annotations:
argocd.argoproj.io/sync-wave: "2" # Deploy after wave 1
Secrets Management#
GitHub Secrets#
| Secret | Purpose | Scope |
|---|---|---|
AWS_ACCOUNT_ID |
ECR registry | Organization |
SLACK_WEBHOOK |
Notifications | Repository |
ARGOCD_TOKEN |
ArgoCD API | Environment |
OIDC Authentication#
yaml
# No long-lived credentials needed
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume:
arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
aws-region: us-east-1
IAM Role for GitHub Actions#
hcl
resource "aws_iam_role" "github_actions" {
name = "github-actions-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/token.actions.githubusercontent.com"
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
}
StringLike = {
"token.actions.githubusercontent.com:sub" = "repo:your-org/lilith:*"
}
}
}
]
})
}
Rollback Procedures#
Quick Rollback#
bash
# Rollback to previous version
argocd app history lilith-production
argocd app rollback lilith-production <REVISION>
# Or via Helm
helm rollback lilith-auth -n lilith
Automated Rollback#
yaml
# ArgoCD sync with rollback on failure
syncPolicy:
automated:
prune: true
selfHeal: true
retry:
limit: 3
backoff:
duration: 5s
factor: 2
maxDuration: 3m
Emergency Rollback Script#
bash
#!/bin/bash
# scripts/emergency-rollback.sh
SERVICE=$1
ENVIRONMENT=${2:-production}
PREVIOUS_TAG=$(aws ecr describe-images \
--repository-name lilith/${SERVICE} \
--query 'sort_by(imageDetails,& imagePushedAt)[-2].imageTags[0]' \
--output text)
echo "Rolling back ${SERVICE} to ${PREVIOUS_TAG}"
argocd app set lilith-${SERVICE}-${ENVIRONMENT} \
--helm-set image.tag=${PREVIOUS_TAG}
argocd app sync lilith-${SERVICE}-${ENVIRONMENT}
Monitoring & Alerting#
Deployment Metrics#
yaml
# Prometheus metrics for deployments
- name: deployment_status
help: 'Deployment status (1=success, 0=failure)'
type: gauge
labels:
- service
- environment
- version
- name: deployment_duration_seconds
help: 'Deployment duration in seconds'
type: histogram
labels:
- service
- environment
Slack Notifications#
yaml
- name: Notify deployment
uses: slackapi/slack-github-action@v1
with:
channel-id: deployments
payload: |
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${{ job.status == 'success' && '✅' || '❌' }} *${{ github.repository }}*"
}
},
{
"type": "context",
"elements": [
{
"type": "mrkdwn",
"text": "Environment: `${{ env.ENVIRONMENT }}`"
},
{
"type": "mrkdwn",
"text": "Version: `${{ github.sha }}`"
}
]
}
]
}
Failure Alerts#
yaml
- name: Alert on failure
if: failure()
uses: slackapi/slack-github-action@v1
with:
channel-id: platform-alerts
payload: |
{
"text": "🚨 Deployment failed!",
"attachments": [
{
"color": "danger",
"fields": [
{"title": "Service", "value": "${{ matrix.service }}", "short": true},
{"title": "Environment", "value": "${{ env.ENVIRONMENT }}", "short": true},
{"title": "Commit", "value": "${{ github.sha }}", "short": true},
{"title": "Author", "value": "${{ github.actor }}", "short": true}
]
}
]
}
Troubleshooting#
Common Issues#
| Issue | Cause | Solution |
|---|---|---|
| Build timeout | Large images | Enable caching, use multi-stage |
| Test failures | Flaky tests | Add retries, isolate tests |
| Deploy stuck | ArgoCD sync issue | Check ArgoCD logs, manual sync |
| Permission denied | Missing IAM | Update OIDC trust policy |
Debug Commands#
bash
# Check workflow runs
gh run list --workflow=ci.yml
# View workflow logs
gh run view <RUN_ID> --log
# Re-run failed jobs
gh run rerun <RUN_ID> --failed
# Check ArgoCD sync status
argocd app get lilith-staging
argocd app sync lilith-staging --dry-run
Related Documentation#
- Container Strategy - Docker builds
- EKS Setup - Kubernetes deployment
- Deployment Runbook - Deployment procedures
- Rollback Runbook - Rollback procedures