# CI/CD Pipeline

Comprehensive guide for the Continuous Integration and Continuous Deployment
pipeline for the Lilith platform.

## Table of Contents

- [Overview](#overview)
- [Pipeline Architecture](#pipeline-architecture)
- [Continuous Integration](#continuous-integration)
- [Continuous Deployment](#continuous-deployment)
- [Environment Strategy](#environment-strategy)
- [GitOps with ArgoCD](#gitops-with-argocd)
- [Secrets Management](#secrets-management)
- [Rollback Procedures](#rollback-procedures)
- [Monitoring & Alerting](#monitoring--alerting)

## Overview

### Pipeline Flow

```
┌─────────────────────────────────────────────────────────────────────────┐
│                           CI/CD Pipeline                                 │
│                                                                          │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐   │
│  │  Push   │──▶│  Lint   │──▶│  Test   │──▶│  Build  │──▶│  Scan   │   │
│  │         │   │  Check  │   │  Suite  │   │  Image  │   │  Image  │   │
│  └─────────┘   └─────────┘   └─────────┘   └─────────┘   └─────────┘   │
│                                                               │          │
│  ┌─────────────────────────────────────────────────────────────┘         │
│  │                                                                       │
│  ▼                                                                       │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐                  │
│  │  Push   │──▶│ ArgoCD  │──▶│ Staging │──▶│  Prod   │                  │
│  │  ECR    │   │  Sync   │   │ Deploy  │   │ Deploy  │                  │
│  └─────────┘   └─────────┘   └─────────┘   └─────────┘                  │
│                                     │             │                      │
│                              ┌──────▼─────┐ ┌─────▼──────┐              │
│                              │  Smoke     │ │  Approval  │              │
│                              │  Tests     │ │  Required  │              │
│                              └────────────┘ └────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘
```

### Key Principles

| Principle         | Implementation                          |
| ----------------- | --------------------------------------- |
| **Fast Feedback** | Parallel jobs, early failure, caching   |
| **Consistency**   | Docker builds, immutable artifacts      |
| **Security**      | Image scanning, secret management, OIDC |
| **Observability** | Detailed logs, metrics, notifications   |

## Pipeline Architecture

### GitHub Actions Workflows

| Workflow              | Trigger            | Purpose                      |
| --------------------- | ------------------ | ---------------------------- |
| `ci.yml`              | Push, PR           | Lint, test, quality checks   |
| `deploy.yml`          | Push to main, tags | Build, push, deploy          |
| `deploy-promote.yaml` | Manual             | Promote between environments |
| `maintenance.yml`     | Schedule           | Dependency updates, cleanup  |

### Job Dependencies

```yaml
# Typical job flow
prelim-checks ──▶ quality-checks ──▶ unit-tests ──▶ build-images
│                                 │ ▼                                 ▼
security-scan              integration-tests │                                 │
└────────────┬────────────────────┘ ▼ deploy-staging │ ▼ smoke-tests │ ▼
deploy-production
```

## Continuous Integration

### CI Workflow Configuration

```yaml
# .github/workflows/ci.yml
name: Continuous Integration

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *' # Nightly builds

env:
  NODE_VERSION: '18'
  CACHE_VERSION: 'v1'

jobs:
  prelim-checks:
    name: Preliminary Checks
    runs-on: ubuntu-latest
    outputs:
      affected-services: ${{ steps.detect-changes.outputs.services }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Cache dependencies
        uses: actions/cache@v4
        with:
          path: |
            node_modules
            services/*/node_modules
          key:
            ${{ env.CACHE_VERSION }}-deps-${{ hashFiles('**/package-lock.json')
            }}

      - name: Install dependencies
        run: npm ci

      - name: Detect changed services
        id: detect-changes
        run: |
          CHANGED=$(git diff --name-only ${{ github.event.pull_request.base.sha }}..${{ github.sha }} | \
            grep -E '^services/[^/]+/' | cut -d'/' -f2 | sort -u | tr '\n' ',')
          echo "services=${CHANGED%,}" >> $GITHUB_OUTPUT
```

### Quality Checks

```yaml
quality-checks:
  name: Quality & Security
  needs: prelim-checks
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Run ESLint
      run: npm run lint

    - name: Check TypeScript
      run: npm run typecheck

    - name: Check formatting
      run: npm run format:check

    - name: Run security audit
      run: npm audit --audit-level=high

    - name: Check for secrets
      uses: trufflesecurity/trufflehog@main
      with:
        path: ./
        base: ${{ github.event.repository.default_branch }}
```

### Unit Tests

```yaml
unit-tests:
  name: Unit Tests
  needs: prelim-checks
  runs-on: ubuntu-latest
  strategy:
    matrix:
      service: [auth, bff, content, conversation, media, ai]
  steps:
    - uses: actions/checkout@v4

    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: ${{ env.NODE_VERSION }}

    - name: Run tests
      run: |
        cd services/${{ matrix.service }}
        npm ci
        npm test -- --coverage

    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        files: services/${{ matrix.service }}/coverage/lcov.info
        flags: ${{ matrix.service }}
```

### Integration Tests

```yaml
integration-tests:
  name: Integration Tests
  needs: [unit-tests, build-images]
  runs-on: ubuntu-latest
  services:
    postgres:
      image: postgres:16
      env:
        POSTGRES_PASSWORD: test
      options: >-
        --health-cmd pg_isready --health-interval 10s
    redis:
      image: redis:7
      options: >-
        --health-cmd "redis-cli ping"
  steps:
    - uses: actions/checkout@v4

    - name: Run integration tests
      run: npm run test:integration
      env:
        DATABASE_URL: postgresql://postgres:test@localhost/test
        REDIS_URL: redis://localhost:6379
```

## Continuous Deployment

### Deployment Workflow

```yaml
# .github/workflows/deploy.yml
name: Continuous Deployment

on:
  push:
    branches: [main]
    tags: ['v*.*.*']
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        type: choice
        options: [staging, production]
      service:
        description: 'Service to deploy (empty for all)'
        required: false

env:
  AWS_REGION: us-east-1
  ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com

jobs:
  build-and-push:
    name: Build & Push
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service:
          [
            auth,
            bff,
            content,
            conversation,
            media,
            ai,
            notification,
            moderation,
          ]
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume:
            arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: services/${{ matrix.service }}/Dockerfile
          push: true
          tags: |
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan image
        run: |
          aws ecr wait image-scan-complete \
            --repository-name lilith/${{ matrix.service }} \
            --image-id imageTag=${{ github.sha }}
```

### Deployment Strategies

| Strategy   | Use Case               | Rollback Time |
| ---------- | ---------------------- | ------------- |
| Rolling    | Standard deployments   | 2-5 minutes   |
| Blue-Green | Zero-downtime critical | Instant       |
| Canary     | High-risk changes      | Gradual       |

### Rolling Deployment

```yaml
deploy-staging:
  name: Deploy to Staging
  needs: build-and-push
  runs-on: ubuntu-latest
  environment:
    name: staging
    url: https://api.staging.lilith.app
  steps:
    - name: Update ArgoCD application
      run: |
        argocd app set lilith-staging \
          --helm-set image.tag=${{ github.sha }}
        argocd app sync lilith-staging --prune
        argocd app wait lilith-staging --health
```

### Production Deployment

```yaml
deploy-production:
  name: Deploy to Production
  needs: [deploy-staging, smoke-tests]
  runs-on: ubuntu-latest
  environment:
    name: production
    url: https://api.lilith.app
  steps:
    - name: Deploy to production
      run: |
        argocd app set lilith-production \
          --helm-set image.tag=${{ github.sha }}
        argocd app sync lilith-production --prune
        argocd app wait lilith-production --health

    - name: Notify on success
      uses: slackapi/slack-github-action@v1
      with:
        channel-id: ${{ secrets.SLACK_DEPLOY_CHANNEL }}
        payload: |
          {
            "text": "✅ Production deployment successful",
            "blocks": [
              {
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "*Deployment Complete*\nVersion: `${{ github.sha }}`\nEnvironment: Production"
                }
              }
            ]
          }
```

## Environment Strategy

### Environment Configuration

| Environment | Branch     | Auto-Deploy | Approval |
| ----------- | ---------- | ----------- | -------- |
| Development | develop    | Yes         | No       |
| Staging     | main       | Yes         | No       |
| Production  | tags (v\*) | No          | Required |

### Environment Protection

```yaml
# GitHub Environment settings
environments:
  staging:
    deployment_branch_policy:
      protected_branches: true
    reviewers: []

  production:
    deployment_branch_policy:
      protected_branches: true
    reviewers:
      - platform-team
    wait_timer: 5 # 5 minutes
```

### Environment Variables

```yaml
# Per-environment configuration
env:
  staging:
    REPLICAS: 2
    LOG_LEVEL: debug
    FEATURE_FLAGS: experimental

  production:
    REPLICAS: 5
    LOG_LEVEL: info
    FEATURE_FLAGS: stable
```

## GitOps with ArgoCD

### Application Definition

```yaml
# deploy/argocd/apps/lilith-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: lilith-staging
  namespace: argocd
spec:
  project: lilith
  source:
    repoURL: https://github.com/org/lilith
    targetRevision: HEAD
    path: deploy/helm/lilith-service
    helm:
      valueFiles:
        - values.yaml
        - values-staging.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: lilith-staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
```

### ApplicationSet for Multiple Services

```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: lilith-services
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - service: auth
          - service: bff
          - service: content
          - service: ai
  template:
    metadata:
      name: 'lilith-{{service}}-staging'
    spec:
      project: lilith
      source:
        repoURL: https://github.com/org/lilith
        path: 'deploy/helm/lilith-service'
        helm:
          values: |
            service:
              name: {{service}}
            image:
              repository: ECR_REGISTRY/lilith/{{service}}
      destination:
        server: https://kubernetes.default.svc
        namespace: lilith-staging
```

### Sync Waves

```yaml
# Control deployment order
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"  # Deploy first

# Dependencies
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"  # Deploy after wave 1
```

## Secrets Management

### GitHub Secrets

| Secret           | Purpose       | Scope        |
| ---------------- | ------------- | ------------ |
| `AWS_ACCOUNT_ID` | ECR registry  | Organization |
| `SLACK_WEBHOOK`  | Notifications | Repository   |
| `ARGOCD_TOKEN`   | ArgoCD API    | Environment  |

### OIDC Authentication

```yaml
# No long-lived credentials needed
- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume:
      arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
    aws-region: us-east-1
```

### IAM Role for GitHub Actions

```hcl
resource "aws_iam_role" "github_actions" {
  name = "github-actions-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/token.actions.githubusercontent.com"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          }
          StringLike = {
            "token.actions.githubusercontent.com:sub" = "repo:your-org/lilith:*"
          }
        }
      }
    ]
  })
}
```

## Rollback Procedures

### Quick Rollback

```bash
# Rollback to previous version
argocd app history lilith-production
argocd app rollback lilith-production <REVISION>

# Or via Helm
helm rollback lilith-auth -n lilith
```

### Automated Rollback

```yaml
# ArgoCD sync with rollback on failure
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  retry:
    limit: 3
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m
```

### Emergency Rollback Script

```bash
#!/bin/bash
# scripts/emergency-rollback.sh

SERVICE=$1
ENVIRONMENT=${2:-production}
PREVIOUS_TAG=$(aws ecr describe-images \
  --repository-name lilith/${SERVICE} \
  --query 'sort_by(imageDetails,& imagePushedAt)[-2].imageTags[0]' \
  --output text)

echo "Rolling back ${SERVICE} to ${PREVIOUS_TAG}"

argocd app set lilith-${SERVICE}-${ENVIRONMENT} \
  --helm-set image.tag=${PREVIOUS_TAG}
argocd app sync lilith-${SERVICE}-${ENVIRONMENT}
```

## Monitoring & Alerting

### Deployment Metrics

```yaml
# Prometheus metrics for deployments
- name: deployment_status
  help: 'Deployment status (1=success, 0=failure)'
  type: gauge
  labels:
    - service
    - environment
    - version

- name: deployment_duration_seconds
  help: 'Deployment duration in seconds'
  type: histogram
  labels:
    - service
    - environment
```

### Slack Notifications

```yaml
- name: Notify deployment
  uses: slackapi/slack-github-action@v1
  with:
    channel-id: deployments
    payload: |
      {
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "${{ job.status == 'success' && '✅' || '❌' }} *${{ github.repository }}*"
            }
          },
          {
            "type": "context",
            "elements": [
              {
                "type": "mrkdwn",
                "text": "Environment: `${{ env.ENVIRONMENT }}`"
              },
              {
                "type": "mrkdwn",
                "text": "Version: `${{ github.sha }}`"
              }
            ]
          }
        ]
      }
```

### Failure Alerts

```yaml
- name: Alert on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    channel-id: platform-alerts
    payload: |
      {
        "text": "🚨 Deployment failed!",
        "attachments": [
          {
            "color": "danger",
            "fields": [
              {"title": "Service", "value": "${{ matrix.service }}", "short": true},
              {"title": "Environment", "value": "${{ env.ENVIRONMENT }}", "short": true},
              {"title": "Commit", "value": "${{ github.sha }}", "short": true},
              {"title": "Author", "value": "${{ github.actor }}", "short": true}
            ]
          }
        ]
      }
```

## Troubleshooting

### Common Issues

| Issue             | Cause             | Solution                        |
| ----------------- | ----------------- | ------------------------------- |
| Build timeout     | Large images      | Enable caching, use multi-stage |
| Test failures     | Flaky tests       | Add retries, isolate tests      |
| Deploy stuck      | ArgoCD sync issue | Check ArgoCD logs, manual sync  |
| Permission denied | Missing IAM       | Update OIDC trust policy        |

### Debug Commands

```bash
# Check workflow runs
gh run list --workflow=ci.yml

# View workflow logs
gh run view <RUN_ID> --log

# Re-run failed jobs
gh run rerun <RUN_ID> --failed

# Check ArgoCD sync status
argocd app get lilith-staging
argocd app sync lilith-staging --dry-run
```

## Related Documentation

- [Container Strategy](./infrastructure/container-strategy.md) - Docker builds
- [EKS Setup](./infrastructure/eks.md) - Kubernetes deployment
- [Deployment Runbook](./runbooks/procedures/deployment.md) - Deployment
  procedures
- [Rollback Runbook](./runbooks/procedures/rollback.md) - Rollback procedures
