# Neo4j Graph Database Operations Runbook

**Purpose**: Operational procedures for Neo4j graph database including
configuration, monitoring, schema management, and disaster recovery.

## Neo4j Architecture Overview

### Development Environment

```
Neo4j Configuration:
├── Image: neo4j:5.15.0-community
├── Ports:
│   ├── 7475 (HTTP/Browser)
│   └── 7688 (Bolt protocol)
├── Memory:
│   ├── Heap: 512MB initial, 2GB max
│   └── Page Cache: 1GB
├── Container Memory: 3GB limit
├── Authentication: neo4j/dev_password
└── Plugins: APOC

Storage:
├── neo4j-data: Graph database files
├── neo4j-logs: Database logs
├── neo4j-import: Import directory
└── neo4j-plugins: Plugin JARs
```

### Services Using Neo4j

| Service   | Purpose                    | Node Types                    |
| --------- | -------------------------- | ----------------------------- |
| knowledge | Knowledge graph operations | Concept, Entity, Relationship |
| rag       | Graph-enhanced retrieval   | Document, Topic, Citation     |

## Configuration Details

### Docker Compose Configuration

```yaml
neo4j:
  image: neo4j:5.15.0-community
  ports:
    - '7475:7474' # HTTP/Browser
    - '7688:7687' # Bolt protocol
  environment:
    - NEO4J_AUTH=neo4j/dev_password
    - NEO4J_PLUGINS=["apoc"]
    - NEO4J_dbms_security_procedures_unrestricted=apoc.*
    - NEO4J_dbms_security_procedures_allowlist=apoc.*
    - NEO4J_apoc_export_file_enabled=true
    - NEO4J_apoc_import_file_enabled=true
    - NEO4J_dbms_memory_heap_initial__size=512m
    - NEO4J_dbms_memory_heap_max__size=2G
    - NEO4J_dbms_memory_pagecache_size=1G
    - NEO4J_server_metrics_enabled=true
    - NEO4J_server_metrics_prometheus_enabled=true
    - NEO4J_server_metrics_prometheus_endpoint=0.0.0.0:2004
  deploy:
    resources:
      limits:
        memory: 3G
      reservations:
        memory: 2G
```

### Memory Configuration Guide

| Workload          | Heap Initial | Heap Max | Page Cache |
| ----------------- | ------------ | -------- | ---------- |
| Development       | 512MB        | 2GB      | 1GB        |
| Small Production  | 1GB          | 4GB      | 2GB        |
| Medium Production | 2GB          | 8GB      | 4GB        |
| Large Production  | 4GB          | 16GB     | 8GB        |

**Rule of thumb**: Page cache should be large enough to hold your graph data.
Heap should be sized for your query complexity.

## Operational Procedures

### Starting Neo4j

```bash
# Development environment
cd /home/eaa/lilith
docker compose -f docker-compose.dev.yml up -d neo4j

# Verify Neo4j is running
docker compose -f docker-compose.dev.yml ps neo4j
docker compose -f docker-compose.dev.yml logs neo4j --tail=20

# Check browser interface
curl -s http://localhost:7475/

# Test Bolt connection
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password "RETURN 1"
```

### Checking Neo4j Health

```bash
# Database status
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listDatabases()"

# Memory usage
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listPools()"

# Active transactions
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listTransactions()"

# Query statistics
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.queryJmx('org.neo4j:instance=kernel#0,name=Transactions')"
```

### Browser Access

Access the Neo4j Browser at: http://localhost:7475

Default credentials:

- Username: `neo4j`
- Password: `dev_password`

### Schema Management

```cypher
// Create indexes for performance
CREATE INDEX entity_name IF NOT EXISTS FOR (e:Entity) ON (e.name);
CREATE INDEX concept_name IF NOT EXISTS FOR (c:Concept) ON (c.name);
CREATE INDEX document_id IF NOT EXISTS FOR (d:Document) ON (d.id);

// Create constraints for data integrity
CREATE CONSTRAINT entity_id_unique IF NOT EXISTS FOR (e:Entity) REQUIRE e.id IS UNIQUE;
CREATE CONSTRAINT concept_id_unique IF NOT EXISTS FOR (c:Concept) REQUIRE c.id IS UNIQUE;

// View all indexes
SHOW INDEXES;

// View all constraints
SHOW CONSTRAINTS;

// Analyze schema
CALL db.schema.visualization();
```

### Common Cypher Queries

```cypher
// Count nodes by label
MATCH (n) RETURN labels(n) as label, count(*) as count ORDER BY count DESC;

// Count relationships by type
MATCH ()-[r]->() RETURN type(r) as type, count(*) as count ORDER BY count DESC;

// Find orphan nodes
MATCH (n) WHERE NOT (n)--() RETURN n LIMIT 10;

// Find highly connected nodes
MATCH (n) RETURN n, size((n)--()) as degree ORDER BY degree DESC LIMIT 10;

// Shortest path between nodes
MATCH path = shortestPath(
  (a:Entity {name: 'Source'})-[*]-(b:Entity {name: 'Target'})
)
RETURN path;

// Find all paths up to depth 3
MATCH path = (a:Entity {name: 'Source'})-[*1..3]-(b:Entity)
RETURN path LIMIT 100;
```

### APOC Procedures

```cypher
// Export graph to JSON
CALL apoc.export.json.all('/var/lib/neo4j/import/backup.json', {});

// Import from JSON
CALL apoc.import.json('/var/lib/neo4j/import/data.json');

// Batch create nodes
CALL apoc.periodic.iterate(
  "UNWIND $nodes as node RETURN node",
  "CREATE (n:Entity) SET n = node",
  {batchSize: 1000, params: {nodes: $nodeList}}
);

// Analyze graph
CALL apoc.meta.stats();

// Schema introspection
CALL apoc.meta.schema();
```

## Troubleshooting

### Common Issues

#### Issue: Neo4j Not Starting

**Symptoms:**

- Container in CrashLoopBackOff
- "Database failed to start" errors

**Resolution:**

```bash
# Check logs
docker compose -f docker-compose.dev.yml logs neo4j --tail=100

# Check if data is corrupted
docker compose -f docker-compose.dev.yml exec neo4j ls -la /data/databases/

# Reset database (WARNING: data loss)
docker compose -f docker-compose.dev.yml down
docker volume rm lilith_neo4j-data
docker compose -f docker-compose.dev.yml up -d neo4j
```

#### Issue: Slow Queries

**Symptoms:**

- High latency on graph queries
- `Neo4jSlowQueries` alert

**Resolution:**

```cypher
// Profile slow query
PROFILE MATCH (n:Entity)-[r*1..5]-(m:Entity)
WHERE n.name = 'Source'
RETURN n, r, m;

// Check if indexes are being used
EXPLAIN MATCH (n:Entity {name: 'Source'}) RETURN n;

// Create missing indexes
CREATE INDEX entity_name FOR (e:Entity) ON (e.name);

// Force index usage
MATCH (n:Entity)
USING INDEX n:Entity(name)
WHERE n.name = 'Source'
RETURN n;
```

#### Issue: Out of Memory

**Symptoms:**

- OOM kills
- `Neo4jHighMemoryUsage` alert

**Resolution:**

```bash
# Check current memory usage
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listPools()"

# Kill long-running queries
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listTransactions()"

# Then terminate specific transaction
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.killTransaction('<transaction-id>')"

# Increase heap in docker-compose.dev.yml
# NEO4J_dbms_memory_heap_max__size=4G
```

#### Issue: High Page Cache Evictions

**Symptoms:**

- `Neo4jHighPageCacheEvictions` alert
- Slow reads

**Resolution:**

```bash
# Check page cache usage
curl -s http://localhost:7475/db/data/transaction/commit \
  -H "Content-Type: application/json" \
  -d '{"statements":[{"statement":"CALL dbms.queryJmx(\"org.neo4j:instance=kernel#0,name=Page cache\")"}]}'

# Increase page cache in docker-compose.dev.yml
# NEO4J_dbms_memory_pagecache_size=2G
```

## Backup & Recovery

### Manual Backup

```bash
# Stop writes (optional but recommended for consistency)
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.checkpoint()"

# Create backup directory
BACKUP_DIR="/backup/neo4j/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Copy data files
docker cp $(docker compose -f docker-compose.dev.yml ps -q neo4j):/data "$BACKUP_DIR/data"

# Export using APOC (alternative)
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL apoc.export.cypher.all('/var/lib/neo4j/import/full-backup.cypher', {})"
docker cp $(docker compose -f docker-compose.dev.yml ps -q neo4j):/var/lib/neo4j/import/full-backup.cypher "$BACKUP_DIR/"

echo "Backup completed to $BACKUP_DIR"
```

### Restore Procedure

```bash
BACKUP_DIR="/backup/neo4j/20240115-120000"

# Stop Neo4j
docker compose -f docker-compose.dev.yml stop neo4j

# Remove current data
docker volume rm lilith_neo4j-data
docker volume create lilith_neo4j-data

# Restore data
docker run --rm -v lilith_neo4j-data:/data -v "$BACKUP_DIR/data":/backup \
  alpine cp -r /backup/* /data/

# Or restore from Cypher export
docker compose -f docker-compose.dev.yml up -d neo4j
sleep 30
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL apoc.cypher.runFile('/var/lib/neo4j/import/full-backup.cypher')"
```

### Disaster Recovery

For complete data loss, recreate the schema and reload from source data:

```cypher
// Recreate schema
CREATE CONSTRAINT entity_id_unique IF NOT EXISTS FOR (e:Entity) REQUIRE e.id IS UNIQUE;
CREATE INDEX entity_name IF NOT EXISTS FOR (e:Entity) ON (e.name);
CREATE INDEX concept_name IF NOT EXISTS FOR (c:Concept) ON (c.name);

// Import from backup file
CALL apoc.import.json('/var/lib/neo4j/import/backup.json');

// Or batch create from application data
// Application needs to re-ingest knowledge graph data
```

## Performance Tuning

### Recommended Production Settings

```yaml
environment:
  - NEO4J_dbms_memory_heap_initial__size=2G
  - NEO4J_dbms_memory_heap_max__size=8G
  - NEO4J_dbms_memory_pagecache_size=4G
  - NEO4J_dbms_memory_off__heap_max__size=2G
  - NEO4J_dbms_tx__state_memory__allocation=ON_HEAP
  - NEO4J_dbms_connector_bolt_thread__pool__max__size=400
  - NEO4J_dbms_connector_bolt_thread__pool__min__size=10
```

### Query Optimization Tips

1. **Always use indexes for lookups**

   ```cypher
   // Bad: Full scan
   MATCH (n:Entity) WHERE n.name = 'Foo' RETURN n;

   // Good: Index lookup (if index exists)
   MATCH (n:Entity {name: 'Foo'}) RETURN n;
   ```

2. **Limit relationship depth**

   ```cypher
   // Bad: Unbounded traversal
   MATCH (n)-[*]-(m) RETURN n, m;

   // Good: Bounded depth
   MATCH (n)-[*1..3]-(m) RETURN n, m;
   ```

3. **Use parameters for repeated queries**

   ```cypher
   // Use $param instead of literal values
   MATCH (n:Entity {name: $name}) RETURN n;
   ```

4. **Profile before optimizing**
   ```cypher
   PROFILE MATCH (n:Entity)-[:RELATED_TO]->(m) RETURN n, m;
   ```

### Key Metrics to Monitor

| Metric                | Warning Threshold | Critical Threshold |
| --------------------- | ----------------- | ------------------ |
| Heap Usage            | > 90%             | > 95%              |
| Page Cache Evictions  | > 100/sec         | > 500/sec          |
| Transaction Rollbacks | > 1/sec           | > 10/sec           |
| Bolt Connections      | > 100             | > 200              |
| Query Latency (P95)   | > 1 second        | > 5 seconds        |

## Graph Schema Reference

### Node Labels

| Label     | Description                        | Properties                     |
| --------- | ---------------------------------- | ------------------------------ |
| Entity    | Named entities                     | id, name, type, description    |
| Concept   | Abstract concepts                  | id, name, category, definition |
| Document  | Source documents                   | id, title, source, created_at  |
| Topic     | Discussion topics                  | id, name, keywords             |
| Tradition | Religious/philosophical traditions | id, name, region, period       |

### Relationship Types

| Type          | Description              | Properties       |
| ------------- | ------------------------ | ---------------- |
| RELATED_TO    | General relationship     | weight, source   |
| PART_OF       | Hierarchical containment | -                |
| REFERENCES    | Citation relationship    | page, chapter    |
| INFLUENCED_BY | Influence relationship   | period, strength |
| BELONGS_TO    | Category membership      | -                |

---

**Last Updated**: 2025-01-XX **Owner**: Platform Team **Review Cycle**:
Quarterly
