Eve library

Neo4j Graph Database Operations Runbook

Runbook

Revision 1

On this page

Purpose: Operational procedures for Neo4j graph database including configuration, monitoring, schema management, and disaster recovery.

Neo4j Architecture Overview#

Development Environment#

text
Neo4j Configuration:
├── Image: neo4j:5.15.0-community
├── Ports:
│   ├── 7475 (HTTP/Browser)
│   └── 7688 (Bolt protocol)
├── Memory:
│   ├── Heap: 512MB initial, 2GB max
│   └── Page Cache: 1GB
├── Container Memory: 3GB limit
├── Authentication: neo4j/dev_password
└── Plugins: APOC

Storage:
├── neo4j-data: Graph database files
├── neo4j-logs: Database logs
├── neo4j-import: Import directory
└── neo4j-plugins: Plugin JARs

Services Using Neo4j#

Service Purpose Node Types
knowledge Knowledge graph operations Concept, Entity, Relationship
rag Graph-enhanced retrieval Document, Topic, Citation

Configuration Details#

Docker Compose Configuration#

yaml
neo4j:
  image: neo4j:5.15.0-community
  ports:
    - '7475:7474' # HTTP/Browser
    - '7688:7687' # Bolt protocol
  environment:
    - NEO4J_AUTH=neo4j/dev_password
    - NEO4J_PLUGINS=["apoc"]
    - NEO4J_dbms_security_procedures_unrestricted=apoc.*
    - NEO4J_dbms_security_procedures_allowlist=apoc.*
    - NEO4J_apoc_export_file_enabled=true
    - NEO4J_apoc_import_file_enabled=true
    - NEO4J_dbms_memory_heap_initial__size=512m
    - NEO4J_dbms_memory_heap_max__size=2G
    - NEO4J_dbms_memory_pagecache_size=1G
    - NEO4J_server_metrics_enabled=true
    - NEO4J_server_metrics_prometheus_enabled=true
    - NEO4J_server_metrics_prometheus_endpoint=0.0.0.0:2004
  deploy:
    resources:
      limits:
        memory: 3G
      reservations:
        memory: 2G

Memory Configuration Guide#

Workload Heap Initial Heap Max Page Cache
Development 512MB 2GB 1GB
Small Production 1GB 4GB 2GB
Medium Production 2GB 8GB 4GB
Large Production 4GB 16GB 8GB

Rule of thumb: Page cache should be large enough to hold your graph data. Heap should be sized for your query complexity.

Operational Procedures#

Starting Neo4j#

bash
# Development environment
cd /home/eaa/lilith
docker compose -f docker-compose.dev.yml up -d neo4j

# Verify Neo4j is running
docker compose -f docker-compose.dev.yml ps neo4j
docker compose -f docker-compose.dev.yml logs neo4j --tail=20

# Check browser interface
curl -s http://localhost:7475/

# Test Bolt connection
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password "RETURN 1"

Checking Neo4j Health#

bash
# Database status
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listDatabases()"

# Memory usage
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listPools()"

# Active transactions
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listTransactions()"

# Query statistics
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.queryJmx('org.neo4j:instance=kernel#0,name=Transactions')"

Browser Access#

Access the Neo4j Browser at: http://localhost:7475

Default credentials:

  • Username: neo4j
  • Password: dev_password

Schema Management#

cypher
// Create indexes for performance
CREATE INDEX entity_name IF NOT EXISTS FOR (e:Entity) ON (e.name);
CREATE INDEX concept_name IF NOT EXISTS FOR (c:Concept) ON (c.name);
CREATE INDEX document_id IF NOT EXISTS FOR (d:Document) ON (d.id);

// Create constraints for data integrity
CREATE CONSTRAINT entity_id_unique IF NOT EXISTS FOR (e:Entity) REQUIRE e.id IS UNIQUE;
CREATE CONSTRAINT concept_id_unique IF NOT EXISTS FOR (c:Concept) REQUIRE c.id IS UNIQUE;

// View all indexes
SHOW INDEXES;

// View all constraints
SHOW CONSTRAINTS;

// Analyze schema
CALL db.schema.visualization();

Common Cypher Queries#

cypher
// Count nodes by label
MATCH (n) RETURN labels(n) as label, count(*) as count ORDER BY count DESC;

// Count relationships by type
MATCH ()-[r]->() RETURN type(r) as type, count(*) as count ORDER BY count DESC;

// Find orphan nodes
MATCH (n) WHERE NOT (n)--() RETURN n LIMIT 10;

// Find highly connected nodes
MATCH (n) RETURN n, size((n)--()) as degree ORDER BY degree DESC LIMIT 10;

// Shortest path between nodes
MATCH path = shortestPath(
  (a:Entity {name: 'Source'})-[*]-(b:Entity {name: 'Target'})
)
RETURN path;

// Find all paths up to depth 3
MATCH path = (a:Entity {name: 'Source'})-[*1..3]-(b:Entity)
RETURN path LIMIT 100;

APOC Procedures#

cypher
// Export graph to JSON
CALL apoc.export.json.all('/var/lib/neo4j/import/backup.json', {});

// Import from JSON
CALL apoc.import.json('/var/lib/neo4j/import/data.json');

// Batch create nodes
CALL apoc.periodic.iterate(
  "UNWIND $nodes as node RETURN node",
  "CREATE (n:Entity) SET n = node",
  {batchSize: 1000, params: {nodes: $nodeList}}
);

// Analyze graph
CALL apoc.meta.stats();

// Schema introspection
CALL apoc.meta.schema();

Troubleshooting#

Common Issues#

Issue: Neo4j Not Starting#

Symptoms:

  • Container in CrashLoopBackOff
  • "Database failed to start" errors

Resolution:

bash
# Check logs
docker compose -f docker-compose.dev.yml logs neo4j --tail=100

# Check if data is corrupted
docker compose -f docker-compose.dev.yml exec neo4j ls -la /data/databases/

# Reset database (WARNING: data loss)
docker compose -f docker-compose.dev.yml down
docker volume rm lilith_neo4j-data
docker compose -f docker-compose.dev.yml up -d neo4j

Issue: Slow Queries#

Symptoms:

  • High latency on graph queries
  • Neo4jSlowQueries alert

Resolution:

cypher
// Profile slow query
PROFILE MATCH (n:Entity)-[r*1..5]-(m:Entity)
WHERE n.name = 'Source'
RETURN n, r, m;

// Check if indexes are being used
EXPLAIN MATCH (n:Entity {name: 'Source'}) RETURN n;

// Create missing indexes
CREATE INDEX entity_name FOR (e:Entity) ON (e.name);

// Force index usage
MATCH (n:Entity)
USING INDEX n:Entity(name)
WHERE n.name = 'Source'
RETURN n;

Issue: Out of Memory#

Symptoms:

  • OOM kills
  • Neo4jHighMemoryUsage alert

Resolution:

bash
# Check current memory usage
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listPools()"

# Kill long-running queries
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.listTransactions()"

# Then terminate specific transaction
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.killTransaction('<transaction-id>')"

# Increase heap in docker-compose.dev.yml
# NEO4J_dbms_memory_heap_max__size=4G

Issue: High Page Cache Evictions#

Symptoms:

  • Neo4jHighPageCacheEvictions alert
  • Slow reads

Resolution:

bash
# Check page cache usage
curl -s http://localhost:7475/db/data/transaction/commit \
  -H "Content-Type: application/json" \
  -d '{"statements":[{"statement":"CALL dbms.queryJmx(\"org.neo4j:instance=kernel#0,name=Page cache\")"}]}'

# Increase page cache in docker-compose.dev.yml
# NEO4J_dbms_memory_pagecache_size=2G

Backup & Recovery#

Manual Backup#

bash
# Stop writes (optional but recommended for consistency)
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL dbms.checkpoint()"

# Create backup directory
BACKUP_DIR="/backup/neo4j/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Copy data files
docker cp $(docker compose -f docker-compose.dev.yml ps -q neo4j):/data "$BACKUP_DIR/data"

# Export using APOC (alternative)
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL apoc.export.cypher.all('/var/lib/neo4j/import/full-backup.cypher', {})"
docker cp $(docker compose -f docker-compose.dev.yml ps -q neo4j):/var/lib/neo4j/import/full-backup.cypher "$BACKUP_DIR/"

echo "Backup completed to $BACKUP_DIR"

Restore Procedure#

bash
BACKUP_DIR="/backup/neo4j/20240115-120000"

# Stop Neo4j
docker compose -f docker-compose.dev.yml stop neo4j

# Remove current data
docker volume rm lilith_neo4j-data
docker volume create lilith_neo4j-data

# Restore data
docker run --rm -v lilith_neo4j-data:/data -v "$BACKUP_DIR/data":/backup \
  alpine cp -r /backup/* /data/

# Or restore from Cypher export
docker compose -f docker-compose.dev.yml up -d neo4j
sleep 30
cypher-shell -a bolt://localhost:7688 -u neo4j -p dev_password \
  "CALL apoc.cypher.runFile('/var/lib/neo4j/import/full-backup.cypher')"

Disaster Recovery#

For complete data loss, recreate the schema and reload from source data:

cypher
// Recreate schema
CREATE CONSTRAINT entity_id_unique IF NOT EXISTS FOR (e:Entity) REQUIRE e.id IS UNIQUE;
CREATE INDEX entity_name IF NOT EXISTS FOR (e:Entity) ON (e.name);
CREATE INDEX concept_name IF NOT EXISTS FOR (c:Concept) ON (c.name);

// Import from backup file
CALL apoc.import.json('/var/lib/neo4j/import/backup.json');

// Or batch create from application data
// Application needs to re-ingest knowledge graph data

Performance Tuning#

yaml
environment:
  - NEO4J_dbms_memory_heap_initial__size=2G
  - NEO4J_dbms_memory_heap_max__size=8G
  - NEO4J_dbms_memory_pagecache_size=4G
  - NEO4J_dbms_memory_off__heap_max__size=2G
  - NEO4J_dbms_tx__state_memory__allocation=ON_HEAP
  - NEO4J_dbms_connector_bolt_thread__pool__max__size=400
  - NEO4J_dbms_connector_bolt_thread__pool__min__size=10

Query Optimization Tips#

  1. Always use indexes for lookups

    cypher
    // Bad: Full scan
    MATCH (n:Entity) WHERE n.name = 'Foo' RETURN n;
    
    // Good: Index lookup (if index exists)
    MATCH (n:Entity {name: 'Foo'}) RETURN n;
    
  2. Limit relationship depth

    cypher
    // Bad: Unbounded traversal
    MATCH (n)-[*]-(m) RETURN n, m;
    
    // Good: Bounded depth
    MATCH (n)-[*1..3]-(m) RETURN n, m;
    
  3. Use parameters for repeated queries

    cypher
    // Use $param instead of literal values
    MATCH (n:Entity {name: $name}) RETURN n;
    
  4. Profile before optimizing

    cypher
    PROFILE MATCH (n:Entity)-[:RELATED_TO]->(m) RETURN n, m;
    

Key Metrics to Monitor#

Metric Warning Threshold Critical Threshold
Heap Usage > 90% > 95%
Page Cache Evictions > 100/sec > 500/sec
Transaction Rollbacks > 1/sec > 10/sec
Bolt Connections > 100 > 200
Query Latency (P95) > 1 second > 5 seconds

Graph Schema Reference#

Node Labels#

Label Description Properties
Entity Named entities id, name, type, description
Concept Abstract concepts id, name, category, definition
Document Source documents id, title, source, created_at
Topic Discussion topics id, name, keywords
Tradition Religious/philosophical traditions id, name, region, period

Relationship Types#

Type Description Properties
RELATED_TO General relationship weight, source
PART_OF Hierarchical containment -
REFERENCES Citation relationship page, chapter
INFLUENCED_BY Influence relationship period, strength
BELONGS_TO Category membership -

Last Updated: 2025-01-XX Owner: Platform Team Review Cycle: Quarterly