Eve library

Safety-Automation Service Test Results

Other

Revision 1

On this page

Test Summary#

Total Tests: 48 Passing: 37 (77%) Failing: 11 (23%)

Test Coverage#

✅ Passing Test Suites (37 tests)#

Health Check (1/1)#

  • ✅ should return healthy status

Red-Team Attack Generation (13/17)#

Single Attack Generation:

  • ✅ should generate a jailbreaking attack
  • ✅ should generate a prompt injection attack
  • ✅ should generate social engineering attacks
  • ✅ should generate data extraction attacks
  • ✅ should reject invalid category
  • ❌ should generate adversarial prompts (Bug: Missing templates)
  • ❌ should generate harmful content attacks (Bug: Missing templates)
  • ❌ should support persona-specific attacks (Bug: Metadata structure mismatch)
  • ❌ should include metadata in attack (Bug: Metadata structure mismatch)

Batch Attack Generation:

  • ✅ should generate a batch of attacks
  • ✅ should generate attacks from specific categories
  • ✅ should respect batch size limits
  • ✅ should generate diverse attack types in batch

Attack Retrieval:

  • ✅ should retrieve attack by ID
  • ✅ should return 404 for non-existent attack

Attack Categories:

  • ✅ should list all attack categories
  • ✅ should include category details

Security Evaluations (11/11)#

Evaluation Initiation:

  • ✅ should start a security evaluation with OWASP LLM Top 10
  • ✅ should start evaluation with NIST AI RMF
  • ✅ should start evaluation with MITRE ATLAS
  • ✅ should use default framework and depth

Evaluation Status Tracking:

  • ✅ should track evaluation status
  • ✅ should return 404 for non-existent evaluation

Evaluation Results:

  • ✅ should retrieve evaluation results when completed
  • ✅ should return 202 when evaluation is still in progress

Evaluation Listing:

  • ✅ should list all evaluations
  • ✅ should include evaluation metadata in list

Vulnerability Management (9/9)#

Vulnerability Reporting:

  • ✅ should report a vulnerability
  • ✅ should require critical fields
  • ✅ should support all severity levels

Vulnerability Retrieval:

  • ✅ should retrieve vulnerability by ID
  • ✅ should return 404 for non-existent vulnerability

Vulnerability Filtering:

  • ✅ should filter vulnerabilities by severity
  • ✅ should filter vulnerabilities by status
  • ✅ should support multiple filters

Automation and Scheduling (0/3)#

  • ❌ should create automated security testing schedule (Bug: this context issue with app.decorate)
  • ❌ should support different frequencies (Bug: this context issue)
  • ❌ should support different test types (Bug: this context issue)

Metrics and Reporting (0/4)#

  • ❌ should provide security metrics dashboard (Bug: this context issue with app.decorate)
  • ❌ should include vulnerability breakdown (Bug: Same as above)
  • ❌ should include attack effectiveness metrics (Bug: Same as above)
  • ❌ should include security trends (Bug: Same as above)

Integration and Edge Cases (5/5)#

  • ✅ should handle concurrent attack generation
  • ✅ should handle concurrent evaluations
  • ✅ should handle malformed requests gracefully
  • ✅ should validate sophistication levels
  • ✅ should validate evaluation frameworks

Bugs Discovered by Tests#

1. Missing Attack Templates (Priority: High)#

Location: src/app.ts line 164 initializeAttackTemplates() Issue: Templates defined for only 4 categories, but 6 categories are defined in attackCategories Missing: adversarial_prompts, harmful_content Failing Tests:

  • should generate adversarial prompts
  • should generate harmful content attacks

2. Metadata Structure Mismatch (Priority: Medium)#

Location: src/app.ts endpoint /v1/red-team/generate-attack Issue: API doesn't return metadata at top level for persona-specific attacks Failing Tests:

  • should support persona-specific attacks
  • should include metadata in attack

3. Fastify Decorate Context Issues (Priority: High)#

Location: src/app.ts lines 1305-1313 (automation endpoint) and 1340 (metrics endpoint) Issue: Using this.getCronPattern() and this.calculateAverageSecurityScore() in route handlers causes context errors Failing Tests:

  • All automation and scheduling tests (3 tests)
  • All metrics dashboard tests (4 tests)

4. Expected Bypass Probability Access (Priority: Low)#

Location: Test expectation vs implementation Issue: Test expects metadata.expected_bypass_probability but implementation may structure it differently Failing Tests: should include metadata in attack

Test Quality Assessment#

✅ Comprehensive Coverage: Tests cover all major functionality ✅ Real Bug Detection: Tests successfully identified 4 categories of bugs ✅ Integration Testing: Tests verify actual HTTP endpoints ✅ Edge Case Handling: Tests verify error scenarios and concurrent access ✅ Attack Categories: Tests verify all 6 attack types ✅ Security Frameworks: Tests verify OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS ✅ Vulnerability Management: Tests verify full CRUD operations and filtering

Recommendations#

Immediate Fixes Required:#

  1. Add templates for adversarial_prompts and harmful_content categories
  2. Fix Fastify decorate context issues in automation and metrics endpoints
  3. Ensure metadata structure consistency across all attack generation endpoints

Future Enhancements:#

  1. Add integration tests with actual AI service
  2. Add performance benchmarks for batch operations
  3. Add security audit logging verification tests
  4. Expand attack template variations for better coverage

Conclusion#

The test suite successfully provides comprehensive coverage of the safety-automation service and has identified real implementation bugs. A 77% pass rate on first test run for a previously untested service is excellent and demonstrates that the tests are working as intended - finding bugs is exactly what tests should do!

Status: Test suite creation COMPLETE ✅ Next Steps: Fix identified bugs to achieve 100% pass rate