homericintelligence/scylla/extract-test-failures

Extract and Summarize Test Failures

Parse test logs to extract failure information and create summary.

When to Use

• CI test run failed, need to understand failures
• Large test suite with many failures, need summary
• Categorizing failures by type (assertion, runtime, timeout)
• Creating PR review comments about failures
• Tracking which tests are failing over time

Quick Reference

# Extract failed test names
grep "FAILED" test_output.log
 
# Get failure details with context
grep -A 10 "FAILED\|Error\|AssertionError" test_output.log
 
# Count failures by type
grep "FAILED\|AssertionError\|ValueError\|TypeError" test_output.log | sort | uniq -c
 
# Extract test summary line
tail -20 test_output.log | grep -E "passed|failed|error"
 
# Get specific failure info
grep -B 5 "AssertionError" test_output.log | head -50

Failure Extraction Workflow

1. Collect log: Get test output from CI or local run
2. Find failures: Extract FAILED markers and error types
3. Group by type: Categorize assertion vs runtime vs timeout
4. Extract details: Get error messages and stack traces
5. Count totals: Summary of how many failures
6. Identify patterns: Common failure causes
7. Summarize: Create concise failure report

Failure Categories

Assertion Failures:

• AssertionError - Expected value check failed
• assert_equal() - Values don't match
• assert_true() - Condition not true
• Fix: Check test logic and expected values

Type/Attribute Errors:

• AttributeError - Missing method/attribute
• TypeError - Wrong argument type
• ValueError - Invalid value for operation
• Fix: Check code syntax and types

Runtime Errors:

• IndexError - Out of bounds access
• KeyError - Dictionary key not found
• ZeroDivisionError - Division by zero
• Fix: Check boundary conditions

Timeout/Hanging:

• Test takes too long
• Infinite loop or deadlock
• Resource exhaustion
• Fix: Optimize or add timeout

Output Format

Report failures with:

1. Total Failures - Count and percentage of total tests
2. Failure Summary - By type (assertion, runtime, timeout, other)
3. Failed Tests - List of test names that failed
4. Top Issues - Most common failure patterns
5. Error Messages - Representative error snippets
6. Recommendations - Which tests to focus on first

Error Handling

References

• See test-diff-analyzer for identifying flaky tests
• See analyze-ci-failure-logs for CI-specific failures
• See generate-fix-suggestions for automated fixes