proffesor-for-testing/agentic-qe/agentic-quality-engineering

Agentic Quality Engineering

<default_to_action> When implementing agentic QE or coordinating agents:

1. SPAWN appropriate agent(s) for the task using Task tool with agent type
2. CONFIGURE agent coordination (hierarchical/mesh/sequential)
3. EXECUTE with PACTS principles: Proactive analysis, Autonomous operation, Collaborative feedback, Targeted risk focus, Structured governance (observability and explainability of agent behavior)
4. VALIDATE results through quality gates before deployment
5. LEARN from outcomes - store patterns in aqe/learning/* namespace

Quick Agent Selection:

• Test generation needed → qe-test-architect
• Coverage gaps → qe-coverage-specialist
• Quality decision → qe-quality-gate
• Security scan → qe-security-scanner
• Performance test → qe-performance-tester
• Full pipeline → qe-fleet-commander

Critical Success Factors:

• Agents amplify human expertise, not replace it
• Human-in-the-loop for critical decisions
• Measure: bugs caught, time saved, coverage improved, and how often agent claims survive verification
• Done means committed, tested, and CI-validated with evidence attached, not "reported as complete"

</default_to_action>

Quick Reference Card

When to Use

• Designing autonomous testing systems
• Scaling QE with intelligent agents
• Implementing multi-agent coordination
• Building CI/CD quality pipelines

PACTS Principles

Core Agents

The full fleet lives in .claude/agents/v3/qe-*.md; these are the agents most core jobs start with.

Coordination Patterns

Hierarchical: fleet-commander → [generators] → [executors] → quality-gate
Mesh: test-gen ↔ coverage ↔ quality (peer decisions)
Sequential: risk-analyzer → test-gen → executor → coverage → gate

Success Criteria

✅ Deployment frequency rises without escaped defects rising (measured against your own baseline) ✅ Coverage gaps detected in real-time ✅ Bugs caught pre-production ✅ Agent claims checked against evidence before anyone relies on them ❌ Agents acting without human oversight on critical decisions ❌ Deploying the whole fleet at once (start with 1-2)

Core Concepts

QE Evolution

Core Premise: Agents amplify human expertise. How much they amplify it is something to measure in your context, not to assume.

Confidence, not trust: trust is a feeling; confidence can be explained. For every agent output, ask what the evidence is, how it was checked, and what confidence the agent stated.

Key Capabilities

1. Intelligent Test Generation

// Agent analyzes code change, generates targeted tests
const tests = await Task("Generate tests", prDiff, "qe-test-architect");
// → Happy path, edge cases, error handling tests

2. Pattern Detection - Scan logs, find anomalies, correlate errors

3. Adaptive Strategy - Adjust test focus based on risk signals

4. Root Cause Analysis - Link failures to code changes, suggest fixes

Completion Theater (the central failure mode)

Agents optimize for appearing done rather than being done, and it gets harder to spot as their output gets more plausible. Patterns to check for:

• Work reported complete that was never committed
• Tests that assert a status code without checking the payload, or check that variables exist instead of behavior
• Coverage inflated by tests that touch code without exercising it
• Validation that checks counts or thresholds instead of behavior
• Report volume standing in for working software
• Stubs accepted as implementations
• Agent claims that the visible evidence contradicts

An honest, low-confidence answer is safer than a confident fabricated one; a false-safe label on a real security issue is the worst outcome. Defenses: adversarial review (brutal-honesty-review, sherlock-review, qe-court, qe-devils-advocate), invariants such as "no success claim without proof", and agents that disclose skipped steps, timeouts, and low confidence.

Agent Coordination

Memory Namespaces

aqe/test-plan/*     - Test planning decisions
aqe/coverage/*      - Coverage analysis results
aqe/quality/*       - Quality metrics and gates
aqe/learning/*      - Patterns and Q-values
aqe/coordination/*  - Cross-agent state

Memory Operations (MCP Tools)

CRITICAL: Always use aqe memory store with persist: true for learnings.

1. Store data to persistent memory:

// Store test plan decisions (persisted to .agentic-qe/memory.db)
aqe memory store \
  --key "aqe/test-plan/pr-123" \
  --namespace "aqe/test-plan" \
  --value '{...}' \
  --json

2. Retrieve prior learnings before task:

// Query patterns before starting test generation
const priorData = await aqe memory get --key "aqe/learning/patterns/test-generation/*" --namespace "aqe/learning" --json
 
// Use patterns to guide current task
if (priorData.success) {
  console.log(`Loaded ${priorData.patterns.length} prior patterns`);
}

3. Store coverage analysis results:

aqe memory store \
  --key "aqe/coverage/auth-module" \
  --namespace "aqe/coverage" \
  --value '{...}' \
  --json

Three-Phase Memory Protocol

For coordinated multi-agent tasks, use the STATUS → PROGRESS → COMPLETE pattern:

// PHASE 1: STATUS - Task starting
aqe memory store \
  --key "aqe/coordination/task-123/status" \
  --namespace "aqe/coordination" \
  --value '{...}' \
  --json
 
// PHASE 2: PROGRESS - Intermediate updates
aqe memory store \
  --key "aqe/coordination/task-123/progress" \
  --namespace "aqe/coordination" \
  --value '{...}' \
  --json
 
// PHASE 3: COMPLETE - Task finished
aqe memory store \
  --key "aqe/coordination/task-123/complete" \
  --namespace "aqe/coordination" \
  --value '{...}' \
  --json

Blackboard Events

Example: PR Quality Pipeline

// 1. Risk analysis
const risks = await Task("Analyze PR", prDiff, "qe-regression-analyzer");
// 2. Generate tests for risks
const tests = await Task("Generate tests", risks, "qe-test-architect");
// 3. Execute + analyze
const results = await Task("Run tests", tests, "qe-parallel-executor");
const coverage = await Task("Check coverage", results, "qe-coverage-specialist");
// 4. Quality decision
const decision = await Task("Evaluate", {results, coverage}, "qe-quality-gate");
// → GO/NO-GO with rationale

Implementation Phases

Advance on evidence, not on the calendar. Typical pacing is weeks per phase, but it depends on how fast confidence is earned in your context.

Phase 1 Example

# Start: deploy a single agent
aqe agent spawn qe-test-architect
 
# Then: generate tests for a handful of real PRs
# Track: bugs found, test quality, review time, claims that failed verification
 
# Measure impact against your pre-agent baseline
aqe agent metrics qe-test-architect
# → Compare tests, bugs found and review time with the baseline; keep what the data supports

Limitations & Strengths

Agents Excel At

• Volume: Scan thousands of logs in seconds
• Patterns: Find correlations humans miss
• Tireless: 24/7 testing and monitoring
• Speed: Instant code change analysis

Agents Need Humans For

• Business context and priorities
• Ethical judgment and trade-offs
• Creative exploration ("what if" scenarios)
• Domain expertise (healthcare, finance, legal)

Best Practices

Autonomy Progression (gated by evidence)

Stage 1: Agent suggests → Human decides
Stage 2: Agent acts → Human reviews after      (when its suggestions consistently survive review)
Stage 3: Agent autonomous on low-risk work     (when post-hoc reviews stop finding real problems)
Stage 4: Agent handles critical with oversight (when the low-risk record and its disclosures hold up)

Agent Coordination Hints

coordination:
  topology: hierarchical
  commander: qe-fleet-commander
  memory_namespace: aqe/coordination
  blackboard_topic: qe-fleet
 
preload_skills:
  - agentic-quality-engineering  # Always (this skill)
  - risk-based-testing           # For prioritization
  - quality-metrics              # For measurement
 
agent_assignments:
  qe-test-architect: [api-testing-patterns, tdd-london-chicago]
  qe-coverage-specialist: [quality-metrics, risk-based-testing]
  qe-security-scanner: [security-testing, risk-based-testing]
  qe-performance-tester: [performance-testing]

Related Skills

• holistic-testing-pact - PACTS principles deep dive
• risk-based-testing - Prioritize agent focus
• quality-metrics - Measure agent effectiveness
• api-testing-patterns, security-testing, performance-testing - Specialized testing

Resources

• Agent definitions: .claude/agents/
• CLI: aqe agent --help
• Fleet status: aqe fleet status

Success Metric: Ship more often with the same or better quality through agent collaboration, and be able to show the evidence for both.