mhylle/claude-skills-collection/e2e-testing

E2E Testing Skill

<!-- contents: generated by tests/skill_structure.py --write; edit headings, not this block -->

Contents

• Overview
• Prerequisites
• Mode Selection
• Setup Mode
• Entry Points
• Setup Workflow
• Run Mode
• CRITICAL: Test Status Integrity
• Pre-Run Checks
• Execution Protocol
• Flexibility Criteria Evaluation
• Evidence Bundle
• History Integration
• Report Mode
• Report Types
• Report Presentation
• Quality Checklist
• Setup Mode
• Run Mode
• Report Mode
• Resources
• references/
• Report Templates

Reference files (open one when a step points to it):

• references/flexibility-criteria-guide.md: Flexibility Criteria Guide
• references/history-schema.md: Test History Schema Reference
• references/test-regime-schema.md: Test Regime Schema Reference

<!-- /contents -->

Overview

A comprehensive E2E testing skill using Playwright MCP for systematic testing of any web-based system. The skill:

• Observes and reports - Never fixes issues, only documents them
• Discovers paths - Finds undocumented functionality at runtime
• Tracks history - Identifies flaky areas and suggests variations
• Produces dual reports - Human-readable and machine-readable formats

Prerequisites

Before using this skill, verify Playwright MCP is available:

1. Check for playwright in MCP server configuration
2. If missing, add to Claude settings:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

If Playwright MCP is unavailable, inform the user and provide setup instructions before proceeding.

Mode Selection

This skill operates in three modes. Determine mode from user request:

If unclear, ask: "Would you like to set up a test regime, run existing tests, or view reports?"

Setup Mode

Purpose: Create or update test regime through interactive discovery.

Entry Points

Determine entry point from user context:

Setup Workflow

Step 1: Gather Initial Context

Ask for any missing information:

• URL: Base URL of the application
• Purpose: What does this system do? (1-2 sentences)
• Key workflows: What are the critical user journeys?
• Existing docs: Any README, user stories, or specs?

Step 2: Explore Application

Use Playwright MCP to explore:

Navigate to base URL
Capture accessibility snapshot
Identify:
  - Navigation elements (menus, links)
  - Interactive elements (buttons, forms)
  - Key pages and sections

For each discovered element, note:

• Element type and purpose
• Alternative paths to reach it
• Required preconditions (login, etc.)

Step 3: Discover Alternative Paths

While exploring, actively look for:

• Multiple ways to accomplish the same goal
• Hidden or non-obvious functionality
• Edge cases in navigation

Document discoveries as: "Found alternative: [description]"

Step 4: Define Test Scenarios

For each key workflow, create scenario with:

scenario: [Descriptive name]
description: [What this tests]
preconditions:
  - [Required state before test]
blocking: [true/false - does failure prevent other tests?]
steps:
  - action: [navigate/click/type/verify/wait]
    target: [selector or description]
    value: [input value if applicable]
    flexibility:
      type: [exact/contains/ai_judgment]
      criteria: [specific rules or judgment prompt]
success_criteria:
  - [What must be true for pass]
alternatives:
  - [Alternative path if primary fails]

Step 5: Create Test Regime File

Write regime to tests/e2e/test_regime.yml:

# Test Regime: [Application Name]
# Created: [YYYY-MM-DD]
# Last Updated: [YYYY-MM-DD]
 
metadata:
  application: [Name]
  base_url: [URL]
  description: [Purpose]
 
global_settings:
  screenshot_every_step: true
  capture_network: true
  capture_console: true
  discovery_cap: 5  # Max new paths to discover per run
 
blocking_dependencies:
  - scenario: login
    blocks: [profile, settings, checkout]  # These won't run if login fails
 
scenarios:
  - scenario: [name]
    # ... scenario definition

Step 6: Present Regime for Review

Show user:

1. Summary of discovered scenarios
2. Blocking dependencies identified
3. Alternative paths found
4. Ask for confirmation or modifications

Run Mode

Purpose: Execute tests sequentially with full evidence capture.

CRITICAL: Test Status Integrity

Principle: No Invalid Skips

A test should only have three outcomes:

Valid Reasons to Skip a Test

• Test environment unavailable (database down, service unreachable)
• Explicit @skip decorator for documented WIP features with ticket reference
• Platform-specific tests running on wrong platform
• External dependency unavailable (third-party API down)

Invalid Reasons to Skip (Mark as FAILED Instead)

Rationale

The purpose of a test is to fail when something doesn't work. Marking missing features or unexecuted tests as "skipped" produces artificially inflated pass rates and hides real issues. A test report that only shows green isn't useful if it achieved that by ignoring problems.

Implementation Examples

When a test cannot find the expected UI element or feature:

Status: FAILED
Notes: "FAILED: Expected 'Add to Cart' button not found. Feature not implemented or selector changed."

When a test is not fully executed:

Status: FAILED
Notes: "FAILED: Test not executed. Checkout flow verification was not completed - stopped at payment step."

When environment is genuinely unavailable (valid skip):

Status: SKIPPED
Notes: "SKIPPED: Payment gateway sandbox unavailable. Ticket: PAY-123"

Pre-Run Checks

1. Verify regime exists: Check for tests/e2e/test_regime.yml

• If missing: "No test regime found. Would you like to run Setup mode first?"

1. Load history: Check for tests/e2e/test_history.json

• If exists: Note previously flaky scenarios for extra attention

1. Verify Playwright MCP: Confirm browser automation is available

Execution Protocol

Rule 1: Always Start from Beginning

Every test run starts fresh from Step 1 of Scenario 1. Never skip steps or use cached state.

Rule 2: Sequential Execution

Execute scenarios in order. For each scenario:

1. Check preconditions
2. Execute each step:
   a. Perform action via Playwright MCP
   b. Capture screenshot
   c. Capture DOM state
   d. Capture network activity
   e. Capture console logs
   f. Evaluate success using flexibility criteria
3. Record result (pass/fail/blocked/skipped)
   - PASS: Step completed successfully
   - FAIL: Step failed OR element not found OR feature missing
   - BLOCKED: Dependent on a failed blocking scenario
   - SKIPPED: Only for valid environmental reasons (see Test Status Integrity)
4. If failed: Try alternatives if defined
5. If blocking failure: Stop dependent scenarios

Rule 3: Failure Handling

When a step fails:

1. Mark as failed - Record failure with evidence
2. Try alternatives - If alternatives defined, attempt them
3. Assess blocking impact:

• Check if this scenario blocks others
• If blocking: Mark dependent scenarios as "blocked"
• If non-blocking: Continue to next scenario

1. Never fix - Document the issue, do not attempt repairs

Rule 4: Runtime Discovery

While executing, watch for undocumented paths:

• New navigation options not in regime
• Alternative ways to complete actions
• Unexpected UI states

For discoveries:

1. Queue for testing (up to discovery_cap limit)
2. Execute after all defined scenarios complete
3. Document findings in report

Flexibility Criteria Evaluation

For each success check, apply the configured flexibility type:

For ai_judgment, provide confidence level:

• High: Clear success/failure
• Medium: Likely success/failure but some ambiguity
• Low: Uncertain, recommend manual review

Evidence Bundle

For each step, capture and store:

evidence/
  scenario-name/
    step-01/
      screenshot.png
      dom-snapshot.html
      network-log.json
      console-log.txt
      accessibility-snapshot.yaml

History Integration

After run completes:

1. Compare to previous runs:

• Same scenario passed before but failed now? Flag regression
• Same scenario failed before? Note persistent issue
• Intermittent pass/fail? Mark as flaky

1. Update history file:

{
  "runs": [
    {
      "timestamp": "ISO-8601",
      "scenarios": {
        "scenario-name": {
          "result": "pass|fail|blocked|skipped",
          "result_notes": "Details about the result",
          "duration_ms": 1234,
          "steps_completed": 5,
          "confidence": "high|medium|low",
          "discoveries": []
        }
      }
    }
  ],
  "flaky_scenarios": ["scenario-1", "scenario-2"],
  "suggested_variations": [
    {
      "scenario": "login",
      "variation": "Test with special characters in password",
      "reason": "Failed 3/10 runs with complex passwords"
    }
  ]
}

Result status rules (see Test Status Integrity):

• pass: Feature works as specified
• fail: Feature doesn't work, doesn't exist, or test incomplete
• blocked: Depends on failed blocking scenario
• skipped: ONLY for valid environmental reasons (with ticket reference)

1. Generate variations for flaky areas:

• If scenario failed 3+ times in last 10 runs: Auto-suggest new test variations
• Add to suggested_variations in history

Report Mode

Purpose: Generate actionable reports from test results.

Report Types

Generate both reports after every run:

Human-Readable Report

Output to tests/e2e/reports/YYYY-MM-DD-HHmmss-report.md:

# E2E Test Report: [Application Name]
 
**Run Date**: YYYY-MM-DD HH:mm:ss
**Duration**: X minutes
**Result**: X passed, Y failed, Z blocked, W skipped
 
## Summary