tercel/spec-forge/test-plan-generation

Test Plan Generation Skill

What Is a Test Plan?

A Test Plan is the master document that defines the scope, approach, resources, and schedule of all testing activities for a software project or feature. As specified by IEEE 829 (Standard for Software and System Test Documentation), a test plan communicates the intent of testing to all stakeholders and provides a framework for organizing, tracking, and evaluating test efforts. It answers fundamental questions: what will be tested, how it will be tested, who will test it, when testing will happen, and what criteria determine whether testing is complete.

Within the software development lifecycle a Test Plan sits downstream of the Software Requirements Specification (SRS) and Technical Design documents. It translates functional and non-functional requirements into concrete, verifiable test cases. A well-constructed Test Plan reduces the risk of undetected defects reaching production, provides measurable quality gates, and serves as the contractual agreement between development, QA, and product teams on what "done" means from a quality perspective.

This skill treats the Test Plan as a living artifact. It is authored once, but it evolves as requirements change, new risks emerge, and test execution reveals areas needing deeper coverage.

Six-Step Workflow

Every Test Plan generated by this skill follows a disciplined six-step process. Each step must be completed before moving to the next.

Step 1 -- Scan Project and Test Infrastructure

Before writing any test documentation, scan the project to build situational awareness of both the application under test and the existing test infrastructure.

Fast path (preferred): resolve <sf_scripts> (see @../shared/scripts.md) and run python3 "<sf_scripts>/sf-scan.py" --root "<project_root>". Its JSON supplies the tech stack, frameworks, DB/auth signals, test.framework/test.command, source_tree, and the existing-doc inventory in one pass — use those directly. Fall back to the manual scan below (silently) only when python3 is unavailable.

1. Glob the project tree to discover the repository structure, source modules, and naming conventions. Use patterns such as **/*.md, **/package.json, **/pyproject.toml, **/*.test.*, **/*.spec.*, **/__tests__/**, or language-specific test directories to map the landscape.
2. Read the README (or equivalent entry-point documentation) to understand the project's purpose, architecture, and deployment model.
3. Identify existing test frameworks and tooling by scanning configuration files such as jest.config.*, pytest.ini, vitest.config.*, .github/workflows/*, Makefile, or docker-compose.test.yml. Understanding what test infrastructure already exists prevents duplicated effort and ensures the plan integrates with the team's workflow.
4. Catalog the tech stack, runtime environments, and external dependencies so the test environment section of the plan is accurate and complete.

Step 1.5 -- Doc-First Discipline (Mandatory)

Before locating upstream documents or writing any test plan content, the doc-first discipline applies. Read the existing test documentation (test-plan.md, qa.md, testing-strategy.md, README test sections), identify what already covers the strategy, scope, environment, or process, and decide for each section whether to REUSE (reference existing), EXTEND (edit existing in place), or NEW (genuinely missing). The default for any topic that already has coverage in an existing test plan is to extend in place — never to create parallel test-plan-v2.md files, never to append ## Update blocks, never to leave deprecated sections strikethrough'd.

The full discipline, the four rules, the pre-generation checklist, and the anti-patterns to avoid:

@../shared/doc-first.md

You MUST run the pre-generation checklist (the five questions) from doc-first.md before proceeding to Step 2. If an existing test plan already covers any of the topics this generation will discuss, you must:

1. List the existing files and sections that overlap (with file:line references)
2. Decide REUSE / EXTEND / NEW for each section (scope, strategy, environment, schedule, defect management)
3. Bias toward EXTEND — opening the existing test plan and modifying it — over NEW
4. Identify which existing references (to test infrastructure, CI configs, requirement IDs) must remain accurate
5. Surface any downstream artifacts (test-cases docs, test code) this generation will affect

If the project shows signs of doc drift (multiple test-plan-*.md files, contradictory strategies, obvious duplication), warn the user and recommend /spec-forge:analyze or /spec-forge:propagate first.

If a usable existing test plan already covers most of what the user is asking for, the right action is edit it in place, not generate a new file.

Step 2 -- Find Upstream SRS and Tech Design Documents

The Test Plan must trace every test case back to a requirement. The sf-scan.py inventory from Step 1 already lists upstream docs under document_index (srs/tech-design/prd) with the requirement IDs each declares (declared_ids); use it instead of re-globbing.

1. Locate upstream docs from the scan's document_index (or, as a fallback, glob docs/ for */srs.md, */tech-design.md, */prd.md).
2. Extract requirement IDs (e.g., FR-XXX-NNN, NFR-XXX-NNN): use python3 "<sf_scripts>/sf-trace.py" ids "<srs.md>" (or the scan's declared_ids) rather than reading them out by hand — these seed the traceability matrix in Step 5.
3. Extract component and module names from the Technical Design to inform test module organization and integration test boundaries.
4. If no upstream documents are found, inform the user and ask whether to proceed with a standalone test plan or wait until requirements documentation is available.

Step 3 -- Clarify Only What You Cannot Infer

After scanning, infer everything you can from the codebase, the upstream SRS/Tech Design, and any existing test plan — and state those inferences as assumptions the user can correct. Then ask only the questions whose answers materially change the plan and cannot be inferred. Typical areas that genuinely need the user (they are rarely in the code) include:

• The scope of testing: which features, modules, or user flows must be covered.
• Quality objectives: what level of confidence is required before release.
• Known risk areas: parts of the system that are fragile, recently changed, or historically defect-prone.
• Non-functional requirements: performance targets, security compliance requirements, compatibility matrix (browsers, devices, OS versions).
• Test environment constraints: availability of staging environments, test data sensitivity, third-party service sandboxes.
• Timeline and resource constraints: how much calendar time is available for testing, and who will execute tests.
• Automation expectations: what percentage of tests should be automated, and what frameworks are preferred.

Proceed to generation once you have enough to fill the core sections — on your inferred defaults (stated explicitly) where the user has not weighed in — rather than blocking on a full questionnaire. Capture any remaining unknowns as open questions in the plan.

Step 4 -- Generate Test Plan

Using the answers from Step 3 and the context from Steps 1 and 2, generate the full Test Plan by filling in every applicable section of references/template.md. Follow the writing guidelines and standards described in the sections below. Generate all Mermaid diagrams inline. Assign test case IDs, priorities, and types as you go.

Real DB enforcement gate: After generating all test cases, scan every test case's DB Approach field. Any test case that performs a Create, Read, Update, or Delete operation against the project's own database MUST have DB Approach: Real DB. If you find Mock for a database-touching test case, rewrite it to use TestContainers or a real test database instance. Mocks are only acceptable for external third-party services the project does not control.

Step 5 -- Build Traceability

After generating the Test Plan, construct the Requirements Traceability Matrix (RTM). Map every SRS requirement ID (both functional and non-functional) to one or more test case IDs. Let the script layer do the coverage math: run

python3 "<sf_scripts>/sf-trace.py" matrix --upstream "<srs.md>" --downstream "<test-plan.md>"

and use its uncovered_upstream (requirements with no test case — the gaps to flag) and orphan_downstream_refs (test references to requirement IDs that do not exist). You decide whether each gap is intentional; the script does the bipartite counting. Fall back to building the matrix by hand if the script is unavailable. The RTM is the primary mechanism for proving that testing is complete and aligned with the specification.

Step 6 -- Quality Check

Run the structural gate first — python3 "<sf_scripts>/sf-verify-doc.py" "<test-plan.md>" --strict deterministically checks the title, sections, TC ID format/uniqueness, and leftover placeholders — and fix everything it flags. Then validate the judgement items against references/checklist.md (the parts the script cannot judge). Summarize the checklist results so the user can see what passed and whether any items were intentionally skipped (with justification). (Run the checklist by hand if python3 is unavailable.)

Test Strategy Design -- The Test Pyramid

The test strategy follows the well-established test pyramid model, with one critical modification: any test that touches the database must use a real database, not a mock.

The Modified Test Pyramid

1. Unit Tests (Pure Logic) form the base for code with no database dependency — business calculations, input validations, data transformations, utility functions. These are fast, isolated, and need no mocks or stubs. They are the cheapest to write and maintain.
2. Unit Tests (DB-Touching) cover any method that reads or writes to the database — repositories, services with queries, data access layers. These MUST use a real database (TestContainers, in-memory test DB, or dedicated test instance). Mocking the database at this level gives false confidence: tests pass, but SQL syntax errors, constraint violations, ORM mapping bugs, and transaction behavior go undetected until production.
3. Integration Tests verify that components work together — API endpoints processing requests through to database, message queue consumers persisting data, service-to-service calls. Always use a real database. Mock only external third-party services you don't control.
4. System Tests validate the application as a whole in a production-like environment with a real database, real cache, and real message queues.
5. Acceptance Tests sit at the peak, verifying business requirements with real data scenarios in staging or production-like environments.

Why Real Database Over Mocks

Mocking the database hides real bugs:

• SQL syntax and query logic errors — the mock doesn't execute SQL, so wrong queries pass
• Constraint violations — unique keys, foreign keys, NOT NULL constraints are invisible to mocks
• Transaction behavior — rollback on error, isolation levels, deadlocks don't exist in mocks
• ORM mapping bugs — field name mismatches, type conversions, lazy loading issues go undetected
• Index and performance issues — mocks return instantly regardless of query efficiency
• Database-specific behavior — NULL handling, string collation, timezone conversion, JSON operators differ between databases