suriyel/longtaskforagent/long-task-st

System Testing — Cross-Feature & System-Wide Verification Before Release

Run cross-feature and system-wide testing after all features are implemented and passing. Per-feature ST test cases (functional, boundary, UI, security) have already been executed during each Worker cycle via long-task-feature-st. This phase focuses on what per-feature testing cannot cover: cross-feature interactions, multi-feature E2E workflows, system-wide NFR verification, compatibility, and exploratory testing.

Announce at start: "I'm using the long-task-st skill. All features are passing — time for cross-feature system testing."

Core principle: Per-feature ST test cases prove individual features meet their requirements. System testing proves the whole works together across feature boundaries.

<HARD-GATE> Do NOT skip any applicable test category. A "Go" verdict requires evidence from EVERY category that applies to this project. "It probably works" is not evidence. </HARD-GATE>

<HARD-GATE> Manual test escalation: When any test scenario in Steps 3-8 cannot be automated (requires physical device, human visual judgment, or external human action), use AskUserQuestion to present the test to the human and collect their verdict.

Format:

Manual Verification Required: {test description}
Category: {ST step — e.g., Integration, E2E, NFR, Compatibility}
What to verify: {specific check with acceptance criteria}
Expected result: {threshold or criterion from SRS}
 
Please report: PASS or FAIL on first line, then what you observed.
To skip: respond SKIP {reason}

• Parse response: first line PASS/FAIL/SKIP. If unparseable, re-prompt once; then record as BLOCKED.
• Record results as MANUAL-PASS or MANUAL-FAIL in the ST report.
• MANUAL-FAIL has the same severity implications as automated failure — it blocks Go verdict for Critical/Major items.
• SKIP {reason} records as BLOCKED with the reason (not silently skipped).

</HARD-GATE>

Checklist

You MUST create a TodoWrite task for each step and complete them in order:

1. ST Readiness Gate

python scripts/check_st_readiness.py feature-list.json

• All features have "status": "passing" — if any failing, invoke long-task:long-task-work instead
• SRS document exists (docs/plans/*-srs.md); Design document exists (docs/plans/*-design.md)
• Load config values if applicable — activate environment per long-task-guide.md; if the project uses a file-based config, ensure it is sourced before running the checks
• Start ST runtime services: Start services using commands from env-guide.md (skip if CLI/library only)
• Read env-guide.md — use "Start All Services" section; run each command with output redirect:

``bash [start command] > /tmp/svc-<slug>-start.log 2>&1 & sleep 3 head -30 /tmp/svc-<slug>-start.log ``

• Extract PID and port from the first 30 lines
• Run "Verify Services Running" health checks from env-guide.md — must respond before proceeding
• If start fails: check the log, diagnose root cause; try corrected commands (port conflict, env vars, missing deps); once a working command is confirmed update `env-guide.md` (Services table + Start command); if the fix needs >2 steps, extract to scripts/svc-<slug>-start.sh and reference from env-guide.md
• Record info: PIDs and ports in task-progress.md — required for Step 11 cleanup
• Read feature-list.json — note tech_stack, quality_gates, constraints[], assumptions[]
• Read SRS — extract all FR-xxx, NFR-xxx, IFR-xxx, CON-xxx requirements; read Stakeholders, User Personas, and Glossary sections
• Read Design doc — extract architecture, API design, testing strategy (§9), third-party dependencies (§8)
• If UI features exist: read UCD doc (docs/plans/*-ucd.md)
• Read task-progress.md — session history context

2. ST Plan

Create docs/plans/YYYY-MM-DD-st-plan.md with:

2a. Test Scope

2b. Requirements Traceability Matrix (RTM)

Map EVERY SRS requirement to ST test approach. Reference per-feature test case documents from Worker Step 9:

| Req ID | Requirement | Feature ST Status | System ST Category | ATS Categories | Test Approach | Priority |
|--------|-------------|-------------------|--------------------|----------------|---------------|----------|
| FR-001 | ... | docs/test-cases/feature-1-xxx.md (PASS) | E2E | FUNC,BNDRY,SEC | Scenario: ... | High |
| NFR-001 | ... | docs/test-cases/feature-5-xxx.md (PASS) | Performance | PERF | Load test: ... | Critical |
| IFR-001 | ... | N/A (cross-feature) | Integration | FUNC,BNDRY | Contract test: ... | High |

Every FR-xxx, NFR-xxx, IFR-xxx must appear in the RTM. Requirements without a test approach = gap.

If any requirements require manual verification (from ATS 自动化可行性 column or Feature-ST manual cases), include a Manual column in the RTM:

| Req ID | ... | Manual | Test Approach | Priority |
|--------|-----|--------|---------------|----------|
| FR-010 | ... | Yes: visual-judgment | Manual visual verification via AskUserQuestion | High |

ATS compliance gate (if ATS document exists):

python scripts/check_ats_coverage.py docs/plans/*-ats.md --feature-list feature-list.json --strict

Must exit 0. Any ATS category gap = finding to resolve before proceeding.

2c. Entry / Exit Criteria

Entry (must ALL be true): all features passing, environment provisioned, all required configs present.

Exit (must ALL be true for Go verdict): all regression/integration/E2E tests pass, all NFR thresholds met with measured evidence, no Critical or Major defects open, RTM shows 100% requirement coverage, ATS category compliance verified (if ATS exists: check_ats_coverage.py --strict exits 0).

2d. Risk-Based Prioritization

1. Critical path — core user workflows, highest business impact
2. Integration boundaries — cross-feature data flows, API contracts
3. NFR thresholds — performance, security (highest technical risk)
4. Edge cases — boundary conditions, error recovery
5. Compatibility — platform/browser variations

3. Regression Testing

1. Run full project test suite using commands from long-task-guide.md

1. Verify ALL tests pass — zero failures, zero errors
2. Verify line and branch coverage thresholds met project-wide
3. Check for new warnings, deprecation notices, dependency conflicts
4. Any failure → STOP — this is a regression. Diagnose before proceeding.

Record: total tests, passed/failed, line/branch coverage vs thresholds.

3b. Full Mutation Regression

Run full-codebase mutation testing. Per-feature mutation during Worker cycles may have only scoped feature tests (when active features > mutation_full_threshold); this step verifies mutation score holds project-wide with the full test suite.

1. Get the mutation_full command from long-task-guide.md
2. Run full mutation testing (all source files, all tests)
3. Verify: mutation score >= quality_gates.mutation_score_min from feature-list.json
4. If surviving mutants found:

• Analyze: equivalent mutant (document + skip) vs real gap (add test → Major severity defect)
• If score below threshold → treat as regression defect (Major severity)

1. Record: mutation score, killed/survived/total, command used

See references/st-recipes.md section "Full Mutation Regression" for per-tool commands and result interpretation.

Record: mutation score vs threshold, surviving mutant count, tool output summary.

4. Integration Testing

Test cross-feature interactions. Read references/st-recipes.md for language-specific patterns and real-vs-contract test classification.

Terminology (see st-recipes.md §1 for details):

• Contract test = mock-based, verifies call signatures — supplementary, not sufficient
• Integration test = real-service, verifies actual data flow — required per boundary

<HARD-GATE> Every internal cross-feature boundary MUST have at least one real integration test (real DB, real HTTP, real file system). Contract tests (mocks) do NOT satisfy this gate.

For external third-party boundaries: if test credentials are available in required_configs or environment, write real integration tests. Otherwise, use contract tests and record the reason in the Mock Authorization column. </HARD-GATE>

For each pair of features sharing data, state, or API boundaries:

• Data flow: Feature A produces data → Feature B consumes it → verify data integrity end-to-end; shared DB/state consistency
• API contracts: internal API calls between modules — verify request/response schemas; error propagation; version compatibility
• Dependency chains: walk dependencies[] graph in feature-list.json; verify features work in dependency order; test each dependency edge

Classification table (include in ST plan):

Minimum per internal boundary:

• ≥1 real test that writes/reads through the real shared resource
• If real service cannot start: boundary is BLOCKED (not skipped) — diagnose via env-guide.md

External boundary protocol:

1. Check required_configs and environment for test credentials/sandbox environment
2. If available → write real integration tests (preferred)
3. If unavailable → use contract tests; record reason in Mock Authorization column

Write integration tests in tests/integration/ or tests/st/. Tag each test:

# Integration: Feature A → Feature B (shared DB) [Real]
def test_feature_a_data_consumed_by_feature_b():
    ...
# Contract: Feature C → External API [Contract]
def test_external_api_response_shape():
    ...

Run and record results per boundary.

4b. Full-Pipeline Smoke Test

Before E2E scenario testing, verify at least one complete data flow path through the entire system. This catches integration bugs that per-boundary tests miss.

<HARD-GATE> At least ONE smoke test must exercise a real end-to-end data path (input → processing → storage → retrieval → output) using only real services. No mocks. </HARD-GATE>

1. Identify the critical path — the single most important data flow through the system (e.g., "create entity → store → query → return")
2. Write one smoke test that:

• Starts from external input (API call, CLI command, UI action)
• Passes through all intermediate processing
• Persists to real storage (if applicable)
• Retrieves and verifies the persisted result
• Uses ONLY real services — no mocks

1. Run the smoke test against running services (started in Step 1)
2. If it fails → Critical severity defect — diagnose before proceeding to E2E

Scaling: