mejbaurbahar/fagun/cicd-testing

CI/CD Testing

Test tiering (don't run everything on every PR)

• PR gate (minutes): unit tests + fast API contract checks + a thin smoke E2E suite covering the money path. This blocks merge — keep it fast and reliable or people start bypassing it.
• Pre-deploy / merge-to-main: fuller regression suite, still bounded (10-20 min).
• Nightly / scheduled: full regression, cross-browser matrix, performance baseline, security scan (SAST/DAST) — anything too slow to gate a PR.
• Post-deploy smoke: a small suite that runs against the just-deployed environment before traffic fully shifts, to catch environment-specific breakage config/secrets differences that staging didn't have.

Quality gates

Define explicit, machine-checkable pass/fail criteria per stage, not "look at the dashboard":

• Merge gate: 0 failing required checks, coverage delta not regressed below threshold (if coverage-gating is used — see the caution about coverage-as-a-metric in [[qa-fundamentals]]).
• Deploy gate: regression suite green, no new Critical/Blocker security findings ([[security-testing]]), perf within budget ([[performance-testing]]).
• Make gates fail loud and specific — a gate that just says "checks failed" without linking the failing job wastes the next person's time.

Parallelization & sharding

• Split large suites across N workers by file/test-duration balancing (not just alphabetical file split, which produces uneven shard times) — most modern runners (Playwright, pytest-xdist) support this natively.
• Keep tests independent (no shared mutable fixture/test-order dependency) — sharding surfaces hidden ordering dependencies as flaky failures, which is actually useful for finding them.
• Cache dependencies (node_modules, pip packages, browser binaries) between runs to cut pipeline time; invalidate cache correctly on lockfile change.

Flaky test handling

• Don't auto-retry-until-green and call it done — track a flakiness rate per test (fails X% of runs over N runs) and treat crossing a threshold as its own bug to investigate, per [[test-automation]]'s flaky-test triage.
• Quarantine a known-flaky test (mark it non-blocking, keep it running, alert on its pattern) rather than deleting it or leaving it silently blocking merges.
• Track flaky-test rate as a first-class pipeline health metric — a suite trending flakier is losing signal even if it's still "green."

Artifact & evidence collection

• On failure, always retain: screenshots, video/trace (Playwright trace: 'retain-on-failure'), console logs, network HAR, the exact commit SHA and environment. Without this, a CI failure is unreproducible days later.
• Set explicit retention windows — indefinite artifact storage silently becomes an infra cost problem.

Notifications & bug creation

• Route failures to the owning team/channel automatically (based on changed files or test tags), not a firehose channel everyone mutes.
• Auto-create/update a tracking ticket on a new regression-suite failure (not on every transient CI blip) — link the failing run, commit range, and evidence.
• De-duplicate: if the same test has an open ticket, comment/bump it rather than opening a new one every run — see [[qa-code-intelligence]] for duplicate-detection approach.

Platform notes

• GitHub Actions: matrix builds for cross-browser/OS; actions/cache; required status checks on branch protection; reusable workflows to avoid duplicating pipeline logic across repos.
• GitLab CI: rules:/only:changes: to scope jobs to relevant file changes; DAG (needs:) to parallelize instead of purely sequential stages.
• Jenkins: declarative pipelines over scripted for readability/maintainability; shared libraries for common QA steps across multiple pipelines.

Reporting

When a pipeline change itself introduces a regression (e.g. a gate that stopped actually blocking), treat it with the same severity discipline as a product bug — a broken quality gate is a Critical finding because it silently removes a safety net. See [[bug-reporting]].