mgifford/accessibility-skills/ci-cd

CI/CD Accessibility Skill

Canonical source: examples/CI_CD_ACCESSIBILITY_BEST_PRACTICES.md in mgifford/ACCESSIBILITY.md

This skill is derived from that file. When in doubt, the example is authoritative.

The canonical guide states synchronization with this skill is not automatic

— re-check this file against canonical periodically.

Apply these rules when adding, reviewing, or maintaining CI/CD accessibility checks.

For guidance on when to use AI skills versus automated scanners during audits, see AI_AUDIT_GUIDE.md in this repository.

Core Mandate

CI can prevent known accessibility regressions, preserve tested semantics, and produce evidence for human review. It cannot establish WCAG conformance by itself. Automated rules, scores, snapshots, and AI output are inputs to an accessibility evaluation, not substitutes for one. This guide targets WCAG 2.2 AA for in-scope content — that target is not itself a conformance claim. A confirmed AAA-level finding under this AA target is obligation: aspirational by default (a visible, non-blocking stretch goal), not advisory, unless a project explicitly elevates that specific criterion to required.

Principles: test complete user tasks and relevant states, not only page loads; run fast deterministic checks locally and repeat in CI; block known regressions using explicit reviewable rules; keep accessibility, performance, security, and functional signals distinct; test the final rendered/sanitized/ optimized output; keep humans responsible for severity, alternatives, exceptions, and release decisions; give workflows and agents the least authority needed for their task.

Severity Scale (this skill)

Tool labels (axe `critical`/`serious`/`moderate`/`minor`, Lighthouse scores) are triage inputs, not project severity — map them into this taxonomy only after evaluating actual user impact on the affected task.

A Layered Testing Model

No single tool covers accessibility requirements. Choose layers by the affected task and the likelihood/consequence of a regression:

Use skills/manual-testing/SKILL.md for checks automation cannot complete.

Critical: Define the CI Contract Before Choosing Tools

For each check, document: the user task/page/component/states covered; the command and config file; tool/browser/runtime/rule-set versions; whether the result warns, blocks, or only produces an artifact; the baseline/expected result; the owner and response process for failures; known false positives and untested requirements; and evidence retained for review.

Do not describe a check as a merge gate unless branch protection actually requires it. Do not describe a warning as a failure.

Scores are not conformance percentages. A Lighthouse accessibility score of 100 does not mean 100% WCAG conformance, and 90 does not mean a page is 90% accessible — it measures the weighted audits Lighthouse ran in one configuration. Use individual audit results to investigate regressions. A score threshold can be an additional signal but should not be the only gate; also fail on specific rules/tasks the team has decided are blocking. Keep performance budgets separate — a performance score must not raise or lower accessibility severity.

{
  "ci": {
    "collect": { "staticDistDir": "./_site", "numberOfRuns": 1 },
    "assert": {
      "assertions": { "categories:accessibility": ["warn", { "minScore": 0.9 }] }
    },
    "upload": { "target": "filesystem", "outputDir": ".lighthouseci" }
  }
}

This warns; it does not block. Raising the threshold (or moving to "error") can be useful after investigation, but the threshold still does not establish conformance.

Critical: axe-core on Every PR, Across States and Preferences

Scan meaningful states after the page is ready AND after relevant interaction — do not scan only the initial route if users must open a menu, submit a form, or complete another action.

import AxeBuilder from '@axe-core/playwright';
import { expect, test } from '@playwright/test';
const colorSchemes = ['light', 'dark'] as const;
for (const colorScheme of colorSchemes) {
  test(`home page in ${colorScheme} mode`, async ({ page }) => {
    await page.emulateMedia({ colorScheme });
    await page.goto('/');
    await expect(page.getByRole('main')).toBeVisible();
    const results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'])
      .analyze();
    expect(results.violations).toEqual([]);
  });
}
test('navigation is tested while expanded', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('button', { name: 'Menu' }).click();
  await expect(page.getByRole('navigation', { name: 'Main' })).toBeVisible();
  const results = await new AxeBuilder({ page }).analyze();
  expect(results.violations).toEqual([]);
});

A clean scan means no violations were found in the tested scope — it does not mean no barriers exist. Missing axe-core checks on PRs is Critical; missing a color-scheme pass is also a coverage gap since theme-specific regressions often only appear in the untested scheme.

Cover applicable combinations without an unnecessarily expensive matrix: desktop and narrow reflow viewports; system/light/dark theme; default and increased contrast; forced colors; reduced motion; default/hover/focus/active/ disabled/loading/success/error states; signed-in/signed-out/timeout/ permission states. Use pairwise or risk-based coverage for routine changes; test every combination for high-risk components (authentication, payment, safety, core navigation).

Serious: Semantic Assertions and ARIA Snapshots

Role and accessible-name queries provide useful contracts:

test('contact form exposes its controls and error', async ({ page }) => {
  await page.goto('/contact');
  const email = page.getByRole('textbox', { name: 'Email address' });
  await expect(email).toBeVisible();
  await page.getByRole('button', { name: 'Send message' }).click();
  await expect(page.getByText('Enter an email address')).toBeVisible();
  await expect(email).toHaveAttribute('aria-invalid', 'true');
});
test('main navigation retains its semantic structure', async ({ page }) => {
  await page.goto('/');
  await expect(page.getByRole('navigation', { name: 'Main' }))
    .toMatchAriaSnapshot(`
      - navigation "Main":
        - list:
          - listitem:
            - link "Home"
    `);
});

Review snapshot changes; do not update them mechanically. An ARIA snapshot represents the accessibility tree, not a transcript of what every screen reader will announce — speech output also depends on the AT product, browser, OS, settings, mode, and user action.

Guidepup virtual screen reader (unit-level, exact spoken output):

import { virtual } from '@guidepup/virtual-screen-reader';
it('announces the dialog title and action buttons', async () => {
  document.body.innerHTML = `
    <dialog open aria-labelledby="dlg-title">
      <h2 id="dlg-title">Confirm deletion</h2>
      <button>Delete</button>
    </dialog>`;
  await virtual.start({ container: document.body });
  const spoken = await virtual.spokenPhraseLog();
  expect(spoken).toContain('Confirm deletion');
  await virtual.stop();
});

Serious: Visual and Preference Regression Tests

Visual tests can find clipped text, missing focus indicators, theme-specific contrast changes, and reflow regressions. Keep separate reviewed baselines for materially different themes and forced-colors behavior. Do not use screenshot equality as the only contrast test — anti-aliasing, transparency, gradients, and rendering differences make pixel comparison unreliable; confirm computed colors and visually inspect where contrast is critical. Emulate reduced motion for deterministic tests, but never remove an animation from the product merely to make a screenshot pass.

Critical: Risk Indicators and Build Gates

Behavioral checks such as Reflow risk and Focus Visible testing (see skills/behavioral-a11y/SKILL.md) produce indicators, not conformance verdicts. A small per-target false-positive probability compounds across many pages, components, states, or builds, and website findings are frequently correlated across shared templates and components. Converting every indicator into a blocking build failure produces noisy, low-trust CI and encourages bypassing the check rather than fixing the underlying pattern.

Do not prescribe automatic blocking merely because a result is machine-readable. A first-seen risk indicator has not been reviewed and has not ruled out a normative exception (for example, the SC 1.4.10 two-dimensional-layout exception); blocking on it treats an unreviewed signal as a confirmed failure.

Map "default CI treatment" onto policy.handling: a deterministic confirmed failure or reviewed regression is report (and may block); a risk indicator or cantTell is review, never suppress, until a human confirms or rejects it; a reviewed exception is suppress only with the full suppression detail below. handling is independent of obligation — do not use CI treatment as a stand-in for whether the underlying standard is required. Never disable an entire rule or engine because one target is noisy; suppress the specific, scoped finding instead. See Accessibility Finding Tracking: Policy Classification.

Fleet-wide and site-wide scans additionally require: separating unique root-cause clusters from affected instances and reporting both — not one issue per affected URL; comparing new findings against a reviewed baseline before deciding severity; recording the detector, browser, and baseline versions used for the scan; giving every reviewed exception an owner, a reason, and an expiry date; and periodically sampling apparently clean results to investigate false negatives, since a check with no findings has not been shown to have zero false negatives.

See skills/behavioral-a11y/SKILL.md for the fixture/PR/scheduled-scan cadence and the definitions of unique finding, affected instance, affected page, and root-cause cluster used above.