mgifford/accessibility-skills/critic

Critic Workflow

The Critic reviews accessibility decisions to ensure they meet professional standards and evidence requirements.

Purpose

The Critic's role is to:

1. Validate ARRM grounding - Verify that findings use real ARRM tasks with correct role assignments
2. Assess automation safety - Determine if deterministic fixes are appropriate
3. Check role engagement - Ensure appropriate professional perspectives are consulted
4. Evaluate evidence quality - Verify claims are supported and evidence types are distinguished
5. Handle perspectives responsibly - Identify user impacts and avoid simulating lived experience
6. Maintain decision quality - Ensure disagreements and unresolved risks are visible

When to Use

Use the Critic for any accessibility finding:

• After Planner assessment of a finding
• For manual review of implementation proposals
• When validating accessibility test results
• When cross-role decisions are being considered

The Critic works best after initial planning but before implementation begins.

When Not to Use

The Critic is not appropriate for:

• Simple technical fixes with clear requirements
• Quick accessibility checks where expert review isn't needed
• Routine validation without professional judgment concerns

Integration with Topic Skills

The Critic checks that AI workflow decisions respect relevant topic skills:

• forms → forms, content-design, keyboard
• keyboard → keyboard, touch-pointer, manual-testing
• color-contrast → color-contrast, light-dark-mode, visual-design
• charts-graphs → charts-graphs, image-alt-text, tables
• And other topic skills as appropriate

ARRM Task Validation

Grounding Requirements

For each accessibility finding, the Critic verifies:

1. Real ARRM Task Usage

• Task IDs must exist in the ARRM CSV
• Must use real ARRM task IDs, not placeholders like "IMG-001"
• Must match actual W3C WAI ARRM dataset

1. Role Assignment Accuracy

• Primary Ownership must match ARRM data
• Secondary Ownership must match ARRM data
• Contributor assignments must match ARRM data
• No invented or made-up role assignments

1. Completeness

• All relevant ARRM tasks should be identified
• No critical ARRM tasks should be omitted
• All role responsibilities should be assigned appropriately

Example Valid ARRM Usage:

{
  "arrm": {
    "task_id": "CSS-010",  // Real ARRM task ID
    "wcag_sc": "1.4.1",
    "level": "A",
    "primary": ["Visual Design"],
    "secondary": ["Content Authoring"],
    "contributors": []
  }
}

Example Invalid ARRM Usage:

{
  "arrm": {
    "task_id": "IMG-001",  // Placeholder ID
    "wcag_sc": "1.1.1",
    "primary": ["Content Authoring"],
    "secondary": ["UX Design"],  // Made-up role
    "contributors": ["Developer"]  // Not in ARRM
  }
}

Automation Safety Assessment

Safety Criteria

The Critic assesses whether an automated fix is appropriate based on:

1. Rule Certainty

• High rule certainty: Clear technical requirements with authoritative source
• Medium rule certainty: Partially clear requirements
• Low rule certainty: Unclear or conflicting requirements

1. Intent Certainty

• High intent certainty: Intended meaning and interaction clearly understood
• Medium intent certainty: Reasonable understanding with some ambiguity
• Low intent certainty: Intent unclear or uncertain

1. Change Risk

• Low change risk: No impact on meaning, workflow, policy, visual hierarchy
• Medium change risk: Moderate impact on one or more areas
• High change risk: Significant impact on functionality or user experience

1. Testability

• High testability: Can be reliably verified with automated tools
• Medium testability: Requires some manual verification
• Low testability: Requires extensive manual testing

Safety Decision Matrix

Safe Automated Fix Example

{
  "decision": {
    "state": "auto-fixable",
    "rule_certainty": "high",
    "intent_certainty": "high", 
    "change_risk": "low",
    "testability": "high"
  }
}

Unsafe Automated Fix Example

{
  "decision": {
    "state": "human-confirmation-needed",
    "rule_certainty": "high",
    "intent_certainty": "medium",  // Unclear what the tabindex should be
    "change_risk": "low",
    "testability": "high"
  }
}

Role Engagement Review

Question Quality Checks

The Critic verifies that questions are properly directed:

1. Primary Role Questions

• Address core responsibility ownership
• Focus on implementation choices within their domain
• Ask for justification of their preferred approach

1. Secondary Role Questions

• Consider broader impact of Primary's decision
• Evaluate interaction with their area of expertise
• Identify potential conflicts with their responsibilities

1. Contributor Role Questions

• Ensure the solution works across all relevant technical areas
• Validate implementation feasibility
• Check for integration with existing systems

Missing Professional Perspectives

The Critic identifies roles that should be involved but aren't:

{
  "missing_perspectives": [
    {
      "role": "Testing",
      "concern": "No keyboard testing planned",
      "recommendation": "Test with keyboard-only navigation",
      "urgency": "high"
    },
    {
      "role": "User Experience Design", 
      "concern": "Impact on cognitive load unclear",
      "recommendation": "User research needed for complexity assessment",
      "urgency": "medium"
    }
  ]
}

Role Conflict Documentation

When different roles suggest conflicting approaches:

1. Record Each Role's Position
2. Document Evidence Supporting Each View
3. Identify the Trade-off Being Made
4. Specify Who Should Make the Final Decision
5. Recommend Who Should Be Consulted
6. Note What Evidence Would Reduce Uncertainty

Evidence Assessment

Evidence Types and Quality

The Critic evaluates the quality and completeness of evidence:

1. Automated Evidence

• Axe-core scan results
• Automated testing tools output
• Static code analysis

1. Manual Evidence

• Keyboard testing results
• Screen reader testing notes
• Visual inspection findings

1. User Research Evidence

• Disabled user testing results
• User interviews and feedback
• Survey data

Evidence Quality Criteria

1. Sufficient Evidence

• Coverage of all relevant scenarios
• Adequate sample size for user testing
• Diverse testing conditions

1. Current vs. Historical

• Up-to-date testing methodology
• Current assistive technology versions
• Relevant for the target user population

1. Documented Process

• Clear reproduction steps
• Test conditions clearly stated
• Results recorded systematically

Perspective Handling

User Impact Analysis

The Critic examines likely impacts for specific user groups:

1. Screen-Reader Users

• Does the change affect information announcement?
• Are semantic landmarks preserved?
• Are navigation patterns disrupted?

1. Motor/Ergonomic Users

• Does the change affect target size or spacing?
• Are keyboard interactions maintained?
• Are gesture-based interactions accessible?

1. Visual Users

• Does the change affect color contrast?
• Are focus indicators preserved?
• Are visual relationships maintained?

1. Cognitive Users

• Does the change affect complexity or clarity?
• Are instructions clear and concise?
• Are error messages helpful?

Appropriate User Research Recommendation

User research should be recommended only when:

• Essential for decision: Cannot resolve uncertainty without direct user input
• High impact: Will significantly affect multiple user groups
• Insufficient internal evidence: Current evidence cannot determine the best approach

Simulating Lived Experience Detection

The Critic ensures it does not:

• Claim first-hand experience: Should not state "I tested with screen readers"
• Make universal claims: Should not say "People with visual impairments can't..."
• Generalize from single anecdote: Should not base conclusions on limited testing

Evidence Gathering Requirements

Decision Quality Review

Disagreement Documentation

When professional roles disagree:

1. Record Each Role's Position
2. Document the Evidence supporting each position
3. Identify the Core Trade-off
4. Specify Decision Authority
5. Recommend Consultation Process
6. Document What Evidence Would Reduce Uncertainty

Decision Owner Identification

The Critic determines who has ultimate decision authority:

Primary Role: Has final decision authority for tasks where they have Primary Ownership

Secondary Role: Can challenge Primary role decisions, especially when their expertise is relevant

Contributor Role: Provides technical implementation but defers to Primary/Secondary decisions

Rationale Documentation

Quality decisions include clear documentation of:

1. Decision Basis: What evidence and analysis supported the decision
2. Stakeholder Input: Which professionals were consulted and their recommendations

3.

2. Trade-offs Considered: What alternatives were evaluated and why they were rejected
3. Risk Assessment: What are the known risks and mitigation strategies

Unclear Risk Documentation

Transparency about uncertainty:

1. Known Limitations: What the decision cannot fully address
2. Monitoring Requirements: What should be tracked post-implementation
3. Revisit Triggers: Under what conditions the decision should be reviewed
4. Evidence Gaps: What information is still needed but unavailable

Structured Critic Output

The Critic produces structured output with the following sections: