10 — Accessibility Testing — Automate the Floor, Human-Test the Ceiling

"Run the Lighthouse audit, fix the red items, ship" was my accessibility testing plan, and it capped my coverage at a third. The split that fixed it: automated accessibility tools find roughly a third of issues — the detectable, rule-violating floor — while the remaining two-thirds, the ones that actually block real users, require human judgment against the accessibility tree and real assistive technology [1][2]. The automation isn't useless; it's just a floor, not a ceiling. Confusing the two is how teams ship a "100/100 Lighthouse" page that a screen-reader user still can't navigate.

The framing that landed is the split between rule violations and experience failures. A rule violation is "this image has no alt attribute" — a tool can detect it deterministically. An experience failure is "the focus order jumps confusingly when the modal opens" or "this heading structure makes the page incomprehensible when navigated by heading" — these require a human (often one using assistive tech) to evaluate. Tools catch the first category reliably; they're blind to the second.

The automated floor: what tools actually find

The roadmap points at three tools, and they're best understood as a layered set rather than alternatives:

• Axe (Deque) is a fast, lightweight engine that checks the document against a ruleset and reports violations, passes, and incomplete items [3]. Its distinctive virtue is that it's a library — axe-core can run inside your existing test automation (Cypress, Playwright, Selenium, Jest), so accessibility checks become part of the CI gate, not a separate manual audit.
• WAVE (WebAIM) is a suite of evaluation tools that identifies accessibility and WCAG errors and facilitates human evaluation of the surrounding context [4]. WAVE's visual overlay — icons on the page marking each issue — is designed for a human reviewer stepping through a page, which is why its own tagline emphasizes human evaluation alongside the automated checks.
• Chrome DevTools' accessibility panel adds the accessibility tree view (how assistive tech sees the page), a color-contrast checker, Lighthouse audits, and vision-deficiency emulation [5].

The shared limitation is the same one Deque publishes openly: automated tooling catches around 30% of accessibility issues. The other 70% — meaningful alt text (not just present alt text), logical heading order, sensible focus management, readable language — need a person.

The accessibility tree — how assistive tech actually sees the page

The single most useful idea in this whole area is the accessibility tree. Browsers don't expose the raw DOM to screen readers; they expose a parallel, cleaned-up tree where each node is an accessible object with a role, name, state, and value. Chrome DevTools lets me inspect this tree directly [5].

Reading the accessibility tree is the skill that separates "I ran the tool" from "I understand what a screen reader will do here." If a button the sighted user sees isn't in the accessibility tree — or is there with a wrong name or role — the screen-reader user has no way to reach it, regardless of how the page looks visually.

The human ceiling: the issues tools miss

The categories that automated tools consistently miss, and that require human evaluation against WCAG [2], include:

• Meaningful text alternatives. A tool flags a missing alt; it cannot tell whether a present alt="image1" is meaningful. Writing good alt text is a judgment call.
• Logical heading order. Tools can flag a skipped heading level (h1 → h4), but whether the heading structure makes sense when navigated linearly is a human call.
• Focus management. When a modal opens, focus should move into it; when it closes, focus should return to the trigger. Tools rarely catch focus-trap or focus-restore bugs.
• Keyboard operability. Can every interaction be reached and operated without a mouse, in a sensible order? This requires actually unplugging the mouse and trying.
• Color contrast at the edges. Tools catch gross contrast failures but miss contextual ones (text over a busy background image).
• Readable language and clear instructions. No tool evaluates whether the page's language is plain and the instructions are clear.

The honest practice is to combine layers: automated checks in CI (axe-core in every build), manual review against the accessibility tree (DevTools panel during development), and periodic testing with actual assistive technology (NVDA, VoiceOver, JAWS) and real users where possible.

WCAG as the shared standard

Underlying all of this is WCAG — the Web Content Accessibility Guidelines — organized around four principles (POUR): content must be Perceivable, Operable, Understandable, and Robust [2]. The guidelines under each principle are the shared vocabulary between automated rules and human judgment: a tool checks "does this image have alt text" (a Perceivable guideline), and a human checks "is that alt text meaningful" (the same guideline, applied with judgment). WCAG is what makes the floor and the ceiling part of the same building.

How I use this

The practical payoff is a layered routine rather than a single audit:

• In CI, on every PR: axe-core assertions in the existing test suite. This catches the rule-violating floor automatically and prevents regressions — a new component without a label fails the build.
• During development: Chrome DevTools' accessibility panel to inspect the tree, the contrast checker on any new color pairing, and vision-deficiency emulation to catch color-blindness issues.
• Before release: a keyboard-only pass (unplug the mouse, navigate the whole flow) and a screen-reader pass (VoiceOver on macOS, or NVDA on Windows) on the critical user journeys.
• Periodically: testing with real users who rely on assistive tech, because they find issues no combination of tools and simulated testing will surface.

The deeper habit is refusing to treat an automated accessibility score as a finish line. A clean axe report means the floor is solid; it says nothing about whether a screen-reader user can actually complete the task. Accessibility is a quality of experience, and experience is what the human ceiling is for.

References

[1] roadmap.sh, "Accessibility tests — QA roadmap node," 2024. [Online]. Available: https://roadmap.sh/qa/accessibility-tests

[2] Guru99, "What is accessibility testing? (Examples)," 2023. [Online]. Available: https://www.guru99.com/accessibility-testing.html

[3] Deque, "Axe — accessibility testing," 2024. [Online]. Available: https://www.deque.com/axe/

[4] WebAIM, "WAVE — web accessibility evaluation tool," 2024. [Online]. Available: https://wave.webaim.org/

[5] Chrome for Developers, "Chrome DevTools — accessibility reference," 2024. [Online]. Available: https://developer.chrome.com/docs/devtools/overview/