--- title: "10 — Accessibility Testing — Automate the Floor, Human-Test the Ceiling" uid: accessibility-testing tags: ["roadmap:qa", "wcag", "accessibility", "a11y", "screen-reader", "chrome-devtools", "wave", "axe"] excerpt: "Automated tools find about a third of issues — the detectable floor. The rest, the ones that block real users, require human judgment against the accessibility tree and real assistive tech." date: 2026-08-13T03:27:47+0000 source: https://www.aveshina.my.id/en/blog/accessibility-testing --- "Run the Lighthouse audit, fix the red items, ship" was my accessibility testing plan, and it capped my coverage at a third. The split that fixed it: **automated accessibility tools find roughly a third of issues — the detectable, rule-violating floor — while the remaining two-thirds, the ones that actually block real users, require human judgment against the accessibility tree and real assistive technology** [1][2]. The automation isn't useless; it's just a floor, not a ceiling. Confusing the two is how teams ship a "100/100 Lighthouse" page that a screen-reader user still can't navigate. The framing that landed is the split between *rule violations* and *experience failures*. A rule violation is "this image has no alt attribute" — a tool can detect it deterministically. An experience failure is "the focus order jumps confusingly when the modal opens" or "this heading structure makes the page incomprehensible when navigated by heading" — these require a human (often one using assistive tech) to evaluate. Tools catch the first category reliably; they're blind to the second. ## The automated floor: what tools actually find The roadmap points at three tools, and they're best understood as a layered set rather than alternatives: - **Axe** (Deque) is a fast, lightweight engine that checks the document against a ruleset and reports violations, passes, and incomplete items [3]. Its distinctive virtue is that it's a *library* — axe-core can run inside your existing test automation (Cypress, Playwright, Selenium, Jest), so accessibility checks become part of the CI gate, not a separate manual audit. - **WAVE** (WebAIM) is a suite of evaluation tools that identifies accessibility and WCAG errors and *facilitates human evaluation* of the surrounding context [4]. WAVE's visual overlay — icons on the page marking each issue — is designed for a human reviewer stepping through a page, which is why its own tagline emphasizes human evaluation alongside the automated checks. - **Chrome DevTools' accessibility panel** adds the accessibility tree view (how assistive tech sees the page), a color-contrast checker, Lighthouse audits, and vision-deficiency emulation [5]. The shared limitation is the same one Deque publishes openly: automated tooling catches around 30% of accessibility issues. The other 70% — meaningful alt text (not just *present* alt text), logical heading order, sensible focus management, readable language — need a person. ## The accessibility tree — how assistive tech actually sees the page The single most useful idea in this whole area is the **accessibility tree**. Browsers don't expose the raw DOM to screen readers; they expose a parallel, cleaned-up tree where each node is an accessible object with a role, name, state, and value. Chrome DevTools lets me inspect this tree directly [5]. ```figure ``` Reading the accessibility tree is the skill that separates "I ran the tool" from "I understand what a screen reader will do here." If a button the sighted user sees isn't in the accessibility tree — or is there with a wrong name or role — the screen-reader user has no way to reach it, regardless of how the page looks visually. ## The human ceiling: the issues tools miss The categories that automated tools consistently miss, and that require human evaluation against WCAG [2], include: - **Meaningful text alternatives.** A tool flags a *missing* alt; it cannot tell whether a *present* alt="image1" is meaningful. Writing good alt text is a judgment call. - **Logical heading order.** Tools can flag a skipped heading level (h1 → h4), but whether the heading *structure* makes sense when navigated linearly is a human call. - **Focus management.** When a modal opens, focus should move into it; when it closes, focus should return to the trigger. Tools rarely catch focus-trap or focus-restore bugs. - **Keyboard operability.** Can every interaction be reached and operated without a mouse, in a sensible order? This requires actually unplugging the mouse and trying. - **Color contrast at the edges.** Tools catch gross contrast failures but miss contextual ones (text over a busy background image). - **Readable language and clear instructions.** No tool evaluates whether the page's language is plain and the instructions are clear. The honest practice is to combine layers: automated checks in CI (axe-core in every build), manual review against the accessibility tree (DevTools panel during development), and periodic testing with actual assistive technology (NVDA, VoiceOver, JAWS) and real users where possible. ## WCAG as the shared standard Underlying all of this is **WCAG** — the Web Content Accessibility Guidelines — organized around four principles (POUR): content must be **P**erceivable, **O**perable, **U**nderstandable, and **R**obust [2]. The guidelines under each principle are the shared vocabulary between automated rules and human judgment: a tool checks "does this image have alt text" (a Perceivable guideline), and a human checks "is that alt text meaningful" (the same guideline, applied with judgment). WCAG is what makes the floor and the ceiling part of the same building. ## How I use this The practical payoff is a layered routine rather than a single audit: - **In CI, on every PR**: axe-core assertions in the existing test suite. This catches the rule-violating floor automatically and prevents regressions — a new component without a label fails the build. - **During development**: Chrome DevTools' accessibility panel to inspect the tree, the contrast checker on any new color pairing, and vision-deficiency emulation to catch color-blindness issues. - **Before release**: a keyboard-only pass (unplug the mouse, navigate the whole flow) and a screen-reader pass (VoiceOver on macOS, or NVDA on Windows) on the critical user journeys. - **Periodically**: testing with real users who rely on assistive tech, because they find issues no combination of tools and simulated testing will surface. The deeper habit is refusing to treat an automated accessibility score as a finish line. A clean axe report means the floor is solid; it says nothing about whether a screen-reader user can actually complete the task. Accessibility is a quality of experience, and experience is what the human ceiling is for. ## References [1] roadmap.sh, "Accessibility tests — QA roadmap node," 2024. [Online]. Available: [https://roadmap.sh/qa/accessibility-tests](https://roadmap.sh/qa/accessibility-tests) [2] Guru99, "What is accessibility testing? (Examples)," 2023. [Online]. Available: [https://www.guru99.com/accessibility-testing.html](https://www.guru99.com/accessibility-testing.html) [3] Deque, "Axe — accessibility testing," 2024. [Online]. Available: [https://www.deque.com/axe/](https://www.deque.com/axe/) [4] WebAIM, "WAVE — web accessibility evaluation tool," 2024. [Online]. Available: [https://wave.webaim.org/](https://wave.webaim.org/) [5] Chrome for Developers, "Chrome DevTools — accessibility reference," 2024. [Online]. Available: [https://developer.chrome.com/docs/devtools/overview/](https://developer.chrome.com/docs/devtools/overview/) ```quiz Q: Automated accessibility tools (axe, WAVE, Lighthouse) catch roughly… - 100% of accessibility issues — a passing audit means the page is accessible - 30% — the rule-violating floor; the remaining issues require human judgment against the accessibility tree and assistive tech correct: 1 explain: Tools deterministically detect rule violations (missing alt, skipped heading level). They cannot evaluate meaningful alt text, logical focus management, keyboard operability, or real experience — those need a human. Q: The accessibility tree is… - the same thing as the DOM, just rendered differently - a parallel, cleaned-up tree the browser exposes to assistive tech, where each node has a role, name, state, and value correct: 1 explain: Browsers don't expose the raw DOM to screen readers; they compute an accessibility tree. Reading it (via DevTools) tells you what a screen reader will actually consume — the DOM noise collapses to signal. Q: A page passes axe-core in CI with zero violations. This means… - the page is fully accessible to all users - the rule-violating floor is solid — but meaningful alt text, focus management, keyboard operability, and real experience are unverified correct: 1 explain: A clean axe report is a floor, not a finish line. The human-ceiling issues — experience failures a tool can't see — still require keyboard-only and screen-reader testing. Q: Which of these would an automated tool most reliably miss? - an image with no alt attribute - a modal that opens but traps focus and never returns it to the trigger on close correct: 1 explain: Missing alt is a rule violation — easily detected. Focus management (where focus goes when a modal opens/closes) requires operating the page with a keyboard and judging the behavior, which tools rarely catch. Q: WCAG's four principles (POUR) state that content must be… - Perceivable, Operable, Understandable, Robust - Performant, Observable, Uniform, Responsive correct: 0 explain: POUR — Perceivable, Operable, Understandable, Robust — is the organizing structure of WCAG. The guidelines under each are the shared vocabulary between automated rules and human judgment. ```