Accessibility Auditing

Using Automated Methods to keep SimpleReport Accessible

Accessibility is a primary concern in keeping SimpleReport available and delightful to all users. SimpleReport uses a combination of methods to ensure the application satisfies accessibility requirements. These include:

• Manual audits with a variety of tools
• Lighthouse page testing
• Automated user flow tests, utilizing cypress axe

✌️ Manual Audits

Manual audits are a good way to confirm that automated testing catches errors properly and to test accessibility implementation locally. This can be done with the official Axe toolkit from Deque, which can be run manually as a browser dev tools extension or utilized to run automated guided tests as well.

Other helpful browser extensions for local testing:

• ARIA dev tools

💡 Lighthouse Testing

Background

Lighthouse is an open-source tool developed by google to improve the quality of web apps. While five categories can be audited with Lighthouse testing, SimpleReport utilizes only the accessibility and performance categories. Lighthouse audits run on a provided URL (or list of URLs) and return results for the specified page. The accessibility results do not seem as granular as Axe tests but offer an easy way to catch egregious regressions in the app. Currently, we run Lighthouse audits on every PR.

Note that Lighthouse provides two tools for running Lighthouse tests, lighthouse, which is a CLI tool, and lighthouse-ci (aka lhci), which is another CLI tool optimized for CI configuration. SimpleReport utilizes lighthouse-ci for testing in CI/CD pipelines. See SimpleReport's Github action and configuration for Lighthouse.

Resources

• lighthouse-ci configuration
• lighthouse CLI options, which can be added to settings in the lighthouse-ci configuration

🔁 e2e Testing

Background

SimpleReport utilizes the cypress-axe and axe-core npm packages to configure accessibility testing in cypress e2e tests. The axe tests reliably catch errors and produce few to no false positives. These tests are run on each PR in the CI/CD pipeline and can also be run locally. See the Cypress wiki page.

To test the accessibility on a page, after visiting a page with cy.visit(), inject axe and check accessibility like so:

cy.injectAxe();
cy.checkA11y();

The Deque blog has a good blog post for setting up axe in cypress.

Though generally not recommended, it is possible to exclude components or set rules to ignore specific errors. For example, see the official GitHub repo docs and this helpful blog.

Gotchas - When adding accessibility checks in cypress, it's best to wait to call cy.checkA11y() until you confirm the page has loaded; otherwise, the check may fail in unexpected and incorrect ways. The checks are most reliable immediately after an assertion like .get or .contains that does not additionally do an action (like a .click) that changes the page.

Also worth noting, because cypress will retry a test if it fails, note that when it re-runs after an accessibility failure, the second run will likely fail for a different reason.

Resources

• Setting up cypress with axe for accessibility
• Deque - How to test for accessibility with cypress

🔖 Other Resources

For the future... There are a variety of other ways to implement automated accessibility testing. This article outlines a few options for GitHub actions.