Accessibility Auditing

Using Automated Methods to keep SimpleReport Accessible

Accessibility is a primary concern in keeping SimpleReport available and delightful to all users. SimpleReport uses a combination of methods to ensure the application satisfies accessibility requirements. These include:

• Manual audits with a variety of tools
• Lighthouse page testing
• Automated user flow tests, utilizing cypress axe

✌️ Manual Audits

Manual audits are a good way to confirm that automated testing is catching errors properly, and to test accessibility implementation locally. This can be done with the official Axe toolkit from Deque, which can be run manually as a browser dev tools extension or utilized to run automated guided tests as well.

Other useful browser extensions for local testing:

• ARIA dev tools

💡 Lighthouse Testing

Background

Lighthouse is an open-source tool developed by google to improve the quality of web apps. While there are 5 categories that can be audited with Lighthouse testing, SimpleReport utilizes only the accessibility and performance categories. Lighthouse audits run on a provided url (or list of urls), and return results for the specified page. The accessibility results do not seem to be as granular as Axe tests, but provide an easy way to catch egregious regressions in the app.

Note that Lighthouse provides two tools for running Lighthouse tests, lighthouse, which is a CLI tool, and lighthouse-ci (aka lhci), which is another CLI tool optimized for CI configuration. SimpleReport utlilizes lighthouse-ci for testing in CI/CD pipelines. See SimpleReport's Github action and configuration for Lighthouse.

Resources

• lighthouse-ci configuration
• lighthouse cli options, which can be added to settings in lighthouse-ci configuration

🔁 e2e Testing

Background

SimpleReport utilizes the cypress-axe and axe-core npm packages to configure accessibility testing in cypress e2e tests. The axe tests reliably catch errors and produce few to no false positives. These tests are run on each PR in the CI/CD pipeline, and can also be run locally. See the Cypress wiki page.

To test the accessibility on a page, after visting a page with cy.visit(), simply inject axe and check accessibility, like so:

cy.injectAxe();
cy.checkA11y();

The Deque blog has a good blog post for setting up axe in cypress.

It is possible, though generally not recommended, to exclude components or set rules to ignore certain errors. See the official GitHub repo docs and this helpful blog.

Gotchas - When adding accessibility checks in cypress, it's best to wait to call cy.checkA11y() until you confirm the page has loaded, otherwise the check may fail in unexpected and incorrect ways. They are most reliable directly after an assertion like .get or .contains that does not do an action (like a .click) that changes the page.

Also worth noting, because cypress will retry a test if it fails, note that when it re-runs after an accessibility failure, the second run will fail for a different reason.

Resources

• Setting up cypress with axe for accessibility
• Deque - How to test for accessibility with cypress

🔖 Other Resources

For the future... There are a variety of other ways to implement automated accessibility testing. This article outlines a few options for GitHub actions.