Auditing and testing

Carrying out audits of your digital products against accessibility guidelines is a great way to find out where many issues are.

How are audits carried out?

Audits are carried out on code, and can be done in the live environment or in test environments. To catch issues pre-launch (which is preferred!), you should carry out testing during multiple stages of your product development process.

Compliance can be the main driver for audits for some businesses, particularly in countries where legislation is more advanced like the USA. We prefer to think about it as a way of making what we do available to all travellers, and in turn, excluding nobody.

What are they tested against?

Audits tend to be run against the Web Content Accessibility Guidelines – or WCAG for short. These are a global set of guidelines widely used in the accessibility industry. We are currently testing against WCAG version 2.2, and we’re aiming for Level AA compliance.

A combination of automated and manual audits is advised. You can choose to run them in-house or ask external experts to do them for you. Automated audits are relatively easy to do yourself, as there are many tools to help you. Manual audits should be carried out by accessibility experts, as they are rather complex. Ideally, audits are done before you go live, and as early as possible in the production process.

Testing with disabled people is vital to gain a true picture of how accessible your products really are.

Automated audits

Automated audits use software to scan your products for accessibility issues against WCAG standards. There are pros and cons of these types of audit.

    1. Easy to repeat tests at different stages of the product lifecycle

    2. Just a few steps to run and very quick results

    3. Little accessibility knowledge is required to run the tests or understand the results

    1. Automated tools don't catch all of the accessibility errors in your product (Source: GOV.UK – What we found when we tested tools on the world’s least-accessible webpage)

    2. Reported false positives (an issue is reported that isn't a true issue)

    3. Multiple tools may be needed for different product types and roles

Automated testing is a great first step to check your website or app for accessibility, but not all checks can be automated.

    1. Lighthouse in the Developer Tools in Google Chrome (free)

    2. Accessibility Insights – a Chrome plugin (free)

    3. WAVE – another Chrome plugin (free)

    4. AQA – by UsableNet. We have licenses for this tool and have set up audits to run weekly on most of our desktop and mobile websites. We share the weekly results with our Engineering teams, and track the “Accessibility Score” the tool gives you, so we can see progress over time. (not free)

    5. Many others – there are many more free auditing tools out there, and more thorough paid options too.

    1. iOS: Automated testing – Accessibility Inspector in Xcode

    2. Android: Automated testing – Accessibility checking in Espresso or Accessibility Scanner

Manual audits

Manual audits cover 100% of WCAG success criteria, so take more time and expert knowledge to complete. They involve a combination of checks that cannot be automated (yet!) such as visual inspections, browser developer tools and assistive technologies to identify failures against the WCAG standards.

An example of the difference between an automated and manual audit is when checking the alt text on an image. An automated audit can tell is there’s an alt tag there, but you need someone to check if the description of the image is correct.

We have separate manual auditing checklists for web and app products (below), which are a great step-by-step guide to manual auditing. They are also a great way to learn what makes a product accessible! We now tend to run manual web audits through our AQA tool, and have manual app audits carried out externally.

Accessible checklist – Web

UX agency Nomensa created this manual auditing checklist for us. Please note – although this covers a lot of WCAG, it only mentions the success criteria relevant to our products, so there are some success criteria not listed here.

Web Auditing Checklist

Accessible checklist – App

This is the same idea, but for auditing an app.

App Auditing Checklist

Product testing

Testing for accessibility during the product development process is vital, and needs to be added to your design, development and QA processes.

We’ll provide more detailed information on how to do this soon (we’re improving our testing documentation as we speak), but here are some handy resources for now:

  1. Our Manual Testing Checklist, which explains how to carry out some of the more complex checks on desktop

  2. For iOS apps, Dani's blog covers it all

  3. For Android apps, there’s an Android Developers guide

User testing

There’s no better way to know if your digital product works for disabled people than by asking them!

There are companies who can organise this for you, or you can organise testing sessions yourself. We’ve done a number of things, including:

  1. Reaching out to disability charities and organising groups of people with a particular disability to test with

  2. Finding screen reader users who test digital products and hiring them on a freelance basis

  3. Working with specialist user research agencies, like Open Inclusion, Fable and Ab11y