Our first A11Y user test
Together with one of our A11Y experts we decided we should organise a qualitative usability study especially aimed at users with a disability. The goal of the study was twofold: on one hand we wanted to learn what we needed to improve on our site, even though we ticked most of the WCAG boxes. On the other hand, we wanted to create empathy; show our colleagues who we are talking about when we discuss users with disabilities. They are not merely a statistic, they are real people.
Although there is a wide spectrum of disabilities, we decided to start off with people who are either (partially) blind or deaf. Two respondents were deaf and had only partial eyesight.
Interviews were done by our partner RuigRok Netpanel, in our usability lab in Amsterdam. The observation room would hold around 20 people who could follow along via a big screen. By actively advertising the sessions, and sending invites to all the teams who were working on the new site I made sure that during sessions the room was packed.
What we discovered during our preparations is that there are a couple of things that fundamentally differ from studies with abled users:
1. You need to take more time per user
Where we would normally take around 60 minutes in total for one participant, we decided that we needed more time for these candidates. Some candidates needed a bit of extra time to get used to the surroundings of the lab; others needed to be picked up and brought to the train station or bus.
With all that in mind, 90 minutes seemed a good length.
2. Users will bring their own equipment
One other complication is that people had to bring their own laptops or tablets. Normally we would test on an ING computer which was in the lab; but as people have their own specific installation in terms of screen readers, browser extensions and other tools, it was imperative for the success of our study that they would bring their own.
Also – we often would use tools like Invision to create a fast but simple click-through prototype for our users to test. Again, this wouldn’t work here as a prototype would not have all the necessary screen reader tagging for example.
Therefore we decided to work with the live website via a test account (as we wouldn’t want users to log in with their own bank account for privacy reasons).
3. Assist users to and from your lab
As some respondents weren’t able to find the lab on their own, we needed to pick them up and drop them off at the main public transport hub. Our usability lab is located in a pretty busy shopping and office area in the South-East of Amsterdam. It is packed with people, scooters and cyclists are weaving in and out of the crowds, vans are haphazardly parked everywhere, loading and unloading god knows what, and more often than not you’ll find parts of the streets obstructed by scaffolding or other building activities. A slightly daunting experience, even with all senses functioning normally.
Helping our participants getting to and from our lab through this mayhem made me realise that this probably is their daily routine. They have to deal with situations like this every day. They might be used to it, but that doesn’t mean it’s easy.
4. Take care of extra guests 😉
Some users will come with their guide dog, so make sure the dog is comfortable as well, by providing a bowl of water and a nice spot to take a nap. Deaf users will bring an interpreter, so you’ll have an extra voice during your interview, which you need to adjust to when conducting the interview.
5. Make sure your lab is accessible
Perhaps an obvious statement, but make sure your lab is accessible. Can wheelchair users access it? Can they reach and use the equipment? If the lab isn’t accessible, do find a location that is.