Reframing usability testing for a remote world

We sat down with Abstract’s Director of Product Design Sarah McIlwain and Senior Product Designer Payam Rajabi to learn about the surprising ways usability testing can benefit from a distributed workforce and user base. Here’s what we learned.

The shift to remote work, once a gradual evolution, has suddenly kicked into high gear. Team chat software has replaced the physical office environment. Meetings are, by default, assumed to be taking place over video calls. 

The world of work has changed quickly, but at Abstract, the concept of distributed work is nothing new. We started bringing on remote team members shortly after opening our San Francisco headquarters in 2015. Before COVID-19 hit, 80% of our workforce was remote, with only 20% working from the office. Now, of course, 100% of us are working remotely. And post-pandemic, we will remain a remote company.

For our team, usability testing and research play a huge part in informing our roadmap and what we release. Traditional user testing, however, is carried out in close quarters: Study participants complete a controlled set of tasks as dictated by a test moderator, who is also in the room.

So how does being remote affect usability studies? The honest answer is, not much. In fact, over the years we’ve discovered a ton of opportunities and advantages to doing this work remotely. In this article we’ll take you through them — but first, a brief history lesson.

The evolution of usability testing 

In the early days of software design, usability testing was often considered a final checkpoint: something you did once everything was built and ready to ship. It was there to make sure all the bits and pieces of your product were in the right place. 

Now, research and testing is done early and often as part of the product design process. Most of today’s design teams will agree there’s a much greater expectation to test early-stage prototypes — as early as drawing boxes and words on a piece of paper and having participants tap on it as if it were an actual interface. 
‍

While the frequency and scope of usability testing has increased, its purpose has remained constant: make sure that the product being delivered is, well, usable.

Regardless of the product, setting, or stage of prototype being tested, the research always aims to mimic a set of actions the user might go through in the real world. That’s as true for larger undertakings like usability baseline testing, where you’re tracking every action a user would take within a product’s interface, as it is for testing a smaller feature improvement. 

User testing should be approached as scientifically as possible. The goal is to create a repeatable story. That way, when you share your prototype with a fixed number of users (usually five to seven), you get a good sampling of how each of them reacts to a specific data point. 

So how does being remote affect that goal?

Challenges of remote usability testing

As with anything, going remote with usability testing presents its own set of challenges. Most of these amount to limitations in what a test moderator can do from a remote setting as opposed to what’s possible when you’re in the same room as the user. 

There are a few clear advantages to on-location testing. It’s easier to build rapport, for one. As the moderator, you can also troubleshoot problems that arise with the technology. 

While our team is used to being distributed, COVID-19 has compounded and added to these challenges. For example, with travel no longer an option, we’re unable to conduct field tests, where we would travel to a locale and be there with users for the duration of the test.

Whether they’re recent developments or not, our team has had to keep all of these considerations in mind. Given that we can’t always be in the same room as the user, how can we fill that gap and ensure repeatability and a controlled test environment?

Advantages to remote usability testing

While remote usability testing comes with certain challenges and considerations, it also presents a number of valuable opportunities — and we find that they far outweigh the limitations.

Remote testing allows for easier observation

Observation is one consistently remote aspect of usability testing. While in traditional user testing the moderator is in the room with the participant, most of these tests feature a camera trained on the participant, and sometimes another camera trained on the computer screen the participant is interacting with. 

Setting it up this way allows others to observe the study remotely, usually on a video feed from a conference room. Observers can watch the study in progress, take notes, and call out specific actions with others in the room. 

Remote observation can be easily recreated, and even improved upon, in fully distributed studies. The ubiquity of video calls makes it easy for anyone on the team to jump on and observe. The “Recording” icon on a video call is also much less imposing than having a camera pointed at you. When participants are comfortable, there’s a more realistic and accurate picture of how they'd interact with the designs in their day-to-day. 

Finally, having the option to access recordings from video conferencing software allows us to take interesting pieces of the study and quickly edit them down into a highlight reel that we can share with the product team. 

The barrier to testing is significantly lower

Another huge advantage of using video conferencing software for usability tests is the amount of effort required. Before, we’d have to secure one location for testing and another for observation. Now, it’s as easy as sending out a Calendly link. The friction of setting up a usability test has virtually vanished, enabling us to work faster and more fluidly.

On top of that, while we’re currently unable to conduct on-location field tests, remote testing allows us to reach a much broader sample of customers. For example, it’s not uncommon for us to jump on a research call with customers in Berlin or Dublin in the morning, and another group of customers in New York or San Francisco in the afternoon. This makes it easier to connect with the right customers at the right time, without all the overhead that travel adds to the equation.

Best practices for remote usability testing

Having worked out the kinks of remote usability testing ourselves, we can say with confidence that a few considerations need to be addressed in order to do it well. 

Make sure your technology is running smoothly 

With remote testing, you don’t have the opportunity to stand over someone’s shoulder and figure out permissions for them. Make sure your test environment is well set-up and populated with the right data ahead of time, so that you don’t risk missing out on the study completely due to technical problems.

Don’t underestimate the power of remote handovers

The remote handover — where the moderator sets up the test environment and then hands over control of their computer to the study participant remotely — is a great tool to try. 

Doing this allows you to bypass all of the permissions, invitations, and feature flags that you would encounter if the participant was using their own account on their own computer. With a remote handover, you're able to tightly control the test environment and give each of your participants the same experience.

Being able to set up participants at state zero, have them go through the steps, and then restart that process with each new participant will save you a ton of time and effort.

Take advantage of transcription services

Trying to take notes while on a video call can result in being less present and missing opportunities to learn more with probing questions. Fortunately, most video conferencing tools come with a transcription service. This allows you to record the call and write down highlights as you watch the study, but also have a full transcription of that conversation to revisit later to get the full picture.

Make sure your findings don't get lost in anyone's inbox 

Having a lower barrier to entry with remote testing means having lots of insights to relay back to your team. Figuring out how to communicate those insights is important. In an environment where everyone's constantly being bombarded by links, messages, and videos, it’s tough to catch people’s attention. 

Experiment with different ways of doing this. Maybe it’s a highlight deck. Maybe it’s an email that includes a summary of insights. Maybe it’s a short presentation in a company all-hands meeting. Pay attention to which communication tools work for your team and which get buried.

Ultimately, any usability testing is only as good as the exposure it gets throughout the company. What you're trying to do is influence outcomes — and you can’t do that if nobody knows what the outcome was. 

Fewer limitations with remote testing

As a team, we’ve been pleasantly surprised by how well usability testing in a distributed environment can work. While we look forward to having the option for on-location field testing soon, we’ve found that remote usability testing allows us to conduct more studies and drive more insight-based outcomes than ever before.