Moderated User Testing - Is it Right For You?

User Acceptance Testing (UAT) is one of the most critical stages of product development. Also known as Beta or End User Testing, UAT is when actual users test the software and make sure it can handle required tasks based on the user’s needs and specifications before being released to the market. Your team has worked for months on designing and developing the product, and now it’s time to test and validate your work with actual users. Does it actually work the way it’s supposed to, and does the product address the user’s needs?
Capturing user feedback before software is officially released is crucial, and figuring out what type of UAT session works for you is the first step in achieving your team’s objectives. Here, we’ll be covering moderated user test sessions, its pros and cons, and how it can work for your team.

An Intro to Moderated User Test Sessions

As one of the more tried and true methods for user testing, moderated user test sessions are usually done in person and can be really beneficial during the early stages of developing your product (like when your product is going live for its very first iteration). These sessions usually include 5 or more users, a moderator (typically an expert in the field), an observer/note-taker, and development/and or data team members who are on call to troubleshoot if needed. During the session, the moderator has users complete real-life scenarios with the software and the team observes how the users interact with the product as they work through each task. The goal is for users to be able to complete the scenarios using the software without any guidance from the team. If they are really stuck, the moderator steps in to assist.

This works well for teams who are just starting out and a need a little bit of structure to their user test environments.

It’s also a great option for teams if:

You’re testing your very first release to a targeted user base. Moderated user sessions allow you to observe testers as they use the product to test real-life scenarios, and can span from an hour to a couple of days. In this setting, you’re able to directly observe each user’s thought process, likes/dislikes, and pain points as they move through (or fail to complete) the tasks you laid out for them using the software itself.

Observing how the testers interact with the product and talk through each scenario allows your team to take note of minor changes you can make before the first release, or sit down with your team during/after the session to review more complex issues that you didn’t anticipate but want to fix before the next iteration.

In the long run, these ongoing sessions would be a “nice to have,” but may not be necessary or even efficient as you release more versions and your users get more familiar with the software.

You have funds and resources to spare.

If your team has enough time, money, and resources, moderated user test sessions are great for real-time collaboration and fixes that may take longer if testing is completed remotely. If you’re short on time and there is a tight window between UAT and going live, you may want to save this type of session for the next release.

You have testers based in various markets.

Having a moderated user test session gives your team the opportunity to get to know your users in person and helps you build a more personalized session catered to that market’s needs. Users in San Francisco, for example, may prioritize one feature and be willing to brainstorm on design improvements, but users in DC may just want to test the product and be trained on using it the way it is. Either way, starting off with a moderated user test session gives you a chance to get to know your users firsthand and helps you build off their feedback for future releases.

When is it time to look into other testing options? Here are things you may need to consider:

Will it work for you as a long-term option?

Moderated user test sessions are great for major milestones in product development, but is it cost-effective for your team in the long run? As you ramp up your release cycle to other markets, your team has to consider whether the time, energy and resources you currently have should still be allocated to a method that can be time consuming and cost intensive (and maybe even counterproductive).

For example, sending three team members to San Francisco for just one session could easily cost up to $3,000 - $4,000 (transportation, flights, meals, etc.), not including costs related to the session and the time it takes to coordinate its logistics (lunch, supplies, etc.). As the team releases the product to other markets, you will have to consider whether frequent test sessions are worth the ongoing cost and resources, or if the team’s valuable time and resources should be spent on other priorities or alternatives.

Is it practical for you at this stage?

This ties into point #1. Moderated user test sessions aren’t designed to be “one size fits all,” and depending on what stage you’re at in product development, it won’t always work for your team. Before continuing with these sessions, think about it from a cost-benefits analysis/point of view. If having an in-person session doesn’t yield better results than other options out there and if there is a chance it could negatively impact your team’s progress and productivity, it’s time to look into more efficient alternatives!

For example, if your team is on a sprint schedule where you release every 2-3 weeks, you have to consider whether it makes sense to keep going with these sessions while investing the same amount of manpower and resources. Is it worth the extra months of prep work and cost to hold these monthly sessions, or could you get the same quality of feedback and work with even more users through cost-effective programs that can help you with virtually moderated or unmoderated testing?

These are just some of the factors your team should think about when deciding if moderated user test sessions are right for you. We’ll be sure to talk more about other options in the testing world, as well as the different types of software available that make it easier to conduct these sessions, as we test them out ourselves.

Until then, happy testing!

Download our Incubator Resources

 

WANT MORE?

We’re known for sharing everything!

HANDBOOK

Save more time, get more done!

FREE HANDBOOK

Innovate from the inside

Written by
Roxanne Janiczek 15 Jan 2018

UAT Lead, foodie, explorer, and dog lover!

YOU MIGHT ALSO LIKE

comments powered by Disqus