Creating an unbiased user feedback environment
There are multiple touch-points where feedback from users on a product may be received: sales, surveys, user testing, and customer success. Every area is important however I will be focusing mostly on user testing.
I have conducted countless user testing sessions and observed many more. There are key patterns I noticed that enhance or breakdown the session. If you are someone that has hosted these before, you’ve probably had times where the feedback was a dead end, wasting time and money. This article is meant to help minimize these issues and help both veterans and juniors alike.
Know your audience
If you are in UX it should go without question that knowing your audience is a key factor to success. There are many approaches and books on how to conduct user testing, however, we should start at a fundamental level to understand where we went wrong. Unless you are working on a product that is with tech-savvy users that understand the process, you may have unintentionally introduced a lot of bias before you start testing users.
When someone tries your product at home or in their work environment, they are in their safe-zone. They fully control and understand the decisions they make have no consequence or judgment.
When you take someone out of their comfort zone they may inadvertently be on the defensive and take actions carefully. Your goal as a host is to remove these notions and bring them towards a neutral state of mind.
A lot of UX researchers focus on the questions they will ask without much thought into the fundamentals on how their visitor may feel in the new environment.
No not because your first language is different than the guest coming in; it’s the wording of the session. “User Testing” feels as if a test is being presented and tasks need to be done correctly. In fact, almost every time I brought in someone for “user testing” they would ask me what happens if they do something wrong? Well, it’s not the user we are testing, it is our products.
It’s not the user we are testing, it is our products.
If you’ve been using the term user testing in your recruitment for feedback, try changing the terminology to “Product feedback.” A phrase along those lines will not only change their tone but empower them to critique freely.
One of the toughest changes I have made is to not refer to the user…as a user. It may be a minutiae gesture, but it may also help build the relationship if you refer to them as your guest.
With the design first movement, some companies have invested a lot into their facilities. I’ve been to some facilities where the user testing room is designed with a one-way mirror and hidden cameras. It seems great at first for the host, but you quickly realize your guest may feel like they are in an interrogation.
Keep the space where feedback is gathered casual and comfortable. Try conducting your session in an open area rather than a closed room. If your company has an open floor plan, find an area that still enables you to do the test but in an open area. This may further relax your guest.
Naturally, people may become nervous if there is a large group observing their actions in the same vicinity. Their psychological response may be the same with the knowledge of a remote group observing too. Try to have a maximum of two people observing at a time and ask permission to record the session for other colleagues to review, give them assurance that the recording will not be viewed outside the office.
Ultimately, give your guest the option to dictate choices from the moment they step through your office doors — this will help alleviate the pressure they may feel and allow for honest feedback without hesitation.
Setting the stage
Your prototype is the meat of this entire session. Creating a proper start and end point is crucial and there are two rules for success.
1: Don’t show static screenshots
As much as static screenshots may allow for quick feedback — it may work within the company with colleagues that know the product. Features can be very complex — and when taken out of context a lot of the potential feedback on discoverability and other related functionality may be lost.
2: Is your guest an existing user or new prospect?
Prototypes need to give the impression of completing a given task without guidance. Dropping someone in the middle of an interaction flow isn’t realistic and doesn’t imitate the real world. The goal of these sessions is to emulate as closely as possible the experience someone would have if they were at home or work with your product.
If the guest you brought in has never used your product, they should start from the very beginning. If the beginning of the usual experience is creating an account, they should start here. There is no problem in fast-forwarding time, but they should complete at least one task, for example:
The goal of your new feature is to analyze data of eating habits. The flow may include the requirements of adding a meal for one day, afterwards display an intermission stating that one week has passed; they should now be returned to the main screen to continue to the primary task.
This not only helps build context, but brings a higher understanding of the product to enable a discussion of the entire user journey from discoverability usability.
If your guest is an existing user, the task obviously becomes much easier. If possible change account names to their name to build the realism and personal connection with your guest, both new and existing users. Using sketch symbols to quickly change elements can make for quick and more relatable sessions.
Your prototype should always have a version catered to existing users and non-existing if you are testing these two personas. Always start with an entrance that would be expected from your product for each persona that exists if you plan to test for each.
This whole notion goes back to knowing your audience. Don’t gather feedback from people that will not have context or understand the usage. If a feature is being developed or redesigned that is only understood by longtime users, perhaps bringing people off the street isn’t the best approach.
Running the session
Create expectations from the beginning. Tell your guest they should be trying out the product as if nobody is around. There is a common practice to tell the users giving feedback to audibly announce what they are thinking. Again this is not natural and may lead them to ask questions or plead for help quicker than normal.
Using a webcam recording is a great way to review emotional response. Faces reflect the thinking of people, there is no reason to force people to speak about their thoughts — that’s what the Q&A interview afterward is for. Certainly, some people may still respond audibly, this is okay as long as it is naturally derived.
Quick tip: If you have recording equipment setup, leaving the room for a few minutes to allow your guest to go it alone.
Responding to feedback
After the requested tasks have been completed, allow your guest to give you any feedback they may have. Hold this as a discussion and do not counter any feedback they have, always agree and converse accordingly — not all feedback has to be implemented.
If feedback talks about the actual usability, or when you ask about other features they may find useful, always follow up with the following questions:
- What is the use case / or reason for this feature (if it isn’t obvious)?
- What is the end-game of this feature?
The responses to these questions can open up further discussion, especially if they yield very contrasting answers. Many times what someone says isn’t always what they intend, and these two questions can bring that truth forward.
Try to write and analyze on a deeper level with coworkers while the session is fresh in your mind. I always suggest leaving at least thirty minutes between sessions to settle in and regroup with colleagues. Remember it’s okay to reach out to clarify any feedback — especially if they are an existing user; this will help build relationships which in turn may retain them for a longer period.