The right environment for user testing

Most games testing, like the user experience testing done at Vertical Slice, is done in a quiet room with a comfy sofa and with access to free food and drink. This isn’t just a needless luxury; instead it’s an integral point of the user testing process, and gives superior results to those found in a typical lab environment, with rows of workstations and one way glass.  Why is this, and what factors are at play here?

What is the environment?

With user testing, the environment comes down to the context in which the software or product is used. This includes where the user is, when they are using something, and who they are using it with.  It’s important to not forget the factor that the environment plays when testing a product, otherwise your final results, conclusions and design decisions may end up way off the mark.

Considering the environment for testing is important for both usability and user experience.  For usability testing, a realistic setting is needed to understand how a system will perform in that environment when out in the field. User experience focuses on how a system makes the user feel, and to truly understand this, you need to test users in an equivalent setting. This is why you’ll find comfy sofas wherever games are tested, since games are typically played at home in front of the TV.

pizza boxes are optional

Getting it wrong

Common mistakes with user testing involve inaccurately representing the environment in which a product is to be used (poor “ecological validity”).

This could include an unfair representation of the conditions in which the product would be used – for example testing a car radio without having the user drive would not give a true understanding of how the radio would function when the user’s attention is diverted.

Another common mistake is failing to test things in a time sensitive manner. For example, ticket machines are often used in high stress situations, minutes before the train is going to leave. Giving the user ample time in a comfortable lab setting does not recreate a typical interaction with the ticket machine, and would give an unrealistic impression of its usability.

User tests should also be separated, though concurrent, to the design process, and the development team should not be allowed to run the user sessions (though it’s a good idea to have them spectate from afar). Having the team in the same room as the player can intimidate them into praising the game or product, and reduce the validity of the results.

you liked it... got that?

Getting it right

So how can you be sure to get the environment right for testing?  The following things must be kept in mind:

• The right place: The ideal place would be the actual environment in which the game or product would be used. However this isn’t always possible, and so it’s important to recreate the environment as closely as possible. It’s also a good tip with games to test somewhere away from where the games are made – a typical player would be excited to be at a game studio, and their opinions would be nicer, and more eager than a typical home experience.
• The right mood: The sofa, and snacks mentioned in the introduction are all part of making the environment comfortable for the user, so that they feel at ease when testing the product or game. Another advantage to the previously mentioned idea of running these tests off-site, away from the people who made the product, is that the user won’t feel intimidated into praising the product, and they’ll be free to give a true opinion.
• The right people: It’s important to recruit the realistically, and test with real users. I’ve written before about how you should test with real users, not your team, and it is one small step which can increase the the validity of your results massively.
• The right friends: Remember when testing multiplayer games that a new environment will initially make the users uncomfortable, and this is magnified if they are playing with strangers. If you want to see how your game is received when played among friends, or with families, there is no shortcut – you have to test with groups of friends to get a true representation.

Taking care when selecting the right environment is important whether you are making a game, a ticket machine, or a washing machine for your cat.  A suitable environment for user tests will increase the validity of your results, and ultimately help make informed design decisions.