Don’t Playtest Your Game

This post is about why you shouldn’t playtest your game. But it’s not about why your game shouldn’t be tested. It just shouldn’t be you, the developer, doing it.

Playtesting during development is invaluable, and it’s great to see how enthusiastic many teams are about running their own tests. However there are a number of biases and pitfalls that we often see in tests run by the developers for their own game, which alter the feedback received and greatly reduce its value.

In this post I cover some of these biases and look at ways in which they can be minimised, to make sure that the playtests are useful, and produce actionable results.

Don’t let players know you’ve made it

One of the first things we say to players when they come in to see a game during development is that we don’t make the game. We tell users that they can be as rude as they like about the game and they won’t offend us, and that we don’t get rewarded if they love the game. We do this to distance ourselves from the game, and let players know this is a safe environment for giving criticism.

People naturally want to please others, and will tell you what you want to hear. This can often be unconcious, and players may not even recognise they’re doing it. However their behaviour and feedback will be different if they think you are invested in the game, and you’ll lose valuable data.

It’s understandable to be proud of your game, but developers have to keep their relationship to the game secret otherwise the results can be biased. It’s unethical to lie about your identity, so teams should wherever possible pick people to run the tests who are not heavily invested on its outcome. If this is not possible, you can just keep quiet about your role unless directly asked, and describe your role as a tester, rather than a developer, of the game.

Be careful with anticipating problems

Another issue we often see with tests led by developers is that the team have an agenda they want to push. Research can be a great tool for answering questions the team may have, however particular care has to be taken during the test and during analysis, to ensure that results are represented fairly.

For example, if a team member hates the design of the menus, they could make the results express this by setting unrealistic menu tasks, asking leading questions, or over-emphasising the feedback from players who didn’t like the menus.

The real danger from anticipating issues is that it can cause unanticipated issues to be missed, as observers will only be looking out for specific things. It introduces lots of biases into how the sessions are remembered, as only the behaviour and feedback from users who behaved as anticipated, or the most entertaining users, will be remembered.

It’s important when running these sessions to act as an impartial observer, and to record everything that happens. Conclusions during the sessions are at a huge risk of bias, and it’s only after all the results have been collated and analysed that reliable results can be found.

Recruit the right people

Another challenge with internal playtests is finding the right people to play the game. Gamers range from kids playing on smartphones, through hardcore console gamers, to OAPs playing Brain Training on their DS. However your game will only target a subset of these players, and it’s important that you find suitable players to playtest your game. This will ensure that the results you find are indicitve of the issues that your real players would encounter after the game comes out.

When teams run their own playtests, it’s often easy to misrecruit. A common mistake is finding people from within the office – not only might they not be the right sort of players, but they will have background exposure to the game, even if they do not work directly on it. This will give them knowledge that real players would not have.

If a professional recruitment agency is too expensive, recruitment through facebook, twitter and word of mouth is possible. However I would definitely recommend setting up a questionnaire about their playing habits, and asking people to fill it out before they get booked in. This will allow you to screen participants and ensure that they are similar to the audience that will buy your game, and that the results will be valid. The extra time it takes to make and analyse the questionnaire will be nothing compared to the time wasted by running sessions for users who are unrepresentative of your audience.

Run tests in a realistic setting

Trade shows are a great opportunity to market your game to consumers. However, they are not a great environment for getting realistic feedback from players.

Trade shows are nothing like how people play games at home. There is a high level of background noise in trade shows, which can obscure audio from the game. Players will also feel under time pressure, either from the queue behind them, or from their desire to see the next game. At trade shows players often have an audience watching them, which will alter player’s behaviour. Also the players may be entirely the wrong sort of people – you haven’t found out anything about them (other than they like trade shows), and so they may not be the sort of player your game should be made for.

To get realistic feedback, always ensure that the location matches a realistic play setting as close as possible – often this means a living room environment. As discussed earlier, you also want to distance players from development, so running these tests off-site can help with that. Hotel rooms are often a useful way of achieving this, as they can be easily rented, often have TVs already, and feel like a living room.

Do playtest your game

Playtesting when making your game is fantastic. It gets invaluable feedback while there is still time to fix issues, and helps prioritise development. Some of the most common biases when teams are running their own playtests are easy to overcome – and an awareness of them will help produce useful and relevent results. Combining well run internal playtests with playtests run by external professionals will improve the quality of your game, while still keeping costs down.