Test…then test again

Sometimes it can be hard enough to convince a team to test their game once. Testing a game once is great at identifying problems and issues with a game. However after the team have addressed these issues, researchers have a new challenge – convincing the team to test again.

Without retesting, it is difficult to tell if the team’s fixes have been effective, and the usability issues that were identified could still be occurring. Additionally the issues from the first round may have obscured secondary issues which weren’t picked up the first time. Today I’ll be looking at some examples of why retesting is important.

Testing difficulty
Have you ever played a game where the difficulty was…wrong? Either the game is much too easy and offers no challenge, or the difficulty is much too hard (and , unlike Dark Souls, it’s not a game type which is meant to be this hard). The problem with these games could well have been not retesting them.

With a single test, the team would have got a finding such as “players believed the game was much too hard”. If the test was done well, the finding may even be quantifiable: “On average, players died 13 times on level 3, however the design goal was that players would only die 3 times”.  The team would have then gone away and altered the game by making it easier. However they now do not know how much easier they have made the game – perhaps their changes were not enough, and the game is still too hard. Or worse, the changes may have been too extensive, and now players complete every challenge with their first attempt.

The only solution to this is retesting after the team makes changes, in order to verify if the changes worked.

Testing fixes
It’s not just with difficulty that retesting is important; it can also help judge which solutions are appropriate. We have tested games before where players haven’t understood a key part of the game. We discovered this in the first test, and reported it to the team. The team’s response was to create tutorials explaining how this bit of the game worked. We then retested – but the issues still occurred. The team redesigned the mechanic entirely.

For this example, it took two tests for the team to find the right solution, and a third test for us to verify that the solution worked, and everyone now understood the mechanic.

Finding more issues…
Retesting is also important when big issues prevent researchers from seeing smaller ones. For example, if a player doesn’t learn how to open the map, we would then have missed any secondary issues about whether they can understand the map. Retesting after the “can’t find the map” issue has been resolved is the only way to uncover these secondary issues with the map itself.

Testing a game through multiple rounds is the best solution to help evaluate the team’s fixes. However it’s not always possible, and so sometimes researchers will be required to rely on their knowledge of best practices, and what they observed in the first test, to help teams find effective solutions to usability problems. But, when in doubt – test, and test again!