How real-world game usability testing is changing

Recently I visited a global game publishers’ usability lab in London to help out for a day, and participated running UX tests on the current revision of an upcoming game. The experience was extremely educational, and enjoyable, and changed my perception of integrating usability within an established company. Unlike the academic settings in which I have previously been working on usability, this companies position as a global entertainments company demands that the needs of the business is taken into consideration as well, and this has an affect on how usability is tested. The role of usability and user experience in game design is changing, and this could be clearly seen.

What’s testing usability of games like in the real world?
Compared to an academic approach, as we have been working towards in our course based at Vertical Slice’s labs, there are a few differences to how large companies typically evaluate the user experience of a game.

• Companies will often consider usability testing a ‘stage’ near the end of the game design process
• A business will tend to have a wider focus when it comes to analysing player data (taking a ‘breadth’ look at the data, rather than the ‘depth’).
• This translates to a testing the game with multiple players at the same time, and observing their play session live looking for problems.
• When a problem is found, a business will focus on how to fix the problem, not why the problem exists
• The business wants results. Now!

NOW!

Why does this difference in testing game usability exist?
To understand the reasons for these differences, you have to understand the demands that the ‘business’ places on their usability specialists. In short, they want to fix the maximum amount of problems, and fast. Working in video games often means tight time frames, and the demand for instant feedback and results. This of course comes down to money – time that programmers are left idle is a wasted expense, and also there is a large degree of pressure to ship the game on schedule, and not lose momentum built up by marketing to competitors. The ‘waterfall’ methodology used by many businesses divide the design process into stages, and usability testing will appear as one of the later stages.

Alan Cooper, in The inmates are running the asylum,  argues that these demands are based on false premises, and looks to the failed early 90’s attempts to design popular PDA’s as an example of why being ‘first to market’, at the expense of a more user friendly product, is not always the most successful game plan. But, nonetheless, these are the demands put on real world usability labs.

wait, when did we want the results again?

What effect does this have on the results?
This ‘wide angle’ mass testing approach imposed on labs has several advantages which are of interest to businesses.

• Major issues, such as problems that most people will encounter, can be seen easily as you will have a room full of people complaining about it.
• Simultaneous testing means you get the opinions of a large data-set quickly
• It is cheaper to the business than a deeper integration of usability into every step of the process.
• Any sort of usability testing has to be applauded, as it’s been shown to generate a better product, improve customer relations and cause a post-launch reduction in troubleshooting and maintenance costs

But, equally the approach forced by the demands of a business has several disadvantages:

• A ‘lets just fix it’ approach means that you won’t learn about the causes of problems found, and can lead to the same problem being created in the next project.
• A business with the attitude of ‘sending it for usability testing’ near the end of the development life cycle limits the amount of changes that can feasibly be made to a game or product.
• ‘Finer grained’ problems, that the player may not be aware of, but that would show up on an close analysis of their play session, will likely be missed.

How should it be done?
This companies’ usability lab is an expert in the field, and so conduct a range of usability tests at different stages in the development process. And this is the correct approach, meeting business demands for fast, cheap results, by changing the nature of usability testing.

Rather than simply being a ‘stage’ of the process, as you will often see in typical ‘waterfall’ project management methodologies, usability and testing the user experience should be ingrained in every aspect of game design, even before the first line of code is written. By doing smaller scale testing, but more frequently throughout game design, we can overcome the disadvantages that the traditional method imposes upon us.

• Time becomes less of an issue, as it runs concurrently to programming and game design
• More problems will be found, but at a more manageable pace
• This means that each problem found can be focused on in more depth, leading to a deeper understanding of their root causes
• Problems will be found with enough time to fix them
• Larger problems will be found, and fixed earlier, reducing bug fixing time.

My impressions from working with this global game publisher and distributor was that this is the direction they are heading, and as market leaders, I imagine that it’s a direction that most companies involved in game design will soon be following.