Is it a good idea to study your own game in-house? Red flags ahead.

Many of our clients, especially the smaller studios, face a dilemma – whether or not to conduct the research themselves or outsource to some external company. The fundamental issue seems to lie with the price of the external service. But – surprisingly – that shouldn’t be the determining factor.

If we consider the internal costs of conducting such a study – including the per person-hours spent on organizing, then conducting, and then systematically reporting the results of such a study – the internal costs are comparable to the price of an external research service.

There is however, another aspect one must consider, which probably is worth more than the cash spent – the reliability of the study.

We recently conducted a think-aloud protocol (TAP) research for one of our clients. In a nutshell: it’s a process where the player discusses their experiences and actions in real-time while playing. You can read more about this method here. If such a study is well carried out, it can produce beneficial, actionable results ready for game developers (or publishers) to use.

TAP allows us to learn about the player’s most important general, conscious pains (frustrations, problems) or gains (joy-bringers) related to the gameplay, interface, specific mechanics, or the smallest of gameplay elements, e.g., particular environmental puzzles or a specific fight with an opponent. Such research methodology provides excellent insights into the causes of particular player behaviour. The TAP results allow game developers to improve certain aspects of their game based on data. The specificity of those research results allow them to immediately target the pain points, making the improvement process faster, easier, more confident and…. cheaper than if they had to figure out those problems themselves or if they were to notice them after the game had already been released.

We must frankly admit that any developer or publisher can easily launch their own game research using the TAP method. It’s definitely not rocket science. And it’s tempting, no doubt. But…

… to conduct an internal study on one’s own game, e.g., using the TAP method, is not a good idea. To better understand why, we must come back to the reliability of the study.

The problem (related particularly to TAP) was covered in a few sentences by Tom Knoll in one of the chapters of Games User Research:

Although the think-aloud protocol is an easy method to learn, this is generally a bad idea due to the emotional investment that game designers are likely to have in their product. Whether intentionally or not, anyone who has been involved in the creation of the game may influence the decision-making of the participant through the use of language or an over-willingness to help when they become stuck (…). It is worth hiring a professional and independent user researcher to conduct the sessions on your behalf. They will be able to maintain an impartial stance throughout the data collection process, which will improve the reliability of collected data and make the findings more robust.

The above warnings apply not only to the TAP methodology but also to all other qualitative methods where the competencies and background of the researcher are crucial. When a development studio decides to launch in-house research, it must also be mindful of at least three psychological effects that reduce the reliability of the entire research process and its results.

People don’t want to make others uncomfortable

People rarely say unpleasant things directly to others. Players are no exception. When they have to say something unpleasant about a game they are playing, e.g. there are terrible moments, ugly graphics, etc., they are more likely to say it to a neutral party who has nothing to do with the creation of that game rather than to the actual developers. Most of us are not malicious or frustrated people looking to make someone uncomfortable. We are often ashamed to tell someone a painful truth; we are not always confident in ourselves and our judgments (“maybe I’m missing something”, “perhaps I’m not the most competent to judge”), and finally – naturally, according to the humankind evolution: we don’t want to spoil our relationship with another person unless there is a severe and vital reason to do so. Therefore, even the mere presence at the developer’s headquarters (if that’s where independent testers would be invited to conduct the study) can inhibit a player from sharing critical feedback with the game’s developers.

Meanwhile, developers should care about discovering the truth, even if it is unfortunate and painful to hear. Such truths are more easily conveyed by people not involved in the studied game’s development.

Paternal effect (aka paternal affect)

The effect of an uncritical attitude to one’s own ideas. It is a natural part of the creative process. It inhibits teamwork and prevents fine-tuning solutions that only seem perfectly valid. It is experienced by everyone who has ever created anything. The only antidote to this affliction is to distance oneself from their work, for example, by having it externally evaluated. Ideally, the evaluation should be a double-masked review – the creator does not know who is evaluating their work, and the evaluator does not know who they are evaluating.

Groupthink (aka groupthink syndrome)

This effect occurs when the development or publishing team’s desire for consensus and consistency outweighs the desire to reach the best possible decision. Such a phenomenon can result in a decision that is not in the interest of either the group or the organisation but is merely a way of avoiding conflict. In other words – maintaining consensus within the group is more important than carefully considering the facts in a realistic way. Symptoms of groupthink also lead to a failure to seek additional information and alternatives or to even discuss them. If you want to know more about this phenomenon, check out this article.

For example, the studio is convinced that the game’s greatest strength and USP will be its highly-developed storyline. If the key people express this opinion in the organisation, it is very likely that everyone else will not even question its relevance. This works as people don’t want to ‘expose’ or show themselves as outsiders from the group. Even if most creators (programmers, designers, artists, etc.) feel that it is not the storyline but the specific mechanics X – let it be a bow mechanic, melee combat, traversal or whatever – that should be put first and in which the most resources should be invested.

Until the team decides to make an independent assessment “from the outside”, not entangled in in-group dependencies, the issue of plot vs mechanics X will not even reach the surface to be discussed. The issue will only become apparent once the game is released, so after hundreds of thousands have been invested not only in production but also in marketing. Also let’s not forget, the first impression on players is only made once.

Groupthink is more dangerous than paternal effect because not only does it block discussions within the team, but it also discourages the team from seeking external evaluation and alternative views of the game.

*

For the abovementioned reasons, the more the development team thinks it’s not worth subjecting the game to external evaluation (because “everything is clear”, “we know what things to improve”, the team “knows very well what they’re doing”, they have “a well-designed roadmap” etc.), and the more they convince investors (or the executive producers, directors, “stakeholders” etc.) of this, the more it should set off alarm bells among the teams’ managers.