A Framework for Exploring and Evaluating Mechanics in Human Computation Games

Kristin Siu School of Interactive ComputingGeorgia Institute of Technology kasiu@gatech.edu, Alexander Zook School of Interactive ComputingGeorgia Institute of Technology a.zook@gatech.edu  and Mark O. Riedl School of Interactive ComputingGeorgia Institute of Technology riedl@cc.gatech.edu

Abstract.

Human computation games (HCGs) are a crowdsourcing approach to solving computationally-intractable tasks using games. In this paper, we describe the need for generalizable HCG design knowledge that accommodates the needs of both players and tasks. We propose a formal representation of the mechanics in HCGs, providing a structural breakdown to visualize, compare, and explore the space of HCG mechanics. We present a methodology based on small-scale design experiments using fixed tasks while varying game elements to observe effects on both the player experience and the human computation task completion. Finally we discuss applications of our framework using comparisons of prior HCGs and recent design experiments. Ultimately, we wish to enable easier exploration and development of HCGs, helping these games provide meaningful player experiences while solving difficult problems.

human computation games; games with a purpose; scientific discovery games; game design; game mechanics †††

1. Introduction

Games are everywhere. Games accompany people out on their mobile phones and await them back at home on their entertainment systems. Games are being integrated into educational curricula, embedded in wearable devices, and used in professional training simulations. For human computation, which harnesses the computational potential of the human crowd, this increasingly diverse audience of players represents new opportunities for computation. Human computation games (HCGs) are games that ask players to help solve complex, computationally-intractable tasks or provide data through gameplay. These games—also known as Games With a Purpose (GWAPs), scientific discovery games, and citizen science games—have been used to solve a wide variety of problems such as image labeling, protein folding, and data collection.

Compared with mainstream games for entertainment, one hurdle compounding HCG development is that these games suffer the design problem of serving two different goals. On the one hand, an HCG must provide a sufficiently-engaging experience for its players. On the other hand, an HCG must enable players to successfully complete the underlying human computation task. Balancing these two goals is difficult and often results in conflicting design decisions. To compound this dilemma, very little design knowledge exists beyond a small number of simple patterns from examples or takeaways from successful games (e.g., (Cooper et al., 2010b; von Ahn and Dabbish, 2008)). This lack of knowledge is intimidating to task providers and HCG developers, who might find it difficult to justify the risk of an expensive development process when there are no guarantees that a game will be successful at a desired computational task. As a result, most HCGs to date are built around specific kinds of templates, leaving the space of possible HCG designs limited and relatively unexplored.

To facilitate broader adoption and ease of game development, human computation game design needs the tools and frameworks to study and communicate about these games in a consistent manner. We need to understand precisely what game elements make certain HCGs successful, that is both effective at engaging players and solving tasks. Building up comprehensive, reusable design knowledge for HCGs that addresses both players and tasks enables us to construct formal representations for tasks, audiences, and game elements. A common language and structure for HCGs would allow us to talk about and explore the space of possible HCG designs, thus broadening the diversity of HCGs as a whole.

Additionally, this knowledge could also be used to build formal models that can predict the effect of tasks, audiences, and game elements on both aspects of the player experience—engagement and retention—and the completed task metrics—data quality, volume, and diversity. All of these affordances would help to make HCGs more successful interfaces for solving new tasks while still able to engage new player audiences with changing player preferences.

In this paper, we contend that reusable design knowledge for human computation games is necessary and should take the needs of both players and tasks into account. We introduce a formal representation of HCG mechanics that provides us with a common vocabulary and structure to visualize, compare, and explore the space of game mechanics in HCGs. We advocate a methodology for building up HCG design knowledge, which uses small-scale, controlled design experiments on tasks with known solutions to understand how variations of game elements affect the player experience and the completion of human computation tasks. Finally, we illustrate how our representation, combined with this methodology, enables the comparative study of existing HCGs and the exploration of HCG mechanics through novel designs.

2. Background

Human computation is the process of using people to solve computational problems as an alternative for current algorithms (Law and von Ahn, 2011). Such problems are often broken into smaller tasks, which are completed by crowdsourced workers, and then aggregated into larger solutions. This paradigm has been used for tasks including classifying objects, ranking items, summarizing texts, and iterative document editing and improvement (Law and von Ahn, 2011). Some human computation or crowdsourcing tasks rely on volunteer efforts, where workers are motivated by participation in scientific processes or altruistic goodwill. More common are online interfaces such as Amazon Mechanical Turk and CrowdFlower, which give task providers a platform to recruit workers, distribute tasks, and provide monetary compensation.

Human computation games have been developed as an alternative to these monetary incentive systems, utilizing game mechanics to enable task completion and providing players an engaging gameplay experience, in addition or as an alternative to financial compensation. The original Game With a Purpose, the ESP Game, addressed the problem of labeling images (von Ahn and Dabbish, 2004), but the breadth of human-computed tasks has greatly diversified since. HCGs have been used to annotate or classify other kinds of information, from music (Barrington et al., 2012; Law and von Ahn, 2009) to galaxies (Lintott et al., 2008), to relational information (e.g., ontology construction) (Krause et al., 2010; Siorpaes and Hepp, 2008), to protein function recognition (Peplow, 2016). Other HCGs have leveraged human players as alternatives to optimization functions in order to manipulate and refine existing input information. Common tasks of this type are often “scientific discovery” problems such as protein (Cooper et al., 2010a) and RNA folding (Lee et al., 2014), DNA multiple sequence alignment (Kawrykow et al., 2012), software verification (Dietl et al., 2012; Logas et al., 2014), and mapping dataflow diagrams onto hardware architectures (Mehta et al., 2013). Additionally, HCGs have used crowdsourced audiences to collect or generate new information, such as creative content or datasets for machine-learning algorithms. These tasks include photo collection (Tuite et al., 2011, 2015), location tagging (Bell et al., 2009; Frazier and Riedl, 2014), and commonsense knowledge acquisition (Kuo et al., 2009). Comprehensive taxonomies (Krause and Smeddinck, 2011; Pe-Than et al., 2013) detail a wide breadth of HCGs and the tasks they have tackled.

Human computation game design has been primarily guided by examples of successful games. These include von Ahn and Dabbish’s templates for classification and labeling tasks (von Ahn and Dabbish, 2008) and the design anecdotes of Foldit (Cooper et al., 2010b) rather than systematic study of HCG elements. As a result, we do not understand what specific elements of these particular design choices work and how to appropriately generalize them or consider alternatives. Confounding this issue is the fact that HCG research remains divided on how game elements, in particular game mechanics, can ensure both successful completion of tasks and engaging player experiences. Some argue that HCG game mechanics should be isomorphic or non-orthogonal to the underlying task, that is game mechanics should map to the process of solving the underlying task (Jamieson et al., 2012; Tuite, 2014). Others argue that incorporation or adaptation of game mechanics from successful digital games designed for entertainment can leverage player familiarity with existing games and keep them more engaged (Krause et al., 2010). This ongoing debate highlights the challenge of designing HCGs that optimize for both a positive player experience and a successfully-completed human computation task.

Prior research has demonstrated that qualitative and quantitative research methods can be used to study game design including methods from game usability (Isbister and Schaffer, 2008), game analytics (Seif El-Nasr et al., 2013), and visual analysis (Wallner and Kriglstein, 2013). Controlled studies have proven successful for understanding the influence of general game design elements, particularly in dual-purpose domains such as educational games. Such studies have investigated game aspects including difficulty (Lomas et al., 2013; Whitlock et al., 2014), controls (Lankes et al., 2014; McEwan et al., 2014), and tutorials (Andersen et al., 2011, 2012; Liu et al., 2014; Shannon et al., 2013).

General crowdsourcing and human computation research has worked to formalize the design and presentation of crowdsourcing tasks (Law and von Ahn, 2011). Controlled studies in this domain focus on evaluating how design variations affect worker efficiency and task completion, examples of which include the kinds of tasks (Doroudi et al., 2016) and the means of motivating workers (Law et al., 2016). How these might apply to human computation games in a way that addresses the necessity to engage players remains unexplored.

It is only recently that researchers have conducted similar controlled studies on specific game elements of HCGs that jointly address aspects of the player experience and the completion of the human computation task (Goh et al., 2011), and advocated for their use (Siu et al., 2014). We discuss these, along with other relevant examples in subsequent sections. Combined with formal crowdsourcing research, we posit that these approaches can enable a formal study of HCG design, allowing HCGs to more effectively address a broader range of tasks in ways more satisfying to players.

3. Formalizing HCG Mechanics

We propose a formal representation of the mechanics of human computation games. This representation serves three core functions:

1. Provides a common vocabulary and visual organization of HCG elements
2. Enables formal comparison of existing HCGs to understand the space of HCG designs and their consequences
3. Facilitates the formulation of controlled design experiments of HCG elements to build further, generalizable knowledge of HCG design

We specifically formalize human computation game mechanics—the rules that define how a player can interact with the game systems—leaving other elements of HCG designs to future work. We divide HCG game mechanics into three types: action mechanics, verification mechanics, and feedback mechanics. As shown in Figure 1, this breakdown reflects the core gameplay loop of most HCGs. HCGs begin with players taking in-game actions, then compare task-relevant input from these actions through verification mechanisms, and finally use verification output to provide feedback or reward for players.

Figure 1. Breakdown of HCG mechanics. Players provide inputs to take actions (shown in blue), which are verified (shown in orange), and receive feedback (shown in gray) from the game. Solid lines represent transitions through the gameplay loop.

We now define and describe these three sets of mechanics in detail, illustrated using three successful HCGs spanning different tasks: the original ESP Game (von Ahn and Dabbish, 2004), Foldit (Cooper et al., 2010a), and PhotoCity (Tuite et al., 2011). Figure 2 shows the mechanical breakdown of these games into action, verification, and feedback mechanics. In addition to these three examples, we also highlight notable instances of other HCGs that have explored these subsets of mechanics and discuss aspects of these mechanics that merit further exploration.

Figure 2. Examples of HCGs (von Ahn and Dabbish, 2004; Cooper et al., 2010a; Tuite et al., 2011) subdivided into action, verification, and feedback mechanics. Arrows from feedback to players have been omitted for clarity.

3.1. Action Mechanics

Action mechanics are the interface for players to complete a human computation task through in-game actions or gameplay. These mechanics align with the process of solving the human computation task, often asking players to utilize skills necessary for solving the task during play. Such mechanics may be as simple as entering text input or as complicated as piloting a space ship in a virtual environment, and tend to vary based on the nature of the task.

Examples. In the ESP Game, players provide labels through text entry to solve the task of labeling given images. In Foldit, players are given a variety of spatial actions, such as handling or rotating components of a protein structure, to solve the task of “folding” a given protein into a minimal energy configuration. In PhotoCity, players navigate to a desired location and take pictures using their camera phones, which are later uploaded to a database and used to construct a 3D representation of the buildings in that location.

Discussion. In general, the action mechanics in human computation games have been closely aligned with the process of solving the human computation task. The three examples in Figure 2 adhere to this alignment of mechanics and task completion, suggesting that the mechanics of these games were designed first and foremost with the task in mind, as opposed to adapting the mechanics of existing games for entertainment.

Actual explorations of adapting mechanics from mainstream or popular digital games do exist for HCGs, but are few. For example, the game OnToGalaxy (Krause et al., 2010) converts an analogous classification task of that previously seen in games such as Ontogame (Siorpaes and Hepp, 2008) to a space shooter game akin to the classic arcade game Asteroids. Unfortunately no comparisons were conducted between these games; while OnToGalaxy was compared against the ESP Game, differences in tasks—not to mention game mechanics—make direct comparison and generalization difficult. Meanwhile, the game Gwario (Siu et al., 2017) was used in an experiment to explore singleplayer and multiplayer mechanics in HCGs, turning a classification task from prior work (Siu et al., 2014) into a platformer game resembling the classic action game Super Mario Bros.. To complement this study of game mechanic variations, the authors also solicited opinions from HCG developers and researchers, asking directly if mechanics from mainstream digital games should be adapted into HCGs. Diverging from prior HCG design theories (Jamieson et al., 2012; Tuite, 2014), these experts were positive towards such adaptation, though not without cautioning that such mechanics ought to be selected carefully to avoid compromising the task results.

One of the major concerns surrounding human computation games is that when compared against other games designed purely for entertainment, HCGs might be perceived as shallow (Tuite, 2014) and therefore risk failing to attract players. Adaptation of familiar or successful mechanics from digital games might serve to address this issue. Given that action mechanics are often the first mechanics that players encounter in a game, an important question is: can actions be used to engage players and attract certain audiences? Players accustomed to certain kinds of (action) mechanics may find some games easier to interact with, easing entry into the game and facilitating early successes. Conversely, familiar mechanics may raise expectations about the game that may or may not be met given the constraints of solving the task alongside expected gameplay. For example, an HCG adopting action mechanics from a successful game may negatively impact players’ experiences if the design quality of the HCG did not match that of the popular game or series.

3.2. Verification Mechanics

Verification mechanics combine the output of player actions to compute task-relevant outcomes. These mechanics can support task completion outcomes including the quality, volume, diversity, and the rate at which the data are acquired.

Examples. For many human computation tasks, consensus on player input often serves as verification. The ESP Game (and many of the games inspired by its structure) verify using an online agreement check that filters correct answers from incorrect answers using agreement between players (Figure 2). The ESP Game later added “taboo word” mechanics to promote data diversity through banning words once consensus on existing data was reached.

By contrast, both Foldit and PhotoCity, which emphasize data manipulation and collection, handle verification through a task-based evaluation function. Foldit’s protein configuration energy function determines the quality of player solutions. In PhotoCity, the game does not explicitly evaluate the provided photos; photos are instead processed on an offline server and then player feedback is based on the resulting alterations to a constructed 3D mesh of the world.

Foldit also makes use of social mechanics, such as allowing players to share solution procedures (called “recipes”) through its community interfaces, as an additional (but optional) instance of verification (Cooper et al., 2011). Players can utilize existing recipes uploaded by other players as a starting point for solving tasks, thus validating and iterating on pre-existing, partial solution strategies, and can also rate their utility.

Discussion. Verification requirements for a task impact design decisions related to the number of players required to play the game and how they might interact with each other. Whether or not verification is handled within the human computation game or as an offline process may impact if a game uses singleplayer mechanics, multiplayer mechanics, or both. Many human computation tasks rely on consensus, and multiple players permits synchronous consensus, as shown in the ESP Game. Other tasks, which can be evaluated using an objective function or comparison against existing data, may permit asynchronous (singleplayer) play, such as Foldit and PhotoCity. Synchronous verification often requires multiple players to be playing at the same time, necessitating a fast verification step for real-time feedback. Note that simulated players can be used to re-verify old solutions (e.g., utilized in the ESP Game), especially where multiple players are required, but are not necessarily available concurrently.

As with action mechanics, few design experiments have tested alternative verification mechanics in HCGs. The game KissKissBan (Ho et al., 2010) modified the original structure of the ESP Game to promote data diversity (i.e., a broader set of image labels), by adding a third player to “ban” common words. Compared with the ESP Game’s eventual use of taboo words, KissKissBan demonstrates an alternative verification mechanic that players found engaging.

In most human computation games (i.e., those which rely on synchronous consensus), players are commonly forbidden from direct communication with each other (often implemented through anonymous pairing with no means of determining the identity of the other player). Very few games (e.g., Foldit) allow players to communicate through channels of communication such as game forums or community interfaces. This is meant to minimize collusion between players, which may lead to players providing deliberately incorrect answers while still succeeding at the game. The game Gwario (Siu et al., 2017) was used to compare singleplayer and co-located multiplayer, which allowed players to communicate directly during side-by-side play. Study results found no negative impact from direct communication; collusion was instead found to be the strongest predictor of high task accuracy. This suggests that direct communication may be permissible for certain tasks, audiences, and kinds of games.

Overall, we believe that verification mechanics are ripe with questions for future exploration. How do different ways of determining agreement influence task results? Would different mechanisms have downstream consequences for the feedback players can receive, for example, highlighting flaws in player-provided inputs?

3.3. Feedback Mechanics

Feedback mechanics provide players with information or digital artifacts based on the results of player actions in terms of partial or full task completion. These mechanics commonly encompass gameplay elements such as rewards and scoring, and can also be mapped to evaluation metrics for the underlying task, thus allowing both researchers and designers to assess player performance at both the completion of the task and progression through the in-game experience.

Examples. For all of the games shown in Figure 2, players receive feedback in the form of a score. However, the scale of the scoring mechanics themselves are unique to the tasks performed. The ESP Game rewards players with points for agreement on an image label. By contrast, Foldit rewards players with points for minimizing an energy function describing the protein structure. PhotoCity rewards players for the number of points their photo choices add to the reconstructed 3D mesh. These examples are all similar in that the feedback “currency” is nominal—points contributing to a numerical score—but vary in what players are rewarded for.

Discussion. Aspects of feedback mechanics are some of the most well-explored design elements in human computation games. These mechanics are synonymous with in-game rewards, which some consider comparable to monetary compensation in other crowdsourcing systems. So what should rewards to the player represent: player actions in the game, player performance at the task (as evaluated by verification mechanics), or a combination of both?

Traditionally, positive feedback in human computation games is designed to encourage participation in the crowdsourcing process, as well as correctness of task completion. For example, players may receive rewards for first completing a task and, if such verification is available, additional rewards for completing it correctly. This kind of positive feedback often rewards collaborative player behavior, which is reasonable given that crowdsourcing is an aggregate process. However, reward systems in games, such as points and leaderboards, are often designed to afford and encourage competitive player behavior. A variety of research has examined in-game rewards for collaborative versus competitive behaviors, specifically in the context of HCGs where competition may be discouraged as a distraction from the task (i.e., players will optimize their behaviors to outperform others, rather than to complete the task sufficiently). Both the game KissKissBan (Ho et al., 2010) and a study by Goh et al. (Goh et al., 2011) examine collaboration and competition in the context of the ESP Game. KissKissBan explored the introduction of competitive mechanics by rewarding a third player for antagonistic behavior that would encourage the other players to provide more diverse image labels. Goh et al. compared collaborative and competitive scoring using two versions of the ESP Game—one which rewarded players for answer agreement and the other which only rewarded the first player to an agreed answer. They found no significant differences when looking at player experience metrics and task results, which suggests that rewarding players for competitive behavior had no negative effects on the completion of the task. Likewise, Siu et al. (Siu et al., 2014) compared collaborative and competitive versions of the game Cabbage Quest, finding no significant differences in task completion such as accuracy and rate of task completion, but that players found the competitive version more engaging. Finally, the previously-mentioned game Gwario (Siu et al., 2017) compared collaborative and competitive scoring in its multiplayer condition, finding that players yielded more accurate (but slower) task results in the collaborative version but found the competitive version more challenging.

In addition to examining what players are rewarded for, one must also consider what players are rewarded with. Players have different motivations for playing games (Yee, 2006) and thus may differ in their preferred kinds of feedback. Crowdsourcing research has found that players who are intrinsically-motivated to solve tasks are often disengaged by monetary compensation (Mason and Watts, 2010) and that curiosity can be a strong incentive for crowdsourced work (Law et al., 2016). This suggests that standard in-game rewards (i.e., points), which may take the place of monetary compensation, may not appeal to all players and that players who are dedicated to the task itself might disengage with games that only provide one kind of reward. In their analysis of Foldit, Cooper et al. (Cooper et al., 2010b) identify a subset of players who are intrinsically motivated to participate in scientific discovery games, which highlights that such intrinsically-motivated players do exist.

So what are the alternatives to points and leaderboards for players who may not find typical feedback systems compelling? Researchers have explored the use of different reward systems. Goh et al. (Goh et al., 2015) compared different versions of a location-based content sharing HCG—one of which rewarded players with points and another which rewarded players with collectible badges—against a control version with no in-game rewards (but which displayed non-gamified progress and statistics). Overall, few differences in player engagement and task results were found between the versions utilizing points and badges, though both were considered more effective and engaging than the control. Siu and Riedl (Siu and Riedl, 2016) investigated multiple reward systems—leaderboards, customizable avatar items, unlockable narrative, and ungamified statistics—using the game Cafe Flour Sack in a multivariate experiment with two different player audiences: expert crowdsourced workers and a non-expert student population. They found that offering players a choice of reward from different systems (e.g., leaderboards, customizable avatar items, and unlockable narrative), as opposed to randomly assigning rewards from these systems, resulted in a better player experience with no differences in task results. When player audiences were taken into account, expert crowdsourced workers proved to be the most effective, but non-experts could perform nearly as well by being offered the ability to choose which rewards they desired. Gaston and Cooper (Gaston and Cooper, 2017) explored the use of three-star reward systems in the context of Foldit. They found that the introduction of three-star reward systems—mechanics awarding players up to three “stars” based on their performance of a game level or challenge—resulted in players completing levels in fewer, longer moves. Additionally players were more likely to replay levels, suggesting that such reward systems could encourage players to interact with the game in desirable ways (e.g., replaying levels to reinforce mastery of concepts in games like Foldit, which must teach players strategies necessary to complete the task). These explorations suggest that different reward systems and their presentation may have effects on the player experience and the completion of the task. Targeted player audiences may also be affected by how and what kinds of rewards are available, and may behave differently under different conditions.

Finally, we highlight Project Discovery (Peplow, 2016) as a unique example of an HCG that is embedded as a playable experience accessible in a mainstream game: the online game EVE Online. Thematically, the HCG is set in the EVE Online universe and players are rewarded with currency that can be spent within the larger game world. Direct integration of HCGs with existing games (moreover, those designed primarily for entertainment) remains otherwise unexplored, but this example demonstrates a potential way to leverage existing player bases of other games using feedback mechanics.

4. A Methodology for HCG Design

Our mechanics representation provides a breakdown of the different kinds of mechanics in human computation games. This enables us to identify where we can focus our explorations of the HCG design space, but not how we should explore the space in order to build up generalizable design knowledge.

We propose a methodology of controlled A/B design experiments that explore the space of human computation game designs, using formal representations for game elements, tasks, and audiences. In the context of HCG mechanics, this manifests as between-subjects (alternatively, within-subjects) experiments comparing separate versions of HCGs with different mechanical variations.

These design experiments should (1) implement a task with a known solution, while (2) focusing on a single element of a HCG’s design. First, testing with a known solution allows us to evaluate task-related metrics objectively without simultaneously solving a novel problem. Such known solutions may be the result of pre-solved human computation problems (e.g., image labeling, as discussed in Section 4.1) or simpler tasks (often those relying on commonsense human knowledge) that are analogous to existing problems. Second, focusing on one particular element of an HCG’s design allows us to understand exactly what kind of impact an element may have on both players and the task with minimal interaction effects. Our mechanics representation can be used to assist us in understanding where and how the introduction of an element (which may affect any of the action, verification, and feedback mechanics) will affect the HCG game loop.

These experiments should simultaneously evaluate how design decisions meet the needs of players and tasks. Optimizing only for the player may result in a game with mechanics that do not effectively solve the human computation task even if the game is considered engaging. Optimizing only for the task may result in a game that players do not find engaging (and thus will not play) even if the game effectively solves the human computation task. We refer to these two axes of metrics as the player experience and the task completion.

Player experience encompasses metrics such as:

• Engagement: how players interact with the game or rate their experience with it
• Retention: how likely are players to continue playing
• Other subjective measures related to how players interact and perceive the game (e.g., preferences, unstructured self-reported feedback)

Task completion refers to the task-related metrics such as:

• Quality: correctness or accuracy of task results
• Volume: amount of completed tasks
• Diversity: the variation or breadth of task results
• Rate of Acquisition: how quickly tasks are completed

The exact metrics to test for often depend on the nature of the human computation task and the HCG’s target player audiences. For example, HCGs for tasks that have a high bar for teaching or training players to solve them effectively may consider metrics such as player retention much more important than HCGs for simpler tasks where maintaining a skilled player base is not a priority.

Moreover, these requirements may change over time. Task providers may find that their initial task completion results may not be sufficient or need additional refinement. Should they wish to change existing mechanics or introduce new ones, it is imperative to understand how those design changes can ensure the necessary results while still maintaining a positive player experience. Likewise, player audiences (not to mention their preferences) may change, especially if an HCG wishes to broaden its reach to new or different target populations of players.

We note that this methodology is not new, as similar experimental testing approaches have been applied to several instances of HCGs. Goh et al. (Goh et al., 2011) compared a non-gamified control application for image labeling against two versions of the ESP Game, one using collaborative scoring mechanisms and the other using competitive scoring mechanisms. They evaluated both results of the image labeling task and aspects of the application or game experience across these three conditions. We will further examine this experiment in Section 4.1. Similarly, Siu et al. (Siu et al., 2014) conducted an experiment using the game Cabbage Quest to test collaborative and competitive scoring mechanisms. The study utilized a task with a known solution—categorizing everyday objects with purchasing locations—and the controlled experiment compared two in-game scoring mechanisms: one collaborative and one competitive. Task results were compared to a gold standard to evaluate task completion metrics, while player actions and survey responses were logged to evaluate player engagement and experience. Both Goh et al.’s ESP Games and Cabbage Quest follow our proposed methodology of taking a problem with a known solution or gold-standard answer set, testing design elements by treating a set of game mechanics as independent variables, and measuring aspects of both the player experience and task completion. We note that these experiments benefited from having a gold-standard answer set, but in some cases, a preexisting solution may not be available for new problems. In such cases, it may be sufficient to evaluate first for player experience, followed by later evaluation of task completion upon verification of initial task results.

We now discuss how a controlled methodology of systematically testing game elements in human computation games, combined with our mechanics representation, can be used to study, compare, and explore HCG designs. We provide two case studies as illustrative examples: (1) a comparison of image labeling games and studies of their design space and (2) a discussion of several studies that have utilized (and expanded) this methodology to test multiple aspects of HCGs, including player audiences and novel game elements.

4.1. Case Study: Comparison and Evolution of Image Labeling Games

Figure 3. Mechanics breakdown of three image labeling HCGs (von Ahn and Dabbish, 2004; Ho et al., 2010; Goh et al., 2011). On the left is the original ESP Game, followed by subsequent variations that modified elements of its original design. The mechanics of the original ESP Game are colored in gray; novel mechanical variations are colored in white.

Image labeling, the task of annotating or classifying images with subject labels or tags, is one of the most iconic and well-studied tasks in human computation games. Figure 3 illustrates our mechanics representation of three HCGs designed to solve the image labeling problem, beginning with the original ESP Game (von Ahn and Dabbish, 2004) followed by KissKissBan (Ho et al., 2010) and Goh et al.’s ESP Games (Goh et al., 2011). Our mechanics representation enables us to identify where changes in game mechanics were made in later games; Figure 3 shows the mechanical structure of the original ESP Game highlighted in gray on top of later iterations. Furthermore, we can visualize how these later games differ, specifically in their explorations of competitive game mechanics.

KissKissBan, previously discussed in Section 3, tackled the problem of generating a wider, more diverse set of labels for given images. While not shown in Figure 3, the original ESP Game eventually adopted the use of “taboo” words based on preexisting (repetitive) solutions that players could not input, forcing players to generate a wider range of labels. In this context, KissKissBan provides an alternative mechanical solution to the problem, specifically by introducing an additional player and with it, an additional verification step that did not rely on preexisting labels. We can see just how these mechanics were injected into the original design of the ESP Game in Figure 3. While the action mechanics of entering text input are identical to those of the first two players, the new design incorporates additional verification steps and feedback mechanics that can penalize the first two players.

Goh et al., as previously discussed in Section 3, conducted a study comparing collaborative and competitive versions of the ESP Game against a non-gamified control application. Their ESP Games are shown Figure 3 with the two experimental conditions emphasized in boldfaced braces. Using our mechanics breakdown, we can identify that the only major mechanical difference between the collaborative and competitive version of their ESP Games is in the particular scoring function used to provide feedback behaviors. Specifically, we can see just how their study isolated and varied one specific feedback mechanic: the scoring function.

Understanding how certain mechanics, or collections of mechanics are modified or added on top of existing games can help provide insight to apply them to other HCGs. Visually, Figure 3 gives us a better understanding of how one might take the changes utilized by KissKissBan or Goh et al.’s ESP Games and apply them to a structurally-similar ESP Game (which vonAhn and Dabbish’s templates refer to as “input-agreement” (von Ahn and Dabbish, 2008)).

Moreover, this helps to hypothesize just how these collections of mechanics are likely to affect the player experience and task completion. Unfortunately, while Goh et al. evaluated both player experience and task completion metrics in their experiment, KissKissBan only evaluated the latter against original ESP Game. This makes it difficult to compare how KissKissBan’s mechanics might compare to those of Goh et al.’s competitive ESP Game, particularly in the context of player experience. Ideally, as more comprehensive design knowledge is built up, HCG developers might be able to treat these collections of mechanics as modular when applying them to new tasks and integrating them into games, assisted by understanding of their potential effects on the player experience and task completion.

4.2. Case Study: Exploring Player Audiences and Novel Designs in HCGs

Figure 4. Mechanical breakdown of the Cafe Flour Sack experiment. Experimental conditions for reward distribution (random versus choice) and player audience (experts versus students) are noted using boldfaced braces.

As previously described, experiments using HCGs such as the ESP Game variations and Cabbage Quest have used a controlled methodology as highlighted in Section 4 to systematically test aspects of games, in particular feedback mechanics. We now report on two recent experiments that have taken this methodology further by exploring aspects of HCGs such as player audience, considering multivariate conditions (i.e., two independent variables), and challenging longstanding design hypotheses with novel game designs.

Cafe Flour Sack. Cafe Flour Sack, previously discussed in Section 3.3, was designed to test aspects of feedback mechanics (Siu and Riedl, 2016). The game uses a classification task with a known answer—pairing cooking ingredients with recipes that can be made from them—with drag-and-drop action mechanics for players to classify answers. A visualization of the experiment using our mechanics representation can be seen in Figure 4.

The primary feedback mechanic tested was the distribution of rewards: whether players were randomly assigned one of three possible reward types or were allowed to choose from those reward types. As shown in Figure 4, the variation occurred only in the feedback mechanics. The only difference between the two versions was a reward selection screen presented to the player: random—which showed players a randomly-assigned (highlighted) reward type—and choice—which allowed players to choose and click on one of three reward types. This interface change was kept deliberately minimal, but when looking only at the reward condition, results show that the choice condition was perceived more positively by players. Otherwise, players showed no significant differences in player experience or task completion.

In addition to testing random versus choice reward distribution, Cafe Flour Sack also tested these conditions across two different player audiences. One audience consisted of expert crowdsourcing workers recruited on Amazon Mechanical Turk, while the other audience consisted of university students. Predictably, experts performed overall better than students at task completion metrics such as task accuracy/correctness and rate of task completion. However, when looking at the interactions between the reward conditions, students given a choice of rewards performed nearly as well as experts given a choice of rewards. This suggests that under certain conditions such as the appropriate choice of mechanics, non-experts (i.e., students) can perform nearly as well as experts. We note that Cafe Flour Sack uses a simple commonsense knowledge task, but such differences could be more impactful for more complex tasks, although further replication and experimentation is needed to understand how these results hold.

While our proposed methodology focuses on using controlled studies on game mechanics, the Cafe Flour Sack experiment highlights the need to also consider other aspects of human computation games. As shown in Figure 4, our mechanics representation can also be extended to other aspects of HCGs, such as player audience. The results from Cafe Flour Sack suggest that testing conditions such as player audience is just as valuable as testing game mechanics, given the effects on the player experience and task completion metrics. We believe that our framework is flexible to such extensions. For example, while we advocate for fixing the task using this experimental methodology, one could hypothetically explore the effect that different tasks have on player experience and task completion by fixing mechanics of an HCG while using similar tasks.

Figure 5. Mechanical breakdown of the Gwario experiment. Experimental conditions for singleplayer and multiplayer are split for clarity, while multiplayer scoring conditions (collaborative versus competitive) are noted using boldfaced braces.

Gwario. Gwario, previously discussed in Section 3, was designed to test co-located multiplayer game mechanics in the context of an HCG adopting the mechanics of the popular platformer Super Mario Bros. (Siu et al., 2017). The game utilizes the task with a known solution as that in Cabbage Quest (Siu et al., 2014). A visualization of the experiment using our mechanics representation can be seen in Figure 5.

The game tested co-located multiplayer mechanics and consisted of two conditions: a singleplayer version and a co-located multiplayer version, in which players are seated side-by-side sharing the same game screen. Figure 5 shows that the mechanics of the singleplayer version are a deliberately-strict subset of the mechanics in the multiplayer version, which adds verbal communication between players as an optional verification step (i.e., players may choose whether or not to talk during gameplay). Additionally, within its multiplayer condition, Gwario also compared the experimental conditions described in Goh et al.’s previously-described ESP Games and Cabbage Quest: collaborative scoring and competitive scoring. Like these prior experiments, the only variation between the two scoring conditions was the scoring function used to evaluate players. The only visual difference between the two was the reward screen at the end of the game levels, showing either a joint score or separate scores.

Adoption of mainstream game mechanics still remains a contentious and relatively-unexplored area of human computation game design. Gwario’s adaptation of collection and 2D platforming mechanics from Super Mario Bros. represents a novel design in the space of HCGs. This raises the question: how can we safely adopt and test mechanics from successful mainstream games? When considering a particular genre or game, which mechanics do we adopt? The Gwario experiment demonstrates possible approaches, such adopting a secondary mechanic from Super Mario Bros.—coin collection—to the task of classifying objects as part of its action mechanics. Meanwhile, Super Mario Bros.’ primary navigation mechanics were left untouched to preserve the feel of the original game, but did not affect the completion of the human computation task. Additionally, direct communication—a social element—was treated as a verification mechanic. By breaking HCGs down using our mechanics representation, we can identify and isolate where adopted mainstream game elements could fit into the HCG game loop. This would allow us to carefully and iteratively test variations while ensuring that design changes to mechanics are isolated to particular parts of the game loop. Comparison through our proposed methodology then could help to identify potential tradeoffs between the player experience and task completion.

Additionally, the Gwario experiment also highlights the potential for different experimental methods in the context of human computation game design. Gwario’s survey of HCG experts, while limited, provides valuable insights into the design philosophies and considerations of HCG developers. While our proposed methodology is very amenable to quantitative methods for evaluating games, we believe that HCG design stands to benefit from qualitative and mixed method approaches, such as grounded-theory approaches and/or semi-structured interviews with both players and HCG developers/task providers.

5. Conclusions

In this paper, we describe a framework for designing and studying human computation games. We believe that generalizable design knowledge for HCGs is necessary, and should consider both players and tasks. We proposed a formal representation of HCG mechanics into three types: action, verification, and feedback, and illustrate it with examples of its potential use. This formalization allows us to visualize and study HCG elements, enable formal comparisons of existing HCGs, and facilitate controlled experiments to advance HCG design knowledge. We highlighted a methodology of running design experiments on known tasks that measure both player experience and task completion when varying game elements. We illustrated how to utilize both representation and methodology in two examples: a comparison of image labeling games and a discussion of recent HCG experiments.

Human computation games have demonstrated the potential to solve complex and difficult problems, but must be both engaging experiences for players and effective at solving their tasks. With the volume and availability of games increasing each day, HCGs must compete for players’ time and attention, and thus must remain relevant and consistent with player expectations. Otherwise, we risk tarnishing their reputation with players, which may ultimately lead to HCGs’ premature dismissal as ineffective interfaces for human computation work. In order to ensure this does not happen, we need to understand how HCGs work, to build better and broader generalizable design knowledge that can adapt to new games, tasks, and audiences, especially when HCG developers do not typically have the training or resources of professional game studios. Our mechanics representation and our proposed methodology are designed to explore and evaluate HCG mechanics so that it will be easier to design and develop successful, effective HCGs. In doing so, we hope to work towards a future where HCGs are engaging, effective, ubiquitous, and empowering.

Acknowledgements.

This material is based upon work supported by the National Science Foundation under Grant No. 1525967. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References

• (1)

• Andersen et al. (2011)

  Erik Andersen, Yun-En Liu, Richard Snider, Roy Szeto, Seth Cooper, and Zoran Popović. 2011. On the Harmfulness of Secondary Game Objectives. In Proceedings of the 6th International Conference on Foundations of Digital Games (FDG ’11). ACM, New York, NY, USA, 30–37.

  DOI:http://dx.doi.org/10.1145/2159365.2159370

• Andersen et al. (2012)

  Erik Andersen, Eleanor O’Rourke, Yun-En Liu, Rich Snider, Jeff Lowdermilk, David Truong, Seth Cooper, and Zoran Popović. 2012. The Impact of Tutorials on Games of Varying Complexity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 59–68.

  DOI:http://dx.doi.org/10.1145/2207676.2207687

• Barrington et al. (2012)

  Luke Barrington, Douglas Turnbull, and Gert Lanckriet. 2012. Game-powered machine learning. Proceedings of the National Academy of Sciences 109, 17 (2012), 6411–6416. DOI:http://dx.doi.org/10.1073/pnas.1014748109 arXiv:http://www.pnas.org/content/109/17/6411.full.pdf

• Bell et al. (2009)

  Marek Bell, Stuart Reeves, Barry Brown, Scott Sherwood, Donny MacMillan, John Ferguson, and Matthew Chalmers. 2009. EyeSpy: Supporting Navigation Through Play. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). ACM, New York, NY, USA, 123–132.

  DOI:http://dx.doi.org/10.1145/1518701.1518723

• Cooper et al. (2011)

  Seth Cooper, Firas Khatib, Ilya Makedon, Hao Lu, Janos Barbero, David Baker, James Fogarty, Zoran Popović, and Foldit players. 2011. Analysis of Social Gameplay Macros in the Foldit Cookbook. In Proceedings of the 6th International Conference on Foundations of Digital Games (FDG ’11). ACM, New York, NY, USA, 9–14.

• Cooper et al. (2010a)

  Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, and others. 2010a. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756–760.

• Cooper et al. (2010b)

  Seth Cooper, Adrien Treuille, Janos Barbero, Andrew Leaver-Fay, Kathleen Tuite, Firas Khatib, Alex Cho Snyder, Michael Beenen, David Salesin, David Baker, Zoran Popović, and Foldit Players. 2010b. The Challenge of Designing Scientific Discovery Games. In 5th International Conference on the Foundations of Digital Games.

• Dietl et al. (2012)

  Werner Dietl, Stephanie Dietzel, Michael D. Ernst, Nathaniel Mote, Brian Walker, Seth Cooper, Timothy Pavlik, and Zoran Popović. 2012. Verification Games: Making Verification Fun. In 14th Workshop on Formal Techniques for Java-like Programs.

• Doroudi et al. (2016)

  Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a Learning Science for Complex Crowdsourcing Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 2623–2634.

  DOI:http://dx.doi.org/10.1145/2858036.2858268

• Frazier and Riedl (2014)

  Spencer Frazier and Mark Riedl. 2014. Persistent and Pervasive Real-World Sensing Using Games. In Second AAAI Conference on Human Computation and Crowdsourcing.

• Gaston and Cooper (2017)

  Jacqueline Gaston and Seth Cooper. 2017. To Three or Not to Three: Improving Human Computation Game Onboarding with a Three-Star System. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 5034–5039.

  DOI:http://dx.doi.org/10.1145/3025453.3025997

• Goh et al. (2011)

  Dion Hoe-Lian Goh, Rebecca P. Ang, Chei Sian Lee, and Alton Y. K. Chua. 2011. Fight or Unite: Investigating Game Genres for Image Tagging. J. Am. Soc. Inf. Sci. Technol. 62, 7 (July 2011), 1311–1324.

• Goh et al. (2015)

  Dion Hoe-Lian Goh, Ei Pa Pa Pe-Than, and Chei Sian Lee. 2015. An Investigation of Reward Systems in Human Computation Games. Springer International Publishing, 596–607.

• Ho et al. (2010)

  Chien-Ju Ho, Tao-Hsuan Chang, Jong-Chuan Lee, Jane Yung-jen Hsu, and Kuan-Ta Chen. 2010. KissKissBan: A Competitive Human Computation Game for Image Annotation. SIGKDD Explor. Newsl. 12, 1 (Nov. 2010), 21–24.

  DOI:http://dx.doi.org/10.1145/1882471.1882475

• Isbister and Schaffer (2008)

  Katherine Isbister and Noah Schaffer. 2008. Game Usability: Advancing the Player Experience. Morgan Kaufmann.

• Jamieson et al. (2012)

  Peter Jamieson, Lindsay Grace, and Jack Hall. 2012. Research Directions for Pushing Harnessing Human Computation to Mainstream Video Games. In Meaningful Play.

• Kawrykow et al. (2012)

  Alexander Kawrykow, Gary Roumanis, Alfred Kam, Daniel Kwak, Clarence Leung, Chu Wu, Eleyine Zarour, Luis Sarmenta, Mathieu Blanchette, Jérôme Waldispühl, and Phylo Players. 2012. Phylo: a citizen science approach for improving multiple sequence alignment. PLoS One 7, 3 (2012), e31362.

• Krause and Smeddinck (2011)

  Markus Krause and Jan Smeddinck. 2011. Human computation games: A survey. In 2011 19th European Signal Processing Conference. IEEE, 754–758.

• Krause et al. (2010)

  Markus Krause, Aneta Takhtamysheva, Marion Wittstock, and Rainer Malaka. 2010. Frontiers of a Paradigm: Exploring Human Computation with Digital Games. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’10). ACM, New York, NY, USA, 22–25.

  DOI:http://dx.doi.org/10.1145/1837885.1837893

• Kuo et al. (2009)

  Yen-ling Kuo, Jong-Chuan Lee, Kai-yang Chiang, Rex Wang, Edward Shen, Cheng-wei Chan, and Jane Yung-jen Hsu. 2009. Community-based Game Design: Experiments on Social Games for Commonsense Data Collection. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’09). ACM, New York, NY, USA, 15–22.

  DOI:http://dx.doi.org/10.1145/1600150.1600154

• Lankes et al. (2014)

  Michael Lankes, Thomas Mirlacher, Stefan Wagner, and Wolfgang Hochleitner. 2014. Whom Are You Looking for?: The Effects of Different Player Representation Relations on the Presence in Gaze-based Games. In Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play (CHI PLAY ’14). ACM, New York, NY, USA, 171–179.

  DOI:http://dx.doi.org/10.1145/2658537.2658698

• Law and von Ahn (2009)

  Edith Law and Luis von Ahn. 2009. Input-agreement: A New Mechanism for Collecting Data Using Human Computation Games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). ACM, New York, NY, USA, 1197–1206.

  DOI:http://dx.doi.org/10.1145/1518701.1518881

• Law and von Ahn (2011)

  Edith Law and Luis von Ahn. 2011. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5, 3 (2011), 1–121.

• Law et al. (2016)

  Edith Law, Ming Yin, Joslin Goh, Kevin Chen, Michael A. Terry, and Krzysztof Z. Gajos. 2016. Curiosity Killed the Cat, but Makes Crowdwork Better. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 4098–4110.

  DOI:http://dx.doi.org/10.1145/2858036.2858144

• Lee et al. (2014)

  Jeehyung Lee, Wipapat Kladwang, Minjae Lee, Daniel Cantu, Martin Azizyan, Hanjoo Kim, Alex Limpaecher, Sungroh Yoon, Adrien Treuille, Rhiju Das, and EteRNA Participants. 2014. RNA design rules from a massive open laboratory. Proceedings of the National Academy of Sciences 111, 6 (2014), 2122–2127.

• Lintott et al. (2008)

  Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven Bamford, Daniel Thomas, M Jordan Raddick, Robert C Nichol, Alex Szalay, Dan Andreescu, Phil Murray, and Jan Vandenberg. 2008. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179–1189.

• Liu et al. (2014)

  Yun-En Liu, Travis Mandel, Emma Brunskill, and Zoran Popović. 2014. Trading Off Scientific Knowledge and User Learning with Multi-Armed Bandits. In Educational Data Mining.

• Logas et al. (2014)

  Heather Logas, Jim Whitehead, Michael Mateas, Richard Vallejos, Lauren Scott, Dan Shapiro, John Murray, Kate Compton, Joseph Osborn, Orlando Salvatore, Zhongpeng Lin, Huascar Sanchez, Michael Shavlovsky, Daniel Cetina, Shayne Clementi, and Chris Lewis. 2014. Software Verification Games: Designing Xylem, The Code of Plants. In Proceedings of the 9th International Conference on the Foundations of Digital Games.

• Lomas et al. (2013)

  Derek Lomas, Kishan Patel, Jodi L. Forlizzi, and Kenneth R. Koedinger. 2013. Optimizing Challenge in an Educational Game Using Large-scale Design Experiments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, 89–98.

  DOI:http://dx.doi.org/10.1145/2470654.2470668

• Mason and Watts (2010)

  Winter Mason and Duncan J. Watts. 2010. Financial Incentives and the ”Performance of Crowds”. SIGKDD Explor. Newsl. 11, 2 (May 2010), 100–108.

  DOI:http://dx.doi.org/10.1145/1809400.1809422

• McEwan et al. (2014)

  Mitchell W. McEwan, Alethea L. Blackler, Daniel M. Johnson, and Peta A. Wyeth. 2014. Natural Mapping and Intuitive Interaction in Videogames. In Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play (CHI PLAY ’14). ACM, New York, NY, USA, 191–200.

  DOI:http://dx.doi.org/10.1145/2658537.2658541

• Mehta et al. (2013)

  Gayatri Mehta, Carson Crawford, Xiaozhong Luo, Natalie Parde, Krunalkumar Patel, Brandon Rodgers, Anil Kumar Sistla, Anil Yadav, and Marc Reisner. 2013. UNTANGLED: A Game Environment for Discovery of Creative Mapping Strategies. ACM Trans. Reconfigurable Technol. Syst. 6, 3, Article 13 (Oct. 2013), 26 pages.

  DOI:http://dx.doi.org/10.1145/2517325

• Pe-Than et al. (2013)

  Ei Pa Pa Pe-Than, Dion Hoe-Lian Goh, and Chei Sian Lee. 2013. A typology of human computation games: an analysis and a review of current games. Behaviour & Information Technology (2013).

• Peplow (2016)

  Mark Peplow. 2016. Citizen science lures gamers into Sweden’s Human Protein Atlas. Nature Biotechnology 34, 5 (2016), 452–452. http://dx.doi.org/10.1038/nbt0516-452c

• Seif El-Nasr et al. (2013)

  Magy Seif El-Nasr, Anders Drachen, and Alessandro Canossa (Eds.). 2013. Game Analytics. Springer London.

• Shannon et al. (2013)

  Amy Shannon, Acey Boyce, Chitra Gadwal, and Tiffany Barnes. 2013. Effective Practices in Game Tutorial Systems. In Proceedings of the 8th International Conference on the Foundations of Digital Games.

• Siorpaes and Hepp (2008)

  Katharina Siorpaes and Martin Hepp. 2008. Games with a Purpose for the Semantic Web. IEEE Intelligent Systems 23, 3 (2008), 50–60.

• Siu et al. (2017)

  Kristin Siu, Matthew Guzdial, and Mark O. Riedl. 2017. Evaluating Singleplayer and Multiplayer in Human Computation Games. In Proceedings of the 12th International Conference on the Foundations of Digital Games. to appear.

• Siu and Riedl (2016)

  Kristin Siu and Mark O. Riedl. 2016. Reward Systems in Human Computation Games. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’16). ACM, New York, NY, USA, 266–275.

  DOI:http://dx.doi.org/10.1145/2967934.2968083

• Siu et al. (2014)

  Kristin Siu, Alexander Zook, and Mark O. Riedl. 2014. Collaboration versus Competition: Design and Evaluation of Mechanics for Games with a Purpose. In Proceedings of the 9th International Conference on the Foundations of Digital Games.

• Tuite (2014)

  Kathleen Tuite. 2014. GWAPs: Games with a Problem. In Proceedings of the 9th International Conference on the Foundations of Digital Games.

• Tuite et al. (2015)

  Kathleen Tuite, Rahul Banerjee, Noah Snavely, Jovan Popović, and Zoran Popović. 2015. PointCraft: Harnessing Players’ FPS Skills to Interactively Trace Point Clouds in 3D. In Proceedings of the 10th International Conference on the Foundations of Digital Games.

• Tuite et al. (2011)

  Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popović. 2011. PhotoCity: Training Experts at Large-scale Image Acquisition Through a Competitive Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 1383–1392.

  DOI:http://dx.doi.org/10.1145/1978942.1979146

• von Ahn and Dabbish (2004)

  Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04). ACM, New York, NY, USA, 319–326.

  DOI:http://dx.doi.org/10.1145/985692.985733

• von Ahn and Dabbish (2008)

  Luis von Ahn and Laura Dabbish. 2008. Designing Games with a Purpose. Commun. ACM 51, 8 (2008), 58–67.

• Wallner and Kriglstein (2013)

  G. Wallner and S. Kriglstein. 2013. Visualization-based analysis of gameplay data — A review of literature. Entertainment Computing 4, 3 (2013), 143 – 155.

• Whitlock et al. (2014)

  Laura A. Whitlock, Anne Collins McLaughlin, William Leidheiser, Maribeth Gandy, and Jason C. Allaire. 2014. Know Before you Go: Feelings of Flow for Older Players Depends on Game and Player Characteristics. In ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play.

• Yee (2006)

  Nick Yee. 2006. Motivations for play in online games. CyberPsychology & behavior 9, 6 (2006), 772–775.