Published on 15.07.16 in Vol 4, No 2 (2016): Jul-Dec
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/5888, first published Apr 21, 2016.
Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy
Background: Cognitive tasks are typically viewed as effortful, frustrating, and repetitive, which often leads to participant disengagement. This, in turn, may negatively impact data quality and/or reduce intervention effects. However, gamification may provide a possible solution. If game design features can be incorporated into cognitive tasks without undermining their scientific value, then data quality, intervention effects, and participant engagement may be improved.
Objectives: This systematic review aims to explore and evaluate the ways in which gamification has already been used for cognitive training and assessment purposes. We hope to answer 3 questions: (1) Why have researchers opted to use gamification? (2) What domains has gamification been applied in? (3) How successful has gamification been in cognitive research thus far?
Methods: We systematically searched several Web-based databases, searching the titles, abstracts, and keywords of database entries using the search strategy (gamif* OR game OR games) AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*). Searches included papers published in English between January 2007 and October 2015.
Results: Our review identified 33 relevant studies, covering 31 gamified cognitive tasks used across a range of disorders and cognitive domains. We identified 7 reasons for researchers opting to gamify their cognitive training and testing. We found that working memory and general executive functions were common targets for both gamified assessment and training. Gamified tests were typically validated successfully, although mixed-domain measurement was a problem. Gamified training appears to be highly engaging and does boost participant motivation, but mixed effects of gamification on task performance were reported.
Conclusions: Heterogeneous study designs and typically small sample sizes highlight the need for further research in both gamified training and testing. Nevertheless, careful application of gamification can provide a way to develop engaging and yet scientifically valid cognitive assessments, and it is likely worthwhile to continue to develop gamified cognitive tasks in the future.
JMIR Serious Games 2016;4(2):e11
Every day, millions of people play games, on computers, consoles, and mobile devices . One explanation for the massive popularity of games is that they can provide easy access to a sense of engagement and self-efficacy which reality may not deliver [ ]. By their nature, games present users with difficult challenges to overcome and use narrative structure, complex graphics, strategic elements, and intuitive rules to engage the user [ ]. This ability to engross users has recently begun to be leveraged for purposes beyond entertainment in the form of gamification. The aim of gamification is to use gamelike features (competition, narrative, leaderboards, graphics, and other game design elements) to transform an otherwise mundane task into something engaging and even fun.
In cognitive science, whether we are gathering data or trying to encourage behavior change, successfully engaging participants is vital. There is evidence that a lack of participant motivation has a negative impact on the quality of data collected , with tasks commonly being viewed as too boring and repetitive for participant attention to be sustained. This is a particular problem for Web-based studies where participants can simply close the browser window and drop out of the study the moment they decide it is not worth their time [ ].
If we are attempting to alter behavior (for example, cognitive bias modification or working memory training), then it is vital that our interventions are engaging. Gamification may help in this regard. If we can successfully import game design elements into cognitive tasks without undermining their scientific value, then we may be able to improve the quality of our data, increase the effectiveness of our interventions, and improve the experience for participants. Furthermore, using games as a vehicle to deliver cognitive training may also be advantageous simply because video games appear to have positive effects on a number of outcomes, including working memory, attentional capacity, problem solving, motivation, emotional control, and prosocial behaviors . In essence, delivering targeted cognitive training through a video game medium might provide a range of benefits.
Much of the literature on gamification is relatively recent, cross-disciplinary, and heterogeneous in nature. This lack of coherence in the field is partly due to poor definition of terms; for example, the gamelike tasks covered in this review could be described as “serious games,” “gamelike,” “gamified,” “games with a purpose,” “gamed-up,” or simply “computer based” [- ]. To our knowledge, there have been no systematic reviews of gamification in cognitive research. There have, however, been several reviews of the concepts core to gamification and how they can be applied successfully (see [ - ]). However, detailed discussion of these reviews is beyond the scope of this paper. Also related but beyond our scope is the body of work on the effects of video games on behavior (see [ , ] for reviews).
This paper aims to systematically review the ways in which gamification has already been used for the purposes of cognitive assessment and training. We were specifically interested in the following questions: (1) Why have researchers opted to use gamification? (2) What domains has gamification been applied in? (3) How successful has gamification been in cognitive research thus far? We deliberately used a broad search strategy so as not to miss any relevant papers. We reviewed only the peer-reviewed literature and have therefore deliberately excluded some cognitive training games available on the iTunes or Play Store (such as Luminosity and Peak) unless supported by peer-reviewed research.
Databases and Search Strategy
The following databases were searched electronically: PsycInfo, Medline, ETHOS, EMBASE, PubMed, IBSS, Francis, Web of Science and Scopus. We searched the titles, abstracts, and keywords of database entries using the search strategy (gamif* OR game OR games) AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*), where * represents a wildcard to allow for alternative suffixes. Searches included papers published in English between January 2007 and October 2015. We searched the bibliographies of included papers to locate further relevant material not discovered in the database search.
Primary Research Paper
Included papers were empirical research studies, not literature reviews, opinion pieces, or design documents.
Novel Gamelike Task
Included papers focussed on newly developed gamelike tasks, created specifically for the study in question. We excluded commercially available video games (ie, “off-the-shelf” games) as well as gamelike tasks that have been in use for many years, such as Space Fortress (see  for a review).
Measure or Train Cognition
Included papers focussed on tasks designed to assess or train cognition. For scoping purposes, we took a narrow definition of cognition: those processes involved in memory, attention, decision making, impulse control, executive functioning, processing speed, and visual perception.
Validated or Piloted
Included papers had to involve an empirical study, either validating the task as a measure of cognition or piloting the intervention. Papers regarding usability testing alone were excluded.
We excluded non–peer-reviewed papers such as abstracts or conference posters.
Gamification in the Behavioral Sciences but not Involving Cognition
We excluded papers on gamification for education purposes, disease management, health promotion, exposure therapy, or rehabilitation.
Game Engines/Three-Dimensional (3D) Environments
We excluded papers that made use of virtual reality or a 3D environment without any game mechanics or gamelike framing.
We did not select papers based on whether they included “gamification.” Rather, papers were selected if they were captured by our search strategy and were considered relevant based on our inclusion/exclusion criteria. We intentionally did not strictly define gamification: as has been discussed famously by Wittgenstein and is alluded to by Deterding , trying to precisely define what elements make a game is both difficult and limiting. Therefore, we decided that a task was gamified if its stated purpose was to increase participant motivation. Where there was insufficient detail to determine whether a paper met our inclusion criteria, we erred on the side of caution to increase our confidence in the relevance of the studies reviewed.
After screening, data were extracted from each paper using a standardized data extraction form. Data relating to the questions of interest such as application of gamification, approach taken, and efficacy were extracted from each paper. Application of gamification refers to the field of psychology in which the gamelike task was used and why a gamelike task was used. Approach taken refers to the specific game mechanics used in the task and what themes and scaffolding were applied. Finally, efficacy refers to the findings produced by the task in practice, as well as details on the participants, methods used for evaluation, and limitations of the study. Categorization of concepts (such as the cognitive domains measured) was done using the paper-authors’ own words where possible. Where not possible, we mapped extracted concepts closely to existing categories.
All papers identified by our search strategy were screened by one reviewer (JL) in 3 stages, to determine whether they were relevant based on our inclusion/exclusion criteria: title, abstract, and full text. A second reviewer (EE) rescreened 20% of the papers from the title stage onward to ensure that no relevant papers were missed. Papers were only included in the review on the agreement of both JL and EE.
Our initial search yielded 33,445 papers (excluding duplicates). Of these, 23 papers from the original search and 4 papers from the manual reference search were included in the review. We repeated the search in October 2015, including papers from January 2015 until October 2015. This search produced 4448 papers (excluding duplicates) and resulted in another 4 papers being included in the review, with a further 2 also included following peer review. The total number of papers included in the review was therefore 33. Seefor a flowchart of the combined searches and for details of all included studies. We used Cohen K to assess inter-rater reliability of paper inclusion at the 20% data check stage (7590 papers checked). There was moderate agreement between the 2 reviewers (k=.526, 95% CI, 0.416 to 0.633, P<.001). All supplementary data referenced in this paper can be found in , whereas contains more detailed information on the games and game mechanics used by studies in the review.
|Author, year||Full title||Game||Category|
|McPherson and Burns, 2007 ||Gs invaders: Assessing a computer gamelike test of processing speed||Space Code||Testing|
|McPherson and Burns, 2008 ||Assessing the validity of computer gamelike tests of processing speed and working memory||Space Matrix/Space Code||Testing|
|Trapp et al, 2008 ||Cognitive remediation improves cognition and good cognitive performance increases time to relapse—results of a 5-year catamnestic study in schizophrenia patients||Xcog||Training|
|Gamberini et al, 2009 ||Eldergames project: An innovative mixed reality table-top solution to preserve cognitive functions in elderly people||Eldergames||Testing|
|Gamberini, Cardullo, Seraglia, and Bordin, 2010 ||Neuropsychological testing through a Nintendo Wii console||Wii Tests||Testing|
|Dovis, Oord, Wiers, and Prins, 2011 ||Can motivation normalize working memory and task persistence in children with attention-deficit/hyperactivity disorder? The effects of money and computer-gaming||Megabot||Testing|
|Delisle and Braun, 2011 ||A context for normalizing impulsiveness at work for adults with attention deficit/hyperactivity disorder (combined type)||Retirement Party||Testing|
|Prins, Dovis, Ponsioen, ten Brink, and van der Oord, 2011 ||Does computerized working memory training with game elements enhance motivation and training efficacy in children with ADHD?||Supermecha||Training|
|Lim et al, 2012 ||A brain-computer interface based attention training program for treating attention deficit hyperactivity disorder||Cogoland||Training|
|Heller et al, 2013 ||A machine learning-based analysis of game data for attention deficit hyperactivity disorder assessment||Groundskeeper||Testing|
|Hawkins et al, 2013 ||Gamelike features might not improve data||EM-Ants and Ghost Trap||Testing|
|Verhaegh, Fontijn, Aarts, and Resing, 2013 ||In-game assessment and training of nonverbal cognitive skills using TagTiles||Tap the Hedgehog||Both|
|Aalbers, Baars, Rikkert, and Kessels, 2013 ||Puzzling with online games (BAM-COG): reliability, validity, and feasibility of an online self-monitor for cognitive performance in aging adults||BAM-COG||Testing|
|Fagundo et al, 2013 ||Video game therapy for emotional regulation and impulsivity control in a series of treated cases with bulimia nervosa||Playmancer||Training|
|Anguera et al, 2013 ||Video game training enhances cognitive control in older adults||Neuroracer||Training|
|van der Oord, Ponsioen, Geurts, Ten Brink, and Prins, 2014 ||A pilot study of the efficacy of a computerized executive functioning remediation training with game elements for children with ADHD in an outpatient setting||Braingame Brian||Training|
|Brown et al, 2014 ||Crowdsourcing for cognitive science—the utility of smartphones||The Great Brain Experiment||Testing|
|Tong and Chignell, 2014 ||Developing a serious game for cognitive assessment: choosing settings and measuring performance||Whack-a-mole||Testing|
|Katz, Jaeggi, Buschkuehl, Stegman, and Shah, 2014 ||Differential effect of motivational features on training improvements in school-based cognitive training||WMTrainer||Training|
|Dunbar et al, 2013 ||Implicit and explicit training in the mitigation of cognitive bias through the use of a serious game||MACBETH||Training|
|Lee et al, 2013 ||A brain-computer interface based cognitive training system for healthy elderly: A randomised control pilot study for usability and preliminary efficacy||Card-Pairing||Training|
|Miranda and Palmer, 2013 ||Intrinsic motivation and attentional capture from gamelike features in a visual search task||Visual Search||Testing|
|Atkins et al, 2014 ||Measuring working memory is all fun and games: A four-dimensional spatial game predicts cognitive task performance||Shapebuilder||Testing|
|Dörrenbächer et al, 2014 ||Dissociable effects of game elements on motivation and cognition in a task switching training in middle childhood||Watermons||Training|
|McNab and Dolan, 2014 ||Dissociating distractor-filtering at encoding and during maintenance||The Great Brain Experiment||Testing|
|O’Toole and Dennis, 2014 ||Mental health on the go: Effects of a gamified attention-bias modification mobile application in trait-anxious adults||ABMTApp||Training|
|Tenorio Delgado, Arango Uribe, Aparicio Alonso, and Rosas Diaz, 2014 ||TENI: A comprehensive battery for cognitive assessment based on games and technology||TENI||Testing|
|De Vries, Prins, Schmand, and Geurts, 2015 ||Working memory and cognitive flexibility-training for children with an autism spectrum disorder: a randomized controlled trial||Braingame Brian||Training|
|Dovis, Van Der Oord, Wiers, and Prins, 2015 ||Improving executive functioning in children with ADHD: Training multiple executive functions within the context of a computer game. A randomized double-blind placebo controlled trial||Braingame Brian||Training|
|Kim et al, 2015 ||Effects of a serious game training on cognitive functions in older adults||Smart Harmony||Training|
|Manera et al, 2015 ||“Kitchen and cooking,” a serious game for mild cognitive impairment and Alzheimer’s disease: a pilot study||Kitchen and Cooking||Both|
|Ninaus et al, 2015 ||Game elements improve performance in a working memory training task||GAME||Training|
|Tarnanas et al, 2015 ||On the comparison of a novel serious game and electroencephalography biomarkers for early dementia screening||VAP-M||Testing|
aIn cases where the game was not named, we assigned a descriptive name.
Why Do Researchers Use Gamification?
We searched each paper for explanations as to why a gamelike task had been used and identified reasons for researchers opting to gamify their cognitive training and testing. These reasons were then coalesced into 7 categories. Some authors listed multiple reasons for gamifying their approach, whereas others gave no motivations at all. Supplementary Table 1 inprovides details of which games fell into each category.
To Increase Participant Motivation
Although we assume that increasing participant motivation was a goal for every study in this review, we found 16 studies that explicitly used gamified tasks to measure or train cognition in a more motivating manner, and the majority (10 of 16) of these studies were assessment studies. Cognitive tests are typically a one off measure, and replayability is not a requirement. As such, these games tended to build simple game-archetypes, such as space invaders or whack-a-mole, around an existing cognitive task; with the goal of encouraging self-motivation, improving participant enjoyment, and even reducing test anxiety . Gamification aims to increase engagement with tasks that might otherwise be perceived as demotivating [ ], and the highly repetitive nature of cognitive tasks means they are ripe for improvement.
To Increase Usability/Intuitiveness for the Target Age Group
Eleven tasks were gamified specifically to enhance appeal with a given age group (see Supplementary Table 2 in). The authors of these studies hypothesized that a more intuitive interface could prevent boredom and anxiety in the target age group, which might damage motivation and concentration on the tasks at hand. Six games were designed to be suitable for the elderly, who may not be familiar working with a mouse and keyboard [ ]. Five other games were aimed at young children, and reframed the cognitive assessment as a game, to test the children under optimal mental conditions [ ].
To Increase Long-Term Engagement
Commercial gamification is often used to create long-term interest around a user experience, product, or event . Similarly, we found 11 studies (8 games) that used gamification to reduce participant dropout rates over a protracted testing or training programme. Many of these games were used in an unobserved, nonlaboratory setting; therefore, the tasks made use of motivational features that made the training and testing intrinsically more appealing and less of a burden to perform on a regular basis. A common feature was that task sessions were kept as short as possible to make them more convenient and to increase likelihood of completion [ , ]. For example, The Great Brain Experiment aimed to keep task sessions below 5 minutes in duration and found that the shortest task, the Stop Signal Task, was the most popular mini game.
To Investigate the Effects of Gamelike Tasks
Obviously, many of the studies in this review investigated the effects of gamelike tasks, however, only 5 studies explicitly stated that their aim was to assess the motivational and cognitive effects of gamelike features. Three of the 6 games were very simplistic, with only a few specific game mechanics and carefully designed nongame controls, to make the effects of the game mechanics on the data as apparent as possible. One game, Watermons , was designed, using Self Determination Theory [ , ], to maximize participant intrinsic motivation and make any motivational effects of game mechanics as apparent as possible.
To Stimulate the Brain
Six studies cited evidence that playing video games can be cognitively beneficial and/or stimulating as a key factor in their decision to gamify. The past decade has seen rapid growth in the investigation of the cognitive effects of video gaming, and findings have typically been positive. There is good evidence that video game players outperform nongamers on tests of working memory  visual attention [ , ] and processing speed [ , ]. The “active ingredient” of this cognitive enhancement effect is not yet known, but it comes as no surprise that researchers who are interested in training cognition are keen to include gamelike features in their training tasks.
To Increase Ecological Validity
Cognitive training has often suffered from a lack of transferability [- ]. Although participants may improve at the training task, these improvements do not generalize to the real world. In a similar vein, cognitive assessment has been accused of being ecologically invalid [ ]. A potential solution is to make tasks more realistic, and these tasks inevitably become gamelike as 3D graphics, sounds, and narrative are added. We identified 6 studies that used gamified tasks to test cognition in an engaging close-to-life environment or enhance transferability of learned skills. As Dunbar et al explain [ ], games are uniquely suited to some forms of cognitive training as they give the player freedom to make choices and experience feedback on the effects of those choices; in other words, they provide opportunities for experiential learning [ ].
To Increase Suitability for the Target Disorder
Gamified tasks may also be more appealing to patients with certain clinical conditions. Specifically, we found 6 studies (covering 4 games) designed for people with attention deficit/hyperactivity disorder (ADHD). It is commonly reported that ADHD patients are compulsive computer game players . Furthermore, patients with ADHD react differently from controls to rewards and feedback: they prefer strong reinforcement and immediate feedback, as well as clear goals and objectives, all of which can easily be delivered in a gamelike environment [ , , ].
What Are the Application Areas of Gamification?
We found comparable numbers of games used for cognitive testing (17) and training (13), with one game that can be used for both testing and training (see Supplementary Table 3 in). It is also worth noting that the numbers of studies investigating cognitive testing (17) and training (15) were very similar, and the slightly smaller proportion of games used to deliver cognitive training is likely explained by the relative recency of the field.
Supplementary Table 4 inshows the cognitive domains assessed. Working memory was the most commonly tested domain. This is likely due to its ease of testing and also because working memory deficit is a common symptom in many disorders. General executive function (EF), attention, and inhibition were also commonly tested domains. Many games tested several cognitive domains and/or general EF, highlighting the difficulty of examining gamification effects on specific cognitive domains (and therefore comparing performance to standard cognitive tasks). Supplementary Table 5 in shows a breakdown of training tasks by the cognitive domains they addressed. Again, working memory was a popular target, closely followed by EF and inhibitory control training. There is a smaller overlap in domains covered by cognitive training tasks; test batteries tend to test a wide range of domains, whereas training focuses on 1 or 2 domains exclusively.
Does Gamification Work?
The studies we reviewed were generally enthusiastic about their use of gamified tasks, although given the diversity of study aims, this does not mean that all games worked as expected. Where reported, subjective and objective measures of participant engagement were positive. All studies that measured intrinsic motivation reported that the use of gamelike tasks improved motivation, compared with nongamified versions. We identified 21 of 33 studies that compared a gamelike task directly against a nongamified counterpart, and these studies can shed light on the specific effects of gamification on testing and training tasks.
Most gamified assessments were validated successfully. Wii Tests , Shapebuilder [ ], The Great Brain Experiment [ , ], BAM-COG [ ], and Tap the Hedgehog [ ] were all found to produce output measures/scores that correlated fairly well with their non–gamelike counterparts, though mixed-domain measures were an issue (see Supplementary Table 1 in for full details of all games). Validation studies varied in their design, and some studies reported complex correlations between gamified and nongamified tasks with multiple outputs. However, sample correlations from some of the simpler validation studies suggest intertask correlations of 0.45-0.60 [ , , , ]. Broadly speaking, these were well-designed and well-powered studies, and together, they provide encouraging evidence that cognitive tests can be gamified and still be useful as a research tool.
Some studies found that their gamified tests were correlated with measures of multiple cognitive domains, in other words, they were mixed-domain measures. For example, use of exploratory factor analysis showed that Whack-a-Mole’s primary output measure was correlated with 2 of the 3 EFs of interest: inhibition (r (22)=.60, P<.001) and updating (r (22) = .35, P<.05) . Space Code [ , ] had similar problems. The initial study was successful, with Space Code’s output measure correlating well with a conventional measure of processing speed (r (58)=.55, P<.001). However, a second paper detailing 2 experiments which aimed to replicate the previous finding found that Space Code’s correlations with measures of working memory, visuospatial ability, and processing speed were not stable [ ]. The fact that Space Code was thought to be a fairly pure measure in one study and then was shown to be mixed-domain in the next highlights the fact that designing gamified cognitive tasks is difficult, and multiple, well-powered validation studies may be required to ensure a task is measuring what is intended.
Gamification also has the potential to invalidate a task. For example, Retirement Party was compared against the Continuous Performance Task-II in healthy controls and adults with ADHD. The Continuous Performance Task-II detected more commission errors from the ADHD adults as expected (mean [M]=56, standard deviation [SD]=13 vs M=46, SD=10), but Retirement Party did not (M=14.4, SD=5.8 vs M=13.2, SD=4.3): this likely invalidates the game as a diagnostic tool for ADHD. However, Delisle and Braun  discuss the possibility that Retirement Party may have detected no deficit in ADHD patients as the nature of the task was such that there was no deficit: the highly structured and feedback-rich multitask environment may have normalized the ADHD patients’ usual inattention. Such a performance boost resulting from gamelike elements is a disadvantage when performing cognitive assessment but is likely to be desirable in a cognitive training scenario.
Several studies in this review focused specifically on adding game mechanics to cognitive tasks to investigate the resultant changes in data, enjoyment, and motivation. Dovis et al  studied whether different types of incentive could normalize ADHD children’s performance on working memory training. They found that regardless of incentive, ADHD children did not perform as well as healthy controls. However, ADHD children also experienced a decrease in performance over time, and the €10 condition and the gaming condition (Megabot) prevented this decrease. These results indicate that performance problems in ADHD training might be somewhat alleviated through the use of gamelike tasks. This is further supported by the study by Prins et al (Supermecha [ ]), which found that ADHD children completed more training trials (M=199.48, SD=47.46 vs M=134.43, SD=34.18), with higher accuracy (69% vs 51%), when trained using a gamified working memory training task as opposed to a non–gamelike one. Children in the gamelike condition were also more engaged and enjoyed the training more, as measured by “absence time.” In a similar vein, The Great Brain Experiment [ ] and GAME [ ] both showed gamelike tasks to be appropriate for measuring and training working memory. With Ninaus et al, presenting evidence that gamification can improve overall participant performance in a working memory training task [ ] and McNab and Dolan showing that data collected from 2 very different gamified and nongamified tasks could fit similar models of working memory capacity.
In contrast, WMTrainer, was assessed across 7 different conditions containing different combinations of game mechanics . They compared training improvement across conditions and found that the greatest training effect was caused by versions with minimal motivational features. The fully gamified condition had one of the shallowest improvement slopes and none of the conditions made any difference on subjective motivation scores. However, even the minimally gamelike version still featured simplistic graphics and displayed a player score at the end of the block. It is possible that even this minimal gamification was enough to induce increased motivation and that adding “distracting” game elements such as persistent score display may have a negative impact on performance by inducing unneeded stress or new cognitive demands [ ].
One of the most theoretically driven games in our review was Watermons , which included many motivational features aimed at delivering a sense of player autonomy and competence. The task-switching training was embedded within a rich storyline and graphically enhanced theme. They found that these gamelike features increased the effect of training, reducing reaction times and switch costs, compared against a non–gamelike version of the training. Participants were also more willing to perform training when in the gamelike condition.
Miranda and Palmer  used a visual search task with 2 forms of reward for fast and accurate responses: sound effects and points. They found that sound effects led to increased reaction times, potentially due to attentional capture and did not improve scores of subjective engagement. Points appeared to have no effect on data and boosted subjective engagement scores. These results highlight the delicate nature of designing gamified cognitive tests because something as innocuous as a few sound effects had deleterious effects on participant performance.
Finally, Hawkins et al,  compared gamelike versions of 2 decision-making tasks against nongame counterparts. No difference between the data collected by the gamified versions and the nongame versions was found. Subjective ratings indicated that both versions of both tasks were equally boring and repetitive, but that the gamelike versions were more interesting and enjoyable. Given the relatively large combined sample size of these studies (N=200), they provide good evidence that game mechanics can be included in cognitive tests without invalidating the data and with the desired effect of increasing motivation.
We identified 7 reasons why researchers use gamification in cognitive research. These include not only the “traditional” applications of gamification such as increasing long- and short-term engagement with a task but also more clinically related reasons such as making tasks more interactive to enhance the effect of cognitive training. Several studies aimed to reduce test anxiety and optimize performance in groups that traditionally dislike being tested, particularly electronically, such as elderly people and children. By hiding the test behind a novel interface and gameplay, the target audience might feel more comfortable.
We saw several games aimed at training and testing people with ADHD, and overall, these games appear highly engaging to users, in some cases, even increasing the time spent training. Gamified tasks may be valuable for assessing ADHD patients as computer games are particularly appealing to them: with rapid rewards, immediate feedback, and time-pressure being exactly the type of stimulus the ADHD brain craves [, ]. The dopaminergic system is thought to be abnormal in ADHD [ , ]. However, it is thought that playing video games can facilitate the release of extrastriatal dopamine, which plays a role in focusing attention and heightening arousal [ , ]; this may improve player performance and motivation. Nevertheless, as Delisle and Braun discuss [ ], we must be cautious that liberal use of game mechanics does not reverse the very deficit we are hoping to measure.
One of the primary reasons that psychologists are keen to utilize gamification is to increase performance and motivation in research populations. The results of references [, , , ], show that gamified tasks can be used to improve motivation, while still maintaining a scientifically valid task. However, Katz et al [ ] and Miranda and Palmer [ ] highlight that this can be difficult balancing act, with several game mechanics having unforeseen deleterious effects on performance. If gamified psychological tasks are to become common in the future, further research is required to disentangle the impact of specific game mechanics on task performance, as these studies have already begun to do.
Differences Between Training and Assessment Tasks
Training games typically contained many features and were similar in appearance to commercial video games. Cognitive training normally requires several sessions to be effective, and as a result, training tasks need to be engaging enough to play for many hours. 3D graphics were quite prevalent, as was the use of avatars, points, levels, and dynamically growing game worlds (see Supplementaryin ). Long-term goals which had to be completed over repeated sessions were also common and served to sustain engagement over a long period. In contrast, assessment games were simpler, predominantly using 2D graphics, sound effects, score, and theme to create the appearance of a game. Several games simply presented themselves as “puzzles” which the participant had to complete. Tasks of this nature represent gamification at its simplest, but they were well received by users, implying that minimal gamification is better than no gamification. The simplicity of gamification employed is likely due to the constant tension between creating an engaging task and the risk of undermining the task’s scientific validity: including unknown game mechanics might have deleterious effects on the data collected.
Validating Gamified Tasks
We found heterogeneous standards for validating gamified tasks. Typically, cognitive assessment games were validated rigorously, using correlation with similar cognitive tasks and factor analysis to determine whether they were performing as expected. Many training games used a gamified task only, meaning the effect of gamification cannot be dissociated from the effect of the intervention. Sample sizes were small in nearly all of the studies we reviewed, and there was little consideration of statistical power when sample sizes were decided upon, with only 5 of 33 papers describing a power calculation. Gamified cognitive tests are novel scientific instruments and must be validated as such. Small pilot studies, followed by larger validation studies including assessment of test–retest reliability, and internal and external validity of the measures taken by the game are needed . Regarding cognitive training, ideally gamified training should be treated as an intervention and so the current gold standard of a blinded randomized control trial is appropriate [ ]. In both testing and training, we would recommend the use of at least 2 controls: a standard cognitive task designed to produce the same output measures/training effect as the newly gamified task and a nongamified version of the gamified task, built on the same software platform and identical in function/interaction, with all game mechanics removed.
One limitation of this review is the necessity of a narrow scope. Gamification in psychology and psychiatry is a rapidly growing field, hence we decided to focus specifically on “cognitive training and testing.” This has resulted in some papers being excluded on the subjective basis of not being “cognitive research.” Nevertheless, to counteract this subjectivity, papers were only included in the review on consensus from both reviewers (JL and EE), and a 20% selection check was performed by EE on papers from the title-screening stage onward. An additional consideration is that many of the studies reviewed were of a preliminary nature, and as such, the findings reported here should be considered tentative.
As discussed by Hawkins et al , it has often been suggested that gamified cognitive tasks may result in higher quality data and more effective training, simply by virtue of heightened engagement. Our review found no evidence to support that gamified tests can be used to improve data quality, either by reducing between-subject noise or by improving participant performance, and there were some indications that it may actually worsen it [ , , , ]. We did, however, find some evidence that gamification may be effective at enhancing cognitive training, but we must take these positive training findings tentatively due to numerous methodologic problems in the studies that we reviewed.
Irrespective of whether gamification can improve data or enhance training, there are still many reasons why gamelike tasks may play an important role in the future of cognitive research. Gamified cognitive tasks are more engaging than traditional tasks, thus making the participant experience less effortful and potentially reducing drop-outs in longitudinal studies. Furthermore, despite concerns that some commonly used game mechanics might reduce participant motivation  (such as by having a visibly low score [ ]), we saw no evidence that this was the case. Indeed, gamification was reported as both motivational and positive by study authors and participants alike. Gamelike tasks may also reduce feelings of test anxiety and allow alternative interfaces to cognitive tests that would otherwise be difficult to deliver in certain populations. The results of this review also show that it is possible to design gamified cognitive assessments that validate well against more traditional measures, providing that caution is taken to avoid developing mixed-domain measures or masking deficits of interest.
As cognitive research begins to move out of the laboratory and onto personal computers and mobile devices, engagement will be the key to collecting high-quality data. Gamification is likely to play an important role in enabling this change; but further research and more rigorous validation is needed to understand the delicate interplay between game mechanics and cognitive processes.
The authors are members of the United Kingdom Centre for Tobacco and Alcohol Studies, a UKCRC Public Health Research: Centre of Excellence. Funding from British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, and the National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged. The UK Medical Research Council and the Wellcome Trust (092731) and the University of Bristol provide core support for ALSPAC. This work was supported by the Medical Research Council (MC_UU_12013/6), a PhD studentship to JL funded by the Economic and Social Research Council and Cambridge Cognition Limited and a NIHR funded academic clinical fellowship to EE. The funders had no role in review design, data extraction and analysis, decision to publish, or preparation of the manuscript.
Conflicts of Interest
Dr Coyle is a Director of Handaxe CIC, a not-for-profit company that develops technology, including computer games, to support mental health interventions for children and adolescents.
Multimedia Appendix 1
Five tables detailing coding and categorization in the main manuscript. Table 1 lists the reasons for using gamification in cognitive training and testing. Table 2 categorizes games by the age group they were aimed at. Table 3 lists games by training or testing category. Tables 4 and 5 list games by the cognitive domains which they targeted.PDF File (Adobe PDF File), 102KB
Multimedia Appendix 2
Additional information on the papers covered by this review that is highly detailed and intended for further analysis in future papers or for other researchers to assess themselves. It contains short text-based descriptions of all 31 games and a screenshot where the paper included one. There is also a detailed breakdown of the game mechanics in each task.PDF File (Adobe PDF File), 1MB
- State of Online Gaming Report.: Spilgames; 2013. URL: http://auth-83051f68-ec6c-44e0-afe5-bd8902acff57.cdn.spilcloud.com/v1/archives/1384952861.25_State_of_Gaming_2013_US_FINAL.pdf [accessed 2014-11-27] [WebCite Cache]
- Cowley B, Charles D, Black M, Hickey R. Toward an understanding of flow in video games. Computers in Entertainment 2008;6(2) Article No. 20. [CrossRef]
- McGonigal J. Reality is Broken: Why games make us better and how they can change the world. New York: Vintage; 2012.
- DeRight J, Jorgensen R. I just want my research credit: frequency of suboptimal effort in a non-clinical healthy undergraduate sample. Clin Neuropsychol 2015;29(1):101-117. [CrossRef] [Medline]
- Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
- Boyle E, Hainey T, Connolly T, Gray G, Earp J, Ott M, et al. An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education 2016 Mar;94:178-192. [CrossRef]
- Boendermaker WJ, Boffo M, Wiers RW. Exploring Elements of Fun to Motivate Youth to Do Cognitive Bias Modification. Games Health J 2015 Dec;4(6):434-443. [CrossRef] [Medline]
- Dunbar N, Miller C, Adame B, Elizondo J, Wilson S, Lane B, et al. Implicit and explicit training in the mitigation of cognitive bias through the use of a serious game. Computers in Human Behavior 2014;37:307-318. [CrossRef]
- Hawkins GE, Rae B, Nesbitt KV, Brown SD. Gamelike features might not improve data. Behav Res Methods 2013 Jun;45(2):301-318. [CrossRef] [Medline]
- Dubbels B. Gamification, Serious Games, Ludic Simulation, and Other Contentious Categories. International Journal of Gaming and Computer-Mediated Simulations 2013 Apr;5(2):1-19. [CrossRef]
- Aparicio A, Vela F, Sánchez J, Montes J. Analysis and application of gamification. In: Proceedings of the 13th International Conference on Interacción Persona-Ordenador. 2012 Presented at: Interacción 2012; October 03-05, 2012; Elche, Spain. [CrossRef]
- Cugelman B. Gamification: what it is and why it matters to digital health behavior change developers. JMIR Serious Games 2013;1(1):e3 [FREE Full text] [CrossRef] [Medline]
- Deterding S, Björk S, Nacke L, Dixon D, Lawley E. Designing gamification: creating gameful and playful experiences. In: Extended Abstracts on Human Factors in Computing Systems. USA: ACM; 2013 Presented at: CHI '13; April 27-May 02, 2013; Paris, France p. 3263-3266. [CrossRef]
- Biddiss E, Irwin J. Active video games to promote physical activity in children and youth: a systematic review. Arch Pediatr Adolesc Med 2010 Jul;164(7):664-672. [CrossRef] [Medline]
- Primack B, Carroll M, McNamara M, Klem M, King B, Rich M, et al. Role of video games in improving health-related outcomes: a systematic review. Am J Prev Med 2012 Jun;42(6):630-638. [CrossRef] [Medline]
- Donchin E. Video games as research tools: The Space Fortress game. Behavior Research Methods, Instruments, & Computers 1995 Jun;27(2):217-223. [CrossRef]
- McPherson J, Burns NR. Gs invaders: assessing a computer game-like test of processing speed. Behav Res Methods 2007 Nov;39(4):876-883. [Medline]
- McPherson J, Burns NR. Assessing the validity of computer-game-like tests of processing speed and working memory. Behav Res Methods 2008 Nov;40(4):969-981. [CrossRef] [Medline]
- Trapp W, Landgrebe M, Hoesl K, Lautenbacher S, Gallhofer B, Günther W, et al. Cognitive remediation improves cognition and good cognitive performance increases time to relapse--results of a 5 year catamnestic study in schizophrenia patients. BMC Psychiatry 2013;13:184 [FREE Full text] [CrossRef] [Medline]
- Gamberini L, Martino F, Seraglia B, Spagnolli A, Fabregat M, Ibanez F, et al. Eldergames project: An innovative mixed reality table-top solution to preserve cognitive functions in elderly people. 2009 May 21 Presented at: 2nd Conference on Human System Interactions; May 21-23, 2009; Catania p. 164-169. [CrossRef]
- Gamberini L, Cardullo S, Seraglia B, Bordin A. Neuropsychological testing through a Nintendo Wii console. Stud Health Technol Inform 2010;154:29-33. [Medline]
- Dovis S, Van der Oord Saskia, Wiers R, Prins Pier J M. Can motivation normalize working memory and task persistence in children with attention-deficit/hyperactivity disorder? The effects of money and computer-gaming. J Abnorm Child Psychol 2012 Jul;40(5):669-681. [CrossRef] [Medline]
- Delisle J, Braun Claude M J. A context for normalizing impulsiveness at work for adults with attention deficit/hyperactivity disorder (combined type). Arch Clin Neuropsychol 2011 Nov;26(7):602-613. [CrossRef] [Medline]
- Prins PJ, Dovis S, Ponsioen A, ten Brink E, van der Oord S. Does computerized working memory training with game elements enhance motivation and training efficacy in children with ADHD? Cyberpsychol Behav Soc Netw 2011 Mar;14(3):115-122. [CrossRef] [Medline]
- Lim, Lee, Guan, Fung Daniel Shuen Sheng, Zhao, Teng Stephanie Sze Wei, Zhang, Krishnan K Ranga Rama. A brain-computer interface based attention training program for treating attention deficit hyperactivity disorder. PLoS One 2012;7(10) [FREE Full text] [CrossRef] [Medline]
- Heller M, Roots K, Srivastava S, Schumann J, Srivastava J, Hale T. A Machine Learning-Based Analysis of Game Data for Attention Deficit Hyperactivity Disorder Assessment. Games Health J 2013 Oct;2(5):291-298. [CrossRef] [Medline]
- Verhaegh J, Fontijn W, Aarts E, Resing W. In-game assessment and training of nonverbal cognitive skills using TagTiles. Pers Ubiquit Comput 2012 Apr 22;17(8):1637-1646. [CrossRef]
- Aalbers T, Baars Maria A E, Olde Rikkert Marcel G M, Kessels Roy P C. Puzzling with online games (BAM-COG): reliability, validity, and feasibility of an online self-monitor for cognitive performance in aging adults. J Med Internet Res 2013;15(12) [FREE Full text] [CrossRef] [Medline]
- Fagundo, Santamaría, Forcano, Giner-Bartolomé, Jiménez-Murcia, Sánchez, Granero, Ben-Moussa, Magnenat-Thalmann, Konstantas, Lam, Lucas, Nielsen, Bults, Tarrega, Menchón, de le Torre, Cardi, Treasure, Fernández-Aranda. Video game therapy for emotional regulation and impulsivity control in a series of treated cases with bulimia nervosa. European Eating Disorders Review 2013 Nov;21(6):493-499. [CrossRef]
- Anguera J, Boccanfuso J, Rintoul J, Al-Hashimi O, Faraji F, Janowich J, et al. Video game training enhances cognitive control in older adults. Nature 2013 Sep 5;501(7465):97-101 [FREE Full text] [CrossRef] [Medline]
- van der Oord S, Ponsioen A J, Geurts HM, Ten Brink EL, Prins PJ. A pilot study of the efficacy of a computerized executive functioning remediation training with game elements for children with ADHD in an outpatient setting: outcome on parent- and teacher-rated executive functioning and ADHD behavior. J Atten Disord 2014 Nov;18(8):699-712. [CrossRef] [Medline]
- Brown H, Zeidman P, Smittenaar P, Adams R, McNab F, Rutledge R, et al. Crowdsourcing for Cognitive Science - The Utility of Smartphones. PloS One 2014 Jul 15;9. [CrossRef]
- Tong T, Chignell M. Developing a Serious Game for Cognitive Assessment: Choosing Settings and Measuring Performance. In: Proceedings of the Second International Symposium of Chinese CHI. 2014 Apr 26 Presented at: Chinese CHI 2014; April 26-27, 2014; Toronto, Canada p. 70-79. [CrossRef]
- Katz B, Jaeggi S, Buschkuehl M, Stegman A, Shah P. Differential effect of motivational features on training improvements in school-based cognitive training. Front Hum Neurosci 2014;8:242 [FREE Full text] [CrossRef] [Medline]
- Lee T, Goh SJ, Quek S, Phillips R, Guan C, Cheung Y, et al. A brain-computer interface based cognitive training system for healthy elderly: a randomized control pilot study for usability and preliminary efficacy. PLoS One 2013 Nov 18;8(11):e79419 [FREE Full text] [CrossRef] [Medline]
- Miranda A, Palmer E. Intrinsic motivation and attentional capture from gamelike features in a visual search task. Behav Res Methods 2014 Mar;46(1):159-172. [CrossRef] [Medline]
- Atkins S, Sprenger A, Colflesh G, Briner T, Buchanan J, Chavis S, et al. Measuring working memory is all fun and games: a four-dimensional spatial game predicts cognitive task performance. Exp Psychol 2014;61(6):417-438. [CrossRef] [Medline]
- Dörrenbächer S, Müller P, Tröger J, Kray J. Dissociable effects of game elements on motivation and cognition in a task-switching training in middle childhood. Front Psychol 2014;5:1275 [FREE Full text] [CrossRef] [Medline]
- McNab F, Dolan R. Dissociating distractor-filtering at encoding and during maintenance. J Exp Psychol Hum Percept Perform 2014 Jun;40(3):960-967 [FREE Full text] [CrossRef] [Medline]
- Dennis TA, O'Toole L. Mental Health on the Go: Effects of a Gamified Attention Bias Modification Mobile Application in Trait Anxious Adults. Clin Psychol Sci 2014 Sep 1;2(5):576-590 [FREE Full text] [CrossRef] [Medline]
- Delgado MT, Uribe PA, Alonso AA, Díaz RR. TENI: A comprehensive battery for cognitive assessment based on games and technology. Child Neuropsychol 2016;22(3):276-291. [CrossRef] [Medline]
- de Vries M, Prins PJ, Schmand B, Geurts H. Working memory and cognitive flexibility-training for children with an autism spectrum disorder: a randomized controlled trial. J Child Psychol Psychiatry 2015 May;56(5):566-576. [CrossRef] [Medline]
- Dovis S, Van der Oord S, Wiers RW, Prins PJ. Improving executive functioning in children with ADHD: training multiple executive functions within the context of a computer game. a randomized double-blind placebo controlled trial. PLoS One 2015;10(4):e0121651 [FREE Full text] [CrossRef] [Medline]
- Kim K, Choi Y, You H, Na D, Yoh MS, Park JK, et al. Effects of a serious game training on cognitive functions in older adults. J Am Geriatr Soc 2015 Mar;63(3):603-605. [CrossRef] [Medline]
- Manera V, Petit P, Derreumaux A, Orvieto I, Romagnoli M, Lyttle G, et al. 'Kitchen and cooking,' a serious game for mild cognitive impairment and Alzheimer's disease: a pilot study. Front Aging Neurosci 2015;7:24 [FREE Full text] [CrossRef] [Medline]
- Ninaus M, Pereira G, Stefitz R, Prada R, Paiva A, Neuper C, et al. Game elements improve performance in a working memory training task. IJSG 2015 Feb 10;2(1):3-16 [FREE Full text] [CrossRef]
- Tarnanas I, Laskaris N, Tsolaki M, Muri R, Nef T, Mosimann U. On the comparison of a novel serious game and electroencephalography biomarkers for early dementia screening. Adv Exp Med Biol 2015;821:63-77. [CrossRef] [Medline]
- Eriksson B, Musialik M, Wagner J. Student essay. 2012 Aug 6. Gamification - Engaging the Future URL: https://gupea.ub.gu.se/handle/2077/30037 [accessed 2016-07-08] [WebCite Cache]
- Sirály E, Szabó Á, Szita B, Kovács V, Fodor Z, Marosi C, et al. Monitoring the early signs of cognitive decline in elderly by computer games: an MRI study. PLoS One 2015 Feb 23;10(2):e0117918 [FREE Full text] [CrossRef] [Medline]
- Ryan R, Deci E. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist 2000;55(1):68-78. [CrossRef]
- Ryan R, Rigby C, Przybylski A. The Motivational Pull of Video Games: A Self-Determination Theory Approach. Motiv Emot 2006 Nov 29;30(4):344-360. [CrossRef]
- Colzato LS, van den Wildenberg M, Zmigrod S, Hommel B. Action video gaming and cognitive control: playing first person shooter games is associated with improvement in working memory but not action inhibition. Psychol Res 2013 Mar;77(2):234-239. [CrossRef] [Medline]
- Durlach P, Kring J, Bowens L. Effects of action video game experience on change detection. Mil Psychol 2009;21(1):24-39.
- Green C, Bavelier D. Enumeration versus multiple object tracking: the case of action video game players. Cognition 2006 Aug;101(1):217-245 [FREE Full text] [CrossRef] [Medline]
- Dobrowolski P, Hanusz K, Sobczyk B, Skorko M, Wiatrow A. Cognitive enhancement in video game players: The role of video game genre. Computers in Human Behavior 2015 Mar;44:59-63. [CrossRef]
- Dye Matthew W G, Green CS, Bavelier D. Increasing Speed of Processing With Action Video Games. Curr Dir Psychol Sci 2009;18(6):321-326 [FREE Full text] [CrossRef] [Medline]
- Abikoff H. Efficacy of cognitive training interventions in hyperactive children: A critical review. Clinical Psychology Review 1985;5(5):479-512. [CrossRef]
- Jaeggi S, Buschkuehl M, Jonides J, Shah P. Short- and long-term benefits of cognitive training. PNAS 2011 Jun 13;108(25):10081-10086. [CrossRef]
- Melby-Lervåg M, Hulme C. Is working memory training effective? A meta-analytic review. Dev Psychol 2013 Feb;49(2):270-291. [CrossRef] [Medline]
- Burgess P, Alderman N, Forbes C, Costello A, Coates L, Dawson D, et al. The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. J Int Neuropsychol Soc 2006 Mar;12(2):194-209. [CrossRef] [Medline]
- Kolb D. Experiential learning: experience as the source of learning and development. Englewood Cliffs, NJ: Prentice-Hall; 1984.
- Prins PJ, Brink E, Dovis S, Ponsioen A, Geurts H, de Vries M, et al. “Braingame Brian”: Toward an Executive Function Training Program with Game Elements for Children with ADHD and Cognitive Control Problems. Games Health J 2013 Feb;2(1):44-49. [CrossRef] [Medline]
- Verhaegh J, Fontijn W, Resing W. On the correlation between children's performances on electronic board tasks and nonverbal intelligence test measures. Computers & Education 2013 Nov;69:419-430. [CrossRef]
- Kok A, Kong T, Bernard-Opitz V. A comparison of the effects of structured play and facilitated play approaches on preschoolers with autism. A case study. Autism 2002 Jun;6(2):181-196. [Medline]
- WebMD. 2012. Video Games and ADHD: What's the Link? URL: http://www.webmd.com/add-adhd/childhood-adhd/features/adhd-and-video-games-is-there-a-link?print=true [accessed 2016-04-19] [WebCite Cache]
- Dougherty D, Bonab A, Spencer T, Rauch S, Madras B, Fischman A. Dopamine transporter density in patients with attention deficit hyperactivity disorder. Lancet 1999;354(9196):2132-2133. [CrossRef] [Medline]
- LaHoste, Swanson, Wigal, Glabe, Wigal, King, Kennedy. Dopamine D4 receptor gene polymorphism is associated with attention deficit hyperactivity disorder. Mol Psychiatry 1996 May;1(2):121-124. [Medline]
- Nieoullon A. Dopamine and the regulation of cognition and attention. Prog Neurobiol 2002 May;67(1):53-83. [Medline]
- Nieoullon A, Coquerel A. Dopamine: a key regulator to adapt action, emotion, motivation and cognition. Curr Opin Neurol 2003 Dec;16 Suppl 2:S3-S9. [Medline]
- Kato P. Pam Kato's Blog. 2013 Apr 05. What do you mean when you say your serious game has been validated? Experimental vs. Test Validity URL: https://pamkato.com/2013/04/25/what-do-you-mean-when-you-say-your-serious-game-has-been-validated-experimental-vs-test-validty/ [WebCite Cache]
- Kato PM. Evaluating Efficacy and Validating Games for Health. Games Health J 2012 Feb;1(1):74-76. [CrossRef] [Medline]
- Mekler E, Brühlmann F, Opwis K, Tuch A. Do Points, Levels & Leaderboards Harm Intrinsic Motivation?: An Empirical Analysis of Common Gamification Elements. In: Proceedings of the First International Conference on Gameful Design, Research, and Applications. 2013 Presented at: Gamification 2013; October 02-04, 2013; Stratford, ON, Canada p. 66-73. [CrossRef]
- Preist C, Massung E, Coyle D. Competing or Aiming to Be Average?: Normification As a Means of Engaging Digital Volunteers. In: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 2014 Presented at: ACM conference on Computer supported cooperative work & social computing; February 15-19, 2014; Baltimore, MD p. 1222-1233. [CrossRef]
|ADHD: attention deficit/hyperactivity disorder|
|EF: executive function|
Edited by G Eysenbach; submitted 21.04.16; peer-reviewed by C Chambers, M Ninaus; comments to author 08.06.16; revised version received 16.06.16; accepted 17.06.16; published 15.07.16
©Jim Lumsden, Elizabeth A Edwards, Natalia S Lawrence, David Coyle, Marcus R Munafò. Originally published in JMIR Serious Games (http://games.jmir.org), 15.07.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.