The following databases were searched electronically: PsycInfo, Medline, ETHOS, EMBASE, PubMed, IBSS, Francis, Web of Science and Scopus. We searched the titles, abstracts, and keywords of database entries using the search strategy (gamif* OR game OR games) AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*), where * represents a wildcard to allow for alternative suffixes. Searches included papers published in English between January 2007 and October 2015. We searched the bibliographies of included papers to locate further relevant material not discovered in the database search.

We did not select papers based on whether they included “gamification.” Rather, papers were selected if they were captured by our search strategy and were considered relevant based on our inclusion/exclusion criteria. We intentionally did not strictly define gamification: as has been discussed famously by Wittgenstein and is alluded to by Deterding [13], trying to precisely define what elements make a game is both difficult and limiting. Therefore, we decided that a task was gamified if its stated purpose was to increase participant motivation. Where there was insufficient detail to determine whether a paper met our inclusion criteria, we erred on the side of caution to increase our confidence in the relevance of the studies reviewed.

After screening, data were extracted from each paper using a standardized data extraction form. Data relating to the questions of interest such as application of gamification, approach taken, and efficacy were extracted from each paper. Application of gamification refers to the field of psychology in which the gamelike task was used and why a gamelike task was used. Approach taken refers to the specific game mechanics used in the task and what themes and scaffolding were applied. Finally, efficacy refers to the findings produced by the task in practice, as well as details on the participants, methods used for evaluation, and limitations of the study. Categorization of concepts (such as the cognitive domains measured) was done using the paper-authors’ own words where possible. Where not possible, we mapped extracted concepts closely to existing categories.

All papers identified by our search strategy were screened by one reviewer (JL) in 3 stages, to determine whether they were relevant based on our inclusion/exclusion criteria: title, abstract, and full text. A second reviewer (EE) rescreened 20% of the papers from the title stage onward to ensure that no relevant papers were missed. Papers were only included in the review on the agreement of both JL and EE.

Our initial search yielded 33,445 papers (excluding duplicates). Of these, 23 papers from the original search and 4 papers from the manual reference search were included in the review. We repeated the search in October 2015, including papers from January 2015 until October 2015. This search produced 4448 papers (excluding duplicates) and resulted in another 4 papers being included in the review, with a further 2 also included following peer review. The total number of papers included in the review was therefore 33. See Figure 1 for a flowchart of the combined searches and Table 1 for details of all included studies. We used Cohen K to assess inter-rater reliability of paper inclusion at the 20% data check stage (7590 papers checked). There was moderate agreement between the 2 reviewers (k=.526, 95% CI, 0.416 to 0.633, P<.001). All supplementary data referenced in this paper can be found in Multimedia Appendix 1, whereas Multimedia Appendix 2 contains more detailed information on the games and game mechanics used by studies in the review.

We searched each paper for explanations as to why a gamelike task had been used and identified reasons for researchers opting to gamify their cognitive training and testing. These reasons were then coalesced into 7 categories. Some authors listed multiple reasons for gamifying their approach, whereas others gave no motivations at all. Supplementary Table 1 in Multimedia Appendix 1 provides details of which games fell into each category.

We found comparable numbers of games used for cognitive testing (17) and training (13), with one game that can be used for both testing and training (see Supplementary Table 3 in Multimedia Appendix 1). It is also worth noting that the numbers of studies investigating cognitive testing (17) and training (15) were very similar, and the slightly smaller proportion of games used to deliver cognitive training is likely explained by the relative recency of the field.

Supplementary Table 4 in Multimedia Appendix 1 shows the cognitive domains assessed. Working memory was the most commonly tested domain. This is likely due to its ease of testing and also because working memory deficit is a common symptom in many disorders. General executive function (EF), attention, and inhibition were also commonly tested domains. Many games tested several cognitive domains and/or general EF, highlighting the difficulty of examining gamification effects on specific cognitive domains (and therefore comparing performance to standard cognitive tasks). Supplementary Table 5 in Multimedia Appendix 1 shows a breakdown of training tasks by the cognitive domains they addressed. Again, working memory was a popular target, closely followed by EF and inhibitory control training. There is a smaller overlap in domains covered by cognitive training tasks; test batteries tend to test a wide range of domains, whereas training focuses on 1 or 2 domains exclusively.

The studies we reviewed were generally enthusiastic about their use of gamified tasks, although given the diversity of study aims, this does not mean that all games worked as expected. Where reported, subjective and objective measures of participant engagement were positive. All studies that measured intrinsic motivation reported that the use of gamelike tasks improved motivation, compared with nongamified versions. We identified 21 of 33 studies that compared a gamelike task directly against a nongamified counterpart, and these studies can shed light on the specific effects of gamification on testing and training tasks.

Most gamified assessments were validated successfully. Wii Tests [21], Shapebuilder [37], The Great Brain Experiment [32,39], BAM-COG [28], and Tap the Hedgehog [27] were all found to produce output measures/scores that correlated fairly well with their non–gamelike counterparts, though mixed-domain measures were an issue (see Supplementary Table 1 in Multimedia Appendix 2 for full details of all games). Validation studies varied in their design, and some studies reported complex correlations between gamified and nongamified tasks with multiple outputs. However, sample correlations from some of the simpler validation studies suggest intertask correlations of 0.45-0.60 [28,37,39,63]. Broadly speaking, these were well-designed and well-powered studies, and together, they provide encouraging evidence that cognitive tests can be gamified and still be useful as a research tool.

Some studies found that their gamified tests were correlated with measures of multiple cognitive domains, in other words, they were mixed-domain measures. For example, use of exploratory factor analysis showed that Whack-a-Mole’s primary output measure was correlated with 2 of the 3 EFs of interest: inhibition (r (22)=.60, P<.001) and updating (r (22) = .35, P<.05) [33]. Space Code [17,18] had similar problems. The initial study was successful, with Space Code’s output measure correlating well with a conventional measure of processing speed (r (58)=.55, P<.001). However, a second paper detailing 2 experiments which aimed to replicate the previous finding found that Space Code’s correlations with measures of working memory, visuospatial ability, and processing speed were not stable [18]. The fact that Space Code was thought to be a fairly pure measure in one study and then was shown to be mixed-domain in the next highlights the fact that designing gamified cognitive tasks is difficult, and multiple, well-powered validation studies may be required to ensure a task is measuring what is intended.

Gamification also has the potential to invalidate a task. For example, Retirement Party was compared against the Continuous Performance Task-II in healthy controls and adults with ADHD. The Continuous Performance Task-II detected more commission errors from the ADHD adults as expected (mean [M]=56, standard deviation [SD]=13 vs M=46, SD=10), but Retirement Party did not (M=14.4, SD=5.8 vs M=13.2, SD=4.3): this likely invalidates the game as a diagnostic tool for ADHD. However, Delisle and Braun [23] discuss the possibility that Retirement Party may have detected no deficit in ADHD patients as the nature of the task was such that there was no deficit: the highly structured and feedback-rich multitask environment may have normalized the ADHD patients’ usual inattention. Such a performance boost resulting from gamelike elements is a disadvantage when performing cognitive assessment but is likely to be desirable in a cognitive training scenario.

Several studies in this review focused specifically on adding game mechanics to cognitive tasks to investigate the resultant changes in data, enjoyment, and motivation. Dovis et al [22] studied whether different types of incentive could normalize ADHD children’s performance on working memory training. They found that regardless of incentive, ADHD children did not perform as well as healthy controls. However, ADHD children also experienced a decrease in performance over time, and the €10 condition and the gaming condition (Megabot) prevented this decrease. These results indicate that performance problems in ADHD training might be somewhat alleviated through the use of gamelike tasks. This is further supported by the study by Prins et al (Supermecha [24]), which found that ADHD children completed more training trials (M=199.48, SD=47.46 vs M=134.43, SD=34.18), with higher accuracy (69% vs 51%), when trained using a gamified working memory training task as opposed to a non–gamelike one. Children in the gamelike condition were also more engaged and enjoyed the training more, as measured by “absence time.” In a similar vein, The Great Brain Experiment [39] and GAME [46] both showed gamelike tasks to be appropriate for measuring and training working memory. With Ninaus et al, presenting evidence that gamification can improve overall participant performance in a working memory training task [48] and McNab and Dolan showing that data collected from 2 very different gamified and nongamified tasks could fit similar models of working memory capacity.

In contrast, WMTrainer, was assessed across 7 different conditions containing different combinations of game mechanics [34]. They compared training improvement across conditions and found that the greatest training effect was caused by versions with minimal motivational features. The fully gamified condition had one of the shallowest improvement slopes and none of the conditions made any difference on subjective motivation scores. However, even the minimally gamelike version still featured simplistic graphics and displayed a player score at the end of the block. It is possible that even this minimal gamification was enough to induce increased motivation and that adding “distracting” game elements such as persistent score display may have a negative impact on performance by inducing unneeded stress or new cognitive demands [34].

One of the most theoretically driven games in our review was Watermons [38], which included many motivational features aimed at delivering a sense of player autonomy and competence. The task-switching training was embedded within a rich storyline and graphically enhanced theme. They found that these gamelike features increased the effect of training, reducing reaction times and switch costs, compared against a non–gamelike version of the training. Participants were also more willing to perform training when in the gamelike condition.

Miranda and Palmer [36] used a visual search task with 2 forms of reward for fast and accurate responses: sound effects and points. They found that sound effects led to increased reaction times, potentially due to attentional capture and did not improve scores of subjective engagement. Points appeared to have no effect on data and boosted subjective engagement scores. These results highlight the delicate nature of designing gamified cognitive tests because something as innocuous as a few sound effects had deleterious effects on participant performance.

Finally, Hawkins et al, [9] compared gamelike versions of 2 decision-making tasks against nongame counterparts. No difference between the data collected by the gamified versions and the nongame versions was found. Subjective ratings indicated that both versions of both tasks were equally boring and repetitive, but that the gamelike versions were more interesting and enjoyable. Given the relatively large combined sample size of these studies (N=200), they provide good evidence that game mechanics can be included in cognitive tests without invalidating the data and with the desired effect of increasing motivation.

We identified 7 reasons why researchers use gamification in cognitive research. These include not only the “traditional” applications of gamification such as increasing long- and short-term engagement with a task but also more clinically related reasons such as making tasks more interactive to enhance the effect of cognitive training. Several studies aimed to reduce test anxiety and optimize performance in groups that traditionally dislike being tested, particularly electronically, such as elderly people and children. By hiding the test behind a novel interface and gameplay, the target audience might feel more comfortable.

We saw several games aimed at training and testing people with ADHD, and overall, these games appear highly engaging to users, in some cases, even increasing the time spent training. Gamified tasks may be valuable for assessing ADHD patients as computer games are particularly appealing to them: with rapid rewards, immediate feedback, and time-pressure being exactly the type of stimulus the ADHD brain craves [64,65]. The dopaminergic system is thought to be abnormal in ADHD [66,67]. However, it is thought that playing video games can facilitate the release of extrastriatal dopamine, which plays a role in focusing attention and heightening arousal [68,69]; this may improve player performance and motivation. Nevertheless, as Delisle and Braun discuss [23], we must be cautious that liberal use of game mechanics does not reverse the very deficit we are hoping to measure.

One of the primary reasons that psychologists are keen to utilize gamification is to increase performance and motivation in research populations. The results of references [9,17,18,38], show that gamified tasks can be used to improve motivation, while still maintaining a scientifically valid task. However, Katz et al [34] and Miranda and Palmer [36] highlight that this can be difficult balancing act, with several game mechanics having unforeseen deleterious effects on performance. If gamified psychological tasks are to become common in the future, further research is required to disentangle the impact of specific game mechanics on task performance, as these studies have already begun to do.

Training games typically contained many features and were similar in appearance to commercial video games. Cognitive training normally requires several sessions to be effective, and as a result, training tasks need to be engaging enough to play for many hours. 3D graphics were quite prevalent, as was the use of avatars, points, levels, and dynamically growing game worlds (see Supplementary Figure 1 in Multimedia Appendix 2). Long-term goals which had to be completed over repeated sessions were also common and served to sustain engagement over a long period. In contrast, assessment games were simpler, predominantly using 2D graphics, sound effects, score, and theme to create the appearance of a game. Several games simply presented themselves as “puzzles” which the participant had to complete. Tasks of this nature represent gamification at its simplest, but they were well received by users, implying that minimal gamification is better than no gamification. The simplicity of gamification employed is likely due to the constant tension between creating an engaging task and the risk of undermining the task’s scientific validity: including unknown game mechanics might have deleterious effects on the data collected.

We found heterogeneous standards for validating gamified tasks. Typically, cognitive assessment games were validated rigorously, using correlation with similar cognitive tasks and factor analysis to determine whether they were performing as expected. Many training games used a gamified task only, meaning the effect of gamification cannot be dissociated from the effect of the intervention. Sample sizes were small in nearly all of the studies we reviewed, and there was little consideration of statistical power when sample sizes were decided upon, with only 5 of 33 papers describing a power calculation. Gamified cognitive tests are novel scientific instruments and must be validated as such. Small pilot studies, followed by larger validation studies including assessment of test–retest reliability, and internal and external validity of the measures taken by the game are needed [70]. Regarding cognitive training, ideally gamified training should be treated as an intervention and so the current gold standard of a blinded randomized control trial is appropriate [71]. In both testing and training, we would recommend the use of at least 2 controls: a standard cognitive task designed to produce the same output measures/training effect as the newly gamified task and a nongamified version of the gamified task, built on the same software platform and identical in function/interaction, with all game mechanics removed.

One limitation of this review is the necessity of a narrow scope. Gamification in psychology and psychiatry is a rapidly growing field, hence we decided to focus specifically on “cognitive training and testing.” This has resulted in some papers being excluded on the subjective basis of not being “cognitive research.” Nevertheless, to counteract this subjectivity, papers were only included in the review on consensus from both reviewers (JL and EE), and a 20% selection check was performed by EE on papers from the title-screening stage onward. An additional consideration is that many of the studies reviewed were of a preliminary nature, and as such, the findings reported here should be considered tentative.

As discussed by Hawkins et al [9], it has often been suggested that gamified cognitive tasks may result in higher quality data and more effective training, simply by virtue of heightened engagement. Our review found no evidence to support that gamified tests can be used to improve data quality, either by reducing between-subject noise or by improving participant performance, and there were some indications that it may actually worsen it [17,18,34,36]. We did, however, find some evidence that gamification may be effective at enhancing cognitive training, but we must take these positive training findings tentatively due to numerous methodologic problems in the studies that we reviewed.

Irrespective of whether gamification can improve data or enhance training, there are still many reasons why gamelike tasks may play an important role in the future of cognitive research. Gamified cognitive tasks are more engaging than traditional tasks, thus making the participant experience less effortful and potentially reducing drop-outs in longitudinal studies. Furthermore, despite concerns that some commonly used game mechanics might reduce participant motivation [72] (such as by having a visibly low score [73]), we saw no evidence that this was the case. Indeed, gamification was reported as both motivational and positive by study authors and participants alike. Gamelike tasks may also reduce feelings of test anxiety and allow alternative interfaces to cognitive tests that would otherwise be difficult to deliver in certain populations. The results of this review also show that it is possible to design gamified cognitive assessments that validate well against more traditional measures, providing that caution is taken to avoid developing mixed-domain measures or masking deficits of interest.

As cognitive research begins to move out of the laboratory and onto personal computers and mobile devices, engagement will be the key to collecting high-quality data. Gamification is likely to play an important role in enabling this change; but further research and more rigorous validation is needed to understand the delicate interplay between game mechanics and cognitive processes.