A Serious Game for Clinical Assessment of Cognitive Status: Validation Study

Introduction

The rapidly aging population and high prevalence of age-related conditions, such as delirium and dementia, are placing increasing burdens on health care systems (eg, [1]). More frequent and accessible methods for cognitive screening are needed to detect early signs of impairment and to prevent or better manage further decline. We envision future development of independent patient-administered methods of cognitive screening that can be completed within a hospital or home. Demonstrating that serious games are highly correlated with other methods of cognitive assessment is necessary but not sufficient to justify their use. In order to ensure adequate motivation and realistic assessment of ability, game-based cognitive assessments should be interactive and engaging. They should also be enjoyable so that patients are willing to complete the assessment task at regular intervals.

Background

In geriatric health care, there are standard mental status tools that screen for cognitive impairment, such as the Mini-Mental State Examination (MMSE) [2], Montreal Cognitive Assessment (MoCA) [3], and Confusion Assessment Method (CAM) [4].

Current cognitive screening methods are only minimally interactive, creating little in the way of engagement or entertainment. They are typically initiated by a health care professional rather than sought out by individuals and they are generally not designed for self-administration or for use by nonclinicians. Some tools such as the CAM require subjective assessments, which may result in administrator bias [4]. Additionally, it may not be feasible for the test administrator to repeatedly assess individuals for changes in their cognitive status over time. The resulting lack of frequent assessment may result in underdiagnosis of a condition such as delirium, where cognitive status can fluctuate widely over the course of a day, making it difficult to detect early stages of delirium and initiate preventive interventions [4].

Software suites, such as CogTest [5] and the Cambridge Neuropsychological Test Automated Battery [6], offer computerized versions of traditional cognitive tests. In addition to validation issues when moving a test to the computer medium, there is also the problem of potential lack of motivation when performing somewhat uninteresting tasks on a computer. To deal with the lack of motivation and engagement, games have been promoted as a way to stimulate cognitive activity in elderly users [7] and to improve brain fitness or to preserve cognitive status. For example, the Games to Train and Assess Impaired Persons game suite is composed of eight different games to evaluate motor and cognitive abilities in individuals with impairments [8]. However, such games do not yet provide validated cognitive assessment, have not been used in the health care setting, and evidence about whether they improve broader measures of intelligence is mixed (eg, [9]).

Manera et al [10] performed a pilot study with a serious game involving patients with mild cognitive impairment (MCI) and Alzheimer disease. They were able to demonstrate that their game correlates with the MMSE and other assessments such as the Trail Making Test Part 2 and Victoria Stroop Test. Because this research [10] was carried out on patients with MCI and dementia, and involved a relatively small pilot sample of 21 people using a kitchen and cooking game, there remains a need for a validated game-like screening tool that can be completed rapidly and independently (or with minimal assistance) by a broad range of older adults with varying cognitive ability.

Serious games are games designed with a primary purpose other than entertainment, such as education and training [11]. Specially adapted games can be leveraged to create an interactive and engaging tool that promotes patient-centered cognitive assessment. Mobile phones and tablets are commonly used devices and can be used as platforms for serious gaming. Previous work has demonstrated that elderly users can use mobile phones [12, 13] and touch-based tablets [14]. Many of these technologies also provide the ability to modify contrast/brightness and text size/font to increase readability. Gaming on mobile platforms is already a growing trend that is enjoyed across a wide range of age groups. Thus, the design of a game-based assessment on a mobile platform would likely increase the accessibility of cognitive assessment.

Although there are many potential benefits of designing games for the elderly, there are possible shortcomings to consider. For instance, some elderly users may not be interested in playing games or may be uncomfortable using technology [8]. A brief comparison between paper-and-pencil–based methods and serious games for cognitive assessment is provided in Table 1.

Serious games have been used in health care for the purpose of brain training in projects such as the ElderGames [7], Smart Aging [15], and the work reported by Anguera et al [16]. The ElderGames project uses a large touchscreen tabletop surface as a gaming platform. The goal of this work is to promote social interactions through gameplay with other elderly adults. A limitation associated with this work is that it requires a large apparatus and is not mobile. Moreover, the Smart Aging platform uses a computer and touchscreen monitor to simulate a virtual loft apartment. It is designed to identify MCI through the completion of a series of tasks that simulate daily activities [15]. This project was reported to be in the pilot phase and was evaluated with a relatively small sample of healthy individuals (N=50). A computer-based serious game has been created [16] that simulates driving a vehicle. However, that research compared serious game performance in elderly users with their performance on psychological tasks rather than with standard cognitive assessments. In contrast, we are explicitly developing a game for cognitive assessment.

Development of a Serious Game

We developed a serious game to assess cognitive status in elderly adults with a focus on detecting small changes in cognition for conditions such as delirium. Our serious game mimics features of the classic psychological Go/No-Go Discrimination Task [17], a measure of inhibition ability. As implemented, our game is similar to the carnival game whack-a-mole (see Figure 1). In a previous study with healthy younger adults, we found that our serious game had a significant relationship (r =.60, P <.001) with the Stroop task [14]. The Stroop task is a test of the inhibitory executive function, which declines with age, and the task has been shown to correlate with white matter loss in the brain [18, 19].

After demonstrating that the game-based screening tool was usable by young and older healthy adult samples, and was predictive of inhibition ability, our next step was to evaluate its usability in a clinical sample. In this paper, we present our findings concerning the process of integrating a game-based cognitive assessment into a clinical environment. We demonstrate that our serious game is usable by an elderly population from an emergency department (ED) and is predictive of scores on standard cognitive assessments. The ED is a promising target for serious game-based cognitive assessment because there is a high prevalence of cognitive impairment in that setting compounded by a high rate of underdetection of delirium [20]. Based on the findings from this research, a set of design guidelines is provided in a later section of this paper to assist future researchers in implementing other serious games for assessing cognitive ability.

‎View this figure

Methods

We conducted a prospective observational clinical study with participants recruited from the Sunnybrook Health Sciences Centre ED (see Figure 2) located in Toronto, Ontario, Canada under a research protocol approved by both the Research Ethics Boards of the Sunnybrook Health Sciences Centre and the University of Toronto. Participants who were 70 years or older and who were present in the ED for a minimum of 4 hours were recruited for the study. Exclusion criteria included patients who were (1) critically ill (defined by the Canadian Triage Acuity Scale score of 1), (2) in acute pain (measured using the Numeric Rating Scale with a score greater than or equal to two out of 10), (3) receiving psychoactive medications, (4) judged to have a psychiatric primary presenting complaint, (5) previously enrolled, (6) blind, or (7) unable to speak English, follow commands, or communicate verbally.

Clinical research assistants (RAs) administered standard cognitive assessments including the MMSE, CAM, Delirium Index (DI) [21], Richmond Agitation-Sedation Scale (RASS) [22], Digit Vigilance Test (DVT) [23], and a choice reaction time (CRT) task. Each participant was then asked to play the serious game and provide feedback. The serious game was played on a 10-inch Samsung Galaxy Tab 4 10.1 tablet. Participants received instructions on how to play the game and interact with the tablet. There was no limit on the number of attempts to play the game. Participants were invited to provide open feedback at the end of the study. At the end of each session, the RA informally interviewed the participant on his/her experience with the game. In addition, RAs provided their own feedback and comments on their experience with the game and their observations of the interaction between each participant and the game.

The RAs recorded the date of the ED visit, whether the cognitive assessments were refused, and the cognitive assessment scores. Usage notes were also recorded and later used to infer usability problems as well as evidence for enjoyment and engagement.

Statistical Analysis

The cognitive data and serious game results were nonnormally distributed based on visual inspection of the data. Tests for normality, including the Kolmogorov-Smirnov and Shapiro-Wilk tests [24], were not used due to the large sample size in this study because they are known to result in oversensitivity to relatively small departures from normality [24]. Transformations of the data were not performed because some of the measures, such as the CAM and DI, are binary/categorical and cannot follow a normal distribution. Our interest was in correlations as a measure of the effect size of the underlying relationship between game performance and the cognitive assessments, but we used nonparametric correlation measures for some of the comparisons [25] that involved categorical or narrow ordinal scales. Correlations between the dichotomous CAM and the other measures were assessed using point-biserial correlations [24]. Correlations involving the DI and RASS (and not involving the CAM) were assessed using Spearman rho because the DI and RASS use a small number of ordered categories. The remaining comparisons were done using Pearson correlations. In order for readers to judge strengths of relationships involving game performance, scatterplots of the relationship between game performance and the MMSE, MoCA, and CAM, respectively, are also presented.

‎View this figure

Results

Description of Sample

We recruited 147 participants (80 males and 67 females) between the ages of 70 and 94 years (mean 80.61, SD 6.08). One participant was excluded for not completing any of the cognitive assessments and five people did not play the serious game (of whom two were CAM-positive), leaving 141 participants who completed the study (age range 70-94, mean age 80.64, SD 6.09; 79 males and 67 females).

Some participants declined to complete some of the cognitive assessments entirely or declined to answer certain questions. The completion rate of each test is shown in Table 2. All participants completed the CAM, DI, and RASS. The serious game had a combined completion rate of 96.6% (141/146), whereas the completion rates for the other assessments were lower with DVT being the worst at 36.3% (37/102) overall. Because the DVT and CRT assessments were initiated partway through the study, the denominators in calculating completion rates for those measures (102 and 99, respectively) were lower than for the other tests (which were initiated at the start of the study).

There were a number of people in the sample with low MMSE and MoCA scores (down to 9 and 8, respectively). There were 129 participants who were negative for the CAM and 12 participants who were positive (a positive result on the CAM suggests that the participant has delirium). Moreover, the DI scores ranged from 0 to 10 (the score indicates the severity of delirium), RASS scores ranged from –2 to 1 (a score >0 suggests that the patient is agitated and a score <0 suggests that the patient is sedated), DVT scores ranged from 81 to 103, and CRT choice accuracy ranged from 34% to 95%. The combined median response time (RT) on the CRT was 1.2 sec (IQR 0.4). The overall median RT on the serious game was 0.9 sec (IQR 0.2), and the mean accuracy was a deviation of 328.5 pixels (SD 59.7) from the center of the target. A summary of the scores on the cognitive assessments can be found in Table 3.

Comparison Between Serious Game Performance and Standard Cognitive Assessments

Game performance was measured based on a participant’s RT and accuracy. In our serious game, RT was measured from the time the target appeared to the time of the user’s response and accuracy was measured as the pixel distance between the center of the target and the center of the user’s touch.

Correlation analysis revealed significant relationships between game median RT and scores on the six cognitive assessments: MMSE, MoCA, CAM, DI, RASS, DVT, and CRT RT (see Table 4). In contrast to the RT results, the corresponding relationships between game accuracy and the standard cognitive assessments were not statistically significant, except for the relationship with DVT. Note that information about which types of correlation were used for each comparison are shown in the footnotes to Table 4.

As a follow-up to our correlation analyses in Table 4, we carried out the same analysis using Spearman rho correlations instead of Pearson correlations. All significant correlations between the cognitive assessments and game RT and game accuracy, respectively, were also observed to be significant using Spearman rho.

In order to examine the separate contributions of speed of processing and executive functioning on cognitive assessment scores, we looked at the partial correlations of serious game and CRT performance (controlling for each other) with the clinical assessments (see Table 5). The partial correlations with game RT (controlling for CRT) remained significant for the MMSE, CAM, and DI, but not for the MoCA and DVT. There was one significant relationship for the partial correlation of game accuracy (controlling for CRT) with DVT. On the other hand, the partial correlations involving CRT, but controlling for serious game performance RT, were not significant except for the MMSE (see Table 5). In addition, the partial correlations involving CRT but controlling for game accuracy were significant for the DI only (Table 5).

Detection of Abnormal State Using Serious Game Performance

A Mann-Whitney U test (see Table 6) was performed to investigate the difference between cognitive ability and serious game performance when the MMSE score was 24 and above (normal cognitive function or possible MCI) versus when that score was below 24 (signs of dementia) [2, 26]. The MMSE was chosen as the grouping criterion because it was a standard in screening for dementia at the time this research was carried out. The test results suggest that there was a significant difference on the CRT in terms of RT between participants with dementia (MMSE <24) and no dementia (MMSE ≥24) [26]. In addition, there was a significant difference between MMSE groups in terms of game RT (U =348.5, z =–4.7; P <.001), but not for game accuracy. For Table 6, the corresponding scatterplot (Figure 3) is also shown. Figure 3 shows the distribution of game RT versus MMSE (“dementia” scores are indicated by triangles) where a tendency for lower MMSE scores to be associated with longer RTs can be seen.

Similar to the analysis reported in Table 6, a Mann-Whitney U test (see Table 7) was performed to investigate the difference between cognitive ability and serious game performance when the MoCA score was 23 and above (normal cognitive function) versus below 23 (MCI) [27]. The MoCA was chosen as the criterion in this comparison because it is a de facto standard in screening for MCI versus normality. There was a significant difference (U =947.5, z =–2.7; P =.001) on the CRT RT between participants with cognitive impairment (MoCA <23) and no impairment (MoCA ≥23). There was also a significant difference between MoCA groups for game RT (U =370.0, z =–3.2; P =.03). For Table 7, the bivariate relationship is illustrated in the scatterplot in Figure 4. This figure illustrates a tendency for lower MoCA scores to be associated with longer RTs, although that relationship appeared to be weaker for the MoCA than it was for the MMSE.

Another Mann-Whitney U test (see Table 8) was performed to investigate the difference between cognitive ability and serious game performance when delirium was present (CAM positive) versus absent (CAM negative). The CAM was chosen as the grouping factor as it is the gold standard in screening for delirium. The test indicated a significant difference on the MMSE, MoCA, RASS, and DI between participants with delirium (CAM positive) and no delirium (CAM negative). In addition, there was a significant difference between CAM groups in terms of RT on the serious game (U =–4.5, P <.001). For Table 8, this relationship is shown in Figure 5. These between-group differences in game RT and MMSE are consistent with findings by Lowery [28], where CAM-negative participants demonstrated faster RT and higher MMSE scores compared to CAM-positive participants.

As a check, we replicated all the Mann-Whitney U tests in Tables 6-8 with their parametric equivalent, in this case the independent samples t-test. The pattern of significant and nonsignificant effects was identical for both tests, with the exception of the comparison of CRT RT between CAM-positive and CAM-negative participants (Table 8). For that comparison, the independent samples t-tests did not show a significant effect, whereas the Mann-Whitney U test did.

‎View this figure

‎View this figure

‎View this figure

Predicting Delirium Status Using Serious Game Performance

In the preceding section, we examined the relationship between game performance and current standards for clinical assessment with respect to MCI, delirium, and dementia. In this section, we examine the question of how well the serious game performance predicted CAM status (delirium).

Discriminant analysis was carried out to see how well game performance could predict CAM status. The two predictors were game RT and accuracy. Game accuracy provided no benefit in prediction and received a zero weight in the discriminant function. Thus, we focused on game RT as a potential screener for further evaluation using the CAM. We examined different possible cutoff values for distinguishing between people who should be screened for possible delirium (using the CAM) and those who should not.

Setting a relatively long median RT for the decision threshold (≥1.88 seconds) resulted in good specificity (127/129, 98.4% CAM-negative patients were correctly identified), but relatively poor sensitivity (only 5/12, 41% CAM-positive patients were correctly identified).

On the other hand, using a more stringent median RT cutoff of 1.13 seconds, there was both good sensitivity (10/12, 83% CAM-positive patients were correctly identified) and good specificity (114/129, 88.3% CAM-negative patients were correctly identified).

We also found that CAM-positive patients hit fewer distractors by mistake (as shown in Figure 6). Since CAM-positive participants had fewer hits in general (to both moles and butterflies), it seems likely that their apparently lower error rate was due to a lower response rate rather than to the presence of a speed-accuracy tradeoff.

‎View this figure

Usability Issues and Evidence of Enjoyment and Engagement

The following brief notes recorded by the RAs during patient use of the serious game are indicative examples of enjoyment and engagement that were observed: “Loved the game, she was playing games on her iPhone before I approached her” “Enjoyed the game, he would play on his own,” “Too easy but don’t make it too challenging, like the game,” and “Really loved the tablet, wanted to keep playing even after testing was over.” However, usability problems were also observed. Some participants placed their palm on the tablet while trying to interact with the serious game. This confused the software because it was unclear which hit points were intentional versus accidental. Some participants claimed that the game was too easy and suggested that we include more difficult levels to make it more interesting. Elderly users also expressed an interest in playing games such as crossword puzzles. Anecdotally, the RAs who supervised the data collection at the hospital reported that this game was easier to administer and more fun to complete compared to standard cognitive assessments such as the MoCA and DVT.

Ergonomic Issues

While interacting with the tablets, the elderly participants assumed numerous positions, such as being seated, lying down, standing, or walking around. Each of these positions had different ergonomic requirements and some brief recommendations based on our experience in this study are provided in the Discussion. Some participants were also frail and required the assistance of the RA to hold the tablet for them.

Discussion

Performance on the serious game in terms of median RT was significantly correlated with MMSE, MoCA, CAM, DI, RASS, DVT, and CRT scores for elderly ED patients and differences were in the expected direction (slower game RT for people with possible MCI and dementia). The correlations suggest a relationship between longer RT on the game and lower cognitive assessment scores. These correlations demonstrate the potential value of serious games in clinical assessment of cognitive status. The correlations between the standard cognitive tests observed in this study are similar to results seen in other research. For example, correlations of r =.43 and r =.60 between MMSE and MoCA scores for healthy controls and patients with MCI, respectively, have been found [29]. In our study, we observed a correlation of r =.63 (P <.001) between the MMSE and MoCA scores. Overall, the correlation of our serious game with existing methods of clinical cognitive assessment appears to be almost as strong as the correlations of the clinical assessment methods with themselves.

In our partial correlation analysis, we observed that our serious game correlates with the MMSE and DI, but that part of that correlation is attributable to speed of processing (CRT speed). Thus, serious game performance in this case involved both speed of processing and executive functioning components. Both components are involved in the correlation of the serious game with the MMSE. However, only the speed of processing component appears to be involved in the correlation with the MoCA. Crucially, the partial correlations of serious game performance (controlling for CRT RT) were higher than the corresponding partial correlations for CRT (controlling for serious game performance) indicating that the serious game is an overall better predictor of cognitive status than simple processing speed as measured by the CRT task.

We found that there was a lack of association between serious game accuracy and scores on cognitive assessments. This may be due to variations in interaction methods where some users used their fingers instead of a stylus to interact with the tablet device. Another reason may be that some users preferred responding more quickly over being accurate in their responses.

One of the goals of this research was to develop a method for predicting the presence of delirium using this serious game. In this study, we found that a median RT cutoff of 1.13 seconds implied relatively good sensitivity and specificity in the clinical decision. However, 25 of the 129 (19.4%) participants were above the median cutoff and only 10 of these were CAM-positive. Thus, in a clinical setting the question remains of how to deal with people who are identified as CAM-positive using this RT cutoff value. One approach would be to give those people full CAM assessment and then treat the CAM-positive patients accordingly. The value of the serious game in this case is that it would allow (based on screening with the serious game) a high rate of delirium detection using CAM assessment in only around 20% of patients (assuming that the current results generalize to other contexts). Ideally, a suitably adapted serious game would also detect risk of delirium onset so that prevention strategies could be used on targeted patients before they developed delirium, but that prospect was beyond the scope of the research reported in this paper.

During our studies, we observed many ergonomic issues that could arise during the administration of the serious game. For instance, there were a variety of positions and methods used to interact with the tablet-based serious game. For participants who are sitting down, we recommend a tablet case that has a hand holder or kickstand to allow them to interact with the tablet in multiple ways. In contrast, for participants lying down on a bed, it may be difficult for them to hold the tablet to play the serious game; thus, a stand affixed to a table or intravenous pole that holds up the tablet would be appropriate. Furthermore, the ergonomic solutions that are adopted should meet hospital standards on hygiene and sanitization for technology. For patients with hand injuries or visual disabilities, the serious game may not be a usable option.

User-centered design and ergonomic interventions were both key in ensuring that the serious game was usable with a challenging user group (elderly patients) and in the fairly unique and demanding context of a hospital ED. The touch interface was modified so that it was more forgiving of the kinds of gestures made by elderly users when interacting with the game and the gameplay was modified so that users with a wide range of ability could play the game. Ergonomic issues that were dealt with in our research included the form factor of the device and the selection and use of accessories to facilitate interactions with the device in different postures and contexts.

Based on our research experience, we present the following recommendations for enhancing tablet-based user interaction between elderly adults and touch-based technologies:

1. Accept multiple gestures, including taps and swipes, as input to maximize interaction.

2. Provide a stylus for users who have difficulties interacting with the tablet with their fingers.

3. For time-sensitive tasks, the time limit should be increased to allow older or more frail users a chance to interact with the software.

4. Tablet screen protectors should be installed to provide more friction between a user’s hand and the screen.

5. A variety of ergonomic stands and mounts should be available to accommodate various interaction positions.

6. Serious games for cognitive assessment should incorporate validated psychological task components (eg, executive functions) and should be easily playable for independent use.

7. Assess the validity of the game across different subgroups of patients. Consider the possibility of using multiple versions of a game, or multiple games, to accommodate the different characteristics and needs of different types of patient.

Limitations

The usability and validation results obtained apply to elderly adults in an emergency setting. Further research would be needed to generalize these results to different types of patient and different clinical settings. The design of this study was cross-sectional, so each participant/patient was only studied during one ED visit and played the game only once. Future research may assess the reliability of the game when played repeatedly by the same patient in the ED. One other limitation is that only one game was examined in this research (the whack-a-mole game that we developed). Other serious games should also be explored to determine which games work best with different types of patients.

This work is an initial validation study of our serious game for cognitive screening, where the game was only administered once. One of the goals of this research is frequent cognitive screening, which can potentially lead to learning effects on the game. Future research that assesses the reliability of the game-based screening tool will need to address how to overcome and differentiate between learning effects on a patient’s game performance on our serious game versus their actual cognitive status. Because we are interested in changes in cognitive status, we are not as concerned with a patient’s improved performance due to learning effects from repeated gameplay, but would aim to track deviations in their performance over time due to cognitive decline.

Conclusions

We believe that serious games are a promising methodology for cognitive screening in clinical settings, including the high-acuity time-pressured ED environment. This work demonstrates the feasibility of implementing a serious game for cognitive screening in a health care environment. To the best of our knowledge, this is the first time that a serious game for cognitive assessment has been tested in an ED and with a full battery of standard cognitive assessment methods for comparison. Based on these results, ergonomically appropriate serious games can potentially revolutionize cognitive assessment of the elderly in clinical settings, allowing assessments to be more frequent, more affordable, and more enjoyable.

This research provides a case study in the development of an interactive serious game for cognitive screening that may be used independently and repeatedly, thus promoting patient-centered health and safety. We have demonstrated in this study that elderly adults older than age 70 years can successfully play our serious game in an ED and that RT performance on the game can be used as an initial screen for cognitive status.

These findings do not yet demonstrate that the serious game evaluated here is ready to be used to screen for delirium in the ED. Only 12 CAM-positive patients were observed in the study and of the game performance measures (RT, accuracy, number of targets hit, number of distractors hit), only game RT was predictive of CAM status. However, due to the known underreporting of delirium in the ED, an efficient and usable method of screening for delirium is clearly needed. In this study, a game median RT cutoff of 1.13 seconds produced a sensitivity of 83% and a specificity of 88% when used retrospectively as a screen for CAM-positive status. Although further research is needed, it seems possible that a suitably revised and validated game might be able to identify approximately 80% to 90% of CAM-positive cases while requiring the screening of no more than approximately 20% of cases.

Outside the ED, the use of the serious game for ongoing patient-administered assessment would ideally involve patients who remain actively engaged with their support network (eg, family and care providers) and with health care professionals. For instance, if patients perform poorly on the serious game or notice a decline in their performance, they could discuss these results with their care providers, which might lead to interventions such as changes to medication or lifestyle that could slow observed declines.