This paper is in the following e-collection/theme issue:

Games for Cognitive Assessment (61) Dementia and Cognitive Decline (481) Diagnostic Tools in Mental Health (242) Cognitive Training for the Elderly (95) mHealth for Screening (213) Methods and New Tools in Mental Health Research (454) Transdiagnostic Mental Health Interventions (160) Extended Reality, Virtual Reality and Virtual Worlds (560) Virtual Reality Interventions in Mental Health (122) XR for Assessment and Diagnosis (9)

Published on in Vol 9, No 4 (2021): Oct-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/30919, first published .
A Virtual Supermarket Program for the Screening of Mild Cognitive Impairment in Older Adults: Diagnostic Accuracy Study

A Virtual Supermarket Program for the Screening of Mild Cognitive Impairment in Older Adults: Diagnostic Accuracy Study

A Virtual Supermarket Program for the Screening of Mild Cognitive Impairment in Older Adults: Diagnostic Accuracy Study

Authors of this article:

Mingli Yan1 Author Orcid Image ;   Huiru Yin1 Author Orcid Image ;   Qiuyan Meng1 Author Orcid Image ;   Shuo Wang1 Author Orcid Image ;   Yiwen Ding1 Author Orcid Image ;   Guichen Li1 Author Orcid Image ;   Chunyan Wang2 Author Orcid Image ;   Li Chen1 Author Orcid Image
Article Authors Cited by (17) Tweetations (3) Metrics

    Original Paper

    1School of Nursing, Jilin University, Changchun, China

    2Senior Officials Inpatient Ward, First Hospital of Jilin University, Changchun, China

    Corresponding Author:

    Li Chen, Prof Dr

    School of Nursing

    Jilin University

    965 Xinjiang Street

    Changchun, 130021

    China

    Phone: 86 043185619366

    Email: chen_care@126.com


    Background: Mild cognitive impairment (MCI) is often a precursor of dementia, and patients with MCI develop dementia at a higher rate than healthy older adults. Early detection of cognitive decline at the MCI stage supports better planning of care and interventions. At present, the use of virtual reality (VR) in screening for MCI in older adults is promising, but there is little evidence regarding the use of virtual supermarkets to screen for MCI.

    Objective: The objectives of this study are to validate a VR game–based test, namely, the Virtual Supermarket Program (VSP), for differentiating patients with MCI and healthy controls and to identify cutoff scores for different age levels.

    Methods: Subjects were recruited from several nursing homes and communities in Changchun, China. They were divided into a healthy control group (n=64) and an MCI group (n=62). All subjects were administered the VSP and a series of neuropsychological examinations. The study determined the optimal cutoff, discriminating validity, concurrent validity, and retest reliability of the VSP. We used the area under the receiver operating characteristic curve (AUC) to evaluate the discriminating validity and obtain the optimal cutoff values. Pearson correlation analysis and the intraclass correlation coefficient were used to evaluate the concurrent validity and retest reliability, respectively.

    Results: A cutoff score of 46.4 was optimal for the entire sample, yielding a sensitivity of 85.9% and specificity of 79.0% for differentiating individuals with MCI and healthy controls, and the AUC was 0.870 (95% CI 0.799-0.924). The median index of VSP score was 51.1 (range 42.6-60.0). There was a moderate positive correlation between the VSP total score and Mini-Mental State Examination score (r=0.429, P<.001). There was a strong positive correlation between VSP total score and Montreal Cognitive Assessment score (r=0.645, P<.001). The retest reliability of the VSP was feasible (r=0.588, P=.048).

    Conclusions: The VSP is interesting and feasible for subjects. It shows high sensitivity and specificity for the identification of MCI in older adults, which makes it a promising screening method. The VSP may be generalized to older adults in other countries, although some cultural adaptation may be necessary.

    Trial Registration: Chinese Clinical Trial Registry ChiCTR2000040074; https://www.chictr.org.cn/showprojen.aspx?proj=64639

    JMIR Serious Games 2021;9(4):e30919

    doi:10.2196/30919

    Keywords

    virtual reality; mild cognitive impairment; dementia; ambient intelligence; digital health; elderly population; aging 


    Dementia is becoming more prevalent worldwide. According to the “World Alzheimer Report 2018” [1], 50 million people worldwide had dementia in 2018. As the most populous country, China has experienced the greatest burden of dementia. The total number of people with dementia in China will reach 23.3 million by 2030 [2], and in 2020, the burden of dementia-related disability and care in China reached US $250 billion, accounting for nearly one-fifth of the global dementia-related costs [3]. However, there is no effective treatment for dementia; therefore, increasing attention has been given to the precursors of dementia. Mild cognitive impairment (MCI) is often a precursor of dementia, but it can also be due to other pathologies (such as Parkinson disease and tumors) [4,5]. Although not all forms of MCI lead to dementia, the mean annual conversion rate of MCI to dementia is approximately 10%, which is far higher than the annual incidence (1%-2%) in healthy older adults [6,7]. Relevant studies have shown that interventions for MCI can effectively delay or even reverse cognitive decline [8]. Therefore, early detection of cognitive decline at the MCI stage is vital for better planning of care and interventions.

    The commonly used cognitive screening methods are paper-and-pencil neuropsychological testing [9] and computerized cognitive testing [10,11]. Compared to neuropsychological testing, the advantages of computerized cognitive testing include standardized administration procedures and presentation of stimuli and accurate measurement of response time [12,13]. Virtual reality (VR) is widely used in research with computerized cognitive testing for MCI [14,15]. VR is an ecologically valid and informative evaluation instrument and provides an opportunity to improve cognitive screening [16-18]. Studies have shown that VR is recommended for the screening of older adults with MCI or dementia [15,19]. Among older adults with cognitive impairment screened through VR, virtual supermarket has been accepted [20] and could be used to detect cognitive impairment in older adults [21,22].

    Virtual supermarkets are a novel method for screening for MCI and assessing cognition through shopping-related activities. Related research using virtual supermarkets is effective in screening for MCI [20,22,23]. Atkins et al [24] developed the Virtual Reality Functional Capacity Assessment Tool, which assesses functional abilities related to shopping, and found a significant relationship between task performance and cognitive performance. Werner et al [22] developed the Virtual Action Planning-Supermarket, which allows subjects to navigate the store, purchase 7 items, and then pay for them, and they distinguished healthy older adults and MCI patients. Significant differences were found between healthy older adults and patients with MCI in the Virtual Action Planning-Supermarket measures. However, they assessed few cognitive domains and screened for cognitive impairment by assessing only executive function. Zygouris et al [20] developed a screening tool called the Virtual Supermarket Test to detect MCI and assessed 4 cognitive abilities in older adults through four shopping-related tasks. The program can operate on Android-based tablet devices or computers. The discriminative validity of the program was good in older adults over 70 years of age. However, Zygouris et al [20] developed virtual supermarket as a cognitive training application that does not specifically screen for MCI, and it screens few cognitive domains. Nevertheless, the literature supports virtual supermarket in screening for MCI [23]. However, the virtual supermarkets developed in other countries may not be suitable screens for older Chinese adults, and there is no virtual supermarket software to screen for MCI in China.

    The Virtual Supermarket Program (VSP) developed in this study screened subjects with MCI by assessing their ability to shop in conditions relevant to daily life. Subjects could actively explore the VSP environment while performing the task. Compared with other versions of the virtual supermarket, VSP provides a more comprehensive screening, involves a rich environment with VR technology, and is more in line with the living and cultural habits of older adults in China. Therefore, we aimed to verify the feasibility of VSP for screening for MCI in older adults in China.


    Study Design

    We evaluated the feasibility of VSP for screening patients with MCI (diagnosed in accordance with Petersen criteria) [25] over 60 years of age in China. The study followed the STARD checklist for the reporting of studies of diagnostic test accuracy [26]. This study was conducted between December 2020 and February 2021. The study protocol was approved by the School of Nursing (approval 2020082805), Jilin University, and it has been registered in the Chinese Clinical Trial Registry (registration ChiCTR2000040074). All subjects were informed about the purpose of the study before providing their verbal consent. The VSP used in this study was a new tool developed by the authors' team at the School of Nursing, Jilin University, China, and a company was entrusted to provide technical support.

    Population

    Subjects were from several nursing homes and communities in Changchun, China. The inclusion criteria of the subjects with MCI were as follows: (1) MCI diagnosis in accordance with the Petersen criteria [25]; (2) age≥60 years; (3) a Montreal Cognitive Assessment (MoCA) scale [27] score of <26; (4) a Mini-Mental State Examination (MMSE) [28] score of ≥24; (5) no impairment in functional daily living activities (Activities of Daily Living [ADL] scale [29] score of ≤26); (6) absence of psychiatric illnesses, with particular reference to depressive symptoms (15-item Geriatric Depression Scale [GDS-15] [30] score of ≤8); and (7) absence of severe auditory/visual loss that can prevent the use of technological devices and from executing the VSP. The inclusion criteria of subjects in the healthy control group were inclusion criteria (2) and (4)-(7) from the aforementioned list and an MoCA score of ≥26.

    Sample Size

    We calculated the sample size using equation 1 [31] , and we set the confidence level (1-α) at 95% and the allowable error (δ) of sensitivity and specificity at 10%. We estimated the sample size required for sensitivity (the sample size for the MCI group) and the sample size required for specificity (the sample size for the healthy control group).

    The sensitivity and specificity of the tools developed by Zygouris et al [21] were 82.35% and 95.24%, respectively, which were used to calculate the sample size for the MCI group and healthy control group in this study. We predicted a dropout rate of 10%. The sample size of the MCI group was 62, and that of the healthy control group was 20 in this study. Therefore, we needed to enroll a minimum of 82 subjects.

    Study Design

    This study recruited subjects over the age of 60 years in nursing homes and communities. The subjects who volunteered for the study were first administered a neuropsychological test to screen for MCI in accordance with the Peterson criteria [25]. The test took place in a quiet and bright room. First, baseline characteristics (age, education level, height and weight, and previous computer use before VSP) and past medical history were collected. Older adults with depression were excluded using the GDS-15. Then, the Clinical Dementia Rating (CDR) and MMSE were used to exclude patients with dementia. Thereafter, MoCA and functional status (ADL scale) measures were collected. Last, VSP tests were performed for all subjects. Prior to each task in the VSP, the subjects were repeatedly instructed on the task, and operation instructions were available throughout the test through the user interface. The VSP was managed by independent researchers, who were blinded to the subjects' cognitive status. All subjects used the VSP application under the same circumstances.

    All data were obtained directly from the subjects and included neuropsychological test scores, VSP scores, and general information. The assessors were blinded to the cognitive status or grouping of the subjects. Figure 1 shows a flowchart of subject recruitment and testing.

    ‎
    Figure 1. Flowchart of subject recruitment and testing. MCI: mild cognitive impairment; VSP: Virtual Supermarket Program.
    View this figure

    Neuropsychological Assessments

    Subjects were administered a battery of neuropsychological tests, including the CDR, MMSE, GDS-15, ADL, and MoCA. The reference standard results were available to the testers. The neuropsychological tests used in this study were as follows.

    CDR

    The CDR assesses 6 abilities on a 5-point scale [32]: no dementia (CDR=0), questionable dementia (CDR=0.5), mild dementia (CDR=1.0), moderate dementia (CDR=2.0), and severe dementia (CDR=3.0). Finally, the results of the 6 ability ratings are combined into one rating on the basis of the scoring standard, and there is no dementia when the CDR was zero [33].

    MMSE

    The MMSE has 30 questions, and the highest potential score is 30. The optimal cutoff values differ for subjects with different educational levels. Subjects who are illiterate and scored ≤17 points, those with a primary school education who scored ≤20 points, and those with a high school education or above who scored ≤24 points are classified as having dementia [34].

    GDS-15

    The GDS-15 comprises 15 questions to which subjects are asked to respond with “yes” or “no.” The total score can range from 0 to 15, and a total score of >8 is associated with depressive symptoms [30].

    MoCA Scale

    The MoCA scale comprises 32 questions, and the highest potential score is 30. If the number of years of education is ≤12 years, 1 point is added to the actual total score, and a final total score of >26 is considered normal [27].

    ADL Scale

    The ADL scale comprises 14 items that are scored at 4 levels. The total score can range from 14 to 56, and a total ADL score of >26 points indicate an impaired ability to perform activities of daily living [29].

    Virtual Supermarket Program

    We developed the VSP to be in line with Chinese cultural habits and with reference to the virtual supermarket of Zygouris et al [20,21]. We mimicked the MoCA and designed the VSP with VR technology. For instance, for the assessment of temporal orientation, the MoCA scale asks the subjects to identify the current date. In the VSP, we designed the link to input the payment password, and the password was the current date. All the tasks in the VSP were designed in this manner. Shopping-related tasks were used to assess learning and memory, executive functions, language, time orientation, and complex attention of older adults and to explore the feasibility of the VSP for screening MCI in older Chinese adults. There were 9 tasks in the VSP, and it was run on a computer (CPU, i5 or higher; memory, ≥16 GB; independent video memory, ≥2 GB; storage capacity, ≥200 GB) for which subjects did not need VR glasses and a handle to operate. The VSP was operated with a computer mouse and a keyboard (Multimedia Appendix 1). The following tasks were included in the VSP: task 1: memorize shopping list; task 2: look at the map to get directions to the supermarket and make purchases on the list; task 3: recall the sales announced by voice in the supermarket; task 4: label the item that have lost their labels; task 5: determine whether the correct items were purchased; task 6: calculate the amount for the purchased items; task 7: enter the correct payment code (current date); task 8: compare lucky draw numbers at the service center; and task 9: choose the correct bus route home.

    The VSP assessed five cognitive domains [35]: learning and memory (tasks 1 and 3), executive functions (tasks 2, 6, and 9), language (tasks 4 and 5), time orientation (task 7), and complex attention (task 8). The VSP score ranges from 0 to 60 (Multimedia Appendix 2). The technician devised a fixed algorithm to calculate the score for each task and the total VSP score on the basis of the accuracy with which the subject completed the task. There was no time limit for subjects to perform each VR task.

    Statistical Analysis

    Statistical analyses were performed using SPSS Statistics (version 23.0, IBM Corp) and MedCalc (version 19.2, MedCalc Software Ltd). The normality of data distributions was assessed using the Shapiro–Wilk test. Mean (SD) values were used to describe continuous variables that were normally distributed, and median (IQR) values were used to describe the continuous variables that were not normally distributed. Descriptive statistics included n (%) values for categorical variables. Independent samples t tests were used to compare continuous variables that were normally distributed between the two groups, nonparametric Mann-Whitney U rank sum tests were used to compare continuous variables that were not normally distributed between the two groups, and chi-square tests were used for the categorical variables. The feasibility of VSP screening for MCI was verified by measures of optimal cutoff, discriminating validity, concurrent validity and test-retest reliability for the VSP. Discriminating validity was examined through receiver operating characteristic (ROC) analysis of VSP total scores. The area under the receiver operating characteristic curve (AUC) can range between 0 and 1. The closer the AUC to 1, the better the diagnosis [36]. Concurrent validity was examined through Pearson correlation analyses of VSP total scores against MMSE scores [34] and MoCA scores [27]. Test-retest reliability was assessed with an intraclass correlation coefficient (ICC). A 2-sided P value of <.05 indicated statistical significance.


    Demographics and Characteristics by Cognitive Status

    The demographics and cognitive characteristics of the subjects are presented in Table 1. In the actual process of recruitment, the healthy control group collected more than the minimum sample size we previously calculated. Considering that a large number of samples can improve the test efficiency, we expanded the sample size. Initially, 181 participants were recruited (Figure 1). No adverse events occurred during the study. Finally, the study included 126 older adults (45 male and 91 female). The mean age was 77.12 years (range 61-94 years), and years of education ranged from 0 to 16 years. In total, 62 older adults were enrolled in the MCI group and 64 were enrolled in the healthy control group. There was a significant difference in age between the 2 groups (P=.02). No between-group differences in sex distribution (P=.06), years of education, or computer experience (having used a computer at least once) were found (P=.08). As expected, the 2 groups had significantly different MMSE scores and MoCA scores (P<.001).

    Table 1. Demographics and cognitive characteristics of subjects (N=126).
    Characteristics Healthy controls (n=64)Patients with mild cognitive impairment (n=62)T, z, or chi-square testP value
    Age (years), mean (SD)73.5 (16.0)82.00 (15.50)–2.486a.01
    Sex, n (%)3.613b.06

    		Female13.0 (20.3)22.0 (35.5)


    		Male51.0 (79.7)40.0 (64.5)

    Years of education, n (%)6.875b.08

    		Primary school10.0 (15.6)4.0 (6.5)


    		Middle school13.0 (20.3)19.0 (30.6)


    		High school13.0 (20.3)20.0 (32.3)


    		University28.0 (43.8)19.0 (30.6)

    Computer experience ( having used a computer at least once), n (%)2.873b.09

    		Yes29.0 (45.3)19.0 (30.6)


    		No35.0 (54.7)43.0 (64.9)

    Cognitive assessment score, mean (SD)


    		Mini-Mental State Examination score (maximum score=30)28.5 (1.3)27.0 (1.6)5.668c<.001

    		Montreal Cognitive Assessment score (maximum score=30)26.6 (1.0)20.8 (2.7)16.106c<.001

    aMann-Whitney U test.

    bChi-square test.

    cIndependent samples t tests.

    Performance in the VSP Between Patients With MCI and Healthy Controls

    The performance scores for the individual tasks on the VSP are summarized in Table 2. The mean scores in each VSP task in the healthy control group were higher than those in the MCI group. With the exception of the difference for task 6, these differences were significant. The total VSP score in the MCI group was significantly lower than that in the healthy control group (P<.001).

    Table 2. Performance scores in the Virtual Supermarket Program.
    Task scoreHealthy controls (n=64), mean (SD)Patients with mild cognitive impairment (n=62), mean (SD)t test (independent samples)P value
    Task 19.9 (0.5)9.6 (1.0)1.999.048
    Task 24.0 (2.0)2.3 (2.5)4.265<.001
    Task 36.0 (3.5)4.1 (3.8)2.940.004
    Task 46.0 (0.5)5.6 (1.2)2.087.04
    Task 55.6 (0.8)4.5 (2.0)4.379<.001
    Task 65.8 (0.9)5.8 (1.1)0.213.83
    Task 75.5 (1.6)4.7 (2.5)2.328.02
    Task 84.0 (1.4)2.0 (1.6)7.407<.001
    Task 94.8 (1.1)2.8 (2.5)5.708<.001
    Total51.4 (4.8)41.3 (7.7)8.894<.001

    Potential of the VSP to Distinguish Between Patients With MCI and Healthy Controls

    An ROC curve analysis was performed with the total VSP scores that were used to distinguish between patients with MCI and healthy controls (Figure 2). The AUC was found to be 0.870 (95% CI 0.799-0.924, P<.001). An optimal statistical cutoff was achieved at a cutoff score of 46.4 points (85.9% sensitivity, 79.0% specificity).

    Among subjects aged under 85 years, the optimal cutoff was 46.5 points, and the sensitivity and specificity were 72.73% and 86.79%, respectively (AUC=0.840, P<.001). Among the subjects aged over 85 years, the optimal cutoff was 47.3 points, yielding a sensitivity of 94.4% and specificity of 80.0% (AUC=0.936, P<.001). The ROC curve stratified by age is shown in Figure 3, and the discriminative validity is shown in Table 3.

    ‎
    Figure 2. Receiver operating characteristic curve analysis for the total score in the Virtual Supermarket Program.
    View this figure
    ‎
    Figure 3. Receiver operating characteristic curve analysis for the total score in the Virtual Supermarket Program among different age groups.
    View this figure
    Table 3. The total score in the Virtual Supermarket Program and the discriminant results of age stratification.

    		Optimal cutoffSensitivity, %Specificity, %Area under the receiver operation characteristic curve95% CIP valueYouden index J
    Entire sample46.485.9079.000.8700.799-0.924<.0010.650
    Age (years)

    		≤8546.572.7386.790.8400.752-0.907<.0010.600

    		>8547.394.4480.000.9360.775-0.993<.0010.744

    Correlations Between the VSP and Routine Cognitive Screening Assessments

    There was a moderate positive correlation between the total VSP scores and MMSE scores (r=0.429, P<.001). There was a strong positive correlation between the total VSP scores and MoCA scores (r=0.645, P<.001). In addition, we found a negative correlation of the total VSP scores with age (r=–0.308, P<.001). Table 4 shows the correlation matrix between the different variables.

    Table 4. Correlation matrix demonstrating the degree of associations among the Mini-Mental State Examination score, Montreal Cognitive Assessment score, age, education, and total performance scores.
    VariableAgeEducationMini-Mental State Examination ScoreMontreal Cognitive Assessment scoreScore in the Virtual Supermarket Program
    Age

    		r—a–0.119–0.342–0.247–0.308

    		P value—0.186<.001.005<.001
    Education

    		r–0.119—0.2690.1100.098

    		P value.19—.002.22.27
    Mini-Mental State Examination score

    		r–0.3420.269—0.5920.429

    		P value<.001.002—<.001<.001
    Montreal Cognitive Assessment score

    		r–0.2470.1100.592—0.645

    		P value.0050.22<.001—<.001
    Score in the Virtual Supermarket Program

    		r–0.3080.0980.4290.645—

    		P value<.0010.27<.001<.001—

    a—: not determined.

    Retest Reliability of the VSP

    A month after the first test, 9 healthy controls were randomly selected to receive the VSP again, and the results showed that the ICC between total VSP scores was feasible (r=0.588, P=.048).


    Principal Findings

    To our knowledge, this is the first study to use a locally developed virtual supermarket task to screen for MCI among older adults in China. The VSP we designed in this study was a computerized application running on a computer to screen for MCI. Through verification of its screening effectiveness, the VSP had the highest sensitivity and specificity when the optimal cutoff was 46.4. The VSP can effectively distinguish older adults with MCI from healthy controls, and it was more suitable for older adults aged 85 years and above. Performance in the VSP had a strong positive correlation with MoCA scores but a moderate positive correlation with MMSE scores. The retest reliability of the VSP scores was feasible.

    Comparing Performance Between Patients With MCI and Healthy Controls

    We found that the overall scores and scores for each task were significantly higher in the healthy control group, but task 6 did not show a significant difference between the 2 groups. In task 6, the subjects were asked to calculate the total cost of the items they had purchased. Given the situation of older adults, the settings related to the task list in the system were set to be relatively simple, which is probably why task 6 did not show a significant difference between the MCI subjects and healthy controls. This is a solvable problem. In the future, we will adjust the difficulty of task 6 by increasing the number of items purchased or adjusting the price of items.

    Compared to the healthy controls, the subjects in the MCI group performed worse on tasks (tasks 1, 2, 3, 5, and 9) that involved learning and memory and executive function. The MCI subjects were more likely to forget or buy the wrong item on their shopping list, and they could not perform the shopping tasks very well. This is evidence that these tasks were better at distinguishing between the 2 groups. Task 5 and task 8 represent the ingenuity in our task design. Task 5 involved an assessment of learning and memory, executive function and language that comprehensively evaluated the subjects’ cognition. Task 8 evaluated the subjects’ complex attention. The subjects needed to use their attention, hearing, and finger reflexes at the same time to complete the task. This task could distinguish the MCI subjects and healthy controls well. Above all, these tasks highlight the effectiveness of the VSP in distinguishing between the MCI subjects and healthy controls.

    Discriminating the Validity of the VSP Total Score

    In this study, the VSP score could effectively distinguish between MCI subjects and healthy controls. The sensitivity and specificity were 85.90% and 79.00%, respectively. The AUC based on the VSP scores was 0.870 in this study. Compared with other studies, this level of accuracy is relatively high. Boz et al [23] used the Virtual Supermarket, which was developed in Greece [21], to screen for MCI among older Turkish adults. They reported that the sensitivity and specificity were 74.00% and 85.00%, respectively; they did not report the AUC. The sensitivity and specificity of the virtual supermarket task were better in our study than in that by Boz et al [23]. This may have resulted from using Euros on the payment screen, which adds additional complexity and cognitive burden [23]. The system of Cognitive Assessment by Virtual Reality [35] is adapted from the RE@CH module, and Chua et al [37] reported an AUC of 0.821, a sensitivity of 78.2%, and a specificity of 75.7% in the Singaporean population. Compared to the task used by Chua et al [37], the VSP is more in line with the culture and living habits of older Chinese adults, and the sample size in our study was larger.

    Age is an important factor in cognition. Among subjects younger than 85 years, the AUC, sensitivity, and specificity were 0.840, 72.73%, and 86.79%, respectively. Among subjects over 85 years of age, the AUC, sensitivity, and specificity were 0.903, 94.44%, and 80.00%, respectively. As people age, their cognitive performance declines. Compared with younger subjects, older subjects committed more task errors and had a harder time concentrating [38]. This may have led to a better discriminative validity of the VSP among individuals over 85 years of age. This finding indicates that our VSP has a wider range of uses and is suitable for older adults of different ages.

    Associations Among VSP Scores, Age, Education, and Neuropsychological Testing

    VSP scores were negatively correlated with age, which is consistent with the work of Chua et al [37,39]. Considered a task involving multiple demands, the VSP was formed by a series of activities consisting of goals during its execution, and it requires activation and cooperation of multiple cognitive domains [40]. The older the subjects, worse the VSP performance and scores. There was no correlation between VSP scores and education level. The literature is not conclusive regarding the impact of education level on cognition [41,42]. The VSP that was developed and designed in this study was suitable for older adults with different educational levels and was intended to have wide applicability. The tasks closely reflect to the habits of older adults, and successful performance does not necessarily depend on education. Kang et al [43] did not report an effect of education levels among older adults with a multiple-order VR task for neuropsychological assessment.

    The total VSP performance scores showed a moderate positive correlation with MMSE scores (r=0.429, P<.001) and a strong positive correlation with MoCA scores (r=0.645, P<.001). The correlation between VSP scores and MoCA scores was significantly higher than that between VSP scores and MMSE scores, which was consistent with our expectation. The VSP used in this study was specifically developed for detecting MCI, and the design of VSP mimicked the MoCA scale. Research has shown that the MoCA scale has more advantages than the MMSE scale in detecting visual executive dysfunction [44]. Therefore, the correlation between VSP and MoCA scores may be stronger than that between VSP and MMSE scores. In addition, although the MMSE has a modest specificity for screening MCI, its sensitivity for screening MCI is low, and the capacity to detect MCI converters is poor [28]. This outcome is consistent with that of Oliveira et al, who also showed a weak correlation between MMSE scores and performance in VR research [39]. Chan et al [45] and Chua et al [37] also conducted correlation analyses with MoCA scores when verifying the screening application for MCI, which they had developed, and showed strong correlations between the measures.

    Clinical Implications

    Our findings provide preliminary evidence that the VSP can be used for MCI screening in Chinese communities and nursing homes, providing a more convenient, concise, and effective tool for MCI screening in China. We introduced several short activities based on VR to assess different cognitive domains, rather than using a single game, which allowed us to assess impairments in the major cognitive domains. The subjects were screened using shopping tasks based on the abovementioned cognitive domains. After completing the task, the obtained scores were compared with the cutoff value of 46.4, which could preliminarily screen whether the subjects had MCI. This screening program could provide rapid screening in Chinese communities and nursing homes. It expands this new method of screening for MCI and provides new ideas and methods for the development of MCI screening software in China. In addition, compared to traditional neuropsychological tests, the VSP has an automatic scoring function and allows data to be extracted and interpreted more easily. The VSP has other advantages, including administration cost savings and better ecological relevance, and is suitable for unsupervised use in a home or clinical setting. We observed that owing to the visual content and interaction style through VR, most subjects found the task more interesting than traditional screening tools and were more willing to take the test. Yun et al [46] also reported elderly people's interest in VR. Implementation studies are needed in the future to evaluate its rollout in clinical practice.

    Study Limitations and Future Research

    There are several limitations in the study, which limit the generalizability of the results. First, subjects in the MCI group were older than those in the healthy control group, which may have affected their performance. Age is an important factor in cognition. To further evaluate the use of VSP in cognitive assessment, demographic data matching and expanding the sample size can be incorporated in future studies.

    Second, even if some of the subjects had previously used a computer, many of the subjects had limited familiarity with new technologies, particularly computer equipment and operations. Oliveira et al [39] showed that experience with the computer was not a relevant predictor for any of the models investigated. We also need to further verify the impact of new technology on the ability of older adults to engage in the task.

    Third, the correlations between VSP subtests assessing specific cognitive domains and existing neuropsychological tests that assess the same domains were not verified. In future research, we need to verify the correlation between the two to verify the feasibility of VSP subtests for assessing specific cognitive domains.

    Owing to technical limitations, this study did not verify the feasibility of the time to complete the VSP in screening for MCI. In the future, it will be necessary to improve the assessment technology of VSP stay time in each area to verify the feasibility of the total time of VSP to detect older adults with MCI. In addition, we need to further verify the retest reliability of the VSP in future research. We tested the VSP again a month after the first test, but the community and nursing home were closed owing to the COVID-19 pandemic, and we could not enter and leave freely. Finally, only nine subjects were retested on the VSP. In the future, it will be necessary to further expand the sample size to verify the retest reliability of the VSP.

    Conclusions

    The VSP we designed in this study is a computerized application running on a computer to identify MCI. This study demonstrated the feasibility of the VSP in distinguishing between MCI and healthy adults among older Chinese adults. The results of this feasibility study are invaluable. Future studies should focus on testing and validating longitudinal data for the ability to track the progression of cognitive decline.

    Acknowledgments

    We would like to extend our gratitude to Ms Xu Yan and Ms Wan En of the Boyuan·Xiangzhiyuan nursing home, Ms Gao Jie of the Wenxin nursing home, and Ms Xu Jun of the Duoen nursing home, who supported us in various ways, including screening and referring patients for the study. We are thankful to Ms Zhao Lijing of Jilin University, Mr Chen Yanqiang of the Jilin University Xinmin Campus from the retirement office, Ms Han Song of the community health service center of Yong Chang, and Ms Huang Xiaoxia of the Changchun city community for their assistance. This study was supported by National Natural Science Foundation of China (grant 81902295) and Graduate Innovation Found of Jilin University, China (grant 101832020CX321). This study also received funding from “13th Five-Year” Science and Technology Research Project of Education Department of Jilin Province, China (grant JJKH20190006KJ) and Sanitation and Health technology Innovation Project of Health Department of Jilin Province, China (grant 2020J036).

    Authors' Contributions

    MY and HY contributed equally as first authors. CW and LC contributed equally as coauthors.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Screenshots of the VSP.

    DOC File , 7619 KB
    ‎
    Multimedia Appendix 2

    VSP score of each task and total score.

    DOC File , 33 KB
    1. Patterson C, Lynch C, Bliss A, Lefevre M. World Alzheimer Report 2018. Alzheimer's Disease International.   URL: https://www.alzint.org/resource/world-alzheimer-report-2018/ [accessed 2021-11-25]
    2. Xu J, Wang J, Wimo A, Fratiglioni L, Qiu C. The economic burden of dementia in China, 1990-2030: implications for health policy. Bull World Health Organ 2017 Jan 01;95(1):18-26 [FREE Full text] [CrossRef] [Medline]
    3. Jia J, Wei C, Chen S, Li F, Tang Y, Qin W, et al. The cost of Alzheimer's disease in China and re-estimation of costs worldwide. Alzheimers Dement 2018 Apr;14(4):483-491. [CrossRef] [Medline]
    4. Weil RS, Costantini AA, Schrag AE. Mild Cognitive Impairment in Parkinson's Disease-What Is It? Curr Neurol Neurosci Rep 2018 Mar 10;18(4):17 [FREE Full text] [CrossRef] [Medline]
    5. Bennett DA, Schneider JA, Bienias JL, Evans DA, Wilson RS. Mild cognitive impairment is related to Alzheimer disease pathology and cerebral infarctions. Neurology 2005 Mar 08;64(5):834-841. [CrossRef] [Medline]
    6. Petersen RC, Lopez O, Armstrong MJ, Getchius TS, Ganguli M, Gloss D, et al. Author response: Practice guideline update summary: Mild cognitive impairment: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology. Neurology 2018 Aug 21;91(8):373-374. [CrossRef] [Medline]
    7. Roberts RO, Knopman DS, Mielke MM, Cha RH, Pankratz VS, Christianson TJH, et al. Higher risk of progression to dementia in mild cognitive impairment cases who revert to normal. Neurology 2014 Jan 28;82(4):317-325 [FREE Full text] [CrossRef] [Medline]
    8. Delbroek T, Vermeylen W, Spildooren J. The effect of cognitive-motor dual task training with the biorescue force platform on cognition, balance and dual task performance in institutionalized older adults: a randomized controlled trial. J Phys Ther Sci 2017 Jul;29(7):1137-1143 [FREE Full text] [CrossRef] [Medline]
    9. Knopman DS, Petersen RC. Mild cognitive impairment and mild dementia: a clinical perspective. Mayo Clin Proc 2014 Oct;89(10):1452-1459 [FREE Full text] [CrossRef] [Medline]
    10. Lim YY, Ellis KA, Harrington K, Ames D, Martins RN, Masters CL, The Aibl Research Group. Use of the CogState Brief Battery in the assessment of Alzheimer's disease related cognitive impairment in the Australian Imaging, Biomarkers and Lifestyle (AIBL) study. J Clin Exp Neuropsychol 2012;34(4):345-358. [CrossRef] [Medline]
    11. Tierney MC, Lermer MA. Computerized cognitive assessment in primary care to identify patients with suspected cognitive impairment. J Alzheimers Dis 2010;20(3):823-832. [CrossRef] [Medline]
    12. Zygouris S, Tsolaki M. Computerized cognitive testing for older adults: a review. Am J Alzheimers Dis Other Demen 2015 Feb;30(1):13-28. [CrossRef] [Medline]
    13. Wild K, Howieson D, Webbe F, Seelye A, Kaye J. Status of computerized cognitive testing in aging: a systematic review. Alzheimers Dement 2008 Nov;4(6):428-437 [FREE Full text] [CrossRef] [Medline]
    14. Tuena C, Pedroli E, Trimarchi PD, Gallucci A, Chiappini M, Goulene K, et al. Usability Issues of Clinical and Research Applications of Virtual Reality in Older People: A Systematic Review. Front Hum Neurosci 2020;14:93 [FREE Full text] [CrossRef] [Medline]
    15. Wang S, Yin H, Li G, Jia Y, Leng M, Meng Q, et al. Detection of Mild Cognitive Impairment Based on Virtual Reality: A Scoping Review. Curr Alzheimer Res 2020;17(2):126-140. [CrossRef] [Medline]
    16. Lecouvey G, Morand A, Gonneaud J, Piolino P, Orriols E, Pélerin A, et al. An Impairment of Prospective Memory in Mild Alzheimer's Disease: A Ride in a Virtual Town. Front Psychol 2019;10:241 [FREE Full text] [CrossRef] [Medline]
    17. Giovannetti T, Yamaguchi T, Roll E, Harada T, Rycroft SS, Divers R, et al. The Virtual Kitchen Challenge: preliminary data from a novel virtual reality test of mild difficulties in everyday functioning. Neuropsychol Dev Cogn B Aging Neuropsychol Cogn 2019 Nov;26(6):823-841. [CrossRef] [Medline]
    18. Rizzo A, Gambino G, Sardo P, Rizzo V. Being in the Past and Perform the Future in a Virtual World: VR Applications to Assess and Enhance Episodic and Prospective Memory in Normal and Pathological Aging. Front Hum Neurosci 2020;14:297 [FREE Full text] [CrossRef] [Medline]
    19. Neguț A, Matu S, Sava FA, David D. Virtual reality measures in neuropsychological assessment: a meta-analytic review. Clin Neuropsychol 2016 Feb;30(2):165-184. [CrossRef] [Medline]
    20. Zygouris S, Iliadou P, Lazarou E, Giakoumis D, Votis K, Alexiadis A, et al. Detection of Mild Cognitive Impairment in an At-Risk Group of Older Adults: Can a Novel Self-Administered Serious Game-Based Screening Test Improve Diagnostic Accuracy? J Alzheimers Dis 2020;78(1):405-412 [FREE Full text] [CrossRef] [Medline]
    21. Zygouris S, Giakoumis D, Votis K, Doumpoulakis S, Ntovas K, Segkouli S, et al. Can a virtual reality cognitive training application fulfill a dual role? Using the virtual supermarket cognitive training application as a screening tool for mild cognitive impairment. J Alzheimers Dis 2015;44(4):1333-1347. [CrossRef] [Medline]
    22. Werner P, Rabinowitz S, Klinger E, Korczyn AD, Josman N. Use of the virtual action planning supermarket for the diagnosis of mild cognitive impairment: a preliminary study. Dement Geriatr Cogn Disord 2009;27(4):301-309. [CrossRef] [Medline]
    23. Eraslan Boz H, Limoncu H, Zygouris S, Tsolaki M, Giakoumis D, Votis K, et al. A new tool to assess amnestic mild cognitive impairment in Turkish older adults: virtual supermarket (VSM). Neuropsychol Dev Cogn B Aging Neuropsychol Cogn 2020 Sep;27(5):639-653. [CrossRef] [Medline]
    24. Atkins AS, Khan A, Ulshen D, Vaughan A, Balentin D, Dickerson H, et al. Assessment of Instrumental Activities of Daily Living in Older Adults with Subjective Cognitive Decline Using the Virtual Reality Functional Capacity Assessment Tool (VRFCAT). J Prev Alzheimers Dis 2018;5(4):216-234. [CrossRef] [Medline]
    25. Petersen RC. Mild cognitive impairment as a diagnostic entity. J Intern Med 2004 Sep;256(3):183-194 [FREE Full text] [CrossRef] [Medline]
    26. Cohen JF, Korevaar DA, Altman DG, Bruns DE, Gatsonis CA, Hooft L, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open 2016 Nov 14;6(11):e012799 [FREE Full text] [CrossRef] [Medline]
    27. Nasreddine Z, Phillips N, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc 2005 Apr;53(4):695-699. [CrossRef] [Medline]
    28. Arevalo-Rodriguez I, Smailagic N, Roqué I Figuls M, Ciapponi A, Sanchez-Perez E, Giannakou A, et al. Mini-Mental State Examination (MMSE) for the detection of Alzheimer's disease and other dementias in people with mild cognitive impairment (MCI). Cochrane Database Syst Rev 2015 Mar 05(3):CD010783 [FREE Full text] [CrossRef] [Medline]
    29. Graf C, Hartford Institute for Geriatric Nursing. The Lawton instrumental activities of daily living (IADL) scale. Medsurg Nurs 2008 Oct;17(5):343-344. [Medline]
    30. Cullum S, Tucker S, Todd C, Brayne C. Screening for depression in older medical inpatients. Int J Geriatr Psychiatry 2006 May;21(5):469-476. [CrossRef] [Medline]
    31. Hajian-Tilaki K. Sample size estimation in diagnostic test studies of biomedical informatics. J Biomed Inform 2014 Apr;48:193-204 [FREE Full text] [CrossRef] [Medline]
    32. Morris JC. The Clinical Dementia Rating (CDR): current version and scoring rules. Neurology 1993 Nov;43(11):2412-2414. [CrossRef] [Medline]
    33. Lim WS, Chong MS, Sahadevan S. Utility of the clinical dementia rating in Asian populations. Clin Med Res 2007 Mar;5(1):61-70 [FREE Full text] [CrossRef] [Medline]
    34. Folstein MF, Folstein SE, McHugh PR. "Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 1975 Nov;12(3):189-198. [CrossRef] [Medline]
    35. Lim JE, Wong WT, Teh TA, Lim SH, Allen JC, Quah JHM, et al. A Fully-Immersive and Automated Virtual Reality System to Assess the Six Domains of Cognition: Protocol for a Feasibility Study. Front Aging Neurosci 2020;12:604670 [FREE Full text] [CrossRef] [Medline]
    36. Park SH, Goo JM, Jo C. Receiver operating characteristic (ROC) curve: practical review for radiologists. Korean J Radiol 2004;5(1):11-18 [FREE Full text] [CrossRef] [Medline]
    37. Chua SIL, Tan NC, Wong WT, Allen JC, Quah JHM, Malhotra R, et al. Virtual Reality for Screening of Cognitive Function in Older Persons: Comparative Study. J Med Internet Res 2019 Aug 01;21(8):e14821 [FREE Full text] [CrossRef] [Medline]
    38. Ben-Sadoun G, Sacco G, Manera V, Bourgeois J, König A, Foulon P, et al. Physical and Cognitive Stimulation Using an Exergame in Subjects with Normal Aging, Mild and Moderate Cognitive Impairment. J Alzheimers Dis 2016 Jun 30;53(4):1299-1314. [CrossRef] [Medline]
    39. Oliveira CR, Lopes Filho BJP, Esteves CS, Rossi T, Nunes DS, Lima MMBMP, et al. Neuropsychological Assessment of Older Adults With Virtual Reality: Association of Age, Schooling, and General Cognitive Status. Front Psychol 2018;9:1085 [FREE Full text] [CrossRef] [Medline]
    40. Burgess PW, Veitch E, de Lacy Costello A, Shallice T. The cognitive and neuroanatomical correlates of multitasking. Neuropsychologia 2000;38(6):848-863. [CrossRef] [Medline]
    41. Farina M, Paloski LH, de Oliveira CR, de Lima Argimon II, Irigaray TQ. Cognitive Reserve in Elderly and Its Connection with Cognitive Performance: A Systematic Review. Ageing Int 2017 May 23;43(4):496-507. [CrossRef]
    42. Salthouse TA. Correlates of cognitive change. J Exp Psychol Gen 2014 Jun;143(3):1026-1048 [FREE Full text] [CrossRef] [Medline]
    43. Kang YJ, Ku J, Han K, Kim SI, Yu TW, Lee JH, et al. Development and clinical trial of virtual reality-based cognitive assessment in people with stroke: preliminary study. Cyberpsychol Behav 2008 Jun;11(3):329-339. [CrossRef] [Medline]
    44. Mai LM, Sposato LA, Rothwell PM, Hachinski V, Pendlebury ST. A comparison between the MoCA and the MMSE visuoexecutive sub-tests in detecting abnormalities in TIA/stroke patients. Int J Stroke 2016 Jun;11(4):420-424. [CrossRef] [Medline]
    45. Chan J, Wong A, Yiu B, Mok H, Lam P, Kwan P, et al. Correction: Electronic Cognitive Screen Technology for Screening Older Adults With Dementia and Mild Cognitive Impairment in a Community Setting: Development and Validation Study. J Med Internet Res 2021 Jan 19;23(1):e26724 [FREE Full text] [CrossRef] [Medline]
    46. Yun SJ, Kang M, Yang D, Choi Y, Kim H, Oh B, et al. Cognitive Training Using Fully Immersive, Enriched Environment Virtual Reality for Patients With Mild Cognitive Impairment and Mild Dementia: Feasibility and Usability Study. JMIR Serious Games 2020 Oct 14;8(4):e18127 [FREE Full text] [CrossRef] [Medline]

    ‎
    ADL: Activities of Daily Living
    CDR: Clinical Dementia Rating
    GDS-15: 15-item Geriatric Depression Scale
    ICC: intraclass correlation coefficient
    MCI: mild cognitive impairment
    MMSE: Mini-Mental State Examination
    MoCA: Montreal Cognitive Assessment
    ROC: receiver operating characteristic
    AUC: Area under the receiver operating characteristic curve
    VR: virtual reality
    VSP: Virtual Supermarket Program

    Edited by N Zary; submitted 04.06.21; peer-reviewed by HG Seo, NC Tan, S Zygouris; comments to author 22.07.21; revised version received 29.08.21; accepted 10.09.21; published 03.12.21

    Copyright

    ©Mingli Yan, Huiru Yin, Qiuyan Meng, Shuo Wang, Yiwen Ding, Guichen Li, Chunyan Wang, Li Chen. Originally published in JMIR Serious Games (https://games.jmir.org), 03.12.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.