This systematic review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [38]. In this study, we present a systematic review to summarize the past studies in the neuropsychological area with VR. Because there were few studies before 2000, the range of publishing years was limited from 2000 to 2019. The earlier studies were also significant and important. However, we assumed most of the concepts and topics in those studies could reflect on future studies, since there were a negligible number of studies that were related to VR and neuropsychological tests.

Initial articles were searched in three web databases: Scopus, Web of Science, and PubMed. These engines are adequate to cover the wide spectrum of topics in the target area. The articles were searched in March 2020, and the search query string was as follows: (“virtual reality” OR “VR” OR “virtual environment”) AND (“executive function(s)” OR “EF(s)” OR “cognitive function(s)” OR “cognition”). The publishing years were limited from 2000 to 2019.

Journal articles and conference (proceeding) papers written in English were chosen, and other forms of publication, such as short reports, chapters of books, and dissertations, were excluded. To aim more deeply at EFs among the cognitive abilities, we included articles that were only related to “executive function” and “virtual reality” were used to see the effectiveness of neuropsychological tests in VR.

After duplicates were removed, all studies were checked by using keywords and abstracts in the screening process. Review and protocol articles and chapters of books were all excluded. Moreover, articles not mainly focusing on EFs were excluded. At the end of the screening process, 234 studies were left.

Evaluation of full-text articles was conducted during the eligibility process. In this phase, we excluded theoretical articles, reporting articles, and non-experimental articles. The term “virtual reality” commonly corresponds not only to articles about immersive environments but also to articles about non-immersive environments, such as the use of a flat-screen monitor, so this study excluded them as not fully immersive VR articles. A few records where we could not find the full text were also excluded.

The overall procedure of the study and the number of selected articles are shown in Figure 1. After searching the web database, 1167 articles were collected (Scopus 868, PubMed 102, and Web of Science 197). After removing duplicates and outliers, the remaining articles were used in the next phase. As a result of the screening phase, 702 articles were excluded by titles, keywords, and abstract review. With abstracts of 234 papers (data set 1), keyword analysis was conducted to examine the research trends. Then, we conducted a full-text review in the eligibility phase, and 187 papers were excluded based on the exclusion criteria. Finally, 47 papers (data set 2) were selected for the descriptive analysis.

Data analysis was conducted by two sub-analysis methods.

First, a keyword analysis was conducted with data set 1. It was important to specify keywords for the analysis, so we conducted keyword extraction with Python NLTK. The Natural Language Toolkit (NLTK) is a Python package that has been developed for natural language processing and document analysis. The NLTK extracted top 200 keywords based on their frequency from each article’s title, keywords, and abstract. Among the results of the extraction, we selected significant keywords in line with the study’s purpose and categorized them according to measuring targets, names of mental disorders, and behavioral and physiological measures. Moreover, some keywords were added from the 47 full-text descriptive analysis.

Second, a descriptive analysis was conducted in detail with data set 2. The analysis was conducted with almost the same categories as the keyword analysis, but we also added some other elements: purpose of article, measuring targets, age of participants, names of mental disorders, real-time walking, and environment of VR.

All the analyses were conducted within a certain timescale. Considering the evolution of yearly publication frequencies, the time periods were divided into 4 stages of 5-year blocks: period 1, 2000-2004; period 2, 2005-2009; period 3, 2010-2014; and period 4, 2015-2019.

In this section, we give an overview of the study of virtual neuropsychological tests in connection with EFs. There was a change of pace of the publication frequency from 2000 to 2019, with an exponential curve, as shown in Figure 2, with 936 articles after removing duplicates and outliers. Studies maintained a certain level all through the 2000s. The publication frequency began to increase conspicuously in 2009 and has rapidly increased in the past 10 years. It expresses the high interest in VR for dealing with mental disorders.

A keyword analysis was applied to data set 1 that mainly focused on EFs. For the keyword analysis, we segmented the total period (2000-2019) into four periods: 2000-2004, 2005-2009, 2010-2014, and 2015-2019. A significant increase in the publications in Figure 2 was the main rationale to classify the periods. In the keyword analysis, we examined the trends in VR applying to neuropsychology.

The final 47 articles included through PRISMA were experimental studies that conducted neuropsychological tests using fully immersive VR. The article set was divided into the same four periods as the keyword analysis. Studies on fully immersive VR were encouraged in the early 2010s, and most of the research was done within the past 5 years (period 4) with every year. Table 1 represents a summary of the 47 studies.

The most noticeable phenomenon is the appearance of treatment studies in period 4. In the early periods, when VR was first used in neuropsychological tests, it was only used for assessing cognitive states. However, high ecological validity, reproducibility of daily life, and the possibility of measuring behavior responses that were not available with traditional neuropsychological tests opened a new venue for cognitive rehabilitation [42,64,89,90]. Therefore, the research trend changed from being assessment oriented toward an additional interest for treatments. The possibility of treatment of cognitive problems using VR seems to expand continuously. In these studies, researchers have tried to train or rehabilitate the participants’ cognitive abilities (mainly inhibition and working memory) through regular and repetitive VR tasks that consisted of daily-living scenarios or game-like applications [42,58,62,64]. Some research argues that the biomarker and the degree of brain activation obtained as kinematic outcomes are indicators of treatment response [91,92].

From the perspective of changes in validation, it seems that there are two main streams. The first perspective is about transplanting existing traditional tests into VR, such as the Stroop test [29,50,57,67,69,70,73] and the trail-making task [76]. This includes most of the usability validation studies in data set 2. Because the Stroop test is a simple and ocular task, it has been considered suitable to convert into a VE. In addition, there is one study that brings the trail-making test into the VE. Second, an interesting thing is that most of the structure validation studies were conducted in supermarket environments based on the multiple-errand test [55,56,68,71]. Supermarket environments are popular due to their deep relation with daily living. Researchers have also tried to design various scenarios that are related to daily living, such as parking simulation [51] and pathfinding with Google Street View [59]. However, continuing to adapt parameters or structural properties to suit VR neuropsychological tests remains a challenge.

The increase in uncategorized studies in period 4 could also be seen in the same context. Most of the uncategorized studies examined the sense of presence and cybersickness [44,47,54,58,63,67,75,77]. These are indeed the two major problems for wider acceptance of VR for more age groups and for obtaining more ecological validity. The increase in studies to solve these problems aims for participants to feel more comfortable, which could further lead to the development of daily-living contents and expand the area of treatment. Therefore, solving these problems seems a key point to improve fully immersive neuropsychological testing in VR, but there is still a long way to go.

As shown in Figure 3, diversification started in line with the popularization of VR in the late 2000s, and the research has grown quantitatively and qualitatively ever since. Initially, studies mostly targeted low-order EFs, such as inhibition, attention, and working memory [61,66,78]. However, in recent years, VR technology has evolved to overcome the problem of presence and cybersickness. This enabled researchers to implement higher ecological ADL with various practical scenarios and VEs. Consequently, it has led to studies that deal with higher-order EFs, such as decision making, planning, and problem solving.

Inhibition and attention are functions deeply related to ADHD. Because most subjects with ADHD are children and youth, it seems that researchers have actively used classroom and apartment (more accurately, the living room) environments that those ages are familiar with [43,45,53,54,65,78]. As the Stroop test is famous for assessing one’s inhibition, many studies on inhibition and attention have translated it into VR. Especially, most studies conducted under virtual apartment environments have developed their application based on the Stroop test [50,57,73,74]. Since familiarity with the participants’ daily-living situation induces psychological comfortability, application can sophisticatedly measure the natural state of cognitive abilities.

The supermarket environment is most frequently used for measuring working memory. Planning [61,63,87] and decision making [75] are also associated with the supermarket environment. For higher-order EFs, various other environments and daily-living scenarios have been tried, such as laboratories [51,77], kitchens [47], and parking simulation [51]. It is unrealistic to precisely measure one isolated EF due to its interdependent property. Higher-order EFs should be comprehensively measured. Only one case measured planning alone, a study that was the first attempt to apply the trail-making test to immersive VR [76]. Lastly, studies on spatial and motor abilities were tried after 2012 using a treadmill or motion-tracking systems.

Since one of the EFs closely connected to other EFs we defined, EFs in ADL tend to be treated comprehensively rather than alone. These deal with specific situations, the emergence of distractors, and well-organized scenarios that assist the original task.

As shown in Figure 4, target symptoms began to diversify after period 2. Stroke and brain injury accounted for the largest percentage until period 3, but in period 4, this changed. This indicates that VR studies on neuropsychology are being conducted in a wide variety of conditions. The thing to keep an eye on is the conspicuous increase of age-related cognitive decline and a relatively high proportion of ADHD.

The many VR studies have focused on ADHD, which is a popular symptom of young age in the early stage of VR research. In Figure 7, the under-14 age group is composed of TBI and ADHD, and ADHD represents the majority of these studies. It is not surprising as almost 5% of children are diagnosed with ADHD [86]. Many previous studies have reported the positive effect of VR on children who have a mental disorder [93-95] and surely on ADHD [43,78,86]. In addition, the ease of converting the Stroop test into immersive VR would have played a role, since the Stroop test is a neuropsychological assessment to measure attention and inhibition, which are major deficit features of ADHD [43]. If we regard under 25 years as one unique group, the activeness of related research can be seen as familiarity and accessibility to VR and resistance to cybersickness of adolescents. Young people are more adaptable to VR, even immersive VR [96].

As VR technology advances and related research accumulates, obstacles that blocked the application of VR neuropsychological tests to the elderly are being resolved [97]. Figure 7 also shows the expansion of the research to the elderly. In the 65+ age group, about 80% of papers are related to age-related cognitive decline. As mentioned before, we consider SCD, MCI, AD, and dementia together as age-related cognitive decline. MCI is a symptom that decreases one’s memory, attention, and cognitive function, and it could lead to dementia or AD [98]. SCD is an early state when someone’s cognitive ability is beginning to decline [99]. In other words, SCD may indicate a pre-MCI condition, even an early marker for the symptomatic manifestation of dementia. Age-related cognitive decline symptoms are common in the elderly. In addition, depression is a quite under-investigated area despite a huge impact on daily-living executive dysfunction in 60% of depressive states in the elderly, with an exponential growth of prevalence with age, a poorer response to antidepressants, and more evolution toward dementia compared to depression without dysexecutive syndrome [100-102].

Investigating VR as a neuropsychological tool for the elderly seems increasingly realistic since VR has become more familiar even for seniors. As we have seen before, there has been a lot of research on and technological advancement in presence and cybersickness in VR, and it is possible that the public and even the elderly could comfortably access it [103-105], which enables expansion to the elderly and causes rapid growth of the research that targets age-related cognitive decline symptoms after period 4.

Overall, immersive VR offers a high number of possibilities to deal with neuropsychological symptoms in various age groups. It shows the versatility of VR, which is applicable from youth with ADHD to the elderly with age-related cognitive decline.

It seems that measuring behavioral data and physiological data is becoming increasingly important. These kinds of data are used to extend the concept of embodied cognition to measure the neuropsychological status and to overcome the weaknesses of VR.

Embodied cognition is a broad notion from embodiment thesis in philosophy, and it states that an agent’s cognition is powerfully affected by aspects of their body beyond the brain itself. This means that sensorimotor experiences and actions are crucial to cognitive processing [106]. These aspects are thought to be crucial for cognitive performances in the elderly [107]. In this context, measuring behavioral and physiological variables in immersive VR would be a key method of assessing the neuropsychological status. Thus, the growth of associated research seems to be increasing because of the ecological characteristics of VR environments. In addition, when we looked deeper into gait or walking, there was a large difference in the appearance of walking on a treadmill and real-time walking in periods 3 and 4. With real walking in a VR neuropsychological test, researchers were able to apply the concept of embodied cognition and measure it in ADL.

Furthermore, an increase in physiological data measurements should be considered as these can provide auxiliary ways to assess the neuropsychological status of a patient. In addition, it is also considered to be a helpful way to overcome cybersickness [108-110]. Cybersickness is a major obstacle in applying VR to older people and people who have difficulties adapting to VR because they feel dizzy. In addition, research on emotion recognition and presence in VR uses physiological data [111-114]. With these efforts added, VR neuropsychological tests’ versatility for all ages continues to improve.

Overall, by using behavioral and physiological data measurement, VR neuropsychological tests are not only getting over existing drawbacks but also expanding to the concept of embodied cognition to improve its measurement capability.

This review had a few limitations. First, keyword analysis was conducted by only using titles and abstracts of the articles. This means that there is a possibility of a difference with the articles’ main texts. Second, we only included articles that described research conducted with real participants, so conceptual and theoretical trends in the area were not covered. Finally, not every article markedly described the target EFs. Likewise, there were some ambiguities in clearly assigning the purpose of each article to assessment, treatment, structure validation, or usability validation. Even though we attempted to judge based on the description of what EFs the articles targeted, and what their intended purpose was, there was still room for error. Future research could include broader studies in the area and use more rigorous methods to analyze the trends.

This review will assist researchers in understanding the trends in VR neuropsychological tests over the past 20 years. Associated research has been on the rise and has sharply increased in recent years. In this process, the advancements in technology and various approaches have led to diversified target cognitive abilities, including EFs, as well as target symptoms. Moreover, collecting behavioral and physiological data enables a wide understanding about treating EFs by using the concept of embodied cognition. As a result, VR neuropsychological tests now cover a wide range of age groups and extend beyond assessment tools to treatment tools. This review shows that there is a continuously increased interest in dealing with neuropsychology by using fully immersive VR, and it is expected that this will help advance research in this area.