This was a pilot study to evaluate the usability and feasibility of a fully immersive VR cognitive training program with an enriched environment. The study consisted of a single session and a survey completed by physiatrists and occupational therapists (OTs) as well as patients with mild cognitive impairment and mild dementia. This study protocol was approved by the Institutional Review Board of Seoul National University Hospital (IRB No. 1809-126-975) on October 16, 2018. All participants provided written informed consent. The study was performed under the principles of Good Clinical Practice and the Helsinki Declaration.

The VR interface system consisted of an HTC Vive head-mounted display and a custom-made hand motion tracking module developed for hand pose estimation and 3-dimensional positions of the hands in the working space.

Figure 1 shows the overall system architecture of the module for hand pose estimation in a virtual environment. The hand motion tracking module has a camera on the palm side of the hand to capture images; the camera is synced with a computer system for the deep learning process in order to estimate the hand poses. Each of the 2 modules for both hands transmits real-time hand images to the deep learning process in the computer system through WiFi communication. The deep learning process computes finger joint positions from the images to estimate hand poses in real time, which is delivered to the interface synchronization process in the computer system for the VR game through Bluetooth. HTC Vive trackers are installed on the backside of the hands to obtain the hands’ positions, which are also used to control the VR game.

The network model of our deep learning process was designed based on the hourglass network [20]. The hourglass network has primarily been used for human body pose estimation. Therefore, the structure of the network is not optimized for hand pose estimation. We mainly focused on the modification of the feature extraction layers of the network so that finger features could be efficiently extracted. However, as it was difficult to obtain sufficient hand image data for training, we used the composite hand data generator to generate simulated hand image data [21]. More than 2 million images were generated to train our deep learning process. As shown at the bottom of Figure 1, the 2-dimensional joint position of each finger was successfully extracted using our proposed method.

With the joint position data, we could estimate the finger joint angles from an inverse kinematics model. Moreover, without the inverse kinematics model, gestures like grip could be easily recognized from the joint position data. For example, grip could be detected if the vertical positions of fingertips were lower than the threshold value. Thumb-to-finger tap gesture could also be easily detected using only the 2-dimensional joint position data [22]. These data were then used to interface with the VR game.

The hand motion tracking module was designed such that patients could wear it easily without much effort. The silicon pad and strap made the module easy to wear, remove, and clean.

We developed virtual harvest and cook games in enriched environments representing Korean rural scenery. In all the games, the patients interacted with the environment from an egocentric point of view (known as “first person point of view”). The developed algorithm is shown in Multimedia Appendix 1.

In the default scene of the program, the user was standing in a stream with a view of a country field and house (Figure 2). Sounds of water, wind, birds, and soft music were played as background music. The scene mimicked an environment familiar and comfortable for Korean elderly patients.

The harvest game aimed to improve sustained, alternating, and selective attention and working memory (Figure 3). In the game, there were 4 backgrounds: farm A (chilis, tomatoes, and cucumbers), farm B (strawberries, paprika, and eggplants), an orchard (apples, mandarins, and pears), and a hen house (brown eggs and white eggs). The type and number of crops and eggs harvested and the time limit were controlled autonomously by the operator, depending on each participant’s performance level. After the operator inputted the harvest target and time frame, an artificial voice informed the patient. If the participants were unable to complete the task in the appointed time, visual cues with green lights around the object were provided. If the participant performed incorrectly, there was no limit but it remained on record. After the participants finished the task, the instructor checked time taken by the participant to complete the task, the number of times the direction was replayed, and the harvest type and number.

The cook game focused on improvements in sustained and selective attention; working, spatial, and procedure memory; and executive function. The cook program included 3 types of recipes: fried eggs, gimbap (dried seaweed roll), and soybean paste stew. The fried egg recipe consisted of 12 steps, gimbap recipe consisted of 11 steps, and soybean paste stew recipe consisted of 14 steps. The recipe was selected by the operator depending on the participant’s level. The participants were verbally instructed by the program on how to execute the allocated recipe. If a participant could not cook the recipe properly, a green arrow around the item necessary for the present step glowed. The strategies of the cook game were based on “errorless learning” [23]. If patients could not progress to the next step, the program did not progress to the next step. After the participants performed the task, the execution time was tabulated for each step.

The VR cognitive training program was evaluated and improved through the initial usability test by 12 physiatrists and 7 OTs before the final application.

Participants including physiatrists, OTs, and patients were provided with a description of the program objectives and procedures before the test. All participants were seated on a fixed chair, wearing hand motion tracking modules, and supervised by an OT during the test (Multimedia Appendix 2). The therapist was trained to manipulate the software for the usability test and to deal with adverse events. The OT adjusted game difficulty to match the patient’s level based on the Mini-Mental State Examination (MMSE) score and Clinical Dementia Rating (CDR). Physical and verbal assistance was provided for participants who struggled to understand the instructions properly. The participants experienced all 4 harvest games and 3 cooking games. The usability and feasibility test was a single session lasting of 30 minutes.

Physiatrists and OTs with more than 1 year of clinical experience were recruited to test the usability and feasibility of the developed program. Patients were recruited from a tertiary hospital. A total of 15 patients were screened for eligibility for study inclusion. Inclusion criteria were age ≥65 years and a score of more than 19 and less than 28 points on the MMSE. Exclusion criteria were patients with moderate to severe dementia who could not understand the research content or agree voluntarily, a history of severe dizziness or epilepsy, psychiatric symptoms or behavioral problems that made it difficult to participate in the study, and other severe medical problems such as neurological or orthopedic diseases.

Physiatrists and OTs evaluated the usability of the developed VR cognitive training system using a self-report questionnaire based on a 7-point Likert-type scale. Questionnaire items consisted of 6 categories and 22 items: overall satisfaction, acceptability (needs, favorability, effectualness for mild cognitive impairment and mild dementia, and distinction), satisfactoriness (intent adequacy, setting adequacy, setting immediacy, inducement of interest, and content diversity), expectation effectiveness (attraction of voluntary participation from patients, labor reduction for therapists, and contribution to therapists’ work), stability (confirmation of proper equipment positioning, warning of system errors, external appearance of safety, durability, and stability satisfaction), and others (willingness to use, additional use intention with other therapeutic tools, and recommendation intention). The higher the score, the more positive the result. The survey also included additional suggestions regarding the developed system.

The patients’ baseline characteristics, including age, sex, medical history, MMSE, CDR, and Geriatric Depression Scale (GDS), were assessed. At the end of the experience, the feasibility of the enriched environment VR games was assessed by patients using a self-report questionnaire with a 7-point Likert-type scale based on 9 items (overall satisfaction, interest, mood, motivation, difficulty, discomfort, anxiety, willingness to use, and expectations for VR rehabilitation). The survey also included additional descriptions of subjective experiences.

Reaction time was assessed to evaluate the activity process of the central nervous system, such as motor preparation and motor program [24]. Using an iPad app (Reaction Test & Speed Test, Wang Haiwen), the instructor instructed patients that when the red screen became green, the screen should be touched as quickly as possible. This app measured response time from the color change to the screen touch. Patients tried for a total of 5 times using their dominant hand.

To assess motor speed, the finger tapping test was used [25]. Participants tapped just an iPad (digital finger tapping test, SyBu Data) with the index finger as fast as they could for 10 seconds. An average of 3 trials with both hands was completed.

Baseline characteristics and assessment data are expressed as mean and SD for continuous variables. The Wilcoxon signed-ranks test was used to evaluate changes in results before and after the usability test. The Mann-Whitney test was employed to compare the number of finger taps between the right and left hands. A P value <.05 was considered statistically significant. SPSS Statistics 21.0 for Windows (IBM Corp, Armonk, NY) was used for all analyses.

We developed a memory and attention training program using enriched VR environments for patients with mild cognitive impairment and mild dementia. Our results suggest that the VR cognitive training system is feasible and usable based on the feedback received from physiatrists, OTs, and patients with mild cognitive impairment and mild dementia. None of the participants complained of motion sickness, and no adverse events occurred. Although not statistically significant, the decreased response time without changing the rate of finger tapping in patients may reflect a temporary increase in attention after the test.

Fully immersive VR can induce psychological immersion and high presence. Lifelike VR may narrow the gap between reality and the virtual world [26]. The high level of immersion and visual realism trigger autobiographical memories [27]. A combination of the enriched environment, which is almost like reality, and VR may enable patients with cognitive decline to feel emotionally stable. Neuropsychiatric symptoms including depression, apathy, and agitation occur in most patients with dementia [28]. The patients in this study were found to be mildly depressive as per their GDS score. After the test, the patients gave positive feedback on the questionnaire with regards to comfort, anxiety, and mood. Therefore, the program might be helpful due to improvement in mood in patients with mild cognitive impairment and mild dementia. This finding is in line with a report stating psychosocial intervention through individualized reminiscence and multisensory stimulation evoked emotional and social benefits for those with dementia, along with a preserved sense of identity [29].

In this study, allowing free hand movements in elderly patients who are not used to the VR machine operation ensured that they could access the VR environment more comfortably. Physiatrists, OTs, and patients did not report incommodiousness of a custom-made hand module. However, 3 rehabilitation specialists suggested that it would be better to add haptic feedback when grasping objects and to enhance the sensitivity of the motion detector. To bring about interactions between the user and the virtual environment, several input devices are needed [30]. A potential problem with hand-held devices is that a stiff controller limits kinematic error and completes the actions without continuous participation from the patient, which may limit the effect of training. Furthermore, an unsolved problem regarding the motion tracking systems and instrumented gloves is that they are limited when creating accurate user movements in the virtual environment [31]. In addition, bacterial contamination and sterilization could be an issue for medical use of hand-held devices and instrumented gloves. There is an ongoing need to make intuitive input devices for the freedom to manipulate devices and increase accuracy. Although the hand motion tracking module we developed is a prototype, it could compensate for several shortcomings of existing products.

It is also necessary to consider the needs of caregiving staff when designing and developing interventional VR programs. Physiatrists and rehabilitation therapists have many training interface options to choose from, including VR, video gaming, and other tablet-based applications [32]. A survey of rehabilitation specialists showed high overall satisfaction and intent to use the VR system for further training. However, they reported that the hardware system needs to be improved with regards to setting the speed and adequacy. It is also worth considering incorporating a haptic device into the hardware system to make patients with cognitive impairment more comfortable with their VR tasks and to provide them the appropriate tactile stimuli. Because the hardware system is not commercialized, continuous development is needed to improve user convenience.

This study has several limitations. First, the short test time may be insufficient to influence cognitive components and motor function. Because the primary purpose of this study was to evaluate the usability and feasibility of the developed system, all participants underwent a single session of VR cognitive training. Response time decreased after the short-term VR experience in patients although it was not statistically significant. This result may suggest improved cognitive process without changes in motor speed after VR cognitive training, as no changes were noted in the finger tapping test. To evaluate the efficacy of the developed program, a further clinical trial should be designed including several training sessions and a conventional cognitive training control group. Second, motion sickness was not evaluated objectively in this study. In order to minimize motion sickness during the VR experience, the researchers observed the condition of participants carefully and continuously asked them about any inconvenience. Participants in this study did not complain of motion sickness. Nevertheless, cybersickness due to VR use must be considered, especially when fully immersive VR is being utilized in cognitively impaired patients [33]. An objective assessment that can measure motion sickness symptoms such as dizziness, nausea, and headache may be required in future studies. Last, the developed program was only used with those with mild cognitive impairment and mild dementia. This study population was chosen based on previous studies that have reported the effect of computerized cognitive training in people with cognitive impairment [3]. However, as the enriched VR environment may be beneficial for reminiscence, relaxation, and enjoyment for different stages of dementia, future studies are warranted in patients with moderate and severe dementia.

We developed a fully immersive VR cognitive training program with an enriched environment for patients with mild cognitive impairment and mild dementia. The feasibility and usability of the program were verified based on the positive satisfaction and willingness to use reported by physiatrists, OTs, and patients. Although we were not able to identify functional changes with the use of a single session, there may be an increase in attention following training. Additional clinical trials are needed to confirm the effects of this program on cognitive function, mood, and physical outcomes.