The difference between serious games (SGs) or transformational games [1] and entertainment video games is that SGs are software apps with a defined goal that goes beyond pure entertainment [2]. In the field of health care, SGs have been developed and used for a variety of purposes, such as training and simulation [3], diagnosis and therapy [4-6], education [7], and other purposes [8]. In recent years, there has been a growing interest in using games to target health conditions, such as stroke [9], Parkinson disease [10], and autism [11]. To a lesser extent, SGs have also been used with people with dementia [12].

Dementia is a neurocognitive disorder [13,14], which impairs cognitive and emotional behaviors, such as memory, language, problem-solving, anxiety, irritability, visuospatial issues, gait and balance-related issues, and other dementia-related aspects [13-18]. Although Alzheimer disease is the most common form of dementia [15], there are other types of dementia, such as vascular dementia [19], Lewy body dementia [20], frontotemporal dementia [21], and mixed dementia [22]. In general, the disease can profoundly affect the carriers, family members, professional caregivers [23], and health care systems [24]. It is estimated that over 35 million individuals worldwide have dementia and that dementia-related expenses reached approximately US $818 billion in 2015 [25]. Portugal alone had expenses ranging between US $1652.8 million and US $2120.4 million in 2009 [24].

Although there are pharmaceutical approaches to treat dementia, these often have side effects, or their desired outcome is only temporary [26]. In addition, the development of new drugs is not only expensive but also time-consuming, as it needs to go through several scientific trials before being approved for human use [27]. As a result, the search for alternative methods, such as SGs, has greatly increased [28].

SGs for people with dementia have been developed as an assistive tool to promote physical, cognitive, and emotional stimulation, leading to a better quality of life [12,29]. Moreover, a novel SG can be used to assess cognitive decline at the early stages (or elevated risk) of dementia [30-32]. Investment in such computer apps can provide an opportunity to reduce institutional health care costs and enhance the quality of life of both family caregivers and people with dementia [33].

Many platform strategies have been considered to develop dementia-related SG apps. Some technologies rely on indirect interactions, such as (PCs) [34], or conventional entertainment systems, such as the Nintendo Wii system [35]. Other technologies are based on direct interaction, such as augmented reality (AR) [36], touchscreen technology [37-39], and gesture recognition systems, such as leap motion (LM), Kinect, and Bracelet Myo [40].

Indirect interaction technologies require an intermediate device to translate human action into interaction with the virtual environment. Indirect interaction devices use more cognitive resources, as they involve conscious spatial and mental translations to convert real-world movements into virtual actions [41].

Using direct interaction technologies, participants do not have an intermediary device to interact with the virtual environment; participants interact directly with the machines using their bodies [41]. In addition, direct interaction devices require less cognitive resources, as there is no movement translation between the real and virtual worlds as opposed to indirect interaction devices [41].

In terms of the development of SGs, previous studies revealed interesting insights regarding the development process of SG. For example, Brian Winn proposed the design, player, experience framework, which depicts the relationship between the designer and the player’s experience [42]. The framework is quite straightforward: the designer designs the game, which is played by the player according to the player’s experience. According to this framework, play is mediated by experience. Thus, the player’s experience (social, cultural, cognitive, and experimental background) influences the design of the game.

The study also considers the learning process in using technological devices to play games, as it can also influence users’ game experience. For example, in a recent study, Vallejo et al [43] evaluated the performance of elderly individuals on a set of technologies while performing 2 different tasks. The study concluded that interaction with technology is dependent on the task, the user’s experience, and motivation. In addition, the interaction with technology also depends on how intuitive both hardware and software interfaces are for people with dementia [28]; many high-tech technologies can overwhelm people with dementia on a cognitive level, which can affect the learning curve of handling technology [28,44].

Thus, to enhance the user experience for people with dementia while using novel technology, additional guidelines have been suggested to help developers in designing technologies while addressing the need for people with dementia, such as the responding, enabling, augmenting, failure-free (REAFF) framework [45,46]. The REAFF framework focuses on 4 principles: (1) responding (technologies should respond to the needs of people with dementia), (2) enabling (technologies should improve the quality of life of people with dementia), (3) augmenting (technologies should be able to adapt to the reserved skills of people with dementia), and (4) failure free (technologies should be as easy to use as possible without discouraging people with dementia).

Another framework—the virtual reality (VR)-Check framework [47]—has been proposed by Krohn et al [47], who evaluated clinical neuropsychology VR apps for cognitive domain specificity ( specifically, the cognitive domain being targeted by the VR app), ecological relevance (if the VR app focuses on activities of daily living), technical feasibility (if the VR app is compatible with the desired technologies), user feasibility (if the VR app is feasible to the target population), user motivation (if the VR app engages the users), task adaptability (if the VR app can be adjusted, for instance, in terms of difficulty), performance quantification (if the VR app can objectively quantify the participant’s performance), immersive capacities (how immersive is the VR app for participants), training feasibility (if the VR app is suitable to foster cognitive training), and predictable pitfalls (estimating resource-related costs when using the VR app).

Despite the existence of several efforts aimed at providing recommendations to develop SGs [48-50], there is still a lack of usability studies that aim to understand how people with dementia interact and accept different types of technologies to perform specific tasks [51,52]. Although elderly individuals are capable of learning and handling new technologies [53], using novel technology can lead to anxious behaviors among elderly populations [54] or lead to undesirable side effects, such as cybersickness [55] and fatigue [28].

To avoid such behaviors, during the prototype playtest phase, it is essential to record the feedback of each player while interacting with the game, as the experience of one player may differ significantly from the experience of another player [42]. In a recent study, Hackner et al [56] analyzed how people with dementia perform different interaction techniques in a tablet, such as a single tap, swipe, and drag-and-drop gestures. The study identified several interaction issues when performing such interaction techniques and presented different solutions to avoid future problems.

Considering the reported potential of SGs as a complementary approach to stimulate people with dementia, the main goal of this study was to better understand how people with dementia accept and interact with out-of-the-shelf technologies and how it influences users’ game experience while performing different activities. Moreover, this study aimed to find the most suitable technology to design a customizable interactive system that can exploit reminiscence and music therapy in people with dementia. We recruited 12 participants with dementia to perform several activities with different technologies to evaluate their performance while answering 6 research questions (RQs):

Following the results of our experiment, we (1) propose a set of guidelines that can help engineers and developers craft better-suited technologies for this population and (2) suggest additional setups of the technologies used to improve user experience in people with dementia.

We recruited 12 participants, 3 males and 9 females, with mean age, 75.08 (SD 8.07) years; mean Mini-Mental State Examination score, 17.33 (SD 5.79); and mean schooling, 5.55 (SD 3.30). This was a convenience sample, and the recruitment of the participants was performed by psychologists at the Madeiran delegation of the Portuguese Alzheimer’s Association (Table 1). Participants were eligible if they (1) could use upper limbs independently, (2) had an intact hearing, and (3) were in the initial or intermediate stages of dementia. For the last inclusion criteria, we relied on the clinical information available and did not perform any further assessments. The study was approved by the board of the association and followed the standard procedures for research with human participants. Before beginning the experimental trial, all participants (or legal guardians) signed an informed consent form, and permission was granted to film the sessions. After signing the consent form, participants were briefed about (1) the activity objectives and (2) how to handle the technologies. In addition, participants were informed that they could drop out of the experimental trial at any time.

We defined patients’ profiles based on their Mini-Mental State Examination (MMSE) [57] scores, age, and years of schooling. The MMSE scores were assessed before the participation of the experimental trial. Only 5 of the participants reported previous experience with technology. For example, participant 1 had experience using a tablet, whereas participants 11 and 12 had experience with PC. Participants 5 and 7 had experience in handling both PC and tablet.

For each of the following technologies, we selected generic tasks that required different types of interaction, such as (1) manipulating virtual objects, (2) playing musical instruments, (3) moving virtual objects from A to B, and (4) observation of virtual environments.

To run the tasks and technologies, we used a Toshiba Satellite L850-1HZ with Windows 10 64 bit equipped with an AMD Radeon HD 7670 and an Intel Core i7-3630QM with 4 GB RAM. Considering that some technologies require a considerable amount of processing power, a desktop computer running Windows 10 64 bits equipped with a Radeon RX 580 Series graphic card and an Intel Core 17-6700 CPU with 16 GB RAM were used. Five different interaction technologies were used in different combinations and tasks.

A within-subject experimental design was used to allow all participants to interact with all technologies and tasks. Each week, different technologies and tasks were randomly introduced. Participants were required to complete tasks, such as manipulating virtual objects, moving virtual objects from A to B, observation of virtual scenarios, and playing musical instruments. Participants were seated in a quiet room and accompanied by 2 researchers and a health professional when needed.

During the experiment, patients sat in front of a table in a silent room of the Madeiran delegation of the Portuguese Alzheimer’s Association (except when performing the task requiring a HMD with LM, which in this case required standing up). Two researchers were present in the room; one researcher was responsible for filming and taking notes on the participants’ performance, whereas the other researcher interacted with the participants during the experimental trial. The video recordings allowed us to analyze participants’ behaviors and study their verbal responses throughout the experiment. The camera was placed behind the shoulders of the participants to conceal their faces and protect their identity. In the case of participants in more intermediate stages of dementia, a health professional was also present to guarantee their well-being and aid researchers during the intervention.

During the experiment, 3 protocols were used: (1) before initiating the task, the researchers instructed participants on how to use a specific technology to complete a task; (2) each task had a maximum duration of 15 min, and participants could repeat tasks if desired; and (3) during the task, participants were encouraged to think aloud and could ask for help at any time. All interventions by the researchers were annotated for later analysis.

To address our RQs, we relied on direct observations and behavioral and verbal responses extracted from the video recordings. To analyze the video recordings, we used Adobe Premiere CC 2017.1.2 release version for coding. This video editing tool allowed us to tag, comment, and export annotations in comma-separated values (.CSV) files. The video analysis went through 2 phases. In the first phase, 2 researchers performed independent video analyses by tagging and annotating events in the video files. In the second phase, the information gathered by both researchers was compared and checked for consistency. In case of disagreement, a third researcher was invited to disambiguate.

To analyze participants’ user experience with a given technology or in each task, we counted the number of issues identified. The issues were grouped into (1) assistance provided by researchers, (2) perception issues, (3) comprehension issues, (4) interaction issues, and (5) discomfort that participants felt. These are described in detail below.

In addition, we studied emotional responses (positive and negative). That is, we counted the number of positive emotions (ie, laughter) and negative emotions (ie, frustration). We also counted the number of software issues (ie, undesirable features or bugs) that occurred during the experiment. For RQ5, we excluded software issues, as these were explicitly related to the actual software and not to the technology per se.

Finally, we calculated the number of times the equipment was exposed to external hazards—equipment at risk. For example, we counted the number of times the equipment was at risk of falling to the ground during user experience.

Data were analyzed using SPSS, version 24 (IBM Corp). For each dependent variable, the normality of the distribution was assessed using the one-sample Kolmogorov-Smirnov test. As most distributions deviated from normality, nonparametric statistical tests were used for the analysis. Descriptive results are presented as median and IQR. For assessing the impact of experimental conditions, the Friedman test was used. For post hoc pairwise comparisons, the Wilcoxon signed-rank test was used. The significance level was set at α=.05. Bonferroni correction was used to account for multiple comparisons. We also used Bonferroni correction for analyzing which combinations of technologies and tasks minimized feasibility performance (RQ4) and for analyzing which technology was exposed less to external hazards. For nonparametric correlations, we used Spearman rank correlation coefficient.

Participants 1, 2, 3, 7, 11, and 12 completed all 10 experimental conditions. Participants 9 and 10 withdrew from the experiment, and participants 4, 5, 6, and 8 were not able to complete all tasks. Consequently, only 9 datasets were considered for the playground activity with LM; 10 datasets were considered for condition LM (piano activity), tablet, and PC. For AR and observation (exploring the forest), 11 datasets were considered, whereas 7 were considered for condition observation (exploring the ghost ship), playing musical instruments, and manipulating virtual objects with both HTC with controllers and HTC with LM, respectively. In addition, some video recordings were corrupted, which did not allow us to computerize the number of issues; instead, we relied on the written notes taken during the experimental trial.

The technology that ranked best in terms of comprehension was the HMD, whereas technologies that scored worse were mouse and tablet. This is probably because of the simplicity of the interaction with HMDs—participants only need to move their heads to interact with the virtual environments. However, when using the mouse, participants showed great difficulties in understanding how to use it. Most of the difficulties were related to the mapping of the mouse, and sometimes, participants lost sight of the mouse cursor. Participants also had difficulties in interacting with the buttons, being distracted by the mouse wheel many times, as it is the most salient button of the device. Participants tended to rotate and click it instead of using actual mouse buttons. Some participants tried to rotate the mouse wheel forward and backward to move the mouse cursor up and down on the screen. Such behaviors even occurred in participants who had previous professional experience with it. For example, participant 12 had previous experience using the mouse, yet was unsuccessful. As a result, the participant cried, and the experiment had to be stopped. Thus, it becomes crucial to develop intuitive interfaces to avoid overwhelming participants in understanding how to use technology to complete virtual tasks [28].

In terms of interaction, the HMD again ranked the highest. Participants presented issues with the tablet’s interface, mostly because of the multi-touch control. When using multi-touch control, participants would tend to rest their hands on the tablet surface and trigger undesired functions that would prevent them from achieving their goal. Once more, an intuitive software interface is vital to enhance performance in people with dementia [28]. As our tablet was not fixed to the table, it also moved around as participants interacted with it. A better setup would have the tablet fixed to a surface, as in the study by Hackner and Lankes [56]. Despite these issues, the participants were able to perform the task gracefully.

Concerning discomfort, participants complained the most when using the LM and HMD with LM. For example, participants 1, 2, 10, and 11 reported fatigue while using LM. Indeed, to interact with the LM, participants’ arms need to be moving in the air, leading to muscle fatigue. In the case of the HMD with LM, only participant 5 did not report any discomfort. The remaining participants reported fatigue, cybersickness, and balance difficulties. Although the HMD alone did not trigger major issues, participants 6 and 3 felt nauseated, and participant 12 reported cybersickness after the virtual video task. Participant 6 complained about the heat generated by the headset. In general, cybersickness and fatigue are some of the negative aspects identified in the scientific literature in terms of the use of technology, whereas balance-related issues are associated with the negative consequences of dementia [15,17,28,55].

We found that the participants’ profile influences the usage of technology. A negative and significant correlation between MMSE scores and the number of assistances provided with AR and HMD with LM were found. In the case of AR technology, we found a significant effect on the level of schooling and the number of perception issues that arose in the experiment. We also saw that a low level of schooling and lack of experience with novel technology could lead to confusion (or even anxiety) [54]. For example, participant 3 was confused when instructed to move the red spheres that were projected on the table; as a result, the participant questioned: “How can I catch the spheres if they are fixed on a table?” Concerning the usage of the tablet, we found a significant correlation between MMSE scores and both comprehension and interaction issues. This is likely because of the multi-touch feature. Some participants failed to understand that by placing the whole hand on the screen of the tablet, multi-touch is triggered. Other issues that were identified included (1) activating the menu buttons of the tablet involuntarily, (2) dragging the tablet involuntarily while interacting with the virtual objects, (3) forgetting to wait for the selection time, and (4) forgetting the task rules. Finally, we found a positive and significant correlation between the MMSE scores and the number of perception issues when using LM. We observed that participants with high MMSE scores were able to interact with technology easily and for longer, which allowed researchers to identify perception issues during user experience, in contrast to participants with lower MMSE scores who struggled to begin a given task. Similar results were found in the study by Alvseike and Brønnick [44], which found that individuals with higher cognitive deficits had more difficulties in using smart house technology than individuals with lower cognitive deficits. Performance may also depend on other variables, such as motivation and experience [43].

Participants required more assistance statistically and had more difficulties in understanding how to use indirect interaction devices. Indirect interaction devices require more cognitive resources [41] and, in a population with cognitive deficits, may hinder performance during the completion of tasks. Conversely, direct interaction devices require less cognitive resources, and, consistent with our observations, participants had fewer complications in using such technologies as they are more intuitive and straightforward to interact with virtual content. Some participants, such as participants 1 and 11, were able to use both direct and indirect interaction technologies with minor problems. However, it is important to take into consideration that these participants had higher MMSE scores, and that participant 11 had experience in using mouse technology.

Participants, in general, did not show many emotional responses when using the studied technologies. However, some interesting reactions were observed. For example, participant 1 was very happy when she was able to grab a cube while using the HMD with LM and said, “Oh good...what a funny thing...it is so beautiful.” The same participant showed pride while playing the xylophone with the HMD with controllers and said that it was a shame that the people in the room could not hear her playing as it was a beautiful song. Participant 11 enjoyed exploring virtual environments with the HMD. She repeatedly said “very beautiful” in both the Exploring the Forest and Exploring the Ghost Ship tasks.

Here, participants used the LM and HMD with controllers to play virtual instruments and showed more perception issues while using the HMD with controllers than LM. Most of the issues identified were visual, auditory, and tactile related. For example, participant 12 complained that she did not hear the xylophones (yet, she confirmed during the experience that she heard the sounds). The same participant also reported that she was not able to see anything several times. In addition, participant 3 complained that she was not able to see or reach the musical instruments (despite being within the participant’s arm range).

In this task, the participants used the LM, HMD with LM, and HMD with controllers to manipulate virtual objects. We found differences in terms of software issues and equipment at risk. In general, the best technology is HMD with controllers. Although there were no statistically significant perception issues, participant 12 raised most visual-related problems, as she had difficulties in identifying the virtual objects in the virtual environments, including the digital representation of her hand. Participant 3 complained because she was expecting to “physically” grab the virtual objects. In terms of software issues, the HMD with controllers scored first place as it did present minor issues.

In contrast, LM technology scored the worst (last place). As participants tried to grab virtual objects, sometimes the objects stayed attached involuntarily to their hands, and they struggled to let go of the objects. Similar behaviors were recorded while participants performed the task while using the HMD with LM. Participants were able to grab virtual objects but had more difficulties dropping them. Finally, in terms of equipment at-risk situations, the HMD with LM triggered more dangerous situations for the equipment. For example, when participants were performing abrupt movements with the head, the HMD was sometimes at risk of falling.

We only found a statistical difference in terms of software issues, with AR and tablet being the ones that scored the worst. AR technology had some camera tracking issues because of environmental issues, such as shadows and reflections. In contrast, most of the issues related to the tablet were because of software bugs. Despite these minor issues, all technologies performed at an acceptable level.

In this task, we did not find any statistical differences. The only issues identified were related to cybersickness in both observation tasks [55].

In this study, we observed that technology had different outcomes in terms of acceptance and performance on people with dementia. Although technologies have been accepted by the majority, some participants had difficulties in managing them to fulfill the tasks. Such differences in the results are mainly because of patient profiles, which, in turn, influence technology configuration (direct interaction versus indirect interaction).

Comprehensibly, most of the technologies used were not aligned with the REAFF framework, as these were not explicitly designed to take into consideration the needs of people with dementia [45,46]. Most of the technologies used did not follow, for example, the augmenting or failure free principles, as participants did not complete the tasks independently. It is also important to consider how to align such technologies with the remaining principles of the REAFF framework for the needs of people with dementia (responding) and how technology can improve their everyday life (enabling).

In addition, it would be interesting to re-evaluate such technologies using similar tasks as presented in this study, but in a clinical context using the VR-Check framework [47]. Thus, more detailed knowledge could be gained regarding the adequacy and therapeutic outcome when using technology and virtual reality with people with dementia.

Although the technologies used are not perfectly aligned with the REAFF framework principles, they are accessible and can be used in their favor if they are set up correctly. By studying the use of the different technologies and tasks by people with dementia, we can provide a set of recommendations for the selection and implementation of different technological solutions when working with this population. Table 4 addresses the main problems encountered and provides recommendations to overcome them. These recommendations can help engineers in the design of technologies for people with dementia and draw attention to health professionals and informal caregivers regarding potential issues that can emerge while using such technologies with this population.

This study involved 12 participants with dementia who performed 5 different tasks using 5 interactive technologies that were available at the time of this study. As participants used the technologies to perform virtual reality tasks, we identified potential issues, such as assistance provided, comprehension issues, perception issues, interaction issues, and discomfort. We also studied how the patient’s profile would affect performance in those different tasks and technologies. Finally, we provided a set of recommendations for the selection, use, and design of virtual tasks for these technologies. Our main findings show significant effects of technology on performance regarding comprehension, interaction, and discomfort.

Overall, the participants were able to complete all tasks using all technologies. However, a clear outcome of the study is that there is no absolute best technology for people with dementia, but this is both task- and patient-profile–dependent. In general, the use of technologies that require direct interaction is advisable, given that cognitive performance gradually declines in people with dementia, as it relies on fewer cognitive resources than indirect interaction devices. We observed that cognitive skills, as assessed by the MMSE test, influenced participants’ perception, comprehension, and interaction and required more assistance. In addition, schooling is also a factor to be considered; the lack of experience and exposure to such technologies can lead to confusion and anxiety, which interferes with user experience.

A cost-effectiveness analysis comparing price and issues identified in all technologies suggests that the best tradeoff between performance and cost is achieved with the mouse, the most effective technology is HMD, and the most expensive one is AR. Through these insights, this study provides newer insights for health professionals, informal caregivers, and engineers regarding the use and design of novel technologies for people with dementia to (1) maximize their success in using such technologies to fulfill virtual tasks and (2) safeguard their psychological well-being. The findings of this study and the proposed guidelines are being implemented on a set of SGs for cognitive stimulation, which explores the potential benefits of music and reminiscence-related approaches in people with dementia.

To conclude, the participants in this study were able to handle the technologies to complete virtual tasks. Interestingly, the overall success in using technologies by people with dementia depends on different variables, such as patient profile, type of task, and interaction modality. Our study provides a quantitative analysis that contributes toward a better understanding of the complex relationships among these factors. Finally, by translating our findings into a set of guidelines, we hope to facilitate technological interventions and to enhance the user experience of people with dementia when performing virtual tasks with out-of-the-shelf technologies.

This study has some limitations. We had a small number of participants, and not all participants interacted with all technologies. Consequently, if we applied Bonferroni correction for multiple comparisons, statistical significances during post hoc analysis do not remain. Hence, a larger sample size would have provided higher statistical power for the analysis. In addition, having a control group of healthy age- and sex-matched participants would have been informative to discriminate age- and dementia-related issues, such as perception problems. Future studies should consider adding a control group to draw additional conclusions regarding the usage of out-of-the-shelf technologies to perform virtual reality tasks. Nevertheless, adding a control group presented some challenges.

First, we interacted with a population that cannot adequately express themselves in the same way as healthy elderly people. Therefore, we had to use very time-consuming methodologies, such as independent annotation of hours of video recordings, categorization, and extraction of data, so that a quantitative analysis could be performed. Consequently, we would have to use the same methodology as the control group (that would not require it), making it not feasible for us, given the time needed and available human resources. Second, even if we did so, our experience tells us that the 2 groups would not be directly comparable even if performing the same activities because people with dementia required constant stimulation and assistance by researchers and health professionals to understand and perform the tasks. Third, the level of autonomy of people with dementia in performing the activities is not comparable with that of a healthy old adult.

In addition, when performing the cost-effectiveness analysis, we considered different approaches, such as the normalization of costs and issues. However, we realized that the resulting values obscured the actual relation to either cost or actual issues, making it very difficult to interpret. Moreover, we considered performing an issues per euro analysis; however, such an approach was also problematic because the metric favored expensive equipment. That is, the more expensive the equipment, the less the issues and cost ratio. Similarly, very cheap equipment, such as the mouse, always presents a very high (comparatively) issue/cost ratio. Therefore, in the cost-effectiveness analysis, we gave the same weight to issues and cost because (1) it is fairer to compare and (2) easier to interpret.

Moreover, the use of assessment tools should be considered for additional qualitative data analysis, such as the Individually Prioritized Problem Assessment and the Psychosocial Impact of Assistive Devices Scale, to evaluate how technology impacts the daily life of people with dementia. Finally, some video recordings were corrupted, and, although we also used written notes, some level of detail may have been lost.