The number of people living with dementia has been rising globally from 20.2 million in 1990 to more than 50 million in 2019 and is estimated to exceed 150 million in 2050 [1,2]. The rising prevalence leads to increasing numbers of patients with dementia who receive inpatient treatment or live in inpatient care facilities. It is important to provide suitable interventions for these patients. Possible intervention targets include improved quality of life, delayed cognitive decline, and successful treatment of comorbid disorders such as depression [3]. Given the broad range of dementia types and courses, these targets are not easy to address [4-7]. To satisfy patient needs, interventions are required to account for the various types of impairments and allow for individual tailoring. Accordingly, the Alzheimer Society of Canada [8] has put forward a person-centered approach as the best practice for dementia care, and dementia-friendly environments are receiving increasing attention [9,10]. In fact, nonpharmaceutical treatment (NPT) has become the recommended treatment for the behavioral and psychological symptoms of dementia [11]. NPT covers a wide range of approaches, including exercise and motor rehabilitation, cognitive rehabilitation, and psychological therapy, as well as information technology and assistive technology [12].

Unfortunately, the need for a person-centered and individually tailored approach puts a high burden on health care systems and private caregivers. Particularly in later disease stages, dementia causes larger health care challenges than other diseases [13]. This often conflicts with the already high costs, the lack of staff, and the short duration of inpatient treatment [14-16]. A possible approach to address these difficulties could be the use of serious games (SGs).

SGs have received increasing attention as a means to apply NPT in dementia treatment [17,18]. SGs are defined as games that are designed to not only entertain but also enhance learning, for example, in the area of psychoeducation or cognitive training [19]. There is a small but growing body of SGs being specifically targeted at people with dementia [20-26]. To accommodate the differential needs of this target group, different author groups propose development guidelines for SGs [27,28]. However, the number of studies involving SGs that meet these guidelines [20,29] or engage in systematic usability testing of SGs [30] is limited. Additionally, more severely impaired target groups are seldom thought of as potential SG users [18]. While unsupervised SG usage seems inherently less likely for people with severe dementia, high user satisfaction (ie, positive feelings toward SG usage) has been found to be independent of cognitive impairment under the premise of adequate app design [25,31]. Given their potential benefits as a delivery method for NPT, we argue that it is a valuable aim to develop SGs specifically for people with dementia, with consideration of severely impaired patients. Since the development of medical apps often tends to pay not enough attention to evidence-based standards [32], further studies are needed to establish utility estimates of SGs for patients with dementia to enhance clinical credibility [12].

This paper presents a pilot study of the usability of a newly developed SG (“Digimenz”) for older patients, promoting participative development. Usability in the context of dementia care has previously been operationalized by the ISO 9241 model [30,33]. It defines usability in terms of effectiveness, efficiency, and user experience. The ability to learn the handling of software is also a common part of different usability definitions [34]. While it is missing in ISO 9241, it appears to be a crucial factor for patients with dementia who have limited learning capacities by definition. Therefore, we have followed Abran et al [35] who propose to augment ISO 9241 by the concept of learnability and use this as an integrative model of usability (Table 1).

The purpose of this study was to generate a feedback loop for the iterative and evidence-based development of an SG app by providing scientific testing and making the results available for further development of this and other SG apps. This will add to a diverse, person-centered, and resource-saving body of intervention options for patients with dementia.

To ensure that the full spectrum of cognitive impairment is covered when examining the usability of the SG app, we recruited participants from two settings: (1) a long-term care residential facility (where people live with, on average, a lesser degree of cognitive impairment) and (2) a geriatric psychiatric inpatient ward (where the degree of deficits is expected to be more severe). We examined whether the Digimenz app is usable across different degrees of impairment while controlling for the effects of the recruitment site.

We were particularly interested in whether the SG shows sufficient usability in terms of global usability, effectiveness, efficiency, user experience, and learnability. We hypothesized that less impaired patients would report higher overall usability of the SG app and show higher effectiveness, efficiency, and learnability, but would have an equally satisfying user experience as patients with a higher degree of impairment. Moreover, usability parameters are expected to be similar in the psychiatric context and in a nursing home.

Following the adapted CONSORT (Consolidated Standards of Reporting Trials) statement for pilot trials [36], the sample size was adapted to the rationale of usability testing. To our knowledge, there is no evidence regarding the effect size for the comparison of the usability parameters of SGs among users with different levels of cognitive impairment. We therefore chose a sample size comparable to that in other usability trials involving people with cognitive impairment [30] in order to allow reliable conclusions about SG usability and inform future large-scale intervention studies. The final sample included 43 participants.

Subsample 1 (n=28) was recruited from a nursing home for older adults specialized in the management of cognitive impairment. Residents are referred to this kind of long-term care facility when care in their original homes is no longer adequate or possible. Subsample 2 (n=15) was recruited from 2 geriatric-psychiatric wards specialized in dementia management in a psychiatric hospital. Patients are referred to this kind of psychiatric ward in case of acute exacerbation of mental impairments or comorbid mental disorders.

The inclusion criterion for all participants was documented cognitive deficits (mild cognitive impairment [MCI] or dementia) diagnosed at admission to the respective institution by a specialist in the field according to the criteria by McKhann et al [37] and Albert et al [38]. Further, participants were included only if they had sufficient German language proficiency. Since we wanted to represent a broad range of cognitive impairments, we tuned the inclusion criteria for sensitivity and not specificity. Therefore, a specific etiology of cognitive impairment was not relevant for inclusion. Moreover, we did not define age limits, numerical cutoffs for cognitive tests, or comorbid mental disorders as exclusion criteria.

The study protocol complies with the Declaration of Helsinki and received ethical approval from the Bielefeld University Ethics Review Board (EUB-2022‐265 s). Potential participants were approached to examine their explicit preferences regarding study participation. The purpose and methodology of the study were explained in simple terms to be accessible for those who were nominally more severely cognitively impaired. If there was any sign of rejection, no further recruitment was attempted. If a potential participant expressed interest in participation, informed consent was acquired. To this end, an information sheet and consent form were read and signed by the participant, if mental capacity allowed it. Otherwise, informed consent was obtained from a legal representative or relative. Study data were deidentified and saved on the secure servers of our institution. To guarantee the anonymity of participating patients, participant-related data will not be made available. No compensation was provided.

In this prospective study, we used a repeated-measures design. Each participant was asked to complete 4 conditions in randomized order. Three of these conditions involved playing 3 different games from the SG app (“Digimenz”). The fourth condition included reading a newspaper serving as a naturalistic control condition. All conditions consisted of 2 runs each (5 min), with one supported by the conducting researcher and the other performed with minimal support. Prior to the experiment and after each individual condition, the participants answered a set of questions and the conducting researcher gave external ratings. Additional unstructured qualitative feedback was registered. Global cognitive functioning was assessed in a separate session. The total duration for each participant was about 60‐90 minutes. To minimize researcher bias while retaining a high data quality, data collection was performed by 2 research assistants who were instructed and supervised by the principal investigators (MT, EMT, and SK), who have extensive experience in dementia research. The study has been registered in the German Clinical Trials Register (DRKS00031363).

To test the applicability of the SGs, we recorded reasons for nonparticipation in the recruitment phase and dropout thereafter. To test usability and its relation to cognitive functioning, as well as differences between the subsamples, we operationalized global usability and the individual usability constructs as follows:

Efficiency, user experience, and learnability were compared between the SGs and the control condition (newspaper). Sufficient usability in the different settings was assumed in the case of noninferiority of these parameters when comparing the SGs to the control condition. A “good” global usability rating (SUS scores >74) was the second indicator of sufficient usability. Error rate cannot be used in this way since it is not defined for the control condition. In addition to these quantitative measures of usability, we registered any additional feedback on usability issues given by the participants during SG usage and while rating the scales on usability. Global cognitive functioning was assessed with the Mini-Mental State Examination (MMSE) [43]. Demographic information (age and sex) and any existing diagnoses were registered from the medical record.

We present descriptive statistics of the rates of nonuse and dropout as operationalization of the applicability of the SGs. We modeled the usability measures according to the conditions and the degree of cognitive impairment (MMSE) and their interaction, which we hypothesized to be the most relevant predictors of usability. We controlled for the effects of the recruitment site (subsample), given the differences between the 2 sites. We used mixed effect models (linear mixed models for continuous outcome variables and cumulative link mixed models [CLMMs] for ordinal outcome variables) with random intercepts to account for the repeated-measures design. These models have the advantage of using all available data at each measurement, precluding the need for listwise deletion (ie, a person does not have to be deleted if missing 1 measurement). Nagelkerke pseudo R² is reported as an effect size measure by comparing the complete model to the model without the respective predictor and its interaction terms. To quantify differences between conditions, we computed Hedges g using the within-subject variance [44] and the adjustment for small samples: g=Mcondition a−Mcondition bSDdiff∗2(1−rcondition a, condition b∗(1−34n−5) [45]. Values of g >0.2 may be interpreted as small, g>0.5 as medium, and g>0.8 as large effect sizes [44]. All analyses were performed in R version 4.4.1 (R Foundation for Statistical Computing). Linear mixed models were fitted with the lme4 package [46], and CLMMs were fitted with the ordinal package [47]. The significance level was set to α=.05 for all inferential tests.

Qualitative feedback data on usability issues were subjected to a thematic content analysis following Kuckartz and Rädiker [48]. This is a multistep, iterative procedure. The steps are as follows:

Steps 4 and 5 were repeated several times to fine-tune the categories.

In subsample 1 (nursing home), of 30 eligible individuals, 30 (100%) consented to participate and 28 (93%) completed the study procedure. In subsample 2 (psychiatric ward), of 41 patients eligible for participation, 21 (51%) consented to participate and 15 (37%) completed the study procedure. Table 2 details the characteristics of both samples. Figure 2 provides an overview of the reasons for nonparticipation and dropout. In subsample 1, all people approached were interested in the SGs, and the reasons for dropout were not specific to the games themselves. In subsample 2, 10 people (24%) denied participation, which was specifically related to SG usage (7 [17%] were disinterested, and 3 [7%] were frustrated by the SGs), while 31 people (76%) were generally interested in the SGs. The odds ratio for dropout between informed consent and study completion was 5.6 (P=.05), that is, the odds of dropping out of the study were 5.6 times higher in the psychiatric ward subsample than in the nursing home subsample.

To compute the total SUS score, missing values were replaced by the scale mean, as recommended by Lewis [49]. In the psychiatric subsample, the SUS total score was 71.5 (SD 24.33), while in the nursing home subsample, the SUS total score was 85.10 (SD 12.67). Less cognitive impairment was associated with better usability ratings in both subsamples, with no significant difference in the correlation coefficients between the subsamples (total sample: Spearman ρ=0.31; P=.03). The results for the individual usability dimensions as defined in Table 1 are presented for the total sample in Table 3. Differences between the subsamples were tested in the following multilevel analyses.

No significant effects of MMSE (F_1,36.50=0.10; P=.75; DR²=0.09) and subsample (F_1,36.45=0.16; P=.69; DR²<0.001) on error rate were found in the multilevel analysis. Among the SGs, there were significant differences in error rates (F_2,74.28=9.43; P<.001; DR²=0.31; Object Recognition game: −0.10; P=.09; Spelling game: +0.15; P=.01; see Table 3 for effect sizes). The interaction of MMSE and condition was not significant (F_2,74.02=0.13; P=.15; DR²=0.09). The Nagelkerke R² for the complete model compared to the null model was R²=0.37. Regarding the difficulty adjustment, Table 4 shows a descriptive analysis of the difficulty levels. Overall, the highest (ie, most difficult) level was seldom reached, and participants with a higher MMSE reached higher levels of difficulty.

Participants with a higher MMSE needed significantly less support than those with a lower MMSE (Table 5; Figure 3). The correlation analysis presented in Table 3 suggests that this effect was of medium size (ρ_{MMSE, Amount Support}=−0.49). For the Spelling game, significantly more support was needed than for the newspaper condition (+1.27; P=.02). This was not the case for the Drag&Drop game (+0.17; P=.77) and the Object Recognition game (−0.09; P=.88) (see Table 3 for effect sizes). No significant differences between the subsamples were found in the multilevel analysis. The Nagelkerke R² for the complete model compared to the null model was R²=0.16.

The multilevel analysis showed that self-reported mood was higher in all 3 game conditions than in the newspaper condition (Drag&Drop: +2.55; P<.001; Object Recognition: +2.17; P<.001; Spelling: +1.56; P=.004; see Table 3 for effect sizes), when correcting for baseline mood (Table 6). The MMSE was not significantly correlated with self-reported mood (Table 6; Figure 4). There were no significant differences between the 2 subsamples. The Nagelkerke R² for the complete model compared to the null model was R²=0.24. The external rating of mood showed a similar pattern with higher mood ratings in all 3 game conditions compared to the newspaper condition (Drag&Drop: +1.65; P<.001; Object Recognition: +1.28; P=.007; Spelling: +1.08; P=.02; see Table 3 for effect sizes). The overall correlation of self-reported and externally rated mood was r=0.64 (95% CI 0.43-0.79; P<.001).

According to the multilevel analysis (Table 7), self-rated difficulty was independent of the MMSE (estimate=−0.06; P=.54). Compared to the newspaper condition, it was significantly lower for the Drag&Drop game (−2.01; P=.002) and Object Recognition game (−1.76; P=.005), but not for the Spelling game (−0.63; P=.54; see Table 3 for effect sizes). The 2 subsamples did not differ significantly in their difficulty ratings. The Nagelkerke R² for the complete model compared to the null model was R²=0.14. The external rating of difficulty, in contrast, showed a strong negative association with MMSE (estimate=−0.22; P<.001) (ie, higher MMSE scores were associated with lower external ratings of difficulty). The overall correlation of self-reported and externally rated difficulty was ρ=0.52 (95% CI 0.25-0.72; P<.001).

Our second indicator of learnability was the proportion of participants needing support in the full-support run versus the minimal-support run (Multimedia Appendix 1).

In the full-support run, more participants needed support in the 3 game conditions than the newspaper condition (Drag&Drop: +0.28; P<.001; Object Recognition: +0.19; P=.02; Spelling: +0.41; P<.001; see Table 3 for effect sizes; F_3,275.18=10.93; P<.001; DR²=0.14). From the full-support run to the minimal-support run, the rate of support decreased significantly (F_1,274.92=9.86; P=.002; DR²=0.05). Accordingly, there were no significant differences in support needed between the game conditions and the newspaper condition in the second run (Drag&Drop: −0.20; P=.09; Object Recognition: −0.11; P=.34; Spelling: −0.16; P=.17; see Table 3 for effect sizes). The interaction effect of run and condition was not significant (F_3,274.9=1.09; P=.36; DR²=0.02). Participants with a higher MMSE score were less likely to need support at all (F_1,37.89=16.76; P<.001; DR²=0.07). There were no significant differences between the 2 subsamples (F_1,37.88=0.004; P=.95; DR² <0.001). The Nagelkerke R² for the complete model compared to the null model was R²=0.25.

The results of the thematic content analysis [48] of usability issues mentioned during SG usage and data collection are presented in Table 8. We chose to categorize usability issues according to the source they were attributed to. Besides usability issues specific to the SGs, participants also mentioned problems with the handling of the tablet and personal impairment that hindered tablet or SG usage.

This study aimed to evaluate the usability of SGs for older adults with cognitive impairment, ranging from MCI to mild and severe dementia. We hypothesized that the SGs would be applicable for a nursing home as well as a psychiatric ward, that the usability would be sufficient, and that it would be differentially influenced by the level of cognitive impairment.

Most people approached were open to testing the SGs; however, a considerably smaller proportion managed to complete the study procedure in the psychiatric ward subsample, given greater impairment there. The global usability was rated good (SUS), and the indicators for individual usability dimensions showed that using the SGs was possible with a moderate error rate and support. Moreover, the support rate decreased from the first to the second usage, implying a learning process, and the gaming experience was associated with positive mood. While the amount of support needed and the ability to learn the handling of the games were related to cognitive impairment, the error rate and mood state after playing were not.

It is a strength of this study that we chose theory-based operationalization of usability, enabling us to draw differentiated conclusions regarding the usability of the SGs and the relation to cognitive functioning. By using different data sources (external and internal ratings, qualitative data, and usage data from the SGs), we can provide comprehensive insights into the usage behavior and user experience. The diverse nature of our sample in terms of cognitive impairment and care settings allows us to set our results in the context of previous studies and generalize our conclusions to more severely impaired people who are often excluded from research.

The inclusive design comes with the drawback of limited formal data quality and the necessity to limit this pilot study to a more constrained procedure. First, we did not include repeated gaming sessions. Therefore, we cannot provide information on the potential long-term effects of the SGs. Second, self-reported data may be inaccurate given limited introspective abilities and difficulties understanding questions, potentially reducing the reliability of the results. We attempted to overcome this limitation by adding observational data and usage data. However, there is some potential for researcher bias in observational data since the assessment of state variables (eg, mood) in patients with dementia generally lacks reliability [51]. We took measures to ensure high-quality observational data by providing scoring guidelines and supervising the ratings done by investigators. Third, the number of study completers in the psychiatric ward was rather small in absolute terms, although it was average for this highly vulnerable population. Therefore, inferential tests must be interpreted with caution. Fourth, the control condition differed from the experimental conditions in more than one way (medium and content). For future research endeavors, it would be preferable to add analog versions of the SGs or a digital version of the newspaper to isolate the effects related to these attributes.

Based on the promising findings from this pilot trial, it would be worthwhile to pursue a more extensive study design with a larger sample size, more recruitment sites, and repeated or longer gaming sessions, as well as repeated assessments over a longer period of time. Future studies should systematically register potential confounding factors that may influence usability at baseline (ie, sensory and motor impairment, and familiarity with technology) and incorporate the perspectives of caregivers and clinical staff. Future studies in the field would also benefit from establishing standards for quantifying the difficulty of SGs to allow the comparison of performance parameters within and across studies. This would help to evaluate the exact support requirements adapted to different profiles of impairment. Studying “people with dementia” without further specification is not sufficient for this aim, as has already been noted by Eichhorn et al [28].

From the quantitative results presented herein, we can conclude that users with a wide variety of cognitive impairments were able to adequately use the SGs under the premise of adequate support. They reported good global usability, reported positive mood after playing, and showed a moderate error rate. Since a decrease in support could be seen from the first to the second usage of each SG, we can conclude that the games were sufficiently learnable.

The qualitative results underlined the importance of a user-centered design, considering cognitive as well as somatosensory impairments. In this regard, the adaptive difficulty levels represented an important step toward personalized care. Less impaired patients reached higher levels, contributing to the broad applicability of the SGs. Nevertheless, additional levels at both the low and high ends of difficulty would have been worthwhile. No significant differences in usability parameters were found between recruitment settings. SGs may therefore be suitable for not only people with MCI or MD but also those with severe cognitive impairment who are treated in an inpatient psychiatric setting. However, this requires overcoming practical restraints like quick patient turnover.