Executive functions (EFs) arise from the interaction of multiple cognitive domains and are defined as the cognitive abilities necessary for goal setting, efficient execution of actions, and flexible modification of behavior to adapt to changing contexts [1,2]. Executive dysfunction (ED), a disruption of these abilities, is a frequent consequence of brain-related disorders, including traumatic brain injury (TBI), dementia, and schizophrenia and, more recently, has been recognized as a sequel of COVID-19 [3]. ED is a major contributor to psychosocial disability in these patients, affecting their ability to perform cognitively demanding activities such as employment and social participation [4-6]. This loss of functional independence has been directly linked to increased health care costs, with a linear relationship observed between the severity of ED and the economic burden associated with these disorders [7,8].

Early recognition of ED is essential to identify patients’ difficulties and facilitate timely access to social and health services. However, current assessment methods demonstrate limited ecological validity and low sensitivity, often failing to detect executive difficulties that manifest in daily life [9], limiting timely access to necessary health care services and psychosocial support. Although a wide range of traditional methods exist, including paper-and-pencil tests and naturalistic observations, expert clinicians frequently identify ED in patients who perform adequately on standard evaluations [10,11]. There is consensus that traditional clinical assessments have lower sensitivity compared to expert clinical judgment [12]. In fact, standard evaluations fail to detect ED in up to 58% of individuals with TBI [13], likely due to their inability to reflect real-world cognitive demands. This detection gap has been attributed to several factors: (1) conventional tests often do not assess critical domains of executive functioning necessary for everyday life, such as contextual adaptation integrating cognitive and emotional information, prospective memory, and multitasking within social contexts; (2) they rely on highly structured tasks; and (3) they measure performance through a limited number of discrete or categorical variables (eg, correct responses, errors, and overall task completion time) [12,14,15].

Information and communication technologies have enabled the development of novel, computer-based tools—notably serious games (SGs)—to enhance the sensitivity and ecological validity of ED assessments. SGs are interactive digital applications designed with purposes beyond entertainment, including assessment, education, or rehabilitation in health contexts [16,17]. SG-based evaluations facilitate the assessment of EF within simulated real-life contexts through unstructured tasks, allowing the inclusion of EF domains that are not typically evaluated [14]. These tools are easy to implement in clinical settings, require only standard, low-cost technology (eg, computers or tablets), are simple for health care personnel to administer, are well tolerated by patients, and are applicable across a broad spectrum of brain disorders. Recent studies have increasingly explored the use of SG for cognitive assessment and rehabilitation, highlighting their potential to enhance ecological validity and user engagement [14,18,19]. However, much of this research has been conducted in nonclinical populations or small samples [20,21]. Although SGs have been proposed as a way to enhance ecological validity by simulating real-world contexts, it is important to keep in consideration that it may also increase cognitive load, which can influence user performance and experience [22,23].

Additionally, performance can be assessed through multiple indicators, including continuous measures that are often overlooked by conventional neuropsychological tests. Furthermore, advanced analytical approaches, such as machine learning models, provide a powerful framework for extracting clinically meaningful insights from these complex, multidimensional data [24].

Beyond diagnostic sensitivity, the clinical applicability of SGs also depends on their usability, particularly for individuals with cognitive impairments. Usability can be defined as the degree to which a product can be used effectively, efficiently, and satisfactorily by specific users in a particular context [16,25]. In neuropsychological applications, this includes learnability, ease of navigation, user control, and satisfaction with game features. Poor usability may overload cognitive resources, hinder learning, and reduce motivation to engage with the task [26-28]. Although usability studies have been conducted for SGs [20,25,29,30], most have involved small samples or participants who do not represent the intended clinical population [21,31,32]. Given that SG users may belong to nongaming populations with heterogeneous cognitive profiles, it is critical to assess usability in the specific clinical groups for whom these tools are designed [30]. Although some studies have examined usability aspects, these have often been limited to specific populations or have not incorporated complex, socially and emotionally embedded scenarios that reflect real-world executive demands [21,33]. As a result, there remains a need for clinically oriented SG that combines ecological validity with systematic usability evaluation in populations affected by ED.

The objective of this study was to introduce a new SG developed to assess EFs in adults. The aims of this study were (1) to describe the development of a clinically oriented SG designed to assess ED in ecologically valid contexts and (2) to evaluate its usability in clinical and healthy populations using standardized measures of usability and perceived difficulty.

Our team developed a novel computer-based tool for the assessment of EF through an SG. The development process followed an iterative, theory-driven design, incorporating input from expert clinicians and cognitive scientists. Expert judges, comprising national and international neuroscientists, neurologists, and neuropsychologists, participated in a structured validation process. They contributed to the selection of the EF domains to be assessed, the design of the SG’s central metaphor, a restaurant setting in which the player assumes the role of a waiter, and the specification of the tasks designed to evaluate core executive processes. This “waiter scenario” was intentionally chosen to simulate cognitively demanding situations grounded in everyday life, particularly those requiring rapid decision-making, emotional regulation, and cognitive flexibility—capacities often compromised in individuals with ED. Working as a waiter involves managing multiple concurrent events (eg, taking orders, responding to customer feedback, and coordinating with kitchen staff), all within a socially and emotionally dynamic environment. These conditions create an ideal context for embedding emotionally salient and contextually rich cues that reflect the complexity of real-life cognitive demands. Special attention was given to enhancing ecological validity and contextual realism. For example, the game’s graphic design emphasized clearly identifiable facial expressions to support emotional recognition and contextual control during gameplay. Tasks integrate context-dependent problem-solving, emotionally charged stimuli, and simultaneous goal management, components that, according to recent theoretical frameworks [14], are essential for assessing ED in ecologically valid settings.

The SG was designed to operationalize 9 key EF domains, selected based on expert consensus and their relevance to real-life functioning. Table 1 presents each cognitive domain, its conceptual definition, and how it is operationalized within the gameplay. The in-game tasks were crafted to embed these domains in naturalistic interactions, ensuring ecological validity by incorporating emotionally salient cues, simultaneous task demands, and context-rich decision-making scenarios. This structure allows for the assessment of ED as it emerges in realistic, socially embedded environments, rather than through artificial, isolated tasks.

To empirically evaluate the alignment between expert ratings and the SG’s proposed indicators of ED, the Cohen κ coefficient [34-36] was used. This statistic is suitable for assessing interrater agreement in categorical or ordinal data across multiple evaluators. Content validity was supported by high levels of expert agreement, with at least 60%—and in most cases over 80%—of experts rating each proposed domain and corresponding task metric as appropriate for assessing ED.

Following the review of these domains, experts also evaluated a set of 6 character illustrations designed to portray basic emotional expressions (sadness, disgust, anger, happiness, neutral, and surprise). These visual stimuli were used to assess emotional recognition and the player’s capacity to adapt behavior according to social signals, a core component of contextual control and social cognition. Special care was taken to ensure that facial expressions were visually distinct and ecologically valid, providing reliable cues for real-time emotional appraisal and decision-making. Figure 1 depicts the examples of customer characters expressing anger and happiness.

Demographic variables, including age, sex, and years of education, were recorded and included as covariates in the statistical analyses. All participants were evaluated by a clinical neuropsychologist who collected qualitative clinical data.

Neuropsychological measures included assessments of general cognitive functioning using the Montreal Cognitive Assessment [39]; attention and working memory using the Digit Span test (forward and backward) [40] and the d2-R Test of attention [41]; processing speed using the Trail Making Test Part A; and executive functioning using the Trail Making Test Part B [42], Frontal Assessment Battery [38], Tower of London Test [43], and the Dysexecutive Questionnaire [37] administrated to the participant and a reliable proxy. Finally, daily functioning abilities were evaluated using the Technology–Activities of Daily Living Questionnaire [44], administered both to the participant and the proxy.

The primary outcomes of interest were usability and perceived difficulty, assessed using the System Usability Scale (SUS) [45] and the Perceived Difficulty Scale (DP13-CL) [46], respectively.

Clinical consensus regarding ED was established through multidisciplinary case discussions involving neurologists and neuropsychologists, after the completion of each participant’s neuropsychological evaluation. ED was operationally defined based on a combination of standardized neuropsychological test performance and expert clinical consensus, which served as the reference standard for group classification.

Participants were classified into 3 categories: no ED, mild ED, or moderate to severe ED, based on the integration of standardized test performance, deviations from normative values, and functional impact. In cases of disagreement, consensus was reached through discussion among clinicians. For the purposes of the present analyses, participants classified as mild or moderate to severe were grouped as presenting ED.

Importantly, ED classification was independent of the clinical group, such that participants within the TBI and COVID-19 groups could be classified as with ED (27 participants with TBI and 32 participants with COVID-19) or without ED (5 participants with TBI and 16 with COVID-19) based on the consensus procedure.

It is worth noting that 12 out of 92 participants within the clinical groups (11 with TBI and 1 with COVID-19), representing approximately 13% of the clinical sample, could not be classified regarding the presence or absence of ED due to missing clinical or informant data required for consensus-based classification. Consequently, these participants were excluded from analyses that required ED classification. No imputation procedures were applied, as the missing data affected only the classification variable and not the primary usability outcomes.

The study sample size (N=181) was determined based on feasibility considerations, including participant availability within the recruitment period and resources for comprehensive neuropsychological assessment. While no formal a priori sample size calculation was conducted given the primary focus on usability evaluation rather than hypothesis testing, a post hoc power analysis confirmed the robustness of our comparative findings. Specifically, for our main contrast regarding perceived usability between clinical patients with ED (n=59) and HC (n=89), the analysis indicated an adequate effect size (Cohen d=0.588), yielding a statistical power of 93.5% (1–β=.935) to detect meaningful differences in SUS scores at an α level of .05. Furthermore, to address estimation precision, 95% CIs were computed and reported for all key usability metrics and point estimates.

Descriptive statistics were computed for all variables. Group differences in continuous variables were assessed using nonparametric tests (Kruskal-Wallis and Mann-Whitney tests), while categorical variables were analyzed using chi-square tests. To indicate the precision of our estimates, 95% CIs were calculated for the main group comparisons. Sociodemographic variables were used as covariates when corresponding.

To control for potential confounders, linear models were used to analyze SUS scores, including group and sex as fixed factors and age and education as covariates. Ordinal logistic regression models were applied to analyze perceived difficulty (DP13-CL), using the same covariates.

All statistical analyses were conducted using Jamovi software (The Jamovi Project), with statistical significance set at P<.05.

This study was approved by the Ethics Committee of Hospital Clínico de la Universidad de Chile (approval number OAIC 1300/22), covering research activities conducted at Hospital Clínico de la Universidad de Chile and Hospital del Salvador, and by the Ethics Committee of Mutual de Seguridad (approval number 20CV-140571). The study was conducted in accordance with the ethical principles of the Declaration of Helsinki and applicable national regulations governing research involving human participants. Participants were recruited and assessed at Hospital Clínico de la Universidad de Chile, Hospital del Salvador, and Mutual de Seguridad. Written informed consent was obtained from all participants prior to study participation. No financial compensation was provided. However, participants were offered a light refreshment during the assessment session and received individualized feedback on their neuropsychological assessment results.

All data were anonymized prior to analysis to ensure participant confidentiality, using coded identifiers. Data were securely stored using REDCap electronic data capture tools hosted at the Faculty of Medicine, Universidad de Chile [47], and were accessible only to authorized members of the research team. No identifiable participant data or images are included in this paper or supplementary materials.

A total of 212 participants were assessed for eligibility in the usability study, of whom 181 were included in the final analyses. The final sample consisted of 43 participants with TBI, 49 participants with COVID-19, and 89 HCs. Table 3 presents the demographic characteristics of the sample, including age, years of education, sex distribution, and the mean scores on the neuropsychological tests and questionnaires administered during the assessment.

The analysis of total scores on the SUS and DP13-CL is presented in Table 3. The item-level breakdown of the SUS is shown in Figure 7. According to the classification proposed by Bangor et al [48], SUS scores indicated good usability across all groups. A Kruskal-Wallis test revealed significant differences between groups (χ²₂=9.18, P=.01). Post hoc pairwise comparisons showed that the TBI group rated usability significantly lower than the HC group (P=.02). No significant differences were found between the TBI and COVID groups (P=.93) or between the COVID and HC groups (P=.06).

For the DP13-CL, the overall mean score across all participants was 5.61 (SD 2.36), corresponding to an average difficulty rating between “easy” and “neither easy nor difficult.” Reported scores ranged from 1 (“extremely easy”) to 12 (“very difficult” to “extremely difficult”), with a total sample size of 181. Group-level analysis revealed that the COVID group reported the highest perceived difficulty (mean 6.43, SD 2.29), followed by the TBI group (mean 5.79, SD 2.61) and the HC group (mean 5.08, SD 2.14). A Kruskal-Wallis test indicated significant differences among the 3 groups (χ²₂=11.6, P=.003). Post hoc pairwise comparisons showed a significant difference between the COVID and HC groups (P=.001), while no significant differences were observed between the TBI and COVID groups (P=.63) or between the TBI and HC groups (P=.23).

When analyses were restricted to participants with ED from the COVID and TBI groups (n=59), they exhibited significantly lower SUS scores (mean 73.1, SD 16.2; P<.001) and higher perceived difficulty (mean 6.34, SD 2.32), compared to HCs (SUS: mean 81.7, SD 13.5; DP13-CL: mean 5.08, SD 2.14; Table 4).

Given the observed group differences in age and education, additional analyses were conducted to control for these demographic variables. A general linear model was used to analyze SUS scores, with group and sex as fixed factors and age and years of education entered as covariates. The model was significant (F_7,171=4.32, P<.001, R²=0.150). However, no significant main effects of group (P=.51) or sex (P=.86) were observed, and the group × sex interaction was not significant (P=.28). Notably, age was negatively associated with SUS scores (β=–.272, P=.003), while education was positively associated (β=.794, P=.04). These findings suggest that the initial group differences observed in unadjusted analyses were largely explained by individual differences in age and education.

To explore whether the same pattern was held for perceived difficulty (DP13-CL), an ordinal logistic regression (proportional odds model) was conducted using group and sex as fixed factors, with age and education as covariates. The model was significant (χ²₇=34.4, P<.001, R²=0.044). As in the SUS analysis, group (P=.08), sex (P=.59), and the group × sex interaction (P=.31) were not significant. However, age was positively associated with perceived difficulty (odds ratio 1.04, 95% CI 1.02-1.07, P<.001), indicating that older participants reported higher difficulty. Years of education showed a trend-level negative association with perceived difficulty (odds ratio 0.91, 95% CI .83-1.00, P=.05), suggesting that higher educational attainment was associated with lower perceived difficulty.

The present study had two aims: (1) to describe the development of a clinically oriented SG designed to assess ED in ecologically valid contexts and (2) to evaluate its usability in clinical and healthy populations. Overall, usability was rated as good across groups, and perceived difficulty fell within a moderate range. Participants with ED reported lower usability and higher perceived difficulty compared to HCs, although these differences were attenuated after adjusting for age and education. Age and educational level were significantly associated with usability and perceived difficulty, indicating that user characteristics influence interaction with the tool.

Differences in perceived usability and difficulty in older and less educated participants are consistent with prior findings indicating that age affects usability perceptions, in which task difficulty may be over- or underestimated regardless of objective performance [49]. Regarding education, it is noteworthy that participants with lower educational backgrounds were mainly from the TBI group, which also showed more pronounced ED. Previous studies have documented the relationship between lower education and greater executive deficits in TBI [50], suggesting that usability differences may partly reflect underlying executive impairment, while also highlighting the role of demographic factors in shaping user experience [49]. The relatively narrow confidence intervals observed for these estimates suggest an acceptable level of precision [51], although some variability remains, particularly in subgroup analyses. Future studies are needed to determine the extent to which demographic factors influence the diagnostic accuracy of the SG.

As highlighted by Martínez-Pernía et al [14], traditional tools lack ecological validity, whereas SGs can more accurately simulate the demands of everyday life. A key advantage of the current SG lies in its ability to recreate realistic, dynamic, and socially embedded situations relevant to everyday functioning. Participants with ED, across both the post–COVID-19 and TBI groups, showed lower usability ratings and higher perceived difficulty compared to HCs. These group differences are consistent with previous literature, suggesting that individuals with ED experience greater difficulties when interacting with complex and dynamic environments [14]. Such contexts often involve emotionally salient and embodied interactions (ie, the integration of cognitive, emotional, physiological, and bodily responses during real-world social interactions) [14,52,53], which are crucial for effective executive functioning in daily life. However, these features may also increase task complexity and cognitive load [22]. Importantly, ecological validity does not necessarily imply lower task demands or greater usability; rather, more realistic scenarios may inherently increase cognitive load [14,22,23]. Therefore, the observed differences may reflect both the sensitivity of the task to executive impairments and the increased complexity associated with these context-rich environments, and its utility for diagnostic accuracy remains to be established. These findings should be interpreted in light of the adjusted analyses, in which group differences were attenuated after controlling for age and education. Accordingly, the results are better understood as reflecting the interaction between user characteristics and task demands within realistic assessment contexts, rather than as evidence of differential sensitivity to ED.

Unlike earlier usability studies that relied exclusively on healthy participants (eg, [16,27,28]), the inclusion of clinical groups in this study provides important evidence for the feasibility and acceptability of the SG in its intended population, thereby enhancing its translational relevance. Although usability testing can yield meaningful insights with relatively small samples, larger sample sizes increase confidence in the identification of usability issues. As Faulkner [54] noted, “more test users means greater confidence that the problems that need to be fixed will be found” (p. 382).

SGs have emerged as promising tools for cognitive assessment, with the potential to address limitations of traditional neuropsychological measures by providing more engaging, dynamic, and ecologically valid contexts [14,18,19,21]. The present findings support the feasibility of this approach in clinical populations, demonstrating that a complex, ecologically valid SG can be used with acceptable usability across diverse groups. Importantly, the influence of age and education on usability highlights the need to consider demographic factors in the design, interpretation, and implementation of SG-based assessments. Future studies should explore the development of adjusted norms, stratification strategies, or interface adaptations to ensure accessibility and appropriate interpretation across populations [18,55].

Although the present study focused on development and usability rather than diagnostic validity, the platform offers a promising foundation for future validation work [14,18,19]. Subsequent research should evaluate its sensitivity and specificity by comparing gameplay-derived performance with gold-standard neuropsychological measures and clinical consensus classifications of ED. Given its digital and interactive nature, the SG also has potential for broader applications, including screening, longitudinal monitoring, and integration into stepped-care models [56]. However, its use was evaluated in a supervised clinical context, and further research is needed to establish its feasibility, reliability, and interpretability in unsupervised or real-world settings.

This study has several limitations that should be considered when interpreting the findings. First, usability was assessed using self-reported measures, which may be subject to response biases and do not fully capture actual user behavior or task engagement [57,58]. Although the platform records detailed behavioral data (eg, action sequences, timing, and errors), these were not analyzed in the present study, which focused on development and initial usability, and therefore represent an important direction for future research. Second, the study did not evaluate the diagnostic validity of the SG; therefore, no conclusions can be drawn regarding its ability to detect ED or differentiate between clinical conditions. Third, the observed influence of age and education highlights potential variability in user experience, which may limit the generalizability of the findings across populations with different demographic characteristics. Finally, the SG was evaluated within a supervised clinical context, and its feasibility, reliability, and user interaction patterns in unsupervised or real-world settings remain to be established and should be examined in future studies. Nonetheless, a major strength of this study is the development of a novel SG for the assessment of ED, as well as the reporting of usability and perceived difficulty data in both healthy participants and individuals with ED from different etiologies.

The SG showed good usability and represents a promising platform for future validation studies. Its design supports the development of more contextually grounded approaches to the assessment of executive functioning and provides the initial evidence of feasibility in clinical populations. These findings contribute to the growing use of interactive, context-rich digital tools in neuropsychological assessment, addressing longstanding limitations of traditional methods in capturing everyday cognitive functioning. As digital health technologies continue to evolve, ensuring usability and accessibility will be critical for successful clinical implementation. Future research should incorporate objective behavioral metrics derived from gameplay, alongside self-report measures, to achieve a more comprehensive characterization of user interaction and performance. Further work is also needed to establish diagnostic accuracy and clinical reliability and to integrate such tools into routine neuropsychological assessment.