Applied games, or serious games, describe (usually digital) games designed to drive desired cognitive, behavior, or other changes in players and the wider community [1-4]. In recent years, applied games have seen increasing interest in areas, such as well-being [5], mental health [6], and education [7,8].

Applied games markedly differ from entertainment games in that they need to fit the capacities, needs, preferences, and contexts of often highly idiosyncratic target end-user audiences; find game mechanics and content that are both appealing or “fun” to the target audience; and deliver the “active ingredients” producing the desired changes, be it learning content, persuasive messages, or medical treatment regimes [4,9]. This is particularly true for applied games targeting children and young people (CYP), whose general and gaming capacities, preferences, and contexts not only drastically differ from those of adult designers and players but also from each other, depending, for example, on their developmental stage [3,10].

As in general design and development, one major successful strategy for sensitizing applied game designers to the specifics of target end users is to directly involve end users (and other relevant stakeholders) in the design, implementation, and evaluation of the game in question [11-13]. Involving users in the design process has been shown to improve use and treatment engagement [13,14], usability [15], and system adoption and adherence across stakeholders [16-18]. On the side of developers, it promises improved understanding of user needs, reduced development costs and time, and improved design quality [19]. Consequently, there are growing calls for regularly using user involvement methods in the design of applied games for health [10,11,20].

However, “user involvement” describes a wide and messy field. Different research and practice communities have developed parallel traditions with confusing differences and overlaps in name, underlying values, and details of implementation, for example, human-centered design [15,21] in computing and human-computer interaction (HCI); participatory design [22,23], co-design [21], or coproduction [24] in design; patient and public involvement in health [25,26]; or action research, participatory research and science, or citizen science across the (social) sciences [15,25,27-29]. Despite regularly involving end users in the form of playtesting, applied game design still has no strong tradition of granting users more agency and input, particularly in the early stages of the design process [30,31]. In addition, recent analyses have shown that digital mental health intervention projects vary greatly in how they approach user involvement and often fail to document how user involvement methods are implemented in detail [10,20]. Bergin et al [16] observed in their recent review that few studies similarly reported any user experience of the involvement processes used—there is no even consensus on how the user experience of study participation should be captured. As a result, there is presently little empirical data on how differences in the detailed implementation of user involvement would affect end-user engagement and the overall efficacy of involvement. Consequently, we do not have evidence-based guidance on how to best involve end users and other stakeholders in the design of applied games [32], for example, for mental health [1,16,33] or education [8,34].

This lack of evidence and guidance is particularly pertinent for user involvement of CYP, and we neither know what roles, activities, or tools help engage and maintain their participation [17,29] nor what implementation factors would affect their involvement or actual adoption, adherence to, and efficacy of developed interventions (an exception was found in a study by Shah et al [35]).

Thus, before we can begin to articulate potential best practice guidelines for the implementation of user involvement in designing applied games with CYP, we need basic stocktaking of actual existing practice and evidence. To this end, this scoping review aimed to describe how user involvement has been implemented in designing games for CYP and what factors (if any) likely affect effective CYP involvement. We articulated this aim in 4 research questions (RQs):

For our scoping review, we combined a systematic literature review [36] with inductive qualitative content analysis [37]. We report our method following the revised 2020 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [36]. We did not preregister this scoping review because of the descriptive and exploratory nature. All study materials can be found in the Open Science Framework repository (Multimedia Appendix 1).

As noted, our review focuses on user involvement methods in the design of applied games with CYP. With “user involvement methods,” we capture any approach that involves end users in the design of a game, such as participatory design, co-design, play-centric design, or user-centered design. With “applied games,” we mean any attempt to create or use game-based software for a nonentertainment change. “CYP” was interpreted verbatim from the records that captured a variety of age ranges, for example, 2 to 4 years or 16 to 25 years.

Unfortunately, there are no well-established standard terms or controlled vocabularies for our search focus; for example, applied games are commonly called “serious game,” “game-based,” “game for X,” or “gamified X.” Following similar reviews [10,11,38], we developed a core search string that combined a range of synonymous keywords for each aspect, such as CYP, games, applied contexts, and user involvement methods (Textbox 1).

We analyzed papers published in the last 10 years (January 1, 2010, to December 7, 2021), and as the field of applied games is quite new, we did not expect a great number of texts before this date. We implicitly focused on selecting papers that had a specified “applied context” (Textbox 1) in which, for example, games were developed with young people for “mental health” or “education.” Furthermore, we focused on the last 10 years of records owing to the development and changes in technologies and methodologies. Capturing recent (within 10 years) studies provides an up-to-date overview of the state of the literature.

We included full length papers that reported on the design of game-based software targeting CYP and explicitly featured and reflected on user involvement methods. As we are interested in understanding the factors affecting user involvement, we excluded papers that did not reflect or evaluate the user involvement methods they used. We did not exclude papers based on application contexts—we wanted to avoid unnecessary focus on a specific domain, such as health-related literature, because insights may already have been made in educational gaming or entertainment game design but ignored by us because of a narrow focus on health-related literature. We excluded short or work-in-progress papers, because they did not provide enough space for detailed reflection and reporting on the methods used. We did not investigate gray literature because of the difficulty of creating a reliable and reproducible search strategy for such a dispersed and unstructured collection of materials with no core databases. See Textbox 2 for the full inclusion and exclusion criteria.

We searched 4 databases: the ACM Digital Library, IEEE Xplore, Scopus, and Web of Science. These databases mirror the interdisciplinary structure of research on applied games: the Association for Computing Machinery Digital Library and IEEE Xplore, which cover computing and HCI conferences and journals in which the bulk of technical games research is published, while Web of Science and Scopus capture disciplines such as medical research, psychology, and education.

Before starting our search, we iterated on variations of search strings implementing our target keywords (Textbox 1) for each database, because each database afforded different search strings tools. The final search strings are presented in Multimedia Appendix 2. We conducted the first full search on May 7, 2021, and the last search on July 5, 2021.

The studies retrieved from the databases were managed using State of the Art through Systematic Review (StART; version 3.3, Beta 03; Laboratório de Pesquisa em Engenharia de Software). StART identified additional studies through snowballing, which were added to the selection for screening. In addition, 14 studies were manually added.

We first removed duplicate records using StART, which identified duplicates across databases. Additional duplicates were then manually removed from the selection. The first author then manually screened titles, keywords, and abstracts against the eligibility criteria; sourced full texts of the eligible studies; and then manually assessed full texts for eligibility. Finally, records that were reported in the same study were merged.

All relevant information was extracted in StART, which was then exported (xlsx format) into the qualitative data analysis software NVivo (version 12; QSR International) for open coding across the extracted information. For each eligible study, the first author extracted standard metadata (title, authors, abstract, and year) in addition to a range of descriptive data.

To describe our sample and study characteristics, the first author coded papers by the following parameters:

To describe user involvement methods and roles, the first author coded papers for the following factors:

Finally, to describe how user involvement was implemented and how to identify emerging factors affecting it, we imported the extracted data fields and full-text PDF documents into NVivo for inductive qualitative content analysis [37]. The first author inductively coded the method, discussion, and conclusion sections of papers for related emerging themes in the first and second focused coding cycle [40]. Descriptively, (10) study structure, (11) activities, and (12) media and tools have emerged as high-level categories. Of these, the activities were clear and distinct enough that we could conduct a follow-up frequency count. In terms of (13) factors affecting user involvement, 4 themes emerged.

Because the aim of our study was narrative description, not establishing summary effects, no bias or certainty assessments were performed.

For descriptive summary reporting, we calculated the frequencies for (1), (2), (4), (5), (6), (7), and (11). Emerging themes and observations were synthesized through standard inductive qualitative content analysis.

Our search returned 1085 records. Title and abstract screening removed 81 duplicate records and further excluded 885 records, of which 164 (18.5%) did not clearly incorporate user involvement methods, 626 (70.7%) lacked explicit reflection of user involvement methods, 63 (7.1%) were not related to game-based software, and 32 (3.6%) were not full papers. At the full-text stage, of the remaining 13.4% (119/885) of records, we excluded a further 69 (58%), of which 22 (32%) were not full papers, 12 (17%) lacked user involvement methods, 22 (32%) lacked a detailed reflection of them, 4 (6%) had no CYP involvement, and 1 (1%) was not related to games; we could not obtain access for 7 (10%) records, and 1 (1%) was not published in English. Of the 50 remaining papers, we merged 3 (6%) paper pairs that reported the same study, resulting in 47 final studies in the analysis. Figure 1 presents the PRISMA flow diagram.

Table 1 presents the characteristics of the included studies; 47% (22/47) of the studies were from computer science and HCI, 19% (9/47) were from education, 17% (8/47) were from serious games or games design, 11% (5/47) were from health, and 6% (3/47) were from psychology. There was no clear upward or downward trend in the publications over time. In total, 85% (40/47) of studies were conducted in the Global North, led by European countries (15/47, 32%), then the United Kingdom (13/47, 28%), Australia and New Zealand (7/47, 15 %) and the United States (4/47, 9%), followed by Brazil (3/47, 6%) in the Global South.

The sample size of CYP involved ranged from 2 to 109, with an average of 23 (SD 24.8) and a median of 13. Table 2 shows the frequency of sample sizes with 26% (12/47) studies involving 6 to 10 participants, 23% (11/47) involving 11 to 25 participants, 19% (9/47) involving 26 to 50 participants, 17% (8/47) involving 1 to 5 participants, and 15% (7/47) involving >51 participants. This is likely because (1) most studies focused on interventions for intersectional groups, such as children with learning difficulties, in which sampling can be challenging; and (2) user involvement methods often value “thick data” with small-sample qualitative methods and a struggle to scale to large participant numbers. Nevertheless, many studies have reported “small sample sizes” as a limitation.

The youngest age sampled across all studies was 3 years, and the oldest was 25 years as a part of a “young people” sample spanning ages 16 to 25 years [52]. The median age sampled was 11.5 years. Most studies (30/47, 64%) sampled an age range from 0 to 3 years (where 0 would be a precise age in years and 3 would be an age range of, eg, 6 to 9 years; Table 3).

Although our review focused on studies involving children, we were also interested in the participant groups that were involved. The plurality of studies (23/47, 49%) involved 3 different participant groups. The predominant groups were CYP (n=47, 100%) owing to inclusion criteria specifying CYP involvement. Studies also label CYP as patients, learners, or target audiences. Subject-matter experts (39/47, 83%), such as health professionals (11/47, 23%), teachers (18/47, 38%), and other experts (10/47, 21%), were cumulatively the second-most frequent participating group (Table 4). Other experts encompassed were from areas such as film, photography, art, and music. The researchers conducting or facilitating the study were the next most frequently involved group (36/47, 77%) in the design process. Researchers included those carrying out the study and references to scientists who were also involved in the study. Some studies also used designers and developers (17/47, 36%), which included game design companies, masters’ students in HCI, game designers, graphic designers, and artists, to help with the development of games or interactive technologies. Support groups (5/47, 11%) included learning assistants, pastoral care coordinators, and special education coordinators who helped support CYP participation.

Study length varied greatly, and in 17% (8/47) of studies, the study length could not be determined from the sampled records. The most frequent duration was 2-3 months (10/47, 21%), followed by studies conducted in ≤1 day (8/47, 17%). Studies conducted in ≤1 day were often short workshops or sessions of 1- to 2-hour activities. Studies conducted over 1 to 7 days (5/47, 11%) were the next most frequent. This was followed by studies that took either 4-6 months (4/47, 9%) or 10-12 months (4/47, 9%). Studies conducted between 2 and 4 weeks (3/47, 6%) and 10-12 months (4/47, 6%) were less frequent, and the lowest frequency was 7 to 9 months (2/47, 4%). A consideration to take into account is that it is not clear whether some studies are reporting the length of the whole study, including recruitment, procedure, and analysis, or whether they are reporting the duration of participants involvement in the study.

The dominant verbatim labels used for CYP involvement roles were “informant” (26/47, 62% of studies) and “co-designer” (20/47, 42%), followed by “playtester/tester” (4/47, 9%), “validation” (4/47, 9%), “co-creator” (4/47, 9%), “end user” (2/47, 4%), and “co-researcher” (1/47, 2%). Overall, 11% (5/47) of studies did not indicate a particular role of involvement (note that a single study could identify multiple roles for CYP; hence, percentage added up to >100%).

Similar to self-labeled methods, these role labels can cover a wide variety of actual degrees and types of involvement. In addition, we coded all studies using the Druin [39] influential taxonomy of 4 possible roles children can play in the design of new technology, in which each successive role can be seen to be more agentic than and encompass the previous ones: ([[[[user]tester]informant]design partner]). Each role also differs in the CYP-adult interaction (from indirect observational input to feedback, dialogue, and elaboration on adult ideas), forms of technology materializations engaged with (from ideas to prototypes to existing products), and goals of inquiry (from developing theory about use to assessing technology effects to improving design and usability):

We coded each included study according to these roles by examining how CYP had been involved in the procedure of the user involvement method and mapping this to the Druin [39] taxonomy (Table 7). According to this classification, in 51% (24/47) of the studies, CYP were involved as design partners; 70% (33/47) of studies involved CYP as informants; and testers and users were involved in 11% (5/47) of studies each. In 2 studies, CYP were only involved as users, and 1 study involved them only as testers. Finally, it is worth noting that 2 studies explicitly adopted Druin [39] taxonomy and ensured the involvement of CYP in each of the 4 roles [43,75].

Inductive coding of the studies’ method descriptions yielded 3 high-level categories of how user involvement was implemented in detail: structure and sessions, activities, and media and tools. Session structure describes recurring stages and an overarching facilitation organization. Activities captures the specific tasks conducted with stakeholders, while media and tools describe the range of media and existing games used in activities.

Out of a total of 47 studies, 36 (77%) of our sample were self-labeled participatory design or co-design (Table 5). In line with previous reviews in related fields, such as co-design in health [90] or CYP involvement in child-computer interaction [91], these 2 broad labels gloss over a wide variety of actual involvement roles, aims, and activities. Most studies sampled (35/47, 74%) involved CYP as “mere” informants and about half (24/47, 51%) as true design partners (Table 7). These numbers appear to indicate deeper user involvement for CYP than in recent systematic review of general serious game development processes by Maheu-Cadotte et al [92] (who found only 1 in 21 processes involved participants as co-designers). However, because their inclusion criteria did not limit eligible studies to those with explicit user involvement, this is an unequal comparison.

The stated aims for user involvement ranged from instrumental ones of making a game or identifying features and functionality to meta-methodological interests, upskilling, and many others. We counted 45 different activities, with prototyping, interviewing, playtesting, and playing existing games being the most common. Studies commonly reported “early” onboarding, ideation, and prototyping phases; we found little mention of “later” development or production stages [64,78,82]. Furthermore, no records linked or referenced openly shared user-created design materials outside of screenshots and workshop photographs. Studies used a wide variety of tools and media, with some consensus that these should be tailored to the (developmental) preferences and abilities of the involved CYP. The reviewed studies did not discuss users’ accreditation or rewards for their contributions, outside of 1 example, which suggests uncertainty on how to motivate or reward CYP for their participation.

Our qualitative coding of method reflection produced five factors that are likely to affect successful CYP involvement:

Perhaps the strongest overarching observation when conducting this systematic review was how differently studies understood and used relevant terms such as “co-design” (2 studies could label their activity “co-design” and “focus group” yet do drastically different things); how differently and unevenly studies reported methods; how differently studies implemented these methods; and in how little detail studies actually documented their involvement methods, despite the fact that our review systematically sampled studies that explicitly reported and reflected on their user involvement. For instance, although we tried to map activities onto “where” in a standard design process, following the example in Vandekerckhove et al [20], we were not able to do so because of the low to absent detail of reporting. This is sadly in line with prior findings on co-design methods: studies rarely report sufficient detail to reproduce them, and there is no standard reporting format [90]. This lack of consensus terminology and reproducible method documentation standards hinders replication. This makes systematic assessment and integration of evidence difficult. In addition, it impedes actual know-how flows beyond tacit and in-person sharing between the members of a research group and project.

Unless we have clear, reliable, and reproducible standards for identifying and reporting types or degrees of user roles and identifying and reporting involvement methods, it is difficult to generate a meaningful systematic body of evidence on which roles or methods work better or worse for which groups of participants, contexts, or aims. This may be one reason why only 4 of the sampled studies reported any qualitative empirical evaluation of what worked and what did not with their involvement methods, even though 46% (22/47) of studies stated meta-methodological aims, such as assessing end-user involvement or involvement methods. Consequently, most reflections and derived recommendations on how to conduct user involvement remain speculative. Thus, we cannot conclude with much certainty anything about, for example, which user roles might work “better.” Although there is a normative preference for the “deepest” possible involvement in much of participatory design, several studies in our review suggested that CYP struggled to act as meaningful co-designers in applied game design [76] and were more able to express themselves as play testers than when sketching their own concepts [63]. This may be partially because of the fact that applied game design requires both domain and game design expertise, which CYP likely does not bring, which may be bridged by more careful onboarding activities or involving multiple stakeholder groups with complementary expertise [60]. Although the involvement of young people has been reported to be essential to realize game prototypes, whether CYP has expertise, or even the availability to consistently collaborate, is a challenge for future research [93]. The important point here remains that we are at present not able to answer such questions or advise on “what works best” because of poor reporting.

Our sample is obviously limited by its English-only focus and 10-year date range. The literature before 2010 would potentially paint a distinctively different picture, which would lead to divergent conclusions in this review. Study selection and coding were all performed by the primary author alone, meaning that we could not quantify the likely reliability and reproducibility of these steps, for example, intercoder reliability. The overall poor quality of method reporting in the sampled studies and the lack of existing controlled vocabularies or taxonomies of involvement methods means that our coding involved substantial degrees of judgment and construction in categorizing different kinds of involvement activities or classifying studies by user role. This does not impact verbatim extractions of self-labeling or similar, and we believe our overall assessment of the sheer diversity of reported practices is not touched by this.

The first obvious direction for future work is to provide a robust shared basis for reporting and assessing user involvement in applied game development (with or without CYP); we need better consensus methods for defining, labeling, and identifying user involvement roles and methods and better guidance and standards on reporting user involvement in sufficient detail for later analysis or replication. Second, on this basis, we can start to analyze and empirically evaluate what kinds of session structures, methods, roles, media, and tools are more engaging in CYP and more effective for different involvement aims. Third, our review suggests that comprehension, cohesion, confidence, accessibility, and time constraints are likely to impact CYP involvement. Here, methodological research can explore whether and how much these factors matter, and then again, which structures, methods, roles, media and tools, and specific implementations thereof are better suited to support them.

This scoping review aimed to explore how CYP have been involved in the design of applied games. We found that a small range of labels (co-design and participatory design) hid a wide variety of actual involvement methods, aims, structures, roles, and implementations. Comprehension, confidence, cohesion, accessibility, and time constraints emerged as 5 likely nonexhaustive factors affecting effective and engaging CYP involvement. However, the reviewed literature documented its user involvement practices inconsistently and in little detail, and its recommendations for future practices are largely not grounded in robust empirical evaluations of (alternative) involvement approaches. Future work is needed to advance more robust and reproducible documentation of user involvement to enable knowledge sharing, as well as more systematic research on “what works” in user involvement of CYP in applied game design.