This study was a descriptive review of end users' involvement (concept) in the development of SGs (context) for health care professionals and students (population). Descriptive reviews allow the identification of trends in a representative sample of published literature regarding prespecified methodological or theoretical elements [13]. This descriptive review builds on the methods used and results found in a previous systematic review aimed at evaluating the efficacy of SGs in health care professions education [14,15].

For this review, “end users’ involvement” was considered an umbrella term for inviting health care professionals and students to contribute to the design or refinement of an SG in any of the three stages of the development process prior to efficacy evaluation [3]. Health care professionals and students with any level of education (from undergraduate to postgraduate education, continuing education) or from any clinical setting were considered. However, SGs for patients were not considered. All SGs that were included aimed to improve learning outcomes (eg, knowledge, skills, attitudes, behaviors) related to various clinical situations or topics.

For a previous systematic review [14], we developed a search strategy to identify randomized controlled trials (RCTs), evaluating the efficacy of SGs among health care professionals and students. The search strategy combined keywords and index terms related to health care professions (eg, nurses, medical students), SGs (eg, game-based learning, educational game), and learning outcomes (eg, knowledge acquisition, skill development). On May 26, 2020, we searched six bibliographical databases: Cumulative Index of Nursing and Allied Health (EBSCO), EMBASE (OVID), ERIC (ProQuest), PsycINFO (APA PsycNET), PubMed (NCBI), and Web of Science—SCI and SSCI (ISI – Thomson Scientific). Two review authors performed the reference selection independently, identifying 45 SGs whose efficacy had been evaluated in 46 published RCTs. Further details regarding the search strategy and selection process for this previous review are published elsewhere [14-16]. The complete search strategy for all bibliographical databases is also presented in Multimedia Appendix 1.

For the current review, we focused on all development work prior to these 46 RCTs. As evaluating the efficacy of an intervention represents one of the last stages in its development [17-19], we considered that SGs that had been the object of RCTs had gone beyond the prototype programing and testing phases [3]. Thus, including only SGs that had been the object of RCTs allowed us to be confident that their development was complete as well as the end users’ involvement in it. One review author performed a backward reference search in the reference lists of the 46 RCTs to identify prior work that described the development of the SGs. When the name of an SG was provided, this review author also performed hand searches in Google to identify additional work describing its development. We included all types of work regarding the development of the SGs (eg, qualitative or quantitative empirical research, discussion on the development process) and all types of reporting (eg, conference abstract, poster, journal article, web page). A second review author also independently identified work related to the development of 8 SGs chosen randomly (18% of the 45 included SGs). This was to ensure that all relevant work was included. In medical record reviews, an independent audit of at least 10% of the sample is frequently recommended [20]; however, as no additional references could be identified for these 8 SGs, it was deemed satisfactory.

The unit of analysis was the included SGs. All documents related to a single SG were considered concurrently to describe the characteristics of end users’ involvement during the development of a particular SG. Thus, all frequency counts are based on the number of SGs rather than the number of papers included in this review.

Using an ad hoc data extraction grid based on the review aims and questions, one reviewer extracted all excerpts regarding end users’ involvement in SG development and categorized them according to the research questions:

We coded data of the elements for which end users’ input was elicited and how their input was integrated into the SG (ie, how the SG was modified following end-user input) based on the SG design framework by Alexiou and Schippers [2]. We further synthesized them using an inductive approach based on data similarities. A second review author independently performed data extraction and coding. We refined the coding scheme until no difference from the results of the first reviewer was noted. This occurred after data extraction and coding were performed in a random sample of 8/45 SGs (18%).The results are presented narratively and by using descriptive statistics (frequencies and percentages) when appropriate.

The number of end users involved was reported for 9 of the 21 SGs (43%) and ranged from 12 to 36, with a median of 27 (IQR: 16) [21-29]. For 3/21 SGs (14%), the number of end users was defined by convenience: all readily accessible individuals were approached, and those who agreed to participate were enrolled [21,23,26]. No justification was found for the number of end users involved in the remaining 6/21 SGs (29%) [22,24,25,27-29].

Eligibility criteria for end-user selection were reported for the development of 1 of the 21 SGs (5%). Researchers selected an equal number of men and women with varying degrees of experience in gaming, but they did not provide a rationale for that decision [24].

We identified the role given to end users in the development process of 17/21 SGs (81%) [21-37]. Table 2 reports the end users’ roles in SG development and the methods used to elicit their input.

As Table 2 shows, end users were more often prototype testers (12/21, 57%), and their input was frequently elicited through questionnaires (7/21, 33%). Details regarding the content of these questionnaires and their development process were rarely provided (4/21, 19%). For one SG, authors used the System Usability Scale [42] with additional items regarding its ease of use and alignment with end users’ design preferences [24,43]. The System Usability Scale consists of 10 statements regarding the ease and speed of use of software, and end users are asked to express their level of agreement with each statement on a 5-point scale. Besides questionnaires, researchers also recorded end users’ interactions with a prototype of the SG and asked them to think aloud during their gaming experience. Based on the effect on end users’ experience with the SG, each interaction was then classified as either positive, neutral, or negative [24].

Only 1 of the 21 SGs (5%) was developed with end users as co-designers who oversaw the development of the clinical content [31]. This SG aimed to improve nurses’ confidence and skills in teaching the correct inhaler technique to patients. With these learning objectives in mind, a group of nurses developed the clinical content, which consisted of a description of seven steps to be followed during the self-administration of inhaled medication. The rest of the development team then developed the narrative of the SG around these seven steps.

Elements for which end-user input was elicited were reported for 15/21 SGs (71%) [21-26,28-31,34-37,39]. Researchers also stated that they modified 10 of the 21 SGs (48%) according to end-user input [23-25,27,30,33-35,37,39]; however, they detailed the modifications for only 5 SGs (24%) [24,25,27,35,39]. Table 3 reports the results regarding end users’ input in SG design. End users’ input was most frequently elicited regarding the goals of the SGs (10/21, 48%); their input regarding hedonic esthetics (2/21, 10%) and narratives (1/21, 5%) rarely received focus.

Our review of 70 references indicated that end users were involved in the development of less than half the 45 SGs in health care professions education. They most often took the role of prototype testers during the later stages of SG development, and they were rarely involved as co-designers or consultants at the onset of development. In addition, researchers often used questionnaires to elicit end-user input. Other methods such as focus groups and individual interviews were rarely used. The level of challenge and functional esthetics were the aspects of SGs for which end users’ input was most frequently elicited.

Several criteria could be used to select end users in the development of SGs. In this review, criteria were mentioned for only one SG and focused on gender and gaming experience [24]. Garber et al [44] underlined that current evidence does not clearly support gender differences in learning preferences. Indeed, suggested gender differences in SGs relate to gameplay preferences (ie, competition for men and collaboration for women) and their perceived educational value (ie, higher in men than in women) [44,45]. However, in health care professions education, once the SGs reached the stage of efficacy evaluation, gender-based analyses have yet to reveal significant differences in learning outcomes [46-48]. Thus, the extent to which SG design should be informed by end users’ gender to improve its efficacy remains unknown. Diehl et al [24] mentioned gaming experience as another criterion for the selection of end users, noting that users with a small or large amount of gaming experience provided the richest input. Similarly, Boeker et al [22] reported that end users with the least amount of gaming experience had the most issues in their interactions with the SG. Given the limited evidence regarding criteria for end user selection, we suggest that researchers should include all end users without considering predetermined characteristics. We encourage them to report on differences in the input of end users based on gender, gaming experience, or other characteristics that could have played a role in the input obtained and in the result of their involvement.

The number of end users involved in SG development ranged from 12 to 36, and these numbers seemed to be mostly based on convenience, as the available end users were approached. Current sample size estimation approaches that are not focused on statistical power only seem suited to testing SG prototypes, as the numbers they suggest can still be considerable [49,50]. However, as the development and refinement of an SG is highly iterative, and many versions of the design of the game are proposed during the first development stages [3], including several end users at the onset of development can represent a challenge. In the broader fields of intervention development and participatory research, authors have suggested that establishing a dialogue with a small number of end users during the initial development stages—aiming for depth rather than breadth—is more important than obtaining a large sample size [17,51]. This suggestion is consistent with qualitative methods such as individual interviews or focus group discussions. However, the use of qualitative methods was described only twice, and only once before the testing of an SG prototype [24,32]. Thus, we suggest that researchers explore methods to elicit end users’ input that are suited for smaller samples, such as qualitative methods. This approach may help researchers acquire a rich understanding of end users’ perspectives on the design of an SG.

The results also elucidate the gap between eliciting and integrating end-user input into the SG design. Few researchers described the changes made to SGs following end-user involvement, and none discussed their decision-making process. Kelly [52] argued that considering end users’ input can be uncomfortable for researchers and, therefore, Kelly recommends that the researchers attempt to find a compromise between making all the decisions themselves and relinquishing all forms of decisions to end users. It has been suggested that not all end users’ input should be directly integrated into an SG and that experts in the field should review what could potentially improve the efficacy of an SG [9,53]. Further research is needed to identify what elements of end users’ input are the most valuable, and at what stage of the development process; thus, researchers should describe their rationale for eliciting end users’ input and their decision-making process for integrating this input.

In this review, we did not identify literature (published or accessible on the web) on the development of most SGs that were the object of an RCT. However, in papers detailing the results of RCTs, we found many instances of authors referring to previous unpublished studies related to the development of their SGs [28,31,39]. This could point to the existence of literature on development that researchers or editors of scientific journals did not consider suitable for publication [54,55]. We argue that sharing this type of experience may prove valuable for planning future SG developments. Further, as current publications of work related to the development of SGs often focus on end users as prototype testers, researchers should consider publishing their experience with end users as consultants, especially in the first stages of SG development or as co-designers. In this review, we found that end users often served the needs of researchers and developers either to answer their concerns or to test a prototype through a fixed protocol. This limits end users’ contribution to designing SGs as well as researchers’ ability to potentially substantially change the design of an SG once it has reached the testing phase.

The strengths of this descriptive review are that it entailed a comprehensive literature search to ensure that all SGs that had been the object of an RCT were found. The selection process was conducted independently and in pairs to ensure that all relevant papers were included; however, SGs that had not been the object of an RCT were not considered in this review. If the development of these SGs were still in progress, only a partial portrait of their development would have been provided. Other limitations include the data extraction process, which was conducted by a single review author for most SGs. This was judged adequate, as the aim of this work was descriptive and no efficacy data were extracted. Finally, as underlined in the discussion, this review was limited to work available on the internet.

Considering that end users’ involvement was poorly described in the SGs under review, we suggest that researchers publish information on the nature of end-user involvement, including the characteristics of the end users selected, content of the instruments used to elicit their input, modifications made to the SGs based on end-user input, and lessons learned throughout the development process. As few researchers reported end users’ involvement in the initial development, those opting for this type of involvement should consider sharing their views on the process. Moreover, researchers should consider involving end users with varying levels of gaming experience and combining different methods to elicit their input to gain further insights into issues that may undermine the efficacy of SGs.