Published on in Vol 7, No 2 (2019): Apr-Jun

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/11565, first published .
Developing Theory-Driven, Evidence-Based Serious Games for Health: Framework Based on Research Community Insights

Developing Theory-Driven, Evidence-Based Serious Games for Health: Framework Based on Research Community Insights

Developing Theory-Driven, Evidence-Based Serious Games for Health: Framework Based on Research Community Insights

Review

1MindBytes BVBA, Merksplas, Belgium

2MindLab Interactive AI Inc, Edmonton, AB, Canada

Corresponding Author:

Sarah Verschueren, BSc, MSc, PhD

MindBytes BVBA

Schoondreef 7

Merksplas, 2330

Belgium

Phone: 32 476753516

Email: sarahverschueren@hotmail.com


Related ArticleThis is a corrected version. See correction statement in: https://games.jmir.org/2020/2/e18515/

Background: The idea of using serious games to effectuate better outcomes in health care has gained significant traction among a growing community of researchers, developers, and health care professionals. Many now recognize the importance of creating evidence-based games that are purposefully designed to address physical and mental health challenges faced by end users. To date, no regulatory resources have been established to guide the development of serious games for health (SGH). Developers must therefore look elsewhere for guidance. Although a more robust level of evidence exists in the research literature, it is neither structured nor is there any clear consensus. Developers currently use a variety of approaches and methodologies. The establishment of a well-defined framework that represents the consensus views of the SGH research community would help developers improve the efficiency of internal development processes, as well as chances of success. A consensus framework would also enhance the credibility of SGH and help provide quality evidence of their effectiveness.

Objective: This research aimed to (1) identify and evaluate the requirements, recommendations, and guidelines proposed by the SGH community in the research literature, and; (2) develop a consensus framework to guide developers, designers, researchers, and health care professionals in the development of evidence-based SGH.

Methods: A critical review of the literature was performed in October to November 2018. A 3-step search strategy and a predefined set of inclusion criteria were used to identify relevant articles in PubMed, ScienceDirect, Institute of Electrical and Electronics Engineers Xplore, CiteSeerX, and Google Scholar. A supplemental search of publications from regulatory authorities was conducted to capture their specific requirements. Three researchers independently evaluated the identified articles. The evidence was coded and categorized for analysis.

Results: This review identified 5 categories of high-level requirements and 20 low-level requirements suggested by the SGH community. These advocate a methodological approach that is multidisciplinary, iterative, and participatory. On the basis of the requirements identified, we propose a framework for developing theory-driven, evidence-based SGH. It comprises 5 stages that are informed by various stakeholders. It focuses on building strong scientific and design foundations that guide the creative and technical development. It includes quantitative trials to evaluate whether the SGH achieve the intended outcomes, as well as efforts to disseminate trial findings and follow-up monitoring after the SGH are rolled out for use.

Conclusions: This review resulted in the formulation of a framework for developing theory-driven, evidence-based SGH that represents many of the requirements set out by SGH stakeholders in the literature. It covers all aspects of the development process (scientific, technological, and design) and is transparently described in sufficient detail to allow SGH stakeholders to implement it in a wide variety of projects, irrespective of discipline, health care segments, or focus.

JMIR Serious Games 2019;7(2):e11565

doi:10.2196/11565

Keywords



Background

Many games and apps market themselves as tools or interventions to address health conditions and disease, yet they provide little explanation on their development, provide minimal information on real-world evaluation of their efficacy, and often reference poorly designed research studies [1-4]. Developers, designers, researchers, and health care professionals involved in the development of serious games for health (SGH) (further simply referred to as developers) now increasingly recognize the importance of creating evidence-based games that are purposefully designed using expertise, knowledge, and validated, quality data [1-7]. To be recognized as a nonpharmacological health care intervention and gain marketing approval from regulators, to obtain reimbursement from health care payers, or to gain CE approval when dispensing medical advice, developers will need to follow rigorous standards and provide a solid rationale for use and clear empirical evidence of the intervention’s safety and efficacy [8]. This trend, together with an increasing focus on incorporating patient needs and preferences in the development process of health care interventions [9-11], has resulted in a paradigm shift in the development of SGH from a mainly game design orientation with a focus on user experience toward a more scientific approach that involves multiple stakeholders such as patients, clinicians, caregivers, payors, as well as regulators [12,13].

Developers who intend to market their SGH interventions to clinicians and patients will also need to deliver convincing evidence of the game’s ability to safely achieve the intended outcomes if they wish to overcome the current barriers to uptake. As only a few validated tangible success stories exist, many clinicians are skeptical about the use of SGH in current health care practice. These barriers may hinder medical and scientific progress in certain fields and impact the investment risk associated with developing SGH. Although the development cost can vary greatly depending on complexity, graphical and technical design features, and the time spent on scientific substantiation and (clinical) evaluation, it typically ranges from ten to several hundred thousand dollars [14]. When complex three-dimensional motion graphics, community platforms, or large-scale clinical evaluation trials are involved, the development costs can even run up to several millions of dollars. Such large investments are risky, given the fact that many SGH address small market niches with limited potential for return on investment. Therefore, any potential barriers to uptake, such as lack of credibility and evidence of effectiveness, compound the investment risk for developers.

The Status Quo

To create theory-driven, evidence-based SGH, developers should collect and integrate scientific evidence and data throughout the entire development life cycle—from early stage theoretical work to later stage evaluation [1-3,5,7,15,16]. However, to date, there is no clear regulatory framework for the development of SGH beyond the type of evaluation data required (ie, evidence of risks and benefits). Regulatory requirements of SGH will likely depend upon their precise claims, and there are few transparent conditions that developers of minimal risk applications must meet before their products can be launched. This may also be the case for applications that are not obviously minimal risk as the developer must first engage regulatory authorities to determine what regulations they need to comply with.

In the absence of a regulatory framework, developers must look elsewhere for guidance on suitable approaches for developing SGH. Although a more robust level of evidence exists in the research literature, it is neither structured nor is there any clear consensus. The few resources that do exist are often focused on only a fragment of the development process, such as technology aspects or pedagogical aspects [17,18]. Others are described at such a high level that it is not possible for developers to implement such recommendations. Without clear consensus on frameworks, guidelines, and recommendations, developers must arbitrarily select which resources to follow.

This is in fact what happens. SGH developers currently use variable frameworks, differing guidelines, and alternative methodologies in SGH development [6,19]. The issue is further compounded by the fact that this emerging field is so multidisciplinary that each segment utilizes its own specific set of principles and frameworks to develop individual components. Moreover, development is often specialized to specific SGH classifications or target audiences [20,21]. Therefore, it is clear that SGH developers would benefit from the establishment of a defined set of requirements that represents the consensus views of SGH stakeholders [21]. Not only would this help increase SGH probability of success but it would also benefit the SGH community by raising the quality of SGH by providing the necessary evidence required by stakeholders. Moreover, it would also enhance the credibility of SGH developers and allow them to achieve a sustainable market share.

Objective

The objective of our research was to search the literature and identify and evaluate the requirements, recommendations, and guidelines proposed by the SGH research community on the development of SGH. This included recommendations on what inputs are required to guide the development, what data should be collected, how games should be tested, which stakeholders should be engaged, and what game design approaches should be considered. On the basis of the findings, a clear and easy-to-implement consensus framework was developed to guide developers in the development of theory-driven, evidence-based SGH.


Databases and Search Strategy

A critical review of the research literature was performed in October to November 2018. The following databases were searched electronically: PubMed, ScienceDirect, Institute of Electrical and Electronics Engineers Xplore, CiteSeerX, and Google Scholar. A 3-step search strategy was used. An initial limited search was undertaken using the search strategy (game OR games) AND (serious OR applied OR health*), where * represents a wildcard to allow for alternative suffixes. This was followed by an analysis of the text words contained in the title and abstract and an analysis of the index terms used to describe article. A second search using all identified keywords and index terms was undertaken across all above databases. Finally, the reference list of all identified reports and articles were searched for additional relevant studies. A supplemental search of guidelines from regulatory authorities was also conducted to capture the requirements of these specific stakeholders. Three researchers independently evaluated the identified articles.

Inclusion Criteria

Included papers were empirical research studies, literature reviews, opinion pieces, preliminary research, randomized controlled trials (RCTs), theoretical models, conference proceedings, conceptual frameworks, or design documents that (1) reported on the development or evaluation of a serious game for use in a health care context, (2) were published in English, (3) were published between January 2007 and November 2018, and (4) were peer reviewed.

Exclusion Criteria

Excluded were any articles that (1) contained abstract only, (2) reported on serious games with applications outside of health care (formal education, corporate training, business decision making, etc), and (3) focused on pedagogical or psychological theory with no link to serious games.

Coding

After screening, requirements relating to the development of SGH were extracted from each of the papers. These requirements were coded using the following steps, on the basis of a thematic analysis approach:

  1. Requirements of SGHs that were described, identified, or documented within the included studies were collected verbatim.
  2. Existing frameworks [2-5] within the included studies were evaluated to identify currently accepted terminology for various requirements. In cases of noncongruent terminology among frameworks, an established term was selected by 1 researcher and confirmed by the other 2 researchers.
  3. The verbatim requirements from step 1 were reviewed individually, and duplicates (variations of the same underlying requirement) were removed. Again, the resulting term was selected by 1 researcher (CB) and reviewed and agreed by the other 2 researchers.
  4. The requirements from step 3 were compared with the accepted terms from step 2 and, if deemed appropriate, the former were categorized within the latter. One researcher (CB) performed this step first and the coding was discussed and confirmed by the other 2 Researchers.

The coding was performed to facilitate understanding and to address parsimony, which is threatened by nonconsensus descriptions, terminology variations, etc.


Search Results

Our initial search yielded 216 papers (excluding duplicates). Of these, 74 papers were included in the review. See Figure 1 for a flowchart of the combined searches, as per Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines [22]. Analysis resulted in a list of 62 requirements for the development of SGH, proposed by the SGH research community. Some requirements were formulated on a meta-level, whereas others were more detailed and concrete. The requirements were therefore categorized to allow for a structured analysis. We identified 5 categories of high-level requirements, as well as 20 detailed (low-level) requirements. See Table 1 for an overview of the identified requirements and categories, and the corresponding papers.

Figure 1. Literature search flowchart.
View this figure
Table 1. List of high-level and low-level requirements and categories, and corresponding papers.
Hierarchy, category, requirements References
High level

Methodological approach


Methodological approach[2,3,5,7,16,23-28]


Multidisciplinary, participatory design process[1,2,5-7,12,13,19,21,23-25,29-37]


Project management approach[28,30,31,38-40]


Quality evaluation[1-3,12,15,16,19,23-26,28,32,35,36,38,41-54]


Publish and disseminate findings[1-3,16,24]
Low level

Inputs


Detailed profile of target audience[1-3,5-7,23,33,43,45,51,55,56]


Target audience needs[2,5,7,16,25,29,43,55,57-60]


Primary research goal[1-3,12,15,23,25,33,43,55,58]

Models and theories


Psychological theory[1,2,4,23-25,27,53,56,58,59,61]


Game mechanics[1,2,5,7,19,20,33,44,45,47,54-58,62-66]


Link between theories, mechanics, and game implementation[7,19,21,23,26,44,45,55,58,62-64]

Game design


Game type (genre)[5-7,23,24,27,33,54,66,67]


Game authoring tool[68,69]


Game platform[5,6,33,67,70]


Game engine[2,19,31,71,72]


Database[2,19,43,70,72]


Data protection[3,2,7]


Game objectives (explicit)[1,3,6,19,20,23,33,47,48,53,54,56,65,73]


Narrative[6,7,20,21,23,27,31,40,43,48,53,56,58,62,65,73-76]


Content[6,7,19,23,36,54,65]


Aesthetics and graphics[7,19,20,23,27,29,40,47,49,53,56,67,73]


Rules[6,7,19,20,23,33,35,40,43,48,53,66,73]


Challenge[6,20,29,31,33,35,40,53,56,58,77]


Interactions[6,7,19,20,23,40,43,47,48,53,67,72,73]


Feedback[7,19,20,24,33,53,54,56,58,59,62,67,78,79]

Evaluation Outcomes: High-Level Requirements

Methodological Approach

Out of 74 articles included for review, 11 articles stress the importance of employing an evidence-based, theoretically driven approach toward developing SGH. We identified both the research methodology and game design methodology requirements. The former includes the selection of clear outcome objectives at an early stage of development, as well as an evaluation of the game’s ability to achieve those objectives at a later stage. Without considering an overarching research methodology at the outset, game developers will be challenged, or will be unable, to evaluate their games with well-designed research studies. Kato identified the following 3 questions that ought to be answered by a research methodological approach: Who is your target audience? What is the primary research goal? How can the goals be reached through gameplay (relevant theories and models) [3]? The latter includes a structured approach toward profiling the target audience, assessing content and technical requirements, selection of relevant game mechanics, and the structured translation of outcome objectives and relevant theories and models into the game design.

Multidisciplinary, Participatory Design Process

Out of 74 articles, 21 articles advocate involving stakeholders from various disciplines in the development process. This is not surprising as SGH have emerged at the nexus of a wide variety of disciplines such as game design, software engineering, user experience design, health care, psychology, pedagogy, and clinical research. Various stakeholders were cited as relevant for inclusion in the design process, including research experts, clinical experts, regulatory authorities, and policy makers. Many acknowledge that a multidisciplinary approach poses a challenge, as individuals with differing backgrounds use differing terminology; it also highlights the importance of different elements and may be unaccustomed to working closely with those outside of their field. Nonetheless, this challenge is considered a necessary one. Importantly, 16 articles explicitly suggest also involving the target audience in the process.

A participatory or user-centered design process uses input and opinion from end users to inform a game developer’s choices. Although many acknowledge this as an important criterion for development, particularly in the field of mental health [37,79], there is currently no clear consensus on how the target audience should participate in game design or which elements it should inform. In fact, a recent meta-analysis on SGH with a behavioral focus indicated that certain types of participatory design may be more effective than others. Involving users in user testing and informant roles may be more beneficial than as co-designers, and the involvement in crucial aspects such as game dynamics elicited higher game effectiveness than involvement in trivial aspects such as esthetic components [29].

An iterative development approach was put forward by 6 articles. By developing SGH in segments, testing, and refining along the way, various stakeholders can inform at critical points of the development cycle, and development costs may be reduced.

Project Management Approach

Out of 74 articles, 7 articles highlighted the need for an approach that helps optimize the use of time and resources during development. This is particularly relevant for projects with limited development or research funding and SGH that focus on small market niches with limited potential for return on investment. There are particular challenges related to SGH that need to be managed: (1) helping multidisciplinary teams communicate and work together, (2) the slow process of research evaluation, and (3) the iterative prototyping process on the basis of user and expert feedback [30,38].

SGH developers increasingly use approaches inspired by Agile project management methods with SCRUM [31,39,80]. Agile is highly suited for complex projects where it is difficult to make a comprehensive implementation plan, and where many changes are expected along the way. Therefore, the approach focuses on conducting work in incremental iterations that can absorb new emerging insights and unanticipated changes. It is also based on the principle that multidisciplinary teams should self-manage their work and focuses on fostering communication and cooperation. However, there is currently no consensus on how to best adapt Agile approaches for participatory or user-centered design-focused projects [81,82] and how limits can be set on how far development iterations should go (how much feedback should be incorporated).

Quality Evaluation

Out of 74 articles, 30 articles cited the need to conduct quality evaluations and trials to validate SGH. This criterion is associated with the need to employ a high-level research methodology. The most important aspect of this criterion is likely the word quality, as there have been many trials of SGH, but only a few have reached a standard that can be considered high quality. Although it is evident that conducting a quality trial to validate SGH is a pivotal criterion, there is little consensus among the SGH community on what constitutes a quality trial. Many SGH trials only evaluate aspects, such as user experience, or technological aspects. Although this provides valuable information, it does not automatically allow for an assessment of how effective the game is at achieving the intended outcomes or the purpose for which it was designed. Drawing from the established research standards, a quality trial should include the use of a control group, participant randomization, an adequately powered trial, and objective measures of the primary and secondary outcomes.

Consistent with this thinking, several articles suggest that game developers should strive to carry out RCTs. Although these types of studies may not be necessary or relevant in all cases, RCTs are the still considered the gold standard for evaluating interventions in health care. Here, it is vital for game developers to work alongside stakeholders with expertise to determine the most relevant trial that will validate their games’ claims. The elements that need to be investigated are dependent upon the development stage of SGH. For example, at an earlier stage, SGH stakeholders suggest investigating usability, user experience, and duration of play.

For true evaluation, which typically occurs at a later stage of development, SGH stakeholders have identified the importance of evaluating a game’s efficacy (level to which it achieves intended objectives) in addition to its safety. The need for empirical evidence of efficacy and safety is consistent with requirements of health regulators, should SGH developers intend to have their product approved as a medical device.

Mixed-methods research, where both qualitative and quantitative data are gathered, is becoming an important methodology to investigate complex health-related topics. We identified several articles that highlight the need for better integration of these methods in the SGH research [41-43].

Publish and Disseminate Findings

SGH developers should endeavor to disseminate their findings to the SGH and wider health care community. This criterion was addressed in 5 articles. Consistent with other areas of health care where researchers are urged to publish all results, even negative ones, game developers should follow suit. This provides valuable evidence to the SGH community and may inform other researchers about what did and did not work for a target audience and game design. This is particularly important in assessing the effectiveness of serious games in relation to the constructs used in the design of the game.

Evaluation Outcomes: Low-Level Requirements

In contrast to the high-level, or meta-level, requirements described above, the researchers also identified 20 low-level requirements from the SGH community. These requirements generally fell into 3 main categories: inputs, models and theories, and game design. Inputs represent the information and evidence that are integrated into the game from a conceptual perspective and cover the clinical or scientific content. SGH stakeholders clearly identified the importance of having a strong understanding of the target audience and their needs, clearly defining a research goal from the outset of a project. Models and theories were also identified as key requirements. This category represented the theories describing why a game would be expected to impact intended outcomes, and it also represented the associated link between these theories and game mechanics at the implementation level. SGH stakeholders suggest that without considering and integrating theories and models into SGH development, the resulting games are bound to be ineffective. The final and largest category of requirements contained all components related to the creation of the game itself. The category of game design comprised everything from defining one’s game authoring tool, engine, platform, and genre to the rules, challenges, and feedback that are integrated within the game. As identified in the models and theories categories, SGH stakeholders noted the importance of mapping the game inputs and model and theories to the game implementation choices and mechanics. Without this link, it is not possible to evaluate if the evidence and thinking captured in the former categories have been truly translated into the game.

Proposed Framework

On the basis of the requirements suggested by the SGH research community, we propose a framework for developing SGH that comprises 5 distinct stages (Figure 2). Each stage has a specific focus and is informed by various stakeholders. Several iterations of development may occur within a given stage, progressively refining the SGH on the basis of testing with and feedback from relevant stakeholders. We will describe these stages as well as the stakeholder involvement in more detail below.

Figure 2. Proposed framework for developing serious games for health. SGH: serious games for health.
View this figure
Stage 1: Scientific Foundations

Sound scientific foundations for the SGH should be established at the earliest stage of development. This will ensure that the final product is relevant, theoretically driven, and evidence based, in line with governing research methodological approaches. Although most developers tend to initiate the development process with a specific idea for an intervention in mind, the overarching objective of this stage is to assess at least conceptually and theoretically, on the basis of objective criteria, whether there is indeed a relevant medical unmet need for a clearly defined target audience who can be addressed with an SGH intervention. This stage typically comprises a topline review of the available literature on the target audience, disease status and impact, available treatment modalities, relevant clinical outcomes, psychosocial aspects (if any), and the governing health care landscape. To approach this task methodically, we propose that developers should focus on answering the following 4 questions: (1) Who is the target audience? (2) What outcome needs to be achieved? (3) How might SGH achieve this outcome? and (4) How can we evaluate whether SGH achieve the intended outcome?

Who Is the Target Audience?

A first, limited profile of the intended end users should be constructed. Information can be obtained through literature review, explorations of Web materials such as patient fora and websites, or consultation with subject matter experts (eg, medical specialists and patient organizations). At this stage, the profile should at minimum cover who the target audience is, the context in which the people function, the specific problems they face, and what their unmet needs are (eg, what alternatives are available to them and how well do these other interventions address those needs?). On top of this, some details may be required that are game- or topic-specific. For example, if the game intends to help children with disabilities improve their motor skills, it is important to understand the disease status and specific medical needs. For a game aimed at helping patients with schizophrenia reintegrate in society, the profile should also include an overview of the other stakeholders who play a role in these patients’ lives and the relations among them (psychosocial details). At the other end of the spectrum, a game intended to educate medical professionals on how to handle ethical dilemmas would require some level of insight into the context in which these professionals make decisions, their learner profiles, and pedagogical needs. In stage 2 of the development, this first limited profile will be broadened to also include information regarding the specific game design needs of the target audience (eg, user experience needs and usability needs).

What Outcome Needs to Be Achieved?

Outcome objectives should be clearly formulated before any creative or technical development starts. These outcomes should be (medically) relevant, based on objective criteria. Due to the nature of SGH and their ability to educate, empower, and address multiple domains of health, we propose a biopsychosocial approach toward identifying relevant outcome objectives [83]. This model considers not only the biological factors of human functioning in the context of health but also the psychological and social factors. An often-cited example of where this is particularly relevant is the issue of therapy compliance. A majority of patients do not comply with the treatment regimens that could save their lives [84]. The solution to compliance issues is clearly complex, and psychological and behavioral factors play a prominent role. This requires developers to evaluate the biological, psychological, and social context of a disease or health condition [85].

Although outcome objectives may range from clinical to pedagogical, psychological, or behavioral, it is important to primarily identify a single primary outcome objective. This will steer the subsequent steps of game development and provide the greatest opportunity for success. This does not exclude a developer from identifying secondary (or even tertiary) outcome objectives as well. In fact, many cases will require the identification of various secondary outcomes that are closely linked to the primary outcome, and these need to be incorporated in the game construct as well. To give a few examples, if improved therapy compliance is the primary outcome objective, secondary outcomes may be to improve the quality of the patient’s relationships or provide a more stable home environment (eg, for psychiatric disorders), or the secondary outcomes may be to overcome the patient’s misconceptions or erroneous beliefs about the therapy (eg, for chronic patients fearing dependence on long-term medication) [86,87] or to make physical rehabilitation exercises more fun and rewarding (eg, for kids with motor skill disorders).

How Might Serious Games for Health Achieve This Outcome?

An often-overlooked step in SGH development is to formulate a hypothesis of how a game might achieve the intended outcome objective(s). Formulating such a hypothesis is a vital step toward the purposeful design of SGH and the evaluation of their causal effect on the outcome. First, developers should identify the outcome determinants. Outcome determinants are the underlying factors or parameters that directly or indirectly determine or influence the outcome objective. For example, if the primary outcome is to reduce peri-operative pain in children and the secondary outcome objectives are to reduce these children’s peri-operative anxiety and stress (which are closely related to the primary outcome objective), it is important to establish what underlying factors contribute to pain, anxiety, and stress in children in this situation and evaluate which of these factors a game might positively or negatively impact, and how. The literature reveals that pain has sensory, emotional, cognitive, and behavioral components that are interrelated with environmental, developmental, sociocultural, and contextual factors [88]. One determinant is the level of pain medication administered to the child after surgery, with inadequate levels resulting in more pain. Delving deeper into the research literature reveals that parents play a key role in managing their child’s pain in the home setting postsurgery. Any misconceptions parents have about pain medication may result in inadequate levels of pain relief, thereby increasing the child’s pain experience. In this case, SGH may leverage pedagogical models and approaches toward educating parents about pain relief and correcting erroneous beliefs. Another determinant of peri-operative pain, anxiety, and stress is lack of control. Developers should therefore investigate whether SGH may help increase a child’s feeling of control of the situation, for instance, through teaching coping skills or by allowing them to freely explore the peri-operative setting and events associated with it, so that they can anticipate what lies ahead.

The identification of these determinants and relevant underlying models or theories should occur in consultation with experts from relevant disciplines. From a biopsychosocial perspective, the link between psychosocial determinants and clinical outcome also needs to be understood and integrated in the game design. In addition, the role of the target audience as well as all relevant stakeholders needs to be evaluated. As such, this step of the process is one of identifying the various factors that contribute to the problem, grouping them, charting how they relate to and impact one another, and establishing hierarchies and relative weights of impact and importance. These insights will inform the game construct, narrative scenarios, and mathematical algorithms further on in the development process.

How Can We Evaluate Whether Serious Games for Health Achieve the Intended Outcome?

Before development starts, developers need to think ahead of how they plan to evaluate whether the game achieves the intended outcome(s). In 2012, Kato first formulated guidelines for conducting high-quality evaluations of SGH [1]. These suggest conducting randomized (clinical) trials that include adequate numbers of participants as well as control groups, the use of objective outcome measures alongside self-reports, monitoring and reporting potential negative side effects, and consulting research experts early on to guide the design of quality trials (eg, measures, n numbers, statistical power, and trial length).

Game evaluation should also include an evaluation of broader intervention characteristics, such as perceived relevance, user experience, and user friendliness. The particular characteristics that need to be evaluated vary depending on the objectives, and they may include satisfaction of needs (competence, autonomy, and relatedness), ability to engage, level of motivation, and competence autonomy. Ideally, a mixed-method methodology is used, in which both quantitative (eg, surveys) and qualitative (eg, focus groups and interviews) data are collected for analysis and evaluation.

This evaluation can be done throughout the development process through a series of iterative tests with the target audience and other stakeholders, using standard measures such as Intrinsic Motivation Inventory and Player Experience of Need Satisfaction Scale [89,90].

Stage 2: Design Foundations

Game developers can draw from a wide range of game mechanics, design, and technological features to construct SGH. If the SGH are to achieve the intended outcomes, the choice of these game mechanics, design, and technological features should be guided by the scientific foundations established in stage 1. It is imperative to translate the theoretical basis into relevant, implementable game design elements. This stage therefore aims to answer the following 3 questions: (1) Which game mechanics are best suited to achieve the intended outcome objectives? (2) What are the design requirements? and (3) How can the game design best accommodate the evaluative trial?

Which Game Mechanics Are Best Suited to Achieve the Intended Outcome Objectives?

Game mechanics are rules or methods that define the interactions and flow of a game session. They describe interactions, game conditions, and triggers in an abstract manner. The most frequently employed game mechanics in electronic health are currently rewards and feedback [44], but other examples include turn-taking, story, penalties, realism, and protégé effect. In the past, game developers often decided upon a game genre before selecting what game mechanics to use. However, as game mechanics are more instrumental toward achieving the intended outcome objectives, developers should first work together with relevant subject matter experts to map the outcome objectives, models, and theories identified in stage 1 onto relevant game mechanics before settling on a particular game genre [44]. In mapping the scientific foundations onto these mechanics, developers gain insight into which game mechanics should be used to effectively achieve the intended outcomes. Although this is a relatively novel approach, there are currently several well-documented examples in the research literature of how this can be done for SGH that have a pedagogical or behavioral focus [45,62,63]. These types of SGH often have outcome objectives that pertain to either understanding or the acquisition of a specific skill set (eg, communication skills and coping skills). Depending on the type of outcome envisaged, there may be layers of intermediate learning objectives that need to be addressed. Here, the pedagogical or behavioral intents should be mapped to a low-level game mechanic implementation. In 2015, Arnab proposed a model for translating learning objectives into learning mechanics and mapping these to relevant game mechanics [45]. This so-called learning mechanics-game mechanics model guides developers in the development of more effective, pedagogy-driven SGH, as it ensures that game mechanics are chosen on the basis of their ability to contribute toward the intended outcomes. In the example of reducing peri-operative pain, stress, and anxiety in children, one such learning objective may include remembering the sequence of events for the upcoming procedure (knowing what to expect and do). This involves the thinking skills understanding and retention. Several learning mechanics address these thinking skills: exploration, repetition, and planning. Each of these learning mechanics can in turn be mapped onto one or more game mechanics, for example, story, cascading information, and strategy and planning. As such, the scientific foundations established in stage 1 can be translated into the game construct.

What Are the Design Requirements?

At this point, the target audience profile needs to be broadened to guide the design choices. Although the specifics will depend on the objective and scope of SGH, the objective is to gain insight into (1) the context of use and (2) the reality of the target audience. What context will the tool be used in? Will there be access to special equipment or technical support? Will the tool be used at home or in hospital? How realistic does the tool need to be (level of fidelity and immersion)? If it needs to be realistic, what characters does the target audience meet or interact with? What type of environments do they move about in? What situations or dilemmas do they typically encounter? In addition, information regarding optimal user experience for the target audience should be collected. This includes computer literacy skill levels, literacy and numeracy levels, and possible physical or mental limitations that may pose restrictions on game design (eg, epilepsy, auditory problems, and limited motor function). This type of information can be gathered through interviews, time-and-motion exercises (shadowing a typical user for a day), or focus groups with the target audience or relevant experts.

How Can the Game Design Best Accommodate the Evaluative Trial?

Are there any design considerations with respect to the future evaluation of the SGH? For instance, if data need to be collected, should this data collection be included in the game design? (eg, tracking user response time, motion ranges) Will it be collected out with the game format (eg, pre and postgame interviews, clinical scales, and biologic sampling). Does it require live feedback or investigator intervention during game play? Are there any design considerations for use in a clinical environment? Should the game design include components that can help track or assess user experience (eg, level of immersive play, eye tracking, etc)?

Stage 3: Game Development

Once stage 2 has been completed, developers should have sufficient scientifically grounded input to guide the practical development of the game. Various approaches can be used depending on the complexity, the developer’s resources, and software and technological skills, but overall, the process comprises the selection and development of the (1) game genre, (2) game rules, (3) content, and (4) visuals and user interface. This stage ideally occurs in an iterative, participatory manner, involving key stakeholders such as clinical experts and target users to informally test and refine the tool along the way.

Game Genre

The scientific and design foundations developed in stages 1 and 2 should now enable developers to select the most appropriate interface genre (eg, first person, third person, and isometric) and procedure genre (eg, strategy game, adventure game, and shooter game) for the intended target audience and context of use. The genre chosen should facilitate the incorporation of the game mechanics and design requirements identified under stage 2.

Game Authoring Tool

There are several authoring tools available for game development, and it is important to assess upfront which authoring tool is most suitable for the project. This will largely depend not only on the technical capabilities available in the team (eg, team members with specific coding skills) but also on whether future administrators of the SGH will need to make their own modifications, such as adding new narratives or including new data measurements. Considerations should be given to open-source authoring tools versus licensed authoring platforms.

Game Rules

Developers can now draw up a set of game rules that specifies how the player’s actions impact the game environment. Depending on the intended purpose of the tool, these rules may need to be consistent and transparent (to allow players to strategize on the basis of their knowledge of the rules) or hidden or unpredictable (to force players to truly reflect on their choices rather than making decisions that help raise game scores). Such rules are often described in mathematical algorithms that govern the tool’s programming. When the SGH need to closely reflect reality to achieve the intended outcome objectives, for instance, through use of realistic narratives or life-like responses to in-game decision making, developers will need to translate the relations, hierarchies, impacts, and relative weights of importance of the outcome determinants, identified in stage 1, into a mathematical algorithm. This will facilitate procedurally generated narrative branching and can drive feedback and reward approaches. To stimulate flow and user engagement, developers can also build in rules that adapt game difficulty and other game-play elements to the performance or physical or mental state of the user.

Game Content

The amount of content required in a game will vary substantially depending on the intended objectives. Many SGH require at least some instructional content or a narrative that ties everything together. When SGH have a large pedagogical or behavioral focus, the narratives can become more elaborate, ranging from linear stories to nonlinear stories that have branching and even offer multiple endings. Within the context of health, narratives are a valuable resource to generate an understanding of the impact of an illness on the patient’s life and well-being [91,92]. Narratives are an everyday medium that people use to communicate information to one another, and therefore they are a familiar format to users [93]. Narratives are perceived as providing essential emotional and social information not usually found within routine resources that lend meaning and perspective to a patient’s predicament [94].

Developers should develop the game content in function of the intended outcomes. Linear story lines may be less time consuming, but they also tend to reduce the potential efficacy of the narrative as a persuasive mechanism as it is not responsive to the users, their state, or in-game decisions. Many SGH are designed using a one-size-fits-all approach; however, recent research shows that this approach may not be effective as different types of people are motivated by different persuasive strategies [95], and a strategy that worked well with one group of people may actually demotivate a different group [95,96]. Personalization has also been shown to be important for successful impact [97]. The relevance of SGH is often directly related to their ability to capture the patient’s unique reality and circumstances in the content [93,98]. In addition, features that allow patients to self-personalize content may promote autonomy and empower patients to take ownership over health care decisions [99]. Developers should also address (health) literacy and numeracy profiles of the intended target audience to maximize chances of success [100-104]. Therefore, building on the target audience profile and design requirements identified in stage 2 allows developers to take a more informed approach toward game content.

Game Visuals and User Interface

On the basis of the specific target audience and design requirements, a theme needs to be chosen, which specifies the overall look and feel of the tool (colors, sounds, environments, characters, navigation, interface, etc). At the same time, developers will need to assess what level of visual conceptualization will be needed. In some cases, the use of archetypical symbols or icons may be warranted to convey complex concepts either to avoid information overload or to eliminate bias [101,105,106]. Graphics have also shown to impact the emotional response of participants [107].

Stage 4: Game Evaluation

The game has now been developed, informally tested, and refined with users, and it should be ready for (clinical) evaluation. Once the trial sites and investigators have been chosen, ethical committee or other approvals have been granted, and users have been recruited for the evaluation study, developers can commence the evaluation, analysis, and assessment of whether the tool successfully achieves the intended outcomes. This stage ideally occurs in consultation with relevant research experts who can guide and oversee the evaluation studies and support the analysis of collected data.

Stage 5: Implementation

On the basis of the findings of stage 4, developers may wish to further refine and reevaluate updated versions of the tool or proceed immediately with rollout toward the intended target audience. Regardless of the outcomes of the game evaluation studies, developers should try to disseminate the study findings to the wider SGH research community, as this will help further advance the field. Even publishing and communicating null results may provide insight on how to optimize SGH interventions and provide guidance on the best practices and pitfalls to avoid. If a game is successfully validated and implemented or marketed for the intended target audience, efforts should be made to collect user data in the field, to help monitor for adverse events (if relevant), or to further explore the validity and use of the SGH.

Stakeholders

More research should be done to identify which stakeholders should be involved in the design process, to understand how to engage them, and at what stage of the development. We propose to consider at least the following 4 stakeholders.

Subject Matter Expert

At the earliest stage of development, subject matter experts are well placed to provide input and guidance on the 4 questions that should be answered when establishing the scientific foundations.

Target Audience

When deciding on the level of user involvement, one should balance the need for input from users with the availability of resources such as time and funding [32]. One should only involve end users at relevant stages of development. We propose to engage them at specific points in stages 2, 3, and 4, when their input and participation are most likely to yield valuable, accurate information and feedback. At stage 2, end users should be engaged to broaden the profile research and identify their specific design requirements that are crucial to tool effectiveness (ie, not esthetic design perspectives). The purpose of the contact is to determine in more detail who the end users are, which subgroups exist, what their socioeconomic backgrounds are, their day-to-day reality, and other aspects that may inform game design, such as literacy levels, numeracy levels, computer skills, and understanding of subject matter. Many established formats to engage end users are available to developers. Examples include time and motion exercises in which developers shadow end users during a typical day in their life or during a relevant event, such as a hospitalization.

At stage 3, end users should be actively engaged in the testing of early prototypes of the game to gather feedback on such aspects as user experience, content relevance, realism, graphic design features (minimal), and preliminary assessment of the achievability of outcome objectives. It is important to note here that not all user feedback needs to be incorporated. To avoid feature creep and ensure effective use of project resources, subject matter experts can help evaluate which feedback should be prioritized. Ideally, an emphasis is placed on feedback that will help enhance user uptake and SGH effectiveness.

At stage 4, end users should be recruited into a quality trial to validate the game. End users who participate in trials should not be involved in the earlier stages of the development.

(Clinical) Research Expert

Research experts should be engaged to advise on scientific approach and trial design at an early stage of development and ideally during the trial.

Business Expert

To ensure market readiness and effectively implement and roll out the SGH to the market, a business expert should be consulted, ideally no later than stage 3.

Aside from these 4 stakeholders, it may be relevant to consult with regulators, health care professionals, patient organizations, health technology assessments, and others throughout the various stages of the development process.

Working with multiple stakeholders carries a specific set of challenges such as clear alignment on roles and responsibilities, finding a common language, understanding of limitations, and consensus on priorities and outcomes [30,55]. These may be overcome, at least in part by (1) educating stakeholders about the development process and the tools and methods used, by means of interactive, hands-on workshops, (2) forming an advisory board of key stakeholders which meets on a regular basis to discuss the project, and (3) by assigning a project manager who functions as the go-to person for all stakeholders involved and who can make final judgment calls in case of conflict, in line with the project’s intended objectives and within the stipulated resource limitations.


A review of existing literature, recommendations, and guidelines on SGH development has allowed us to formulate a framework for developing theory-driven, evidence-based SGH that represents many of the requirements set out by SGH stakeholders in the research literature. The framework covers all aspects of the development process (scientific, technological, and design) and is transparently described in sufficient detail to allow developers to implement it in a wide variety of projects, irrespective of discipline, health care segments, or focus. Adoption of such a consensus framework by the wider SGH research community is a first step toward increasing probability of success and raising the quality of SGH by providing the necessary evidence required by stakeholders. Moreover, it would also enhance the credibility of SGH developers and allow them to achieve a sustainable market share.

Conflicts of Interest

SV is a paid consultant of MindBytes BVBA, CB is an employee of MindLab Interactive AI Inc and previously served as a paid consultant of MindBytes BVBA, and GVS is the owner of MindBytes BVBA and MindLab Interactive AI Inc.

  1. Kato PM. Evaluating efficacy and validating games for health. Games Health J 2012 Feb;1(1):74-76. [CrossRef] [Medline]
  2. Kato PM. The role of the researcher in making serious games for health. In: Arnab S, Debattista K, Dunwell I, editors. Serious Games for Healthcare: Applications and Implications. Hershey, Pennsylvania: IGI Global; 2013:213-231.
  3. Graafland M, Dankbaar M, Mert A, Lagro J, De Wit-Zuurendonk L, Schuit S, et al. How to systematically assess serious games applied to health care. JMIR Serious Games 2014 Nov;2(2):e11 [FREE Full text] [CrossRef] [Medline]
  4. Drummond D, Hadchouel A, Tesnière A. Serious games for health: three steps forwards. Adv Simul 2017;2(1):3 [FREE Full text] [CrossRef]
  5. De Troyer O, Janssens E. Supporting the requirement analysis phase for the development of serious games for children. Int J Child-Computer Interact 2014 May;2(2):76-84. [CrossRef]
  6. Wattanasoontorn V, Boada I, Garcia R, Sbert M. Serious games for health. Entertain Comput 2013;4(4):231-247. [CrossRef]
  7. Mitgutsch K, Alvarado N. Purposeful by design? A serious game design assessment framework. In: Proceedings of the International Conference on the Foundations of Digital Games. 2012 Presented at: FDG '12; 2012; Vancouver p. 121-128. [CrossRef]
  8. Goldstein N, Kemp K, Leff S, Lochman J. Guidelines for adapting manualized interventions for new target populations: a step-wise approach using anger management as a model. Clin Psychol (New York) 2012 Dec 01;19(4):385-401 [FREE Full text] [CrossRef] [Medline]
  9. Facey K, Boivin A, Gracia J, Hansen H, Lo Scalzo A, Mossman J, et al. Patients' perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care 2010 Jul;26(3):334-340. [CrossRef] [Medline]
  10. FDA. Draft Guidance for industry, FDA staff, and other stakeholders: patient preference information- submission, review in PMAs, HDE applications, and De Novo requests, and inclusion in device labeling. Silver Spring, MD: FDA; 2015:a.
  11. Anderson M, Kimberly McCleary K. On the path to a science of patient input. Sci Transl Med 2016 Dec 27;8(336):11. [CrossRef] [Medline]
  12. Tate R, Haritatos J, Cole S. HopeLab's approach to re-mission. Int J Learn Media 2009;1(1):29-35. [CrossRef]
  13. Loerzel V, Clochesy J, Geddie P. Using a community advisory board to develop a serious game for older adults undergoing treatment for cancer. Appl Nurs Res 2018 Feb;39:207-210 [FREE Full text] [CrossRef] [Medline]
  14. Aldrich C. Virtual worlds, simulations, and games for education: a unifying view. Innovate 2009;5(5) [FREE Full text]
  15. Kato PM. Video games in health care: closing the gap. Rev Gen Psychol 2010;14(2):113-121. [CrossRef]
  16. Baranowski T, Blumberg F, Buday R, DeSmet A, Fiellin LE, Green CS, et al. Games for health for children - current status and needed research. Games Health J 2016 Feb;5(1):1-12 [FREE Full text] [CrossRef] [Medline]
  17. Oliver M. An introduction to the evaluation of learning technology. J Educ Techno Soc 2000;3(4):20-30.
  18. de Freitas S, Oliver M. How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Comput Educ 2006 Apr;46(3):249-264. [CrossRef]
  19. Thillainathan N. Kassel University. 2013. A model driven development framework for serious games   URL: http://pubs.wi-kassel.de/wp-content/uploads/2013/07/JML_427.pdf [accessed 2019-02-19] [WebCite Cache]
  20. Shi Y, Shih J. Game factors and game-based learning design model. Int J Comput Games Technol 2015;2015:1-11. [CrossRef]
  21. Van Broeckhoven F, Vlieghe J, De Troyer O. Mapping between pedagogical design strategies and serious game narrarives. 2015 Presented at: 7th International Conference on Games and Virtual Worlds for Serious Applications; 12 October, 2015; Skovde, Sweden p. 123-130.
  22. Moher D, Liberati A, Tetzlaff J, Altman D, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
  23. Bul KC, Kato PM, Van der Oord S, Danckaerts M, Vreeke LJ, Willems A, et al. Behavioral outcome effects of serious gaming as an adjunct to treatment for children with attention-deficit/hyperactivity disorder: a randomized controlled trial. J Med Internet Res 2016 Feb 16;18(2):e26 [FREE Full text] [CrossRef] [Medline]
  24. DeLaughter KL, Sadasivam RS, Kamberi A, English TM, Seward GL, Chan SW, et al. Crave-out: a distraction/motivation mobile game to assist in smoking cessation. JMIR Serious Games 2016 May 26;4(1):e3 [FREE Full text] [CrossRef] [Medline]
  25. De Smet A, Van Cleemput K, Bastiaensens S. Bridging behavior science and gaming theory: using the intervention mapping protocol to design a serious game against cyberbullying. Comput Human Behav 2016;56:337-351. [CrossRef]
  26. Gentile M, Guardia DL, Grande V, Ottaviano S, Allegra M, La V. International Journal of Serious Games. 2014. An Agent Based Approach to designing Serious Game: the PNPV case study   URL: http://www.academia.edu/19591978/An_Agent_Based_approach_to_design_Serious_Game [accessed 2019-02-21] [WebCite Cache]
  27. Marfisi-Schottman I, George S, Tarpin-Bernard F. Tools and methods for efficiently designing serious games. 2010 Presented at: 4th Europeen Conference on Games Based Learning ECGBL2010; 21-22 October, 2010; Copenhagen, Denmark p. 226-234.
  28. Amengual Alcover E, Jaume-I-Capó A, Moyà-Alcover B. PROGame: a process framework for serious game development for motor rehabilitation therapy. PLoS One 2018;13(5):e0197383 [FREE Full text] [CrossRef] [Medline]
  29. DeSmet A, Thompson D, Baranowski T, Palmeira A, Verloigne M, De Bourdeaudhuij I. Is participatory design associated with the effectiveness of serious digital games for healthy lifestyle promotion? A meta-analysis. J Med Internet Res 2016 Apr 29;18(4):e94 [FREE Full text] [CrossRef] [Medline]
  30. Korhonen T, Halonen R, Ravelin R, Kemppainen J, Koskela K. A multi-disciplinary approach to serious game development in the health sector. 2017 Presented at: Mediterranean Conference on Information Systems Proceedings; 2017; Mediterranean p. 1-14.
  31. Rico-Olarte C, López D, Narváez S, Farinango C, Pharow P. HapHop-Physio: a computer game to support cognitive therapies in children. Psychol Res Behav Manag 2017;10:209. [CrossRef]
  32. de Beurs D, van Bruinessen I, Noordman J, Friele R, van Dulmen S. Active involvement of end users when developing web-based mental health interventions. Front Psychiatry 2017;8:72 [FREE Full text] [CrossRef] [Medline]
  33. Brox E, Fernandez-Luque L, Tøllefsen T. Healthy gaming - video game design to promote Health. Appl Clin Inform 2011;2(2):128-142 [FREE Full text] [CrossRef] [Medline]
  34. Khaled R, Vasalou A. Bridging serious games and participatory design. Int J Child Comput Interact 2014 May;2(2):93-100. [CrossRef]
  35. Mader S. A game design method for therapeutic games. 2016 Presented at: 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES); September 7-9, 2016; Barcelona, Spain. [CrossRef]
  36. Castel A, Qasmieh S, Greenberg D, Ellenberger N, Howell TH, Griffith C, et al. Digital gaming to improve adherence among adolescents and young adults living with HIV: mixed-methods study to test feasibility and acceptability. JMIR Serious Games 2018 Oct 15;6(4):e10213 [FREE Full text] [CrossRef] [Medline]
  37. Dekker MR, Williams AD. The use of user-centered participatory design in serious games for anxiety and depression. Games Health J 2017 Dec;6(6):327-333. [CrossRef]
  38. Kato P, de Klerk S. Serious games for assessment: welcome to the jungle. JATT 2017;18(1):1-6 [FREE Full text]
  39. Aulia A, Tanzil F, Yudhawanto MF, Ramadhan IA, Ismail R. The influence of game "nutrients hero" for healthy diet. Procedia Comput Sci 2017;116:81-86. [CrossRef]
  40. Baaden M, Delalande O, Ferey N, Pasquali S, Waldispühl J, Taly A. Ten simple rules to create a serious game, illustrated with examples from structural biology. PLoS Comput Biol 2018 Mar 8;14(3):e1005955. [CrossRef]
  41. Thompson D. Talk to me, please!: the importance of qualitative research to games for health. Games Health J 2014 Jun;3(3):117-118. [CrossRef] [Medline]
  42. Guetterman TC, Fetters MD, Creswell JW. Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Ann Fam Med 2015 Nov 09;13(6):554-561. [CrossRef]
  43. Ingadottir B, Blondal K, Thue D, Zoega S, Thylen I, Jaarsma T. Development, usability, and efficacy of a serious game to help patients learn about pain management after surgery: an evaluation study. JMIR Serious Games 2017 May 10;5(2):e10 [FREE Full text] [CrossRef] [Medline]
  44. Mert A. A medical serious games framework hierarchy for validity. 2018 Presented at: 2018 10th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games); September 5-7, 2018; Würzburg, Germany p. 1-2. [CrossRef]
  45. Arnab S, Lim T, Carvalho MB, Bellotti F, de Freitas S, Louchart S, et al. Mapping learning and game mechanics for serious games analysis. Br J Educ Technol 2014 Jan 05;46(2):391-411. [CrossRef]
  46. All A, Nunez Castellar E, Van Looy J. Measuring effectiveness in digital game-based learning: a methodological review. Int J Serious Games 2014 Jun 19;1(2):3-21. [CrossRef]
  47. Catalano CE, Luccini AM, Mortara M. Guidelines for an effective design of serious games. Int J Serious Games 2014 Feb 03;1(1). [CrossRef]
  48. Dondlinger M. Educational video game design: a review of the literature. Journal of Applied Educational Technology 2007;4(1) [FREE Full text]
  49. Iten N, Petko D. Learning with serious games: is fun playing the game a predictor of learning success? Br J Educ Technol 2014 Nov 11;47(1):151-163. [CrossRef]
  50. Landers R. Developing a theory of gamified learning. Simul Gaming 2015 Jan 15;45(6):752-768. [CrossRef]
  51. Mayer I. Towards a comprehensive methodology for the research and evaluation of serious games. Procedia Comput Sci 2012;15:233-247. [CrossRef]
  52. Shepherd M, Fleming T, Lucassen M, Stasiak K, Lambie I, Merry SN. The design and relevance of a computerized gamified depression therapy program for indigenous māori adolescents. JMIR Serious Games 2015 Mar 03;3(1):e1 [FREE Full text] [CrossRef] [Medline]
  53. Starks K. Cognitive behavioral game design: a unified model for designing serious games. Front Psychol 2014;5:28 [FREE Full text] [CrossRef] [Medline]
  54. Yusoff A, Crowder R, Gilbert L. A conceptual framework for serious games. 2009 Presented at: Ninth IEEE International Conference on Advanced Learning Technologies; 15-17 July, 2009; Riga, Latvia.
  55. Kelley C, Wilcox L, Ng W, Schiffer J, Hammer J. Design features in games for health: disciplinary and interdisciplinary expert perspectives. DIS (Des Interact Syst Conf) 2017 Jun;2017:69-81 [FREE Full text] [CrossRef] [Medline]
  56. Cugelman B. Gamification: what it is and why it matters to digital health behavior change developers. JMIR Serious Games 2013 Dec 12;1(1):e3 [FREE Full text] [CrossRef] [Medline]
  57. Alaribe I. Design a serious game to teach teenagers with intellectual disabilities how to use public transportation. Procedia Soc Behav Sci 2015 Feb;176:840-845. [CrossRef]
  58. Kamel Boulos MN, Gammon S, Dixon MC, MacRury SM, Fergusson MJ, Miranda Rodrigues F, et al. Digital games for type 1 and type 2 diabetes: underpinning theory with three illustrative examples. JMIR Serious Games 2015 Mar 18;3(1):e3 [FREE Full text] [CrossRef] [Medline]
  59. Alves T. Flow adaptation in serious games for health. 2018 Presented at: 6th International Conference on Serious Games and Applications for Health; May 16-18, 2018; Vienna, Austria. [CrossRef]
  60. Tziraki C, Berenbaum R, Gross D, Abikhzer J, Ben-David B. Designing serious computer games for people with moderate and advanced dementia: interdisciplinary theory-driven pilot study. JMIR Serious Games 2017 Jul 31;5(3):e16 [FREE Full text] [CrossRef] [Medline]
  61. Protopsaltis A. Experiential learning in a virtual environment.: Serious Games Institute; 2010.   URL: http//www.seriousgamesinstitute.co.uk [accessed 2015-06-04] [WebCite Cache]
  62. Lim T, Louchart S, Suttie N. Narrative Serious Game Mechanics (NSGM) - Insights into the narrative-pedagogical mechanism. In: Games for Training, Education, Health and Sports. Cham: Springer; 2014:23-34.
  63. Lim T, Carvalho M, Bellotti F, Arnab S, de Freitas S, Louchart S, et al. Citeseerx.ist.psu. 2012. The LM-GM framework for Serious Games Analysis   URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.678.6172&rep=rep1&type=pdf
  64. Suttie N, Louchart S, Lim T, Macvean A, Westera W, Brown D, et al. In persuit of a 'serious games mechanics' a theoretical framework to analyse relationships between 'game' and 'pedagogical aspects' of serious games. Procedia Comput Sci 2012;15:314-315. [CrossRef]
  65. Vallejo V, Wyss P, Rampa L, Mitache A, Müri RM, Mosimann U, et al. Evaluation of a novel Serious Game based assessment tool for patients with Alzheimer's disease. PLoS One 2017;12(5):e0175999 [FREE Full text] [CrossRef] [Medline]
  66. Kim S, Kim HJ, Hwang HC, Hong JS, Bae S, Min KJ, et al. The effects of a serious game on depressive symptoms and anxiety in breast cancer patients with depression: a pilot study using functional magnetic resonance imaging. Games Health J 2018 Oct 31 (forthcoming). [CrossRef] [Medline]
  67. Bellotti F, Berta R, De Gloria A. Designing Serious Games for Education: from Pedagogical principles to Game Mechanisms. 2011 Presented at: Proceedings of the European Conference on Games-based Learning; 2011; Athens, Greece p. 26-34.
  68. Martínez-Pernía D, Núñez-Huasaf J, del Blanco Á, Ruiz-Tagle A, Velásquez J, Gomez M, et al. Using game authoring platforms to develop screen-based simulated functional assessments in persons with executive dysfunction following traumatic brain injury. J Biomed Inform 2017 Oct;74:71-84. [CrossRef]
  69. Slootmaker A, Hummel H, Koper R. Evaluating the usability of authoring environments for serious games. Simul Gaming 2017 Apr 24;48(4):553-578. [CrossRef]
  70. Johnson CM, McIlwain S, Gray O, Willson B, Vorderstrasse A. Creating a sustainable collaborative consumer health application for chronic disease self-management. J Biomed Inform 2017 Dec;71:198-206 [FREE Full text] [CrossRef] [Medline]
  71. Carbajal S, Polomeni F, Mugica J. An emotional engine for behavior simulators. Int J Serious Games 2015;2(2):57-67. [CrossRef]
  72. Cangas A, Navarro N, Parra JM, Ojeda JJ, Cangas D, Piedra JA, et al. Stigma-stop: a serious game against the stigma toward mental health in educational settings. Front Psychol 2017;8:1385 [FREE Full text] [CrossRef] [Medline]
  73. Plohn T, Louchart S, Aalberg T. Dynamic pervasive storytelling in long lasting learning games. EJEL 2015;13(3):193-205.
  74. Ayramo S. Narrative-Based Serious Games. In: Serious Games on the Move. Vienna: Springer; 2011.
  75. Fanfarelli J. Stars, University of Central Florida. 2014. The effects of narrative and achievements on learning in a 2D platformer video game   URL: https:/​/stars.​library.ucf.edu/​cgi/​viewcontent.cgi?referer=https:/​/www.​google.com/​&httpsredir=1&article=5772&context=etd
  76. Gaydos M. Seriously considering design in educational games. Educ Res 2015 Dec;44(9):478-483. [CrossRef]
  77. Hamari J, Shernoff DJ, Rowe E, Coller B, Asbell-Clarke J, Edwards T. Challenging games help students learn: an empirical study on engagement, flow and immersion in game-based learning. Comput Human Behav 2016 Jan;54:170-179. [CrossRef]
  78. Burgers C, Eden A, Van Engelenburg M. How feedback boosts motivation and play in a brain-training game. Comput Human Behav 2015;48:94-103. [CrossRef]
  79. Ninaus M, Pereira G, Stefitz R. Game elements improve performance in a working memory training task. Int J Serious Games 2015;2(1):3-16.
  80. Baharum A, Bahar IA, Fatah NS, Ismail R, Hanapi R, Fabeil NF. Serious game for phobia treatment in Malaysia. Adv Sci Lett 2018 Mar 01;24(3):1823-1828. [CrossRef]
  81. Salah D, Paige R, Cairns P. A systematic literature review for agile development processesuser centred design integration. 2014 Presented at: Proceedings of the 18th international conference on evaluation and assessment in software engineering; 2014; New York.
  82. Lárusdóttir M, Cajander Å, Gulliksen J, Cockton G, Gregory P, Salah D. On the integration of user centred design in agile development. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. 2014 Presented at: NordiCHI '14; October 26-30, 2014; Helsinki, Finland. [CrossRef]
  83. Engel GL. The need for a new medical model: a challenge for biomedicine. Science 1977 Apr 8;196(4286):129-136. [Medline]
  84. Partridge A, Kato P, DeMichele A. Adherence to oral cancer therapies: challenges and opportunities. In: ASCO 2009 Educational Book. Alexandria, VA: American Society of Clinical Oncology; 2009:124-128.
  85. Jin J, Sklar GE, Min SO, Chuen LS. Factors affecting therapeutic compliance: a review from the patient's perspective. Ther Clin Risk Manag 2008 Feb;4(1):269-286 [FREE Full text] [Medline]
  86. Apter AJ, Boston RC, George M, Norfleet AL, Tenhave T, Coyne JC, et al. Modifiable barriers to adherence to inhaled steroids among adults with asthma: it's not just black and white. J Allergy Clin Immunol 2003 Jun;111(6):1219-1226. [Medline]
  87. Bender BG, Bender SE. Patient-identified barriers to asthma treatment adherence: responses to interviews, focus groups, and questionnaires. Immunol Allergy Clin North Am 2005 Feb;25(1):107-130. [CrossRef] [Medline]
  88. American Academy of Pediatrics. Committee on Psychosocial Aspects of Child and Family Health, Task Force on Pain in Infants‚ Children‚ and Adolescents. The assessment and management of acute pain in infants, children, and adolescents. Pediatrics 2001 Sep;108(3):793-797. [Medline]
  89. Birk M, Atkins C, Bowey J, Mandryk R. Fostering intrinsic motivation through avatar identification in digital games. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016 Presented at: CHI '16; May 07-12, 2016; San Jose, California, USA p. 2982-2995. [CrossRef]
  90. Ryan R, Rigby C, Przybylski A. The motivational pull of video games: a self-determination theory approach. Motiv Emot 2006;30(4):344-360. [CrossRef]
  91. Winterbottom A, Bekker HL, Conner M, Mooney A. Does narrative information bias individual's decision making? A systematic review. Soc Sci Med 2008 Dec;67(12):2079-2088. [CrossRef] [Medline]
  92. Khangura S, Bennett C, Stacey D, O'Connor AM. Personal stories in publicly available patient decision aids. Patient Educ Couns 2008 Dec;73(3):456-464. [CrossRef] [Medline]
  93. Kreuter MW, Wray RJ. Tailored and targeted health communication: strategies for enhancing information relevance. Am J Health Behav 2003 Dec;27(Suppl 3):S227-S232. [Medline]
  94. Keeney RL, Raiffa H. Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge & New York: Cambridge University Press; 1993:569.
  95. Orji R, Mandryk R, Vassileva J, Gerling K. Tailoring persuasive health games to gamer type. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2013 Presented at: CHI '13; April 27-May 02, 2013; Paris, France p. 2467-2476. [CrossRef]
  96. Kaptein M. Personalized persuasion in Ambient Intelligence. J Ambient Intell Smart Environ 2012;4(3):279-280. [CrossRef]
  97. Reichlin L, Mani N, McArthur K, Harris AM, Rajan N, Dacso CC. Assessing the acceptability and usability of an interactive serious game in aiding treatment decisions for patients with localized prostate cancer. J Med Internet Res 2011 Jan 12;13(1):e4 [FREE Full text] [CrossRef] [Medline]
  98. Lerner JS, Li Y, Valdesolo P, Kassam KS. Emotion and decision making. Annu Rev Psychol 2015 Jan 03;66:799-823. [CrossRef] [Medline]
  99. Syrowatka A, Krömker D, Meguerditchian AN, Tamblyn R. Features of computer-based decision aids: systematic review, thematic synthesis, and meta-analyses. J Med Internet Res 2016 Jan 26;18(1):e20 [FREE Full text] [CrossRef] [Medline]
  100. Medical Device Innovation Consortium. 2015. Medical Device Innovation Consortium (MDIC) Patient centered benefit-risk project report: A framework for incorporating information on patient preferences regarding benefit and risk into regulatory assessments of new medical technology   URL: http://mdic.org/wp-content/uploads/2015/05/MDIC_PCBR_Framework_Web1.pdf [accessed 2019-02-19] [WebCite Cache]
  101. Zikmund-Fisher B, Ubel P, Smith D. Communicating side effect risks in a tamoxifen prophylaxis decision aid: the debiasing influence of pictographs. Patient Educ Couns 2008;73(2):209-214. [CrossRef]
  102. Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008 Dec;73(3):448-455. [CrossRef] [Medline]
  103. Brown SM, Culver JO, Osann KE, MacDonald DJ, Sand S, Thornton AA, et al. Health literacy, numeracy, and interpretation of graphical breast cancer risk estimates. Patient Educ Couns 2011 Apr;83(1):92-98 [FREE Full text] [CrossRef] [Medline]
  104. Weiss BD, Coyne C. Communicating with patients who cannot read. N Engl J Med 1997 Jul 24;337(4):272-274. [CrossRef] [Medline]
  105. Bunge M, Mühlhauser I, Steckelberg A. What constitutes evidence-based patient information? Overview of discussed criteria. Patient Educ Couns 2010 Mar;78(3):316-328. [CrossRef] [Medline]
  106. Feldman-Stewart D, Brundage MD. Challenges for designing and implementing decision aids. Patient Educ Couns 2004 Sep;54(3):265-273. [CrossRef] [Medline]
  107. Ancker J, Chan C, Kukafka R. Interactive graphics to demonstrate health risks: formative development and qualitative evaluation. J Health Commun 2009;14(5):461-475. [CrossRef]


RCT: randomized controlled trial
SGH: serious games for health


Edited by G Eysenbach; submitted 11.07.18; peer-reviewed by T English, K Bul; comments to author 16.08.18; revised version received 03.01.19; accepted 25.01.19; published 02.05.19

Copyright

©Sarah Verschueren, Connor Buffel, Geert Vander Stichele. Originally published in JMIR Serious Games (http://games.jmir.org), 02.05.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.