Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 24.07.20 in Vol 8, No 3 (2020): Jul-Sep

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/19037, first published Apr 03, 2020.

This paper is in the following e-collection/theme issue:

    Viewpoint

    Quality Criteria for Serious Games: Serious Part, Game Part, and Balance

    1Multimedia Communications Lab, Technical University of Darmstadt, Darmstadt, Germany

    2Institute of Sport Science, Technical University of Darmstadt, Darmstadt, Germany

    3Research Group Didactics of Mathematics, Technical University of Darmstadt, Darmstadt, Germany

    Corresponding Author:

    Polona Caserman, MSc

    Multimedia Communications Lab

    Technical University of Darmstadt

    Rundeturmstraße 10

    Darmstadt, 64289

    Germany

    Phone: 49 6151 16 20391

    Email: polona.caserman@kom.tu-darmstadt.de


    ABSTRACT

    Serious games are digital games that have an additional goal beyond entertainment. Recently, many studies have explored different quality criteria for serious games, including effectiveness and attractiveness. Unfortunately, the double mission of serious games, that is, simultaneous achievement of intended effects (serious part) and entertainment (game part), is not adequately considered in these studies. This paper aims to identify essential quality criteria for serious games. The fundamental goal of our research is to identify important factors of serious games and to adapt the existing principles and requirements from game-related literature to effective and attractive serious games. In addition to the review of the relevant literature, we also include workshop results. Furthermore, we analyzed and summarized 22 state-of-the-art serious games for education and health. The selected best-practice serious games either prove their effectiveness through scientific studies or by winning game awards. For the analysis of these games, we refer to “DIN SPEC 91380 Serious Games Metadata Format.” A summarized text states quality criteria for both the serious and the game part, and especially the balance between them. We provide guidelines for high-quality serious games drawn from literature analysis and in close cooperation with domain experts.

    JMIR Serious Games 2020;8(3):e19037

    doi:10.2196/19037

    KEYWORDS



    Introduction

    Serious games are digital games that not only entertain but also intend to achieve at least one additional goal, a so-called characterizing goal [1]. Serious games aim to accomplish this characterizing goal without compromising the experience of playing a game (player experience). Examples include educational games, such as Meister Cody – Talasia (Meister Cody GmbH) [2], and games for health, such as ExerCube (Sphery AG) [3] and Pokémon GO (Nintendo Co) [4]. They should, in general, motivate the player to achieve the characterizing goal through appropriate methods, an engaging player experience, and the use of appropriate interaction technology.

    Serious games are not limited to training (exercising) or learning. They can also be used for other purposes and can be applied in almost every area. For example, America’s Army (US Army) [5] is used as a recruiting tool; Re-Mission (HopeLab) [6,7] aims to change attitude, knowledge, and behavior; Trash Monsters (Bunny and Gnome) [8] improves knowledge about waste separation; and Orwell: Keeping an Eye on You (Osmotic Studios) [9] raises awareness about government surveillance. In some cases, games that have not been intentionally developed for serious purposes can also achieve additional effects. For example, the Civilization (Take-Two Interactive Software) [10], Age of Empires (Microsoft Corp) [11], and Assassin’s Creed (Ubisoft) [12] series are primarily developed for entertainment purposes; however, when playing these games, players also learn about ancient organizations or the history of civilizations. Nevertheless, the question remains: what are important constituents of serious games and, in particular, what are important aspects of high-quality serious games? Although many useful heuristics are presented in the game-related literature [13,14], no model yet exists that focuses equally on the serious and game aspects, as well as on the balance between them.

    Existing quality criteria for video games often focus on appropriate game design, (eg, they consider only the player enjoyment [13] and are identified based on game reviews and rating systems [14,15]). Rating systems cover the different kinds of expertise of involved game reviewers and the complexity of testing processes [16]. For example, IGN (IGN Entertainment), Gamespot (CBS Interactive), and PC Gamer (Future US Inc) primarily rely on the expertise and opinions of their in-house editors. The metareview system Metacritic (CBS Interactive) aggregates these expert scores into a single metascore, in addition to letting users vote on a secondary user score. Conversely, studies that evaluate quality criteria for serious games are often specific to an application field and focus on didactic aspects (ie, they propose specific guidelines for educational [17] or motion-based serious games [18]). In particular, existing quality criteria for serious games often lack in the combining of both aspects (ie, serious as well as game aspects).

    In this paper, we aimed to gather criteria for high-quality serious games, considering both the serious and game aspects and the balance between them. First, we described some of the successful serious games according to the proven “DIN SPEC 91380 Serious Games Metadata Format” (SG-MDF) [16,19]. Existing serious games taxonomies include specific classification systems for rehabilitation games [20], educational games [21,22], or serious games in general [23-25]; however, they usually select classification criteria arbitrarily and are not generally accepted. In particular, they do not take the aspects of the characterizing goal or the quality of serious games into consideration. SG-MDF overcomes these limitations and covers all crucial aspects of serious games (ie, the characterizing goal as well as quality criteria). Finally, based on the description of serious games, relevant game-related work, and close cooperation with domain experts, we refined and extended these quality criteria to define high-quality serious games.

    To summarize, our primary goal is to identify quality criteria for the serious and game part, as well as the balance between them. We have provided guidelines for high-quality serious games drawn from literature analysis and workshops with domain experts.


    State of the Art

    Game Classification and Selection

    We studied serious games to classify them according to SG-MDF. SG-MDF is also used in a metadata-based information system [26], which allows providers of serious games (eg, developers and publishers) to systematically describe the games so that users (eg, teachers, trainers, coaches, doctors, and therapists) can find suitable games accordingly. Using this format, we provide a summary of games for health [3,4,6,7,27-52] and educational games [2,8,9,53-64] (see Multimedia Appendices 1 and 2). Note that the list is not complete and should only serve as a foundation for developing and explaining the quality criteria. We selected serious games that prove their effectiveness either through scientific studies or by winning game awards. Furthermore, the selected serious games have a certain level of familiarity in the community.

    We use SG-MDF because it overcomes the limitations of the existing taxonomies [23-25] and covers crucial aspects of serious games, such as the characterizing goal and the quality (based on scientific studies, game awards, professional ratings, recommendation by experts, and number of players/downloads). However, we did not include all categorizations as proposed by SG-MDF and included only measures that are important to present the quality criteria in this paper. For example, game modes are important for the “support social interactions” criterion, and the target group is essential for the “appropriate interaction technology” and “media presentation” criteria. In general, all serious games should use appropriate interaction technology for the target group, game purpose, and application area. Furthermore, the progress indicator is essential for the “appropriate feedback and reward” criterion.

    Games for Health

    Serious games are not only fun to play but are also beneficial for health. For example, they can motivate players to increase physical exercise. Due to insufficient physical activity, the risk of diseases such as obesity, diabetes, cancer, and cardiovascular diseases are increasing. The World Health Organization reports that physical activity has decreased over time in high-income countries [65]. These results show that it is crucial to motivate people to become more physically active. However, games for health do not only cover physical exercises but are also often used for prevention, rehabilitation and, in general, supporting health (ie, enforcing a behavior change towards a better, more active, and healthier lifestyle, including better nutrition).

    Popular exergames such as Pokémon GO [27] and Dance Dance Revolution (Konami) [28] aim to provide an effective and attractive workout experience for a wide variety of users. Pokémon GO, for example, has over 1 billion downloads on Google Play Store (Google Corp) [44], making over $800 million US dollars worldwide in 2019 [45]. The study by Althoff et al [4] shows that it indeed increases players’ activity level compared with their prior activity level; however, the researchers could only confirm short-term effects. Additional studies show that Dance Dance Revolution significantly increases energy expenditure [29] and improves aerobic fitness in overweight children [30]. Some schools have even included the game in their physical education courses to motivate children to exercise [66]. However, studies also report that exergames are often only capable of providing light to moderate exercise and thus, often fail to significantly increase physical activity or exercise attendance [67].

    Furthermore, Wii Sports games (Nintendo Co) are best-selling video games [68] that contribute to weight loss [31] or to increased muscle strength [32]. However, they lack proper training concepts and disregard performance aspects that are essential for a successful workout (eg, accuracy, precision, and intensity of movement). Similarly, Beat Saber (Beat Games) [33] is one of the top virtual reality games in 2019 [46,47]; however, it focuses on game design and ignores the extensive knowledge of movement and training science in sports. On the contrary, the ExerCube [34] was developed by an interdisciplinary team of sports scientists, game designers, and researchers in the field of human-computer interaction. The results of a user study with 40 participants show that the ExerCube is on par with personal training [3].

    Other serious games intend to improve the physical status of older people. For example, BalanceFit aims to improve coordination, strength, and balance [35]. A study by Hardy et al [36] shows that an adaptive approach enables people with heterogeneous skills to play this game (eg, fit players as well as players with gait impairments or wheelchairs). Furthermore, the game ErgoActive provides adaptive cardio training on an ergometer bike to increase the physical activity of its players [35]. The results of a feasibility study with 16 participants demonstrated the effectiveness of cardio training based on personalized heart rate control [37].

    The characterizing goal of a game for health does not necessarily need to aim at a physical training effect. The serious game Re-Mission [38] intends to inform patients about cancer treatments and aims to change their attitude positively. Studies confirmed the effectiveness of the game in randomized controlled trials with cancer patients [6,7]. Other serious games, such as Escape from Diab and Nanoswarm: Invasion from Inner Space (Archimage) are persuasive and are able to change health-related behavior among children [39]. Further serious games for health are used as prevention (eg, PlayForward: Elm City Story) [40] or rehabilitation (eg, SnowWorld) [41]. Moreover, Dr Kawashima’s Brain Training (Nintendo Co) includes a set of minigames that are designed to improve cognitive functions in elderly persons. However, even though randomized controlled trials report benefits [42,43], no long-term effects and relevance for everyday functioning could be confirmed.

    Educational Games

    In addition to motivating players to become more physically active, serious games are often used to increase players’ motivation levels to learn and improve learning outcomes. Educational games can be a reliable and effective tool compared with traditional methods [69]. According to the Entertainment Software Association, 74% of parents believe that video games can be educational for their children [70]. Educational games are also the second most popular Google Play app category [71]. Serious games are effective in terms of learning and some of them are even better at teaching than traditional methods [72].

    Studies in the field of game-based learning show the benefits of educational games, including improvement of mathematical skills (eg, Meister Cody – Talasia [2]), reading performance (eg, Meister Cody – Namagi [53]), and programming skills (eg, Debugger 3.16: Hack’n’run [Spiderworks Games] [54,61]). Serious games can also be used to assess knowledge (eg, Semideus [Flow Factory] [55,56]). VocabiCar (Westermann Digital GmbH) [57] is another educational game for children and intends to expand the English vocabulary of pupils.

    In addition to improving players’ skills, educational games can also raise awareness. The game Trash Monsters [8] raises awareness of waste separation and teaches children how to recycle correctly. Educational games do not necessarily need to be intended for children but can be dedicated to students or adults in general. Orwell: Keeping an Eye on You serves to raise awareness of state surveillance [9,63], and Orwell: Ignorance is Strength (Osmotic Studios) serves to raise awareness of fake news [58,64]. By representing different moral values, the game strengthens or enforces players’ opinions. However, both serious games are reading intensive and therefore (similar to all serious games) only suitable for specific player types.

    Another application area for serious games are simulations, in particular, corporate games for training purposes. ViPOL (TriCAT) enables police forces to train in virtual reality for scenarios that are too expensive, complex, or dangerous to be trained for in the real world [59]. The simulation was developed in close cooperation with police officers. In a study by Bertram et al [59], the results show that virtual training can be as efficient as regular training for complex collaborative tasks.


    Identifying Quality Criteria for Serious Games

    Development of the Criteria

    A review of the state of the art was conducted to determine quality criteria for attractive and effective serious games. We propose essential aspects of high-quality serious games, including characteristics for the serious and game part, as well as for the balance between them. Although we focus on educational games and games for health, the quality criteria are transferable to all kinds of application areas. For example, the criteria can be used not only for games that improve players’ skills/performance but also for games that raise the players’ awareness of a certain topic or that positively change their attitudes.

    We furthermore discussed the derived quality criteria in workshops with experts from the respective areas. The aim of the workshops was to identify the requirements and needs for high-quality serious games. Therefore, domain experts, such as game developers and companies that deploy serious games, as well as scientists from different areas (eg, sports education and computer science), critically discussed the quality criteria of serious games. The quality criteria are shown in Figure 1 and further detailed in the following section. High-quality serious games must achieve both the serious and the game aspects; they must systematically support players to reach the characterizing goal (serious part) and they must elicit and maintain player experience (game part). Furthermore, both parts should be perfectly matched and integrated rather than addressed in isolation.

    Figure 1. Quality criteria for the serious part and game part, as well as the balance between them.
    View this figure

    In the following section, we summarize key findings, including the strength of evidence for each criterion. These results should help other researchers and game developers gain a deeper understanding of high-quality serious games.

    Serious Part

    This section describes the core elements for the serious part of the game: existence of a characterizing goal, development of appropriate methods for achieving this characterizing goal, and evaluation of the quality (see also Table 1).

    Table 1. Summary of quality criteria for the serious part.
    View this table
    Characterizing Goal

    Serious games must ensure that players achieve the characterizing goal.

    Focus on the Characterizing Goal

    The characterizing goal of a serious game is closely linked to the application area. In educational games, the characterizing goals include learning or training effects. In games for health, the characterizing goals include changes in vital status or general changes in attitude and behavior (eg, nutrition or mobility behavior). It may also be beneficial to combine physical and cognitive training. For example, in the game Lü (Lü Interactive Playground) [73], the players not only improve mathematical skills but also stay physically active. Similarly, ExerCube [34] provides training for body and mind. Furthermore, recent evidence on exergame-based therapy for Parkinson disease shows that exergames can enhance cognitive skills and are at least as effective as traditional therapies [74,75].

    Thus, serious games should always focus on achieving the characterizing goal and should support the player in achieving this goal. Learning or training content must remain in focus during gameplay and game elements should not interfere with the learning or training process.

    Clear Goals

    Similar to entertainment games, in which the game goals should be clear [13,14], the characterizing goal of a serious game should also be transparent so that players can work towards achieving this goal. In particular, a serious game should ensure that players always know what to do to complete the tasks or exercises; otherwise, a tutorial is required (see also the “Intuitive Game Mechanics and Natural Mapping” section). For example, in the ExerCube, a virtual avatar demonstrates required movements in a short tutorial to make sure that players know how to execute them [3]. The educational games Meister Cody – Talasia and Meister Cody – Namagi [60] show how each exercise should be solved before players start to solve the tasks.

    Indispensability of the Characterizing Goal

    Engaging in the serious part of the game should be mandatory for playing the game. Otherwise, players may avoid the serious part to get to the fun part more quickly [76], preventing the characterizing goal from being reached. In particular, the training and learning tasks should not be a hurdle while playing a serious game. In other words, the characterizing goal should be embedded in the gameplay (see also the “Integrated Serious Part With Gameplay” section).

    Methods

    The methods for serious games need to be appropriate for the specific application area and target group.

    Correctness of the Domain Expert Content

    The most evident requirement for serious games is that they must not contain any errors with respect to their subject matter, such as erroneous mathematical equations, incorrect information on historical events, or inadequate information on physical exercise. If typing or presenting errors occur, these errors must not mislead players. Furthermore, conveyed information must not only be factually correct but also be imparted using appropriate technical language.

    Moreover, even though there is usually no connection between neutrality and the correctness of a given opinion or issue, serious games should remain neutral, especially on political and social issues. For example, the games Orwell: Keeping an Eye on You [9] and Orwell: Ignorance is Strength [58] do not convey a specific political opinion to the player. Instead, they only show the effects of the player’s actions without judging them. The player then has the opportunity to assess and question their own decisions. In particular, serious games should be appropriate for the target group, depending on religion, culture, and traditions.

    Appropriate Feedback on Progress

    A serious game should provide appropriate feedback to players so that they can assess their progress. Thus, to enhance player performance, effects should be visible and recognizable (eg, through a progress bar). Continuous feedback on progress is essential in all serious games as the players work towards achieving the characterizing goal [77]. Moreover, multimodal feedback (eg, visual, audio, or haptic feedback) can be beneficial [14]. For example, in the ExerCube, players receive immediate visual and audio feedback to enhance their movements [3].

    Apart from in-game feedback, Ravyse et al [78] furthermore show that postgame feedback also improves learning. Similarly, the level number can indicate the player’s progress (ie, the higher the level, the more skills the player has developed). A higher in-game level seems to have a more significant effect on motivation (desire to practice) than individual rewards (eg, achievements) [79]. Game statistics additionally show the player’s progress after ending the game or a level and are not only advantageous for players themselves, but also useful for their therapists (especially in games for health [20]) or their parents and educators (in educational games [60]).

    Appropriate Reward

    Games should provide positive reinforcement and in-game awards [1] to immerse players more deeply in the game [14]. For example, in the educational game VocabiCar, players who accomplish a learning task or complete a challenge gain points and can access their progress in a high-score table [57]. High-score tables allow players to compare their performance against other players. In addition to points, in-game awards consist of virtual badges, achievements, power-ups, and desirable objects. Fancy animations or a possibility to change their avatar (eg, new clothes, hair color, or equipment) as a reward for accomplishing a task can further motivate players. However, there is ample evidence that “rewards or feedback delivered in a controlling manner undermine intrinsic motivation and deeper forms of learning” [80]. Therefore, rewards should be deliberately deployed in serious games.

    Quality

    High-quality serious games should measure the effects and benefits in a scientific study. Furthermore, awards and user or domain expert ratings can confirm the quality of a serious game.

    Proof of Effectiveness and Sustainable Effects

    A serious game is effective when players achieve the characterizing goal and the learning or training effects are sustainable. Researchers often validate effectiveness with a study, such as a scientific, clinical, or empirical evaluation, by monitoring heart rate (eg, ExerCube [3] and ErgoActive [37]), number of steps (eg, Pokémon GO [4]), or aerobic fitness (eg, Dance Dance Revolution [30]). However, potential aversions to video games, certain game genres, or specific interaction technologies among players have to be considered when designing and evaluating a study. In educational games, the results of a group exposed to a serious game and a group exposed to traditional methods can be compared (eg, Meister Cody – Talasia [2] and ViPOL [59]). However, empirical studies often suffer from numerous sources of error (eg, bias) [81]. In this regard, randomized controlled trials with a sufficient number of participants are the gold standard for empirical proof of effects.

    Awards and Ratings

    In addition to scientific studies that evaluate usability and player experience, game awards (eg, German Computer Games Award, European Innovative Game Award, and International Educational Games Competition) are also an important aspect of identifying high-quality serious games. Further quality criteria include professional or user ratings, the number of players or downloads (eg, Google Play, App Store [Apple Inc], and Steam [Valve Corp]), and recommendations by domain experts and game reviews (eg, IGN, GameSpot, and PC Gamer).

    Game Part

    This section describes core elements for appropriate game design and suitable interaction technology, as seen in Table 2. Note that there is considerable overlap in the various concepts within the game part.

    Table 2. Summary of quality criteria for the game part.
    View this table
    Enjoyment

    Serious games should not only ensure positive player experience, flow, and sense of control but should also support social interaction.

    Ensure Player Engagement and Experience

    Player engagement is tightly associated with enjoyment. Koster, the author of A Theory of Fun for Game Design, addresses the importance of a game being engaging, enjoyable, and fun [82]. The GameFlow approach proposed by Sweetser and Wyeth [13] includes 8 dimensions with numerous criteria and recommendations to ensure player enjoyment in games. Calvillo-Gámez et al [87] furthermore present the core elements of the gaming experience to provide a positive experience while playing video games. Moreover, Desurvire and Wiberg [14] propose heuristics of playability for game developers to develop better games.

    However, due to different kinds of players, not every player will find all components equally important. Therefore, the game should provide different fun components to provide an engaging experience for different player types (eg, Bartle’s player types [88]).

    Ensure Flow

    For the optimal player experience, the game has to establish a satisfying balance between challenges and skills. Csikszentmihalyi’s well-known flow theory describes the feeling of enjoyment when the task difficulty and skill levels are in balance [83]. This theory is complemented by Bushnell’s theorem of “easy to learn, difficult to master” [84]. Games that are easy to learn enable flow because they are not overwhelming, whereas games that are hard to master keep players from dropping out because of boredom. Thus, as the player’s performance improves, the complexity or difficulty of the game should also increase. In other words, serious games should adapt to the current performance level of the player (ability vs skills).

    To avoid boredom, game developers should also ensure that the gameplay varies. As proposed by Desurvire and Wiberg [14], any fatigue or boredom should be minimized by varying activities and pacing during the gameplay. Furthermore, research by Scoresby and Shelton [89] identified that content, emotion, motivation, and engagement associated with the game are necessary criteria for flow.

    Moreover, serious games should automatically adapt to the players to motivate them to keep learning/training and to increase the effectiveness. One of the primary advantages of educational games is their ability to engage the learner so that they voluntarily complete sufficient repetitions of activities, ensuring that learning takes place [90]. For example, both of the Meister Cody educational games adapt their difficulty depending on the player’s skills [60]. The results show that, due to adaptivity, game-based learning is particularly promising for children who want to learn in a home environment or do not have access to individual reading support [53]. Similarly, exergames should ensure that the intensity matches the player’s fitness level in order to motivate players to repeat the exercises continuously and regularly [91]. For example, ExerCube [3] or ErgoActive [37] identify the individual’s optimal strain to adapt the game difficulty and complexity gradually based on the player’s heart rate.

    Establish an Emotional Connection

    Additionally, players should get emotionally involved in a serious game. Dillon [85] has drawn attention to the fact that emotions and instincts increase players’ engagement to continue playing the game. The game designer and developer proposed the 6-11 framework, which contains 6 basic emotions and 11 instincts. For example, various serious games use an instinct to survive and thus to fight (eg, Re-Mission [38]) or to collect something (eg, VocabiCar [57] and Debugger 3.16: Hack’n’run [54]). Furthermore, LeBlanc’s theory of 8 kinds of fun describes the desirable emotional responses evoked in players when they interact with the game [86].

    Sense of Control

    Players should feel in control over their actions in the game world [14]. In particular, players should have control and influence on the game world. For example, in the serious game PlayForward: Elm City Story, players can see how different actions lead to different outcomes [48]. Similarly, Escape from Diab allows players to influence the storyline and the characters [51,52]. As a result, the ability to influence the game world and in particular the story progress can motivate players to keep playing the game. Furthermore, serious games should support an optimal relationship between the player’s actions and the game’s reactions. For example, increased pedaling frequency in ErgoActive will always cause the character to rise [35].

    Support Social Interactions

    Bond and Beale [15] identified that good games offer some form of social interaction (see also the self-determination theory [92]). Social interactions in games are important for players who perform better depending on the game mode (eg, playing with friends or against them). For example, Pokémon GO [27] lets friends feed the player’s creatures, and friends and family playing Dr Kawashima’s Brain Training [49] can compete against each other. Vorderer et al [93] suggest that competitive elements are important for enjoyment. On the contrary, the work of Staiano et al [31] reveals that playing a cooperative version of the Nintendo Wii Active game is more effective than playing a competitive version. Thus, different game modes motivate players more or less. Especially if the players perform better in groups, collaborative and competitive multiplayer settings can contribute to motivating players [3]. For specific players, playing in a group (ie, multiplayer games) is more motivating than playing alone (ie, single-player games) [86]. Depending on the game purpose and target group, the game developers and designers should try to include different game modes so that a serious game is enjoyable for a broad player base.

    Ensure Immersive Experience

    Immersion in virtual environments can be increased by stimulating different human senses, especially by including appropriate audiovisual elements in the game. The game should use visceral, audio, and visual content to immerse players more deeply in the game [14]. Slater and Wilbur [94] describe immersion as the extent to which the computer system can deliver an illusion of a virtual environment to players. Thus, an immersive virtual environment should accommodate a wide range of appropriate synchronized sensory modalities. Recent studies already provide evidence that multimodal sensory stimulation improves the sense of presence (ie, the sense of “being there”) and immersion [95]. For example, SnowWorld [50] successfully distracts players during rehabilitation (wound treatment) by immersing them in a virtual world. For a fully immersive virtual reality experience, serious games should include visual (eg, current-generation head-mounted displays), audio (eg, noise-canceling headphones), and haptic (eg, data gloves with force feedback or vibrations) feedback, as well as sense of smell (eg, smell dispenser) [96].

    Media Presentation

    One of the most apparent factors for immersive serious games is that they should have visually appealing graphics and appropriate sound effects. In particular, audiovisual elements in the game seize the attention of players [78].

    Attractive Graphics

    The included graphics should look attractive and engaging, as well as appropriate for the game purpose, application area, and the target group. For example, a game should have different designs for children, adults, and people with disabilities. The game designer should ensure clear interfaces without unnecessary information. Ravyse et al [78] furthermore suggest creating games that are high in realism; however, it should not be overloaded with unnecessary objects so that players do not get distracted from a specific task. For effective training, particularly for firefighters [97] or police training [59,98], the simulations should provide realistic virtual environments. However, in contrast to high-end graphics, reduced graphics can also be appropriate for some game types. For example, Minecraft (Mojang Studios) [99] is one of the most successful video games and it contains a world of blocks.

    Appropriate Sound

    Serious games should not only be visually appealing but should also include appropriate background music and sound effects. Previous studies have shown that audio influences the sense of presence, particularly in immersive virtual reality applications [100,101]. Martin-Niedecken et al [3] have expressed a similar view. The researchers show that music increases the motivation and immersion of the test subjects while playing the ExerCube. However, due to the players’ varying music preferences, the choice of music could also be an important factor in motivation [102]. For example, the game Beat Saber allows players to create levels with custom songs [33].

    Balance Between Serious and Game Part

    The serious part and the game part of the game should be integrated and strongly connected, as seen in Table 3.

    Table 3. Summary of quality criteria for balance between the serious and game part.
    View this table

    Integrated Serious Part With Gameplay

    Serious games should embed the characterizing goal into the gameplay and the characterizing goal should not be avoidable.

    Embedding Characterizing Goal Into the Gameplay

    The gameplay experience includes, among other factors, an imaginative immersion (ie, immersion in the game world and the story) [95]. Thus, to motivate players, learning/training tasks need to be embedded into the immersive gameplay, such as in a story or narrative. Learning and training tasks should be directly related to the game and should be connected with the game elements and environment. For example, the educational game Addy (Coktel Vision) does not integrate learning into the game mechanics but uses a game only as a reward for learning. In this case, the serious part is not integrated into the game.

    High-quality serious games should always integrate the serious part with the gameplay. For example, in adventure-based games, such as Meister Cody – Namagi and Meister Cody – Talasia, the tasks are embedded in a narrative and the story only proceeds when the player solves a problem. Similarly, Semideus [62] integrates the player’s knowledge of rational numbers seamlessly into the gameplay. This close connection between the learning/training tasks and the storytelling makes the game even more motivating for children. An engaging story can motivate players; however, games without any story can still be very successful (eg, ExerCube [34]). Therefore, a story does not necessarily have to be profound or fascinating, but depending on the type of game, it can still be motivating.

    Scientific Foundation

    To develop an effective and attractive serious game, members of an interdisciplinary team of game designers, programmers, artists, and domain experts have to work together throughout the entire development process. As proposed by Martin-Niedecken et al [3], the target group should also be involved in the design process from the beginning (ie, participatory game design). This interdisciplinary team has to establish a balance between the disciplinary standards and requirements and the interdisciplinary integration under the twofold mission of serious games. Thus, for a high-quality serious game, one needs scientific foundations on both sides [1]. However, due to the limited development budget for serious games, there are often no resources for hiring professional game designers or artists. Thus, the challenge of an optimal balance between the serious and the game parts remains, making it difficult for the teams to develop a high-quality serious game that is entertaining and also fulfills its characterizing goal. Nevertheless, teams should continue to ensure an appropriate balance.

    Interaction Technology

    Depending on the serious game and the target group, adequate interaction technology must be deliberately chosen based on the specific needs of users and the game’s purpose.

    Appropriate Interaction Technology

    The interaction technology must be suitable for the physical and mental ability of the target group. For example, the Kinect sensor (Microsoft Corp) and the Leap Motion Controller (Ultraleap) are particularly suitable for rehabilitation games, such as games for players with Parkinson disease or players who have had strokes, since these players often cannot hold or wear additional sensors [74]. Thus, depending on the target group, serious games must be presented by appropriate technology, including the appropriate visual display, speakers, and haptic interfaces, as well as suitable game-specific controllers (eg, gamepads or joysticks) [103]. The usage of an innovative technology should be appropriate for the game purpose as well; for instance, a head-mounted display can be annoying during sports, especially if the player is excessively sweating. However, commercial virtual reality games such as Beat Saber [33] have already proven that immersive environments are engaging and motivating despite sweating.

    Intuitive Game Mechanics and Natural Mapping

    It should be easy for players to understand how to play. Otherwise, the game should provide a tutorial. A tutorial should introduce not only game mechanics (how to do something) but also the gameplay (what to do). For inexperienced players, the controls should be basic enough to learn quickly, whereas experienced players can use advanced options. Desurvire and Wiberg [104] proposed principles for game designers to create better tutorials. In general, the game should be developed in such a way that players do not need to read a tutorial in order to play [14]. The work of Andersen et al [105] shows that tutorials are only appropriate for complex games and that simpler games should allow the players to discover the game mechanics themselves. Regardless of which interaction technology is used, the game controls should be intuitive, such as using the WASD keys to move and the space bar to jump (well-known game controls).

    In addition to intuitive game mechanics, a game should also ensure natural mapping between technology and gameplay; in other words, the game should naturally map its controls to changes in the virtual environment [14]. For example, games for health should sense the movements of players to trigger the corresponding game responses [103]. Beat Saber [33] fulfills this requirement and allows players to intuitively move in the real world by simply walking around and using arm movements to slash the blocks.

    No Simplifying of the Learning/Training Process Due to Technical Features

    The interaction technology must support players in achieving the characterizing goal and must not impair or even disturb the learning/training processes. In exergames, players should not have advantages due to poor interaction technology (eg, cheating in Nintendo Wii sports games). The system must ensure that players are moving their bodies as required by the game [25]. Without accurate tracking, Wii players can sit on a couch and successfully play the game with a controller without performing the desired physical exercise. In this case, the system can be tricked and players only pretend to carry out a movement correctly. A study by Marks et al [106] shows that games played with Wii controllers require less physical activity than games played with the Kinect. To address this, exergames should track and detect motions accurately and in real time to ensure that players correctly execute all exercises. If the movement of a specific body part is required, this body part should be explicitly tracked to avoid cheating. For example, the ExerCube uses virtual reality trackers (HTC Vive; HTC Corp) to track upper body movements accurately [3].

    Avoid Adverse Effects

    Serious games should ensure that no accidents or injuries can happen, as older people can easily stumble and fall. Especially in exergames, the movements have to be tracked accurately so that players perform the exercises correctly and do not make mistakes that cause injuries. If required, the technology should be adapted and personalized to the specific target groups. For instance, the game BalanceFit uses a stability frame for secure balance training so that older people with heterogeneous skills are able to play the game [35]. Furthermore, physical or mental overload should be prevented by adequate monitoring of the player’s psychophysical state or through regular breaks suggested by the game.

    Independent of the application area, all games should avoid technical issues and should be easy to maintain [15]. Moreover, in immersive virtual reality, it is important to maintain a high frame rate, low latencies, and fast synchronization to avoid cybersickness. Cybersickness is caused by perceived motion or sensory mismatch, usually reported in a virtual roller coaster or car simulations. The latency between a user’s movement and visual feedback in virtual reality has a significant impact on user experience and performance [107].


    Conclusion

    In this paper, we proposed criteria for high-quality serious games. We examined various serious games and existing heuristics from game-related literature to specify quality criteria for effective and attractive serious games. The suggested quality criteria were furthermore discussed in workshops with domain experts. We introduced quality criteria for the serious and game aspects, as well as the balance between them. First, high-quality serious games should keep the characterizing goal in focus and should use appropriate methods for the specific application area and target group. Serious games should provide suitable feedback so that players can assess their progress and work towards achieving the characterizing goal. The effectiveness of serious games should be proven in scientific studies or by winning game awards. Second, high-quality serious games should be fun and enjoyable. They must ensure player engagement and should keep the players in flow (ability vs skills). Finally, the double mission of serious games, that is, the balance between the serious and the game part, must be ensured. Therefore, high-quality serious games should embed the characterizing goal into the gameplay so that engaging in the serious part is mandatory for playing the game. Furthermore, the interaction technology should be suitable for the target group and game purpose.

    In future work, we want to evaluate the proposed quality criteria, weight them, and determine a score to specify high-quality serious games, thereby proposing a quality mark. We hope that the proposed criteria will encourage game designers, developers, and researchers in the future to develop more high-quality serious games.

    Acknowledgments

    This work has been funded by the European Union (IWB-EFRE-Program Hessen) and the Hessian Ministry of Economics, Energy, Transport and Housing (Digitales Hessen) as part of the Wissens- und Technologietransfer Serious Games project.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Classification of serious games for health.

    PDF File (Adobe PDF File), 83 KB

    Multimedia Appendix 2

    Classification of educational games.

    PDF File (Adobe PDF File), 41 KB

    References

    1. Dörner R, Göbel S, Effelsberg W, Wiemeyer J. Serious Games: Foundations, Concepts and Practice. Cham, Switzerland: Springer International Publishing; 2016.
    2. Kuhn J, Holling H. Number sense or working memory? The effect of two computer-based trainings on mathematical skills in elementary school. Adv Cogn Psychol 2014;10(2):59-67 [FREE Full text] [CrossRef] [Medline]
    3. Martin-Niedecken A, Rogers K, Vidal L, Mekler E, Segura E. ExerCube vs personal trainer: valuating a holistic, immersive, and adaptive fitness game setup. 2019 Presented at: Proceedings of the CHI Conference on Human Factors in Computing Systems; May 4-9, 2019; Glasgow, United Kingdom p. 1-15. [CrossRef]
    4. Althoff T, White RW, Horvitz E. Influence of Pokémon Go on Physical Activity: Study and Implications. J Med Internet Res 2016 Dec 06;18(12):e315 [FREE Full text] [CrossRef] [Medline]
    5. Zyda M. From visual simulation to virtual reality to games. Computer 2005 Sep;38(9):25-32. [CrossRef]
    6. Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 2008 Aug;122(2):e305-e317 [FREE Full text] [CrossRef] [Medline]
    7. Beale IL, Kato PM, Marin-Bowling VM, Guthrie N, Cole SW. Improvement in cancer-related knowledge following use of a psychoeducational video game for adolescents and young adults with cancer. J Adolesc Health 2007 Sep;41(3):263-270. [CrossRef] [Medline]
    8. Trash Monster. Bunny & Gnome. 2016.   URL: https://muell-ag.de/
    9. Orwell: Keeping an Eye on You. Osmotic Studios. 2016.   URL: https://www.osmoticstudios.com/orwell-keeping-an-eye-on-you/
    10. Civilization. Take-Two Interactive Software, Inc. 2016.   URL: https://civilization.com/
    11. Age of Empires. Microsoft. 1997.   URL: https://www.ageofempires.com
    12. Assassin's Creed. Ubisoft Entertainment. 2018.   URL: https://assassinscreed.ubisoft.com [accessed 2020-05-01]
    13. Sweetser P, Wyeth P. GameFlow: a model for evaluating player enjoyment in games. Comput Entertain 2005 Jul 01;3(3):3. [CrossRef]
    14. Desurvire H, Wiberg C. Game usability heuristics (PLAY) for evaluating and designing better games: The next iteration. In: Lecture Notes in Computer Science. Berlin, Germany: Springer Berlin Heidelberg; 2009 Presented at: International Conference on Online Communities and Social Computing; July 19-24, 2009; San Diego, CA p. 557-566. [CrossRef]
    15. Bond M, Beale R. What makes a good game? Using reviews to inform design. 2009 Presented at: People and Computers XXIII Celebrating People and Technology; Sep 1-5, 2009; Churchill College, Cambridge, UK p. 418-422. [CrossRef]
    16. Göbel S, Gutjahr M, Steinmetz R. What makes a good serious game-conceptual approach towards a metadata format for the descriptionevaluation of serious games. 2011 Presented at: 5th European Conference on Games Based Learning; Oct 20-21, 2011; Athens, Greece p. 202-210.
    17. Annetta LA. The “I's” Have It: A Framework for Serious Educational Game Design. Review of General Psychology 2010;14(2):105-112. [CrossRef]
    18. Mueller F, Isbister K. Movement-based game guidelines. 2014 Presented at: SIGCHI Conference on Human Factors in Computing Systems; Apr 28-May 1, 2014; Toronto, Ontario, Canada p. 2191-2200. [CrossRef]
    19. Göbel S, Caserman P, Hansen J, et al. DIN SPEC 91380. Serious Games Metadata Format 2018. [CrossRef]
    20. Rego P, Moreira PM, Reis LP. Serious games for rehabilitation: A survey and a classification towards a taxonomy. 2010 Presented at: 5th Iberian Conference on Information Systems and Technologies; June 16-19, 2010; Santiago de Compostela, Spain p. 1-6.
    21. Krathwohl D, Anderson L. A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. London, England: Longman; 2009.
    22. Carvalho MB, Bellotti F, Berta R, De Gloria A, Sedano CI, Hauge JB, et al. An activity theory-based model for serious games analysis and conceptual design. Computers & Education 2015 Sep;87:166-181. [CrossRef]
    23. Djaouti D, Alvarez J, Jessel J. Classifying serious games: The G/P/S model. In: Handbook of Research on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches. Hershey, PA: IGI Global; 2011:118-136.
    24. Sawyer B, Smith P. Serious games taxonomy. 2008 Presented at: Serious Games Summit at the Game Developers Conference; Feb 18-22, 2008; San Francisco, CA. [CrossRef]
    25. Laamarti F, Eid M, El Saddik A. An Overview of Serious Games. International Journal of Computer Games Technology 2014;2014:1-15. [CrossRef]
    26. Serious Games Information Center. 2018.   URL: https://seriousgames-portal.org [accessed 2020-05-01]
    27. Pokémon GO. Niantic, Inc. 2016.   URL: https://pokemongolive.com/ [accessed 2020-05-01]
    28. Dance Dance Revolution. Konami. 2010.   URL: https://www.konami.com/games/asia/en/products/ddr_a/ [accessed 2020-05-01]
    29. Lanningham-Foster L, Jensen TB, Foster RC, Redmond AB, Walker BA, Heinz D, et al. Energy expenditure of sedentary screen time compared with active screen time for children. Pediatrics 2006 Dec;118(6):e1831-e1835. [CrossRef] [Medline]
    30. Murphy EC, Carson L, Neal W, Baylis C, Donley D, Yeater R. Effects of an exercise intervention using Dance Dance Revolution on endothelial function and other risk factors in overweight children. Int J Pediatr Obes 2009;4(4):205-214. [CrossRef] [Medline]
    31. Staiano AE, Abraham AA, Calvert SL. Adolescent exergame play for weight loss and psychosocial improvement: a controlled physical activity intervention. Obesity (Silver Spring) 2013 Mar;21(3):598-601 [FREE Full text] [CrossRef] [Medline]
    32. Sohnsmeyer J, Gilbrich H, Weisser B. Effect of a six-week-intervention with an activity-promoting video game on isometric muscle strength in elderly subjects. International Journal of Computer Science in Sport (International Association of Computer Science in Sport) 2010;9(2):75-79 [FREE Full text]
    33. Beat Saber. Beat Games. 2018.   URL: https://beatsaber.com [accessed 2020-05-01]
    34. The ExerCube. Sphery Ltd. 2018.   URL: https://sphery.ch/der-exercube/ [accessed 2020-05-01]
    35. Hardy S, Dutz T, Wiemeyer J, Göbel S, Steinmetz R. Framework for personalized and adaptive game-based training programs in health sport. Multimed Tools Appl 2014 May 21;74(14):5289-5311. [CrossRef]
    36. Hardy S, Göbel S, Steinmetz R. Adaptable and personalized game-based training system for fall prevention. 2013 Presented at: 21st ACM international conference on Multimedia; Oct 21-25, 2013; Barcelona, Spain p. 431-432. [CrossRef]
    37. Hoffmann K, Hardy S, Wiemeyer J, Göbel S. Personalized Adaptive Control of Training Load in Cardio-Exergames-A Feasibility Study. Games for Health Journal 2015 Dec;4(6):470-479. [CrossRef] [Medline]
    38. Re-Mission. Hopelab. 2006.   URL: https://www.re-mission2.org [accessed 2020-05-01]
    39. Baranowski T, Baranowski J, Thompson D, Buday R, Jago R, Griffith MJ, et al. Video game play, child diet, and physical activity behavior change a randomized clinical trial. Am J Prev Med 2011 Jan;40(1):33-38 [FREE Full text] [CrossRef] [Medline]
    40. Fiellin LE, Hieftje KD, Pendergrass TM, Kyriakides TC, Duncan LR, Dziura JD, et al. Video Game Intervention for Sexual Risk Reduction in Minority Adolescents: Randomized Controlled Trial. J Med Internet Res 2017 Sep 18;19(9):e314 [FREE Full text] [CrossRef] [Medline]
    41. Virtual Reality Pain Reduction. College of Engineering, University of Washington.   URL: http://www.vrpain.com [accessed 2020-05-01]
    42. Nouchi R, Taki Y, Takeuchi H, Hashizume H, Akitsuki Y, Shigemune Y, et al. Brain training game improves executive functions and processing speed in the elderly: a randomized controlled trial. PLoS One 2012 Jan;7(1):e29676 [FREE Full text] [CrossRef] [Medline]
    43. Nouchi R, Taki Y, Takeuchi H, Hashizume H, Nozawa T, Kambara T, et al. Brain training game boosts executive functions, working memory and processing speed in the young adults: a randomized controlled trial. PLoS One 2013;8(2):e55518 [FREE Full text] [CrossRef] [Medline]
    44. Pokémon GO. Niantic, Incn. Google Play Store. 2016.   URL: https://play.google.com/store/apps/details?id=com.nianticlabs.pokemongo [accessed 2020-05-01]
    45. Gough C. Pokémon Go player spending worldwide from 2016 to 2019. Statista. 2020 Mar 5.   URL: https://www.statista.com/statistics/882474/pokemon-go-all-time-player-spending-countries/ [accessed 2020-05-01]
    46. Best of 2019 virtual reality: The top VR-only experiences as measured by gross revenue this year. Steam. 2019.   URL: https://store.steampowered.com/sale/2019_top_vr [accessed 2020-05-01]
    47. Beat Saber. Steam. 2018.   URL: https://store.steampowered.com/app/620980/Beat_Saber/ [accessed 2020-05-01]
    48. Play­For­ward: Elm City Stories. Schell Games. 2013.   URL: https://www.schellgames.com/games/playforward/ [accessed 2020-05-01]
    49. Dr. Kawashima’s Brain Training (Brain Age). Nintendo. 2005.   URL: https://www.nintendo.com/games/detail/brain-age-train-your-brain-in-minutes-a-day-wii-u/ [accessed 2020-05-01]
    50. Hoffman HG, Chambers GT, Meyer WJ, Arceneaux LL, Russell WJ, Seibel EJ, et al. Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures. Ann Behav Med 2011 Apr;41(2):183-191 [FREE Full text] [CrossRef] [Medline]
    51. Thompson D, Baranowski T, Buday R, Baranowski J, Thompson V, Jago R, et al. Serious Video Games for Health How Behavioral Science Guided the Development of a Serious Video Game. Simul Gaming 2010 Aug 01;41(4):587-606 [FREE Full text] [CrossRef] [Medline]
    52. Escape from Diab and Nanoswarm: Invasion from inner space. Archimage, Inc. 2006.   URL: http://archimage.com [accessed 2020-06-19]
    53. Görgen R, Huemer S, Schulte-Körne G, Moll K. Evaluation of a digital game-based reading training for German children with reading disorder. Computers & Education 2020 Jun;150:103834. [CrossRef]
    54. Debugger 3.16 Hack'n'Run. Spiderwork Games. 2017.   URL: http://www.spiderwork-games.com/debugger.htm [accessed 2020-05-01]
    55. Ninaus M, Kiili K, McMullen J, Moeller K. Assessing fraction knowledge by a digital game. Computers in Human Behavior 2017 May;70:197-206. [CrossRef]
    56. Kiili K, Ketamo H. Evaluating Cognitive and Affective Outcomes of a Digital Game-Based Math Test. IEEE Trans. Learning Technol 2018 Apr 1;11(2):255-263. [CrossRef]
    57. VocabiCar. Quantumfrog. 2017.   URL: https://vocabicar.de [accessed 2020-05-01]
    58. Orwell: Ignorance is Strength. Osmotic Studios. 2018.   URL: https://www.osmoticstudios.com/orwell-ignorance-is-strength/ [accessed 2020-05-01]
    59. Bertram J, Moskaliuk J, Cress U. Virtual training: Making reality work? Computers in Human Behavior 2015 Feb;43:284-292. [CrossRef]
    60. Learning game for children struggling in math and reading. Meister Cody. 2013.   URL: https://www.meistercody.com [accessed 2020-05-01]
    61. Debugger 3.16: Hack'n'Run. Steam. 2017.   URL: https://store.steampowered.com/app/339720/Debugger_316_HacknRun/ [accessed 2020-05-01]
    62. Semideus. Flow Factory Oy. 2016.   URL: http://www.flowfactory.fi/semideus/ [accessed 2020-05-01]
    63. Orwell: Keeping an Eye On You. Steam. 2016.   URL: https://store.steampowered.com/app/491950/Orwell_Keeping_an_Eye_On_You/ [accessed 2020-05-01]
    64. Orwell: Ignorance is Strength. Steam. 2018.   URL: https://store.steampowered.com/app/633060/Orwell_Ignorance_is_Strength/ [accessed 2020-05-01]
    65. Guthold R, Stevens GA, Riley LM, Bull FC. Worldwide trends in insufficient physical activity from 2001 to 2016: a pooled analysis of 358 population-based surveys with 1·9 million participants. Lancet Glob Health 2018 Oct;6(10):e1077-e1086 [FREE Full text] [CrossRef] [Medline]
    66. Schiesel S. Classes turn to video game that works legs. The New York Times 2007 Apr 30 [FREE Full text]
    67. Peng W, Crouse JC, Lin J. Using active video games for physical activity promotion: a systematic review of the current state of research. Health Educ Behav 2013 Apr;40(2):171-192. [CrossRef] [Medline]
    68. Gough C. All-time best-selling console games based on global unit sales as of April 2020. Statista. 2020 Apr 1.   URL: https://www.statista.com/statistics/264530/all-time-best-selling-console-games-worldwide/ [accessed 2020-04-01]
    69. Bartolomé NA, Zorrilla A, Zapirain B. Can game-based therapies be trusted? Is game-based education effective? A systematic review of the Serious Games for health and education. 2011 Presented at: 16th International Conference on Computer Games (CGAMES); July 27-30, 2011; Louisville, KY p. 275-282. [CrossRef]
    70. 2019 Essential Facts About the Computer and Video Game Industry. Entertainment Software Association. 2019.   URL: https://www.theesa.com/esa-research/2019-essential-facts-about-the-computer-and-video-game-industry/ [accessed 2020-05-01]
    71. Clement J. Most popular Google Play app categories 2020. Statista. 2020 May 5.   URL: https://www.statista.com/statistics/279286/google-play-android-app-categories/ [accessed 2020-05-01]
    72. Wouters P, van Nimwegen C, van Oostendorp H, van der Spek ED. A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology 2013;105(2):249-265. [CrossRef]
    73. Lü. Lü Interactive Playground. 2017.   URL: https://www.play-lu.com [accessed 2020-05-01]
    74. Garcia-Agundez A, Folkerts A, Konrad R, Caserman P, Tregel T, Goosses M, et al. Recent advances in rehabilitation for Parkinson's Disease with Exergames: A Systematic Review. J Neuroeng Rehabil 2019 Jan 29;16(1):1-17 [FREE Full text] [CrossRef] [Medline]
    75. Garcia-Agundez A, Folkerts A, Konrad R, Caserman P, Göbel S, Kalbe E. PDDanceCity: An Exergame for Patients with Idiopathic Parkinson’s Disease and Cognitive Impairment. 2017 Presented at: Mensch und Computer; Sep 10-13, 2017; Regensburg, Germany p. 381-386. [CrossRef]
    76. Giessen HW. Serious Games Effects: An Overview. Procedia - Social and Behavioral Sciences 2015 Feb;174:2240-2244. [CrossRef]
    77. Whyte EM, Smyth JM, Scherf KS. Designing Serious Game Interventions for Individuals with Autism. J Autism Dev Disord 2015 Dec;45(12):3820-3831. [CrossRef] [Medline]
    78. Ravyse WS, Seugnet Blignaut A, Leendertz V, Woolner A. Success factors for serious games to enhance learning: a systematic review. Virtual Reality 2016 Sep 20;21(1):31-58. [CrossRef]
    79. Graham K, Schofield D. Rock god or game guru: Using Rocksmith to learn to play a guitar. Journal of Music, Technology & Education 2018 Aug 01;11(1):65-82. [CrossRef]
    80. Ryan RM, Weinstein N. Undermining quality teaching and learning: A self-determination theory perspective on high-stakes testing. Theory and Research in Education 2009 Jun 25;7(2):224-233. [CrossRef]
    81. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, CONSORT. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg 2012;10(1):28-55 [FREE Full text] [CrossRef] [Medline]
    82. Koster R. Theory of fun for game design. Sepastopol, CA: O'Reilly Media Inc; 2013.
    83. Czikszentmihalyi M. Flow: The psychology of optimal experience. New York, NY: Harper & Row; 1990.
    84. Crockett L. Easy to learn, difficult to master: Accessible front-ends to challenging science assignments. 2016 Presented at: 10th European Conference on Games Based Learning: ECGBL 2016; Oct 6-7, 2016; University of the West, Paisley, Scotland p. 144-151.
    85. Dillon R. On the Way to Fun: An Emotion-Based Approach to Successful Game Design. Natick, MA: A K Peters, Ltd; 2010.
    86. Hunicke R, LeBlanc M, Zubek R. MDA: A formal approach to game design and game research. : Proceedings of the AAAI Workshop on Challenges in Game AI; 2004 Presented at: AAAI Workshop on Challenges in Game AI; July 25-29, 2004; San Jose, CA p. 1-5.
    87. Calvillo-Gámez E, Cairns P, Cox A. Assessing the core elements of the gaming experience. In: Game User Experience Evaluation. Cham, Switzerland: Sprinter International Publishing; 2015:37-62.
    88. Bartle R. Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD Research 1996;1(1):19 [FREE Full text]
    89. Scoresby J, Shelton BE. Visual perspectives within educational computer games: effects on presence and flow within virtual immersive learning environments. Instr Sci 2010 Feb 4;39(3):227-254. [CrossRef]
    90. Pivec M. Editorial: Play and learn: potentials of game-based learning. Br J Educ Technol 2007 May;38(3):387-393. [CrossRef]
    91. Whitehead A, Johnston H, Nixon N, Welch J. Exergame effectiveness: what the numbers can tell us. New York, NY, United States: Association for Computing Machinery; 2010 Presented at: 5th ACM SIGGRAPH Symposium on Video Games; July 28-29, 2010; Los Angeles, CA p. 55-62. [CrossRef]
    92. Ryan RM, Rigby CS, Przybylski A. The Motivational Pull of Video Games: A Self-Determination Theory Approach. Motiv Emot 2006 Nov 29;30(4):344-360. [CrossRef]
    93. Vorderer P, Hartmann T, Klimmt C. Explaining the enjoyment of playing video games: the role of competition. 2003 Presented at: Second International Conference on Entertainment Computing; May 8-10, 2003; Pittsburgh, PA p. 1-9.
    94. Slater M, Wilbur S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 1997 Dec;6(6):603-616. [CrossRef]
    95. Ermi L, Mäyrä F. Fundamental components of the gameplay experience: Analysing immersion. Worlds in play: International perspectives on digital games research 2005;37(2):37-53 [FREE Full text]
    96. Caserman P, Garcia-Agundez A, Goebel S. A Survey of Full-Body Motion Reconstruction in Immersive Virtual Reality Applications. IEEE Trans. Visual. Comput. Graphics 2019:1-20. [CrossRef]
    97. Eller C, Bittner T, Dombois M, Rüppel U. Collaborative immersive planning and training scenarios in VR. 2018 Presented at: Workshop of the European Group for Intelligent Computing in Engineering (EG-ICE); June 10-13, 2018; Lausanne, Switzerland p. 164-185. [CrossRef]
    98. Caserman P, Zhang H, Zinnäcker J, Göbel S. Development of a directed teleport function for immersive training in virtual reality. 2019 Presented at: International Conference on Virtual Worlds and Games for Serious Applications (VS-Games); Sep 4-6, 2019; Vienna, Austria p. 1-8. [CrossRef]
    99. Minecraft. Mojang. 2009.   URL: https://www.minecraft.net/ [accessed 2020-05-01]
    100. Cummings JJ, Bailenson JN. How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence. Media Psychology 2015 May 15;19(2):272-309. [CrossRef]
    101. Kern AC, Ellermeier W. Audio in VR: Effects of a Soundscape and Movement-Triggered Step Sounds on Presence. Front. Robot. AI 2020 Feb 21;7(20). [CrossRef]
    102. Sanders T, Cairns P. Time perception, immersion and music in videogames. 2010 Presented at: 24th BCS Conference on Human Computer Interaction; Sep 6-10, 2010; Dundee, UK p. 160-167. [CrossRef]
    103. Wiemeyer J. Towards a generic framework for serious games. 2019 Presented at: 12th International Symposium on Computer Science in Sport (IACSS); July 8-10, 2019; Moscow, Russia p. 193-200. [CrossRef]
    104. Desurvire H, Wiberg C. Game User Experience Evaluation. In: User Experience Design for Inexperienced Gamers: GAP—Game Approachability Principles. Cham, Switzerland: Springer; 2015:169-186.
    105. Andersen E, O'Rourke E, Liu YE, Snider R, Lowdermilk J, Truong D, et al. The impact of tutorials on games of varying complexity. 2012 Presented at: SIGCHI Conference on Human Factors in Computing Systems; May 5-10, 2012; Austin, TX p. 59-68. [CrossRef]
    106. Marks DW, Rispen L, Calara G. Greater physiological responses while playing XBox Kinect compared to Nintendo Wii. International Journal of Exercise Science 2015;8(2):7 [FREE Full text] [CrossRef]
    107. Caserman P, Martinussen M, Göbel S. Effects of end-to-end latency on user experience and performance in immersive virtual reality applications. 2019 Presented at: Joint International Conference on Entertainment Computing and Serious Games; Nov 11-15, 2019; Arequipa, Peru p. 57-69. [CrossRef]


    Abbreviations

    SG-MDF: DIN SPEC 91380 Serious Games Metadata Format


    Edited by M Birk; submitted 03.04.20; peer-reviewed by M Kickmeier-Rust, H Söbke; comments to author 02.05.20; revised version received 14.05.20; accepted 29.05.20; published 24.07.20

    ©Polona Caserman, Katrin Hoffmann, Philipp Müller, Marcel Schaub, Katharina Straßburg, Josef Wiemeyer, Regina Bruder, Stefan Göbel. Originally published in JMIR Serious Games (http://games.jmir.org), 24.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.