A Serious Game (Immunitates) About Immunization: Development and Validation Study

Background: Vaccination is a fundamental part of all levels—local to worldwide—of public health, and it can be considered one of humanity's greatest achievements in the control and elimination of infectious diseases. Teaching immunization and vaccination can be monotonous and tiring. It is necessary to develop new approaches for teaching these themes in nursing school. Objective: We aimed to develop and validate a serious game about immunization and vaccination for Brazilian nursing students. Methods: We developed a quiz-type game, Immunitates, using design and educational theoretical models and Brazilian National Health Guidelines. The game’s heuristics and content were evaluated with 2 different instruments by a team of experts. A sample of nursing students evaluated the validity of the game’s heuristics only. We calculated the content validity index (CVI) for each evaluation. Results: The study included 49 experts and 15 nursing students. All evaluations demonstrated high internal consistency (Cronbach α≥.86). The game’s heuristics (experts: CVI 0.75-1.0; students: CVI 0.67-1.0) and the game’s contents demonstrated validity (experts: CVI 0.73-1.0). Participants identified some specific areas for improvement in the next version. Conclusions: The serious game appears to be valid. It is intended as a support tool for nursing students in the teaching–learning process and as a tool for continuing education for nurses. (JMIR Serious Games 2022;10(1):e30738) doi: 10.2196/30738


Introduction
Vaccination, a fundamental part of public health, is one of humanity's greatest achievements in the control and elimination of infectious diseases [1]. The benefits of vaccines have been proven; in the last few decades, as an example, smallpox was eradicated through immunization [2]. Vaccines have saved lives; reduced the incidence of polio by 99% in the world; and reduced diseases, disabilities resulting from diseases, and deaths caused by diphtheria, tetanus, and whooping cough [2].
Immunization of the population can decrease the transmission of infectious diseases, in turn decreasing hospitalizations, public expenditures on health care, and population mortality, which may increase the life expectancy of a population. Immunization will always be a necessary health action. Each year, 130 million babies are born [3], and they all have the right to receive protection against vaccine-preventable diseases.
In Brazil, it is necessary to abide by the recommended vaccination schedule [4] for children to be considered immunized. Thus, it is important that those responsible for a child are knowledgeable about the importance of vaccination-why the entire population must take the recommended vaccines-and that myths are debunked and explained.
Therefore, it is important that health professionals, especially nurses, have theoretical knowledge that offers a foundation and security to organize and promote health education. In this sense, nursing students must learn enough to provide health education to their future patients. Furthermore, in Brazil, nursing duties in primary care include administering vaccinations and managing everything related to them to ensure the safety of the immunobiological materials, and consequently, that of the patient [5].
In Brazil, the theoretical topics related to vaccination are mostly found in manuals and ordinances-technical texts that can be long, boring, and unappealing-resulting in dense and exhaustive reading for the student. We believe the use of a different methodology (serious games) in the teaching-learning process will help teachers to stimulate learning in the classroom and students to reinforce and review their knowledge on immunization. By using new methodologies, teaching can be more dynamic and the pattern of traditional vertical teaching, wherein the teacher demonstrates and the student repeats, can be broken [6]. Breaking this pattern of traditional education is important given the profile of the current generation of students-they are no longer just listeners but critical protagonists in the process of building knowledge [7,8].
The use of educational technologies in nursing has gradually increased over the years in Finland [9] and in other regions (Norway, other European countries, Asia, and Brazil [10]). To the best of our knowledge, there are no digital serious games related to vaccination in Brazil that are specifically for nursing students and nurses. We aimed to develop serious game about vaccine-preventable diseases and immunization and validate the game's contents and heuristics.

Framework
As a theoretical frame of reference for the creation and development of the game, we used the Elemental Tetrad (Story, Aesthetics, Mechanics, Technology) [11] and the first 3 levels (Remembering, Understanding, and Applying) of the Revised Bloom's Taxonomy [12], as they have been commonly used in studies [13][14][15][16] that addressed the development of educational health technologies.
The contents of the game (a database with questions, answers, and feedback) were based on the Brazilian Ministry of Health's Manual of Rules and Procedures for Vaccination [17].

Story
In the game Immunitates (Figure 1), the player assumes the role of nurse in the vaccination room and answers questions on immunization. When the players correctly answers a sufficient number of questions, their nurse avatar is promoted. The aim is to reach the last level of the game when the nurse reaches the role of Minister of Health.
Feedback on each question is immediate, thus maintaining the user's motivation and engagement in the game [18].

Aesthetics
The player is asked to choose a female or male avatar ( Figure  2), which changes job positions as the player advances to the next levels.

Mechanics
The game has a quiz format, and players progress by levels ( Figures 3-5), which are unlocked as the player answers questions correctly. The game has 7 levels, each corresponding to a job position that the nurse avatar achieves ( Figure 6). Each level has a bank of questions, from which a specific number of questions are randomly chosen (Table 1). For example, level 1 has a bank of 5 questions, and each time the player accesses the level, 3 of the 5 questions are chosen. The player must correctly answer at least 1 question for the next level to be unlocked. If the user does not correctly answer the minimum number of questions, they fail the level but can immediately play it again.     Correctly answered questions to progress to the next level, n Questions played, n Questions in database, n Level 1 3 5 1 Immunization room nurse  (Figure 7), and text is automatically displayed to explain the correct answer, as automatic rationale feedback, if the player selects the incorrect option ( Figure 8). Questions written with direct and simple wording and that require players to remember information were considered easy questions. Questions that require complex clinical reasoning were considered difficult and are presented later on in the game. For better visualization of the question and answer texts, questions could be up to 150 characters long, while each answer and feedback could be up to 120 characters.
Throughout the game, the player listens to a theme song (which can be muted on the achievements screen). The player also has the option to reset the game on the information screen (not pictured), this action restarts the game completely and all progress is lost.

Technology
The game was programmed in January 2019 using C++ and JavaScript for mobile phones with Android and iOS operating systems; the game is available in Portuguese for free from the Google Play Store and the Apple App Store.

Game Validation
This study was undertaken with 2 different independent samples: (1) experts in nursing education technologies, nursing games, or immunization and (2) nursing students.

Experts
Inclusion criteria for experts were adapted from an existing model [19] that is widely used in Brazil to select experts for validation studies [20]. It was a mandatory criterion to have a Bachelor of Science degree in Nursing. An additional criterion was having specialized knowledge, clinical practice, or authorship in areas related to immunization, technology, or health education.
We actively searched 90 public nursing colleges in Brazil for nursing professors who had a history of researching nursing education technologies, nursing games, or immunization. We contacted 983 professors by email, inviting them to validate the game's heuristics and content. The email had 2 different links: one to download the game, and the other to provide demographic information (age, gender, time working as a nurse, and additional degrees or certifications) to indicate that they accepted to be part of the study. Participants received a second email with links to heuristics and content evaluation forms.

Students
Inclusion criteria for students were being an undergraduate nursing student enrolled at University of Brasília Ceilândia College in the discipline Comprehensive Care for Women and Children's Health at the time of data collection. In this discipline, students learn about the Brazilian vaccination calendar and immunization, and students were invited, during a class, to take part in this study. Students who agreed to participate were sent an email with 3 links: one to download the game, one to provide demographic information (age and gender), and a third to validate the game's heuristics only.

Ethics
The study was conducted in accordance with the Helsinki

Evaluations
Game heuristics were evaluated by both experts and nursing students using Avaliação Heurística para Jogos Educacionais Digitais (AHJED, Heuristic Evaluation for Digital Educational Games) [22]), which has 8 dimensions-interface, playability, multimedia, artificial intelligence, game's story, educational elements, contents, and educational agent-evaluated on a Likert scale (Multimedia Appendix 1).
Game contents were evaluated by experts using an instrument [23] with 3 dimension-objectives, structure and presentation, and relevance (Multimedia Appendix 2) and by omitting items about functionalities that were neither present nor intended to be present in the game.
Participants also had the option to leave comments about their experience playing the game and suggestions about what could be improved in the next version.

Analysis
Statistical analysis was performed using SPSS software (version 22; IBM Corp).
The content validity index (CVI) is widely used in the field of health [24][25][26] and is used to assess the proportion of agreement for an item of an instrument [27]. The index was calculated in 2 ways: For the game heuristics evaluation, the number of agree and strongly agree responses were summed and divided by the total number of responses. For the game content evaluation, the number of adequate and totally adequate were summed and divided by the total number of responses. Items with CVI≥0.80 would remain as they are, whereas items with CVI<0.80 should be changed for the next version of the game.

Results
Of those invited, 184 nursing professors provided demographic information in response to the first email, but only 49 completed the game validation and met inclusion criteria; therefore, 49 experts took part in this study. Of the 49 experts, 43 experts (88%) were female, and 6 (12%) experts were male. Experts ranged from 28 to 63 years old (mean 44.04 years old), with time working as a nurse ranging from 3 to 40 years. Of 48 students enrolled in the discipline, 15 took part in the study. Of the 15 students, 14 (93%) students were female, and 1 (7%) student was male. Students ranged from 20 to 28 years old (mean 22.13 years old).
Overall, the CVI ranged from 0.77 to 0.97. For heuristics, the CVI ranged from 0.75 to 1 in the expert group and 0.67 to 1 in the student group. Cronbach α was always greater than 0.86 (  (Table 4).  Table 4. Comments and suggestions from experts and nursing students.

Comment Participant
"I suggest including more in-depth questions on the topic, to stimulate research and interest in the continuity of the game." Expert 2 "I believe that the feedback when we make mistakes should come with a suggestion for reading." Expert 4 "I suggest you remove the circles that are moving around, it takes your attention and makes it tiring." Expert 7 "Suggestion: enable a 'learn more' hyperlink; the player will be able to download the vaccination calendar, vaccination manuals and be redirected to the Ministry of Health webpage."

Discussion
The results present evidence of validity and suggest that the game can be used by both nurses and nursing students alike, as the game was tested and well accepted by them. We noticed that digital serious games about vaccination have up until now been developed for children [30,31].
Evaluations demonstrated high internal consistency (Cronbach α≥.86) [32]. Although most items within the evaluated heuristics were greater than 0.80, a few-level of difficulty (students: CVI 0.73), partition of the content (students: CVI 0.67), player's performance (students: CVI 0.67), and aesthetics of the game (experts: CVI 0.75; students: CVI 0.67)-did not reach this stipulated number; these features will be reworked for the next version of the game based on suggestions: The content will also be reworked for the next version of the game.
According to Korhonen [38], playability should not be the main objective of a game, although it is necessary for a positive experience, as good playability positively affects the user experience. Our results confirm this, since both experts (CVI 0.87) and students (CVI 0.88) indicated playability was acceptable, and expert and student participants commented on their good experience and how they liked the game:

I really liked the game. [Expert 49]
Content is perhaps the most important part of an educational tool. From participants' suggestions, it was clear that both experts and students would have liked to have an external link that could provide theoretical support as a way to further their knowledge.
Expert 2 suggested including more in-depth questions to stimulate research and interest in the player, which corroborates the findings of a previous study [39], namely, that the difficulty of a game can affect player motivation. The game must have challenging elements but still leave the player confident enough to overcome them. Experts 27 and 28 also commented on the need for more complex questions that would need clinical reasoning, but expert 25 thought that the questions should be simpler, shorter, and easier to understand. However, players may have different skills; what is classified as difficult, a player may perceive to be very easy or very difficult, which can lead players to feeling frustrated or even bored, which decreases their motivation and engagement with a game [40].
It is also true that technology-specific problems can arise when using educational technologies, such as issues downloading or installing and log-in, audio, and video-related problems. Students can find this method of teaching to be unengaging or they may simply not find the time to study using these new tools [41].
During the COVID-19 pandemic, when teachers and students worldwide had to stay at home, a new challenge arose in health care education. Somehow training in a field that is traditionally taught with a hands-on approach had to continue in alternative manners. During the first wave, educators began to utilize new educational technologies to continue teaching. The impact of these technologies in the field of health care education is unique because educators must continue training future professionals who will soon be working in-person at hospitals amid the pandemic, despite social distancing [42]. Although many educators have had to improvise and quickly learn how to use these new technologies during the COVID-19 pandemic, the use of technology in the teaching-learning process should be driven by pedagogical needs and goals and not by technological pressures [43].
When elements of gamification, such as points, achievements, and levels, are incorporated into the undergraduate teaching-learning process, there is a positive effect on student motivation and performance [44]. Motivation is the state in which the individual feels moved to do something. According to the Self-Determination Theory, motivation is separated into levels (from a little to a lot of motivation) and orientation (intrinsic and extrinsic motivation). In extrinsic motivation, the individual seeks reward or escapes punishment, whereas intrinsic motivation refers to pleasure to do or an inherent personal satisfaction [45]. Harandi [46] highlights that when students are motivated to learn, they are likely to be involved; when they are involved, they are more likely to achieve educational goals.
There were a few limitations in this study. The budget available for the construction of the game did not allow for all intended features to be included in this version. Suggestions made by the participants were documented so that, in the future, they can be implemented.
The number of students who participated was small, which can alter the results of the evaluation of educational technology. Students who had already completed the discipline Comprehensive Care for Women and Children's Health were not included, even though they already had the necessary knowledge to validate the game, because these students were engaged in course work outside the university, and we had difficulty contacting them.
In future studies, we intend to test the effectiveness of the game by comparing knowledge acquisition from class only compared with that from the combination of class and playing the game to determine if this game is useful in helping nursing students. We also intend to test graduate nurses' satisfaction with the game and knowledge acquisition from the game to determine if it is useful to this target audience. With these results, we would be able to categorically affirm that the game is useful to both nurses and nursing students.
Immunitates presents evidence of validity, even though some areas of the game require improvement. Immunitates is designed to be a support tool for nursing students (ie, not to replace face-to-face class instruction) and to be a tool for continuing education for nurses.