Published on in Vol 10, No 1 (2022): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/31644, first published .
How, for Whom, and in Which Contexts or Conditions Augmented and Virtual Reality Training Works in Upskilling Health Care Workers: Realist Synthesis

How, for Whom, and in Which Contexts or Conditions Augmented and Virtual Reality Training Works in Upskilling Health Care Workers: Realist Synthesis

How, for Whom, and in Which Contexts or Conditions Augmented and Virtual Reality Training Works in Upskilling Health Care Workers: Realist Synthesis

Review

1Division of Nursing, Midwifery and Social Work, University of Manchester, Manchester, United Kingdom

2Centre for Health Informatics, Division of Informatics, Imaging and Data Sciences, University of Manchester, Manchester, United Kingdom

3Division of Population Health, Health Services Research and Primary Care, University of Manchester, Manchester, United Kingdom

4Manchester Academic Health Science Centre, University of Manchester, Manchester, United Kingdom

Corresponding Author:

Norina Gasteiger, BA, BHSc, MPH

Division of Nursing, Midwifery and Social Work

University of Manchester

Jean McFarlane Building

Oxford Road

Manchester, M13 9PL

United Kingdom

Phone: 44 01613067798

Email: norina.gasteiger@postgrad.manchester.ac.uk


Background: Using traditional simulators (eg, cadavers, animals, or actors) to upskill health workers is becoming less common because of ethical issues, commitment to patient safety, and cost and resource restrictions. Virtual reality (VR) and augmented reality (AR) may help to overcome these barriers. However, their effectiveness is often contested and poorly understood and warrants further investigation.

Objective: The aim of this review is to develop, test, and refine an evidence-informed program theory on how, for whom, and to what extent training using AR or VR works for upskilling health care workers and to understand what facilitates or constrains their implementation and maintenance.

Methods: We conducted a realist synthesis using the following 3-step process: theory elicitation, theory testing, and theory refinement. We first searched 7 databases and 11 practitioner journals for literature on AR or VR used to train health care staff. In total, 80 papers were identified, and information regarding context-mechanism-outcome (CMO) was extracted. We conducted a narrative synthesis to form an initial program theory comprising of CMO configurations. To refine and test this theory, we identified empirical studies through a second search of the same databases used in the first search. We used the Mixed Methods Appraisal Tool to assess the quality of the studies and to determine our confidence in each CMO configuration.

Results: Of the 41 CMO configurations identified, we had moderate to high confidence in 9 (22%) based on 46 empirical studies reporting on VR, AR, or mixed simulation training programs. These stated that realistic (high-fidelity) simulations trigger perceptions of realism, easier visualization of patient anatomy, and an interactive experience, which result in increased learner satisfaction and more effective learning. Immersive VR or AR engages learners in deep immersion and improves learning and skill performance. When transferable skills and knowledge are taught using VR or AR, skills are enhanced and practiced in a safe environment, leading to knowledge and skill transfer to clinical practice. Finally, for novices, VR or AR enables repeated practice, resulting in technical proficiency, skill acquisition, and improved performance. The most common barriers to implementation were up-front costs, negative attitudes and experiences (ie, cybersickness), developmental and logistical considerations, and the complexity of creating a curriculum. Facilitating factors included decreasing costs through commercialization, increasing the cost-effectiveness of training, a cultural shift toward acceptance, access to training, and leadership and collaboration.

Conclusions: Technical and nontechnical skills training programs using AR or VR for health care staff may trigger perceptions of realism and deep immersion and enable easier visualization, interactivity, enhanced skills, and repeated practice in a safe environment. This may improve skills and increase learning, knowledge, and learner satisfaction. The future testing of these mechanisms using hypothesis-driven approaches is required. Research is also required to explore implementation considerations.

JMIR Serious Games 2022;10(1):e31644

doi:10.2196/31644

Keywords



Background

As in most businesses, upskilling health care workers is vital to improving and advancing existing skills and practices and closing gaps in knowledge so that employees may continue practicing with ease [1,2]. By definition, upskilling is the process of refining existing skills or learning new skills [1]. Within the health care sector, upskilling is required to promote workforce flexibility, skill delegation, and adaptation during times of change, restructuring, or crisis [3-5]. Ultimately, this also ensures that health care delivery is safe, aligns with best practice, and is standardized across staff.

Traditional health care training consists of role modeling, shadowing, and the see one, do one method [6-9], along with learning through textbooks, e-learning, workshops, and seminars, as well as reading peer-reviewed journal articles. Simulation-based methods have also traditionally been used in upskilling, training, and engaging health and care providers in continued education, with the ultimate purpose of practical learning to improve patient safety [10]. These include part- or full-body manikins, synthetic latex–based simulation models, bench-top simulators, human actors, and live animal and cadaveric procedures. However, a lack of time, inaccessible resources, and a tendency to rely on experiential knowledge limit the ability to upskill [11,12]. Furthermore, training with traditional simulators is becoming difficult because limits are placed on work hours [13], and opportunities for learners to practice technical procedures on live animals and humans or cadavers are reduced because of ethical issues, commitment to patient safety, cost, and limited availability of resources [13-15].

Virtual reality (VR) and augmented reality (AR) training programs may help to overcome these barriers because they can be continuously available and used independently by learners, and they do not increase costs with use [16,17]. Akin to traditional simulation methods, VR and AR training programs enable repeated practice within safe environments away from patients and stress or time pressure [13,18,19]. VR and AR have already helped to upskill registered health care professionals on disaster response [20,21], technical and behavioral skills [7,17,22-24], and nontechnical cognitive skills [25-27].

VR is a computer-generated simulated environment in which users are immersed [28,29]. However, immersion levels can vary greatly. For example, in nonimmersive VR, environments can simply be projected onto computer screens, whereas in fully immersive VR, users wear a headset to feel as though they have been transported into a digital environment. In contrast, AR is the projection of computer-generated imagery (eg, objects) onto real-world environments [28-30], with mixed reality enabling the objects to be responsive, interactive, and spatially aware [28,29].

The effectiveness and success of VR and AR training programs is often nonlinear and complicated. This is because fidelity and perceptions of immersion depend on various dimensions. Fidelity refers to the extent to which an experience is close to reality [31]. Accordingly, the five dimensions that influence fidelity include physical (ie, a simulated environment), psychological (eg, stress and emotions), social, group culture, and open-mindedness of the user [32,33]. The extent to which a simulation is perceived as good or realistic also depends on a user’s willingness to believe in it [34]. Ultimately, this may require detail such as object collision detection (and response) or haptic technology for physical force feedback and tactile sensation [34]. These tools can introduce an additional dimension to VR by enabling users to interact with systems or manipulate digital objects through touch.

Previous literature reviews have focused on the novelty, application, and effectiveness of VR and AR training programs for health professionals, including for surgical training [13,15,18,19,35-37], nontechnical skills training [25], urology [38], disaster training [21], and dementia care [39], as well as to assess their cost-effectiveness compared with traditional simulators [40]. The reviews suggest that VR and AR may be effective for training various health care providers in both technical and nontechnical skills. However, research has also found that VR and AR training programs do not work for all learners, such as those who already have experience in a skill [14,41]. VR and AR learning methods are also sometimes reported as equal to, but not better than, traditional learning methods when used by nursing students [42-44] and other tools used in phlebotomy training [45]. In addition, the literature on implementing VR and AR in training for practicing health professionals is limited.

This realist review explores why there is variation in the effectiveness of VR and AR training programs and what factors influence their implementation and maintenance. Realist reviews can help to understand how, for whom, and in which contexts and conditions interventions or programs (such as the use of AR or VR for training) work. They offer a theory-driven approach to producing causal explanations of how different mechanisms of action may be triggered, which then lead to intended and unintended outcomes [46,47]. Mechanisms are changes in reasoning or individual or collective reactions (eg, behaviors, perceptions of fidelity, or cybersickness) to an intervention’s resources [46]. These mechanisms are triggered under certain circumstances, contexts, or conditions, which may relate to training scenarios, populations, or diverse AR and VR technologies.

Ultimately, a program theory developed in alignment with realist methods will result in a collection of context-mechanism-outcome (CMO) configurations that consider context, mechanisms, and outcomes. The program theory explains how an intervention may contribute to a chain of events (ie, mechanisms) that result in expected and desired or unexpected outcomes. The realist approach also considers how interventions may work differently within different contexts or conditions. CMO configurations are presented as follows:

Context + Mechanisms = Outcomes

Underlying the realist methodology is the expectation that the VR or AR intervention does not produce outcomes by itself but is instead influenced by underlying social entities, processes, or social structures (mechanisms) [46,48]. This means that it also uncovers how an intervention works in practice and results in a transferable program theory [48] that considers demi-regularities (semipredictable outcomes), which may result in varying outcomes but consistent CMO patterns [47].

Objectives

The aim of this realist review is to develop, test, and refine an evidence-informed program theory on how, for whom, and to what extent training using AR or VR works for upskilling health care workers and to understand what facilitates or constrains their implementation and maintenance.

The review addressed the following questions:

  1. How, for whom, and to what extent does training using AR or VR for upskilling health care workers work?
  2. What facilitates or constrains the implementation (and maintenance) of training using AR or VR in health and care settings?

Overview

This realist review adheres to the processes explained in the RAMESES (Realist and Meta-narrative Evidence Syntheses: Evolving Standards) training documents [48]. Our protocol describes the methods in more detail [2]. In addition, we report the review in accordance with the RAMESES publication standards for realist syntheses [49]. The review followed a 3-step process, consisting of theory elicitation, theory testing, and theory refinement.

Step 1. Elicit Theory

Search and Screening

The purpose of the first step was to elicit an initial program theory from candidate theories found within existing literature, which could then be refined and tested. Academic and practitioner theories were located by searching a range of databases and practitioner journals for literature on using AR or VR to upskill health professionals. The databases, search terms, and eligibility criteria are presented in Textbox 1. No constraints were imposed on the dates of publication. Learning and technology adoption theories were identified within this literature. The search was conducted between January 18 and January 25, 2021.

Search strategy and eligibility criteria.

Search locations

  • Databases
    • MEDLINE
    • Scopus
    • CINAHL
    • Embase
    • Education Resource Information Centre
    • PsycINFO
    • Web of Science
  • Journals
    • Academic Medicine
    • MedEdPORTAL
    • Medical Teacher
    • International Journal of Medical Education
    • Journal of Continuing Education in the Health Professions
    • GMS Journal for Medical Education
    • Focus on Health Professional Education
    • Medical Education
    • Journal of Nursing Education and Practice
    • Nurse Education Today
    • International Journal of Nursing Studies

Search strategy keywords

  • Keywords with Boolean operators AND and OR (asterisk [*] indicates other variations that are covered (eg, nurs* includes nurses, nurse, nursing)
    • augmented reality OR virtual reality AND health* OR care* OR nurs* OR doctor OR surgeon AND training OR upskilling OR skill OR education AND evaluation OR implementation OR feasibility OR effectiveness
  • Search example (Scopus)
    • TITLE-ABS-KEY (augmented AND reality OR virtual AND reality) AND TITLE-ABS-KEY (health* OR care* OR nurs* OR doctor OR surgeon) AND TITLE-ABS-KEY (training OR upskilling OR skill OR education) AND TITLE-ABS-KEY (evaluation OR implementation OR feasibility OR effectiveness)

Eligibility criteria for papers identified in databases and journals

  • Inclusion criteria
    • Using simulation technologies (any type of immersion)
    • Health workers, care workers, and postgraduate or registered learners
    • Any health, care, or university-based setting
    • Covers detail on what contexts, how, and for whom they worked or on implementation (or maintenance)
    • Published in English
  • Exclusion criteria
    • Simulation technologies that do not use augmentation or virtual reality (eg, web-based e-learning interventions or manikins)
    • Undergraduate students
    • Published in languages other than English
  • Exceptions
    • Work including undergraduate learners or other simulation technologies can be included if the data for postgraduate or registered learners and augmented reality or virtual reality can be separated
Textbox 1. Search strategy and eligibility criteria.

In alignment with previous realist reviews (eg, the study by Wong et al [50]), we conducted a 2-stage screening process, with a second researcher independently screening a random subset of papers. First, an author (NG) screened the title and abstract of each paper against the inclusion and exclusion criteria (Textbox 1) and generated a shortlist of possibly eligible papers. The full texts of these papers were then screened in the second stage. A second author (DD) independently screened a random selection of 20.2% (39/193) of the abstracts and titles and 20% (18/90) of the full texts. The raw interrater agreement rates for the 2 screening rounds were 85% and 89%, respectively. Discussion helped to reach consensus, with a third author (SNvdV) acting as a moderator.

Data Extraction

Data were extracted by 2 authors (NG and DD) into a coding sheet on Excel (Microsoft Corporation). This included information on the study (eg, author, date, title, research design, and sample), the intervention, contexts, mechanisms, outcomes, learning or technology adoption theories mentioned, and barriers and facilitators to implementation (or maintenance; see Table S1 in Multimedia Appendix 1). One author extracted all the data, whereas the second author reviewed 20% (16/80) of the papers for consistency. When complete CMO configurations were not provided, fragments were recorded.

Analysis

A narrative synthesis was conducted to determine overlapping CMO configurations and the most common barriers and facilitators to implementation and maintenance. We aggregated authors’ hypothesized mechanisms, regardless of whether they had been tested, to identify the common ways in which VR or AR affect and lead to the outcomes. The learning and technology adoption theories were also summarized and used to discuss and make meaning of the CMO configurations (in step 2).

Finally, the research team discussed the initial program theory and selected a number of CMO configurations to test, focusing on those that were expected to be most feasible, measurable, and likely to apply or transfer to future AR and VR interventions aimed at upskilling health care workers.

Step 2. Test Theory

Search and Screening

The purpose of step 2 was to test the initial program theory, using existing evidence. Empirical literature was identified in a 2-step process. First, empirical studies were identified from the first search by removing nonempirical and non–full-length papers. Second, the same search as in step 1 was repeated but with a time frame of 3-6 months to identify recently published work that may have been missed. This search was conducted on March 8, 2021. We used the same screening process as in step 1 to assess the relevance of newly identified articles. The first author (NG) screened the papers to identify a shortlist of possibly eligible papers. The second author (DD) then independently screened a random selection of these papers (abstracts and titles: 2/9, 20%; full texts: 1/2, 50%), with interrater agreement rates of 100%.

Data Extraction and Quality Appraisal

The same items as in step 1 were extracted, along with specific evidence for the mechanisms (where applicable) and the expected outcomes identified in the initial program theory. Studies that did not provide evidence relating to the outcomes were excluded. Studies were assessed for quality using the Mixed Methods Appraisal Tool (MMAT; version 2018) [51].

The MMAT consists of 2 screening questions and 5 study design–specific criteria that could be scored 1 (yes) or 0 (no) [51]. In keeping with the studies by Pluye et al [52], Mogharbel et al [53], and Vusio et al [54], we calculated quality scores for each article and classified them as low quality (≤40%, ie, meeting 1-2 criteria), moderate quality (60%-80%, ie, meeting 3-4 criteria), or high quality (100%, ie, meeting all 5 criteria).

The quality of all the studies was assessed by 1 author (NG), whereas a second author (DD) assessed the quality of 22% (10/46) of the studies. We calculated the Cohen κ using SPSS software (version 23; IBM Corp) to determine the interrater reliability between the 2 authors.

Step 3. Refine Theory

To refine the theory, evidential fragments (parts of studies, rather than entire studies, that provided evidence) from the second search were compared and matched to the initial program theory. We made revisions by identifying differences and presented the final theory as a narrative and diagrammatic summary. The most commonly identified learning or technology adoption theories were used to discuss the program theory.

We then assessed our confidence in each CMO configuration as high, moderate, low, or very low according to the criteria presented in Table 1. The confidence level was determined by the criterion with the lowest level. For example, if a CMO configuration had 7 supporting studies, with 4 (57%) of them contesting, and an average MMAT score of 90%, the CMO configuration was deemed low confidence.

Table 1. Criteria used to determine confidence in each context-mechanism-outcome configuration.
ConfidenceNumber of supporting studiesContesting studies (if applicable), %MMATa average score, %
High≥80-2076-100
Moderate5-721-2951-75
Low430-7426-50
Very low≤375-1000-25

aMMAT: Mixed Methods Appraisal Tool.


Search Outcome

The extended PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart [55] in Figure 1 shows the identification and screening process.

Figure 1. Extended PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart depicting the literature search and screening process.
View this figure

Theory Elicitation

The initial search identified 1042 papers. After deduplication and abstract and title screening, 186 full texts, including 8 studies snowballed from the literature, were reviewed, of which we excluded 106 (56.9%), leaving 80 (43.01%) papers for inclusion for eliciting the initial theory. The most common reasons for exclusion were not including health care workers (39/106, 36.8%), not focusing on education and training (29/106, 27.4%), or relevant information not being separable (17/106, 16%).

Theory Testing

The second search identified 46 recently published empirical studies. After deduplication and abstract and title screening, 7 full texts were screened, of which 5 (71%) were excluded because they did not cover AR or VR (3/5, 60%), did not include health care workers (1/5, 20%), or did not focus on education or training (1/5, 20%). Of the 7 studies, the 2 (29%) that remained were combined with the empirical literature from the first search (n=54). Of these 56 studies, 46 (82%) were included in testing and refining the theory, after 10 (18%) were excluded for not providing evidence on the CMO configurations.

Characteristics of the Included Articles

Theory Elicitation

The 80 papers identified in the first search consisted of empirical research (55/80, 69%), literature reviews (22/80, 28%), case reports (2/80, 3%), and cost-benefit analyses (1/80, 1%). Of these, 83% (66/80) focused on VR, 11% (9/80) on AR, and 6% (5/80) focused on both.

Theory Testing

Of the 46 empirical studies included in the second stage of the review, almost half (22/46, 48%) were quantitative descriptive studies [8,9,14,22,41,56-72], 11 (24%) were randomized controlled trials [6,7,17,20,23,73-78], 7 (15%) were quantitative nonrandomized studies [16,24,79-83], 5 (11%) were mixed methods studies [84-88], and 1 (2%) was a qualitative study [89]. They were published between 1999 [80] and 2021 [59,60,83]. Of the 46 studies, 21 (46%) were conducted in the United States [7,14,17,20,22,56,58,61,63-66,72,73,75,77,78, 80-82,88], 4 (9%) in the United Kingdom [68,84-86], 3 (7%) each in China [24,67,70] and India [23,76,89], and 2 (4%) each in Germany [59,87], Taiwan [6,71], Italy [69,83], and France [57,79]. Of the 46 studies, 2 (4%) did not provide a location [8,62] and 1 (2%) study was conducted in each of the following countries: Spain [16], Canada [74], Malaysia [60], the Netherlands [9], and Australia [41].

A range of health care professionals participated, including surgeons, nurses, physicians, pharmacists, technicians, social workers, radiologists, community health workers, ophthalmologists, dentists, and respiratory therapists. Clinical experience ranged from <2 months [17] to 30 years [67]. Sample sizes ranged from 6 [24] to 109 [71] health care professionals and trainees, with an overall mean of 34.3 (SD 25.8) participants and a total of 1543 participants (of the 46 studies, 1, 2%, did not report a sample size). For those that provided a mean age, participants ranged in age from 19 years [71] to 43.7 years [87]. The characteristics of the included studies are presented in Table S2 in Multimedia Appendix 1.

The Initial Theory

In the initial program theory, a total of 12 contexts were identified. Table 2 presents all potential CMO configurations. Informed by the initial literature screening and discussion within the research team, two contexts (1 and 6) were combined because of considerable overlap in the mechanisms and outcomes. In all, 6 contexts were chosen to be tested with empirical evidence in the next step. We had low confidence that there would be evidence available to test the remaining CMO configurations.

Table 2. The context-mechanism-outcome configurations identified in our initial program theory.
ContextMechanismsOutcomesa
1. Realistic (high-fidelity) simulations
  • Perceptions of realistic haptics and imagery
  • Triggers interactive learning
  • Lack of perceived realism in haptics or tactile sensation
  • Enhanced skills and proficiency
  • Learner satisfaction with realism
  • More effective learning
  • Preference for non-VRb learning, for example, laboratory dissection or physical reality
2. Artificial intelligence–enabled VRc
  • Provides feedback and highlights deficiencies
d
3. VR or ARe that immerses learners
  • Engages or exposes learners in deep immersion
  • Provides a safe environment free from patient harm
  • Cybersickness
  • Higher engagement and participation in training
  • Improved learning, knowledge, and comfort with knowledge
  • Improved skill performance
4. Comfortable devicesc
  • Cybersickness
  • Poor learning experience
5. VR or AR that delivers standardized teaching
  • Provides feedback to leaners
  • Enables repeated practice
  • Improves skill or performance
  • Leads to better patient outcomes in the future
6. Visualization through VR or AR
  • Interactive experience
  • Easier and more detailed visualization of patient anatomy
  • Perceived realism of the imagery
  • Learner satisfaction with tool and realism
  • Increased understanding or learning of content
  • Improved performance or skill
7. Accounts for physical and mental workloadc
  • Psychological improvements (reduced stress and improved self-confidence)
  • Decreased mental demand, effort, and physical workload scores
8. Team training delivered by AR or VRc
  • Interaction between learners and environment, as well as real-time collaboration and communication
  • Improves teamwork
  • Results in learner satisfaction
9. Knowledge or skill transfer
  • Enhances skills
  • Practice in safe environment (with no risk to patients)
  • Deliberate practice
  • Knowledge transfer to clinical practice
  • Skills transfer to cadaver, box trainer, and surgery and procedure
  • Better patient care in the future
10. Used with a teacherc
  • Improved instruction
11. Embedded in curriculumc
12. Limited training opportunities
  • Provides feedback on performance, skill or technique
  • Repeated practice
  • Access to experiential learning opportunities
  • Safe and stress-free learning environment
  • Skill improvement, technical proficiency, and reduced incidence of complications or errors
  • Learner satisfaction
  • Improvements for learners with less experience
13. Novices
  • Feedback and objective measurement of skills or knowledge
  • Independent or self-directed training
  • Safe, static, and risk-free environment without endangering patients
  • Repeated practice
  • Exposure to experience
  • Technical proficiency and skill acquisition
  • Improved performance (including operative performance)
  • Learner satisfaction: VR was preferred
  • Novices (less experienced people) improved most

aContext + Mechanisms = Outcomes.

bVR: virtual reality.

cThe context-mechanism-outcome configurations for which we had low confidence that there would be evidence available to test them.

dNot available.

eAR: augmented reality.

Summary of the AR and VR Training Interventions

The interventions presented in the empirical literature aimed at improving technical, behavioral, or nontechnical skills. The technical skills included laparoscopic procedural skills and camera navigation [8,9,59-61,78], evacuation procedures [20], dental drilling techniques [70], and vesicourethral anastomosis during robot-assisted radical prostatectomy [24]. Nontechnical skills were less commonly focused on but included decision-making, communication, teamwork [56,85], and patient counseling and communication [89]. In keeping with the Kirkpatrick et al [90] criteria for evaluation outcomes, 78% (36/46) of the studies explored behavior or skill improvement, 67% (31/46) explored reaction to the simulators (eg, satisfaction, attitudes, opinions on user experience, or intention to use the simulator), 20% (9/46) explored knowledge or learning outcomes, and 7% (3/46) explored patient results (eg, vaccine refusal rates, patient pain, and medical errors).

Of the 46 studies, 22 (48%) used nonimmersive VR simulators, of which computer-based programs and the LapSim, AnthroSim, and MIST-VR simulators were the most commonly used [6,8,9,14,41,56,58-61,64,65,70,73,75,78,80,81,85-88]; 12 (26%) used fully immersive VR, with the most common headsets used being the Oculus Rift and HTC Vive [16,17,20,23,57,62,69,71,74,76,77,79]; and 2 (4%) used the stereographic CrystalEyes shutter glasses, which enabled 3D visualization when connected to an immersive workbench, for partially immersive VR [68,84]. Of the 46 studies, 6 (13%) used AR, with the Microsoft HoloLens glasses being the most commonly used device [7,22,66,82,83,89]. Other devices included smartphone apps, the ODG R-7 Smartglasses, and the Brother AiRScouter WD-200B headset. Of the 46 studies, 3 (7%) combined AR with VR [63,67,72]. For example, Luciano et al [63] used the ImmersiveTouch VR system in addition to high-resolution AR stereoscopic glasses. Qin et al [67] included nonimmersive VR, fully immersive VR, and AR in a comprehensive multimodal simulation training program. The study by Wang et al [24] did not clearly state the level of immersion.

Of the 46 studies, 24 (52%) used haptic technology for force feedback or tactile sensation [6,8,9,14,41,58-64,67-70, 72,73,78-80,82,84,87]. Some used other tools such as manikins [22,69,82]. For example, Semeraro et al [69] connected the commercial Laerdal HeartSim 4000 manikin to a VR headset, tracking device, and sensor gloves. Robots were also used in some studies [24,62,85] to, for example, simulate operating with the da Vinci surgical robot. Finally, a training program also used human actors who were prompted by the simulator to provide patient feedback (eg, making groaning sounds to convey pain) during endoscopy training [86].

Quality of the Included Studies

There was substantial agreement for the MMAT appraisals between the 2 raters (NG and DD; 90%; κ=0.778, 95% CI 0.625-0.931; P<.001).

Overall, of the 46 studies, 13 (28%) were of high quality and 3 (7%) were of low quality, whereas the remaining 30 (65%) were of moderate quality. Of the 46 studies, 9 (20%) quantitative descriptive [8,56,58,61,63,64,66,68,70] and 4 (9%) quantitative nonrandomized studies did not include participants who were representative of the target population [24,80,82,83]; in addition, 9 (20%) quantitative descriptive studies did not clearly state their sampling strategy [41,56-58,60,62,66-68]; in 5 (11%) randomized controlled trials, randomization was not conducted properly [6,23,75-77]; in 6 (13%), blinding was either not possible or not conducted [6,7,17,20,23,76]; the qualitative approach was not reported for 2 (4%) mixed methods studies [84,85]; and the only qualitative study did not meet any of the criteria [89].

Final CMO Configurations

In all, 6 contexts were identified. We distinguished technology-related conditions (Table S3 in Multimedia Appendix 1) from training-related circumstances (Table S4 in Multimedia Appendix 1). Figure 2 provides a diagrammatic summary of the CMO configurations in which we had moderate or high confidence. These are discussed in detail next. The configurations in which we had very low or low confidence are presented in Tables S3 and S4 in Multimedia Appendix 1 but without further discussion in the text.

Figure 2. Diagram of our program theory on AR and VR training for health care workers built from the context-mechanism-outcome configurations in which we had moderate or high confidence. AR: augmented reality; VR: virtual reality.
View this figure

Realistic Simulations and Visualization (CMO 1)

The first condition relates to when VR (all levels of immersion, with and without haptics), AR, and a combination of VR and AR training programs portray realistic (high-fidelity) simulations or imagery (eg, on patient anatomy). This triggered perceptions of reality, enabled visualization of patient anatomy, and triggered an interactive experience [9,22,56,59,61,63-65,67-70, 72,79,80,82-87]. Easier visualization was explained through the use of 3D imagery, which often helped to reduce cognitive load and limit extraneous data [68,83]. The interactive experience was characterized by users interacting with the imagery in real time or when engaging in multiuser team training [56].

Across the mechanisms, 2 expected outcomes included more effective learning (increased understanding and learning of content as well as enhanced skills, proficiency, and performance) and increased learner satisfaction. There was strong supporting evidence for more effective learning when perceptions of realism and easier visualization were triggered. For example, in the study by Balian et al [22], half of the 51 participants delivered more than 80% of the cardiopulmonary resuscitation compressions with complete chest recoil and reduced leaning on the chest. This was attributed to perceptions of a realistic simulation, whereby realistic feedback included auditory (heartbeat metronome) and visual cues (increase or decrease in blood flow to vital organs). We had moderate to high confidence that easier visualization, interaction, and perceptions of realism lead to more effective learning.

Increased learner satisfaction was contested within the evidence. Some studies identified that their haptic tools hindered perceptions of realism [59,80,87]. Burdea et al [80] stated that the learners in their study were not satisfied with the VR simulator because it was not perceived as realistic. In addition, the lack of perceived realism might be why their VR group performed worse than the control group (using a rubber simulator) in diagnosing prostate cancer (33% vs 92%, respectively). It was expected that a more realistic VR simulator would have improved performance and learner satisfaction. However, most of the studies provided evidence that learners were satisfied with the tools in general [9,22,56,59,61,63,67-70,72,79,82,84-87]. We had the lowest confidence that an interactive experience resulted in learner satisfaction but moderate to high confidence that easier visualization and perceptions of realism result in satisfaction with the realism and tools, respectively.

Immersion (CMO 2)

The second condition relates to when fully immersive VR (with and without haptics) or AR with a manikin immersed learners in the training environment [16,20,22,23,57,62,71,74,76,77,79]. This triggered perceptions of deep immersion, whereby learners were transported into their training environments and a safe learning environment, free from patient harm. Bhowmick et al [23] explained that isolation from the outside world and use of realistic scenarios (eg, environments, characters, and tasks) promoted feelings of deep immersion. This resulted in improved learning, knowledge, and comfort with knowledge and skill performance.

Improved learning, knowledge, and comfort with knowledge and skill performance were observed by 22% (10/46) of the studies [16,22,23,57,62,71,74,76,77,79]. For example, residents in the study by Luca et al [62] made significantly fewer major errors after the training on average (1.8 compared with 5.2). Barré et al [79] also reported decreased mental demand (thinking, deciding, and calculating) for those in the VR intervention group compared with increases in the control group. In the study by Bracq et al [57], the more users felt immersed in the environment, the more they perceived it to be useful for their learning. Increases in confidence were also observed over the training periods [23,74,76]. Given the strong evidence, lack of contrasting evidence, and the high MMAT score (78%), we had high confidence in this CMO configuration.

Knowledge or Skill Transfer (CMO 4)

In the training-related context of knowledge and skill transfer, AR, combined AR and VR, and VR (all levels of immersion, with and without haptics) were used. When teaching transferable skills, three mechanisms may be triggered: enhancement of existing skills, practice in a perceived safe environment (away from patient harm, time restraints, and stress), and deliberate practice [14,17,20,24,41,59,71-74,77,78,81,88,89]. This leads to transfer of knowledge and skills to clinical practice and other simulators.

Empirical evidence was found for transferable skills, especially enhancing skills. Enhanced skills through VR or AR training helped to transfer knowledge and skills to clinical settings [71,88,89], other simulators (eg, sheep) [41], and surgical or invasive procedures [24,73,81]. For example, the percentage of medical and nurse trainees who experienced >1 occupational needlestick injury in the first 2 months of clinical internship was 31%-35% compared with the percentage of past senior trainees (80%) [71]. In addition, in a study by Wang et al [24], the average time required for real-life anastomosis procedures was shortened from 40.0 (SD 12.4) minutes to 25.1 (SD 7.1) minutes. However, the VR and AR simulators were not always superior and sometimes presented similar outcomes to traditional training [73,81]. In the live procedures, some medical errors (leakages) were still found [24], showing that despite improvements, performance was not perfect. We had moderate confidence that skills are transferable to clinical practice.

Training Novices (CMO 6)

The last training-related context relates to when VR (nonimmersive and fully immersive, with and without haptics) or AR were used to train novices (learners with little or no experience). The programs were expected to trigger various resources and mechanisms, including feedback and objective measurement of skills or knowledge; independent and self-directed learning; a safe, static, and risk-free learning environment; repeated practice; and exposure to experience [6,8,9,14,17,41,58-62,65,70-72,76,79,81]. This may result in technical proficiency, skill acquisition and improved performance (including operative performance), learner satisfaction, and the most improvement in novices.

Evidence showed that repeated attempts and practice on VR or AR simulators significantly improved skills such as speed of decision-making [71], catheter-insertion depth [72], efficiency of endoscopies [58], 30° laparoscopic camera manipulation skills [60], and syringe aspiration time for central venous catheterization [65]. Given the strong supporting evidence, lack of contested evidence, and the high MMAT score (77.5%), we had high confidence that repeated practice results in technical proficiency, skill acquisition, and improved performance. Evidence for the remaining CMO configurations was very limited and often contested; thus, our confidence in them was very low or low (Table S4 in Multimedia Appendix 1).

Implementation and Maintenance of VR and AR Training Programs

Information regarding barriers and facilitating factors for implementing and maintaining VR or AR training programs for health care professionals was extracted from the studies included in creating (step 1) and refining (step 2) the program theory.

Cost

Some argued that high up-front expenses created barriers to implementation and maintenance, including purchasing simulators and headsets as well as software licenses, technology maintenance, staff training, and programming requirements [26,76,91-95]. Integrating VR or AR with manikins was reported to significantly increase costs further [22]. Others argued that these costs were justified because VR can be used repeatedly at no additional cost per learner [16,17,34,40]. VR can provide a complete training tool (unlike box trainers) [60], does not wear out quickly (like manikins), and can represent any anatomy or body type, making it more cost-effective [65]. VR can also reduce time related to clinical teaching [6] and travel for trainees and educators [88].

The cost of VR and AR was expected to decrease with commercialization and market competition in this area [17,40,77], facilitating implementation as cheaper options become available [27]. A number-needed-to-train metric may also encourage hospital trusts and universities to implement VR programs [13]. This considers how many clinical costs each hour of training can reduce. Lohre et al [74] exemplified this metric, whereby 1 hour of training on their VR simulator was equivalent to 48 minutes of real-world training time. The simulator was therefore deemed at least 34.1 times more cost-effective.

Attitudes and Experience

A lack of acceptance (ie, negative attitudes) of VR and AR [56,91] and negative experiences may reduce uptake and behavioral intention [57,91,94,96]. Symptoms of cybersickness and perceptuomotor aftereffects when using VR included nausea, eye fatigue, dizziness, vomiting, and ataxia [57,91,94,96]. Other negative experiences could include addiction to VR gaming [91] and increased cognitive load and stress during initial use [57].

It was expected that a cultural change toward acceptance will occur when VR gains traction [56], which may help to increase VR as a standard teaching tool [97] and ultimately improve acceptance. Researchers have already observed positive attitudes toward these novel training tools [9,77,94,95]. For example, Ryu et al [95] reported that 81% of the 45 program directors and residents in their study expressed that VR would be a useful training tool.

Developmental and Logistical Considerations

Developmental and logistical considerations further create barriers because implementing and maintaining VR and AR programs requires imagination, resources, and planning [94]. From conception, the design and development of training resources can be a lengthy and complicated process, requiring specific programming and animation expertise [17,26,34,91]. Design needs to be multidisciplinary (to convey accurate content), attractive, and user centered [91]. Including external tools may further complicate development and implementation because haptic interaction systems and robotic arms may be cumbersome and limit use [61,84]. Logistical considerations also included storage space, maintenance, cleaning headsets between learners, and providing hazard-free and private learning spaces [27].

Access to Training

The studies highlighted access to training as a facilitator to uptake [18,27,34,40,58,62,77,98]. The mobility of AR and VR training can increase learning opportunities [18,34,62,77], which may fill educational gaps created by geographic or socioeconomic barriers [62]. Health professionals can also learn asynchronously, at their convenience [18,40], whereby self-guided training can be available to all shift workers [27,58]. These benefits also enable the potential scalability of VR and AR training [23,79,88].

Conversely, some studies reported that learners were not able to complete the training because of scheduling conflicts with patients and time constraints [27,60,95,99]. Stefanidis et al [100] clarified that initially, enthusiasm was high, but no one monitored training. Attendance only improved from 6% to 71% after a scheduling coordinator was hired.

Creating a Curriculum

The complexity involved in developing a standardized curriculum created barriers to implementation [8,34,37,57,91,93]. This required personnel to develop the program and schedule learners, validated training devices, and clearly defined objective criteria that aligned with existing curricula and could be used to evaluate learning outcomes [34,37,57,91,93]. Nationwide implementation was further challenged by locally established priorities, regional training budgets [93], and an unequal distribution of VR or AR resources between training centers and institutions [17,34,60].

According to the studies, leadership and collaboration are crucial to facilitate implementation [8,27,93,101]. At a local level, health professionals can develop credentialing committees [8], whereas at a higher level, national organizations and committees can help to ensure a standardized approach to training. With regard to localized training programs (eg, within hospitals), subspecialties could develop a shared training program [93,101]. Support from senior clinicians, boards of directors, and other organizational leaders is helpful to facilitate uptake [27,93,101].


Principal Findings and Comparison With Prior Work

To our knowledge, this is the first realist review to explore AR and VR training programs for health care professionals. It contributes a transferable program theory that may be applicable to diverse health professionals and across AR and VR technologies with varying levels of fidelity and use of haptics or additional tools.

A total of 80 published papers were used to develop an initial program theory, and 46 empirical studies that reported on VR, AR, or mixed simulation training programs for health professionals then helped to refine and test the theory. A total of 41 individual CMO configurations were identified, across 6 contexts and conditions. Of the 41 CMO configurations, we had moderate to high confidence in 9 (22%) and low and very low confidence in 5 (11%) and 27 (59%), respectively. Our low confidence was often due to contesting studies as well as the outcomes (especially those on patient results) not being substantiated with sufficient empirical evidence.

We also identified barriers and facilitators to implementation and maintenance, which must be acknowledged for the CMO configurations to be operationalized. The most common barriers were up-front costs, poor acceptance, negative experiences (ie, cybersickness), logistics, and the complexity involved in developing a curriculum. Decreasing costs due to commercialization and the cost-effectiveness of training, a cultural shift toward acceptance, access to training opportunities, and leadership and collaboration facilitated implementation.

The CMO configurations can be explained by applying learning theories identified within some of the reviewed literature [57,83,84,87,96]. Constructivism assumes that learning is an active process, building on previous skills, knowledge and interaction with the physical and social environment [102]. Through active construction [103] and learning by doing [104], trainees interact with the environment to adapt and learn. In the same way, VR and AR can be used by health professionals who already have some previous experiences and acquired knowledge or skills in their clinical fields. VR and AR programs may enable upskilling through active learning by immersing health professionals within simulated real-life environments. This is reflected in the mechanism of immersing learners in deep immersion. The mechanisms of repeated practice, enhancing skills, and interactive experiences are also explained by constructivism because learners can interact with VR or AR environments to practice their skills.

Cognitive load theory (CLT) can also help to explain the mechanisms, especially in the context of realistic simulations and visualization. CLT assumes that people have a finite amount of working memory available [105,106]. However, we have an unlimited long-term memory, which holds cognitive schemas (experiential knowledge). Learning is then the process of constructing and automating these schemas so that it can be stored in long-term memory. Cognitive load is categorized into intrinsic load (task-specific cognitive effort), extraneous load (irrelevant cognitive effort), and germane load (residual working memory capacity).

Some of the CLT literature suggests that VR and AR may help to reduce extraneous load (ie, processes not related to learning) by providing cues and feedback in real time [68,83]. For example, AR glasses and 3D and realistic imagery can provide real-time visual clues to learning to reduce the cognitive effort of remembering this information. However, it is also possible that VR or AR learning tools may unintentionally increase task-specific or extraneous cognitive load because they may complicate learning processes. This is because learners may need to adapt to using VR or AR tools if they are not familiar with them. In some of the reviewed studies [23,57,79], health professionals reported discomfort with the VR headset because of either fatigue or cybersickness, which may also increase extraneous cognitive load because they focus on this discomfort and consequently impair their learning ability. Pretraining to gain familiarity is therefore crucial [57,107].

It was evident that the literature on implementation is premature, with little focus on implementation experiences [17,68,85,100,101]. Some of the considerations were context dependent, highlighting that when implementing VR and AR training programs, the contexts and conditions must be acknowledged. For example, novices (eg, residents and postgraduate medical and health students) may have already been exposed to VR or AR learning tools and may be more accepting of them as well as tolerant of cybersickness. This is because VR and AR is being implemented in new training curricula [77,100] and discomfort decreases with familiarity and use [79,108]. This consideration might be more relevant for those less familiar with the technologies.

Future Research

There was a clear absence of AR and VR training programs for allied health staff, care workers, and within care- and community-based settings. There was also less focus on simple behavioral skills such as disposing of hazardous medical waste or practicing hand hygiene, for which AR and VR smartphone apps have already been developed [109]. In addition, many of the VR and AR devices were used along with haptics, robotic arms, actors, or manikins, which may introduce confounding factors when exploring effectiveness. As also identified by Kyaw et al [110], the applicability of VR or AR training within care and community settings and use as a stand-alone training tool warrants further investigation.

As is common in realist reviews [111] and evident in the literature, most of the mechanisms were not measured, except for repeated practice where authors accounted for repetitions. Control groups were rarely used, and qualitative data on experiences were limited. Future work should use robust and hypothesis-driven methods to objectively measure the impact of the mechanisms. For example, the 14-item Igroup Presence Questionnaire [112] can measure spatial presence (deep immersion), involvement (interaction), and experienced realism, whereas the 16-item Simulator Sickness Questionnaire [113] can measure cybersickness and discomfort. These validated questionnaires should be used in addition to a control group, whereas qualitative data (eg, through interviews) may help to further understand why and when the mechanisms are (or are not) triggered.

More work is also needed to increase the confidence in some of the CMO configurations for which we had low or very low confidence and to understand context-dependent implementation outcomes, along with updating the barriers and facilitators to implementation. Cost and acceptance, for example, may not be a barrier in the future, given that commercialization and market demand will reduce up-front costs, whereas increasing use may create a cultural change that favors acceptance.

Strengths and Limitations

Unlike some realist reviews [111,114], we first used nonempirical literature to form our theory and then tested and refined it with empirical literature. This was crucial to helping us to refine the program theory; in addition, it helped to ensure that the program theory was evidence informed and more reliable. Unlike others [50], we also assessed the quality of the research used to test and refine the theory and ultimately determined our confidence in each CMO configuration. The criteria used to determine confidence were conservative and also considered contesting studies and quantity of evidence. This transparency is important because program theories developed through realist reviews are only as good as the quality and quantity of the evidence they include. To our knowledge, this is the first realist review to consider all these factors.

Limitations included not sense checking our CMO configurations with AR or VR training experts as well as not comprehensively searching for gray literature. This meant that some initial theories might have been missed. In addition, only 20% (9/46) of the included studies were assessed for quality by 2 researchers. As such, interpretation of our quality assessments may be subject to some caution. However, we did not exclude research because of low quality and amalgamated the quality of the studies to determine our confidence in the CMO configurations; therefore, we do not expect this to bias our results. Interrater reliability was also substantial.

Conclusions

This review explored the complex nature of AR and VR training programs for health care staff, highlighting how they may actually work in practice, for whom they are most likely to work, and in which contexts and circumstances or under which conditions they may work. We found evidence for improved skills, learning and knowledge, and learner satisfaction, but there was little evidence on patient results. We had moderate to high confidence that VR and AR training programs trigger perceptions of realism and deep immersion as well as enable easier visualization of patient anatomy, interactivity, enhanced skills, and repeated practice in a safe environment. Future testing of these mechanisms using hypothesis-driven approaches is required. More research is also required to explore implementation and maintenance considerations. Ultimately, our evidence-informed program theory can be used to support the development and implementation of AR and VR training programs for health care providers and as a starting point for further research.

Acknowledgments

This work is funded by the National Institute for Health Research Applied Research Collaboration Greater Manchester. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health Research or the Department of Health and Social Care.

Authors' Contributions

NG conceived and designed the review with support from DD, SNvdV, and PW. NG and DD identified and analyzed the literature and conducted the quality assessments. All authors contributed to developing the program theory. NG wrote the first draft of the manuscript. All authors revised and approved the final manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Data extraction items, summary of the empirical articles (n=46), context-mechanism-outcome configurations, and our confidence in the configurations.

DOCX File , 45 KB

  1. Ross JR. Upskilling could be an ideal treatment for healthcare's employment woes. Cardiovascular Business. 2019.   URL: https:/​/www.​cardiovascularbusiness.com/​topics/​healthcare-economics/​upskilling-could-be-ideal-rx-healthcares-employment-woes [accessed 2021-06-26]
  2. Gasteiger N, van der Veer SN, Wilson P, Dowding D. Upskilling health and care workers with augmented and virtual reality: protocol for a realist review to develop an evidence-informed programme theory. BMJ Open 2021 Jul 05;11(7):e050033 [FREE Full text] [CrossRef] [Medline]
  3. Wright P, Bretthauer K. Strategies for addressing the nursing shortage: coordinated decision making and workforce flexibility. Dec Sci 2010;41(2):373-401. [CrossRef]
  4. Fralic MF. Upskilling for new-era healthcare: lessons from high-impact readings. Nurse Leader 2013 Aug;11(4):29-31. [CrossRef]
  5. Almomani E, Sullivan J, Hajjieh M, Leighton K. Simulation-based education programme for upskilling non-critical care nurses for COVID-19 deployment. BMJ Simul Technol Enhanc Learn 2020 Oct 21. [CrossRef]
  6. Tsai S, Chai S, Hsieh L, Lin S, Taur F, Sung W, et al. The use of virtual reality computer simulation in learning Port-A cath injection. Adv Health Sci Educ Theory Pract 2008 Mar 30;13(1):71-87. [CrossRef] [Medline]
  7. Huang CY, Thomas JB, Alismail A, Cohen A, Almutairi W, Daher NS, et al. The use of augmented reality glasses in central line simulation: "see one, simulate many, do one competently, and teach everyone". Adv Med Educ Pract 2018 May;Volume 9:357-363. [CrossRef]
  8. Aggarwal R, Grantcharov TP, Eriksen JR, Blirup D, Kristiansen VB, Funch-Jensen P, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006 Aug;244(2):310-314. [CrossRef] [Medline]
  9. Graafland M, Bok K, Schreuder HW, Schijven MP. A multicenter prospective cohort study on camera navigation training for key user groups in minimally invasive surgery. Surg Innov 2014 Jun 16;21(3):312-319. [CrossRef] [Medline]
  10. Aggarwal R, Mytton OT, Derbrew M, Hananel D, Heydenburg M, Issenberg B, et al. Training and simulation for patient safety. Qual Saf Health Care 2010 Aug 06;19 Suppl 2(Suppl 2):34-43 [FREE Full text] [CrossRef] [Medline]
  11. Sarre S, Maben J, Aldus C, Schneider J, Wharrad H, Nicholson C, et al. The challenges of training, support and assessment of healthcare support workers: a qualitative study of experiences in three English acute hospitals. Int J Nurs Stud 2018 Mar;79:145-153 [FREE Full text] [CrossRef] [Medline]
  12. Malik G, McKenna L, Plummer V. Perceived knowledge, skills, attitude and contextual factors affecting evidence-based practice among nurse educators, clinical coaches and nurse specialists. Int J Nurs Pract 2015 May 30;21 Suppl 2:46-57. [CrossRef] [Medline]
  13. Lewis T, Aggarwal R, Rajaretnam N, Grantcharov T, Darzi A. Training in surgical oncology - the role of VR simulation. Surg Oncol 2011 Sep;20(3):134-139. [CrossRef] [Medline]
  14. Ode G, Loeffler B, Chadderdon RC, Haines N, Scannell B, Patt J, et al. Wrist arthroscopy: can we gain proficiency through knee arthroscopy simulation? J Surg Educ 2018 Nov;75(6):1664-1672. [CrossRef] [Medline]
  15. Hart R, Karthigasu K. The benefits of virtual reality simulator training for laparoscopic surgery. Curr Opin Obstet Gynecol 2007 Aug;19(4):297-302. [CrossRef] [Medline]
  16. Price MF, Tortosa DE, Fernandez-Pacheco AN, Alonso NP, Madrigal JJ, Melendreras-Ruiz R, et al. Comparative study of a simulated incident with multiple victims and immersive virtual reality. Nurse Educ Today 2018 Dec;71:48-53. [CrossRef] [Medline]
  17. Zackoff MW, Lin L, Israel K, Ely K, Raab D, Saupe J, et al. The future of onboarding: implementation of immersive virtual reality for nursing clinical assessment training. J Nurses Prof Dev 2020 Mar 31;36(4):235-240. [CrossRef] [Medline]
  18. Rogers MP, DeSantis AJ, Janjua H, Barry TM, Kuo PC. The future surgical training paradigm: virtual reality and machine learning in surgical education. Surgery 2021 May;169(5):1250-1252. [CrossRef] [Medline]
  19. Xuan W, Cui L, Lv X. Augmented reality animation for pancreatic surgery. In: Transactions on Edutainment XVI. Berlin, Heidelberg: Springer; Apr 12, 2020:146-153.
  20. Farra S, Hodgson E, Miller ET, Timm N, Brady W, Gneuhs M, et al. Effects of virtual reality simulation on worker emergency evacuation of neonates. Disaster Med Public Health Prep 2019 Apr 8;13(2):301-308 [FREE Full text] [CrossRef] [Medline]
  21. Farra SL, Miller ET. Integrative review: virtual disaster training. J Nurs Edu Pract 2012 Nov 04;3(3). [CrossRef]
  22. Balian S, McGovern SK, Abella BS, Blewer AL, Leary M. Feasibility of an augmented reality cardiopulmonary resuscitation training system for health care providers. Heliyon 2019 Aug;5(8):e02205 [FREE Full text] [CrossRef] [Medline]
  23. Bhowmick S, Darbar R, Sorathia K. Pragati: design and evaluation of a mobile phone based head mounted virtual reality interface to train community health workers in rural India. In: Proceedings of the 10th Nordic Conference on Human-Computer Interaction. 2018 Presented at: NordiCHI'18: Nordic Conference on Human-Computer Interaction; September 29 - October 3, 2018; Oslo Norway p. 299-310. [CrossRef]
  24. Wang F, Zhang C, Guo F, Sheng X, Ji J, Xu Y, et al. The application of virtual reality training for anastomosis during robot-assisted radical prostatectomy. Asian J Urol 2021 Apr;8(2):204-208 [FREE Full text] [CrossRef] [Medline]
  25. Bracq M, Michinov E, Jannin P. Virtual reality simulation in nontechnical skills training for healthcare professionals. Sim Healthcare 2019 Jan 2;14(3):188-194. [CrossRef]
  26. Chheang V, Fischer V, Buggenhagen H, Huber T, Huettl F, Kneist W, et al. Toward interprofessional team training for surgeons and anesthesiologists using virtual reality. Int J Comput Assist Radiol Surg 2020 Dec;15(12):2109-2118 [FREE Full text] [CrossRef] [Medline]
  27. Michael S, Villarreal P, Ferguson M, Wiler J, Zane R, Flarity K. Virtual reality–based resilience programs: feasibility and implementation for inpatient oncology nurses. Clin J Oncol Nurs 2019 Dec 1;23(6):664-667. [CrossRef]
  28. Hu H, Feng X, Shao Z, Xie M, Xu S, Wu X, et al. Application and prospect of mixed reality technology in medical field. Curr Med Sci 2019 Feb 13;39(1):1-6. [CrossRef] [Medline]
  29. Pantelidis P, Chorti A, Papagiouvanni I, Paparoidamis G, Drosos C, Panagiotakopoulos T, et al. In: Tsoulfas G, editor. Virtual and Augmented Reality in Medical Education, Medical and Surgical Education - Past, Present and Future. London, UK: IntechOpen; 2017.
  30. Mertz L. Virtual reality pioneer tom furness on the past, present, and future of VR in health care. IEEE Pulse 2019 May;10(3):9-11. [CrossRef]
  31. Meakim C, Boese T, Decker S, Franklin AE, Gloe D, Lioce L, et al. Standards of best practice: Simulation Standard I: terminology. Clin Simul Nurs 2013 Jun;9(6):3-11. [CrossRef]
  32. Dieckmann P, Gaba D, Rall M. Deepening the theoretical foundations of patient simulation as social practice. J Society Simul Healthc 2007;2(3):183-193. [CrossRef]
  33. SIRC glossary. National League for Nursing Simulation Innovation Resource Center. 2013.   URL: http://sirc.nln.org/mod/glossary/view.php?id14183 [accessed 2021-06-23]
  34. Olasky J, Sankaranarayanan G, Seymour NE, Magee JH, Enquobahrie A, Lin MC, et al. Identifying opportunities for virtual reality simulation in surgical education: a review of the proceedings from the innovation, design, and emerging alliances in surgery (IDEAS) conference: VR surgery. Surg Innov 2015 Oct;22(5):514-521 [FREE Full text] [CrossRef] [Medline]
  35. Fiani B, De Stefano F, Kondilis A, Covarrubias C, Reier L, Sarhadi K. Virtual reality in neurosurgery: "Can You See It?"-a review of the current applications and future potential. World Neurosurg 2020 Sep;141:291-298. [CrossRef] [Medline]
  36. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou M, Davidson BR. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev 2013 Aug 27(8):CD006575. [CrossRef] [Medline]
  37. Fried M, Uribe J, Sadoughi B. The role of virtual reality in surgical training in otorhinolaryngology. Curr Opin Otolaryngol Head Neck Surg 2007 Jun;15(3):163-169. [CrossRef] [Medline]
  38. Childs BS, Manganiello MD, Korets R. Novel education and simulation tools in urologic training. Curr Urol Rep 2019 Nov 28;20(12):81. [CrossRef] [Medline]
  39. Hirt J, Beer T. Use and impact of virtual reality simulation in dementia care education: a scoping review. Nurse Educ Today 2020 Jan;84:104207. [CrossRef] [Medline]
  40. Farra S, Gneuhs M, Hodgson E, Kawosa B, Miller ET, Simon A, et al. Comparative cost of virtual reality training and live exercises for training hospital workers for evacuation. Comput Inform Nurs 2019 Sep;37(9):446-454 [FREE Full text] [CrossRef] [Medline]
  41. Hart R, Doherty DA, Karthigasu K, Garry R. The value of virtual reality-simulator training in the development of laparoscopic surgical skills. J Minim Invasive Gynecol 2006 Apr;13(2):126-133. [CrossRef] [Medline]
  42. Alamrani MH, Alammar KA, Alqahtani SS, Salem OA. Comparing the effects of simulation-based and traditional teaching methods on the critical thinking abilities and self-confidence of nursing students. J Nurs Res 2018 Jun;26(3):152-157. [CrossRef] [Medline]
  43. Blanié A, Amorim M, Benhamou D. Comparative value of a simulation by gaming and a traditional teaching method to improve clinical reasoning skills necessary to detect patient deterioration: a randomized study in nursing students. BMC Med Educ 2020 Feb 19;20(1):53 [FREE Full text] [CrossRef] [Medline]
  44. William A, Vidal V, John P. Traditional instruction versus virtual reality simulation: a comparative study of phlebotomy training among nursing students in Kuwait. J Edu Pract 2016;7(9):18-25 [FREE Full text]
  45. Scerbo MW, Bliss JP, Schmidt EA, Thompson SN. The efficacy of a medical virtual reality simulator for training phlebotomy. Hum Factors 2006 Sep 06;48(1):72-84. [CrossRef] [Medline]
  46. Kaboub F. Realistic evaluation. Soc Sci J 2019 Dec 09;41(1):153-154. [CrossRef]
  47. Jagosh J, Macaulay A, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q 2012 Jun;90(2):311-346 [FREE Full text] [CrossRef] [Medline]
  48. Wong G, Westhorp G, Pawson R, Greenhalgh T. Realist synthesis. RAMESES training materials. London: The RAMESES Project 2013 [FREE Full text]
  49. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med 2013 Jan 29;11(1):21 [FREE Full text] [CrossRef] [Medline]
  50. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010 Feb 02;10:12 [FREE Full text] [CrossRef] [Medline]
  51. Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Edu Inform 2018 Dec 18;34(4):285-291. [CrossRef]
  52. Pluye P, Gagnon M, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. Int J Nurs Stud 2009 Apr;46(4):529-546. [CrossRef] [Medline]
  53. Mogharbel A, Dowding D, Ainsworth J. Physicians' use of the computerized physician order entry system for medication prescribing: systematic review. JMIR Med Inform 2021 Mar 04;9(3):e22923 [FREE Full text] [CrossRef] [Medline]
  54. Vusio F, Thompson A, Birchwood M, Clarke L. Experiences and satisfaction of children, young people and their parents with alternative mental health models to inpatient settings: a systematic review. Eur Child Adolesc Psychiatry 2020 Dec;29(12):1621-1633 [FREE Full text] [CrossRef] [Medline]
  55. Moher D, Liberati A, Tetzlaff J, Altman D, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
  56. Abelson JS, Silverman E, Banfelder J, Naides A, Costa R, Dakin G. Virtual operating room for team training in surgery. Am J Surg 2015 Sep;210(3):585-590. [CrossRef] [Medline]
  57. Bracq M, Michinov E, Arnaldi B, Caillaud B, Gibaud B, Gouranton V, et al. Learning procedural skills with a virtual reality simulator: an acceptability study. Nurse Educ Today 2019 Aug;79:153-160. [CrossRef] [Medline]
  58. Clark JA, Volchok JA, Hazey JW, Sadighi PJ, Fanelli RD. Initial experience using an endoscopic simulator to train surgical residents in flexible endoscopy in a community medical center residency program. Curr Surg 2005 Jan;62(1):59-63. [CrossRef] [Medline]
  59. Elessawy M, Mabrouk M, Heilmann T, Weigel M, Zidan M, Abu-Sheasha G, et al. Evaluation of laparoscopy virtual reality training on the improvement of trainees' surgical skills. Medicina (Kaunas) 2021 Feb 02;57(2):130 [FREE Full text] [CrossRef] [Medline]
  60. Khoo H, Chik I, Azman A, Zuhdi Z, Harunarashid H, Jarmin R. Virtual reality laparoscopic simulator: training tool for surgical trainee in Malaysia. Formos J Surg 2021;54(1):11. [CrossRef]
  61. Kurenov S, Cendan J, Dindar S, Attwood K, Hassett J, Nawotniak R, et al. Surgeon-authored virtual laparoscopic adrenalectomy module is judged effective and preferred over traditional teaching tools. Surg Innov 2017 Feb 07;24(1):72-81 [FREE Full text] [CrossRef] [Medline]
  62. Luca A, Giorgino R, Gesualdo L, Peretti GM, Belkhou A, Banfi G, et al. Innovative educational pathways in spine surgery: advanced virtual reality-based training. World Neurosurg 2020 Aug;140:674-680. [CrossRef] [Medline]
  63. Luciano C, Banerjee P, Bellotte B. Learning retention of thoracic pedicle screw placement using a high-resolution augmented reality simulator with haptic feedback. Neurosurgery 2011;69(1 Suppl Operative):s14-s19. [CrossRef]
  64. Luciano CJ, Banerjee PP, Sorenson JM, Foley KT, Ansari SA, Rizzi S, et al. Percutaneous spinal fixation simulation with virtual reality and haptics. Neurosurgery 2013;72:89-96. [CrossRef]
  65. Pepley DF, Gordon AB, Yovanoff MA, Mirkin KA, Miller SR, Han DC, et al. Training surgical residents with a haptic robotic central venous catheterization simulator. J Surg Educ 2017 Nov;74(6):1066-1073 [FREE Full text] [CrossRef] [Medline]
  66. Ponce BA, Jennings JK, Clay TB, May MB, Huisingh C, Sheppard ED. Telementoring: use of augmented reality in orthopaedic education: AAOS exhibit selection. J Bone Joint Surg Am 2014 May 21;96(10):e84. [CrossRef] [Medline]
  67. Qin Z, Tai Y, Xia C, Peng J, Huang X, Chen Z, et al. Towards virtual VATS, face, and construct evaluation for peg transfer training of Box, VR, AR, and MR Trainer. J Healthc Eng 2019 Jan 06;2019:6813719 [FREE Full text] [CrossRef] [Medline]
  68. Sakellariou S, Ward B, Charissis V, Chanock D, Anderson P. Design and implementation of augmented reality environment for complex anatomy training: inguinal canal case study. In: VMR 2009: Virtual and Mixed Reality. Berlin, Heidelberg: Springer; 2009:605-614.
  69. Semeraro F, Frisoli A, Bergamasco M, Cerchiari EL. Virtual reality enhanced mannequin (VREM) that is well received by resuscitation experts. Resuscitation 2009 Apr;80(4):489-492. [CrossRef] [Medline]
  70. Wang D, Zhao S, Li T, Zhang Y, Wang X. Preliminary evaluation of a virtual reality dental simulation system on drilling operation. Biomed Mater Eng 2015 Aug 17;26(s1):747-756. [CrossRef]
  71. Wu S, Huang C, Huang S, Yang Y, Liu C, Shulruf B, et al. Effect of virtual reality training to decreases rates of needle stick/sharp injuries in new-coming medical and nursing interns in Taiwan. J Educ Eval Health Prof 2020 Jan 20;17:1 [FREE Full text] [CrossRef] [Medline]
  72. Yudkowsky R, Luciano C, Banerjee P, Schwartz A, Alaraj A, Lemole GM, et al. Practice on an augmented reality/haptic simulator and library of virtual brains improves residents' ability to perform a ventriculostomy. Simul Healthc 2013 Feb;8(1):25-31. [CrossRef] [Medline]
  73. Cannon WD, Garrett WE, Hunter RE, Sweeney HJ, Eckhoff DG, Nicandri GT, et al. Improving residency training in arthroscopic knee surgery with use of a virtual-reality simulator. J Bone Joint Surg Am 2014;96(21):1798-1806. [CrossRef]
  74. Lohre R, Bois AJ, Pollock JW, Lapner P, McIlquham K, Athwal GS, et al. Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: a randomized clinical trial. JAMA Netw Open 2020 Dec 01;3(12):e2031217 [FREE Full text] [CrossRef] [Medline]
  75. Maytin M, Daily TP, Carillo RG. Virtual reality lead extraction as a method for training new physicians: a pilot study. Pacing Clin Electrophysiol 2015 Mar 12;38(3):319-325. [CrossRef] [Medline]
  76. Pulijala Y, Ma M, Pears M, Peebles D, Ayoub A. Effectiveness of immersive virtual reality in surgical training-a randomized control trial. J Oral Maxillofac Surg 2018 May;76(5):1065-1072. [CrossRef] [Medline]
  77. Real FJ, DeBlasio D, Beck AF, Ollberding NJ, Davis D, Cruse B, et al. A Virtual Reality Curriculum for Pediatric Residents Decreases Rates of Influenza Vaccine Refusal. Acad Pediatr 2017;17(4):431-435. [CrossRef] [Medline]
  78. Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002 Oct;236(4):458-463. [CrossRef] [Medline]
  79. Barré J, Michelet D, Truchot J, Jolivet E, Recanzone T, Stiti S, et al. Virtual reality single-port sleeve gastrectomy training decreases physical and mental workload in novice surgeons: an exploratory study. Obes Surg 2019 Apr 28;29(4):1309-1316. [CrossRef] [Medline]
  80. Burdea G, Patounakis G, Popescu V, Weiss R. Virtual reality-based training for the diagnosis of prostate cancer. IEEE Trans Biomed Eng 1999 Oct;46(10):1253-1260. [CrossRef] [Medline]
  81. McIntosh KS, Gregor JC, Khanna NV. Computer-based virtual reality colonoscopy simulation improves patient-based colonoscopy performance. Can J Gastroenterol Hepatol 2014 Apr;28(4):203-206 [FREE Full text] [CrossRef] [Medline]
  82. Qian M, Nicholson J, Tanaka D, Dias P, Wang E. Augmented reality (AR) assisted laryngoscopy for endotracheal intubation training. In: Chen J, Fragomeni G, editors. Virtual, Augmented and Mixed Reality. Applications and Case Studies. Cham: Springer; 2019:355-371.
  83. Triberti S, Petrella F, Gorini A, Pappalardo O, Sebri V, Savioni L, et al. Augmenting surgery: medical students’ assessment and ergonomics of 3d holograms vs. CT scans for pre-operative planning. EAI Endors Trans Pervas Health Technol 2021 Jan 08;7(25):167844. [CrossRef]
  84. Charissis V, Ward B, Naef M, Rowley D, Brady L, Anderson P. An enquiry into VR interface design for medical training: VR augmented anatomy tutorials for breast cancer. In: Proceedings Volume of the Engineering Reality of Virtual Reality 2008. 2008 Presented at: The Engineering Reality of Virtual Reality 2008; 2008; San Jose, California, United States. [CrossRef]
  85. Khan MS, Ahmed K, Gavazzi A, Gohil R, Thomas L, Poulsen J, et al. Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity. BJU Int 2012 Aug 29;111(3):518-523. [CrossRef]
  86. Kneebone R, Nestel D, Moorthy K, Taylor P, Bann S, Munz Y, et al. Learning the skills of flexible sigmoidoscopy - the wider perspective. Med Educ 2003 Nov;37 Suppl 1(1):50-58. [CrossRef] [Medline]
  87. Koch A, Pfandler M, Stefan P, Wucherer P, Lazarovici M, Navab N, et al. Say, what is on your mind? Surgeons' evaluations of realism and usability of a virtual reality vertebroplasty simulator. Surg Innov 2019 Apr 15;26(2):234-243. [CrossRef] [Medline]
  88. Pickering CE, Ridenour K, Salaysay Z, Reyes-Gastelum D, Pierce SJ. EATI Island - A virtual-reality-based elder abuse and neglect educational intervention. Gerontol Geriatr Educ 2018 Nov 15;39(4):445-463. [CrossRef] [Medline]
  89. Dixit R, Sinha V. Leveraging augmented reality for training transfer: a case of healthcare service providers in ophthalmology. Dev Learn Organ 2019 Dec 09;34(6):33-36. [CrossRef]
  90. Kirkpatrick DL. The Four Levels of Evaluation: Measurement and Evaluation. Alexandria: American Society for Training and Development; 2007.
  91. Baniasadi T, Ayyoubzadeh SM, Mohammadzadeh N. Challenges and practical considerations in applying virtual reality in medical education and treatment. Oman Med J 2020 May 19;35(3):e125 [FREE Full text] [CrossRef] [Medline]
  92. Bashir G. Technology and medicine: the evolution of virtual reality simulation in laparoscopic training. Med Teach 2010 Jul 23;32(7):558-561. [CrossRef] [Medline]
  93. Burden C, Oestergaard J, Larsen C. Integration of laparoscopic virtual-reality simulation into gynaecology training. Br J Obstet Gynaecol 2011 Nov;118 Suppl 3:5-10. [CrossRef] [Medline]
  94. Persky S, McBride CM. Immersive virtual environment technology: a promising tool for future social and behavioral genomics research and practice. Health Commun 2009 Dec 10;24(8):677-682 [FREE Full text] [CrossRef] [Medline]
  95. Ryu WH, Chan S, Sutherland GR. Supplementary educational models in Canadian neurosurgery residency programs. Can J Neurol Sci 2017 Mar 07;44(2):177-183. [CrossRef] [Medline]
  96. Uruthiralingam U, Rea P. Augmented and virtual reality in anatomical education - a systematic review. Adv Exp Med Biol 2020;1235:89-101. [CrossRef] [Medline]
  97. Bartella A, Kamal M, Scholl I, Schiffer S, Steegmann J, Ketelsen D, et al. Virtual reality in preoperative imaging in maxillofacial surgery: implementation of "the next level"? Br J Oral Maxillofac Surg 2019 Sep;57(7):644-648. [CrossRef] [Medline]
  98. McElhinney B, Beard A, Karthigasu K, Hart R. Virtual reality simulation: a valuable adjunct to surgical training. In: Virtual Reality in Psychological, Medical and Pedagogical Applications. London, UK: IntechOpen; 2012.
  99. Fleming M, Olsen D, Stathes H, Boteler L, Grossberg P, Pfeifer J, et al. Virtual reality skills training for health care professionals in alcohol screening and brief intervention. J Am Board Fam Med 2009 Jul;22(4):387-398 [FREE Full text] [CrossRef] [Medline]
  100. Stefanidis D, Acker CE, Swiderski D, Heniford BT, Greene FL. Challenges during the implementation of a laparoscopic skills curriculum in a busy general surgery residency program. J Surg Educ 2008 Jan;65(1):4-7. [CrossRef] [Medline]
  101. Servotte J, Guillaume M, Boga D, Coucke P. Methodological approach for the implementation of a simulator in augmented reality in a radiation therapy department. Int J Healthc Manag 2016 Dec 09;10(3):154-159. [CrossRef]
  102. Phillips DC. The good, the bad, and the ugly: the many faces of constructivism. Edu Res 2016 Jul 01;24(7):5-12. [CrossRef]
  103. Mascolo M, Fischer K. Constructivist theories. 2005.   URL: https://www.academia.edu/8906476/Constructivist_Theories [accessed 2021-06-22]
  104. Dewey J. Experience and Education. New York: Collier Books; 1938.
  105. Sweller J. Cognitive load during problem solving: Effects on learning. Cogn Sci 1988;12:85. [CrossRef]
  106. Sweller J. Element interactivity and intrinsic, extraneous, and germane cognitive load. Educ Psychol Rev 2010 Apr 23;22(2):123-138. [CrossRef]
  107. Lackey SJ, Salcedo JN, Szalma J, Hancock P. The stress and workload of virtual reality training: the effects of presence, immersion and flow. Ergonomics 2016 Aug 15;59(8):1060-1072. [CrossRef] [Medline]
  108. Chang E, Kim HT, Yoo B. Virtual reality sickness: a review of causes and measurements. Int J Hum Comput Interact 2020 Jul 02;36(17):1658-1682. [CrossRef]
  109. Gasteiger N, Dowding D, Ali S, Scott AJ, Wilson P, van der Veer SN. Sticky apps, not sticky hands: A systematic review and content synthesis of hand hygiene mobile apps. J Am Med Inform Assoc 2021 Aug 13;28(9):2027-2038. [CrossRef] [Medline]
  110. Kyaw BM, Saxena N, Posadzki P, Vseteckova J, Nikolaou CK, George PP, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019 Jan 22;21(1):e12959 [FREE Full text] [CrossRef] [Medline]
  111. Shams F, Wong JS, Nikoo M, Outadi A, Moazen-Zadeh E, Kamel MM, et al. Understanding eHealth cognitive behavioral therapy targeting substance use: realist review. J Med Internet Res 2021 Jan 21;23(1):e20557 [FREE Full text] [CrossRef] [Medline]
  112. Schubert T, Friedmann F, Regenbrecht H. The experience of presence: factor analytic insights. Presen Teleoperat Virt Environ 2001 Jun;10(3):266-281. [CrossRef]
  113. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol 1993 Jul;3(3):203-220. [CrossRef]
  114. Wiese A, Kilty C, Bergin C, Flood P, Fu N, Horgan M, et al. Protocol for a realist review of workplace learning in postgraduate medical education and training. Syst Rev 2017 Jan 19;6(1):10 [FREE Full text] [CrossRef] [Medline]


AR: augmented reality
CLT: cognitive load theory
CMO: context-mechanism-outcome
MMAT: Mixed Methods Appraisal Tool
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
RAMESES: Realist and Meta-narrative Evidence Syntheses: Evolving Standards
VR: virtual reality


Edited by N Zary; submitted 29.06.21; peer-reviewed by TT Meum, N Moore, S Jung; comments to author 10.09.21; revised version received 16.09.21; accepted 12.10.21; published 14.02.22

Copyright

©Norina Gasteiger, Sabine N van der Veer, Paul Wilson, Dawn Dowding. Originally published in JMIR Serious Games (https://games.jmir.org), 14.02.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.