Published on in Vol 9, No 4 (2021): Oct-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/29586, first published .
Effectiveness and Utility of Virtual Reality Simulation as an Educational Tool for Safe Performance of COVID-19 Diagnostics: Prospective, Randomized Pilot Trial

Effectiveness and Utility of Virtual Reality Simulation as an Educational Tool for Safe Performance of COVID-19 Diagnostics: Prospective, Randomized Pilot Trial

Effectiveness and Utility of Virtual Reality Simulation as an Educational Tool for Safe Performance of COVID-19 Diagnostics: Prospective, Randomized Pilot Trial

Original Paper

1Department of Emergency Medicine, Inselspital, University Hospital Bern, Bern, Switzerland

2Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Oslo, Norway

3ORamaVR SA, Geneva, Switzerland

4Institute of Computer Science, Foundation for Research and Technology, Hellas, Heraklion, Greece

5Department of Computer Science, University of Crete, Heraklion, Greece

Corresponding Author:

Tanja Birrenbach, MME, MD

Department of Emergency Medicine, Inselspital

University Hospital Bern

Freiburgstrasse 16C

Bern, 3010

Switzerland

Phone: 41 31 6322111

Email: tanja.birrenbach@insel.ch


Background: Although the proper use of hygiene and personal protective equipment (PPE) is paramount for preventing the spread of diseases such as COVID-19, health care personnel have been shown to use incorrect techniques for donning/doffing of PPE and hand hygiene, leading to a large number of infections among health professionals. Education and training are difficult owing to the social distancing restrictions in place, shortages of PPE and testing material, and lack of evidence on optimal training. Virtual reality (VR) simulation can offer a multisensory, 3-D, fully immersive, and safe training opportunity that addresses these obstacles.

Objective: The aim of this study is to explore the short- and long-term effectiveness of a fully immersive VR simulation versus a traditional learning method regarding a COVID-19–related skill set and media-specific variables influencing training outcomes.

Methods: This was a prospective, randomized controlled pilot study on medical students (N=29; intervention VR training, n=15, vs control video-based instruction, n=14) to compare the performance of hand disinfection, nasopharyngeal swab taking, and donning/doffing of PPE before and after training and 1 month later as well as variables of media use.

Results: Both groups performed significantly better after training, with the effect sustained over one month. After training, the VR group performed significantly better in taking a nasopharyngeal swab, scoring a median of 14 out of 17 points (IQR 13-15) versus 12 out of 17 points (IQR 11-14) in the control group, P=.03. With good immersion and tolerability of the VR simulation, satisfaction was significantly higher in the VR group compared to the control group (median score of User Satisfaction Evaluation Questionnaire 27/30, IQR 23-28, vs 22/30, IQR 20-24, in the control group; P=.01).

Conclusions: VR simulation was at least as effective as traditional learning methods in training medical students while providing benefits regarding user satisfaction. These results add to the growing body of evidence that VR is a useful tool for acquiring simple and complex clinical skills.

JMIR Serious Games 2021;9(4):e29586

doi:10.2196/29586

Keywords



The COVID-19 pandemic is a global health emergency that places massive demands on health systems and health care workers [1]. Proper use of hygiene and personal protective equipment (PPE) is paramount to prevent spreading of disease and contamination of health care workers. One possible reason for the high infection rate of COVID-19 is ineffective use of PPE. In Italy, up to 20% of health care workers were initially infected with the disease [2].

PPE recommendations from international organizations are largely consistent (eg, those from the US Centers for Disease Control and Prevention [CDC] and World Health Organization [WHO]) [3-6]; however, the actual use of PPE is not. Health care personnel in all professions and at all levels of training have been shown to use incorrect techniques for donning and doffing of PPE and hand hygiene [7-11]. The main reason appears to be inadequate training in correct PPE technique and lack of assessment of proficiency [7,11,12].

Simulation proves to be a powerful tool to test the accurate use of hygiene skills that are relevant for the treatment of patients with COVID-19 (ie, PPE and hand hygiene) [13]; nevertheless, there is still ambiguity regarding which training method works best. A recent Cochrane review of evidence relating to PPE and protection of health care staff exposed to contaminated body fluids highlights the lack of robust evidence in this area [11].

Furthermore, education and training of health care personnel is difficult with social distancing restrictions in place and shortages of both PPE [2] and testing materials.

Virtual reality (VR) uses computer systems to generate realistic pseudoenvironments that provide users with visual, tactile, and auditory sensations, with the possibility of realistic interaction with the virtual environment [14]. Milgram and Kishino [15] referred to mixed reality (MR) as the technologies which involve the merging of real and virtual worlds.

VR simulation with the use of head-mounted devices (HMDs) can offer a multisensory, 3-D, fully immersive, and safe training opportunity, avoiding the restrictions of social distancing and material shortages [16,17]. Through the concept of immersion, sense of presence, and interaction with the virtual environment in a real-time and realistic manner, VR simulation can create emotional experiences that facilitate experiential learning, exceeding other 2-D learning modalities [18].

The value of VR in medical education has already been demonstrated for various tasks [16,19-30]. VR is often used for training skills of varying complexity, ranging from simple nursing skills (Foley catheter placement, gaining venous access [31]) to laparoscopic/endoscopic/endovascular skills [23,32] or complex surgical procedures [26-29,33,34]. It is suggested that skills training as a first step to acquiring a competency can be better taught with VR than with traditional learning methods, because VR allows for a more active and immersive learning experience [21].

Recent studies suggest that VR improves postintervention knowledge and skills of health professionals better than traditional education or other types of digital education [21,31,35]. VR offers several advantages, as it provides possibilities for flexible learning and self-learning, providing standardization, reproducibility, and stimuli control; it enables automated generation of data about the details of simulations, including performance measurements that can be used for research or to provide automated individualized feedback [24]. The simple novelty of interactive technologies themselves, such as VR, can improve student motivation [36]. The initial cost and effort of creating the program can easily be compensated by broad distribution [16,37], as VR training is gradually finding its way into the medical curriculum [38].

Only a few virtual or mixed reality simulations exist for training hand hygiene [39-42]. In addition, high-quality studies evaluating the effectiveness and long-term retention of a VR simulation compared to conventional training methods are lacking.

Therefore, we hypothesized that a VR simulation could be an effective and useful tool with high student satisfaction to teach COVID-19 diagnostics, and we performed a randomized pilot study in medical students to explore (1) the effectiveness of a fully immersive VR simulation versus a traditional learning method regarding a COVID-19–related skill set (ie, proper hand hygiene, proficiency in PPE use, and correct acquisition of a nasopharyngeal specimen) tested in a simulated clinical scenario, and, as a secondary outcome, its long-term effectiveness 1 month after training; and (2) media-specific variables influencing training outcomes, such as usability, satisfaction, simulator sickness, and the experience of presence and immersion.


Study Design, Setting, and Ethical Approval

This is a prospective randomized controlled pilot study, taking place at the emergency department of the Inselspital, University Hospital Bern, Switzerland [43], from September to November 2020.

The study population consisted of a convenience sample of medical students at the University of Bern. All participants attended on a voluntary basis; no remuneration was provided. Informed consent was obtained. Data were collected, analyzed, and stored in anonymized form.

The local ethics committee deemed our study exempt from full ethical approval (Business Administration System for Ethics Committees Req-2020-00889).

Inclusion/Exclusion

The inclusion criteria were as follows: medical students (years 3-6 out of a 6-year curriculum) at the University of Bern.

The exclusion criteria consisted of unwillingness to participate or to provide informed consent.

Baseline Investigations

Baseline Survey

A brief survey about sociodemographic factors; prior training; and experience in hand hygiene and PPE use, taking of respiratory samples (nasopharyngeal swab), and prior experience with VR was performed after enrollment.

Assessments/Measurements

We evaluated the performance of hand disinfection, taking a nasopharyngeal swab on a manikin, and contamination during doffing of PPE (Figure 1).

Figure 1. Flowchart of the study. VR: virtual reality.
View this figure
Hand Disinfection

Hand disinfection performance was evaluated using a fluorescent marker (Visirub conc and Sterillium, Hartmann AG) and UV-light scanning was performed using the Derma Litecheck UV Multimedia device (KBD Ltd) at the time of enrollment (pretest), directly after the intervention (posttest 1), and 1 month after the intervention (posttest 2). Participants were blindfolded during the assessment and were unable to assess their results. A performance analysis scheme (documentation of missed locations; n=38 locations for each hand, with a total of 76 areas investigated) was developed by the institution’s infection control and medical educator team as adapted from Pan et al [44]. The outcome was the number of missed locations (range from 0 to 76; a lower number is better). Performance was supervised by an independent and trained rater, and the images were electronically recorded and analyzed according to the predefined scheme by a rater blinded to the intervention (Figures 2 and 3).

Figure 2. UV-light scanning of a perfectly disinfected palmar surface of the right hand.
View this figure
Figure 3. UV-light scanning of right hand dorsal surface. Missing areas of hand disinfection, right hand: digitus I, dorsal, distal phalanx; digitus II, dorsal, distal, and middle phalanx; digitus III, dorsal, distal, and middle phalanx; digitus IV, dorsal, distal phalanx; digitus V, dorsal, distal, and middle phalanx (total number of missed locations=8).
View this figure
Obtaining the Nasopharyngeal Swab and Evaluating Contamination During Doffing

A simulation setup for conducting a nasopharyngeal swab for COVID-19 testing on a manikin (Little Anne, Laerdal Medical) using proper hand hygiene and PPE was installed.

The correct procedure of taking a nasopharyngeal swab sample as well as possible contamination while doffing were evaluated directly after the intervention (posttest 1) and 1 month after the intervention (posttest 2).

An independent and trained rater blinded to the intervention assessed each participant’s performance using a 17-item checklist adapted from [8,10] based on the CDC guidelines for PPE [3,6], WHO guidelines for hand hygiene [5], and international recommendations for taking a nasopharyngeal swab [6]; this checklist was developed by the institution’s infection control and medical educator team (Multimedia Appendix 1). The outcome was the number of points achieved on the checklist (range 0-17; a higher number of points indicated a better result).

Contamination during the procedure was evaluated using fluorescent lotion (Dermalux Testlotion S, KBD Ltd), which was applied to the participants’ hands, forearms, and torso before the doffing of PPE. After doffing, 10 areas (right hand, right forearm, right upper arm, left hand, left forearm, left upper arm, torso ventral, torso dorsal, neck, head/ears) were analyzed by UV lighting for contamination by an independent rater. The outcome was the number of contaminated areas (range from 0 to 10; a lower number indicated a better result).

Intervention

Participants were randomized to either the intervention group (VR simulation) or control group in a 1:1 ratio using a computer-generated system.

VR Simulation

The intervention group was trained in COVID-19–related skills using the VR simulation (the Covid-19 VR Strikes Back (CVRSB) module, version 1.1.6), a software platform developed by ORamaVR SA, and the Oculus Rift S head mounted device and hand controllers (Facebook Inc). The proprietary ORamaVR software medical VR training application is available free of charge for all VR desktop and mobile HMDs [45] (Figure 4).

The participants performed two runs in the simulation using the single player modus.

Figure 4. Screenshot of the virtual reality (VR) application, Covid-19 VR Strikes Back, showing the taking of a nasopharyngeal swab.
View this figure

Control Group

The control group was trained using traditional learning methods: printed instructions and local instruction videos on COVID-19–related skills, ie, PPE donning and doffing, as well as formal videos on proper hand hygiene according to the WHO and on taking a correct nasopharyngeal sample [46].

Intervention Survey

Both groups were evaluated regarding variables of media use according to established questionnaires.

Usability for both training modules was assessed using the After-Scenario Questionnaire (ASQ) [47], which assesses the ease of task completion, satisfaction with completion time, and satisfaction with supporting information on a 7-point Likert scale (total score ranges from 1, full satisfaction, to 7, poor satisfaction).

The User Satisfaction Evaluation Questionnaire (USEQ) [48] contains 6 questions with a 5-point Likert scale to evaluate user satisfaction (total score ranges from 6, poor satisfaction, to 30, excellent satisfaction).

For the VR simulation, “visually induced motion sickness” was assessed with 4 items (nausea, headache, blurred vision, dizziness) according to the Simulator Sickness Questionnaire (SSQ) adapted from Kennedy et al [49] (total score ranges from 1, no simulator sickness, to 5, strong simulator sickness).

Presence and immersion in the virtual world was determined according to the 6-item questionnaire developed by Slater-Usoh-Steed [50] (total score ranges from 1, no immersion, to 7, full immersion).

Statistical Analysis

Data were analyzed using SPSS, version 22 (IBM Corporation), and Stata 16.1 (StataCorp).

Baseline characteristics are presented as numbers and percentages or medians and interquartile ranges using descriptive statistics as appropriate.

The intervention and control group were compared regarding the baseline characteristics by chi-square test and Wilcoxon rank sum test as applicable.

The Wilcoxon rank sum test was used at a specific time point between the study groups for the comparison of all four outcome groups: (1) the number of missed areas during hand disinfection, (2) achieved items from the 17-item checklist during nasopharyngeal swab acquisition, (3) the number of contaminated areas during doffing, and (4) variables of media use. Within-group differences for different time points were tested using the Wilcoxon matched-pairs signed-rank test.

For all tests, a P value <.05 was considered significant. For this pilot study, no adjustment for multiple testing was performed. Furthermore, pairwise comparisons were favored over more complex analyses, such as mixed linear regression analysis.


Sample Characteristics

In total, 29 students completed the study (control group, n=14; intervention group, n=15) (Figure 1). All students included completed the whole study. There were no dropouts. The baseline characteristics of the participants are detailed in Table 1. No significant differences were found regarding gender, mean age, educational level in medical school, need to wear glasses, previous experience with computer games, or previous experience with VR. Likewise, previous education and experience regarding hand disinfection, use of PPE, and taking nasopharyngeal swabs did not show any significant differences.

Table 1. Baseline characteristics of the study sample (N=29).

VRa group (n=15)Control group (n=14) P value
Sociodemographic factors

Female gender, n (%)9 (60)9 (64).81

Age (years), median (IQR)23 (22-25)22.5 (22-24).56

Year of medical school, n (%).44


31 (7)2 (14)


414 (93)11 (79)


50 (0)1 (7)

Wears glasses, n (%)8 (53)8 (57).84

Plays computer games regularly (Likert scale responseb), n (%).95


110 (67)10 (71)


24 (27)3 (21)


30 (0)0 (0)


40 (0)0 (0)


51 (7)1 (7)

Uses VR regularly (Likert scale responseb), n (%)


115 (100)14 (100)
Previous education and experience

Previous education in hand disinfection, n (%)14 (93)11 (79).25

Previous education in PPE,c n (%)6 (40)3 (21).28

Previous no. of swabs, median (IQR)0 (0-0)0 (0-0).54

Uses PPE regularly (Likert scale responseb), n (%).36


110 (67)8 (57)


23 (20)2 (14)


30 (0)3 (21)


41 (7)0 (0)


51 (7)1 (7)

aVR: virtual reality.

bLikert scale: 1, completely disagree, to 5, completely agree.

cPPE: personal protective equipment.

Hand Disinfection

There was no significant difference in the number of missed areas during hand disinfection at baseline (intervention group: median 21, IQR 11-27, vs control group: median 20, IQR 14-21; P=.47) (Table 2). Both groups performed significantly better after training without a significant group difference (posttest 1) (median 7, IQR 4-14, in the intervention group vs median 10, IQR 6-14, in the control group; P=.34). For the secondary outcome, at posttest 2, again, no significant difference was noted between the intervention and control groups (median 14, IQR 8-17, in the intervention group vs median 11, IQR 7-16, in the control group; P=.74). In both groups, no significant difference was found between posttests 1 and 2 (intervention group, P=.11; control group, P=.25) (Table 3).

Table 2. Comparison between the VR group and the control group regarding hand disinfection, nasopharyngeal swab testing, and contamination during doffing.
 Values, median (IQR)P value

VRa group (n=15)Control group (n=14)
Number of missing areas during hand disinfection (out of 76 possible areas)

Pretest21 (11-27)20 (14-21).47

Posttest 17 (4-14)10 (6-14).34

Posttest 214 (8-17)11 (7-16).74
Nasopharyngeal swab testscore (out of 17 points)

Posttest 114 (13-15)12 (11-14).03

Posttest 214 (14-16)14 (14-15).79
Number of contaminated body areas during doffing (out of 10 possible areas)

Posttest 12 (2-4)3 (1-4).64

Posttest 21 (0-2)0 (0-1).18

aVR: virtual reality.

Table 3. Comparison between posttests 1 and 2 for the VR group and the control group.

Values, median (IQR)P value


Posttest 1Posttest 2
Hand disinfection (number of missing areas)

VRa group7 (4-14)14 (8-17).11

Control group10 (6-14)11 (7-16).25
Swab test (score out of 17)

VR group14 (13-15)14 (14-16).28

Control group12 (11-14)14 (14-15).02
Doffing (number of contaminated areas)

VR group2 (2-4)1 (0-2).005

Control group3 (1-4)0 (0-1).003

aVR: virtual reality.

Nasopharyngeal Swab Acquisition

At posttest 1, the intervention group performed significantly better in taking a nasopharyngeal swab, scoring a median of 14 points on the 17-item checklist (IQR 13-15) versus 12 points (IQR 11-14) in the control group (P=.03) (Table 2). No significant differences between the groups were found after 1 month at posttest 2 (P=.79) for long-term retention as the secondary outcome.

The number of actual nasopharyngeal swabs performed in real life between posttests 1 and 2 did not differ between the groups (VR group, median no. of swabs 0, IQR 0-0; control group, median swabs 0, IQR 0-0; P=.56).

Contamination During Doffing

No significant difference between the number of contaminated areas during doffing was found between the groups at both time points (Table 2). However, in both groups, a significant reduction of contamination was noted at posttest 2 compared to posttest 1 (intervention group: posttest 1, median contaminated areas 2, IQR 2-4; posttest 2, median contaminated areas 1, IQR 0-2, P=.005; control group: posttest 1, median contaminated areas 3, IQR 1-4; posttest 2, median contaminated areas 0, IQR 0-1, P=.003).

Variables of Media Use

The results of the intervention survey regarding usability, satisfaction, simulator sickness, sense of presence, and immersion are detailed in Table 4.

Table 4. Variables related to media use.

Values, median (IQR)P value
 VRa group (n=15)Control group (n=14)
After-Scenario Questionnaire scoreb1 (1-3)3 (2-4).002
User Satisfaction Evaluation Questionnairec27 (23-28)22 (20-24).01
Simulator Sickness Questionnaire, abbreviatedd1 (1-3)N/AeN/A
Presence and immersion according to Slater-Usoh-Steedf5 (5-5)N/AN/A

aVR: virtual reality.

bRange 1 to 7 (1=full satisfaction).

cRange 6 to 30 (30=optimal satisfaction).

dRange 1 to 5 (1=no simulator sickness).

eN/A: not applicable.

fRange 1 to 7 (7=full presence and immersion).

The ASQ revealed a significantly better result for the VR module (median score in the intervention group: 1, IQR 1-3; median score in the control group: 3, IQR 2-4; P=.002), as did the USEQ (median score in the intervention group: 27/30, IQR 23-28; median score in the control group: 22/30, IQR 0-24; P=.01).

The median score in the 4-item SSQ in the intervention group was 1 (IQR 1-3), thus revealing good tolerability of the VR simulation.

Presence and immersion in the virtual world according to the questionnaire of Slater, Usoh, and Steed was high (median 5 on a scale from 1 to 7, IQR 5-5).


In this study, we demonstrated that our VR simulation was at least as effective as traditional learning methods (video and written instruction) in training medical students in COVID-19–related skills—namely, the correct performance of hand hygiene, use of PPE, and taking of a nasopharyngeal swab specimen—and that it provides a benefit in user satisfaction.

Effectiveness of Training

For the most investigated steps of the training, both educational methods improved performance to a similar extent without a significant difference. However, students in the VR group performed significantly better in acquiring a nasopharyngeal swab specimen on a manikin directly after the intervention than the control group, but this finding may be a result of multiple testing. Furthermore, the medians of the two groups only differed by 2 points on a 17-point outcome scale, and the IQRs overlapped.

There are few VR and MR simulations for training hand hygiene, and high quality evidence of their effectiveness is limited [39,40,42]. Shimada and colleagues [41] targeted preschool children, using a data glove instead of hand-held controllers to obtain the posture of user’s hand as a VR device. They used Leap Motion to obtain the posture of a hand, in contrast to our setting, which used only commercially available standard hardware. They found that their VR system was more effective than conventional hand hygiene instruction in a small group (n=12) of young children.

Performance of correct hand hygiene in our study was poor at baseline, in accordance with the existing evidence, stressing the need for effective instruction [7,11]. However, the performance significantly improved in both groups after the instruction, despite the limited technical possibilities to simulate the hand movements of disinfection with the need to hold the standard controllers in both hands while performing the correct movements. To minimize this limitation, participants could see the correct movements of their avatar’s hands (without the controllers) in a mirror.

This study adds to the body of evidence that VR simulation can help with the acquisition of simple skills, in combination with increased user satisfaction. Furthermore, our study highlights the necessity of strong collaborations between developers, users, and educators to ensure that these new technologies can complement and enhance existing educational curricula.

Variables of Media Use

Satisfaction is considered to be one of the key components of usability [48]. The satisfaction of participants in the VR group measured by the USEQ was significantly higher than that of the participants in the control group. As most of our students were inexperienced in VR, the novelty effect may have added to the results. This effect consists of an increase in perceived usability of a technology due to its newness or the tendency for performance to initially improve when new technology is instituted, not because of any actual improvement in learning or achievement but in response to increased interest in the new technology. However, Huang and colleagues [51] state that novelty does not necessarily increase learning achievement. According to them, the increase of learning achievement is more dependent on a match between the learning content and the learning method. The embodied learning method in VR is particularly appropriate for instructing difficult knowledge and spatial knowledge [51]. It remains to be elucidated whether satisfaction with and efficiency of VR simulations will decrease over time as the technology becomes more widespread. However, with potentially increasing technological advances, a certain novelty effect will remain.

To make our VR training available to as many users as possible, we avoided using specialized hardware such as that used in many other studies of VR skill training. It could be speculated that this use of off-the-shelf controllers may reduce the realism of the VR simulation. However, we were able to demonstrate that a high degree of immersion and satisfaction could be achieved with our simulation even with standard hardware. This may be even more pronounced with future developments, such as hand tracking without the need for traditional controllers.

This “experience of presence” in VR, which could be demonstrated in our study, is known to correlate positively with training effectiveness [52,53].

Skill Retention

In the follow-up after 1 month, there were no significant differences between the groups regarding any outcome. This cannot be explained by different exposures, as there was no difference in the mean number of swabs performed in real life in the meantime. One possible explanation is that the participants prepared more deliberately for the second appointment, as they might have suspected that it would involve a repetition of the first assessment (“assessment drives learning”) [54]. Most motor skills are lost over time, or at least the level of performance deteriorates, starting as soon as 1 day after training [55]. Maagaard et al [56] detected that the laparoscopic skills of novices acquired in VR simulator training deteriorated in a period between 6 and 18 months without further training.

We were able to show that the observed learning effect was maintained over the observed time frame of 1 month in both learning groups. Whether there will be a difference in skill decay between the two learning methods in the long run remains an important open question.

Strengths and Limitations

Our study has several strengths to point out. First, the study assesses outcomes of direct clinical relevance, not surrogates such as performance on multiple-choice tests or user satisfaction only. Second, the study not only assesses the effect of training on performance gains but also includes a quantification of skill retention over time. Third, we compared the novel VR intervention to an established educational alternative rather than to no intervention.

In addition, attrition bias was nonexistent because all participants completed the study protocol without dropping out.

As a practical benefit, our VR simulation program is available for free; thus, program directors and educators are able to enhance their existing curricula with an effective novel adjunct or alternative, or replicate the study setting.

This study has several limitations, including its single center design, which restricts external validity. The number of participants in our study was limited due to the large logistical and human resources required to conduct the study during a pandemic. Therefore, the detection of small differences between training modalities is not possible with our study design. There is the possibility of selection bias, based on volunteer convenience sampling of medical students, as well as a possible performance bias, with allocation to the interventional group leading to higher motivation, satisfaction, and performance. The need to use hand-held controllers instead of hand-tracking might have further impacted the efficiency of the VR simulation; however, we wanted to apply technological equipment that is widely available.

Furthermore, the correlation of these findings to clinical, patient-oriented outcomes remains to be validated.

Conclusion

To our knowledge, this is the first study using a VR simulation to train health care personnel in the correct use of hand hygiene, PPE, and taking a nasopharyngeal swab specimen and to compare the effectiveness to established traditional training (video and written instructions). VR simulation was at least as effective as traditional learning methods in training medical students while providing a benefit in user satisfaction. These results add to the growing body of evidence that VR is a useful tool for acquiring and maintaining simple and complex clinical skills.

Conflicts of Interest

WEH has received research funding from the European Union, the Swiss National Science Foundation, the Zoll Foundation, Dräger Medical Germany, Mundipharma Research UK, MDI International Australia, and Roche Diagnostics Germany, all outside the submitted work. WEH has provided paid consultancies to AO Foundation Switzerland, outside the submitted work. WEH received financial support for a congress he chaired from EBSCO Germany, Isabel Healthcare UK, Mundipharma Medical Switzerland, and VisualDx USA, all outside the submitted work. TCS holds an endowed professorship supported by the Touring Club Switzerland. The sponsor has no influence on the research conducted, in particular on the results or the decision to publish. GP is Chief Technical Officer of ORamaVR, the company that developed the software for this study. The company had no influence on the study protocol, the analysis, the results, or the decision to publish. This study was funded in part by a grant from the Swiss National Science foundation (grant number 31CA30_196615) for the project “Mixed-method evaluation of an online forward triage tool within the COVID-19 pandemic” to TCS and WEH. The other authors report no conflict of interest.

Editorial note: This randomized study was not registered, explained by authors as follows: "As our trial was not a clinical trial and did not involve patients or patient data, but instead was a pilot study involving medical students, it was not registered as a clinical trial." The editor granted an exception from ICMJE rules mandating prospective registration of randomized trials, given that the subjects were medical students. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness.

Multimedia Appendix 1

Checklist used for performance assessment.

DOCX File , 13 KB

Multimedia Appendix 2

CONSORT-EHEALTH checklist (V. 1.6.1).

PDF File (Adobe PDF File), 693 KB

  1. Lu H, Stratton CW, Tang Y. Outbreak of pneumonia of unknown etiology in Wuhan, China: the mystery and the miracle. J Med Virol 2020 Apr 12;92(4):401-402 [FREE Full text] [CrossRef] [Medline]
  2. The Lancet. COVID-19: protecting health-care workers. Lancet 2020 Mar;395(10228):922. [CrossRef]
  3. Using personal protective equipment (PPE). US Centers for Disease Control and Prevention.   URL: https://www.cdc.gov/coronavirus/2019-ncov/hcp/using-ppe.html [accessed 2020-05-11]
  4. Infection prevention and control during health care when novel coronavirus (nCoV) infection is suspected. World Health Organization.   URL: https:/​/www.​who.int/​publications-detail/​infection-prevention-and-control-during-health-care-when-novel-coronavirus-(ncov)-infection-is-suspected-20200125 [accessed 2020-05-12]
  5. WHO guidelines on hand hygiene in health care. World Health Organization. 2009 Jan 15.   URL: https://www.who.int/gpsc/5may/tools/9789241597906/en/ [accessed 2020-05-12]
  6. Protecting healthcare personnel. US Centers for Disease Control and Prevention.   URL: https://www.cdc.gov/hai/prevent/ppe.html [accessed 2020-05-12]
  7. John A, Tomas ME, Hari A, Wilson BM, Donskey CJ. Do medical students receive training in correct use of personal protective equipment? Med Educ Online 2017 Jan 04;22(1):1264125 [FREE Full text] [CrossRef] [Medline]
  8. Nagoshi Y, Cooper LA, Meyer L, Cherabuddi K, Close J, Dow J, et al. Application of an objective structured clinical examination to evaluate and monitor intern's proficiency of hand hygiene and personal protective equipment use in the United States. J Educ Eval Health Prof 2019 Oct 15;16:31 [FREE Full text] [CrossRef] [Medline]
  9. Hung P, Choi K, Chiang V. Using interactive computer simulation for teaching the proper use of personal protective equipment. Comput Inform Nurs 2015 Feb;33(2):57. [CrossRef]
  10. Beam EL, Gibbs SG, Boulter KC, Beckerdite ME, Smith PW. A method for evaluating health care workers' personal protective equipment technique. Am J Infect Control 2011 Jun;39(5):415-420 [FREE Full text] [CrossRef] [Medline]
  11. Verbeek J, Rajamaki B, Ijaz S, Sauni R, Toomey E, Blackwood B. Personal protective equipment for preventing highly infectious diseases due to exposure to contaminated body fluids in healthcare staff. Cochrane Database Syst Rev 2020;4:1-150. [CrossRef]
  12. John A, Tomas ME, Cadnum JL, Mana TS, Jencson A, Shaikh A, et al. Are health care personnel trained in correct use of personal protective equipment? Am J Infect Control 2016 Jul 01;44(7):840-842. [CrossRef] [Medline]
  13. Lockhart SL, Naidu JJ, Badh CS, Duggan LV. Simulation as a tool for assessing and evolving your current personal protective equipment: lessons learned during the coronavirus disease (COVID-19) pandemic. Can J Anaesth 2020 Jul 27;67(7):895-896 [FREE Full text] [CrossRef] [Medline]
  14. Pensieri C, Pennacchini M. Overview: virtual reality in medicine. J Virtual Worlds Res 2014 Jan 19;7(1):1-34 [FREE Full text] [CrossRef]
  15. Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 1994 Dec 24;E77-D(12):9 [FREE Full text]
  16. Li L, Yu F, Shi D, Shi J, Tian Z, Yang J, et al. Application of virtual reality technology in clinical medicine. Am J Transl Res 2017;9(9):3867-3880 [FREE Full text] [Medline]
  17. Slater M, Sanchez-Vives M. Enhancing our lives with immersive virtual reality. Front Robot AI 2016 Dec 19;3 [FREE Full text] [CrossRef]
  18. Babini MH, Kulish VV, Namazi H. Physiological state and learning ability of students in normal and virtual reality conditions: complexity-based analysis. J Med Internet Res 2020 Jun 01;22(6):e17945 [FREE Full text] [CrossRef] [Medline]
  19. Gerup J, Soerensen CB, Dieckmann P. Augmented reality and mixed reality for healthcare education beyond surgery: an integrative review. Int J Med Educ 2020 Jan 18;11:1-18 [FREE Full text] [CrossRef] [Medline]
  20. Bin KM, Zary N. Augmented reality, virtual reality and mixed reality in medical education: a comparative Web of Science scoping review. Preprints.org. Preprint posted online on April 29, 2019. [FREE Full text] [CrossRef]
  21. Kyaw BM, Saxena N, Posadzki P, Vseteckova J, Nikolaou CK, George PP, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019 Jan 22;21(1):e12959 [FREE Full text] [CrossRef] [Medline]
  22. Bracq M, Michinov E, Jannin P. Virtual reality simulation in nontechnical skills training for healthcare professionals. Sim Healthcare 2019 Jan 2;14(3):188-194. [CrossRef]
  23. Pantelidis P, Chorti A, Papagiouvanni I, Paparoidamis G, Drosos C, Panagiotakopoulos T. Virtual and augmented reality in medical education. In: Medical and Surgical Education - Past, Present and Future. London, UK: IntechOpen; 2017:77-97.
  24. Ustun AB, Yilmaz R, Yilmaz FGK. Virtual reality in medical education. In: Mobile Devices and Smart Gadgets in Medical Sciences. Hershey, PA: IGI Global; 2020:73.
  25. Semeraro F, Scapigliati A, Ristagno G, Luciani A, Gandolfi S, Lockey A, et al. Virtual reality for CPR training: how cool is that? Dedicated to the "next generation". Resuscitation 2017 Dec;121:e1-e2. [CrossRef] [Medline]
  26. Clarke E. Virtual reality simulation-the future of orthopaedic training? A systematic review and narrative analysis. Adv Simul (Lond) 2021 Jan 13;6(1):2 [FREE Full text] [CrossRef] [Medline]
  27. Lee R, Raison N, Lau WY, Aydin A, Dasgupta P, Ahmed K, et al. A systematic review of simulation-based training tools for technical and non-technical skills in ophthalmology. Eye (Lond) 2020 Oct 13;34(10):1737-1759 [FREE Full text] [CrossRef] [Medline]
  28. Lohre R, Bois AJ, Pollock JW, Lapner P, McIlquham K, Athwal GS, et al. Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: a randomized clinical trial. JAMA Netw Open 2020 Dec 01;3(12):e2031217 [FREE Full text] [CrossRef] [Medline]
  29. Polce EM, Kunze KN, Williams BT, Krivicich LM, Maheshwer B, Beletsky A, et al. Efficacy and validity of orthopaedic simulators in surgical training: a systematic review and meta-analysis of randomized controlled trials. J Am Acad Orthop Surg 2020 May 8;28(24):1027-1040. [CrossRef]
  30. Hassandra M, Galanis E, Hatzigeorgiadis A, Goudas M, Mouzakidis C, Karathanasi EM, et al. Α virtual reality app for physical and cognitive training of older people with mild cognitive impairment: mixed methods feasibility study. JMIR Serious Games 2021 Mar 24;9(1):e24170 [FREE Full text] [CrossRef] [Medline]
  31. Rourke S. How does virtual reality simulation compare to simulated practice in the acquisition of clinical psychomotor skills for pre-registration student nurses? A systematic review. Int J Nurs Stud 2020 Feb;102:103466. [CrossRef] [Medline]
  32. Khan R, Plahouras J, Johnston BC, Scaffidi MA, Grover SC, Walsh CM. Virtual reality simulation training in endoscopy: a Cochrane review and meta-analysis. Endoscopy 2019 Jul 09;51(7):653-664. [CrossRef] [Medline]
  33. Hooper J, Tsiridis E, Feng JE, Schwarzkopf R, Waren D, Long WJ, NYU Virtual Reality Consortium. Virtual reality simulation facilitates resident training in total hip arthroplasty: a randomized controlled trial. J Arthroplasty 2019 Oct;34(10):2278-2283. [CrossRef] [Medline]
  34. Grantcharov TP. Is virtual reality simulation an effective training method in surgery? Nat Clin Pract Gastroenterol Hepatol 2008 May 1;5(5):232-233. [CrossRef] [Medline]
  35. Zhao G, Fan M, Yuan Y, Zhao F, Huang H. The comparison of teaching efficiency between virtual reality and traditional education in medical education: a systematic review and meta-analysis. Ann Transl Med 2021 Feb;9(3):252-252 [FREE Full text] [CrossRef] [Medline]
  36. Kavanagh S, Luxton-Reilly A, Wuensche B, Plimmer B. A systematic review of virtual reality in education. Themes Sci Technol Educ 2017;10(2):119 [FREE Full text]
  37. King D, Tee S, Falconer L, Angell C, Holley D, Mills A. Virtual health education: scaling practice to transform student learning: using virtual reality learning environments in healthcare education to bridge the theory/practice gap and improve patient safety. Nurse Educ Today 2018 Dec;71:7-9. [CrossRef] [Medline]
  38. Hautz S, Hoffmann M, Exadaktylos A, Hautz W, Sauter T. Digital competencies in medical education in Switzerland: an overview of the current situation. GMS J Med Educ 2020;37(6):Doc62 [FREE Full text] [CrossRef] [Medline]
  39. Gill B, Tsang Y. Technological innovation in nursing education and practice. The Open University of Hong Kong. 2019.   URL: http:/​/www.​ouhk.edu.hk/​NHS/​Conference&Symposium/​STINE/​eBook/​Technological%20Innovation%20in%20Nursing%20Education%20and%20Practice.​pdf [accessed 2021-09-28]
  40. Clack L, Hirt C, Wenger M, Saleschus D, Kunz A, Sax H. VIRTUE - a virtual reality trainer for hand hygiene. 2018 Presented at: 9th International Conference on Information, Intelligence, Systems and Applications (IISA); July 23-25, 2018; Zakynthos, Greece. [CrossRef]
  41. Shimada S, Funahashi K, Ito K, Tanase Y, Iwazaki K. VR hand hygiene training system that visualizes germs to be washed and removed. 2017 Presented at: InIEEE 6th Global Conference on Consumer Electronics (GCCE); October 24-27, 2017; Nagoya, Japan. [CrossRef]
  42. Kanazawa A, Asai T, Minazuki A, Hayashi H. Construction of a MR space using projection mapping to promote hand hygiene. 2014 Presented at: InIIAI 3rd International Conference on Advanced Applied Informatics; Kokura Kita-ku, Japan; Aug 31-Sept 4, 2014. [CrossRef]
  43. Exadaktylos A, Hautz WE. Emergency medicine in Switzerland. ICU Manag Pract 2015;15:3 [FREE Full text]
  44. Pan S, Chen E, Tien K, Hung I, Sheng W, Chen Y, et al. Assessing the thoroughness of hand hygiene: "seeing is believing". Am J Infect Control 2014 Jul;42(7):799-801. [CrossRef] [Medline]
  45. Project Covid-19 VR Strikes Back. ORamaVR.   URL: http://elearn.oramavr.com/cvrsb/ [accessed 2021-04-13]
  46. Marty FM, Chen K, Verrill KA. How to obtain a nasopharyngeal swab specimen. N Engl J Med 2020 May 28;382(22):e76. [CrossRef]
  47. Lewis JR. Psychometric evaluation of an after-scenario questionnaire for computer usability studies. SIGCHI Bull 1991 Jan;23(1):78-81. [CrossRef]
  48. Gil-Gómez JA, Manzano-Hernández P, Albiol-Pérez S, Aula-Valero C, Gil-Gómez H, Lozano-Quilis J. USEQ: a short questionnaire for satisfaction evaluation of virtual rehabilitation systems. Sensors (Basel) 2017 Jul 07;17(7):1589 [FREE Full text] [CrossRef] [Medline]
  49. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol 1993 Jul;3(3):203-220. [CrossRef]
  50. Usoh M, Catena E, Arman S, Slater M. Using presence questionnaires in reality. Presence: Teleoperators Virtual Environ 2000 Oct;9(5):497-503. [CrossRef]
  51. Huang W. Investigating the Novelty Effect in Virtual Reality on STEM Learning. Thesis. Arizona State University. 2020.   URL: http://hdl.handle.net/2286/R.I.57391 [accessed 2021-06-02]
  52. Lerner D, Mohr S, Schild J, Göring M, Luiz T. An immersive multi-user virtual reality for emergency simulation training: usability study. JMIR Serious Games 2020 Jul 31;8(3):e18822 [FREE Full text] [CrossRef] [Medline]
  53. Mantovani F, Castelnuovo G. The sense of presence in virtual training: enhancing skills acquisition and transfer of knowledge through learning experience in virtual environments. In: Riva G, Davide F, Ijsselsteijn WA, editors. Being There: Concepts, Effects and Measurement of User Presence in Synthetic Environments. Amsterdam, the Netherlands: IOS Press; 2003:167-181.
  54. McLachlan JC. The relationship between assessment and learning. Med Educ 2006 Aug;40(8):716-717. [CrossRef] [Medline]
  55. Arthur Jr W, Bennett Jr W, Stanush PL, McNelly TL. Factors that influence skill decay and retention: a quantitative review and analysis. Hum Perform 1998 Mar;11(1):57-101. [CrossRef]
  56. Maagaard M, Sorensen JL, Oestergaard J, Dalsgaard T, Grantcharov TP, Ottesen BS, et al. Retention of laparoscopic procedural skills acquired on a virtual-reality surgical trainer. Surg Endosc 2011 Mar 7;25(3):722-727. [CrossRef] [Medline]


ASQ: After-Scenario Questionnaire
CDC: US Centers for Disease Control
CVRSB: Covid-19 VR Strikes Back
HMD: head-mounted device
MR: mixed reality
PPE: personal protective equipment
SSQ: Simulator-Sickness Questionnaire
USEQ: User Satisfaction Evaluation Questionnaire
VR: virtual reality
WHO: World Health Organization


Edited by N Zary; submitted 26.04.21; peer-reviewed by R Lundin, S Gallagher; comments to author 26.05.21; revised version received 12.06.21; accepted 09.09.21; published 08.10.21

Copyright

©Tanja Birrenbach, Josua Zbinden, George Papagiannakis, Aristomenis K Exadaktylos, Martin Müller, Wolf E Hautz, Thomas Christian Sauter. Originally published in JMIR Serious Games (https://games.jmir.org), 08.10.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.