This is a prospective randomized controlled pilot study, taking place at the emergency department of the Inselspital, University Hospital Bern, Switzerland [43], from September to November 2020.

The study population consisted of a convenience sample of medical students at the University of Bern. All participants attended on a voluntary basis; no remuneration was provided. Informed consent was obtained. Data were collected, analyzed, and stored in anonymized form.

The local ethics committee deemed our study exempt from full ethical approval (Business Administration System for Ethics Committees Req-2020-00889).

The inclusion criteria were as follows: medical students (years 3-6 out of a 6-year curriculum) at the University of Bern.

The exclusion criteria consisted of unwillingness to participate or to provide informed consent.

Participants were randomized to either the intervention group (VR simulation) or control group in a 1:1 ratio using a computer-generated system.

The intervention group was trained in COVID-19–related skills using the VR simulation (the Covid-19 VR Strikes Back (CVRSB) module, version 1.1.6), a software platform developed by ORamaVR SA, and the Oculus Rift S head mounted device and hand controllers (Facebook Inc). The proprietary ORamaVR software medical VR training application is available free of charge for all VR desktop and mobile HMDs [45] (Figure 4).

The participants performed two runs in the simulation using the single player modus.

The control group was trained using traditional learning methods: printed instructions and local instruction videos on COVID-19–related skills, ie, PPE donning and doffing, as well as formal videos on proper hand hygiene according to the WHO and on taking a correct nasopharyngeal sample [46].

Both groups were evaluated regarding variables of media use according to established questionnaires.

Usability for both training modules was assessed using the After-Scenario Questionnaire (ASQ) [47], which assesses the ease of task completion, satisfaction with completion time, and satisfaction with supporting information on a 7-point Likert scale (total score ranges from 1, full satisfaction, to 7, poor satisfaction).

The User Satisfaction Evaluation Questionnaire (USEQ) [48] contains 6 questions with a 5-point Likert scale to evaluate user satisfaction (total score ranges from 6, poor satisfaction, to 30, excellent satisfaction).

For the VR simulation, “visually induced motion sickness” was assessed with 4 items (nausea, headache, blurred vision, dizziness) according to the Simulator Sickness Questionnaire (SSQ) adapted from Kennedy et al [49] (total score ranges from 1, no simulator sickness, to 5, strong simulator sickness).

Presence and immersion in the virtual world was determined according to the 6-item questionnaire developed by Slater-Usoh-Steed [50] (total score ranges from 1, no immersion, to 7, full immersion).

Data were analyzed using SPSS, version 22 (IBM Corporation), and Stata 16.1 (StataCorp).

Baseline characteristics are presented as numbers and percentages or medians and interquartile ranges using descriptive statistics as appropriate.

The intervention and control group were compared regarding the baseline characteristics by chi-square test and Wilcoxon rank sum test as applicable.

The Wilcoxon rank sum test was used at a specific time point between the study groups for the comparison of all four outcome groups: (1) the number of missed areas during hand disinfection, (2) achieved items from the 17-item checklist during nasopharyngeal swab acquisition, (3) the number of contaminated areas during doffing, and (4) variables of media use. Within-group differences for different time points were tested using the Wilcoxon matched-pairs signed-rank test.

For all tests, a P value <.05 was considered significant. For this pilot study, no adjustment for multiple testing was performed. Furthermore, pairwise comparisons were favored over more complex analyses, such as mixed linear regression analysis.

In total, 29 students completed the study (control group, n=14; intervention group, n=15) (Figure 1). All students included completed the whole study. There were no dropouts. The baseline characteristics of the participants are detailed in Table 1. No significant differences were found regarding gender, mean age, educational level in medical school, need to wear glasses, previous experience with computer games, or previous experience with VR. Likewise, previous education and experience regarding hand disinfection, use of PPE, and taking nasopharyngeal swabs did not show any significant differences.

There was no significant difference in the number of missed areas during hand disinfection at baseline (intervention group: median 21, IQR 11-27, vs control group: median 20, IQR 14-21; P=.47) (Table 2). Both groups performed significantly better after training without a significant group difference (posttest 1) (median 7, IQR 4-14, in the intervention group vs median 10, IQR 6-14, in the control group; P=.34). For the secondary outcome, at posttest 2, again, no significant difference was noted between the intervention and control groups (median 14, IQR 8-17, in the intervention group vs median 11, IQR 7-16, in the control group; P=.74). In both groups, no significant difference was found between posttests 1 and 2 (intervention group, P=.11; control group, P=.25) (Table 3).

At posttest 1, the intervention group performed significantly better in taking a nasopharyngeal swab, scoring a median of 14 points on the 17-item checklist (IQR 13-15) versus 12 points (IQR 11-14) in the control group (P=.03) (Table 2). No significant differences between the groups were found after 1 month at posttest 2 (P=.79) for long-term retention as the secondary outcome.

The number of actual nasopharyngeal swabs performed in real life between posttests 1 and 2 did not differ between the groups (VR group, median no. of swabs 0, IQR 0-0; control group, median swabs 0, IQR 0-0; P=.56).

No significant difference between the number of contaminated areas during doffing was found between the groups at both time points (Table 2). However, in both groups, a significant reduction of contamination was noted at posttest 2 compared to posttest 1 (intervention group: posttest 1, median contaminated areas 2, IQR 2-4; posttest 2, median contaminated areas 1, IQR 0-2, P=.005; control group: posttest 1, median contaminated areas 3, IQR 1-4; posttest 2, median contaminated areas 0, IQR 0-1, P=.003).

The results of the intervention survey regarding usability, satisfaction, simulator sickness, sense of presence, and immersion are detailed in Table 4.

The ASQ revealed a significantly better result for the VR module (median score in the intervention group: 1, IQR 1-3; median score in the control group: 3, IQR 2-4; P=.002), as did the USEQ (median score in the intervention group: 27/30, IQR 23-28; median score in the control group: 22/30, IQR 0-24; P=.01).

The median score in the 4-item SSQ in the intervention group was 1 (IQR 1-3), thus revealing good tolerability of the VR simulation.

Presence and immersion in the virtual world according to the questionnaire of Slater, Usoh, and Steed was high (median 5 on a scale from 1 to 7, IQR 5-5).

For the most investigated steps of the training, both educational methods improved performance to a similar extent without a significant difference. However, students in the VR group performed significantly better in acquiring a nasopharyngeal swab specimen on a manikin directly after the intervention than the control group, but this finding may be a result of multiple testing. Furthermore, the medians of the two groups only differed by 2 points on a 17-point outcome scale, and the IQRs overlapped.

There are few VR and MR simulations for training hand hygiene, and high quality evidence of their effectiveness is limited [39,40,42]. Shimada and colleagues [41] targeted preschool children, using a data glove instead of hand-held controllers to obtain the posture of user’s hand as a VR device. They used Leap Motion to obtain the posture of a hand, in contrast to our setting, which used only commercially available standard hardware. They found that their VR system was more effective than conventional hand hygiene instruction in a small group (n=12) of young children.

Performance of correct hand hygiene in our study was poor at baseline, in accordance with the existing evidence, stressing the need for effective instruction [7,11]. However, the performance significantly improved in both groups after the instruction, despite the limited technical possibilities to simulate the hand movements of disinfection with the need to hold the standard controllers in both hands while performing the correct movements. To minimize this limitation, participants could see the correct movements of their avatar’s hands (without the controllers) in a mirror.

This study adds to the body of evidence that VR simulation can help with the acquisition of simple skills, in combination with increased user satisfaction. Furthermore, our study highlights the necessity of strong collaborations between developers, users, and educators to ensure that these new technologies can complement and enhance existing educational curricula.

Satisfaction is considered to be one of the key components of usability [48]. The satisfaction of participants in the VR group measured by the USEQ was significantly higher than that of the participants in the control group. As most of our students were inexperienced in VR, the novelty effect may have added to the results. This effect consists of an increase in perceived usability of a technology due to its newness or the tendency for performance to initially improve when new technology is instituted, not because of any actual improvement in learning or achievement but in response to increased interest in the new technology. However, Huang and colleagues [51] state that novelty does not necessarily increase learning achievement. According to them, the increase of learning achievement is more dependent on a match between the learning content and the learning method. The embodied learning method in VR is particularly appropriate for instructing difficult knowledge and spatial knowledge [51]. It remains to be elucidated whether satisfaction with and efficiency of VR simulations will decrease over time as the technology becomes more widespread. However, with potentially increasing technological advances, a certain novelty effect will remain.

To make our VR training available to as many users as possible, we avoided using specialized hardware such as that used in many other studies of VR skill training. It could be speculated that this use of off-the-shelf controllers may reduce the realism of the VR simulation. However, we were able to demonstrate that a high degree of immersion and satisfaction could be achieved with our simulation even with standard hardware. This may be even more pronounced with future developments, such as hand tracking without the need for traditional controllers.

This “experience of presence” in VR, which could be demonstrated in our study, is known to correlate positively with training effectiveness [52,53].

In the follow-up after 1 month, there were no significant differences between the groups regarding any outcome. This cannot be explained by different exposures, as there was no difference in the mean number of swabs performed in real life in the meantime. One possible explanation is that the participants prepared more deliberately for the second appointment, as they might have suspected that it would involve a repetition of the first assessment (“assessment drives learning”) [54]. Most motor skills are lost over time, or at least the level of performance deteriorates, starting as soon as 1 day after training [55]. Maagaard et al [56] detected that the laparoscopic skills of novices acquired in VR simulator training deteriorated in a period between 6 and 18 months without further training.

We were able to show that the observed learning effect was maintained over the observed time frame of 1 month in both learning groups. Whether there will be a difference in skill decay between the two learning methods in the long run remains an important open question.

Our study has several strengths to point out. First, the study assesses outcomes of direct clinical relevance, not surrogates such as performance on multiple-choice tests or user satisfaction only. Second, the study not only assesses the effect of training on performance gains but also includes a quantification of skill retention over time. Third, we compared the novel VR intervention to an established educational alternative rather than to no intervention.

In addition, attrition bias was nonexistent because all participants completed the study protocol without dropping out.

As a practical benefit, our VR simulation program is available for free; thus, program directors and educators are able to enhance their existing curricula with an effective novel adjunct or alternative, or replicate the study setting.

This study has several limitations, including its single center design, which restricts external validity. The number of participants in our study was limited due to the large logistical and human resources required to conduct the study during a pandemic. Therefore, the detection of small differences between training modalities is not possible with our study design. There is the possibility of selection bias, based on volunteer convenience sampling of medical students, as well as a possible performance bias, with allocation to the interventional group leading to higher motivation, satisfaction, and performance. The need to use hand-held controllers instead of hand-tracking might have further impacted the efficiency of the VR simulation; however, we wanted to apply technological equipment that is widely available.

Furthermore, the correlation of these findings to clinical, patient-oriented outcomes remains to be validated.

To our knowledge, this is the first study using a VR simulation to train health care personnel in the correct use of hand hygiene, PPE, and taking a nasopharyngeal swab specimen and to compare the effectiveness to established traditional training (video and written instructions). VR simulation was at least as effective as traditional learning methods in training medical students while providing a benefit in user satisfaction. These results add to the growing body of evidence that VR is a useful tool for acquiring and maintaining simple and complex clinical skills.