From premedical students to practicing physicians, competency-based medical education (CBME) has been applied in the design, implementation, and evaluation of medical education programs according to competency-based outcomes [1]. CBME for undergraduate medical students can help to improve task-specific confidence and test performance after the course has been completed, resulting in better performance and patient care before residency training [2]. CBME has been widely employed to promote greater learner-centeredness and curricular outcomes since 2010 [3]. Furthermore, CBME must use multiple assessment tools that meet minimum requirements for quality. Therefore, a robust and multifaceted assessment system, including quantitative and qualitative methods, is essential for evaluating the progress of learners [4].

CBME is used to improve graduates’ competency levels, ensuring they are skillful and qualified in all critical areas of their occupation. To this end, CBME emphasizes the use and importance of simulation-based training, which considers patient safety and for which real-life opportunities are limited [5]. This consideration is crucial for medical learners and teachers during the COVID-19 pandemic. History taking and physical examination (H&P) is a principal competency, incorporating knowledge, skills, and behavior to initially approach a patient. Therefore, H&P is a key performance level of otorhinolaryngology–head and neck surgery (ORL-HNS) [6]. To enhance the development of H&P skills, commonly used methods to assess this competency include in-training examinations such as the Milestone [7], Mini Clinical Evaluation Exercise [8], and oral examinations of clinical practice [9]. Recently, simulations such as part-time trainers, integrated simulators, virtual reality (VR), and wearable devices have become increasingly popular for CBME [10]. Simulation-based training, such as VR, has been shown to improve health professionals’ knowledge and skills outcomes and be an integrative step toward supervised clinical practice [11]. In particular, VR allowed students to practice rounding skills, facilitate their education during the COVID-19 pandemic, and supplement their in-person clerkship education [12].

VR consists of a computer-generated 3D simulation in which the user both explores and manipulates the contents of the environment to learn and assess clinical knowledge and skills [13]. VR provides experiential learning and provides standardized, controlled exposure to situated events, patient-caregiver communication, and teamwork. The use of VR has been shown to be highly acceptable by learners in a wide range of health care settings [14] and to play an essential role in improving performance [15]. As a subtype of image-based VR, 360° VR video represents an immersive 3D medium featuring authenticity and fidelity using a VR head-mounted display. The use of 360° VR video opens up many possibilities in many domains of medical education [16], such as an independent teaching aid or an adjunct to traditional face-to-face teaching [17]. The application of 360° VR video has enhanced the effectiveness of medical education and training, raised the level of diagnosis and treatment, improved the doctor-patient relationship, and boosted medical execution efficiency [18]. Using 360° VR videos to facilitate the acquisition of new clinical skills has been suggested to be a valuable step in developing a clinical teaching curriculum [19].

However, transferring practical skills to a clinical setting using a criterion-based training program with VR simulators is difficult [20]. In addition to the poor mechanical performance of the simulated haptic feedback, complex tasks such as H&P and surgical procedures can induce excessive cognitive load during simulation training, which can harm learning, especially for novices [21]. Since the learner’s cognitive capacity during learning is limited, improper instructional design may waste precious cognitive resources, impair the essential processing, and cause generative underutilization during learning [22]. Therefore, cognitive overload negatively impacts learning outcomes in simulation training [23,24]. For example, the increased cognitive load was significantly associated with the declined correct identification of a trained murmur during simulation training [23]. A study of laparoscopic training [24] reported that immersive VR simulation training resulted in the cognitive overload that impeded actual learning and skills acquisition compared with conventional VR simulation training. However, structured and distributed VR simulation practices may induce a lower cognitive load when the learning situation is increased in complexity [21]. Recently, VR-based simulation training under cognitive load control has improved performance under similar conditions to an actual surgical task [25]. Therefore, estimating cognitive load during 360° VR and 2D video learning is essential for practical instruction and reform.

Estimations of cognitive load and specific load types are still challenging based on current cognitive load theory [26,27]. For example, subjective cognitive load questionnaires, such as the Paas Cognitive Load Scale (Paas-CLS) [28] and the National Aeronautics and Space Administration Task Load Index (NASA-TLX) [29] have been shown to be good tools to measure intrinsic load, but not extraneous and germane loads [30]. However, a lowering of intrinsic cognitive load can reduce total cognitive load, thus releasing working memory capacity [31]. Furthermore, instructional techniques for reducing cognitive load have been shown to improve learning [32]. In addition to estimating cognitive load, the NASA-TLX has been successfully applied to determine user acceptance when evaluating innovative applications [33]. Nevertheless, a negative carryover effect due to failure in a VR simulation training program could affect the subjective cognitive load estimations [34]. Therefore, objective estimates of cognitive load, including secondary-task performance [35], during the training program have been applied in many medical education studies [34-36]. Secondary-task performance is a task that assesses participants’ attention and is limited by the storage capacity of visual short-term memory [37]. Although secondary-task performance is a cognitive function test, it frequently deteriorates from baseline to dual-task among novices and is particularly useful for tracking changes in cognitive load in the early phases of simulation-based skills [35].

VR immersion, especially containing stressful content, can evoke acute stress reactions accompanied by autonomic dysfunction [38]. Heart rate variability (HRV) is a sensitive indicator of cardiac autonomic modulation [39] and responds to any psychophysical changes immediately. Several physiological or psychological changes, such as stress [40] and anxiety [41], can change HRV. Reduced HRV indexes might reflect loss of cognitive efficiency [36] and might predict increased cognitive load [42] and poor cognitive performance [43]. Nevertheless, immersive VR using the first-person perspective can induce body ownership illusion in an uncomfortable posture, reduced HRV indexes, and more mistakes in a cognitive task [44]. Despite it being challenging to assess associations between HRV and subjective cognitive load due to sizeable interindividual variability [45], measuring HRV during a learning task may help evaluate whether learning materials impact learning outcomes through cardiac changes autonomic function.

This study aimed (1) to evaluate differences between two learning materials (ie, 360° VR video and 2D video) in the subjective and objective cognitive loads, autonomic function, outcome, and learning experience while students were learning H&P skills and (2) to study how learning materials (ie, 360° VR or 2D video) impact learning outcomes through the changes in cognitive load and HRV. The initial hypotheses were that (1) subjective and objective cognitive loads and autonomic function would be changed while using different learning materials and (2) learners with higher learning outcomes and experience would have different cognitive load and autonomic function during the learning tasks than those with lower learning outcomes and experience.

Therefore, we used cognitive questionnaires immediately after completing a video learning module to assess subjective cognitive load. We also used secondary-task reaction time and HRV to objectively estimate the dynamic changes of cognitive load and cardiac autonomic function during a video module for providing ongoing feedback to improve video learning. Subsequently, we used the Milestone report form to evaluate the real-patient H&P. The Milestones have been designed according to the grounding principles of CBME and have become a significant formative component of the current accreditation model of graduate medical education [46]. Although the Milestones were not designed to assess undergraduate medical students, we considered that some descriptors and targets of the Milestones were suitable for evaluating learner performance of H&P as an undergraduate medical student moves from entry into clerkship through graduation. Furthermore, we applied the AttrakDiff2 questionnaire to assess the acceptance of technical innovations [47]. The AttrakDiff2 questionnaire has been used to investigate learners’ experience of many educational innovations, such as mobile e-learning [48] and augmented reality [49]. Finally, we used anonymous qualitative feedback to determine the learning experience. We considered that the learning outcomes and the learning experience could meet learners’ needs and prompt their use of learning through different video modules.

We conducted a prospective, randomized controlled, pilot system–design study from June 1 to October 30, 2018, at an academic teaching hospital (Department of ORL-HNS, Linkou Chang Gung Memorial Hospital, Taoyuan, Taiwan). The Institutional Review Board of the Chang Gung Medical Foundation approved this study (No. 201601821B0), and we conducted all procedures in compliance with the Declaration of Helsinki 1975. We informed the participants about the aims of the study and then obtained written informed consent from them. We registered the entire study proposal at ClinicalTrials.gov (NCT03501641). Figure 1 shows the study flowchart following the CONSORT 2010 guidelines [50].

A convenience sampling approach was used, and 32 consecutive volunteers were recruited during the study period. The inclusion criteria were as follows: (1) aged >20 years and (2) novices in ORL-HNS (ie, undergraduate medical students). The exclusion criteria were as follows: (1) contraindications for using VR, such as pregnancy, hypertension, motion sickness, inner ear infections, claustrophobia, recent surgery, pre-existing binocular vision abnormalities, heart disorders, or epilepsy, and (2) declining to participate. All of the volunteers had at least a basic level of computer literacy and could use VR headsets and controllers after instruction. We used the 25-item Group Embedded Figures Test (GEFT) (score range 0-18) to assess the participants’ cognitive style [54]. The GEFT has been shown to have high reliability in medical education [55], and we have previously validated its effectiveness of classifying learning preference in millennial undergraduate medical students [56]. Field-independent learners prefer and have better performance in computer-assisted learning. We stratified the students into two subgroups: “field-dependent” (GEFT score ≤12) and “field-independent” (GEFT score >12) [57].

The participants were blinded to the purpose of the study during recruitment to minimize preparation bias. After the participants had provided consent and completed the GEFT, we randomly assigned them, with a 1:1 allocation, to the 360° VR video group and the 2D video group, matched by age, sex, and cognitive style (Figure 1). The Random Number Generators tool in SPSS software (version 24; IBM Corp) was used to create a list of random numbers for allocating the students, who were stratified by center with a 1:1 allocation using a fixed block size of 8 in both parallel subgroups. We concealed the allocation sequence before implementing the video module, and the module adhered to our computer-generated randomization protocol.

After randomization, the participants reviewed their allocated video through a head-mounted display in the same space for 10 minutes. To reduce the effect of the head-mounted display on learning experience, both groups used the same VR device (VIVE VR headset; HTC Corp). We explained the functionality of the VR device to the participants before the intervention. In the 360° VR video group, the learners arbitrarily reviewed the instructor’s demonstrations and responses from standard patients and other medical staff from a first-person perspective in an immersive 360° environment (Figure 2). In the 2D video group, the learners simply watched the instructor’s demonstrations from a third-person perspective in a theater environment (Figure 3). During the learning course, the participants watched the movie by themselves at a time of their choosing.

Each participant in this study provided anonymous feedback about their experience of the module used.

The primary outcome measure of this study was the Milestone level after completing the video learning module. The secondary outcomes were the AttrakDiff2 questionnaire scales.

The sample size was estimated using primary outcome effects (the Milestone) based on a priori study (360° VR video: mean 3.1, SD 0.7; 2D video: mean 2.3, SD 0.6). A two-tailed Wilcoxon-Mann-Whitney U test was used to calculate a sample size of 16 in each group (normal parent distribution; calculated effect size, 1.23; type I error, 0.01; power, 70%). For a block size of 8, we decided to enroll a total of 32 students to show the difference in the Milestone level.

The D’Agostino-Pearson omnibus normality test showed that most of the continuous variables were nonnormally distributed, and they were presented as median and IQR. Differences between groups were analyzed using the Wilcoxon signed-rank test, the Mann-Whitney U test, or the Fisher exact test as appropriate. Effect sizes were calculated using the Hodges-Lehmann method for the Mann-Whiney U test and the Wilcoxon signed-rank test, or odds ratios for the Fisher exact test. The Spearman correlation test was used to analyze relationships between variables of interest. The Bonferroni correction was used to adjust P values because of the increased risk of a type I error when conducting multiple statistical tests at the same time [72]. Continuous variables were dichotomized using the median split. Variables of interest were analyzed for multivariate logistic regression models. All P values were two‐sided, and statistical significance was accepted at P<.01. Statistical analyses were performed using G*Power software (version 3.1.9.2; Heinrich-Heine-Universität Düsseldorf), Prism for Windows (version 7.0; GraphPad Software Inc), and SPSS Statistics for Windows (version 25; IBM Corp).

A total of 32 volunteers were recruited, of which 20 (63%) were males and 12 (38%) were females. The median age was 24 (IQR 23-25) years. There were 3 (9%) field-dependent and 29 (91%) field-independent participants. Table 2 summarizes the variables of interest for the overall study cohort. As expected, there were no significant differences in age, sex, or cognitive style between the 360° VR video and 2D video groups at baseline. After randomization, all participants received the intended intervention. There were no protocol deviations in this study.

Differences across the RR at 5 to 10 minutes, RR at 0 to 5 minutes, and RR at baseline were not statistically significant in the overall cohort, the 360° VR video group, and the 2D video group, respectively. Furthermore, the RR at 5 to 10 minutes, RR at 0 to 5 minutes, and RR at baseline were comparable between both groups (Table 4). Differences in the SDNN index were not statistically significant for either intragroup comparisons or intergroup comparisons. Differences in the RMSSD index were not statistically significant for either intragroup comparisons or intergroup comparisons. Differences in the LF/HF index were not statistically significant for either intragroup comparisons or intergroup comparisons.

Overall, the participants had a significantly higher Milestone level (median 3, IQR 2-4) than the reference value of “1” (P<.001) after 10 minutes of video instruction. Although the Milestone level of the 360° VR video group (median 3, IQR 3-4) was higher than that of the 2D video group (median 2, IQR 2-3), the difference did not reach statistical significance (effect size=1, 95% CI 0-1; P=.02) (Figure 4, lower).

The overall cohort and the 360° VR and 2D video groups had significantly positive learning experiences in terms of pragmatic quality, hedonic stimulation, hedonic identification, and attractiveness, compared with the reference value of “0” with all corrected P<.01, after 10 minutes of video instruction (Table 5). There were no statistically significant differences in pragmatic quality, hedonic stimulation, hedonic identification, or attractiveness between the two groups.

To investigate how learning materials impact learning outcomes through the changes in cognitive load and HRV, we dichotomized the Milestone level, AttrakDiff2 questionnaire scales, cognitive load variables, and HRV indexes. Figure 6 demonstrates the associations between variables of interest.

This study’s main findings highlighted the complexities of cognitive load estimations and HRV indexes inherent in developing and evaluating 360° VR and 2D video learning. Our results showed that video learning resulted in a higher total cognitive load and mental demand and prolonged the reaction time at the 10th minute; however, video learning also enhanced a Milestone level of H&P skills with positive learning experiences. Using the same head-mounted display, both the 360° VR and 2D videos were efficient learning methods with positive learning experiences for novice learners. Notably, the 360° VR video learners demonstrated a higher Milestone level of H&P in actual patients with SDB than the 2D video learners. Although both video modules produced comparable total cognitive load and subscales, the 360° VR video learners had a more prolonged reaction time at the 10th minute than the 2D video learners.

Interestingly, in the second portion of the video, the 360° VR video learners had a longer reaction time than the baseline data. These findings suggested that the immersive 360° VR video could reduce secondary-task performance without increasing subjective cognitive loads. Furthermore, the elevated LF/HF ratio at 0 to 5 minutes correlated with a reduced hedonic stimulation scale. Combined with qualitative feedback, we found that the students seemed to consume more cognitive resources to fit the immersive 360° VR video than the 2D video learning. This preliminary study suggests that the immersive 360° VR video might lead to a better Milestone level than the 2D video, which seemed to be unrelated to cognitive load estimates and HRV indexes in novice learners.

This study used quantitative and qualitative measures and methods to evaluate different video learning materials and showed its potential clinical applicability for learning H&P skills. However, several limitations should be addressed. First, this study needs external validation due to the small sample size. Although H&P is one of the essential training topics for undergraduate medical students, our outcomes may not be generalizable to junior residents who already have basic knowledge, skills, and attitudes. Second, assessments of the learner’s performance in the workplace, such as the Mini Clinical Evaluation Exercise, were not included in this study. Third, there were several significant associations between variables of interest in this study. However, the potential relationships between video module, cognitive load, and HRV might be underestimated using such stringent criteria for multiple tests [73]. Accordingly, further studies with a larger sample size with the extended competence spectrum and learning outcome assessments are warranted to confirm our results and inferences.

Our results indicate that both 360° VR and 2D video learning modules can increase H&P competency. Traditionally, students learn clinical skills, such as H&P, from clinical teachers. Instructional videos offer an excellent pedagogical approach to enhance medical students’ clinical competencies and self-confidence levels [74,75]. Moreover, using videos as a delivery format can improve the effects and attractiveness of text-based e-learning [76]. Therefore, clinical teachers can use videos as learning resources for students’ independent learning [77]. Recently, many universities and teaching institutes have promoted the use of instructional videos as a resource for self-directed education and student-centered programs, especially during the COVID-19 pandemic.

Immersive 360° VR videos allow for a virtual experience where learners can improve their competencies by experiencing all of the sights and sounds of teaching clinics and responses from the patients, families, and medical staff from a first-person perspective. When working memory and information processing capacities are limited, 360° VR video learning can trigger a higher cognitive load or stress than conventional 2D video learning. In this study, 360° VR video learning might reduce participants’ attention toward the secondary task without changing subjective cognitive load estimates and HRV indexes regardless of the learning outcome. Therefore, the quality of 360° VR video learning modules still needs to be improved concerning reducing objective cognitive load.

Approximately 25% of our volunteers reported high vibration, out-of-focus blur, eye fatigue, and motion sickness in the preparation phase of the 360° VR video production. A literature review revealed that immersive 360° VR video might induce 3D visual fatigue [78] and visually induced motion sickness [79,80] and may interrupt autonomic balance [81]. Therefore, we used spherical stabilization to smooth quick movements and vibrations and dynamic depth of field to reduce visual discomfort [82]. In addition, we chose a 10-minute instructional course to reduce these known side effects. Although levels of cardiac autonomic balance, as reflected by the LF/HF ratio, were similar during the 360° VR video learning and equivalent to those of the 2D video learning, our 360° VR video learners still experienced a more prolonged secondary-task reaction time, mainly in the second portion of the video.

In contrast, the 2D video learners did not report these problems while using the same head-mounted display. Although the participants might have an adequate cognitive function to overcome the impaired attention for obtaining the H&P competence, the instructors need to design a 360° VR video with less distraction and lower cognitive load. A motion capture unit to synchronize vestibular response and visual information may reduce motion sickness [83], and further studies are warranted to investigate this issue.

Interestingly, the elevated LF/HF ratio at 5 to 10 minutes seemed to be associated with lower hedonic stimulation. Since these cognitive load estimates were not related to Milestone level, they could be used to improve the quality of video learning. For example, the virtual environment sustained the increased LF/HF ratio and acted as a stimulatory driver for cardiac autonomic activity after mental stress [84]. Therefore, limited time to immerse into the virtual environment could improve the learners’ experience.

This study applied the adaptive EMD method and the well-behaved Hilbert transform technique to process the nonstationary electrocardiogram signals [63,64]. Importantly, we found that all RR intervals at different time points significantly and positively correlated with the total cognitive load. Furthermore, the significant association between the RR at 5 to 10 minutes and the Paas-CLS score was independent of the video module. Using similar analytic technology, Ghaderyan and Abbasi also observed that an increased RR interval was associated with higher cognitive load estimations [85]. Moreover, prolonged reaction time at the 5th minute seemed to be related to the RR at 5 to 10 minutes. These observations reflected that novice learners need to improve their cognitive efficiency when learning from dynamically filmed video from a first-person perspective for dealing with elevated cognitive load. These results also indicated that subjective and objective estimates of cognitive load and HRV monitoring have considerable potential to help us better understand a learner’s processing strategy to video learning modules, enhance our comprehension of contextual information, and improve video learning. Future studies are warranted to confirm the benefits of cognitive load and HRV measures in video learning.

Our preliminary results suggested that 360° VR video learning may result in a better H&P performance despite causing a more prolonged secondary-task reaction time than 2D video learning. Furthermore, the increased LF/HF ratio was associated with lower acceptance of video learning. Qualitative evaluation reflected varying benefits and faults between the 360° VR and 2D videos. Consequently, the students seemed to consume more cognitive resources to fit the immersive 360° VR video than 2D video learning. Without cognitive overload, the undergraduate medical students could adjust their mental efficiency to handle the decreased cognitive reserve when learning through the 360° VR video module. This study indicates that the ubiquitous and diverse roles of multimodal cognitive load estimations and HRV measures in video learning may function as part of an integrated CBME curriculum. Further research is necessary to assess their effects on other performance outcomes.