For this experiment, we chose a classic surgical OSCE station from trauma surgery. It presents patients with a trauma diagnosis (fracture of distal radius) in different clinical cases.

The underlying disease was added to the ALICE simulator framework as a blueprint. The creation of medical content and programming of additional ALICE features were completed according to the steps of Kern's curriculum planning cycle [5].

The design and technical realization of ALICE have previously been described by our group [6]. In short, ALICE is a Web-based immersive patient simulator which allows students to navigate through a virtual “game like” environment from a first-person perspective (Figure 1).

ALICE simulates a treatment room with a simulated patient. The user can interact with the virtual patient and additional non-player characters such as nurses and other doctors. The simulator starts with a short introduction teaching the user basic simulator controls and usage of the simulator. The user is able to freely interact with the environment and treat the virtual patient. The user is free to choose between all available tests and there is no restriction on specific or medically indicated tests. When the student chooses an examination not medically indicated, the test shows a normal finding. The simulation ends when the student chooses a diagnosis and initiates the necessary treatment. ALICE stores the students’ behavior at the server level, logging students’ decisions.

Participants in the study were 160 medical students. Participation was on a voluntary basis without any financial compensation. The study group was comprised of 100 students and the remaining 60 students were allocated to the control group. The study was approved by the Educational Committee of the Medical Faculty at the University of Cologne. The Institutional Review Board was informed and there were no objections. The impact of ALICE on OSCE performance was tested using a trauma case examined by 100 students. In the first stage of the test, students answered 11 multiple-choice questions (MCQs). They then participated in a real OSCE. The correct results of the MCQs and OSCE were not revealed to the students at this point. In the next stage, the students worked with the simulator (ALICE), after which they repeated the MCQ questionnaire and the OSCE with the same clinical scenario as before (Figure 2). The correct results were then revealed and discussed in a peer-to-peer debriefing and with an experienced doctor.

The control group consisted out of 60 students who prepared for the OSCE without ALICE. All students passed the study completely without any drop outs.

Declarative knowledge was tested by comparison of students’ pre- and postsimulator performance on the MCQs. Process knowledge was tested by comparison of their pre- and postsimulator OSCE performance.

Process knowledge was measured on two different levels: (1) comparison of treatment of virtual patient 1 versus virtual patient 4 in the study group and (2) comparison of pre- and postsimulator OSCE performance. The following parameters were defined: correct diagnosis, correct therapy, and correct workflow in anamnesis and diagnostics. These workflows were designed as a blueprint reflecting the “optimal” workflow suggested by 2 independent senior surgeons.

Validation of ALICE’s new features and curricular content was tested on different levels according to the “Consensus guidelines for validation of virtual reality surgical simulators” (Table 1) [7].

Face validity was determined by descriptive analysis. Students were asked to judge the degree of resemblance between ALICE and the real OSCE situation. Content validity was defined as the degree to which the system covers the subject matter of the real activity. It was examined by comparison of ALICE performance in the fourth case and with OSCE performance of the control group. Impact of previous knowledge as a degree of discrimination between the different experience levels was tested using a subgroup comparison between students up to third study year and students in fourth year or higher. Predictive validity as sign of impact on future performance was tested by comparison between the OSCE and control group.

Student acceptance and their opinion about the effectiveness, applicability, and impact on motivation were determined by means of a questionnaire using a (forced choice) 6-point Likert scale (1=extremely satisfied, 2=very satisfied, 3=somewhat satisfied, 4=somewhat dissatisfied, 5=very dissatisfied, 6=extremely dissatisfied).

Data were analyzed with the SPSS Statistics Version 25 (IBM Corp, Armonk, NY, USA) for Windows (Microsoft Corp, Redmond, WA) and Microsoft Excel Version 2013 for Windows (Microsoft Corp, Redmond, WA). Group comparisons were conducted using t tests and P ⩽.05 was considered statistically significant.

The participants in the study were 160 medical students (31 males, 129 females; mean age 27 (SD 2.5) years, age range 21-34 years) who participated in the study on a voluntary basis with no financial compensation. All students were in the clinical phase of their studies (up to third study year n=57; fourth study year and above n=103) and had passed the first medical exam after 2 years of study. All students were recruited at the University Hospital of Cologne. Students worked on all 4 clinical cases. ALICE was accessible one day before the OSCE.

ALICE’s content was extended by adding diagnostic procedures for trauma patients. Four clinical cases were presented in a firm sequence according to the curricular demands of the specific curricular needs (Table 2). All students worked on all four cases. The framework of medical content was based on the national learning objectives catalogue.

The learning goals were defined and are summarized in Textbox 1.

The simulation itself covers different teaching methods. ALICE starts with a short animation which covers simulator usage. The user is asked to explore the surroundings and familiarize themselves with their avatar in the virtual world. After treatment of the first virtual patient, the user is given structured feedback and a keynote speech from the virtual instructor. These 2 methods allow teaching of both declarative and process knowledge. After finishing the cases, students first have a group discussion with other students after which they are finally debriefed by a real instructor on the educational level of a medical doctor. These two methods consolidate declarative and process knowledge.

Cognitive Engagement was designed according to the ICAP framework (Interactive, Constructive, Active, and Passive) [8]. Working with the simulator puts students in an active learner role and finding the right diagnostic and therapeutic workflow is constructive. Virtual instructors accompanied the students during their cases in the sense of a virtual cognitive apprenticeship. Instructors reduced their support step-by-step according to the number of correct clinical decisions and correct diagnoses made by the students.

The current study reveals that ALICE is a suitable tool for OSCE preparation. It showed that ALICE has validity (content, construct, and predictive) and leads to a positive impact on knowledge gain. It is inevitable that each modification of such a simulator requires internal validity testing as major changes may affect the effect of such a curricular intervention.

ALICE was designed as a low-cost 3D immersive framework which makes it possible to add different clinical cases in a Web-based scenario with learning that is not time or location dependent. This enables learning at the individual’s own pace and with the number of repetitions appropriate for them. Differences in knowledge levels can be evened out and future performance can potentially be raised. However, IPS can only be used as a support for a real OSCE training in terms of a blended learning concept since the degree of reality created by these simulators nowadays is not yet of suitable quality.

Moreover, IPS are limited to simulating clinical decision-making based on defined guidelines or clinical blueprints. In this current feasibility study, we used a comparatively simple OSCE station, namely interpretation of X-rays and a limited number of possible clinical decisions enables a steep learning curve [9]. More complex scenarios, however, are technically possible [10] but their impact on OSCE performance has not yet been proven for these modules. Furthermore, these educational tools are often used as an additional tool for preparation. A study that compares students who used ALICE exclusively for preparation of a complex case and students who prepared with conventional learning methods is desirable and part of future studies. However, the current study reflects reality, as most learners do not rely on only one knowledge source when preparing for an examination.

ALICE shows a positive impact on learners’ motivation and both their declarative and procedural knowledge. However, these findings may be influenced by the fact that participation in this study was on a voluntary basis which is known to possibly bias the result as motivated students more often agree to participation than less motivated students [11]. It is important to note that learners have different learning styles and thus not all learners are equally responsive to this educational tool [12]. Hence, the results cannot be generalized for all students.

Although ALICE stored user data, there are little information about the time spent on the simulator and the number of repetitions. ALICE logged the decisions and corresponding time points, however there are no information about students’ behavior like breaks, restarts, or distractions. There was no “full logging” of students’ behavior such as “time logged on,” IP address etc as these program algorithms would require routines similar to spyware, and such implementation was out of the question. Therefore, the influence of number of repetitions and time spent on ALICE cannot be answered in this study.

Embedding a simulator such as ALICE in an educational environment requires thorough curricular planning and subsequent implementation of the curriculum into the simulator. This initial investment pays off once the simulator can be used as an alternative to established teaching methods with high running costs. The current student generation puts high requirements on the quality and motivation of their education [13]. Motivation is known to have a strong effect on knowledge gain [14]. ALICE supports motivation by offering exploration of an immersive world with freedom of choices and treatment of the user’s own virtual patients [15]. These factors promote experience of autonomy and competency which can directly affect intrinsic motivation [16]. Extrinsic motivation was enhanced by applying features that are commonly used in video games such as reward systems (badges, points, rank), continuous feedback, and leader boards for students [17]. The use of IPS for psychomotor or communicative learning goals is already described in literature and its effect on knowledge gain in these settings is still not proven. Adding these competences to the ALICE framework is part of future studies.

ALICE was designed to cover all qualities of the ICAP framework: Interactive, Constructive, Active and Passive. Hence ALICE promotes not only knowledge gain but also influences students’ intrinsic motivation [18]. However, ALICE is most effective when used in a blended learning context as the group discussion is highly interactive and this is thought to have a positive effect on learners’ performance [19].

ALICE is a valuable tool for teaching declarative and process knowledge in a Web-based setting. It shows positive impact on knowledge gain and student motivation and therefore enriches the toolbox of didactic methods for OSCE preparation.