This paper is in the following e-collection/theme issue:

Viewpoints and Personal Experiences on Gaming and Games (13) Serious Games for Health and Medicine (683) Extended Reality, Virtual Reality and Virtual Worlds (560) Digital Health, Telehealth and e-Innovation in Clinical Settings (401) Clinical Integration of XR and Spatial Computing (32) Viewpoints on XR and Spatial Computing (7)

Published on in Vol 12 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/52785, first published .
Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

Authors of this article:

Jan Egger1, 2, 3 Author Orcid Image ;   Christina Gsaxner1, 4, 5, 6 Author Orcid Image ;   Gijs Luijten1, 6 Author Orcid Image ;   Jianxu Chen7 Author Orcid Image ;   Xiaojun Chen8, 9 Author Orcid Image ;   Jiang Bian10 Author Orcid Image ;   Jens Kleesiek1, 3, 11, 12 Author Orcid Image ;   Behrus Puladi4, 5 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    Viewpoint

    1Institute for Artificial Intelligence in Medicine, Essen University Hospital (AöR), Essen, Germany

    2Center for Virtual and Extended Reality in Medicine (ZvRM), Essen University Hospital (AöR), Essen, Germany

    3Cancer Research Center Cologne Essen (CCCE), University Medicine Essen (AöR), Essen, Germany

    4Department of Oral and Maxillofacial Surgery & Institute of Medical Informatics, University Hospital RWTH Aachen, Aachen, Germany

    5Institute of Medical Informatics, University Hospital RWTH Aachen, Aachen, Germany

    6Institute of Computer Graphics and Vision, Graz University of Technology, Graz, Austria

    7Leibniz-Institut für Analytische Wissenschaften (ISAS), Dortmund, Germany

    8Institute of Biomedical Manufacturing and Life Quality Engineering, State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China

    9Institute of Medical Robotic, Shanghai Jiao Tong University, Shanghai, China

    10Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, United States

    11German Cancer Consortium (DKTK), Partner Site Essen, Essen, Germany

    12Department of Physics, TU Dortmund University, Dortmund, Germany

    Corresponding Author:

    Behrus Puladi, MD, DMD

    Department of Oral and Maxillofacial Surgery & Institute of Medical Informatics

    University Hospital RWTH Aachen

    Pauwelsstraße 30

    Aachen, 52074

    Germany

    Phone: 49 241 80 88231

    Fax:49 241 80 82430

    Email: bpuladi@ukaachen.de


    At the Worldwide Developers Conference in June 2023, Apple introduced the Vision Pro. The Apple Vision Pro (AVP) is a mixed reality headset; more specifically, it is a virtual reality device with an additional video see-through capability. The video see-through capability turns the AVP into an augmented reality (AR) device. The AR feature is enabled by streaming the real world via cameras on the (virtual reality) screens in front of the user’s eyes. This is, of course, not unique and is similar to other devices, such as the Varjo XR-3 (Varjo Technologies Oy). Nevertheless, the AVP has some interesting features, such as an inside-out screen that can show the headset wearer’s eyes to “outsiders,” and a button on the top, called the “digital crown,” that allows a seamless blend of digital content with the user’s physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to “The Ultimate Display,” which Ivan Sutherland had already sketched in 1965. After a great response from the media and social networks to the release, we were able to test and review the new AVP ourselves in March 2024. Including an expert survey with 13 of our colleagues after testing the AVP in our institute, this Viewpoint explores whether the AVP can overcome clinical challenges that AR especially still faces in the medical domain; we also go beyond this and discuss whether the AVP could support clinicians in essential tasks to allow them to spend more time with their patients.

    JMIR Serious Games 2024;12:e52785

    doi:10.2196/52785

    Keywords

    Apple Vision Pro; mixed reality; augmented reality; virtual reality; health care; mobile phone 


    In his highly influential paper “The Ultimate Display” [1], Ivan Sutherland, the pioneer of computer graphics and creator of the first augmented reality (AR) head-mounted display (HMD) in the early 1960s, predating the advent of PCs, described aspects that Apple’s Vision Pro aims to implement. The Apple Vision Pro (AVP) can span the wide range of the reality-virtuality continuum, as explained by Milgram et al [2], starting from unaltered reality, achieved by streaming the real world without additional information to the user; progressing through mixed reality (MR) [3], involving the integration of digital information into the real world (referred to as AR) and vice versa (augmented virtuality); and extending to complete immersion, also known as virtual reality (VR), where the user experiences only computer-generated virtual content.

    In the realm of VR, there is currently a broader selection of headsets at various price points, including offerings from Meta, Vive, Varjo, PlayStation, and Google (with the most economical probably being the now-discontinued Google Cardboard or a similar variant from another vendor), compared to AR. AR faces a distinct challenge because of the need to analyze reality to augment it at the right position with digital content for the user. An exception is AR used for a pure simulation, where the AR hologram is indeed shown in the real world but has no meaningful relation to or interactions with real-world objects. An example of such a scenario is a surgical simulation [4]. The AR hologram is shown in front of the user to allow them to inspect and interact with it. Still, the underlying hardware and software need to capture and model the real world to anchor the hologram within it so that the hologram does not drift away from the user’s view.

    In a medical scenario, AR can enable equips a physician with x-ray vision into a patient [5]. This capability is a valuable advancement from the current practice of viewing patient information, such as preoperative imaging, on PC monitors, contributing to the “switching focus problem,” where a physician must divide their attention between the patient and the digital information. Hansen et al [6] outlined that such a division leads to an increased mental workload, disorientation, and deteriorated hand-eye coordination. Instead, AR can display patient information directly in the physician’s view of the patient, additionally offering spatial perception even before any incisions are made. A recent systematic review of the AR headset HoloLens (Microsoft Corp) in medicine [7] showed its broad applications and the massive research efforts during the recent years in all these areas (medicine and health care with applications for patients, physicians or surgeons and students, which should ultimately lead to the development of an intelligent health care metaverse [8,9]). Hence, we highlight the user groups and applications targeted by AR in the next sections and discuss how common challenges in AR can be overcome with the AVP’s capabilities. Furthermore, we share our findings about the new AVP gathered through a structured survey among ourselves and colleagues who had the opportunity to try out the AVP.


    Physicians and Health Care Professionals

    Physicians and health care professionals comprise, by far, the largest target group for AR applications [7]. The adoption and use cases of AR within nursing departments are growing, focusing on wound care, information retrieval, injections, and education [10]. In this context, the accuracy of the visualization is not of utmost importance, but having a wearable “sensing” computer can increase time at the bedside through automatic information retrieval and documentation. However, especially for AR-supported image guidance and navigation, such as in surgery, very high accuracy and reliability are needed [11]. Applications for which submillimeter precision is not necessary are, for example, ablations, ventriculostomies [12-16], and certain orthopedic interventions [17]. Here, the HoloLens is, with its inside-out tracking, already a promising tool, but for applications that need, for example, submillimeter precision, it cannot be used reliably yet. An example is the deep brain stimulation procedure used for treating essential tremors and Parkinson disease, where millimeter-to-submillimeter accuracy in deep brain stimulation targeting (an electrode placement inside the brain) can be important [18]. Another study exploring the clinical accuracy of the HoloLens for neuronavigation concluded that it is currently not within clinically acceptable levels [19]. The same holds true for some application scenarios in orthopedic surgery [20], such as screw placement, where there is still room for improvement [21]. This improvement can be achieved through external tracking hardware [22,23], but it may come at the cost of mobility and increased expenses. The AVP is promising due to its superior sensing technology, as discussed in the Accuracy and Reliability subsection. More importantly, the powerful Apple M1 and R1 chipsets can handle the sensor data appropriately to exploit them completely. This technology comes at a high price, but the often-criticized end-user price of US $3499 (without tax) for the AVP does not seem to be an issue for health professionals who want to use it in health care. The price is similar to that of the HoloLens and much cheaper than (a fraction of) those of existing and clinically used conventional medical navigation systems, for example, those from Brainlab (Stefan Vilsmeier) or Medtronic (Earl Bakken and Palmer Hermundslie). In this context, it is important to mention that the use of AR HMDs with direct patient reference, outside of research in the context of feasibility studies, must obtain appropriate regulatory approval, and the AT HMDs must be certified and classified as medical devices. The Magic Leap 2 (Rony Abovitz), for example, received certification for use in an operating room last year (2023). The requirements to be met are still up to debate, for example, when it comes to active or passive cooling. In contrast to the HoloLens 2, both AVP and Magic Leap have active cooling, potentially compromising the sterile airflow. Furthermore, sterilization is a problem because Apple recommends not using isopropyl alcohol or other solvents for cleaning. This should not be an absolute impediment, given the approval of the Magic Leap 2 for use in the operating theater and the use of regular glasses in the operating theater. We do, however, think that the time is coming when clinicians, technicians, and manufacturers should come together to meet binding guidelines.

    Another challenge for clinical adoption is the integration of the device within the digital infrastructure of the hospital, for example, to retrieve data from and transfer data to the electronic health records. Moreover, and especially in a medical context, the display may also require an image focus at surgical table distance [24]. In combination with the outward sensors, this would be ideal for medical scenarios.

    Students

    Students are the second most common intended target group [7] for AR applications, for example, in educational training, such as the HoloPointer, a virtual AR pointer for laparoscopic surgery training [25]. Another example is the use of AR to teach medical students catheter placement [26] and the use of AR for training on central venous access [27]. Notably, the aforementioned partially outward-oriented sensors, unlike those in other headsets, allow for the accurate tracking of all “normal” (hand) movements for interaction, regardless of (head) position. This is also valuable for health care professionals when practicing or working at the patient’s bedside. When combined with speech commands and eye-controlled interactions, physicians can expectantly implement the AVP easily and in a sterile manner.

    However, we found that the effects of HoloLens-based learning compared to those of conventional learning, for example, using cadavers or other computerized methods, seem to be rather small [7]. A reason for this is that the learning tools usually used are relatively simple, conventional 3D models, and more innovative visualizations, including interactive, dynamic content, which cannot be easily delivered through regular computerized methods, have not been explored in depth yet [7]. Here, we believe that the AVP can raise the bar because of its high-resolution visualization (consisting of 2 micro-OLED displays with 23 megapixels) that uses eye-tracking to make the headset also usable as a desktop screen with its fine textual details. In addition, the 3D user interface, using precise and intuitive finger tracking in combination with eye interactions, can be a game changer. Despite these advances, it remains to be seen whether an AR headset has a major advantage over a conventional screen. The cost of the AVP is also a concern, especially for students with a limited financial budget, but the “pro” might indicate that a “regular” version will also be released. For medical simulation training, however, an examination of costs [28] suggests that using the AVP is a financially viable option. The AVP is, however, highly personalized to the individual user to optimize the experience and accuracy. Luckily for the student group, the AVP can be shared, and the interpupillary distance setting (IDP) is adaptable, compromising the experience minimally, but sharing greatly reduces cost for simulation purposes.

    Patients

    Patients are currently the least frequented target group for extended reality (XR) devices [7]. An example is the use of MemHolo, which provides MR experiences for patients with Alzheimer disease [29]. Other examples in this area include the use of the HoloLens as an assistance and monitoring tool for medication adherence [30] and the use of a HoloLens-based system for functional mobility assessment [31]. We expect the advanced eye-tracking to enhance the user experience because visual cues will not merely exist within the user’s field of view but can also be precisely situated in the user’s focal point. However, many interesting assistance and monitoring applications are limited by the restricted possible use time of untethered AR headsets, primarily due to battery life, and this problem seems to exist in the AVP, which has a similar battery life to that of the HoloLens, approximately 2 hours. The only consolation is the “external” battery pack of the AVP, which can be swapped. Finally, the Apple M2 chip and the new R1 chip of the AVP could potentially boost the relatively small number of applications that have been explored so far, for example, by increasing the frames per second, minimizing display latency, and thus reducing motion sickness, especially useful for older patients, who are more susceptible to MR sickness than younger users [32,33].

    Differences Between Target Groups

    An example of different use cases per user group is shown in Textbox 1. The main target group is health care professionals, which can be divided into several subgroups, such as surgeons, physicians, and nurses. Unlike the other groups, price is not a major issue for this group, but the use cases are different from those of the other groups. For nurses, data retrieval, documentation, and guidance without the need for high accuracy are the estimated use cases, freeing up time at the bedside. The AVP’s high data transfer rate and processing power are more important. For a surgeon, however, accuracy is critical for surgical navigation [34]. For the general practitioner, seamless integration and unparalleled view during transport may be the greatest values of the AVP. For other groups, price is an important consideration, with the caveat that it could reduce overall costs for medical simulation training. The average patient is not expected to be able to afford this device on their own. For now, it could be used for therapies in the clinic, where it can be shared among several patients. However, this makes cleaning the device more important than if there is a single user. We expect to see silicon-replaceable parts for the device, which would ultimately benefit all groups. The AVP has similar challenges to other headsets in terms of battery life and motion sickness, which affects all groups, especially nurses during long, hectic shifts with movement and sometimes no breaks. The integration of an external battery makes battery life easy to address in future hardware updates. Regarding motion sickness, we expect Apple to offer a lower resolution for a higher refresh rate in combination with its foveated rendering to combat this. In summary, the biggest differences between target groups are in the need for high accuracy and the willingness to accept the high price.

    Textbox 1. Possible health care scenarios with the Apple Vision Pro.

    Health care professionals

    • Patient record retrieval
    • Automated documentation
    • Guidance in medical procedures and equipment
    • Surgical preoperative planning
    • Surgical navigation
    • Peerless public medical record viewer

    Students

    • Complex anatomy education
    • Complex pathophysiology education
    • Training simulators
    • Automated skill evaluation

    Patients

    • Patient treatment preparation
    • Patient education
    • (Automated) diagnostics
    • Rehabilitation
    • Improved patient well-being

    Despite significant research efforts in medical AR, clinical translation remains limited so far [35]. Several technical challenges require ongoing multidisciplinary efforts among software developers, hardware manufacturers, regulatory entities, and clinical personnel to create safe, effective, and seamlessly integrated AR solutions in health care. Some of these challenges could be addressed by the AVP’s hardware and user interface.

    Accuracy and Reliability

    The most significant challenge in medical AR systems is their accuracy and reliability in registration and calibration, which are critical for aligning virtual objects with real-world elements. Failures in tracking or registration components can misguide physicians, potentially leading to serious medical accidents. Although recent developments in tracking technology are encouraging [36,37], clinical settings pose additional challenges with limited space, unconventional lighting, and dynamic environments. A solution is to combine an AR device with external tracking systems to increase precision [22]. On the downside, such combinations require complicated setups and the calibration of bulky (external) devices, diminishing the simplicity and lightweight nature of purely head-worn systems. The latest sensor technology, combined with state-of-the-art tracking and registration algorithms, could eliminate the need for external systems to meet clinical requirements. We expect the AVP to move the needle in terms of accuracy because of its strong inside-out tracking through 12 built-in cameras and light detection and ranging sensing method, which is the key aspect for increasing the accuracy of AR. We expect that enhanced eye-tracking capabilities and the incorporation of multiple outward-facing sensors will substantially enhance the reliability and stability of user interactions. The increase in both the quantity and range of sensors, coupled with a more potent chipset, is bound to increase spatial resolution and visualization accuracy significantly in comparison to untethered alternatives. Future research should focus on exploiting this improved hardware setup.

    Usability and Human Factors

    In complex clinical environments, a seamless integration of AR systems into existing clinical processes is crucial. Existing AR solutions often add undue complexity and disrupt established workflows, requiring intricate hardware, many manual steps, and elevated technical knowledge and understanding. Health care professionals, who possess varying degrees of technical background, or patients, who represent diverse demographics, cannot be reasonably expected to meet these demands. Hence, AR systems need to be intuitive and user-friendly. Apple products are known for their user-centric design and seamless integration of hardware and software. We expect that the AVP’s user experience, in terms of interface, interaction, and feedback, will be more intuitive for a diverse audience than existing HMDs and VR glasses.

    As a demonstration of this ease of use, the AVP connects automatically to nearby Apple MacBooks, commonly used by physicians, to display the screen and access files in real time without any effort. An obvious use case is secure, peerless access to patient files while using public transportation. This is valuable for working with all medical records. Apple products work seamlessly with each other, but in a clinical setting, not all products may come from Apple or its partners. A combined challenge and focus should be to enable cross-manufacturer integration.

    In addition, as mentioned earlier, more natural and user-guided interactions are possible with the outward and eye-tracking sensors. This will allow developers to create more intuitive applications. The improved hardware should not be underestimated. For example, Gsaxner et al [38] developed stand-alone marker-less image registration, which has potential in many medical applications, but the setup of an additional laptop is cumbersome and another potential risk for data breach. With the AVP, this type of setup is most likely not needed.

    Visualization and Perception

    Current AR devices face limitations in terms of view, resolution, and brightness, impacting visual fidelity [39]. In addition to these hardware constraints, challenges include depth perception and occlusion between virtual and real content [40]. Current optical see-through (OST) displays, such as that of the popular HoloLens, struggle to address these issues adequately, as they cannot modify the reality presented. By contrast, the AVP’s video see-through (VST) system provides significantly more possibilities for seamlessly and adaptively integrating virtual elements with the real environment. Visual accuracy is difficult to achieve and user dependent, but eye-tracking and automatic eye calibration, combined with the high-resolution displays inside the headset, make us believe that the AVP could set new standards in terms of the quality and perception of visualization.

    In summary, hardware limitations are the main bottleneck in developing AR applications suitable for the clinical translation of AR that cannot be addressed by software developers. Another bottleneck is the regulation and list of requirements for 3D smart glasses in surgery. Combining hardware technology and software from different companies, especially for data transfer, is the final major issue, however, but it is not limited to 3D smart glasses and has been a persistent obstacle in health care [41].

    We believe that the AVP can provide significant advancements as the new state-of-the-art device, with much-needed improvements in precision, reliability, usability, workflow, and perception. The combination of AR and VR in an XR device has the potential to transform health care by aiding in diagnostics, improving surgical procedures, facilitating remote patient care, and enhancing medical training. Many other novel applications may emerge because of the advancements in AVP hardware and software platform. Returning to our main question, the AVP has the potential to enhance medical education and therapy, medical record evaluation, data retrieval, and documentation automation while improving patient outcomes and freeing up clinicians to spend more time at the patient’s bedside. Of course, this will require a collaborative effort among engineers, researchers, and clinicians to develop or enhance these applications and to prove cost-effectiveness to ensure clinical adoption.


    Highlighted Features

    Apple announced some interesting (and still unique) features of the AVP. In the upcoming sections, we introduce and discuss these features in the context of medical MR with a focus on AR.

    Digital Crown

    The digital crown is a turning button that allows the user to blend digital content easily and seamlessly with the physical environment surrounding them. The button is named the same as that in the Apple Watch, but it serves a different function in the AVP by providing an easy option to quickly escape VR without the need to remove the headset. Depending on the capabilities and accessibility of developers to exploit the digital crown, it could also enable diminished reality (DR) [3,42]. This feature would allow users, such as physicians, students, and patients, to remove distracting real-world objects or environments, thereby reducing visual overstimulation. Notably, DR may even extend to diminishing or removing the headsets of other users. To elaborate, the AVP can create realistic avatars of its users. When using the FaceTime application, the movement and direction of the eyes, tongue, and facial muscles are correctly tracked and visualized for other users of the AVP inside the same FaceTime application. This could be used to give the wearer of the headset the ability to perform DR, that is, to diminish the headset and in paint the face when communicating with other AVP users in the real world but via the pass-through mode. The idea of DR is not new, but its implementation, especially in medicine, is a relatively novel [43]. The AVP may be the first untethered device capable of providing DR in a medically relevant and accurate context without the need for external hardware.

    Notably, DR needs, in general, a VST capability and cannot be realized by OST displays, such as that in the HoloLens. The medical HMD from VOSTARS has addressed this by combining an OST display, that is, the view through a semitransparent glass, with a VST function [44]. VOSTARS has realized this with a semitransparent glass that can transition into a nontransparent mode. The AVP, by contrast, does not conform to the OST versus VST dichotomy but rather operates on the VST versus VR HMD spectrum, aligning with the reality-virtuality continuum, postulated by Milgram and Kishino in 1994 [45].

    Virtual Eyes

    Spooky at first sight, the outward display reveals a user’s eyes while wearing the AVP. According to Apple, this feature is meant to let others know when the AVP wearer is using apps or is completely immersed in a virtual world. This feature is not needed in OST AR devices, such as the HoloLens, because the headset wearer’s eyes can be seen through the transparent display. This feature is particularly crucial when considering patients who are dealing with serious illnesses. The transparency ensures that medical professionals, while using the device, do not create an additional layer of detachment by being obscured behind a screen, potentially alleviating concerns about the impact on patient-physician interactions.

    3D Camera

    The AVP features a 3D camera, which allows a user to capture spatial photos and spatial videos in 3D. This could hold significant potential for the professional health care sector. The ability to capture spatial photos and videos in 3D could prove immensely valuable for documenting medical interventions. This is certainly also of interest for medical training, allowing residents to review and reenact treatments in 3D from different viewpoints. It could also serve as a tool to uncover malpractice. The 3D capabilities can also provide an alternative or an addition to the 3D documentation of crime and crash scenes, which is common practice during forensic and medicolegal investigations [46].

    Interaction

    According to Apple, a user can interact with virtual content by simply looking at an element, tapping the fingers together to select it, and using the virtual keyboard or dictation to type. There also seems to be a “visual search” function, which may be similar to the “visual look up” feature found on iPhones and iPads. “Visual search” will allow users to interact with items and text for opening webpages and translating into other languages in real time. Interestingly, this was already described by Sutherland [1] as “the language of glances” to interact with a computer, with an example where looking at a corner of a triangle makes it become rounded. Looking something up is definitely of interest, for example, to surgeons who have their hands full with (surgical) instruments or nonverbal users.

    Optic ID

    The AVP is supposed to support the optical identification of a user for authentication, such the iPhone’s face identification. The biometric method is enabled by scanning the iris of the user wearing the headset. This is not unique to the AVP because the HoloLens 2 already supported an iris authentication. However, this is also an important feature for the AVP in health care, where you have sensitive patient data, and unlocking the device and accessing the data should be permitted only for authorized users. Furthermore, it enables hands-free log-in in a potentially sterile environment.

    ZEISS Optical Inserts

    Carl Zeiss AG is working together with Apple to provide precision optics for users who require vision correction. This will improve the visual experience and provide greater comfort, as users do not need to wear their prescription glasses and frames inside the limited space within the AVP. In addition, it may help avoid side effects such as “overheating,” headaches, and light bleeding. The additional lenses can be attached magnetically to the main lens, which makes insertion and replacement easy.

    Head Strap

    The head strap of the AVP seems to be easily exchangeable, and there are already several alternatives (in terms of materials, colors, etc) available on the web. This will ultimately enable more comfort in comparison to the HoloLens and the Varjo XR-3, which are adjustable but still consist of quite stiff (plastic) frames around the head. It will definitely be more comfortable than the Sword of Damocles, developed by Ivan Sutherland and his student Bob Sproull [47], where the user had to be strapped into the AR or VR HMD device, which was suspended from the ceiling because it was so heavy. The head strap also contains the AVP’s speaker, which is placed directly over the user’s ears and is supposed to virtualize surround sound. However, the strap means that all main components are integrated with the visor. The HoloLens 2, for example, distributes the weight more evenly by integrating components in a plastic case located at the rear, including the battery and system on a chip board. This was done by Microsoft based on its experiences with the HoloLens 1 to make the HoloLens 2 more comfortable. On the contrary, the AVP seems to be worn more like a VR device that fits tightly onto the face instead of partly hovering in front of the user’s eyes, such as other AR devices that are more similar to prescription glasses. The AVP has a similar weight range (600-650 g) to other popular AR and VR headsets, even though the battery is not integrated into the device and is separate. In part, this could be attributed to the material, for example, the aluminum alloy frame. Furthermore, in a medical context, it is usually necessary to be able to easily disinfect the equipment used. A special version with a disinfectable head strap could meet this requirement.

    3D Persona Avatars

    During the setup of the AVP, the user’s face is scanned by the headset to create a photorealistic avatar, which is then used by the device’s operating system, called VisionOS. For better fitting purposes, the user’s face can also be scanned with the TrueDepth camera of an iPhone, as seen at a public demo at the Worldwide Developers Conference in June 2023. Furthermore, the user’s ears can be scanned to optimize the speakers that are inside of the head strap because, aside from the shape and size of the head, the ear sizes, positions, and distances can vary a lot between users. All these user-specific options will make the headset more comfortable to wear, and this will finally increase the acceptability of the new device, not only in the health care sector but also in other sectors.

    Unity

    The Unity engine already supports pretty much all common headset devices, whether it be AR, VR, or MR, such as the Oculus (Palmer Luckey), HoloLens, or Google VR (Google LLC). Unity also supports the OpenVR application programming interface that has already been used in medicine [48]. Therefore, the support of the Unity engine of the AVP makes the development and porting of apps much easier. This also makes a comparison between and an evaluation of devices with the “same” app easier. Notably, Unity currently already allows the development of apps for iPhone, iPad, and Mac while deployment occurs via the Apple app store or Xcode, suggesting future development for the AVP. Overall, Unity support will enable faster development of apps, hopefully fast enough to reach a critical mass before end users may turn their backs, which is needed for the long-term success of the new Apple device.

    Medical Device

    Talking about the health care sector, we need to mention that the AVP is not certified as a medical device yet. We are not aware of whether Apple has plans in this direction, at least not publicly. Apple, however, promoted Complete HeartX as an education app for medical students by providing hyperrealistic 3D models and animations of the heart and medical issues. In comparison, Microsoft advertised the HoloLens 2 specifically for medical scenarios, for example, with a plenary presentation by Bernard Kress (who was at that time Principal Optical Architect, HoloLens team, Microsoft Corp) at the International Society for Optics and Photonics Medical Imaging in February 2020. Nevertheless, we are sure that the AVP will be used by researchers and companies for medical applications, similar to other devices, if Apple does not explicitly prohibit this somehow, for example, by monitoring their devices and locking them. The integration of the AVP with other software and hardware commonly used in clinical settings, for example, integration with electronic health records, will be interesting [49,50].

    First Experience

    In March 2024, we had the opportunity to try out the AVP ourselves for the first time (Figure 1). We, therefore, conducted a standardized expert survey among colleagues (and ourselves) at our institute (the Institute for Artificial Intelligence in Medicine). The inclusion criterion was an experience of at least 10 minutes with the AVP. After obtaining written consent, we conducted a structured survey on the AVP. We collected baseline data from the respondents including experience with HMDs (Table 1), as well as asked Likert questions on how far the AVP corresponds to an “Ultimate Display” (Table 2). We also administered the User Experience Questionnaire (UEQ; Figure 2) as well as asked open-ended questions about what is missing from an “Ultimate Display” (Textbox 2).

    ‎
    Figure 1. The illustration of a possible Apple Vision Pro scenario with one of the authors.
    Table 1. Characteristics of the experts interviewed (N=13).
    ParameterValue
    Sex

    		Female3 (23)

    		Male10 (77)
    Age (y), mean (SD; range)30.9 (6.5; 23-48)
    Work experience (y), mean (SD; range)4.3 (5.6; 0-20)
    Previous experience with HMDsa

    		Yes9 (69)

    		No4 (31)
    Previous experience with HMDs (hours)

    		Mean (SD)213.3 (400.8)

    		Median (range)1.0 (0-1000)

    aHMD: head-mounted display.

    Table 2. Likert questionnaire (N=13).

    		Likert item (1=disagree; 7=agree)Value, mean (SD)
    1.The Apple Vision Pro comes very close to an “Ultimate Display” in terms of realism.5.5 (1.3)
    2.The Apple Vision Pro comes very close to an “Ultimate Display” in terms of interactivity.5.0 (1.1)
    3.The Apple Vision Pro is more comfortable than other HMDsa.4.9 (1.5)
    4.The Apple Vision Pro is more user-friendly than other HMDs.5.1 (1.9)
    5.The Apple Vision Pro surpasses other HMDs in immersion and sense of presence.5.4 (1.6)
    6.The Apple Vision Pro represents a significant technological advance.5.7 (1.4)

    aHMD: head-mounted display.

    ‎
    Figure 2. The results (means and SDs) of the User Experience Questionnaire (UEQ) regarding the attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty of the Apple Vision Pro compared to those of a benchmark (bar plot in the background).
    Textbox 2. Open questionnaire summary.

    Negatives

    • Suboptimal weight distribution
    • Comparatively heavy among head-mounted display (HMDs)
    • Detachable light seal cushion poses potential damage risks and necessitates screen touching before and after use
    • Requirement for specialized prescription glasses
    • Short battery life (2 hours under normal use)
    • Forced interaction method with eyes and pinch
    • Inconsistent functionality of eye navigation near the borders of the field of view
    • The absence of intelligent automatic transition between real and virtual worlds via the crown button
    • Limited field of view compared to other HMDs
    • Blurry resolution of real-world pass-through, hindering text legibility
    • Clear differentiation between virtual and real environmental components by users

    Positives

    • Lightweight compared to the currently used hardware and materials
    • Natural interaction via outward sensors
    • Effective eye-tracking for seamless user interaction
    • Rapid and simple adjustment of glasses for user interpupillary distance
    • Innovative crown button design with wheel and push functionalities
    • Smooth transition between real and virtual worlds facilitated by the crown button
    • High-resolution internal display for realistic virtual objects
    • High-quality spatial sound experience
    • Seamless visualization of the real world
    • Minimal app start-up and file loading times
    • Autofocus and person blending and unblending for safety precautions
    • Simple and fast to learn

    Responses to the Likert questions on whether the AVP comes close to an “Ultimate Display” comprised values between 4.9 and 5.7 on a scale of 1 (disagree) to 7 (agree). The UEQ showed good mean results for attractiveness (1.77, SD 0.65), efficiency (1.58, SD 0.70), stimulation (1.56, SD 1.33), and novelty (1.29, SD 1.11) but above average mean result for perspicuity (1.54, SD 0.68) and below average mean result for dependability (1.10, SD 0.46) compared to a benchmark. This fits in with the general perception (Textbox 2) that the AVP is a significant improvement over other HMDs but still behind the “Ultimate Display.”


    To summarize, we see Apple entering the MR headset market with its own device very positively. Another company’s contribution can benefit the technical progress in this area, not only from a hardware perspective but also from a software perspective, by enabling new user input concepts and apps. With the current hardware specifications and Apple’s history of delivering cutting-edge devices and beyond, there is a very good chance of another AR “evolution” or even “revolution” (Table 3). This is a very important aspect because (1) AR devices are sparse in comparison to VR devices, and (2) the product life cycle to a possible successor is in the range of years, which feels like ages when compared to the current pace of technological developments, especially in (medical) computer science [51]. For example, the first version of the HoloLens was introduced by Microsoft in 2016, and it took Microsoft >3 years to introduce its successor, the HoloLens 2 (released in limited numbers on November 7, 2019).

    Table 3. Side-by-side comparison of the specifications of several existing mixed reality headsets with those of the upcoming Apple Vision Pro [52].
    ParameterApple Vision ProPico 4Meta Quest 3HTC Vive XR EliteVarjo XR-3HoloLens 2Magic Leap 2
    ARaVSTbVSTVSTVSTVSTOSTcOST
    VRdYesYesYesYesYesNoNo
    Resolution (per eye)3800×30002160×21602064×22081920×19201920×1920 focus and 2880 ×2720 peripheral2048×10801440×1760
    Refresh rate (Hz)10090120909012060
    Visible field of viewEstimate 100 horizontal104 horizontal and 103 vertical110 horizontal and 96 vertical102.13 horizontal and 91,27 vertical115 horizontal and 90 vertical43 horizontal and 29 vertical44 horizontal and 53 vertical
    CameraDual 6.5 MPe pass-through cameras16 MP RGBf cameraDual 18 PPDg color pass-through cameras16 MP RGB cameraDual 12 MP 90 Hz camerasNative pass-through 8 MP camera 30 HzNative pass-through 12.6 MP 60 fps RGB camera
    Tracking type6 DoFh inside-out tracking via 12 integrated cameras; includes depth sensor and LIDARi6 DoF inside-out tracking6 DoF inside-out tracking via 4 integrated cameras; includes depth sensor6 DoF inside-out tracking via 4 integrated cameras; includes depth sensor6 DoF marker-based tracking and non–marker-based tracking; includes LiDAR sensor6 DoF inside-out tracking via 4 integrated cameras6 DoF inside-out tracking
    TrackingEyes, face, and handsHandsHands, upper body, and leg positionHandsEyes and handsEyes and handsEyes and hands
    Depth sensingLiDAR+TrueDepthDepth cameraDepth cameraDepth cameraLiDAR+RGB fusionToFj depthiToFk depth
    CPUlApple M2 and Apple R1Qualcomm Snapdragon XR2Qualcomm Snapdragon XR2 Gen 2Qualcomm Snapdragon XR2 Gen 2NoneQualcomm Snapdragon 850Quad-core Zen2
    GPUmApple M2 10-core GPUAdreno 650Adreno 740Adreno 650NoneAdreno 630AMD GFX10.2
    RAMUp to 16 GB8 GB8GB12 GBNone4 GB16 GB
    Storage64 GB128 or 256 GB128, 256, and 512 GB128 GBNone64 GB256 GB
    OSnvisionOSPICO OS 5.0 (Android)AndroidAndroidNoneUniversal Windows PlatformLumin OS
    SDKovisionOSPICO Unity Integration SDK and OpenXRMeta XR All-in-One SDK and OpenXRWave andOpenXRVarjo Native and OpenXRWindows Mixed Reality platform and OpenXRMagic Leap and OpenXR
    Battery life2 hours3 hours2 hours2 hoursWired3 hours3.5 hours
    Weight600 g + headband 50 g + external battery 353 g295 g + headband 291 g509 g625 g594 g + headband 386 g566 g260 g
    UntetheredYesYesYesNoNoYesYes
    Price (US $)34994304991100650035003299

    aAR: augmented reality.

    bVST: video see-through.

    cOST: optical see-through.

    dVR: virtual reality.

    eMP: megapixel.

    fRGB: red, green, blue.

    gPPD: peak pixel density.

    hDoF: degrees of freedom.

    iLIDAR: light detection and ranging.

    jToF: time-of-flight.

    kiToF: indirect time-of-flight.

    lCPU: central processing unit.

    mGPU: graphics processing unit.

    nOS: operating system.

    oSDK: software development kit.

    Steve Jobs released the iPhone in 2007, creating a new category of smartphones. What made it special was that Apple combined existing technologies and software in a manner that made previously unexpected things possible, which Apple repeated with the iPad in 2010 and then with the Apple Watch in 2015. Hence, the AVP could lead to the breakthrough of not only AR, VR, or MR but also DR (because it is a pass-through technique) due to its enormous technological progress. Moreover, technological progress can not only enable more precise, patient-specific treatments but also make these treatments more efficient, for example, through hands-free, instant authentications; documentation; and the visualization of search queries and their results directly within the field of vision of the HMD user. Further, well-engineered AR can help overcome communication barriers in a face-to-face manner. Thus, it supports clinicians in a range of essential tasks and reduces the administrative burden on clinicians to allow them to spend more time with their patients, which is a key challenge for upcoming artificial intelligence (AI)-based foundation models [53].

    Moreover, the recent rise of large language models [54], such as ChatGPT, cannot be ignored in the context of the upcoming AVP because it changed how we communicate with computers, and the AVP can certainly be enhanced with the integration of a ChatGPT-like bot. Even though ChatGPT is still in its infancy, when it comes to hard facts, especially in health care [55,56], it already enables a fluent, partly fact-based, conversation. This will, in the short or long run, also completely enter the headset market, where someone can discuss upcoming treatment steps on a humanlike level based on a trained large language model. However, we do not think that this will be specific to the AVP, and with ChatGPT, Microsoft seems to currently have the edge.

    Another application is the metaverse, which integrates physical reality and VR [8]. The idea is that users and their avatars interact in an environment with access to an unlimited amount of health data. Metaverse applications focus primarily on AI-based medical practice and medical image–guided diagnosis and therapy [8]. The AVP could make an important contribution as a hardware enabler that allows users to move and interact in the metaverse in a more intuitive manner.

    Nevertheless, digitization holds promise for making medical care more efficient. But will screens come between doctors and patients [57]? At the same time, for every hour doctors spend with their patients, they spend 2 hours in front of the computer [58]. The AVP’s untethered, external battery, and spooky-eye design with improved sensor and computing hardware has the potential to help health care professionals improve efficiency and patient outcomes, allowing more time for patient care. The improved hardware should allow the levels of accuracy and computing needed for the device to truly function as a stand-alone device for surgery or preoperative planning. Integration with patient records combined with AI allows for patient recognition and provides the health care professional with accurate data retrieval and documentation. The provision of guidance for devices and medical procedures can occur naturally through its outward-facing sensors, making reading books a thing of the past. The VST design opens up futuristic possibilities for DR, while the ghostly eyes maintain a sense of patient contact. We recognize that this is an optimistic outlook and will require the combined efforts of experts in the field.

    Whether the AVP will blur the line between patient encounters and computer time also depends on patient and physician acceptance. In ophthalmology, the AVP is seen as a breathtaking application [59]. However, this must first be confirmed in clinical trials with rigorous scientific methodology and hard clinical end points. When the AVP is used for patient scenarios, the focus must be on the patient, not the technology, which is always a vehicle for existing problems.


    To sum up, the AVP enriches the current headset market and provides another alternative, especially in the limited AR segment. The AVP will likely be at the technological forefront in this sector, which puts pressure on other vendors to compete by drawing level with and surpassing Apple. The price is certainly a burden for entertainment consumers, but for professionals, especially in the health care sector, with the Microsoft HoloLens being in the same price range and the Varjo XR-3 costing twice as much, it is not a deal-breaker. This becomes even more irrelevant if seen in relation to the other hardware costs in health care, such as those of navigation systems and imaging equipment (computed tomography and magnetic resonance imaging scanners), which can reach millions. In Sutherland’s [1] sense and our experience, it is not yet the Wonderland that Alice or the doctors (added by us) walked into, but perhaps a bit closer.

    Acknowledgments

    This work was supported by the REACT-EU project KITE (Plattform für KI-Translation Essen; EFRE-0801977), FWF enFaced 2.0 (KLI 1044), National Institutes of Health (NIH) and National Institute of Allergy and Infectious Diseases (NIAID; R01AI172875), and NIH and NCATS (UL1 TR001427). BP was funded by the Medical Faculty of the RWTH Aachen University in Germany as part of the Clinician Scientist Program. CG was funded by the Advanced Research Opportunities Program from the RWTH Aachen University. JC was funded by the Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung [BMBF]) in Germany under the funding reference 161L0272 and supported by the Ministry of Culture and Science of the State of North Rhine-Westphalia (Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen [MKW NRW]). Furthermore, the authors acknowledge the Center for Virtual and Extended Reality in Medicine (ZvRM) of the University Hospital in Essen, Germany. Finally, the authors want to thank ZhaoDi Deng, Ahmad Idrissi-Yaghir, and Rainer Röhrig. Parts of Figure 1 were created using the generative artificial intelligence tool DALL·E 3 by OpenAI [60].

    Conflicts of Interest

    BP is an associate editor of the Journal of Medical Internet Research. All other authors declare no other conflicts of interest.

    1. Sutherland IE. The ultimate display. In: Proceedings of the 3rd IFIP Congress 1965. 1965. Presented at: IFIP Congress 1965; May 24-29, 1965; New York, NY.
    2. Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: a class of displays on the reality-virtuality continuum. In: Proceedings of SPIE's 1995 Symposium on Microlithography. 1995. Presented at: SPIE 1995; February 19-24, 1995; Santa Clara, CA. [CrossRef]
    3. Egger J, Gsaxner C, Kleesiek J, Puladi B. What is diminished virtuality? A directional and layer-based taxonomy for the reality-virtuality continuum. JMIR XR Spatial Comput. Jan 31, 2024;1:e52904. [CrossRef]
    4. Lungu AJ, Swinkels W, Claesen L, Tu P, Egger J, Chen X. A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery. Expert Rev Med Devices. Jan 16, 2021;18(1):47-62. [CrossRef] [Medline]
    5. Livingston MA, Dey A, Sandor C, Thomas BH. Pursuit of “X-ray vision” for augmented reality. In: Huang W, Alem L, Livingston M, editors. Human Factors in Augmented Reality Environments. New York, NY. Springer; 2013.
    6. Hansen C, Black D, Lange C, Rieber F, Lamadé W, Donati M, et al. Auditory support for resection guidance in navigated liver surgery. Int J Med Robot. Mar 28, 2013;9(1):36-43. [CrossRef] [Medline]
    7. Gsaxner C, Li J, Pepe A, Jin Y, Kleesiek J, Schmalstieg D, et al. The HoloLens in medicine: a systematic review and taxonomy. Med Image Anal. Apr 2023;85:102757. [FREE Full text] [CrossRef] [Medline]
    8. Wang G, Badal A, Jia X, Maltz JS, Mueller K, Myers KJ, et al. Development of metaverse for intelligent healthcare. Nat Mach Intell. Nov 15, 2022;4(11):922-929. [FREE Full text] [CrossRef] [Medline]
    9. Randazzo G, Reitano G, Carletti F, Iafrate M, Betto G, Novara G, et al. Urology: a trip into metaverse. World J Urol. Oct 2023;41(10):2647-2657. [FREE Full text] [CrossRef] [Medline]
    10. Wüller H, Behrens J, Garthaus M, Marquard S, Remmers H. A scoping review of augmented reality in nursing. BMC Nurs. May 16, 2019;18(1):19. [FREE Full text] [CrossRef] [Medline]
    11. Gsaxner C, Li J, Pepe A, Schmalstieg D, Egger J. Inside-out instrument tracking for surgical navigation in augmented reality. In: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. 2021. Presented at: VRST '21; December 8-10, 2021; Osaka, Japan. [CrossRef]
    12. Ferraguti F, Minelli M, Farsoni S, Bazzani S, Bonfe M, Vandanjon A, et al. Augmented reality and robotic-assistance for percutaneous nephrolithotomy. IEEE Robot Autom Lett. Jul 2020;5(3):4556-4563. [CrossRef]
    13. Li Y, Chen X, Wang N, Zhang W, Li D, Zhang L, et al. A wearable mixed-reality holographic computer for guiding external ventricular drain insertion at the bedside. J Neurosurg. Nov 01, 2019;131(5):1599-1606. [CrossRef] [Medline]
    14. Azimi E, Niu Z, Stiber M, Greene N, Liu R, Molina C, et al. An interactive mixed reality platform for bedside surgical procedures. In: Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention. 2020. Presented at: MICCAI 2020; October 4-8, 2020; Lima, Peru. [CrossRef]
    15. Van Gestel F, Frantz T, Vannerom C, Verhellen A, Gallagher AG, Elprama SA, et al. The effect of augmented reality on the accuracy and learning curve of external ventricular drain placement. Neurosurg Focus. Aug 2021;51(2):E8. [CrossRef] [Medline]
    16. Schneider M, Kunz C, Pal'a A, Wirtz CR, Mathis-Ullrich F, Hlaváč M. Augmented reality-assisted ventriculostomy. Neurosurg Focus. Jan 2021;50(1):E16. [CrossRef] [Medline]
    17. Dennler C, Bauer DE, Scheibler AG, Spirig J, Götschi T, Fürnstahl P, et al. Augmented reality in the operating room: a clinical feasibility study. BMC Musculoskelet Disord. May 18, 2021;22(1):451. [FREE Full text] [CrossRef] [Medline]
    18. Frey J, Cagle J, Johnson KA, Wong JK, Hilliard JD, Butson CR, et al. Past, present, and future of deep brain stimulation: hardware, software, imaging, physiology and novel approaches. Front Neurol. Mar 9, 2022;13:825178. [FREE Full text] [CrossRef] [Medline]
    19. van Doormaal TP, van Doormaal JA, Mensink T. Clinical accuracy of holographic navigation using point-based registration on augmented-reality glasses. Oper Neurosurg (Hagerstown). Dec 01, 2019;17(6):588-593. [FREE Full text] [CrossRef] [Medline]
    20. Jud L, Fotouhi J, Andronic O, Aichmair A, Osgood G, Navab N, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. Feb 15, 2020;21(1):103. [FREE Full text] [CrossRef] [Medline]
    21. Elmi-Terander A, Nachabe R, Skulason H, Pedersen K, Söderman M, Racadio J, et al. Feasibility and accuracy of thoracolumbar minimally invasive pedicle screw placement with augmented reality navigation technology. Spine (Phila Pa 1976). Jul 15, 2018;43(14):1018-1023. [FREE Full text] [CrossRef] [Medline]
    22. Li R, Si W, Liao X, Wang Q, Klein R, Heng PA. Mixed reality based respiratory liver tumor puncture navigation. Comput Visual Media. Jan 17, 2020;5(4):363-374. [CrossRef]
    23. Frisk H, Lindqvist E, Persson O, Weinzierl J, Bruetzel LK, Cewe P, et al. Feasibility and accuracy of thoracolumbar pedicle screw placement using an augmented reality head mounted device. Sensors (Basel). Jan 11, 2022;22(2):522. [FREE Full text] [CrossRef] [Medline]
    24. Carbone M, Cutolo F, Condino S, Cercenelli L, D’Amato R, Badiali G, et al. Architecture of a hybrid video/optical see-through head-mounted display-based augmented reality surgical navigation platform. Information. Feb 08, 2022;13(2):81. [CrossRef]
    25. Heinrich F, Huettl F, Schmidt G, Paschold M, Kneist W, Huber T, et al. HoloPointer: a virtual augmented reality pointer for laparoscopic surgery training. Int J Comput Assist Radiol Surg. Jan 23, 2021;16(1):161-168. [FREE Full text] [CrossRef] [Medline]
    26. Schoeb DS, Schwarz J, Hein S, Schlager D, Pohlmann PF, Frankenschmidt A, et al. Mixed reality for teaching catheter placement to medical students: a randomized single-blinded, prospective trial. BMC Med Educ. Dec 16, 2020;20(1):510. [FREE Full text] [CrossRef] [Medline]
    27. Suzuki K, Morita S, Endo K, Yamamoto T, Fujii S, Ohya J, et al. Learning effectiveness of using augmented reality technology in central venous access procedure: an experiment using phantom and head-mounted display. Int J Comput Assist Radiol Surg. Jun 16, 2021;16(6):1069-1074. [CrossRef] [Medline]
    28. Hippe DS, Umoren RA, McGee A, Bucher SL, Bresnahan BW. A targeted systematic review of cost analyses for implementation of simulation-based education in healthcare. SAGE Open Med. Mar 19, 2020;8:2050312120913451. [FREE Full text] [CrossRef] [Medline]
    29. Aruanno B, Garzotto F. MemHolo: mixed reality experiences for subjects with Alzheimer’s disease. Multimed Tools Appl. Jan 5, 2019;78(10):13517-13537. [CrossRef]
    30. Blusi M, Nieves JC. Feasibility and acceptability of smart augmented reality assisting patients with medication pillbox self-management. Stud Health Technol Inform. Aug 21, 2019;264:521-525. [CrossRef] [Medline]
    31. Sun R, Aldunate RG, Sosnoff JJ. The validity of a mixed reality-based automated functional mobility assessment. Sensors (Basel). May 11, 2019;19(9):2183. [FREE Full text] [CrossRef] [Medline]
    32. Oh H, Son W. Cybersickness and its severity arising from virtual reality content: a comprehensive study. Sensors (Basel). Feb 09, 2022;22(4):1314. [FREE Full text] [CrossRef] [Medline]
    33. Li X, Luh DB, Xu RH, An Y. Considering the consequences of cybersickness in immersive virtual reality rehabilitation: a systematic review and meta-analysis. Appl Sci. Apr 20, 2023;13(8):5159. [CrossRef]
    34. Malhotra S, Halabi O, Dakua SP, Padhan J, Paul S, Palliyali W. Augmented reality in surgical navigation: a review of evaluation and validation metrics. Appl Sci. Jan 27, 2023;13(3):1629. [FREE Full text] [CrossRef]
    35. Seibold M, Spirig JM, Esfandiari H, Farshad M, Fürnstahl P. Translation of medical AR research into clinical practice. J Imaging. Feb 14, 2023;9(2):44. [FREE Full text] [CrossRef] [Medline]
    36. Macario Barros A, Michel M, Moline Y, Corre G, Carrel F. A comprehensive survey of visual SLAM algorithms. Robotics. Feb 10, 2022;11(1):24. [CrossRef]
    37. Andrews CM, Henry AB, Soriano IM, Southworth MK, Silva JR. Registration techniques for clinical applications of three-dimensional augmented reality devices. IEEE J Transl Eng Health Med. 2021;9:1-14. [CrossRef]
    38. Gsaxner C, Pepe A, Wallner J, Schmalstieg D, Egger J. Markerless image-to-face registration for untethered augmented reality in head and neck surgery. In: Proceedings of the Medical Image Computing and Computer Assisted Intervention. 2019. Presented at: MICCAI 2019; October 13-17, 2019; Shenzhen, China. [CrossRef]
    39. Xiong J, Hsiang EL, He Z, Zhan T, Wu ST. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light Sci Appl. Oct 25, 2021;10(1):216. [FREE Full text] [CrossRef] [Medline]
    40. Bichlmeier C, Sielhorst T, Heining SM, Navab N. Improving depth perception in medical AR: a virtual vision panel to the inside of the patient. In: Proceedings of the Bildverarbeitung für die Medizin 2007. 2007. Presented at: BVM 2007; March 25-27, 2007; Munich, Germany. [CrossRef]
    41. Innocente C, Ulrich L, Moos S, Vezzetti E. Augmented reality: mapping methods and tools for enhancing the human role in healthcare HMI. Appl Sci. Apr 24, 2022;12(9):4295. [FREE Full text] [CrossRef]
    42. Mori S, Ikeda S, Saito H. A survey of diminished reality: techniques for visually concealing, eliminating, and seeing through real objects. IPSJ T Comput Vis Appl. Jun 28, 2017;9:17. [CrossRef]
    43. Gsaxner C, van Meegdenburg T, Luijten G, Vu VD. DREAMING - diminished reality for emerging applications in medicine through inpainting dataset. Zenodo. Jan 8, 2024. URL: https://zenodo.org/records/10471365 [accessed 2024-09-06]
    44. Cutolo F, Cattari N, Fontana U, Ferrari V. Towards the development of a quasi-orthoscopic hybrid video/optical see-through HMD for manual tasks. In: Proceedings of the Augmented Reality, Virtual Reality, and Computer Graphics. 2019. Presented at: AVR 2019; June 24-27, 2019; Santa Maria al Bagno, Italy. [CrossRef]
    45. Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans Inform Syst. 1994;77:1321-1329.
    46. Kottner S, Thali MJ, Gascho D. Using the iPhone's LiDAR technology to capture 3D forensic data at crime and crash scenes. Forensic Imaging. Mar 2023;32:200535. [CrossRef]
    47. Sproull R, Brock DC. Interview of Ivan Sutherland. IEEE Ann Hist Comput. 2023;45(1):64-76. [CrossRef]
    48. Egger J, Gall M, Wallner J, Boechat P, Hann A, Li X, et al. HTC Vive MeVisLab integration via OpenVR for medical applications. PLoS One. Mar 21, 2017;12(3):e0173972. [FREE Full text] [CrossRef] [Medline]
    49. Boonstra A, Versluis A, Vos JF. Implementing electronic health records in hospitals: a systematic literature review. BMC Health Serv Res. Sep 04, 2014;14(1):370. [FREE Full text] [CrossRef] [Medline]
    50. Jha AK, DesRoches CM, Campbell EG, Donelan K, Rao SR, Ferris TG, et al. Use of electronic health records in U.S. hospitals. N Engl J Med. Apr 16, 2009;360(16):1628-1638. [CrossRef]
    51. Egger J, Gsaxner C, Pepe A, Pomykala KL, Jonske F, Kurz M, et al. Medical deep learning-a systematic meta-review. Comput Methods Programs Biomed. Jun 2022;221:106874. [FREE Full text] [CrossRef] [Medline]
    52. VRcompare. URL: https://vr-compare.com/ [accessed 2024-02-22]
    53. Moor M, Banerjee O, Abad ZS, Krumholz HM, Leskovec J, Topol EJ, et al. Foundation models for generalist medical artificial intelligence. Nature. Apr 12, 2023;616(7956):259-265. [CrossRef] [Medline]
    54. Brants T, Popat AC, Xu P, Och FJ, Dean J. Large language models in machine translation. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2007. Presented at: EMNLP-CoNLL 2007; June 28-30, 2007; Prague, Czech Republic.
    55. Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: a taxonomy and systematic review. Comput Methods Programs Biomed. Mar 2024;245:108013. [FREE Full text] [CrossRef] [Medline]
    56. Barnard F, Van Sittert M, Rambhatla S. Self-diagnosis and large language models: a new front for medical misinformation. arXiv. Preprint posted online July 10, 2023. 2024. [FREE Full text]
    57. Gawande A. Why doctors hate their computers: digitization promises to make medical care easier and more efficient. But are screens coming between doctors and patients? The New Yorker. Nov 5, 2018. URL: https:/​/www.​newyorker.com/​magazine/​2018/​11/​12/​why-doctors-hate-their-computers#:~:text=A%202016%20study%20found%20that,screen%20to%20do%20electronic%20tasks [accessed 2024-09-06]
    58. Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L, et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med. Sep 06, 2016;165(11):753. [CrossRef]
    59. Waisberg E, Ong J, Masalkhi M, Zaman N, Sarker P, Lee AG, et al. The future of ophthalmology and vision science with the Apple Vision Pro. Eye (Lond). Feb 2024;38(2):242-243. [FREE Full text] [CrossRef] [Medline]
    60. DALL·E 3. OpenAI. URL: https://openai.com/index/dall-e-3/ [accessed 2024-09-06]

    ‎
    AI: artificial intelligence
    AR: augmented reality
    AVP: Apple Vision Pro
    DR: diminished reality
    HMD: head-mounted display
    MR: mixed reality
    OST: optical see-through
    VR: virtual reality
    VST: video see-through
    XR: extended reality

    Edited by T Leung, G Eysenbach; submitted 15.09.23; peer-reviewed by M Hirota, N Silva; comments to author 12.11.23; revised version received 26.03.24; accepted 02.07.24; published 18.09.24.

    Copyright

    ©Jan Egger, Christina Gsaxner, Gijs Luijten, Jianxu Chen, Xiaojun Chen, Jiang Bian, Jens Kleesiek, Behrus Puladi. Originally published in JMIR Serious Games (https://games.jmir.org), 18.09.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.