This paper is in the following e-collection/theme issue:

Usability of Games and Gamification (63) Extended Reality, Virtual Reality and Virtual Worlds (509) Usability of Virtual Reality Simulations (30)

Published on in Vol 8, No 4 (2020): Oct-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/18153, first published .
A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality

A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality

A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality

Authors of this article:

Timothy Zhang1, 2 Author Orcid Image ;   Richard Booth3 Author Orcid Image ;   Royce Jean-Louis4 Author Orcid Image ;   Ryan Chan3 Author Orcid Image ;   Anthony Yeung5 Author Orcid Image ;   David Gratzer1, 5 Author Orcid Image ;   Gillian Strudwick1, 4 Author Orcid Image
Article Authors Cited by (22) Tweetations (10) Metrics

    Viewpoint

    1Centre for Addiction and Mental Health, Toronto, ON, Canada

    2Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    3Arthur Labatt Family School of Nursing, Western University, London, ON, Canada

    4Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

    5Department of Psychiatry, University of Toronto, Toronto, ON, Canada

    Corresponding Author:

    Gillian Strudwick, RN, PhD

    Centre for Addiction and Mental Health

    Bell Gateway Building, Room 7343

    1001 Queen St W

    Toronto, ON, M6J 1H4

    Canada

    Phone: 1 4165358501 ext 39333

    Email: gillian.strudwick@camh.ca


    Health-related virtual reality (VR) applications for patient treatment, rehabilitation, and medical professional training are on the rise. However, there is little guidance on how to select and perform usability evaluations for VR health interventions compared to the supports that exist for other digital health technologies. The purpose of this viewpoint paper is to present an introductory summary of various usability testing approaches or methods that can be used for VR applications. Along with an overview of each, a list of resources is provided for readers to obtain additionally relevant information. Six categories of VR usability evaluations are described using a previously developed classification taxonomy specific to VR environments: (1) cognitive or task walkthrough, (2) graphical evaluation, (3) post hoc questionnaires or interviews, (4) physical performance evaluation, (5) user interface evaluation, and (6) heuristic evaluation. Given the growth of VR in health care, rigorous evaluation and usability testing is crucial in the development and implementation of novel VR interventions. The approaches outlined in this paper provide a starting point for conducting usability assessments for health-related VR applications; however, there is a need to also move beyond these to adopt those from the gaming industry, where assessments for both usability and user experience are routinely conducted.

    JMIR Serious Games 2020;8(4):e18153

    doi:10.2196/18153

    Keywords

    virtual reality; simulated environment; usability; evaluation; assessment methods; medical informatics; nursing informatics 


    In the last decade, there has been a tremendous increase in the use of virtual reality (VR) technology in a variety of global contexts, including entertainment (eg, gaming), education, marketing, and design. VR broadly describes digitally created simulations where a person can be immersed in a computer-generated reality and complete tasks or interact with a virtual environment. Equipment such as VR headsets that allow individuals to experience the sounds and sights of a virtual world are often utilized to create an immersive experience.

    More recently, numerous applications of VR specific to the health context have been identified and used [1-7], as research involving VR for health-related applications is gaining interest. As of July 2020, over 1000 studies were registered on ClinicalTrials.gov—a registry of clinical trials in the United States—for assessing VR interventions, such as anxiety management, distraction during painful procedures, gait training, rehabilitation, phobias, and medical education [8]. VR has been shown to be able to act as a low-cost and effective analgesic for pain arising in cases such as invasive medical procedures or even cancer in pediatric patients [9-11]. Hospitals may be able to leverage VR to reduce preoperative anxiety in patients, as well as a treatment method for those with generalized anxiety disorder [12]. A recent study by Donker et al showed that patients with acrophobia who received exposure therapy through a gamified, VR-enabled, self-help app had significant reductions in acrophobic symptoms [13]. Notably, in addition to the lack of need for a psychiatrist to be directly present during this intervention, the total cost per patient came to approximately US $24 through the use of Google Cardboard as the VR headset [13], exemplifying the ability of VR to increase treatment access while also significantly reducing costs. These examples only scratch the surface of the exciting potential of VR in health care.

    To complement the significant amount of benefits that VR applications bring to health-related contexts, a focus on the usability of health information technologies needs to be maintained, particularly given the diverse needs and abilities of the user base (eg, patients, health professionals, family members, etc). By usability, we refer to how easily the technology can be utilized by an individual based on three cycles or steps [14]. Often, the effort invested into ensuring the usability of a technology or application goes unnoticed until the user interacts with a poorly designed system. A user’s proficiency with a technology may originate from a combination of their own self-exploratory learning as well as more formal, structured lessons and walkthroughs. Given the novel nature of VR in health care, the likely paucity of the latter places a greater emphasis on ensuring VR technologies are intuitive and easy to adopt for those who are new to the technology.

    In the context of VR specifically, this includes both the use of the hardware (eg, headset) as well as the immersive software and VR experience as perceived by the user.


    The purpose of this viewpoint paper is to conduct the following: (1) highlight the need to conduct usability assessments for VR apps, (2) provide a primer on the potential usability assessment approaches that can be applied to VR in health-related contexts and their potential challenges, and (3) direct readers to several resources where additional information on the topic can be found.


    One of the challenges of VR for health-related applications is assessing and addressing issues related to usability. Health-related applications of VR may warrant an even greater focus on usability testing than nonhealth-related applications, given that the user base (ie, those typically with illnesses, chronic conditions, or disabilities) is diverse in terms of ages, abilities, and beyond, and may have special needs that need to be accounted for when utilizing the technology. In addition, one of the most common problems associated with VR is motion sickness, which is often related to the quality of the virtual space mapping to the replicated physical setting [15]. This can be a significant barrier to users looking to obtain health-related benefits from using VR. Yet, there are methods in which motion sickness may be evaluated and addressed before the technology is implemented. The integration of VR into treatment plans can also meet commonly seen elements of friction associated with new technologies, such as distrust during adoption, although in some situations these can dwindle following introductory exposure [16]. Other limitations of contemporary VR include the challenge of generating varied types of tactile sensations [17] and other types of multisensory integration [18].

    Assessment approaches for analyzing and evaluating the usability of various VR technologies for health-related applications have generally been understudied and not well described in the research literature. We conducted a cursory search ourselves of several academic databases and found limited explanations of usability methods utilized in the development stage of VR applications and an even more limited body of literature on how to conduct usability assessments for VR used for health-related purposes. While reasons for this gap in knowledge are likely due to the nascent nature of the field, further work must be completed toward generating best practices related to VR usability to assist practitioners and researchers in the development and diffusion of these sorts of innovations. For instance, outside of the VR context, there is an extensive literature base identifying the need for technologies that are used for health-related applications to be user friendly and have a high degree of ease of use, often incorporating in lessons from the human factors discipline [19-22]. Numerous papers, including one reporting on the System Usability Scale [23], have been published describing ways to assess usability for non-VR technologies, including electronic health records and mobile health apps [24-26]. Yet, there is limited guidance for those developing or researching health-related VR environments. Often, usability evaluation approaches used for other health information technology applications are difficult to implement within VR contexts. Thus, VR applications used in health contexts may not always undergo a thorough usability assessment. In the meantime, however, methods developed outside of the VR context will continue to be used until the scientific approaches for assessing VR usability further develop and until methods from the VR gaming industry become commonplace in health technology–related research.


    Overview

    The following section describes VR usability assessment methods that have been employed in past research. It is important to note that these methods may be hybridized and blended together to suit the goals of each unique evaluation and are not mutually exclusive. The approaches are described using a previously developed classification of usability methods in virtual environments developed by Bowman and colleagues in 2000 [27] and updated by Martens in 2016 [28]. These approaches include (1) cognitive or task walkthrough, (2) graphical evaluation, (3) post hoc questionnaire or interview, (4) physical performance evaluation, (5) user interface (UI) evaluation, and (6) heuristic evaluation.

    Table 1 [14,21,29-36] summarizes key information related to each of the identified VR assessment approaches, including some considerations for assessment requirements excluding basic needs, such as an appropriate space to conduct a VR assessment and the VR hardware and software itself.

    It is recommended that some assessment methods should favor the involvement of specific user groups, such as external users (ie, a group of testers not involved in the development process). Some assessment method requirements also lend themselves to requiring representative users, meaning a sample of users who may reflect the appropriate end-user population. The following sections provide an explanation of each of the VR usability assessment approaches.

    Table 1. Overview of virtual reality (VR) usability assessment approaches.
    ApproachAssessment requirementsaVR aspects evaluatedTypical output of assessmentResults type
    Cognitive or task walkthrough [14,29]Representative external users, developed task or scenario, and recording and timing equipmentEnvironment navigation, object interaction, and user-system interactionTask performance and user feedbackDiscrete and descriptive
    Graphical evaluation [30,31]Multiple relevant graphical environments, recording equipment, questionnaires, and interview guidesQuality of graphics and image renderingsUser feedbackDescriptive
    Post hoc questionnaires and interviews [32]External users, questionnaires or interview guides, and recording equipmentNonspecificUser feedbackDescriptive
    Physical performance evaluation [33]External users, developed task or scenario, and recording and timing equipmentPhysical immersion and VR performanceTask performance, system performance metrics, and user feedbackDiscrete and descriptive
    User interface evaluation [14,34,35]Developed task or scenario, recording and timing equipment, questionnaires, and interview guidesIntegration of VR environment and real-life tools and VR performanceUser feedback and task performanceDiscrete and descriptive
    Heuristic evaluation [14,21,36]Experienced users, developed task or scenario, questionnaires, interview guides, and recording equipmentVariousRefer to list in Heuristic Evaluation sectionDescriptive

    aItalicized text indicates optional requirements depending on specific assessment approaches being used (eg, recording equipment is only required if incorporating think-aloud methods).

    Cognitive or Task Walkthrough

    The cognitive or task walkthrough is a formative assessment method that assesses the user, or hypothetical user, based on the completion of task-based VR scenarios, response to system changes, and the user’s exploration and navigation of the VR environment [14]. While other measures for task load performance exist, such as the NASA-TLX (Task Load Index) [37], this assessment is based on Norman's 1986 [38] model of interaction and assesses the user’s mental and physical actions in VR environments founded on the premise that users learn to use a technology through a process of self-exploration rather than didactic training or lessons [39]. Originally designed to assess simple UIs, such as automated teller machines and kiosks, this assessment method is increasingly used to assess VR usability as well [40].

    One way to perform such an assessment is by employing the following three cycles or steps. The first cycle assesses a user’s actions when they are trying to achieve a goal [14]. An observer will document the overall path the user takes to complete a task or whether they behave in an intended way in the VR scenario. Challenges or issues in achieving the goal of each cycle are noted by the observer. Behaviors in this first cycle are largely dictated by the user having to make decisions and how the environment facilitates such decision pathways. For example, if the user’s goal is to pick up an object, but the object is missing, then the environment should allow the user to locate the object. Locating the object itself leads to the second cycle or step called “exploration and navigation in virtual environments” [14].

    In the second cycle, the user explores and moves around the environment to identify a path toward an object of interest. The VR environment should allow for intuitive navigation, recognizing user movements and responsively adapting to changes in user location as the user explores to locate the object of interest [14]. The observer records any challenges or issues in achieving this goal.

    In the third cycle, the user’s behaviors in response to a system initiative are assessed [14]. The purpose of this cycle or step is to examine how the VR system supports user activity when the user manipulates an object. The user and system are required to reciprocally recognize and interpret the feedback or actions of one another and respond appropriately [14]. For instance, if the user decides to throw a vase, the system should interpret this action and produce an appropriate response, such as depicting the vase flying and being shattered when contacting another object such as a wall. Correspondingly, the system may also take the initiative and act, meaning it is the user’s role to interpret and respond to this action [14]. For example, if a helium balloon (ie, the object) suddenly detaches from its base and starts floating away (ie, the system’s action or initiative), the user in this case may then choose to intervene and attempt to catch the balloon or allow the balloon to float away.

    In summary, a cognitive or task walkthrough is a task-based assessment that assesses a user’s actions when they are trying to achieve a goal (see Table 1) and incorporates further assessments of user navigation (ie, cycle two) and system response (ie, cycle three). For each of these cycles, users should be allowed to freely walk through the task or interaction without interruption by the observers. Since this approach is largely driven by scripts and dialogues within the VR environment, usability issues and the system’s ability to support user interaction is primarily assessed through descriptive, qualitative feedback (eg, user comments, think-aloud method, and observer observations) [14,28].

    Graphical Evaluation

    This assessment method focuses on the quality of graphics generated in the VR environment and how it influences the user's experience. This may include, but is not limited to, how different color combinations, shapes, textures, and renderings depicted will impact the user’s interaction with the VR environment and system [30,41]. There are numerous ways to assess graphics, which can be attuned to hardware (eg, view, resolution, color contrast, update rate, etc); fidelity (eg, geometry and colors); camera placement, if applicable; the precision of the tracking system; stereoscopic image quality [42]; and beyond [43].

    Many methods that vary in degrees of complexity exist for assessing graphics. In one common approach, users are exposed to different iterations of graphical environments to get a better understanding of its impact on user experience [30,41]. Depending on the purpose of the overall VR environment, the graphical object of interest may vary. For example, to examine a user’s behavior in a large city, the graphical evaluation may be more focused on image depth, complexity, and breadth of the city and its 3D renderings. If the focus is narrower, such as assessing how a user reacts to seeing smoking paraphernalia, then focusing on meticulous, realistic details for an object such as a cigarette will be of greater importance. To assess a user’s response to graphics in a VR environment, users may be asked to think aloud or be given a set of questionnaires to collect user feedback about the graphical output in the VR system (see Post Hoc Questionnaires and Interviews section) [30,41].

    Post Hoc Questionnaires and Interviews

    Post hoc questionnaires and interviews are often used to identify a user’s general overall experience in using VR. However, some of these questionnaires and interviews may be targeted toward specific usability concepts, such as graphics, the physical hardware, and motion sickness [37,44,45].

    This assessment method is often performed following the conclusion of a user’s interaction with the VR system [28]. Since VR remains a relatively new technology, responses may be highly influenced by the individual’s comfort and experience with using VR. Thus, unless the usability evaluation is already tailored to a target or only includes a subset of users based on experience (eg, inexperienced VR users), demographic information about users’ opinions, views, and experiences with VR should also be collected to help better interpret user feedback [28]. Due to its versatility, this assessment can be viewed as a complement to many of the assessments covered in this article rather than a stand-alone method. Its overall purpose is to serve as a simple, straightforward way of collecting targeted feedback. In order to collect specific feedback pertaining to the specific evaluation tied to a post hoc questionnaire or interview, special care must be given to the semantics and framing of questions [28].

    Often, post hoc questionnaires are also used during the VR prototyping stage by engaging end users as a form of iterative quality improvement, but often in conjunction with another evaluation method such as a cognitive or task walkthrough [28].

    Physical Performance Evaluation

    Physical performance in the context of VR is defined by the performance of the hardware and environment. The smoothness and quality of the virtual environment are evaluated not unlike how a website can be evaluated on its loading time. Performance metrics with this assessment method include lag time (ie, the time delay between the user’s intended action and the system’s response within VR) and synchronization (ie, whether the system accurately reflects the user’s intended actions). VR should be as convincingly realistic as possible to users, and the physical performance of a VR system is the key determinant of mental and physical immersion [33]. Immersion is defined as a state of being fully absorbed and/or deeply engaged within a simulated environment and is a key factor in determining the quality of VR [21]. This assessment method can facilitate user-centered design and can also yield information on the physical space required for users to fully explore the VR environment [33].

    To gauge the VR system’s physical performance, data can be obtained through a combination of approaches, such as questionnaires, task performance scores, or by leveraging back-end data to examine factors such as retrieval and load times. Simple but physically demanding VR precision tasks are highly informative for this type of assessment. For example, a task involving manipulating small objects with virtual chopsticks will quickly reveal any performance issues related to the precision of translated movements. Such a task can be timed and scored, and the user can be asked to describe their satisfaction and feelings to identify physical performance issues [33]. As another example, tasks involving actions that require users to reach out around their body to interact with nearby objects can be used to highlight unaddressed issues with distance compression, a frequent phenomenon within VR environments where objects are perceived by the user to be closer than their actual position [46]. Following a given task, a user may achieve high task performance scores but still report heavy cognitive overload (ie, mental exhaustion) while using the system, for example, finding that performing the task in VR was significantly more difficult than performing the same task with real objects or tools. Such a situation would signal that some probing questions (eg, Was there a specific action of the task that was particularly difficult to perform?) or further back-end evaluations may be required to identify possible underlying physical performance–related issues [33].

    User Interface Evaluation

    The purpose of a UI evaluation is to help determine the usability of a VR system’s front-end UI [14,35,47]. This approach can also help identify a UI design or solution that appropriately balances factors such as intuition and immersion against usability [34]. An optimized UI solution should provide the user with the best combination between immersion and usability, such that users feel immersed but unencumbered in accomplishing their tasks relative to outside a VR environment [14,34]. A feeling of immersion is especially pertinent when considering VR applications that notably outperform real-world counterparts, such as a simulated environment used to manage phobias or pain [8]. In these unique situations where the UI itself is deeply interrelated with the intervention (ie, phobia exposure tool), a comprehensive UI design evaluation may only be feasibly accomplished by a wider-scale clinical trial measuring treatment outcomes. Returning to more general VR applications, a UI evaluation allows for the identification of the type of UI solution that will provide the best immersion-to-efficiency ratio between a VR environment and real-life tools [31]. In a proof-of-concept case study by Kasurinen [34], users were instructed to complete one of five training scenarios with three varying levels of VR and real-life tools [34]:

    1. No VR: participants move throughout an environment with keyboard and mouse controls; other activities are completed with real-life tools in a simulated workspace setting.
    2. Semi-VR: participants move within a virtual environment with a VR headset; other activities are completed with real-life tools in a simulated workspace setting. The real-life workstation also displays the current state of the VR.
    3. Full VR: participants move and complete their activities fully within a VR environment. Real-life tools are replaced with virtual equivalents (eg, virtual keyboard) and other real-life displays (eg, workstation screen) are virtually broadcasted to the VR headset.

    For each iteration, data on user preferences can be collected alongside discrete data, such as task completion times and the number of errors [34]. Questions related to UI elements should also be asked throughout each iteration, as follows [14]:

    1. Can the user form or remember the task goal?
    2. Are the appropriate objects or parts of the environment viable?
    3. Can the necessary objects be located?
    4. Can the user execute movement and navigation actions?
    5. Can the user recognize objects in the environment?

    Each of these questions can help to reveal a specific area with potential for improvement within the UI. This method can aid in assessing both the appropriate amount of real-life integration and the quality of said integration so the VR intervention can best accomplish its intended purpose. If the integration between virtual and real-life tools is insufficient, it has been shown that this friction will cause users to prefer the No VR option, which may also be partially related to physical performance (see Physical Performance Evaluation section) [34].

    Heuristic Evaluation

    A heuristic evaluation is a UI approach that involves several topic experts or an expert evaluator, rather than soliciting direct user feedback. A VR usability expert will typically evaluate a UI’s design against an accepted set of usability principles or standards already published in the literature [48]. While there are several sets of accepted standards or heuristics, for traditional UIs little research exists on defining heuristics for VR environments. Nielsen’s [21] heuristics set is the most commonly referenced and utilized set of heuristics for UI design. Sutcliffe and Gault [48] further defined a set of 12 heuristic guidelines based on Nielsen’s set, as shown in Textbox 1.

    A set of 12 heuristic guidelines.
    • Natural engagement
    • Compatibility with the user’s task and domain
    • Natural expression of action
    • Close coordination of action and representation
    • Realistic feedback
    • Faithful viewpoints
    • Navigation and orientation support
    • Clear entry and exit points
    • Consistent departures
    • Support for learning
    • Clear turn taking
    • Sense of presence
    Textbox 1. A set of 12 heuristic guidelines.

    Expert results are then aggregated and used to identify priority areas of action [28]. Heuristic assessments also require a set of tasks for the experts to experience. The nature of these tasks and the VR environments themselves should also be subjectively considered when carrying out a heuristic assessment, given the lack of standardization between various types of VR equipment and software [28]. While not all heuristics may apply to a given VR application, such an evaluation has great potential to glean a rich, overall picture of the state of the application. For example, if the VR application is intended to be designed in a way that the user is automatically placed in an “inescapable” environment, then there is no relevance in assessing clear entry and exit points (ie, the eighth heuristic guideline, clear entry and exit points) [28]. Since heuristics are broad rules of thumb rather than specific guidelines, they should not be treated as binary checkboxes, but rather as individual continuums that can each be an area for improvement, although binary elements may exist within. To illustrate, perhaps the heuristic guideline of realistic feedback is of particular interest, which outlines that the VR application should help users effectively recognize and recover from errors [21]. The presence or absence of a feature such as, for example, tangible error messages would constitute a binary checkbox, but the palatability and effectiveness of said error messages would be of higher importance. Is the problem or error precisely and concisely indicated? Is a potential solution suggested? Is the language user friendly and free of codes or abbreviations, such as “A 50 (0x32) error has occurred”? Ultimately, considering and tracking multiple granular elements within each heuristic will aid greatly in obtaining actionable results to direct improvement.


    Table 2 [14,21,23,29-34,36,37,39,40,42-45,49-54] provides a list of references specific to each of the approaches where readers can access additional information.

    Table 2. Additional resources.
    Assessment approachReferences and resources
    Cognitive or task walkthrough[14,29,39,40,49,50]
    Graphical evaluation[30,31,42,43]
    Post hoc questionnaires and interviewsa[23,32,37,44,45,51]
    Physical performance evaluation[33]
    User interface evaluation [14,34]
    Heuristic evaluation[14,21,36,52]
    Other[53,54]

    aThis represents a sample of many that can be employed, depending on what usability concept is to be measured.


    As previously noted, many of the approaches presented in this paper can be blended or hybridized together to suit the goals or needs of a given VR application evaluation. They are certainly not mutually exclusive. Some of the methods already incorporate a level of hybridization, most often with the inclusion of a post hoc questionnaire or interview. Given the lack of standardization across approaches, this warrants future research regarding the development of a comprehensive framework incorporating multiple methods of VR evaluation to provide, at a minimum, a strategic work plan for those looking to perform a baseline evaluation of any new VR application. This should include the incorporation of more up-to-date methods already used in the gaming industry. Those who employ usability methods for VR that have been developed for other kinds of health information technologies should be encouraged to share their experiences with the broader scientific community, placing an emphasis on the practical experiences of doing so. The current literature base lacks practical examples of how to best use these approaches, which could be of great use to those employing them.

    When developing VR interventions and applications, particularly in the context of health, the comfort of the end user is paramount. Alongside the numerous benefits of VR technology, VR still carries the risk of imposing symptoms similar to motion sickness during use as a result of visual distortions and asynchronies, among other effects [45]. While these issues are peripherally related to performance issues and may be identified in user feedback, these data are inherently subjective and the effects are, thus, not easily quantifiable enough to measure improvements. Thus, the authors recommend that any VR assessment also explicitly consider the possible effect of motion sickness on its users by incorporating tools such as the Simulator Sickness Questionnaire (SSQ), originally developed to help measure motion sickness for pilots in flight simulators [45]. The results from the SSQ or another similar questionnaire may identify specific considerations for certain populations, age groups, diagnoses, and beyond. Additionally, the repeated occurrence of specific symptoms or combinations of such from the SSQ (eg, eyestrain, nausea, and vertigo) can provide additional direction in identifying the root issues within the VR software and hardware [45].


    Health-related applications using VR are a rapidly advancing area of development. Like all emerging technologies in health care, there is a need to ensure the quality and safety of these novel tools [55]. For VR, validated usability and assessment approaches are an important step before its deployment in real-world clinical settings. The assessment methods described here give developers and researchers a high-level overview of important elements to consider regarding the usability of their VR implementations and to make iterative changes prior to clinical implementation. However, once these approaches are employed for VR, sharing practical experiences in doing so would be of tremendous value. This area of science is in its infancy and comprehensive knowledge translation would be critical to its growth.

    Overall, this paper provides a description and discussion of six different contemporary VR usability assessment methods. As an emerging area for research, the development of formative usability assessment methodologies for health-related VR applications is an important area for future development. Further, while the six approaches discussed in this paper have been discussed in isolation, further future hybridization of approaches to develop more robust and multidimensional interpretations of VR usability should be considered. For instance, like other usability evaluation approaches [21,56], a purposeful mixed methods approach may assist in generating more holistic and robust interpretations of a system’s usability. We see value in the triangulation of data related to user feedback and other task performance metrics in health-related VR applications. Due to the nascent nature of the domain, a pluralistic approach to usability evaluation should be considered in an effort to develop broader and more nuanced understandings of the state of the art in VR.

    Given that the VR industry is projected to grow to over US $9 billion in sales of VR devices alone by 2021 [57], it is no surprise the industry is marked with large financial investments, such as the acquisition of Oculus for US $2 billion, as many large technology companies continue to invest heavily in VR [58]. As a collective, health care organizations and professionals should emphasize ensuring the mitigation and prevention of potential growing pains that may arise if VR interventions are churned out without rigorous evaluation and proper regard for quality, allowing for VR to usher in a new field of innovative, technology-enabled health care. With this foundation, the potential benefits to providers and patients alike will only continue to grow with continuous improvements in technology and reductions in cost.

    Acknowledgments

    The authors would like to acknowledge the Centre for Addiction and Mental Health library in Toronto, Canada, for providing access to the necessary scholarly resources to carry out this work.

    Authors' Contributions

    This work was first conceived by GS. All authors contributed to the writing and editing of the manuscript through feedback and discussion, met ICMJE (International Committee of Medical Journal Editors) author requirements, and have approved the final manuscript.

    Conflicts of Interest

    None declared.

    1. Hu J, Luo E, Song E, Xu X, Tan H, Zhao Y, et al. Patients' attitudes towards online dental information and a web-based virtual reality program for clinical dentistry: A pilot investigation in China. Int J Med Inform 2009 Mar;78(3):208-215. [CrossRef] [Medline]
    2. Tossavainen T, Juhola M, Pyykkö I, Aalto H, Toppila E. Development of virtual reality stimuli for force platform posturography. Int J Med Inform 2003 Jul;70(2-3):277-283. [CrossRef]
    3. Vogt S, Skjæret-Maroni N, Neuhaus D, Baumeister J. Virtual reality interventions for balance prevention and rehabilitation after musculoskeletal lower limb impairments in young up to middle-aged adults: A comprehensive review on used technology, balance outcome measures and observed effects. Int J Med Inform 2019 Jun;126:46-58. [CrossRef] [Medline]
    4. Hone-Blanchet A, Wensing T, Fecteau S. The use of virtual reality in craving assessment and cue-exposure therapy in substance use disorders. Front Hum Neurosci 2014;8:844 [FREE Full text] [CrossRef] [Medline]
    5. Bordnick PS, Graap KM, Copp HL, Brooks J, Ferrer M. Virtual reality cue reactivity assessment in cigarette smokers. Cyberpsychol Behav 2005 Oct;8(5):487-492. [CrossRef] [Medline]
    6. Clus D, Larsen ME, Lemey C, Berrouiguet S. The use of virtual reality in patients with eating disorders: Systematic review. J Med Internet Res 2018 Apr 27;20(4):e157 [FREE Full text] [CrossRef] [Medline]
    7. Galvez J, Eisenhower M, England W, Wartman E, Simpao A, Rehman M, et al. An interactive virtual reality tour for adolescents receiving proton radiation therapy: Proof-of-concept study. JMIR Perioper Med 2019 Mar 05;2(1):e11259 [FREE Full text] [CrossRef]
    8. ClinicalTrials.gov. Bethesda, MD: US National Library of Medicine   URL: https://clinicaltrials.gov/ [accessed 2020-10-06]
    9. Gershon J, Zimand E, Lemos R, Olasov Rothbaum B, Hodges L. Use of virtual reality as a distractor for painful procedures in a patient with pediatric cancer: A case study. Cyberpsychol Behav 2003 Dec;6(6):657-661. [CrossRef] [Medline]
    10. Freitas DM, Spadoni VS. Is virtual reality useful for pain management in patients who undergo medical procedures? Einstein (Sao Paulo) 2019;17(2):1-3 [FREE Full text] [CrossRef]
    11. Eijlers R, Utens EMWJ, Staals LM, de Nijs PFA, Berghmans JM, Wijnen RMH, et al. Systematic review and meta-analysis of virtual reality in pediatrics. Anesth Analg 2019;129(5):1344-1353. [CrossRef]
    12. Tarrant J, Viczko J, Cope H. Virtual reality for anxiety reduction demonstrated by quantitative EEG: A pilot study. Front Psychol 2018;9:1280 [FREE Full text] [CrossRef] [Medline]
    13. Donker T, Cornelisz I, van Klaveren C, van Straten A, Carlbring P, Cuijpers P, et al. Effectiveness of self-guided app-based virtual reality cognitive behavior therapy for acrophobia: A randomized clinical trial. JAMA Psychiatry 2019 Jul 01;76(7):682-690 [FREE Full text] [CrossRef] [Medline]
    14. Sutcliffe AG, Kaur KD. Evaluating the usability of virtual reality user interfaces. Behav Inf Technol 2000 Jan;19(6):415-426. [CrossRef]
    15. Nichols S, Patel H. Health and safety implications of virtual reality: A review of empirical evidence. Appl Ergon 2002 May;33(3):251-271. [CrossRef] [Medline]
    16. Huygelier H, Schraepen B, van Ee R, Vanden Abeele V, Gillebert CR. Acceptance of immersive head-mounted virtual reality in older adults. Sci Rep 2019 Mar 14;9(1):4519 [FREE Full text] [CrossRef] [Medline]
    17. Han P, Chen Y, Lee K, Wang H, Hsieh C, Hsiao J, et al. Haptic around: Multiple tactile sensations for immersive environment and interaction in virtual reality. In: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST '18). 2018 Presented at: 24th ACM Symposium on Virtual Reality Software and Technology (VRST '18); November 28-December 1, 2018; Tokyo, Japan p. 1-10. [CrossRef]
    18. Riva G, Wiederhold BK, Mantovani F. Neuroscience of virtual reality: From virtual exposure to embodied medicine. Cyberpsychol Behav Soc Netw 2019 Jan;22(1):82-96 [FREE Full text] [CrossRef] [Medline]
    19. Walji MF, Kalenderian E, Piotrowski M, Tran D, Kookal KK, Tokede O, et al. Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. Int J Med Inform 2014 May;83(5):361-367 [FREE Full text] [CrossRef] [Medline]
    20. Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017 Jan;24(1):218-226. [CrossRef] [Medline]
    21. Nielsen J. Enhancing the explanatory power of usability heuristics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). 1994 Presented at: SIGCHI Conference on Human Factors in Computing Systems (CHI '94); April 24-28, 1994; Boston, MA p. 152-158.
    22. Dey A, Billinghurst M, Lindeman RW, Swan JE. A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Front Robot AI 2018 Apr 17;5:1-28 [FREE Full text] [CrossRef]
    23. Brooke J. SUS: A 'quick and dirty' usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL, editors. Usability Evaluation in Industry. London, UK: Taylor & Francis; 1996:189-194.
    24. Saleem JJ, Haggstrom DA, Militello LG, Flanagan M, Kiess CL, Arbuckle N, et al. Redesign of a computerized clinical reminder for colorectal cancer screening: A human-computer interaction evaluation. BMC Med Inform Decis Mak 2011 Nov 29;11:74 [FREE Full text] [CrossRef] [Medline]
    25. Johnson CM, Johnston D, Crowley PK, Culbertson H, Rippen HE, Damico DJ, et al. EHR Usability Toolkit: A Background Report on Usability and Electronic Health Records. Rockville, MD: Agency for Healthcare Research and Quality, US Department of Health and Human Services; 2011 Aug.   URL: https:/​/digital.​ahrq.gov/​sites/​default/​files/​docs/​citation/​EHR_Usability_Toolkit_Background_Report.​pdf [accessed 2020-10-06]
    26. Karahoca A, Bayraktar E, Tatoglu E, Karahoca D. Information system design for a hospital emergency department: A usability analysis of software prototypes. J Biomed Inform 2010 Apr;43(2):224-232 [FREE Full text] [CrossRef] [Medline]
    27. Bowman DA, Gabbard JL, Hix D. A survey of usability evaluation in virtual environments: Classification and comparison of methods. Presence 2002 Aug;11(4):404-424 [FREE Full text] [CrossRef]
    28. Martens D. Virtually Usable: A Review of Virtual Reality Usability Evaluation Methods [master's thesis]. New York, NY: Parsons School of Design; 2016 May 14.   URL: https://danamartensmfadt.files.wordpress.com/2016/08/virtuallyusable.pdf [accessed 2020-10-06]
    29. Costalli F, Marucci L, Mori G, Paterno F. Design criteria for usable web-accessible virtual environments. In: Proceedings of the International Cultural Heritage Informatics Meeting (ICHIM 2001). 2001 Presented at: International Cultural Heritage Informatics Meeting (ICHIM 2001); September 3-7, 2001; Milan, Italy p. 413-426.
    30. McMahan RP, Bowman DA, Zielinski DJ, Brady RB. Evaluating display fidelity and interaction fidelity in a virtual reality game. IEEE Trans Vis Comput Graph 2012 Apr;18(4):626-633. [CrossRef]
    31. Vaden EA, Ehrlich JA, Kolasinski EM. Usability evaluation of low-end virtual reality systems. In: Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting. 1996 Presented at: Human Factors and Ergonomics Society 40th Annual Meeting; September 2-6, 1996; Philadelphia, PA. [CrossRef]
    32. Castilla D, Garcia-Palacios A, Bretón-López J, Miralles I, Baños RM, Etchemendy E, et al. Process of design and usability evaluation of a telepsychology web and virtual reality system for the elderly: Butler. Int J Hum Comput Stud 2013 Mar;71(3):350-362. [CrossRef]
    33. Rezazadeh IM, Firoozabadi M, Wang X. Evaluating the usability of virtual environment by employing affective measures. In: Wang X, editor. Mixed Reality and Human-Robot Interaction. Intelligent Systems, Control and Automation: Science and Engineering, vol 1010. Dordrecht, the Netherlands: Springer; 2011:95-109.
    34. Kasurinen J. Usability issues of virtual reality learning simulator in healthcare and cybersecurity. Procedia Comput Sci 2017;119:341-349. [CrossRef]
    35. Chin J, Diehl VA, Norman KL. Development of an instrument measuring user satisfaction of the human-computer interface. In: Proceedings of the Human Factors in Computing Systems Conference (ACM CHI '88). 1988 Presented at: Human Factors in Computing Systems Conference (ACM CHI '88); May 15-19, 1988; Washington, DC p. 213-218. [CrossRef]
    36. Sutcliffe AG, Deol Kaur K. A usability evaluation method for virtual reality user interfaces. CiteSeerX. 2006.   URL: http:/​/citeseerx.​ist.psu.edu/​viewdoc/​download;jsessionid=B5295890A3F6C06F50E9FC12DF1EB4F3?doi=10.​1.​1.​115.​945&rep=rep1&type=pdf [accessed 2020-10-06]
    37. Hart S, Staveland L. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Adv Psychol 1988;52:139-183. [CrossRef]
    38. Norman DA. Cognitive engineering—Cognitive science. In: Carroll JM, editor. Interfacing Thought: Cognitive Aspects of Human-Computer Interaction. Cambridge, MA: MIT Press; 1987:325-336.
    39. Polson P, Lewis C, Rieman J, Wharton C. Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. Int J Man Mach Stud 1992 May;36(5):741-773. [CrossRef]
    40. Wharton C, Bradford J, Jeffries R, Franzke M. Applying cognitive walkthroughs to more complex user interfaces experiences, issues, and recommendations. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). 1992 Presented at: SIGCHI Conference on Human Factors in Computing Systems (CHI '92); May 3-7, 1992; Monterey, CA p. 381-388. [CrossRef]
    41. Van Orden KF, DiVita J. Effects of structure and color on symbol visibility. In: Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting. 1996 Presented at: Human Factors and Ergonomics Society 40th Annual Meeting; September 2-6, 1996; Philadelphia, PA. [CrossRef]
    42. Moorthy A, Su C, Mittal A, Bovik A. Subjective evaluation of stereoscopic image quality. Signal Process Image Commun 2013 Sep;28(8):870-883. [CrossRef]
    43. Menzies RJ, Rogers SJ, Phillips AM, Chiarovano E, de Waele C, Verstraten FAJ, et al. An objective measure for the visual fidelity of virtual reality and the risks of falls in a virtual environment. Virtual Real 2016 Jun 6;20(3):173-181. [CrossRef]
    44. Golding J. Motion sickness susceptibility questionnaire revised and its relationship to other forms of sickness. Brain Res Bull 1998 Nov 15;47(5):507-516. [CrossRef] [Medline]
    45. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. Int J Aviat Psychol 1993 Jul;3(3):203-220. [CrossRef]
    46. Finnegan D, O'Neill E, Proulx M. An approach to reducing distance compression in audiovisual virtual environments. In: Proceedings of the IEEE 3rd VR Workshop on Sonic Interactions for Virtual Environments (SIVE). 2017 Presented at: IEEE 3rd VR Workshop on Sonic Interactions for Virtual Environments (SIVE); March 19, 2017; Los Angeles, CA p. 1-6. [CrossRef]
    47. Charfi S, Kolski C. RITA: A useR Interface evaluaTion frAmework. J Univers Comput Sci 2015;21(4):526-560 [FREE Full text] [CrossRef]
    48. Sutcliffe A, Gault B. Heuristic evaluation of virtual reality applications. Interact Comput 2004 Aug;16(4):831-849. [CrossRef]
    49. Wilson C. Cognitive walkthrough. In: User Interface Inspection Methods: A User-Centered Design Method. Waltham, MA: Morgan Kaufmann; Nov 2013:65-80.
    50. Mahatody T, Sagar M, Kolski C. State of the art on the cognitive walkthrough method, its variants and evolutions. Int J Hum Comput Interact 2010 Jul 30;26(8):741-785. [CrossRef]
    51. Golding JF. Predicting individual differences in motion sickness susceptibility by questionnaire. Pers Individ Dif 2006 Jul;41(2):237-248. [CrossRef]
    52. Murtza R, Monroe S, Youmans R. Heuristic evaluation for virtual reality systems. In: Proceedings of the Human Factors and Ergonomics Society International Annual Meeting. 2017 Presented at: Human Factors and Ergonomics Society International Annual Meeting; October 9-13, 2017; Austin, TX p. 2067-2071. [CrossRef]
    53. Dias P, Pimentel A, Ferreira C, Madeira J. Usability in virtual and augmented environments: A qualitative and quantitative study. In: Proceedings of the International Society for Optics and Photonics (SPIE): Stereoscopic Displays and Virtual Reality Systems XIV. 2007 Mar Presented at: International Society for Optics and Photonics (SPIE): Stereoscopic Displays and Virtual Reality Systems XIV; January 29-31, 2007; San Jose, CA. [CrossRef]
    54. Livatino S, Koffel C. Handbook for evaluation studies in virtual reality. In: Proceedings of the IEEE Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems. 2007 Presented at: IEEE Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems; June 25-27, 2007; Ostuni, Italy p. 1-6. [CrossRef]
    55. Mytton OT, Velazquez A, Banken R, Mathew JL, Ikonen TS, Taylor K, et al. Introducing new technology safely. Qual Saf Health Care 2010 Aug;19 Suppl 2:i9-i14. [CrossRef] [Medline]
    56. Xiao Y, Montgomery DC, Philpot LM, Barnes SA, Compton J, Kennerly D. Development of a tool to measure user experience following electronic health record implementation. J Nurs Adm 2014;44(7/8):423-428. [CrossRef]
    57. Flavián C, Ibáñez-Sánchez S, Orús C. The impact of virtual, augmented and mixed reality technologies on the customer experience. J Bus Res 2019 Jul;100:547-560. [CrossRef]
    58. Cipresso P, Giglioli IAC, Raya MA, Riva G. The past, present, and future of virtual and augmented reality research: A network and cluster analysis of the literature. Front Psychol 2018;9:2086 [FREE Full text] [CrossRef] [Medline]

    ‎
    ICMJE: International Committee of Medical Journal Editors
    SSQ: Simulator Sickness Questionnaire
    TLX: Task Load Index
    UI: user interface
    VR: virtual reality

    Edited by M Birk; submitted 06.02.20; peer-reviewed by T Risling, V vanden Abeele; comments to author 07.04.20; revised version received 09.07.20; accepted 03.09.20; published 28.10.20

    Copyright

    ©Timothy Zhang, Richard Booth, Royce Jean-Louis, Ryan Chan, Anthony Yeung, David Gratzer, Gillian Strudwick. Originally published in JMIR Serious Games (http://games.jmir.org), 28.10.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.