The following section describes VR usability assessment methods that have been employed in past research. It is important to note that these methods may be hybridized and blended together to suit the goals of each unique evaluation and are not mutually exclusive. The approaches are described using a previously developed classification of usability methods in virtual environments developed by Bowman and colleagues in 2000 [27] and updated by Martens in 2016 [28]. These approaches include (1) cognitive or task walkthrough, (2) graphical evaluation, (3) post hoc questionnaire or interview, (4) physical performance evaluation, (5) user interface (UI) evaluation, and (6) heuristic evaluation.

Table 1 [14,21,29-36] summarizes key information related to each of the identified VR assessment approaches, including some considerations for assessment requirements excluding basic needs, such as an appropriate space to conduct a VR assessment and the VR hardware and software itself.

It is recommended that some assessment methods should favor the involvement of specific user groups, such as external users (ie, a group of testers not involved in the development process). Some assessment method requirements also lend themselves to requiring representative users, meaning a sample of users who may reflect the appropriate end-user population. The following sections provide an explanation of each of the VR usability assessment approaches.

The cognitive or task walkthrough is a formative assessment method that assesses the user, or hypothetical user, based on the completion of task-based VR scenarios, response to system changes, and the user’s exploration and navigation of the VR environment [14]. While other measures for task load performance exist, such as the NASA-TLX (Task Load Index) [37], this assessment is based on Norman's 1986 [38] model of interaction and assesses the user’s mental and physical actions in VR environments founded on the premise that users learn to use a technology through a process of self-exploration rather than didactic training or lessons [39]. Originally designed to assess simple UIs, such as automated teller machines and kiosks, this assessment method is increasingly used to assess VR usability as well [40].

One way to perform such an assessment is by employing the following three cycles or steps. The first cycle assesses a user’s actions when they are trying to achieve a goal [14]. An observer will document the overall path the user takes to complete a task or whether they behave in an intended way in the VR scenario. Challenges or issues in achieving the goal of each cycle are noted by the observer. Behaviors in this first cycle are largely dictated by the user having to make decisions and how the environment facilitates such decision pathways. For example, if the user’s goal is to pick up an object, but the object is missing, then the environment should allow the user to locate the object. Locating the object itself leads to the second cycle or step called “exploration and navigation in virtual environments” [14].

In the second cycle, the user explores and moves around the environment to identify a path toward an object of interest. The VR environment should allow for intuitive navigation, recognizing user movements and responsively adapting to changes in user location as the user explores to locate the object of interest [14]. The observer records any challenges or issues in achieving this goal.

In the third cycle, the user’s behaviors in response to a system initiative are assessed [14]. The purpose of this cycle or step is to examine how the VR system supports user activity when the user manipulates an object. The user and system are required to reciprocally recognize and interpret the feedback or actions of one another and respond appropriately [14]. For instance, if the user decides to throw a vase, the system should interpret this action and produce an appropriate response, such as depicting the vase flying and being shattered when contacting another object such as a wall. Correspondingly, the system may also take the initiative and act, meaning it is the user’s role to interpret and respond to this action [14]. For example, if a helium balloon (ie, the object) suddenly detaches from its base and starts floating away (ie, the system’s action or initiative), the user in this case may then choose to intervene and attempt to catch the balloon or allow the balloon to float away.

In summary, a cognitive or task walkthrough is a task-based assessment that assesses a user’s actions when they are trying to achieve a goal (see Table 1) and incorporates further assessments of user navigation (ie, cycle two) and system response (ie, cycle three). For each of these cycles, users should be allowed to freely walk through the task or interaction without interruption by the observers. Since this approach is largely driven by scripts and dialogues within the VR environment, usability issues and the system’s ability to support user interaction is primarily assessed through descriptive, qualitative feedback (eg, user comments, think-aloud method, and observer observations) [14,28].

This assessment method focuses on the quality of graphics generated in the VR environment and how it influences the user's experience. This may include, but is not limited to, how different color combinations, shapes, textures, and renderings depicted will impact the user’s interaction with the VR environment and system [30,41]. There are numerous ways to assess graphics, which can be attuned to hardware (eg, view, resolution, color contrast, update rate, etc); fidelity (eg, geometry and colors); camera placement, if applicable; the precision of the tracking system; stereoscopic image quality [42]; and beyond [43].

Many methods that vary in degrees of complexity exist for assessing graphics. In one common approach, users are exposed to different iterations of graphical environments to get a better understanding of its impact on user experience [30,41]. Depending on the purpose of the overall VR environment, the graphical object of interest may vary. For example, to examine a user’s behavior in a large city, the graphical evaluation may be more focused on image depth, complexity, and breadth of the city and its 3D renderings. If the focus is narrower, such as assessing how a user reacts to seeing smoking paraphernalia, then focusing on meticulous, realistic details for an object such as a cigarette will be of greater importance. To assess a user’s response to graphics in a VR environment, users may be asked to think aloud or be given a set of questionnaires to collect user feedback about the graphical output in the VR system (see Post Hoc Questionnaires and Interviews section) [30,41].

Post hoc questionnaires and interviews are often used to identify a user’s general overall experience in using VR. However, some of these questionnaires and interviews may be targeted toward specific usability concepts, such as graphics, the physical hardware, and motion sickness [37,44,45].

This assessment method is often performed following the conclusion of a user’s interaction with the VR system [28]. Since VR remains a relatively new technology, responses may be highly influenced by the individual’s comfort and experience with using VR. Thus, unless the usability evaluation is already tailored to a target or only includes a subset of users based on experience (eg, inexperienced VR users), demographic information about users’ opinions, views, and experiences with VR should also be collected to help better interpret user feedback [28]. Due to its versatility, this assessment can be viewed as a complement to many of the assessments covered in this article rather than a stand-alone method. Its overall purpose is to serve as a simple, straightforward way of collecting targeted feedback. In order to collect specific feedback pertaining to the specific evaluation tied to a post hoc questionnaire or interview, special care must be given to the semantics and framing of questions [28].

Often, post hoc questionnaires are also used during the VR prototyping stage by engaging end users as a form of iterative quality improvement, but often in conjunction with another evaluation method such as a cognitive or task walkthrough [28].

Physical performance in the context of VR is defined by the performance of the hardware and environment. The smoothness and quality of the virtual environment are evaluated not unlike how a website can be evaluated on its loading time. Performance metrics with this assessment method include lag time (ie, the time delay between the user’s intended action and the system’s response within VR) and synchronization (ie, whether the system accurately reflects the user’s intended actions). VR should be as convincingly realistic as possible to users, and the physical performance of a VR system is the key determinant of mental and physical immersion [33]. Immersion is defined as a state of being fully absorbed and/or deeply engaged within a simulated environment and is a key factor in determining the quality of VR [21]. This assessment method can facilitate user-centered design and can also yield information on the physical space required for users to fully explore the VR environment [33].

To gauge the VR system’s physical performance, data can be obtained through a combination of approaches, such as questionnaires, task performance scores, or by leveraging back-end data to examine factors such as retrieval and load times. Simple but physically demanding VR precision tasks are highly informative for this type of assessment. For example, a task involving manipulating small objects with virtual chopsticks will quickly reveal any performance issues related to the precision of translated movements. Such a task can be timed and scored, and the user can be asked to describe their satisfaction and feelings to identify physical performance issues [33]. As another example, tasks involving actions that require users to reach out around their body to interact with nearby objects can be used to highlight unaddressed issues with distance compression, a frequent phenomenon within VR environments where objects are perceived by the user to be closer than their actual position [46]. Following a given task, a user may achieve high task performance scores but still report heavy cognitive overload (ie, mental exhaustion) while using the system, for example, finding that performing the task in VR was significantly more difficult than performing the same task with real objects or tools. Such a situation would signal that some probing questions (eg, Was there a specific action of the task that was particularly difficult to perform?) or further back-end evaluations may be required to identify possible underlying physical performance–related issues [33].

The purpose of a UI evaluation is to help determine the usability of a VR system’s front-end UI [14,35,47]. This approach can also help identify a UI design or solution that appropriately balances factors such as intuition and immersion against usability [34]. An optimized UI solution should provide the user with the best combination between immersion and usability, such that users feel immersed but unencumbered in accomplishing their tasks relative to outside a VR environment [14,34]. A feeling of immersion is especially pertinent when considering VR applications that notably outperform real-world counterparts, such as a simulated environment used to manage phobias or pain [8]. In these unique situations where the UI itself is deeply interrelated with the intervention (ie, phobia exposure tool), a comprehensive UI design evaluation may only be feasibly accomplished by a wider-scale clinical trial measuring treatment outcomes. Returning to more general VR applications, a UI evaluation allows for the identification of the type of UI solution that will provide the best immersion-to-efficiency ratio between a VR environment and real-life tools [31]. In a proof-of-concept case study by Kasurinen [34], users were instructed to complete one of five training scenarios with three varying levels of VR and real-life tools [34]:

For each iteration, data on user preferences can be collected alongside discrete data, such as task completion times and the number of errors [34]. Questions related to UI elements should also be asked throughout each iteration, as follows [14]:

Each of these questions can help to reveal a specific area with potential for improvement within the UI. This method can aid in assessing both the appropriate amount of real-life integration and the quality of said integration so the VR intervention can best accomplish its intended purpose. If the integration between virtual and real-life tools is insufficient, it has been shown that this friction will cause users to prefer the No VR option, which may also be partially related to physical performance (see Physical Performance Evaluation section) [34].

A heuristic evaluation is a UI approach that involves several topic experts or an expert evaluator, rather than soliciting direct user feedback. A VR usability expert will typically evaluate a UI’s design against an accepted set of usability principles or standards already published in the literature [48]. While there are several sets of accepted standards or heuristics, for traditional UIs little research exists on defining heuristics for VR environments. Nielsen’s [21] heuristics set is the most commonly referenced and utilized set of heuristics for UI design. Sutcliffe and Gault [48] further defined a set of 12 heuristic guidelines based on Nielsen’s set, as shown in Textbox 1.

Expert results are then aggregated and used to identify priority areas of action [28]. Heuristic assessments also require a set of tasks for the experts to experience. The nature of these tasks and the VR environments themselves should also be subjectively considered when carrying out a heuristic assessment, given the lack of standardization between various types of VR equipment and software [28]. While not all heuristics may apply to a given VR application, such an evaluation has great potential to glean a rich, overall picture of the state of the application. For example, if the VR application is intended to be designed in a way that the user is automatically placed in an “inescapable” environment, then there is no relevance in assessing clear entry and exit points (ie, the eighth heuristic guideline, clear entry and exit points) [28]. Since heuristics are broad rules of thumb rather than specific guidelines, they should not be treated as binary checkboxes, but rather as individual continuums that can each be an area for improvement, although binary elements may exist within. To illustrate, perhaps the heuristic guideline of realistic feedback is of particular interest, which outlines that the VR application should help users effectively recognize and recover from errors [21]. The presence or absence of a feature such as, for example, tangible error messages would constitute a binary checkbox, but the palatability and effectiveness of said error messages would be of higher importance. Is the problem or error precisely and concisely indicated? Is a potential solution suggested? Is the language user friendly and free of codes or abbreviations, such as “A 50 (0x32) error has occurred”? Ultimately, considering and tracking multiple granular elements within each heuristic will aid greatly in obtaining actionable results to direct improvement.