Hearing loss is a major health concern for the global society due to its negative consequences on individuals’ lives. These consequences include, but are not limited to, communication, employment, cognitive decline, social participation, and quality of life [1-5]. Hearing loss primarily affects communication, and for those who are diagnosed with sensorineural hearing loss, a prescription of hearing aids is typically the first step of the aural rehabilitation process [6]. Hearing aids amplify sounds and provide various features (ie, noise reduction) to substantially mitigate the negative consequences of hearing loss by improving audibility. However, even with these advancements, there is a gap between clinical assessment and real-world performance [7-13], such as the wearer’s complaint of persistent hearing difficulties in noisy situations [14-17].

One contributing factor for this issue could be limitations of current measurement tools. Taylor [18] mentioned difficulties in constructing a laboratory environment that closely replicates real-world settings and in measuring individuals’ unique auditory environments. In clinical practice, aided threshold and speech perception testing is often performed to assess the benefits provided by hearing aids. Aided threshold testing involves presenting warbles tones (250 Hz to 8000 Hz) through a loudspeaker in the sound field [19]. The patient is asked to respond (ie, “say yes or press the button”) when he or she hears the tone, even if the tone is soft. Speech testing is also performed in the sound field with one or more loudspeakers [19]. Words and sentences can be used as test materials and the patient is asked to listen and repeat words and sentences back to the tester. Some outcome measures include noise and multi-talker conditions to simulate real-world auditory environments, but they lack an important piece of information that people use for communication: visual cues. In real-world conversations, nonverbal cues, such as lip movements, are readily available and their significant influence on speech perception has been demonstrated in previous studies [20-25]. Summerfield [21] examined changes in the accuracy of phonetic perception in noise depending on the amount of visual information given to 10 listeners with normal hearing (NH). The participants heard a total of 125 sentences with 100 keywords and repeated the keywords under various conditions. Participants’ overall speech performance was the best when they were able to see the whole face of the speaker (65.3%), followed by the lips (54.0%), the four points (30.7%), nothing (22.7%), and a circle (20.8%). A more recent study investigated the speech perception performance of 77 NH participants who were divided into five age groups under three conditions: auditory only, visual only, and audiovisual. The highest accuracy rate was observed for the audiovisual condition, followed by the auditory-only and the visual-only conditions [26]. Benefits of audiovisual integration are well noted in literature and, yet, there are no audiological evaluation tools that use visual cues. Thus, even with well-programmed devices and good test results, hearing aid wearers often do not perceive this benefit in the real world. This mismatch reduces device satisfaction and can ultimately result in discontinuance of hearing aid use [7-9,11-13,27,28]. The MarkeTrak survey conducted in 2000 by Sergei Kochkin reported poor benefit as one of the top 10 reasons for not using hearing aids [27]. Results from previous studies emphasize the need for closing the gap between clinical assessment and real-world performance.

With the emergence of the fourth industrial revolution, researchers and industries have been putting in efforts to fuse technologies and health care. Among these technologies, virtual reality (VR) has been applied not only to gaming but to education and health care. There are five key components of VR systems: virtual space (VS), immersion, interactivity, creators, and users [29]. To be more specific, there is an imaginary place, VS, and through interaction and immersion, individuals feel more present, or connected, to the VS. VR’s biggest strength is that auditory and visual information are provided simultaneously to generate realistic environments. There are studies showing the efficacy of VR in certain areas, such as pain management, stroke rehabilitation, and chronic subjective tinnitus [30-34]. For audiology, VR has been researched for sound localization, but research into the effect of VS on speech performance has been sparse [35-37].

Ahrens et al [37] tested the sound localization ability of 10 NH listeners under eight conditions involving blindfolding, a head-mounted display (HMD) headset, virtual and real environments, loudspeakers, acoustic or visual stimuli, and a simulated laser pointer. The results revealed that the headset had a negative impact on individuals’ sound localization ability, as differences in interaural time and levels, which are important cues for sound localization, were larger when wearing the headset. Azimuth and elevation errors decreased when the source locations were visible to the participants in both the virtual and real environments. Sechler et al [38] explored the potential use of VR in sound localization testing among bilateral cochlear implant users. A total of 12 NH listeners and four bilateral cochlear implant users performed sound localization testing in a VR environment that was created for the study. A total of 13 sound cues were presented and the participants selected where they heard the sound cues in the VR environment. Bilateral cochlear implant users completed testing under first implant–only, second implant–only, and both implants conditions. Comparing the localization performance, individuals with NH showed better performance than bilateral cochlear implant users as to response time, left or right discrimination, percent correct, and root mean square error. Better sound localization performance was observed under the implants condition and the first implant–only condition. Overall, both studies suggest VR’s potential to be employed in clinical audiology.

The purpose of this study was to investigate the impact of VS on speech-in-noise performance and its feasibility as a viable instrument for speech testing in clinical practice. Findings from this study will shed light on VR’s potential to overcome the limitations of current assessment tools and, ultimately, to be utilized in clinical practice, which is an unexplored territory in the field of audiology.

In our study, speech recognition improved with the provision of visual information, regardless of the presence of hearing loss. Hearing aids facilitated better speech recognition with lower listening effort for hearing aid wearers. All of these findings are consistent with previous literature [44-49]. After summarizing participants’ subjective responses, we saw that the quality of the VS was excellent, which was demonstrated by high scores for audiovisual synchronization, audio quality of the recorded sentences, immersion, and the amount of reality reflected in the VS. A café was an appropriate place to use as the VS. Participants were interested in the new testing method (ie, VSHMD), and a high percentage of participants showed inclination toward completing the test once available in clinic.

Our study is meaningful in terms of diversity of participant characteristics, a relatively simple test environment, and a more naturalistic stimulus. Most studies utilizing VR for speech performance recruited individuals with NH and HI [36,50-52] and involved a test setup that may be difficult to establish in clinical settings. For example, in Salanger et al [50], 40 children and 8 young adults with NH were enrolled in the study. Acoustic treatments (ie, acoustic wall and ceiling tiles) and objects (ie, chalkboards) were included to create a VR 3D classroom. Video recordings of the talkers, which were less naturalistic, were presented to the participants. Hendrikse et al [51] also recruited 14 young NH listeners for localization and speech performance testing with animated characters as test stimuli. A 16-loudspeaker array and a projector were used to present auditory and visual stimuli, and a metal frame covered by a cloth was used to reduce environmental sounds, light, and room reflections [51]. Setting up such test environments in clinical practice may be challenging, as they require a number of loudspeakers, large space, and other necessary materials for the creation of a realistic environment. Our study, on the other hand, is the first study to include hearing aid users with a more naturalistic stimulus and a potentially more implementable test setup for clinical practice. The VS presented in this study was created using a real café and actors instead of avatars. The testing was performed using a relatively fewer number of loudspeakers (five) and, yet, the results were comparable to that obtained in previous studies.

Incorporation of visual information into speech testing can be beneficial for both patients and professionals. Since communication entails visual and auditory information, this type of testing could assess communication difficulties in conjunction with speech recognition performance more accurately. Patients would be more engaged in testing since the test is more interesting, as reported on the questionnaire. For hearing aid users, reality-reflected test results could foster realistic expectations, ownership of hearing loss, and better optimization of the devices. This would lead to increased satisfaction toward the device and reduced hearing aid return and discontinuance rate. If hearing aid wearers experience higher device satisfaction and perceived hearing aid benefit, the number of clinic visits for further adjustments would also decrease, which can be a critical issue for individuals who live far away from hospitals and clinics.

Although a VS is shown to be beneficial for speech recognition in noise in this study, ample work is still needed to address some limitations of our research. Each K-HINT list was broken down into two separate lists so that hearing aid users could complete tests under unaided and aided conditions. It is highly likely that phoneme distribution was affected during this process and, therefore, test materials with more sentence lists need to be used for subsequent studies. The weight of the headset also needs to be improved. The authors believe that it is crucial to not only examine the effect of visual cues on speech performance but to test the device that will be used, as it could be one of many factors that professionals and patients would consider before employing and performing the test in clinics. The weight of the system was reported to be heavy on the questionnaire and was mentioned as a weakness of the VSHMD condition. Use of a lighter device could possibly address this concern. Another concern was that individuals who are unfamiliar with the HMD system might have difficulty performing the test. Designing user interfaces that are easy to use and providing tester assistance regarding the HMD system before and during testing could address the issue. It is also worth mentioning that in-depth investigation as to the amount contributed by each sensory modality for speech-in-noise performance is necessary. Gonzalez-Franco et al [53] examined the impact of selective attention on individuals’ speech perception when visual and auditory cues were asynchronous. Two speakers were simultaneously speaking sentences, and participants were asked to recall the “target CALL,” which consisted of eight words (“Arrow,” “Baron,” etc), and to remember the content of the target sentences under four conditions (ie, synchronized visual and audio cues; auditory only with no visual information; asynchronized visual and audio cues, in which the target speaker’s lips matched the audio of the other talker; and asynchronized visual and audio cues, in which the target speaker’s lips did not match any talker’s audio). Participants were able to identify the “target CALL” more accurately when auditory and visual information was synchronous. In terms of remembering the content of the sentences, more errors were observed with asynchronous information, especially when the target speaker’s lips matched the audio of the other speaker, demonstrating the dominance of visual cues [53]. Measuring one’s reliance on each sensory system might allow researchers to recognize whether a test containing visual cues reflects one’s speech performance in a real environment; if one’s communication is actually interfered with by visual signals occurring naturally in real life and the test scores are poor, this might mean that the test is reflective of his or her real-world performance. In addition, vision screening was not performed prior to testing. Although the authors made sure all participants were able to clearly see the VS for the VSHMD condition, as the rationale behind the experiment is visual and auditory input representing real-world conditions, it is necessary to include vision screening. There is a possibility of different hearing aid settings affecting the HI group’s speech performance. As mentioned earlier, the authors did not make any changes to the hearing aid settings because those are the settings that are used by hearing aid users in the real world. However, some features, such as noise reduction, might have influenced the results of the HI group. It is worth noting that in-depth investigations regarding the actual impact of VR audiological testing in clinical practice is necessary. For example, the Technology Acceptance Model is commonly used for implementation-focused research to examine user acceptance of information technology by evaluating individuals’ willingness to use technology, perceived ease of use, and so on [54]. It is important to not only compare performance but also assess end-user acceptance of VR audiological testing to fully understand how VR audiological testing works and compares to other testing methods. Further studies with larger sample sizes, a larger variety of participant characteristics, and correlational analysis between speech performance with visual cues and standardized hearing aid questionnaires would be beneficial. Development of sentences that are appropriate for the VS and examination of their effect would be valuable in taking a step forward toward the development and standardization of reality-reflecting test methods and materials. In sum, we hope our findings open up opportunities for future studies and support the necessity of VR in being utilized in the field of audiology. It might still be challenging to set up a test environment that closely resembles individuals’ everyday listening environments and to accurately evaluate one’s unique hearing difficulties and needs. However, VR audiological testing would be another way for professionals to serve diverse clinical populations more competently.