Between June 2020 and November 2020, 12 patients with left-sided visual neglect after subacute right hemispheric stroke were recruited and included in the study.

They were inpatients at the Neurorehabilitation Clinics of the Inselspital, Bern University Hospital, or the Kantonsspital Luzern, Switzerland. All patients had normal or corrected-to-normal vision. The mean age of the patients was 58.2 years (SD 9.70 years), and 4 were female. See Table 1 for detailed information and Figure 1 for the recruitment flowchart.

The inclusion criterion was a pathological score in 1 of the following 3 assessments: Catherine Bergego Scale (CBS) [32], Sensitive Neglect Test (SNT) [35], and Line Bisection Test [34].

The CBS is an established questionnaire allowing to detect the presence and severity of neglect based on the observation of everyday life activities (cutoff for neglect: CBS score >1). The SNT is a paper-pencil cancellation task. The spatial distribution and number of missed targets are evaluated using the center of cancellation (CoC; cutoff: CoC >0.081) [33]. The CoC reflects the mean position from the center to the missed targets and is normalized to values from −1 to 1. Zero indicates no spatial bias, negative values indicate a shift towards the left, and positive values indicate a shift towards the right. Furthermore, the Line Bisection Test [34] is another frequently used neuropsychological task to assess neglect severity, with a cutoff value of ≥11% mean relative rightward deviation [34].

Somatosensory impairment, and hearing and auditory extinction were assessed clinically. Somatosensory impairment was assessed by comparing the sensitivity for touch on the forehead and the temporal head region between the left and right. To assess hearing and auditory extinction, rustling was presented to each ear individually or to both ears simultaneously, and patients’ reports were compared. Based on this assessment, patients were assigned to the following 2 subgroups: patients with and without sensory impairment. As no patient had auditory extinction, no auditory extinction group was formed.

The study was approved by the Ethics Committee of the Cantons of Bern and Lucerne (ID 2017-02195), and was conducted in accordance with the latest version of the Declaration of Helsinki. All participants signed written informed consent forms before participation.

A selection of different questionnaires was administered to assess the usability and side effects of the VR task, as previously described by Gerber et al [36,37].

For assessments of acceptance, usability, and participant perception of the visual search task and the VR system, the System Usability Scale (SUS) [38] was used. To assess side effects and cybersickness [39], the Simulator Sickness Questionnaire (SSQ) [40] was used.

The VR hardware consisted of a stand-alone head-mounted display (HMD) and a hand-held controller. During the task, the positions of the controller and the HMD were continuously recorded (20 Hz sample rate).

The resolution of the HMD (Oculus Quest, Facebook Technologies) was 1440×1600 pixels, with a horizontal field of view of 110 degrees and a frame rate of 72 Hz. For the development of our bird search task, the platform Unity [41] was used.

To monitor the patient’s behavior during the task, the patient’s view was projected to a laptop computer using a wireless streaming tool via Sidequest [42]. The experimenter was thus able to see what the patient saw in real-time and could intervene if needed.

Additionally, for the application of the tactile cue, a special cushion as an inlay piece for the HMD was used, as described by Knobel et al [29]. The cushion contains 6 coin vibrators symmetrically distributed over the forehead. Each coin vibrator could be started individually. The sound was presented via over-ear headphones.

The main goal of the bird search task was to detect the appearing birds as quickly as possible. The task took place in a virtual environment and contained 3 main features (Figure 2).

First, there was a plane surface, looking like a floor, which prevented the patient from feeling like floating or even falling. Second, directly in front of the patient, a colored fixation cross was presented. The patient’s initial position was aligned, so the fixation cross was straight ahead. Third, there were 4 presentation angles at which the stimuli (blue birds), one at a time, could appear. These presentation angles were on a horizontal line at −70°, −30°, 30°, and 70°, as defined with respect to the patient’s trunk (see the presentation angles of 4 birds in Figure 2). In this study, we will consider the 30° and −30° angles as paracentral angles and the 70° and −70° angles as lateral angles.

At the beginning of each trial (a full round in Figure 3), the patients were asked to orient the HMD toward a central fixation cross (red cross in Figure 2; Figure 3A), thereby aligning their head to the trunk straight ahead. The fixation cross had the property to turn green if aligned with the HMD, thereby giving feedback for being correctly fixated (Figure 3B). After fixating for 2 seconds on the central fixation cross, it disappeared and, at the same time, a target bird appeared (Figure 3C). Consequently, the disappearance of the central fixation cross was a signal for the patient to start searching for the target bird. The patient was asked to confirm the detection of the target bird by flagging it with a hand-held controller. A flagged bird disappeared from the scene by falling. This started a new trial (ie, central fixation cross for 2 seconds, disappearance of the latter, and concomitant appearance of the next target bird; Figure 3D). If the target bird was not found within 11 seconds, it simply disappeared, and the next trial started.

Each target bird could appear either with or without additional spatial cues. There were 2 different possible spatial cues (auditory or tactile), resulting in the following 4 different conditions: None (no cue), Audio (only auditory cue), Tactile (only tactile cue), and Combo (auditory and tactile cues; for more details, see the “Cues” section below).

In the None condition, the patient had to start looking for the target as soon as the fixation cross disappeared, but no additional hint for the direction was given. In the 3 other conditions, the patient received an additional hint from the presentation angles of the newly appeared target bird at the same time the target appeared. The cues were repeated after 3 seconds until the maximum presentation time of 11 seconds.

The bird search task was organized into 4 sessions, each containing a single spatial cue type (ie, None, Audio, Tactile, and Combo). In each session, 80 target birds were presented (ie, each of the 4 presentation angles were tested 20 times). The order of the presentation angles at which the target birds appeared was random, with the constraint that 2 target birds could not appear at the same presentation angle in 2 consecutive trials. The order of the 4 sessions was randomized. The 4 sessions took place over 2 consecutive days (ie, 2 sessions per day). On the first day, all the conditions were explained to the patient, and 3 practice trials were performed. During the practice trials, each cue type and each presentation angle was presented. First, the patient was verbally guided to the target, and if the position was not detected correctly, the corresponding practice trial was repeated up to 3 times. If the position was detected correctly, the next angle/cue was presented. After completing the practice, the first 2 sessions were performed. On the second day, the corresponding practice trials of the 2 remaining cueing conditions were repeated, and then, the 2 remaining sessions were performed.

The task included auditory and tactile spatial cues. The auditory cue was a 1-second tone of 500 Hz with 0.1 seconds fading in and out to reduce the sound’s sharpness. The spatial information of the auditory cue was generated by the audio spatializer software development tool kit that is part of the Unity game development platform [41]. The head-related transfer function of the spatializer is based on the KEMAR data set (set of pre-ear impulse recordings of a dummy head; Bill Garnder at MIT Media Lab [43]). In the task, the sound was realistically perceived as coming from the left (due to different sound volume levels between the 2 ears) if the sound source was on the left side of the head and as coming from the front if the head was oriented toward the sound source.

The tactile cue (1-second vibration) was applied using punctual vibration that came from a special inlay of the HMD that contained 6 symmetrically positioned individually controllable coin vibrators [29]. The coin vibrators were controlled in a way that mimicked the spatial behavior of sound. This means that if the head is oriented to the front and a cue is given on the left side, the most left coin vibrator is activated, signaling that the cue is on the left side. If the head is then turned and faces the cue, the middle coin vibrators are activated, signaling that the cue is now directly in the front of the patient. If the head is turned even more, the right coin vibrators are activated, signaling that the cue is now on the right side.

In the Combo condition, auditory and tactile cues were given simultaneously for 1 second each.

Statistical analyses were performed once for all patients together, and comparisons were performed between the with and without somatosensory impairment groups (both groups had no auditory extinction).

First, we analyzed the effects of the different cueing conditions and of the different spatial positions on in-game parameters by means of a 2-way repeated measures ANOVA, with cue type (levels: None, Audio, Tactile, and Combo) and angle (levels: −70°, −30°, 30°, and 70°) as within-subject factors for all participants.

Second, since it is assumed that auditory and tactile cueing require intact somatosensory processing to be effective, we aimed to assess the effect of impairment in the respective modality. For this purpose, we grouped the patients according to somatosensory impairment (yes/no) and auditory extinction (yes/no), and reran the ANOVA with group as a between-subject factor (levels: with and without somatosensory impairment), and cue type (levels: None, Audio, Tactile, and Combo) and angle (levels: −70°, −30°, 30°, and 70°) as within-subject factors.

Post-hoc analyses with least significant difference were applied for the identification of significant differences for cues within presentation angles. The study data were managed using Research Electronic Data Capture [48,49], a web-based tool to support data handling for research studies. Data analyses were performed with R (R Foundation for Statistical Computing) and MATLAB (MathWorks).

With regard to the neuropsychological measures, the mean results were as follows: 10.0 (SD 4.79) points for the CBS score [32], 0.30 (SD 0.37) for the CoC [33], and 13.6% (SD 19.0%) relative deviation toward the right for the Line Bisection Test [34].

On separation for somatosensory impairment, there were 7 patients without and 5 with somatosensory impairment of the front head. On comparing age and neuropsychological measures between the groups, there were no significant differences for age (t_9.94=0.480; P=.64) and the neuropsychological measures (CBS: t_10.0=0.376, P=.72; CoC: t_7.78=−0.289, P=.78; Line Bisection Test: t_9.99=0.343, P=.74).

Usability (based on the SUS ratings) was rated with the sum of 3 questions (responses ranging from 0 [unusable] to 4 [highly usable]) and reached a value of 10.2 (SD 1.85) out of a maximum of 12. Both groups provided similar ratings, with no significant difference between the groups (t_8.95=−1.54; P=.16).

The occurrence of side effects was assessed with the sum of 7 frequent items from the SSQ (responses ranging from 0 [no side effects] to 3 [severe side effects]), and the score was 0.833 (SD 0.834) out of a theoretical maximum of 21, indicating a very low rate of side effects. The results were not significantly different between the groups (SSQ: t_9.86=−1.77; P=.11).

This study showed that our new multimodal VR setup, including visual, auditory, and tactile cues, is highly usable and that it is suitable to provide multimodal spatial cues to patients who have spatial attention deficits. Furthermore, the measurements provided some important information concerning the peculiarities of the application of such a setup in this population. Indeed, all patients seemed to benefit from auditory cues; however, the positive effect of tactile cues on in-task performance seemed to depend on the presence of somatosensory impairment. Patients without somatosensory impairment did benefit from tactile cues. Nevertheless, there seemed to be a consistent alerting effect for all cue conditions (auditory, tactile, and combined) among all patients, which could help them to overcome the initial tendency to orient toward the right, a common symptom observed in neglect. Finally, we did not find an additive effect of combining auditory and tactile cues in any measure or group.

In line with our first hypothesis, patients rated the device’s usability as very high and did not report any relevant side effects. The high usability of our new VR setting, including tactile stimulation, is similar to that in one of the few previous studies in stroke patients with a similar visual and auditory setting, which however did not entail any tactile stimulation [14,37]. Moreover, similar levels of usability have been reported by healthy participants using new VR tools including visual and auditory stimulation, but without tactile stimulation [36,50,51]. Hence, in our study, the additional presentation of tactile stimulation did not change usability or cause discomfort. Interestingly, in our last study in healthy participants [29], tactile stimulation, similar to that presented here, caused some side effects. We hypothesize that one reason for the minor discomfort caused by tactile stimulation is the duration of application of the tactile stimulation. Indeed, in our previous study, tactile stimulation was continuously presented over several minutes, whereas in this study, it was only presented during short intervals (ie, up to a maximum of 4 seconds per target). This study and our previous study [29] thus represent in some way maximum and minimum usage of tactile stimulation. Future studies should therefore target an optimal tactile stimulation time for maximum effectiveness, but without side effects.

Our second hypothesis concerned in-task parameters and assumed that patients with spatial attention disorder would benefit from spatial cues, as assessed by means of a compound performance measure, considering both search time and percentage of found targets. Usually, those cues are presented on a 2D screen with a limited visual angle. The advantage of our VR approach is that we can present the patients with far more lateral cues in a virtual environment, supporting them in orienting toward the left by means of not only eye movements, but also head movements. Even though some tools have been developed and tested to examine the effects of visual, auditory, or even tactile cues in patients with impaired spatial attention [21,52], to the best of our knowledge, so far, no study has examined the effects of tactile cues in neglect patients using a VR system.

The auditory cueing of our VR setup was effective in all patients, increasing performance and confirming our hypothesis. The new approach with tactile cueing in VR was also effective, as long as patients had preserved somatosensory processing. This is in line with the principles of unimodal or multimodal cueing, a common approach to guide the attention of patients with spatial attention disorders toward the neglected side [24,53,54].

The fact that tactile cueing can induce positive effects in patients without somatosensory impairment is particularly remarkable. Indeed, tactile cues were presented at different locations in space than visual stimuli; the tactile cues were presented on the skin of the forehead (ie, within the personal space), whereas visual targets were presented in the peripersonal space. This configuration is different from the one concerning auditory cues, were visual targets and auditory cues are both presented at the same location in the peripersonal space [55,56]. In this group of patients, this suggests the activation of spatial attention networks through not only spared supramodal mechanisms [52,57] but also “supraspatial” (ie, entailing different spatial reference frames) mechanisms.

Our results seem to not provide support for the third hypothesis, that is, an advantage of the combined application of tactile and auditory cues in comparison with unimodal cues, since this did not lead to a significant increase in performance or a higher rate of correct early orientation. Some evidence in the literature shows that the use of multimodal cues can have an additive effect [29,31], whereas other studies have shown no benefit of using multimodal cues when compared with unimodal cues [54]. A possible explanation for these disparities could be associated with the level of spatiotemporal matching [58,59] and cross-modal intensity matching [60] needed for cross-modal cues to have an additive effect. In our study, auditory cues had a high spatiotemporal match with the visual target, but tactile cues did not [54]. This might be even more relevant for neglect patients, as they typically show problems with spatial transformations [61-63].

Besides performance parameters (eg, search time and percentage of found targets), we also assessed early orientation behavior. Interestingly, in this case, all types of cues (auditory, tactile, and combination) had positive effects on early orientation, even in patients who had a somatosensory impairment and could not correctly localize a tactile cue. This might be explained through a general alerting effect, as it has been shown that a temporary alertness enhancement can ameliorate attentional orienting in neglect patients [64-66], but this effect is short lived [67,68]. Therefore, it seems reasonable to hypothesize that all types of cues can ameliorate early orientation through their alerting effect and that this effect is not long enough to further support performance to find targets.

In this pilot study, we aimed to show the feasibility and usability of our VR setup in a diverse and complex patient group. On the one hand, the variance increases the difficulty of the interpretability of the task results, and on the other hand, the variety increases the representativeness for this patient group. Neglect severity shows spontaneous fluctuations and intraday variations [69]. We tried to minimize this effect by having the 2 sessions on consecutive days and at the same time; nevertheless, factors like medication changes and exhausting therapies beforehand could not be controlled. Future studies with a larger sample size should try to control or at least assess such confounders

This study showed the usability and feasibility of a new approach entailing auditory, tactile, and combined audio-tactile cueing in VR among patients with different combinations of attentional and somatosensory impairments. Our data suggest that auditory and tactile cues may be equally efficient in ameliorating attentional performance in neglect patients, at least in those with spared somatosensory processing. Moreover, combined audio-tactile stimulation did not show an additive effect in our setup. Future studies are needed to assess these preliminary findings in a larger group of patients. One of the possible directions to reach intermodal additive effects would be to assess the effects of tactile cues presented “nearer” to the source of the visual and auditory stimuli (eg, tactile stimulation on the hand, with variation in presentation side and intensity according to how near the patient moves the hand to the target).

Overall, audio-tactile cueing seems to be a promising method to guide patient attention. For instance, in the future, it could be used as an add-on approach that supports attentional orientation during established therapeutic approaches (eg, optokinetic stimulation).