A total of 22 unimpaired participants took part in the experiment. The participants were all university students aged between 23 and 27 years without known sensorimotor, cognitive, visual, or auditory impairments, including 11 women and 11 men. Handedness was not an inclusion criterion; however, task interaction was restricted to the participant's right hand due to the placement of electrodermal activity (EDA) sensors on the left hand. While EDA data were recorded, these were excluded from the analysis as hand movements caused frequent data corruption for most participants.

Participants were recruited from TU Delft, the Netherlands, via word of mouth (convenience sampling). The experiment was conducted in a laboratory at TU Delft between October 27, 2023, and November 3, 2023.

The sample size of 22 participants was based on our previous study [14], which used a comparable within-subjects design and task with 24 participants. No formal a priori power analysis was conducted. A post-hoc sensitivity analysis indicated that, at the Bonferroni-corrected significance level (α=.008) and 80% power, the design could reliably detect within-subject effects of |d_z|≥0.81 in a 2-tailed paired t test (df=21).

The study was approved by the Human Research Ethics Committee (HREC) of TU Delft under approval number 3417. All participants provided written informed consent for participation in the study and for the collection and use of their data, including physiological recordings (electrocardiography [ECG], eye-tracking, and EDA), audio and screen recordings, and questionnaire responses. Participant data were de-identified using numerical codes, signed consent forms were stored separately in a locked cabinet, and only the research team had access to the study data. Participants received a 15-euro gift card as compensation. No adverse events, such as motion sickness requiring session termination, were reported. No identifiable images of individual participants are included in the manuscript or supplementary material.

Participants were not informed of the specific condition before each trial, except for the mental low and mental high conditions, which required pretrial instructions for the concurrent n-back task. Visual and auditory manipulations became apparent once the trial began. The researcher administering the experiment was aware of all condition assignments. No formal blinding of outcome assessors was implemented, as all dependent measures were recorded automatically via the HMD, eye tracker, and ECG system.

The experiment was conducted in a laboratory room with a walkable area of approximately 5.0 m × 2.5 m (12.5 m²), providing enough space for participants to perform physical movements without risk of collision.

The experimental setup featured an HTC VIVE Pro Eye HMD. This HMD provided a 110-degree field of view, dual OLED screens with a resolution of 1440 × 1600 pixels per eye (2880 × 1600 combined), and integrated eye-tracking with a native sampling rate of 120 Hz. The setup also included 1 HTC VIVE tracked controller (HTC VIVE Pro Controller 2.0) and 2 base stations. Eye-gaze data, participants' actions, and HMD tracking data were all logged once per rendered frame at approximately 86 Hz and interpolated to a constant frequency of 100 Hz using nearest-neighbor interpolation. Each frame, Unity’s raycasting engine assigned the gaze to a single AOI based on the nearest object intersected by the gaze ray. No temporal smoothing was applied to AOI labels in the statistical analysis. Frames with gaze tracking loss, identified by a null ray direction vector, were classified as “Tracking loss”, a category separate from the 4 AOIs and from “Other.” “Other” comprises gaze on nontask elements (eg, characters) and valid gaze rays landing outside all AOIs. This distinction is described further in data diagnostics.

We developed the virtual environment in Unity (v2021.3.11f1) and displayed it via the SteamVR plugin (v2.7.3). Eye-tracking data were collected within Unity using the integrated Tobii XR and SRanipal SDKs. The experiment used a virtual supermarket environment modified from Cucinella et al [14], populated with product and character assets (eg, onions, bread, NPC populator, low poly animated people packages) from the Unity Asset Store. Auditory stimuli (soundtracks from Envato Elements, Pixabay) were delivered through the HTC VIVE Pro Eye headphones. The participant's view was screen-recorded using Windows Xbox Recorder.

ECG data were acquired at 1024 Hz via Bluetooth using BioTrace+ software (MindMedia, NeXuS-4) with electrodes in a Lead II configuration. The experiment ran within the Unity Editor on a Windows 10 64-bit PC (AMD Ryzen 9 5900X CPU @ 3.70 GHz, 32 GB DDR4 RAM, NVIDIA GeForce RTX 3080 GPU).

Participants engaged in a virtual shopping task, collecting products from 4 shelves in front of them according to a sequentially ordered list of 30 products, and placing them in a cart (Figure 1). Each trial lasted 122.4 seconds.

To investigate how external task demands influence task performance, perceived workload, physiological activity, and behavioral strategies, we varied the order and locations of products across trials. The 4 identical, equidistant shelves contained 33 or 34 products (6-11 per shelf), depending on the condition. Each shopping list contained 30 items distributed across 4 shelves; in some conditions, 1 or 2 products appeared twice on the list. The listed subset, list order, and product-shelf assignments varied across conditions but were fixed across participants within a condition. Consecutive list items were grouped in sets of 4, 1 per shelf. This constraint held for positions 1-16 in all conditions and for positions 1-28 in baseline. With a mean of 19 products collected at baseline, the majority of task execution fell within the balanced range.

Participants could move freely and used the controller button (hold to grab, release to drop) to interact with products via a visible virtual right hand (no virtual body was shown). In a within-subjects design, all participants performed 7 conditions, whose order was randomized per participant.

The experiment included 7 conditions, summarized in Tables 1 and 2. The baseline condition consisted of the shopping task without any additional stimuli and was used as the reference condition for all comparisons. The 6 experimental conditions added visual, auditory, or cognitive demands on top of the same core shopping task; the core task procedure was the same across all conditions.

All experimental manipulations were delivered as programmed in Unity. No technical failures affecting the visual, auditory, or cognitive manipulations were observed during data collection.

The following dependent measures were calculated per trial:

After providing informed consent upon arrival, participants received instructions for the shopping task. They learned the primary goal was to read products from a list, retrieve them from designated shelves using the controller in their right hand (with specific button presses for grab/release), and place them into the shopping cart. Participants were asked to keep their left hand still, avoid throwing products, follow the list chronologically, and not search outside the specified areas. It was pointed out that the list was intentionally long and not meant to be completed, and that they should not rush. They were also informed that products would vanish from the cart shortly after placement, advised about potential motion sickness, and reminded that they could withdraw at any time.

A familiarization session before putting on the HMD specifically addressed potentially confusing products. Examples given were a blue foil pack, a Fanta bottle labeled “Santa,” and leeks, which the researchers noted might be an unfamiliar term for Dutch participants. The need to pay attention to color was also highlighted, using different colored yogurts as an example.

Participants then self-applied ECG electrodes (lead II configuration), and the researcher started screen-recording the HMD view. Once the HMD was fitted, the eye tracker was calibrated. During calibration, the virtual height was set to 185 cm to ensure consistent visibility across participants, particularly for shorter individuals viewing top-shelf products.

An initial practice trial of approximately 1.5 minutes allowed participants to acclimatize to the environment and controls without task demands. Subsequently, participants underwent 7 experimental trials, each featuring a different condition (eg, visual low and auditory high) presented in random order. Eye-tracking recalibration occurred before each trial. For the n-back trials, participants were briefed on the concurrent task requirements, informed of its equal priority with shopping, and provided an audio example.

Posttrial procedures involved removing the HMD and completing a condition-specific NASA-TLX questionnaire via Qualtrics. A final NASA-TLX for item weighting was completed after the last trial, although these weights were not used in the analysis.

The experiment concluded with a gift card compensation. Each participant’s session took roughly 75 to 90 minutes.

No participants were excluded after data collection. Heart rate data were unavailable for 3 out of 154 trials (3/154, 1.9%; participant 22: visual low and visual high; participant 12: auditory low), likely due to electrode displacement. Given the small proportion of missing data, no imputation was performed; these trials were excluded pairwise for heart rate analyses, resulting in 20 degrees of freedom for the affected comparisons. Little’s missing completely at random (MCAR) test, conducted across the 8 trial-level dependent measures (MDE, performance, heart rate, head-stillness duration, and the 4 gaze AOI durations), did not reject the MCAR assumption (χ²₇=5.26; P=.63). IBIs were screened for artifacts using the Berntson et al [36] criterion with the placement parameter set to 0.75 (ie, 75% from the maximum expected difference toward the minimum artifact difference), resulting in the rejection of 902 of 27,063 IBIs (3.3%) across all trials. One participant had substantially higher rejection rates in individual trials (up to 66.5%), though their remaining valid IBIs were retained for analysis.

Gaze tracking loss occurred on a mean of 5.3% of frames per trial (SD 2.8%; mean 561 of 10,516 frames per trial at the ~86 Hz logging rate, across 154 trials). In total, 86,364 of 1,619,535 frames (86,364/1,619,535, 5.3%) were affected. These frames were classified as Tracking loss, separate from the other gaze category, which includes only gaze with a valid ray direction that did not intersect any AOI. Tracking loss was similar across conditions (range 4.7%-5.2%) except in the mental high condition (7.8%), where it was significantly elevated relative to baseline (baseline: mean 5.83, SD 2.72 s; mental high: mean 9.41, SD 4.38 s; mean difference 3.58 s, 95% CI 1.84-5.31; t₂₁=4.29; P<.001). An episode-level analysis based on episode duration showed that across all conditions, 4213 of 7339 tracking-loss episodes (4213/7339, 57%) lasted 100-400 ms, consistent with the typical duration of spontaneous eye blinks. Under mental high, the elevated tracking loss reflected increases across multiple duration categories: typical blinks (100-400 ms) rose from 4.56 seconds to 6.39 seconds per trial (t₂₁=3.64; P=.002), long blinks (400-1000 ms) rose from 0.37 seconds to 1.51 seconds (t₂₁=3.26; P=.004), and very long gaps (>1000 ms), absent in all other conditions, contributed 0.47 seconds per trial. By contrast, the duration of short jitter episodes (<100 ms) did not increase significantly (mental high: 1.07 s vs baseline: 0.98 s; t₂₁=0.73; P=.48).

No statistical outliers were removed. The degree of nonnormality of the paired difference scores (condition minus baseline) was assessed descriptively. Of 48 comparisons, 31 had |skewness| < 0.5, 8 had |skewness| between 0.5 and 1.0, and 9 had |skewness| > 1. The most skewed comparisons involved gaze on other elements (auditory low: 3.06; mental low: 2.24; mental high: 2.06) and heart rate, auditory high (2.23). In each case, the skew was traced to a single participant with an extreme difference score; excluding the respective participant did not change the significance of any comparison for gaze on other elements or heart rate. The 2-tailed paired-samples t test is relatively robust to outlier contamination [37].

To quantify the size of the differences between conditions, we calculated Cohen d (measuring the overall difference between conditions) and Cohen d_z (measuring the degree of change within participants). Since we performed 6 comparisons against the baseline, we applied a Bonferroni correction, adjusting our significance threshold to P<.008 (α=.05/6) to control the risk of false positives. As a robustness check, we fitted linear mixed-effects models for each dependent measure (Multimedia Appendix 1). Model B corrects for presentation order (trial position 1-7) as a linear covariate to control for learning or fatigue effects across the experimental session. Effect sizes were computed following the convention of the emmeans package [38]. Full results are reported in Multimedia Appendix 1.

While the above dependent measures provide global indications of participants’ behaviors, understanding how they adapted their strategies requires examining their behaviors more closely. Gaze behavior on AOIs, task performance, and product-grabbing actions were used to create “behavioral charts” for a representative participant in the baseline, visual high, auditory high, and mental high conditions. To reduce high-frequency jitter in the visualized AOI labels, a 1-dimensional mode filter with a window size of 19 samples (190 ms) was applied. This chart shows how either the sensory or the cognitive demands affected a participant's shopping routine of reading the shopping list, locating products on the main shelves, and grabbing and placing them in the cart. The x-axis represents the elapsed time within the trial in seconds. The y-axis categorizes gaze targets (ie, list, main shelves, and cart) and activities (ie, grabbing products). These actions are visualized in the form of colored bars of variable length, corresponding to the duration of each behavior. Additionally, we analyzed the percentage of time spent looking at the shopping list as a function of normalized time between consecutive product placements to examine differences in the participants’ strategies compared to the baseline.

Table 3 shows descriptive statistics (ie, mean, SD) calculated over the 22 participants and the 2-tailed paired-samples t test results for each dependent measure (ie, MDE, performance, heart rate, head-stillness duration, gaze duration). We also included Cohen d and the within-subject effect size Cohen d_z, as well as the mean difference from baseline with 95% CIs.

The mental high condition produced the largest and most consistent effects, with statistically significant differences from baseline on all dependent measures except heart rate, gaze on the cart, and gaze on other elements (Table 3). The mental low condition also showed significant effects on MDE, heart rate, gaze on the shopping list, and main shelves.

In the 2-back task, 38 items were presented, of which the first 2 items in each block were not scorable, yielding 30 scorable items. Mean accuracy was 19.1 (SD 6.3) out of 30 scorable items (19.1/30, 64%; scored for 21 of 22 participants; audio recordings failed for 1 participant), confirming that the task was cognitively demanding.

For some dependent measures, d_z was larger than d, notably heart rate (Figure 4B). Among the sensory conditions, only the auditory high condition significantly increased MDE, and only the visual conditions significantly redistributed gaze (for visual high: less on main shelves; for visual low and visual high: more on other elements, ie, likely the characters). No sensory condition significantly affected performance.

Figure 5 illustrates the average percentage of time participants gazed at the shopping list during the interval between placing consecutive products in the cart. Participants during the baseline condition showed a peak of attention toward the shopping list early in the interval, which then dropped, indicating they likely encoded the next product and focused on searching for the product on the shelves. Conversely, participants in the mental high condition maintained a relatively higher percentage of gaze time on the shopping list throughout the mid-to-late stages of the interval, indicating more frequent rechecking or consultation under increased mental demand.

Behavioral charts in Figure 6 show, over time, the gaze patterns on the shopping list, the main shelves, and the shopping cart, as well as the time products were held, for a single participant (Participant 18, selected because their baseline performance was close to the group mean) in the baseline, visual high, auditory high, and mental high conditions.

Observations from the behavioral charts indicate that participants in the baseline condition generally followed a consistent strategy: first checking the shopping list to identify the target product, then locating it on the main shelves. After locating the product, participants grabbed it and placed it in the shopping cart. However, we observed some variations: some participants reduced the frequency of checking the shopping list, collecting multiple products consecutively before revisiting it, while others checked the list for the next product while placing the previously collected product in the cart. As illustrated in Figure 6, this general strategy remained largely the same in the auditory high condition, with gaze durations and products collected comparable to the baseline. In the visual high condition, the overall collection strategy was similar, but gaze was redirected away from the shelves (see also Table 3), suggesting that the visual distractors competed for attention without disrupting task execution. Yet, in the mental high condition, participants’ behaviors changed considerably: there were fewer grabs (reflecting lower performance, as reported in Table 3), more frequent revisits to the shopping list, and a longer overall proportion of time spent attending to the shopping list compared to the baseline (also observed in Table 3).

To further understand participants' behavior and strategies, we qualitatively analyzed video recordings showing their first-person perspective from the HMD during the shopping task. This analysis helped identify specific behaviors within the baseline and mental high conditions that could not be seen directly in the behavioral charts. For example, in the mental high condition, we observed that many participants adopted a step-by-step approach, prioritizing either the shopping or the 2-back task rather than performing both simultaneously; they would pause 1 task to execute the other. In 2 participants, we observed them using a finger-pointing strategy, presumably to assist with checking the shopping list (Figure 7), whereas in another case, 1 participant switched from English to Dutch (their native language) when performing the 2-back task.

This study examined how visual, auditory, and cognitive demands at 2 intensity levels influenced perceived workload, task performance, heart rate, gaze behavior, and head movement during a virtual grocery shopping task in immersive VR. The results are discussed below in relation to the 2 hypotheses.

Hypothesis 2, which predicted that cognitive demands would negatively affect task execution, was supported. Under mental high, the 2-back task substantially increased MDE, reduced task performance, prolonged head-stillness, and shifted gaze away from the shelves toward the shopping list. The 0-back condition showed a similar but attenuated pattern. This interference likely reflects competition for working memory, as both the n-back and the shopping task require maintaining and updating information in memory, creating a central bottleneck despite their different input and output modalities [15]. Heart rate was significantly elevated under mental low demand; the mental high effect did not reach corrected significance in the 2-tailed paired-samples t test, possibly because reduced physical activity under mental high partly offset any cognitive heart rate increase. Both mental conditions involved speaking aloud, which may also have contributed to the heart rate elevation relative to baseline.

Hypothesis 1, which predicted that high sensory demands would primarily distract attention and increase head movement, was only partially supported. Visual distractors redirected gaze toward other elements without reducing performance. Auditory demands increased self-reported MDE but had limited effects on behavior or performance. Neither modality increased head movement. Unlike the n-back task, these sensory manipulations were passive distractors that did not compete for working-memory resources, which likely explains their limited behavioral impact.

In the visual high condition, participants spent significantly less time gazing at the main shelves and significantly more time gazing at other elements compared to the baseline. A similar pattern was observed in the visual low condition, although there the decrease in gaze on the main shelves did not reach statistical significance after Bonferroni correction. These findings suggest that the nontask-relevant visual elements captured attention at the expense of task-relevant areas. However, this gaze redistribution did not affect performance or perceived workload. Contrary to the head-movement prediction in hypothesis 1, head-stillness duration did not differ from baseline under any visual or auditory condition. This pattern may be explained by perceptual load theory [39]: if the shopping task involved low perceptual load, spare capacity would allow processing of peripheral elements, explaining the gaze redistribution. The absence of a performance cost may reflect that the time spent glancing at distractors was short enough to be absorbed without reducing the total number of products placed in the cart. In the auditory high condition, participants reported significantly higher MDE than at baseline. The spatialized, semantically varied, and dynamically changing sounds from varying distances were likely difficult to ignore, imposing additional cognitive effort to sustain attention on the core task and thereby increasing perceived workload. Although task-irrelevant sounds and demanding acoustic conditions can affect performance in other settings [12,40], the auditory demands in this study did not significantly affect shopping performance, visual attention distribution, or head-stillness.

Mean heart rate was approximately 90 bpm across all 7 conditions, compared with approximately 105 bpm in [14], where participants were instructed to work as accurately and quickly as possible. The present instructions not to rush likely reduced the physical intensity of the task. Because heart rate reflects both physical [23] and mental demands [25], interpreting condition differences requires considering both components. Under mental high, participants collected significantly fewer products (~11 vs ~19 at baseline), meaning they probably moved less. This reduced physical activity may have suppressed heart rate, partly offsetting any increase associated with higher cognitive load and helping to explain why the raw mental high vs baseline difference was modest. An exploratory linear mixed-effect model adjusting for the total number of products placed in the cart suggested that a cognitive component may remain after accounting for reduced number of products collected (Table S3 and Figure S1 in Multimedia Appendix 1). However, the model estimates heart rate at product counts that rarely occurred within each condition and should be interpreted as an extrapolation-dependent sensitivity analysis. Speaking aloud in both mental conditions could also have contributed to their heart rate elevation relative to baseline, as speech has been shown to increase autonomic arousal [41]; the present data cannot fully separate this speaking component from a cognitive one. Converging evidence for a cognitive contribution comes from our earlier study [14], in which all participants collected the same number of products by design; heart rate was still significantly higher under cognitive demands (~108 vs ~102 bpm), supporting a cognitive effect that does not depend on the product-adjusted model.

The behavioral analysis revealed several adaptations to increased cognitive demands. Under mental high, participants tended to prioritize either the shopping or the 2-back task rather than performing both simultaneously, a pattern broadly consistent with the compensatory task-serialization strategies described by Boer et al [42] in older drivers at intersections. Participants in the mental high condition also frequently looked back at the shopping list and, in some cases, pointed toward it with their virtual hand (Figure 7). Despite collecting fewer products, participants spent substantially more time gazing at the shopping list than at baseline, suggesting that the increase in list-checking reflects cognitive load rather than simply being proportional to the number of products collected. This increased list-checking may reflect cognitive offloading, where individuals rely on external information to reduce demands on working memory [33,35]. The finger-pointing behavior observed in some participants may have served a comparable function, as pointing can support the processing of spatially separated information [34] and gestures have been shown to reduce working memory load [43,44] and improve comprehension [45], consistent with cognitive-load theory principles [46]. Gaze on other elements did not differ significantly between mental high and baseline. However, tracking loss duration did increase significantly under mental high. This suggests that high cognitive load may have increased spontaneous blinking, in line with research showing that blink rate may increase under higher cognitive load [47,48]. Yet, hardware tracking loss is an imperfect blink proxy, as gaps can also reflect headset slippage, extreme gaze angles, or downward gaze toward the cart. Only the 100-400 milliseconds range corresponds to canonical blink durations; the longer gaps observed under mental high are more susceptible to these alternative sources.

Prolonged gaze on AOIs may also reflect increased processing demands, consistent with research showing that higher cognitive load affects oculomotor behaviors such as fixation duration [27-29,49]. Head-stillness also increased significantly under high mental demand. However, because participants collected fewer products and therefore moved less in the mental high condition, this increase may partly reflect reduced physical task activity rather than a purely cognitive effect. Monitoring such behavioral adaptations may serve as a noninvasive method for estimating user state in IVR, as eye-tracking can provide behavioral information in IVR [50], and IVR behavioral metrics, including head and hand movement patterns, have been proposed as objective indicators of cognitive load [51]. Cucinella et al [14] previously noted that physiological measures exhibit large individual differences and require person-specific baselines to be practical. Combining performance metrics with observable behaviors such as gaze patterns and pointing gestures could offer a more accessible approach to workload assessment, potentially enabling IVR systems to adapt task complexity in real time to the user’s capacity.

This study has several limitations. First, the sample consisted of 22 unimpaired university students, and it is unclear whether the findings generalize to other populations, such as older adults or individuals with cognitive impairments. Older adults may also use compensatory strategies to support everyday functioning [52]. In contrast, cognitively impaired populations, for example, individuals with acquired brain injury, often show altered sensitivity to visual and auditory demands, ranging from hyposensitivity to hypersensitivity [53], and may rely less on compensatory behaviors, as awareness deficits following brain injury can limit recognition of one’s own impairments and the spontaneous use of compensatory strategies [52,54,55]. Accordingly, the present findings may underestimate the impact of sensory manipulations in clinical populations. Future studies should involve larger and more diverse samples, including participants of different ages, cognitive abilities, and cultural backgrounds.

Second, product-shelf assignments were fixed per condition rather than randomized per participant. The 4-shelf grouping constraint held fully only for the first 16 list positions in the experimental conditions (the first 28 positions in baseline), though most participants collected fewer than 19 products. We cannot fully exclude that differences in product placement across conditions influenced task difficulty. Third, participants’ search strategies (eg, scanning shelves left-to-right) were not quantified. Future studies should examine whether such strategies influence vulnerability to distractions. Fourth, while the increased gaze duration on the shopping list may reflect cognitive offloading [33], alternative explanations such as forgetting the target product, losing one’s place on the list, or pausing at the list during task switching cannot be excluded. Future studies could include task-related error metrics (eg, order of collection, missing products) to more directly assess memory strategies. Finally, although the n-back task is a widely used cognitive task [56], it may not fully reflect real-world shopping demands. Future studies could adopt realistic secondary tasks, such as answering a phone call or sending a text message [57], embedded within a naturalistic virtual shopping environment.

The cognitive (2-back) task had a strong impact on perceived workload, performance, and behavioral measures, while visual and auditory demands primarily affected gaze distribution and perceived workload, respectively. Unlike prior work, this study compares 2 demand levels against a no-demand baseline and adds gaze and head-movement measures not previously examined. The absence of significant performance decrements under visual and auditory demands suggests that IVR environments can incorporate sensory richness without large reductions in task execution in this sample of unimpaired young adults, although the present design had limited power to detect small effects. Heart rate effects may have been partly masked by reduced physical activity under high cognitive load, limiting the use of heart rate as a standalone workload indicator in this task. By contrast, participants spontaneously adopted observable behavioral strategies to cope with high cognitive load, such as task serialization and increased list-checking. These behavioral adaptations, detectable from gaze and head movement data without physiological sensors, may offer a more practical approach to workload monitoring in IVR. As these findings were obtained with unimpaired young adults, their generalizability to clinical populations remains to be established. Future work in neurorehabilitation and training applications should investigate whether such behavioral markers can support the dynamic adjustment of task difficulty and environmental complexity to the individual user’s capacity.