This paper is in the following e-collection/theme issue:

Serious Games for Health and Medicine (647) Extended Reality, Virtual Reality and Virtual Worlds (520) Anxiety and Stress Disorders (1013) Social Anxiety Disorder (SAD) (49) Virtual Reality Interventions in Mental Health (111) Machine Learning (1400)

Published on in Vol 10, No 3 (2022): Jul-Sep

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/38284, first published .
Prediction of Specific Anxiety Symptoms and Virtual Reality Sickness Using In Situ Autonomic Physiological Signals During Virtual Reality Treatment in Patients With Social Anxiety Disorder: Mixed Methods Study

Prediction of Specific Anxiety Symptoms and Virtual Reality Sickness Using In Situ Autonomic Physiological Signals During Virtual Reality Treatment in Patients With Social Anxiety Disorder: Mixed Methods Study

Prediction of Specific Anxiety Symptoms and Virtual Reality Sickness Using In Situ Autonomic Physiological Signals During Virtual Reality Treatment in Patients With Social Anxiety Disorder: Mixed Methods Study

Authors of this article:

Joo Young Chun1 Author Orcid Image ;   Hyun-Jin Kim2 Author Orcid Image ;   Ji-Won Hur3 Author Orcid Image ;   Dooyoung Jung4 Author Orcid Image ;   Heon-Jeong Lee5 Author Orcid Image ;   Seung Pil Pack6 Author Orcid Image ;   Sungkil Lee7 Author Orcid Image ;   Gerard Kim8 Author Orcid Image ;   Chung-Yean Cho9 Author Orcid Image ;   Seung-Moo Lee9 Author Orcid Image ;   Hyeri Lee9 Author Orcid Image ;   Seungmoon Choi10 Author Orcid Image ;   Taesu Cheong1 Author Orcid Image ;   Chul-Hyun Cho5 Author Orcid Image
Article Authors Cited by (3) Tweetations (7) Metrics

    Original Paper

    1School of Industrial and Management Engineering, Korea University, Seoul, Republic of Korea

    2Department of Psychiatry, Chungnam National University Sejong Hospital, Sejong, Republic of Korea

    3School of Psychology, Korea University, Seoul, Republic of Korea

    4Department of Biomedical Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

    5Department of Psychiatry, Korea University College of Medicine, Seoul, Republic of Korea

    6Department of Biotechnology and Bioinformatics, Korea University, Sejong, Republic of Korea

    7Department of Software, Sungkyunkwan University, Suwon, Republic of Korea

    8Department of Computer Science and Engineering, Digital Experience Laboratory, Korea University, Seoul, Republic of Korea

    9Department of Film & Multimedia, Korea National University of Arts, Seoul, Republic of Korea

    10Department of Computer Science and Engineering, Pohang University of Science and Technology, Pohang, Republic of Korea

    *these authors contributed equally

    Corresponding Author:

    Chul-Hyun Cho, MD, PhD

    Department of Psychiatry

    Korea University College of Medicine

    73 Goryeodae-ro, Seongbuk-gu

    Seoul, 02841

    Republic of Korea

    Phone: 82 029205505

    Fax:82 0262805810

    Email: david0203@gmail.com


    Background: Social anxiety disorder (SAD) is the fear of social situations where a person anticipates being evaluated negatively. Changes in autonomic response patterns are related to the expression of anxiety symptoms. Virtual reality (VR) sickness can inhibit VR experiences.

    Objective: This study aimed to predict the severity of specific anxiety symptoms and VR sickness in patients with SAD, using machine learning based on in situ autonomic physiological signals (heart rate and galvanic skin response) during VR treatment sessions.

    Methods: This study included 32 participants with SAD taking part in 6 VR sessions. During each VR session, the heart rate and galvanic skin response of all participants were measured in real time. We assessed specific anxiety symptoms using the Internalized Shame Scale (ISS) and the Post-Event Rumination Scale (PERS), and VR sickness using the Simulator Sickness Questionnaire (SSQ) during 4 VR sessions (#1, #2, #4, and #6). Logistic regression, random forest, and naïve Bayes classification classified and predicted the severity groups in the ISS, PERS, and SSQ subdomains based on in situ autonomic physiological signal data.

    Results: The severity of SAD was predicted with 3 machine learning models. According to the F1 score, the highest prediction performance among each domain for severity was determined. The F1 score of the ISS mistake anxiety subdomain was 0.8421 using the logistic regression model, that of the PERS positive subdomain was 0.7619 using the naïve Bayes classifier, and that of total VR sickness was 0.7059 using the random forest model.

    Conclusions: This study could predict specific anxiety symptoms and VR sickness during VR intervention by autonomic physiological signals alone in real time. Machine learning models can predict the severe and nonsevere psychological states of individuals based on in situ physiological signal data during VR interventions for real-time interactive services. These models can support the diagnosis of specific anxiety symptoms and VR sickness with minimal participant bias.

    Trial Registration: Clinical Research Information Service KCT0003854; https://cris.nih.go.kr/cris/search/detailSearch.do/13508

    JMIR Serious Games 2022;10(3):e38284

    doi:10.2196/38284

    Keywords

    social anxiety; virtual reality; autonomic physiological signals; machine learning; virtual reality sickness 


    Anxiety involves uncertainty about the expectancy of a threat. It is a normal emotional response that helps individuals heed and cope with potential signs of danger [1]. Anxiety induces anticipatory stress and various neurophysiological responses. Previous research has shown that certain brain regions that process threatening information can biologically trigger stress response systems, such as the autonomic nervous system and hypothalamic-pituitary-adrenal axis. Hyperactivity of the autonomic response can induce the expression of anxiety signals, perception of sensations from inside the body, and central periphery interactions [2].

    Social anxiety disorder (SAD) is the fear of negative evaluation and embarrassment in social situations. For instance, anxiety can occur when thinking about being observed while eating and speaking publicly [3]. The physiological symptoms of SAD include heart palpitations, sweating, tremors, shaking, and blushing. Many existing studies have focused on anxiety-induced heart reactions, and have discovered that anxiety increases blood catecholamine concentrations [4] and causes excess sympathetic activation and parasympathetic withdrawal [5].

    Existing treatment methods for SAD include pharmacotherapy and psychotherapy, such as cognitive behavioral therapy (CBT) [6]. During CBT, an exposure technique allows the patient to experience and participate in a feared situation to create a natural process related to fear reduction [7]. However, individuals with social anxiety rarely request for support from professionals owing to fear of the stigma of mental health treatment. Therefore, the demand for accessibility to CBT via online content has been increasing. Moreover, the range of virtual reality (VR) treatments is expanding [8].

    Recent VR intervention research has created virtual environments, such as restaurants. Individuals undergoing VR intervention showed reduced anxiety regarding daily social interactions. They also showed a decrease in social interaction anxiety and depression, and improved satisfaction with daily life after the VR intervention [9].

    Exposure therapy is the mainstay VR treatment for SAD. Measuring anxiety symptoms in real time while applying the exposure technique enables the confirmation of actual SAD symptom levels and the appropriate application of personalized therapeutic methods.

    However, VR sickness influences user experience when using VR systems [10]. The symptoms of VR sickness include eye fatigue, disorientation, and nausea [11]. Sickness resulting from VR is referred to by many names, including motion sickness, cybersickness, and simulator sickness. These uncomfortable feelings disturb VR intervention; therefore, VR sickness is a crucial problem that must be solved. The most popular method to assess VR sickness is the Simulator Sickness Questionnaire (SSQ) [12]. The symptoms of simulator motion sickness resemble those of motion sickness, but are less severe. Three distinct symptom groups, namely nausea, oculomotor, and disorientation, were clustered across 16 symptom variables. Because simulator sickness–related symptoms differ from motion sickness patterns, a measurement system was established through this grouping. The SSQ scores based on factor analysis models provide a scale score as a good indicator of overall simulator sickness severity. This quantifies simulator sickness for activities that can lead to symptoms. However, the method cannot be used for real-time measurements because it measures psychological scales through a questionnaire after VR intervention. There are several technical approaches for overcoming VR sickness, and measuring and evaluating it in real time is crucial. This is one of the factors determining the difference in individual sensitivity and future compliance. If it is not addressed during VR intervention, patient compliance can be greatly reduced [13].

    Several studies have proposed interventions for social anxiety using VR treatment technology. The results showed that social interaction anxiety and quality of life improved when treatment was applied [9]. In particular, previous studies have shown that exposure to scenarios of various social situations through VR is effective for alleviating social anxiety symptoms [14].

    Recently, a study applied VR treatment to anxiety disorders, such as SAD [15], proposed a VR intervention for SAD treatment, and verified its clinical effect by reducing scores on the Social Anxiety Disorder Scale. This method included CBT to enable participants to overcome SAD by exposing them to a self-introduction situation. The treatment had exposure therapy for patients with SAD through VR for over 6 sessions. Compared with previous studies that applied relatively unidirectional VR content, this study applied a customized approach that divided the difficulty according to the patient’s own symptom report. However, this study used a subjective self-assessment scale, which was influenced by patient bias. Furthermore, survey-oriented progress is lengthy and may have different results based on subjective judgment.

    We have attempted to investigate SAD in relation to VR treatment using objective data such as brain imaging data [16]. Objective physiological signals, which reflect the degree of anxiety symptoms or sickness symptoms, can be assessed in real time during VR treatment sessions by several sensors. This would provide a basis for stand-alone interactive content to be provided during VR treatment for anxiety. Moreover, reducing VR motion sickness could help enhance user experience and compliance using physiological signals.

    In this study, we developed a machine-learning model to predict the severity of specific anxiety symptoms and VR sickness using physiological signal data that were measured in situ during VR treatment sessions in patients with SAD.


    Participants

    Participants were recruited through advertisements on the universities’ online sites. The inclusion criteria for the SAD group were set as follows: (1) men and women aged between 19 and 31 years using the Korean language; (2) condition meeting the Diagnostic and Statistical Manual of Mental Disorders, fourth edition (DSM-IV) criteria for SAD, as evaluated by the Mini-International Neuropsychiatric Interview [17]; (3) no psychiatric comorbidity, except for depression and panic disorder, and no experience in psychotropic drug treatment; (4) not currently receiving psychotherapy; (5) not currently diagnosed with medical or neurological disorders; (6) no history of psychotic symptoms that can be triggered by VR interventions; and (7) no susceptibility to visual stimuli. We set the exclusion criteria as follows: (1) any history of organic brain damage or intellectual disability; (2) a history of psychotic symptoms that can be triggered by VR interventions; (3) vulnerability to visual stimuli; and (4) ineligibility for participation in a magnetic resonance imaging assessment (during the study, we performed a functional magnetic resonance imaging evaluation for another project in the same subjects) [18].

    Forty individuals with SAD participated in the study. Among them, 8 individuals quit for personal reasons (eg, time limitations). Ultimately, 32 participants completed the study. All participants completed all VR sessions. The number of people who participated in the entire session was 32, but 6 of them, for whom the sensor data for a specific session were not collected because of problems such as sensor errors, were excluded from data analysis. Consequently, data from 26 individuals were used for specific anxiety symptom and VR sickness predictions after the data qualification process.

    Ethics Approval

    The study was approved by the Institutional Review Board of Korea University Hospital (2018AN0377). It was conducted in accordance with the principles of the Declaration of Helsinki. All participants were informed of the study procedure and provided written informed consent before the experiment. This study was registered on the Clinical Research Information Service (KCT0003854).

    Study Design

    VR Intervention for SAD

    The study design included VR sessions, psychological scale evaluations, and in situ autonomic physiological signal data measurements (Figure 1). A total of 6 VR sessions and 4 psychological scale assessments were performed. A recent study found that the amount of anxiety that increased during actual conversation and the anxiety that increased during a conversation in a VR environment was similar [19]. In a study comparing VR exposure therapy and real-world exposure therapy for SAD [20], 97 participants with a major diagnosis of SAD were randomly assigned to VR exposure therapy, real-world exposure therapy, or a waiting list. A standardized self-report scale showed a statistically significant improvement in those who completed active treatment than in those on the waiting list. Thus, VR exposure therapy was as effective as real-world exposure therapy for treating SAD. In this study, participants completed VR treatment sessions that included interactive content on social anxiety situations, and the VR intervention was performed face-to-face and provided individually. The intervention proceeded in a VR environment, and each participant entered a meeting room with several nonplaying characters [15]. The content had introduction, core, and finishing stages. In the introduction phase, participants selected their avatars and were informed how to manipulate the VR device to progress through the stages [21]. Participants performed a warm-up session with mediation to support their adaptation. The core stage had an intervention in which college student’s participated in social situations where they introduced themselves to each other. Participants proceeded with the treatment session by selecting the difficulty of the core level as easy, medium, or hard. In the first session, all participants started the VR intervention at the easy level. From the second session, participants freely chose the level of difficulty they wanted based on the difficulty they experienced during the treatment process. This is related to the degree of difficulty in interacting in social situations. The reaction of the nonplaying characters was based on the difficulty level selected in the previous stage. In the easy-level session, the nonplaying characters concentrated well on the patients’ self-introduction. Nonplaying characters appeared distracted in the hard level, such as yawning or engaging in chatting. The finishing stage provided cognitive and behavioral safety guidelines for SAD with text and audio information during the VR intervention. The VR headset used was VIVE (HTC Corporation). The in situ autonomic physiological signals (heart rate [HR] and galvanic skin response [GSR]) of the participants were measured during the VR experience.

    ‎
    Figure 1. Virtual reality (VR) treatment process with psychological scale measurement and physiological signal data extraction. Psychological scale assessments included Beck Anxiety Inventory, State-Trait Anxiety Inventory, Social Phobia Scale, Social Interaction Anxiety Scale, Brief-Fear of Negative Evaluation Scale, Internalized Shame Scale, Post-Event Rumination Scale, and Liebowitz Social Anxiety Scale. SSQ: Simulator Sickness Questionnaire.
    View this figure
    Data Collection and Separation

    This study analyzed specific anxiety symptoms and in situ autonomic physiological signal data. For this purpose, physiological signal data were measured in real time during 4 VR sessions (#1, #2, #4, and #6), where psychological scale tests were performed after VR treatment. Data were extracted as independent variables. The SSQ was evaluated during all treatment sessions; therefore, in situ physiological signal data for all sessions were extracted. We created a data frame with the HR and GSR data. Values ​​below zero in the HR and GSR measurements were considered outliers. Outliers were removed, and a moving average was used to reduce the noise of the data.

    Measurement

    Psychological Scales

    We measured psychological scales using subjective surveys after the VR sessions #1, #2, #4, and #6. The Beck Anxiety Inventory [22], State-Trait Anxiety Inventory [23], Social Phobia Scale [24], Social Interaction Anxiety Scale [25], Brief-Fear of Negative Evaluation Scale [26], Internalized Shame Scale (ISS) [27], and Post-Event Rumination Scale (PERS) [28] were used. After building a simple linear regression model for the scales, their subdomains, and the SSQ, the best performing group of scales based on the mean squared error of the models’ prediction results was analyzed. Subsequently, we generated a machine learning model to predict the survey results. The linear regression coefficient was included as an independent variable for prediction; therefore, the specific anxiety symptoms were selected in the order of the smallest root mean square error deviation when predicting with simple linear regression. The subdomains of the corresponding specific anxiety symptoms were also included and used for prediction. The ISS and PERS showed a good reflection of the physical response to SAD. In this study, the machine learning model used the ISS and PERS as independent variables. Multimedia Appendix 1 provides the complete regression results for our machine learning model.

    The ISS assesses the patients’ shame and self-esteem [27]. It has 30 items that measure shame and self-esteem (24 and 6 items, respectively). ISS data were measured using the ISS written in the Korean language [29]. Factor analysis involved the following 4 factors: inadequacy, emptiness, self-punishment, and fear of mistakes, with 10, 5, 5, and 4 items, respectively.

    The PERS measures postevent ruminations during social situations [28,30]. It comprises 2 scales, negative and positive rumination, with 5 and 9 items, respectively. Each answer was measured on a numerical scale from 0 (low score) to 4 (high score), with a higher score indicating that the individual frequently experiences rumination.

    VR Sickness Scale

    The SSQ was developed for VR sickness, and is the gold standard for assessing physical sickness after exposure to a simulator or VR environment [12]. The SSQ can measure 16 symptoms of VR sickness. The subdomains are grouped into nausea (general discomfort, increased salivation, sweating, nausea, difficulty concentrating, stomach awareness, and burping), oculomotor (general discomfort, fatigue, headache, eyestrain, difficulty focusing, difficulty concentrating, and blurred vision), and disorientation (difficulty focusing, nausea, fullness of the head, blurred vision, dizziness with eyes open, dizziness with eyes closed, and vertigo). The total VR sickness score was the sum of nausea, oculomotor, and disorientation group scores, with a higher score indicating a greater level of simulator sickness.

    Data Labeling and Processing

    We used k=2 to categorize the specific anxiety symptoms in the severe and nonsevere groups, where k is the number of median points ​​of the clusters to be classified. Data are grouped as per the number of hyperparameters k. In k-means clustering, the algorithm can be executed when the user determines the number of clusters. K-means clustering is a representative detached clustering algorithm that uses unsupervised learning [31].

    We labeled the severe and nonsevere groups using k-means clustering for each specific anxiety symptom and VR sickness. The ISS cutoff values for dividing the severe groups were 48 (total score), 11 (mistake anxiety), 9 (self-punishment), 10 (emptiness), and 16 (inappropriate). The PERS cutoff values were 41 (total score), 17 (positive), and 27 (negative). For the SSQ, severity classification was performed using k-means clustering via the sum of the response scores for each category. The cutoff values for dividing the severe group were 9 (total score), 7 (nausea), 8 (oculomotor), and 4 (disorientation).

    Physiological Signal Data

    We measured all participants’ HR and GSR during the VR sessions in real time. These values are closely related to in situ physiological signaling responses related to anxiety. We recorded the data using a Shimmer3 GSR+ Unit (Shimmer) with 3 channels. Two channels tracked the electrodermal skin conductance level signals via hand electrodes attached to the first and middle fingers. We monitored the fingers of the nondominant hand at a sampling frequency of 52 Hz. The third channel recorded cardiac volume data from an infrared sensor attached to the left earlobe. Data were converted into HR data using a software system.

    Preprocessing removed any sections where the HR and GSR were not measured or had negative values. The noise was reduced using the moving average method at 2-s intervals. We extracted all the data from the time series data set of the participants. We also calculated the mean, standard deviation, minimum, maximum, linear regression coefficient, maximum difference, peak ratio, and mean difference for each in situ physiological signal. These created the data frame. To calculate the peak ratio [32], we divided the number of peaks by the length of the physiological signal data.

    We used real data for the prediction model; therefore, generating data was a challenge. The severe and nonsevere groups varied in size; therefore, we manipulated the data set to improve the model performance. We did not consider the undersampling technique because the number of participants was insufficient. Conversely, we used the oversampling technique on the minority group to improve classifier performance.

    The SMOTE (Synthetic Minority Oversampling Technique) approach generated insufficient data [33]. This method helps construct classifiers from imbalanced data sets (where the classification categories are not approximately equally distributed), such as real-world data sets with a small number of abnormal cases.

    Statistical Analysis

    Group-Based Analysis

    Differences between groups (severe and nonsevere) were assessed using independent t tests in Python with SciPy version 1.5.2. Differences were considered statistically significant at P<.05. Furthermore, we analyzed correlations between the extracted variables. The variables with the strongest positive correlations were mean and peak GSR ratios. The variables with the second strongest positive correlations were the mean and standard deviation of the GSR. The variables with the strongest negative correlations were the linear regression coefficient and standard deviation of the GSR. The second most negatively correlated variables were the GSR difference and HR linear regression coefficient. All correlations are illustrated in Multimedia Appendix 2. There was a high correlation between the mean and peak GSR (0.92); however, removing this feature decreased the prediction score. Therefore, this feature was retained. All other correlations were <0.86.

    Machine Learning Techniques

    All machine learning techniques were performed using Python version 3.7.7. Logistic regression [34], random forest [35], and naïve Bayes [36] classification were performed using Scikit-learn version 0.23.2. The random forest model avoids overfitting, thereby improving model accuracy. Additionally, we can select relatively important ranks of variables in the classification model. The naïve Bayes model is a conditional probability-based classification method that calculates the probability of features belonging to each class and handles noise and missing values reasonably.

    In addition, when training, it works excellently regardless of the size of the data, and it is easy to obtain the estimated probabilities for prediction. Although 32 samples of data could be insufficient, we amplified the patients’ data using SMOTE methodology after segmentation for each session. We used machine learning models that can perform reasonable prediction with a few data points to minimize performance degradation because of insufficient data. We calculated the accuracy, F1 score (weighted average of precision and recall), and area under the curve (AUC) for the prediction.

    All machine learning models, except for the random forest model, were performed using the oversampling technique on the training data set. This supplemented insufficient data after the training-test split. We performed 5-fold cross-validation for the random forest model to improve the prediction score. During the prediction, sampling of the severe and nonsevere groups was stratified for training-test splits to ensure equally frequent severity.


    Classified Specific Anxiety Symptoms and VR Sickness

    Specific anxiety symptom and VR sickness results were compared between the severe and nonsevere groups using k-means clustering. The smallest difference between group sizes was observed for ISS self-punishment (severe vs nonsevere group, 53 vs 51). Based on the average score of the specific anxiety symptoms, the most significant difference between the groups was the ISS score (severe vs nonsevere group, 59.951 vs 33.222). ISS emptiness showed the smallest difference between groups (severe vs nonsevere group, 12.91 vs 5.75). After labeling with k-means clustering, the anxiety symptom with the lowest distortion was ISS mistake anxiety and that with the highest distortion was total ISS (1.263 vs 7.439).

    After 4 VR sessions with 26 participants, 104 sets of VR sickness data were collected. These were labeled as severe and nonsevere groups using k-means clustering. Distortion of k-means clustering was 2.57 for total VR sickness, 1.44 for nausea, 1.71 for oculomotor, and 1.16 for disorientation. Table 1 presents the statistical analysis results of the severe and nonsevere groups. The boxplot for the severe and nonsevere groups is illustrated in Multimedia Appendix 3.

    Table 1. Severe and nonsevere groups clustered by k-means clustering.
    MeasureGroupsaMinimum valueCutoff valueMaximum valueDistortionb

    		Nonsevere group countSevere group countNonsevere group meanSevere group mean



    ISSc634133.22259.9513<48927.439
    ISS mistake anxiety34708.17612.9571<11161.263
    ISS self-punishment51534.96112.4531<9201.835
    ISS emptiness60445.75012.9100<10202.104
    ISS inappropriate58469.15521.9782<16363.344
    PERSd475732.76648.14019<41644.203
    PERS positive366810.05921.0002<17353.007
    PERS negative505416.90036.0163<27565.163
    Total VRe sickness73313.24713.7100<9292.568
    Nausea group9861.78610.5000<7151.445
    Oculomotor group63412.2388.3410<6151.709
    Disorientation group69350.9866.0000<4121.164

    aAfter labeling into severe and nonsevere groups through k-means clustering, the numerical characteristics and differences between the groups are shown for each group.

    bDistortion was calculated using the k-means clustering model with k=2.

    cISS: Internalized Shame Scale.

    dPERS: Post-Event Rumination Scale.

    eVR: virtual reality.

    Differences in Physiological Signals According to Specific Anxiety Symptom Severity

    There were significant differences in the GSR min between the severe and nonsevere groups stratified by the ISS (t102=1.39; P=.17). In addition, groups stratified by the ISS mistake anxiety showed a significant difference in the HR peak ratio (t102=2.07; P=.04) and GSR average change (t102=2.02; P=.046). Groups stratified by ISS self-punishment showed significantly different GSR min values (t102=2.34; P=.02). Groups stratified by ISS emptiness showed a difference in mean HR (t102=−2.44; P=.02). Groups stratified by ISS inappropriate showed a significant difference in mean HR (t102=−2.23; P=.03). Groups stratified by PERS had significantly different mean HRs (t102=−1.99; P=.050). Furthermore, groups stratified by PERS positivity showed significantly different mean HRs (t102=2.51; P=.01), HR min values (t102=2.49; P=.01), and HR peak ratios (t102=2.15; P=.03). Groups stratified by PERS negative had significantly different mean HRs (t102=−2.79; P=.006), HR min values (t102=−2.49; P=.01), HR peak ratios (t102=−2.15; P=.03), and GSR max values (t102=−2.00; P=.048). The severe and nonsevere groups by anxiety symptoms are illustrated in Multimedia Appendix 4.

    We performed t tests after labeling the severity of VR sickness, for each VR sickness subdomain. The most significant difference in total VR sickness was for the HR min value (t102=−1.63; P=.02). In the nausea group, the HR max value was significantly different between the groups (t102=−2.47; P=.02). Additionally, the oculomotor group differed in the HR min value (t102=−2.19; P=.03). The disorientation group showed a difference in total HR change (t102=−2.14; P=.04) and total GSR change (t102=−2.09; P=.04). Differences in the severe and nonsevere groups according to VR sickness are illustrated in Multimedia Appendix 5.

    Specific Anxiety Symptom and VR Sickness Prediction Based on the Physiological Signal Data

    After oversampling the minority data using SMOTE, the F1 score [37], accuracy, and area under the receiver operating characteristic curve were calculated after predicting the survey results using logistic regression analysis, random forest, and naïve Bayesian methodology. Additionally, we calculated the feature importance of specific anxiety symptoms. The variable importance can be calculated based on the impurity index [38]. We used the feature importance of a random forest–based prediction model.

    Anxiety Symptom Prediction

    The classifications of specific anxiety symptom results were analyzed based on 3 models. Logistic regression [34], random forest [35], and naïve Bayes classifier [36] analyzed the severity of specific anxiety symptoms for participants. The data were amplified using the SMOTE method to prevent performance degradation due to unbalanced data.

    The classification performance was measured using the F1 score. The F1 scores of the ISS mistake anxiety subdomain were 0.8421, 0.7368, and 0.7647, as calculated by logistic regression, random forest, and naïve Bayesian classifier, respectively. These values were higher than those of the other ISS subdomains. The classification performance of the PERS negative subdomain was relatively higher than that of the PERS positive subdomain, with 0.6667, 0.6154, and 0.6452 as measured by logistic regression, random forest, and naïve Bayes classifier, respectively.

    Specifically, we classified the ISS using a logistic regression model with an F1 score of 0.7619. The logistic regression model revealed an F1 score of 0.8421 for ISS mistake anxiety, whereas the naïve Bayes classification model revealed a score of 0.7857 for ISS self-punishment. The logistic regression model revealed an F1 score of 0.6429 for ISS emptiness and 0.7200 for ISS inappropriate. The random forest model revealed an F1 score of 0.7568 for PERS. Further, the naïve Bayes classification model showed an F1 score of 0.7619 for PERS positivity. Finally, the random forest model revealed an F1 score of 0.7097 for PERS negativity. The specific anxiety symptom prediction results are illustrated in Table 2 and Multimedia Appendix 6.

    For ISS, ISS mistake anxiety, ISS self-punishment, ISS inappropriate, PERS, PERS positive, and PERS negative, physiological signal data related to GSR had the highest importance, and for ISS emptiness, physiological signal data related to HR had the highest importance. The feature importance of specific anxiety symptoms in the random forest model is illustrated in Figure 2.

    Table 2. Specific anxiety symptom classification model evaluation with F1 score, accuracy, and area under the curve.
    VariableaLogistic regressionRandom forestNaïve Bayes classifier

    		F1 scoreAccuracyAUCbF1 scoreAccuracyAUCF1 scoreAccuracyAUC
    ISSc0.7620.8080.8270.6320.7310.7060.7270.7690.775
    ISS mistake anxiety0.8420.7690.7330.7900.6920.6080.7650.6920.660
    ISS self-punishment0.7500.6920.6730.7270.7690.7690.7860.7690.769
    ISS emptiness0.6430.6150.6250.5710.6540.6390.6000.6920.673
    ISS inappropriate0.7200.7310.7320.5710.6540.6390.7140.6920.702
    PERSd0.6670.6540.6760.7570.6540.6250.6670.6150.607
    PERS positive0.6150.6150.6300.5000.7690.6600.7620.8080.861
    PERS negative0.6670.6540.6540.7100.6540.6540.6450.5770.577

    aAfter predicting the severity of each specific anxiety symptom using logistic regression, random forest, and naïve Bayes classifier models, the performance of each model was evaluated.

    bAUC: area under the curve.

    cISS: Internalized Shame Scale.

    dPERS: Post-Event Rumination Scale.

    ‎
    Figure 2. Feature importance of specific anxiety symptoms (random forest model). After predicting the subdomains of each anxiety symptom using the random forest model, the feature importance for each model was calculated and sorted in descending order. GSR: galvanic skin response; ISS: Internalized Shame Scale; PERS: Post-Event Rumination Scale.
    View this figure
    VR Sickness Prediction

    We classified the severity of total VR sickness with an F1 score of 0.7059, that in the nausea group with a score of 0.4000, that in the oculomotor group with a score of 0.6667, and that in the disorientation group with a score of 0.6364. The classification performance for the nausea group subdomain had the highest AUC of 0.94. The VR sickness results are illustrated in Table 3 and Multimedia Appendix 7.

    When feature importance was calculated for the classification of the severity of VR sickness for each type, physiological signal data related to GSR were selected as essential variables for all subdomains. All subdomains of VR sickness symptoms showed HR-related features as the second most crucial factor in variation. Figure 3 illustrates the importance of VR sickness features from the random forest model.

    Table 3. Virtual reality sickness classification model evaluation with F1 score, accuracy, and area under the curve.
    VariableaRandom forest

    		F1 scoreAccuracyAUCb
    Total VRc sickness0.70590.80770.7917
    Nausea group0.40000.88460.9400
    Oculomotor group0.66670.65380.7000
    Disorientation group0.63640.69230.7124

    aAfter predicting the severity of each VR sickness scale using the random forest model, the model’s performance was evaluated.

    bAUC: area under the curve.

    cVR: virtual reality.

    ‎
    Figure 3. Feature importance of virtual reality (VR) sickness (random forest model). After predicting the subdomains of each VR sickness scale using the random forest model, the feature importance for each model was calculated and sorted in descending order. GSR: galvanic skin response.
    View this figure

    Principal Findings

    We developed and tested machine learning models to predict specific anxiety symptoms and VR sickness for SAD using in situ autonomic physiological signal data measured during participatory and interactive VR treatment. The severity of specific anxiety symptoms and the side effects of VR treatment are an essential part of digital therapy. Usually, these kinds of evaluations rely on subjective reporting by patients. However, autonomic physiological responses, such as blushing, sweating, and shivering, could play a central role in assessing symptoms [39]; therefore, they have the potential to be used for monitoring various symptoms related to anxiety or VR sickness.

    Previous studies have assessed the relationship between physiological cues and SAD. However, real-time analysis is limited because of current sensor technology and evaluation methods [40]. Furthermore, individuals with SAD cannot be treated because of methodological problems [41-43].

    VR treatment techniques extend beyond the traditional psychiatric therapeutic approach [44]. To create a therapeutic VR system, we must build an interactive system rather than simply provide a VR environment. Interactive VR may have several benefits for psychiatric disorders in terms of treatment. First, it enhances ecological validity by immersing individuals in emotionally engaging virtually constructed therapeutic environments. Second, it can be flexibly used to present patients in various contexts, enabling personalized treatment according to the assessment of individual differences in symptoms during VR treatment [45]. We expect our findings to be useful in interactive VR treatment, especially for specific anxiety symptoms and VR sickness.

    We showed that the data extracted from the time series with different in situ autonomic physiological signal data lengths could be used as the independent variables of a predictive model. As 32 samples of physiological data were not sufficient for the machine learning model, we increased the data by dividing it per session using only 32 participants’ intervention data and supplementing the insufficient data using SMOTE methodology. This technique can minimize performance degradation due to a lack of data. We generated cutoff values for the classification groups for labeling via k-means clustering. This is an unsupervised learning method based on the absence of a label to evaluate the model performance. We could explain which variable plays the most crucial role in predicting specific anxiety symptoms and VR sickness.

    Among the anxiety symptoms related to the ISS, the most predictable was ISS mistake anxiety. Emotion comprised 5 main subsystems, namely, “cognitive component (appraisal), neurophysiological component (bodily symptoms), motivational component (action tendencies), motor expression component (facial and vocal expression), and subjective feeling component (emotional experience)” [46]. Shame is highly correlated with the body’s physiological responses because of its mechanism [47]. Mistake anxiety is the core symptom of SAD. The high predictability of these symptoms is related to the reliability of HR and GSR predictions of SAD symptoms.

    Among the scales related to PERS, the most predictable was PERS positivity. Rumination may negatively affect physical and mental health [48,49]. However, this is inconsistent with previous studies reporting that positive rumination either is not associated with social anxiety or is low [28,30]. In contrast, other studies have reported that positive and negative rumination is high in social anxiety. Socially anxious individuals ruminate broadly, positively, or negatively about all aspects of social interactions when faced with ambiguity [50]. However, no previous study has examined the relationship between in situ autonomic physiological responses, and negative and positive rumination. Additional studies are required to determine whether these results are reproducible.

    Several factors can cause VR sickness. Humans perceive their direction and movement through various sensory organs. Individuals may experience motion sickness if they repeatedly receive sensory information that differs from their prediction [51]. Visual movement may cause motion sickness [52,53]. When visual stimulation is the primary cause of motion sickness, it is called visually induced motion sickness. Previous studies have assessed virtual perceptions related to recognizing self-motion [54]. If the degree of physiological arousal (HR, blood pressure, skin conductance, respiration, skin temperature, and blood volume pulse amplitude) is high, the possibility of VR motion sickness is high [55].

    We used HR and GSR for machine learning–based prediction in our study. These are important indicators of physiological arousal. SAD-related psychological symptoms and VR sickness during VR treatment can be determined in advance through the proposed model. The intervention of medical staff can also improve patient compliance. However, we must develop additional real-time measurement factors to better predict other VR sickness subdomain symptoms.

    We predicted specific anxiety symptom severity and VR sickness severity via in situ physiological signal data from actual cases. We used supervised and unsupervised learning and data generation to build and evaluate the SAD and VR sickness predictive models. Considering these advantages, specific anxiety symptoms and VR sickness could be assessed more accurately. The data were labeled using an unsupervised learning method. After separating the severe and nonsevere groups, significant differences were found for each classification group using a t test. The real-time evaluation of VR motion sickness can help improve patient compliance with treatment. Moreover, we can reduce the time required for a survey by predicting participants’ anxiety using objective data. Predicting survey results using objective in situ autonomic physiological signal data makes less subjective intervention possible compared with the conventional survey-based method. In addition, the evaluation of symptoms through real-time autonomic physiological signals shows the possibility of evaluating psychiatric symptoms in real time to increase interactivity, an essential element in the personalized VR treatment process.

    Limitations

    This study had several limitations. First, our study evaluated in situ autonomic physiological signals and symptom-related data in a SAD group alone. Subsequently, it included no control groups. This study used machine learning models to discriminate between severe and nonsevere groups because real-time physiological signal data were measured only for participants above a certain Korean Social Interaction Anxiety Scale (K-SAD) score. In some cases, after labeling the severe and nonsevere groups, there was no significant difference between groups. These results can be inferred as a limitation of the methodology that divides the group with a K-SAD score above 82 into severe and nonsevere groups according to specific anxiety symptoms. Although an unsupervised learning methodology distinguished respondents, it was challenging to explain clear criteria other than to the participants in the experiment. Second, this study used self-rating score-based scales, which could be associated with bias. Third, this study could only classify the severity of SAD (K-SAD ≥82) in the group with a particular score or higher. Fourth, there was insufficient evaluation of the response to a specific task or exposure situation during a VR session. Because insufficient data are obtained using a SMOTE model, there is a limitation in not using the actual data. In addition, it is unknown how performance changes as an individual progresses during a session. Furthermore, we cannot determine how in situ autonomic physiological signals change when a specific situation occurs in each session. Fifth, the number of participants who participated in the VR treatment session was 32, but the number of participants used for the analysis was 26 (a small sample size). The decrease in performance can be inferred to be limited to the improvement in performance because the number of individuals who participated in the VR sessions was small. Therefore, we need to carefully interpret the results and secure a larger number of samples for future studies. However, since it is challenging to obtain real-time in situ data, we believe that the strength of this study is that real valuable data from participants were utilized for analysis.

    Conclusion

    This study showed that using in situ autonomous physiological signal data measured during a VR intervention can predict specific anxiety symptoms and VR sickness in patients with SAD. Using real-time physiological data from VR sessions, we can classify the severity of specific SAD symptoms and utilize the findings for personalized digital treatment. Machine learning models can assist in the decisions of medical staff and the construction of interactive VR treatment. Future research should focus on various predictive methodologies to enhance the tailored interactive function and to maximize the convenience of VR treatment. Additionally, to improve the clinical prediction performance and increase accuracy, more abundant and appropriate data will need to be collected.

    Acknowledgments

    This work was supported by a National Research Foundation (NRF) of Korea grant funded by the Ministry of Science and Information and Communications Technology (MSIT), the Government of Korea (NRF-2020R1C1C1007463 and NRF-2021R1A5A8032895), and Information and Communications Technology (ICT) and Future Planning for Convergent Research in the Development Program for R&D Convergence over Science and Technology Liberal Arts (NRF-2022M3C1B6080866).

    Authors' Contributions

    Authors CHC (david0203@gmail.com) and TC (tcheong@korea.ac.kr) are co-corresponding authors for this article, responsible for data and materials, manuscript submission, peer review, publication process, authorship details, and ethics committee approval.

    JWH, DJ, HJL, SL, GK, CYC, SML, SC, TC, and CHC conceived and designed the study. JYC, HJK, TC, and CHC performed statistical analyses. JYC, HJK, JWH, DJ, TC, and CHC wrote the first draft of this manuscript. JYC, HJK, JWH, DJ, SL, GK, CYC, SML, HL, SC, TC, and CHC participated in data collection. All authors have edited all versions of the manuscript. All authors were involved in interpreting the results, and they read, commented on, and approved the final version of the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Psychological scale prediction (linear regression model).

    PDF File (Adobe PDF File), 344 KB
    ‎
    Multimedia Appendix 2

    Correlation table for each feature.

    PDF File (Adobe PDF File), 298 KB
    ‎
    Multimedia Appendix 3

    Box plot of specific anxiety symptoms and virtual reality sickness severe and nonsevere group clustering results.

    PDF File (Adobe PDF File), 864 KB
    ‎
    Multimedia Appendix 4

    Results of specific anxiety symptoms.

    PDF File (Adobe PDF File), 385 KB
    ‎
    Multimedia Appendix 5

    Results of virtual reality sickness.

    PDF File (Adobe PDF File), 380 KB
    ‎
    Multimedia Appendix 6

    Receiver operating characteristic curve of specific anxiety symptom prediction models (logistic regression model, random forest model, and naïve Bayesian model).

    PDF File (Adobe PDF File), 853 KB
    ‎
    Multimedia Appendix 7

    Receiver operating characteristic curve of a virtual reality sickness prediction model (random forest model).

    PDF File (Adobe PDF File), 382 KB
    1. Cannistraro P, Rauch S. Neural circuitry of anxiety: evidence from structural and functional neuroimaging studies. Psychopharmacol Bull 2003;37(4):8-25. [Medline]
    2. Garfinkel SN, Eccles JA, Critchley HD. The heart, the brain, and the regulation of emotion. JAMA Psychiatry 2015 Nov;72(11):1071-1072. [CrossRef] [Medline]
    3. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. Washington, DC: American Psychiatric Association; 2013.
    4. Thurston RC, Rewak M, Kubzansky LD. An anxious heart: anxiety and the onset of cardiovascular diseases. Prog Cardiovasc Dis 2013 May;55(6):524-537. [CrossRef] [Medline]
    5. Friedman BH. An autonomic flexibility-neurovisceral integration model of anxiety and cardiac vagal tone. Biol Psychol 2007 Mar;74(2):185-199. [CrossRef] [Medline]
    6. Rapee RM, Heimberg RG. A cognitive-behavioral model of anxiety in social phobia. Behaviour Research and Therapy 1997 Aug;35(8):741-756. [CrossRef]
    7. Heimberg RG. Cognitive-behavioral therapy for social anxiety disorder: current status and future directions. Biological Psychiatry 2002 Jan;51(1):101-108. [CrossRef]
    8. Maples-Keller JL, Bunnell BE, Kim S, Rothbaum BO. The use of virtual reality technology in the treatment of anxiety and other psychiatric disorders. Harv Rev Psychiatry 2017;25(3):103-113. [CrossRef]
    9. Geraets CN, Veling W, Witlox M, Staring AB, Matthijssen SJ, Cath D. Virtual reality-based cognitive behavioural therapy for patients with generalized social anxiety disorder: a pilot study. Behav. Cogn. Psychother 2019 Mar 27;47(6):745-750. [CrossRef]
    10. McCauley ME, Sharkey TJ. Cybersickness: perception of self-motion in virtual environments. Presence: Teleoperators & Virtual Environments 1992 Jan;1(3):311-318. [CrossRef]
    11. LaViola JJ. A discussion of cybersickness in virtual environments. SIGCHI Bull 2000 Jan 01;32(1):47-56. [CrossRef]
    12. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology 1993 Jul;3(3):203-220. [CrossRef]
    13. Sevinc V, Berkman MI. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure of cybersickness in consumer virtual environments. Appl Ergon 2020 Jan;82:102958. [CrossRef] [Medline]
    14. Bouchard S, Dumoulin S, Robillard G, Guitard T, Klinger É, Forget H, et al. Virtual reality compared with exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial. Br J Psychiatry 2017 Apr;210(4):276-283. [CrossRef] [Medline]
    15. Kim H, Lee S, Jung D, Hur J, Lee H, Lee S, et al. Effectiveness of a participatory and interactive virtual reality intervention in patients with social anxiety disorder: longitudinal questionnaire study. J Med Internet Res 2020 Oct 06;22(10):e23024 [FREE Full text] [CrossRef] [Medline]
    16. Lee H, Choi J, Jung D, Hur J, Cho C. The effects of virtual reality treatment on prefrontal cortex activity in patients with social anxiety disorder: participatory and interactive virtual reality treatment study. J Med Internet Res 2021 Dec 17;23(12):e31844 [FREE Full text] [CrossRef] [Medline]
    17. Yoo S, Kim Y, Noh J, Oh K, Kim C, NamKoong K, et al. Validity of Korean version of the mini-international neuropsychiatric interview. Anxiety and Mood 2006;2(1):50-55 [FREE Full text]
    18. Hur J, Shin H, Jung D, Lee H, Lee S, Kim GJ, et al. Virtual reality-based psychotherapy in social anxiety disorder: fMRI study using a self-referential task. JMIR Ment Health 2021 Apr 14;8(4):e25731 [FREE Full text] [CrossRef] [Medline]
    19. Powers MB, Briceno NF, Gresham R, Jouriles EN, Emmelkamp PMG, Smits JAJ. Do conversations with virtual avatars increase feelings of social anxiety? J Anxiety Disord 2013 May;27(4):398-403. [CrossRef] [Medline]
    20. Anderson PL, Price M, Edwards SM, Obasaju MA, Schmertz SK, Zimand E, et al. Virtual reality exposure therapy for social anxiety disorder: a randomized controlled trial. J Consult Clin Psychol 2013 Oct;81(5):751-760. [CrossRef] [Medline]
    21. Social Anxiety Disorder VR treatment. YouTube.   URL: https://www.youtube.com/channel/UCCpzKK_qEqz8LRfxl_TfGXw/featured [accessed 2022-08-29]
    22. Beck A, Epstein N, Brown G, Steer R. Beck anxiety inventory. J of Consult Clin Psychol 1993:23-40 [FREE Full text] [CrossRef]
    23. Spielberger CD, Gonzalez-Reigosa F, Martinez-Urrutia A, Natalicio LF, Natalicio DS. The state-trait anxiety inventory. Revista Interamericana De Psicología/Interamerican Journal of Psychology 1971;5:145-158 [FREE Full text]
    24. Davidson J, Potts N, Richichi E, Ford S, Krishnan K, Smith R, et al. The Brief Social Phobia Scale. J Clin Psychiatry 1991 Nov;52 Suppl:48-51. [CrossRef] [Medline]
    25. Mattick RP, Clarke JC. Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behav Res Ther 1998 Apr;36(4):455-470. [CrossRef] [Medline]
    26. Watson D, Friend R. Measurement of social-evaluative anxiety. J Consult Clin Psychol 1969 Aug;33(4):448-457. [CrossRef] [Medline]
    27. Cook DR. Measuring shame. Alcoholism Treatment Quarterly 1988 Mar 07;4(2):197-215. [CrossRef]
    28. Abbott MJ, Rapee RM. Post-event rumination and negative self-appraisal in social phobia before and after treatment. J Abnorm Psychol 2004 Mar;113(1):136-144. [CrossRef] [Medline]
    29. Lee I, Choi H. Assessment of shame and its relationship with maternal attachment, hypersensitive narcissism and loneliness. Korean Journal of Counseling and Psychotherapy 2005;17(3):651-670 [FREE Full text]
    30. Edwards S, Rapee R, Franklin J. Postevent rumination and recall bias for a social performance event in high and low socially anxious individuals. Cognitive Therapy and Research 2003;27:603-617. [CrossRef]
    31. Jain AK. Data clustering: 50 years beyond K-means. Pattern Recognition Letters 2010 Jun;31(8):651-666. [CrossRef]
    32. Duarte M. detecta: A Python module to detect events in data (Version v0.0.5). Zenodo.   URL: https://zenodo.org/record/4598962#.YwzRTLRBzDc [accessed 2022-08-29]
    33. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. jair 2002 Jun 01;16:321-357. [CrossRef]
    34. LaValley MP. Logistic regression. Circulation 2008 May 06;117(18):2395-2399. [CrossRef]
    35. Pal M. Random forest classifier for remote sensing classification. International Journal of Remote Sensing 2007 Feb 22;26(1):217-222. [CrossRef]
    36. Murphy KP. Naive Bayes classifiers. University of British Columbia. 2006.   URL: https://www.ic.unicamp.br/~rocha/teaching/2011s2/mc906/aulas/naive-bayes.pdf [accessed 2022-08-29]
    37. Sokolova M, Japkowicz N, Szpakowicz S. Beyond Accuracy, F-Score and ROC: A family of discriminant measures for performance evaluation. In: Sattar A, Kang B, editors. AI 2006: Advances in Artificial Intelligence. AI 2006. Lecture Notes in Computer Science, vol 4304. Berlin, Heidelberg: Springer; 2006:1015-1021.
    38. Breiman L. Random forests. Machine Learning 2001;45:5-32. [CrossRef]
    39. Gerlach AL, Mourlane D, Rist F. Public and private heart rate feedback in social phobia: a manipulation of anxiety visibility. Cogn Behav Ther 2004 Mar;33(1):36-45. [CrossRef] [Medline]
    40. Alvares GA, Quintana DS, Kemp AH, Van Zwieten A, Balleine BW, Hickie IB, et al. Reduced heart rate variability in social anxiety disorder: associations with gender and symptom severity. PLoS One 2013;8(7):e70468 [FREE Full text] [CrossRef] [Medline]
    41. Rösler L, Göhring S, Strunz M, Gamer M. Social anxiety is associated with heart rate but not gaze behavior in a real social interaction. J Behav Ther Exp Psychiatry 2021 Mar;70:101600 [FREE Full text] [CrossRef] [Medline]
    42. Wen W, Liu G, Mao Z, Huang W, Zhang X, Hu H, et al. Toward constructing a real-time social anxiety evaluation system: exploring effective heart rate features. IEEE Trans. Affective Comput 2020 Jan 1;11(1):100-110. [CrossRef]
    43. Zhang X, Wen W, Liu G, Hu H. Recognition of public speaking anxiety on the recurrence quantification analysis of GSR signals. 2016 Presented at: Sixth International Conference on Information Science and Technology (ICIST); May 06-08, 2016; Dalian, China. [CrossRef]
    44. Park MJ, Kim DJ, Lee U, Na EJ, Jeon HJ. A literature overview of virtual reality (VR) in treatment of psychiatric disorders: recent advances and limitations. Front Psychiatry 2019 Jul 19;10:505 [FREE Full text] [CrossRef] [Medline]
    45. Verhoef REJ, van Dijk A, Verhulp EE, de Castro BO. Interactive virtual reality assessment of aggressive social information processing in boys with behaviour problems: A pilot study. Clin Psychol Psychother 2021 May 17;28(3):489-499 [FREE Full text] [CrossRef] [Medline]
    46. Scherer KR. What are emotions? And how can they be measured? Social Science Information 2016 Jun 29;44(4):695-729. [CrossRef]
    47. Hedman E, Ström P, Stünkel A, Mörtberg E. Shame and guilt in social anxiety disorder: effects of cognitive behavior therapy and association with social anxiety and depressive symptoms. PLoS One 2013;8(4):e61713 [FREE Full text] [CrossRef] [Medline]
    48. Brosschot JF, Verkuil B, Thayer JF. Conscious and unconscious perseverative cognition: is a large part of prolonged physiological activity due to unconscious stress? J Psychosom Res 2010 Oct;69(4):407-416. [CrossRef] [Medline]
    49. Glynn LM, Christenfeld N, Gerin W. The role of rumination in recovery from reactivity: cardiovascular consequences of emotional states. Psychosom Med 2002;64(5):714-726. [CrossRef] [Medline]
    50. Dannahy L, Stopa L. Post-event processing in social anxiety. Behav Res Ther 2007 Jun;45(6):1207-1219. [CrossRef] [Medline]
    51. Sherman C. Motion sickness: review of causes and preventive strategies. J Travel Med 2002;9(5):251-256 [FREE Full text] [CrossRef] [Medline]
    52. Bonato F, Bubka A, Palmisano S, Phillip D, Moreno G. Vection change exacerbates simulator sickness in virtual environments. Presence: Teleoperators and Virtual Environments 2008 Jun 01;17(3):283-292. [CrossRef]
    53. Keshavarz B, Riecke BE, Hettinger LJ, Campos JL. Vection and visually induced motion sickness: how are they related? Front Psychol 2015 Apr 20;6:472 [FREE Full text] [CrossRef] [Medline]
    54. Lubeck AJ, Bos JE, Stins JF. Motion in images is essential to cause motion sickness symptoms, but not to increase postural sway. Displays 2015 Jul;38:55-61. [CrossRef]
    55. Macedonio MF, Parsons TD, DiGiuseppe RA, Weiderhold BA, Rizzo AA. Immersiveness and physiological arousal within panoramic video-based virtual reality. Cyberpsychol Behav 2007 Aug;10(4):508-515. [CrossRef] [Medline]

    ‎
    AUC: area under the curve
    CBT: cognitive behavioral therapy
    GSR: galvanic skin response
    HR: heart rate
    ISS: Internalized Shame Scale
    K-SAD: Korean Social Interaction Anxiety Scale
    PERS: Post-Event Rumination Scale
    SAD: social anxiety disorder
    SMOTE: Synthetic Minority Oversampling Technique
    SSQ: Simulator Sickness Questionnaire
    VR: virtual reality

    Edited by N Zary; submitted 27.03.22; peer-reviewed by H Kim, K Lee, K Uludag; comments to author 18.04.22; revised version received 30.06.22; accepted 21.07.22; published 16.09.22

    Copyright

    ©Joo Young Chun, Hyun-Jin Kim, Ji-Won Hur, Dooyoung Jung, Heon-Jeong Lee, Seung Pil Pack, Sungkil Lee, Gerard Kim, Chung-Yean Cho, Seung-Moo Lee, Hyeri Lee, Seungmoon Choi, Taesu Cheong, Chul-Hyun Cho. Originally published in JMIR Serious Games (https://games.jmir.org), 16.09.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.