Background: Virtual reality can play an important role during the COVID-19 pandemic in the health care sector. This technology has the potential to supplement the traditional in-hospital medical training and treatment, and may increase access to training and therapies in various health care settings.
Objective: This systematic review aimed to describe the literature on health care–targeted virtual reality applications during the COVID-19 crisis.
Methods: We conducted a systematic search of the literature on the PsycINFO, Web of Science, and MEDLINE databases, according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. The search string was as follows: “[(virtual reality)] AND [(COVID-19) OR (coronavirus) OR (SARS-CoV-2) OR (healthcare)].” Papers published in English after December 2019 in peer-reviewed journals were selected and subjected to the inclusion and exclusion criteria. We used the Mixed Methods Appraisal Tool to assess the quality of studies and the risk of bias.
Results: Thirty-nine studies met the inclusion criteria. Seventeen studies showed the usefulness of virtual reality during the COVID-19 crisis for reducing stress, anxiety, depression, and pain, and promoting physical activity. Twenty-two studies revealed that virtual reality was a helpful learning and training tool during the COVID-19 crisis in several areas, including emergency medicine, nursing, and pediatrics. This technology was also used as an educational tool for increasing public understanding of the COVID-19 pandemic. Different levels of immersion (ie, immersive and desktop virtual reality), types of head-mounted displays (ie, PC-based, mobile, and standalone), and content (ie, 360° videos and photos, virtual environments, virtual reality video games, and embodied virtual agents) have been successfully used. Virtual reality was helpful in both face-to-face and remote trials.
Conclusions: Virtual reality has been applied frequently in medicine during the COVID-19 pandemic, with positive effects for treating several health conditions and for medical education and training. Some barriers need to be overcome for the broader adoption of virtual reality in the health care panorama.
Trial Registration: International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY) INPLASY202190108; https://inplasy.com/inplasy-2021-9-0108/
On March 11, 2020, the World Health Organization declared the spread of the novel coronavirus (COVID-19) a global pandemic . As a primary measure to control the spread of the virus, many governments worldwide recommended staying at home and practicing social distancing, dramatically affecting people’s daily lives [ - ]. Health care represents one of the sectors most affected by the adverse effects of the COVID-19 pandemic [ ]. Prolonged rules of social distancing and stay-at-home directions caused relevant difficulties in carrying out several in-hospital clinical activities [ , ], and the sudden interruption of medical education and training programs [ - ].
In such times of isolation and limited resources due to the COVID-19 crisis, information and communication technologies (ICTs) have empowered medical institutions to meet mental health and learning needs with scalable solutions [, ]. Virtual reality (VR) represents an ICT development with the potential to revolutionize clinical support and education [ - ].
By definition, VR is a set of technologies, including head-mounted displays (HMDs), computers, and mobile devices, that can immerse users in a 3D environment to different degrees [- ], from a simple presentation on a 2D display screen system (ie, desktop VR) up to highly immersive systems (ie, immersive VR) that use HMDs.
VR in the last decade has become a game changer for the health care sector in more than one way [- ], representing a helpful instrument both for the treatment of several health conditions and for medical education and training [ , - ]. This technology has been successfully applied to a wide range of mental disorders [ , ], including anxiety [ - ] and depression [ ]. Furthermore, VR is being used in physical rehabilitation for improving motor function, fitness, movement quality, and mobility [ , ], and it has also been adopted as an enjoyable method for managing pain [ - ]. Regarding education and training, VR was greatly appreciated by medical and nursing students [ - ], and proved to play a crucial role in improving medical knowledge [ - ] and fostering surgical skills [ , , ].
Therefore, during the COVID-19 crisis, VR has the potential to supplement the traditional in-hospital medical training and treatment , and may increase access to training and therapies in various health care settings [ , ].
Since, to the best of our knowledge, no previous work has investigated the use of health care–targeted applications of VR during the COVID-19 crisis, this systematic review aimed to describe the literature on this topic.
A systematic search of the literature was performed on March 3, 2022, by 2 of the authors (FP and AP) following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines . The review was preregistered (September 29, 2021) in the International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY; INPLASY202190108). The search databases were PsycINFO, Web of Science, and MEDLINE.
Two authors (FP and AP) established clear inclusion criteria to determine papers’ eligibility for inclusion in the review. Only studies meeting the following criteria were considered eligible for inclusion: (1) type of participant: all human participants (clinical and nonclinical population); (2) intervention: VR; (3) comparators: usual intervention, non-VR group, or none; (4) outcomes: focused on health care–targeted applications of VR during the COVID-19 pandemic; and (5) study design: randomized controlled trial (RCT), quantitative nonrandomized study (eg, nonrandomized controlled trial and case-control study), quantitative descriptive study (eg, survey, case series, and case report), or mixed methods study.
Papers published in English between December 2019 and March 2022 in peer-reviewed journals were selected and subjected to the inclusion criteria outlined above.
Studies were excluded if they (1) did not use VR; (2) did not focus on the applications of VR for the treatment of health conditions, or for medical education and training during the COVID-19 pandemic; (3) did not describe research conducted after the outbreak of the COVID-19 crisis; (4) did not specify the period when the study was conducted; (5) did not include outcome measures or did not include complete results; (6) did use only qualitative data; and (7) were letter to the editor articles, commentaries, or reviews.
Search Terms and Selection of Papers for Inclusion
The search string was as follows: “[(virtual reality)] AND [(COVID-19) OR (coronavirus) OR (SARS-CoV-2) OR (healthcare)].”
Initially, 2 authors (FP and AP) checked the titles and abstracts of identified articles to determine their eligibility. Subsequently, they independently reviewed the full text of potentially eligible papers. A consensus between the authors (FP and AP) resolved disagreements. When papers provided insufficient data for inclusion in the analysis, the corresponding authors were contacted to provide additional data. Five additional articles emerged via hand-searching and reviewing the reference lists of relevant papers.
Study Quality and Risk of Bias Assessment
We used the Mixed Methods Appraisal Tool (MMAT)  to assess the methodological quality of the studies included and the risk of bias. Studies could be awarded a score of 0, 25, 50, 75, or 100 (with 100 being the highest quality). The MMAT has high reliability and efficiency as a quality assessment protocol and can concomitantly appraise methodological quality across various empirical research [ ]. Two authors (FP and AP) independently assessed the studies’ quality. Interrater reliability was calculated with Cohen kappa [ ], using the software package SPSS (IBM Corp).
Two authors (FP and AP) independently extracted the following data.
The study characteristics included (1) the study outcomes (treatment, education, and training); (2) the study design used (RCT, quantitative nonrandomized study, quantitative descriptive study, or mixed methods study); (3) the populations included in the study (sample size, profession or health condition, gender, mean age or age range, and nationality); and (4) the measures used for the assessment of outcomes (eg, self-report questionnaires, semistructured interviews, and users’ session data). An indication of the mean age or age range identified studies conducted on children (ie, under 12 years old), adolescents (12-18 years old), young adults (18-35 years old), middle-aged adults (36-55 years old), and older adults (over 55 years old). The division in these age ranges followed previous studies [- ]. Regarding the study outcomes, we divided the selected papers into 2 main domain-specific categories related to VR applications in health care: (1) treatment, and (2) education and training.
The VR characteristics included (1) the level of immersion (high, medium, or low); (2) in the case of immersive VR, the specific type of HMD (ie, PC-based, console-based, mobile, or standalone system); (3) the content (virtual environments, 360° photos and videos, embodied virtual agents, VR video games, social VR platforms, or hybrid); (4) the site of use (face-to-face or remotely), the user mode (single user or multiuser); and (5) the time of use (the total amount of sessions and VR duration of use). The level of immersion, defined as a quantifiable feature of a technology that includes the extent to which it is possible to immerse oneself in the virtual world through interfaces , was considered because, based on it, it is possible to distinguish VR in different categories [ - ] ( ). Second, in the case of studies where an HMD was used, the model was specified to describe its specific type based on its implemented technologies ( ). Third, since there are different types of experiences in VR, information on the content was included ( ). Fourth, the site of use was considered since it represents crucial information concerning how VR can be offered. Fifth, information on user mode was included since VR content can be single user (ie, usable by a single user) or multiuser (ie, 2 or more users can share the same VR experience and communicate or interact with it). Finally, in the studies that indicated the time of VR use, this information was included, offering valuable insights about how and how long to use this technology to treat mental and physical health conditions, and in medical education and training.
|High (immersive VRa)||The user is immersed within a 3D content.||HMDsb|
|Medium (semi-immersive VR)||Rooms within which computer-generated environments are projected onto the walls.||C-Automatic Virtual Environment (CAVE)|
|Low (desktop VR)||Computer-generated environments made in 3D but which are shown on 2D displays.||PC, television, mobile phone, or tablet screen|
|Types of HMDs|
|PC-based||Requires a connection between an HMD and a computer with advanced computational and graphics capabilities.||Oculus Rift S and HTC Vive|
|Console-based||Needs a connection between an HMD and a specific game console.||PlayStation VR|
|Mobile||Involves the integration of VR systems on mobile devices thanks to specific HMDs.||Samsung Gear VR and low-cost HMDs compatible with mobile phones, such as Google Cardboard|
|Standalone (all-in-one)||They do not need other technologies to work.||Meta Quest II, HTC Vive Focus, and Pico Interactive Neo|
|Virtual environments||Simulation on a computer of 3D environments, which can be explored in real-time and in which the user can interact with objects contained within it . They are created through specific software such as Unreal or Unity.||N/Ac|
|360° videos and photos||Images or videos of real or digital environments presented in a spherical version. They provide a spherical view with multiple viewing angles and perspectives. The contents are omnidirectional and can either be computer generated or captured from the real world .||YouTube VR and Google Earth VR|
|Embodied virtual agents||Graphical representations of the individuals controlled by the computer itself using an artificial intelligence program .||N/A|
|VR video games||Video games played through HMDs in which the player can interact with virtual content not only through a joypad or a keyboard, but also using head rotation, eye movements, or specially designed controllers that respond to the position and movements of the player in a defined space . They include the following:|
- Commercial off-the-shelf video games that are “games that one can purchase on the high street” [ ], or rather purchasable in online or physical stores
- Custom-made games, often defined in the literature as “serious games” [ ], that are games created ad hoc by researchers to educate, train, or change behavior
|Beat Saber, Half-Life: Alyx, Superhot VR, and Fruit Ninja VR|
|Social VR platforms||3D virtual spaces where multiple geographically remote users can interact with one another through VR HMDs [, ]. At present, several online VR applications with a social component exist.||AltspaceVR, Horizon Worlds, VRChat, and VRzone|
aVR: virtual reality.
bHMD: head-mounted display.
cN/A: not applicable.
The search strategy retrieved 2503 records published after December 2019. A total of 1687 studies remained after deduplication and language examination, and 905 records were excluded after the first screening, and title and abstract analysis. Full-text copies of the 782 remaining studies were obtained and subjected to further evaluation. After reading the full-text copies, 743 studies were excluded based on our exclusion criteria, resulting in 39 studies being included in our systematic review ().
Quality Assessment Outcomes
Interrater reliability was 0.827, representing substantial agreement . Within this systematic review, the distribution of MMAT scores varied significantly with the study design ( ). Nineteen studies (49%) met the MMAT quality assessment score of 75% or above, implying that much of the research in this area is of high quality ( ). Nevertheless, the quality scores varied substantially according to the study design.
|MMATa score distribution||References||Value, n (%)|
|Quantitative randomized controlled trial (N=4)|
|50||[, ]||2 (50)|
|100||[, ]||2 (50)|
|Quantitative nonrandomized study (N=10)|
|50||[- ]||5 (50)|
|75||[- ]||4 (40)|
|Quantitative descriptive study (N=19)|
|25||[, - ]||4 (21)|
|50||[, - ]||6 (32)|
|75||[- ]||5 (26)|
|100||[- ]||4 (21)|
|Mixed methods study (N=6)|
|75||[, ]||2 (33)|
|100||[, ]||2 (33)|
aMMAT: Mixed Methods Appraisal Tool.
bN/A: not applicable.
Seventeen studies focused on the applications of VR during the COVID-19 pandemic to treat different health conditions, while 22 studies investigated its use for medical education and training ().
Ten studies showed the usefulness of VR for reducing stress, anxiety, and depression during the COVID-19 pandemic, and there were several findings. First, a self-administered at-home VR-based intervention through low-cost HMDs was useful during the COVID-19 lockdown for diminishing depression levels, stress levels, general distress, and perceived stress in healthy individuals up to the 2-week follow-up . Second, immersive VR experiences showing calming nature scenes (ie, VRelax and Tranquil Cinematic-VR simulation) were helpful in reducing perceived stress among COVID-19 intensive care unit (ICU) health care workers [ , ]. Third, guided meditation and exploration of natural environments using immersive VR were highly satisfactory for patients recovering in a COVID-19 ICU, with perceived benefits in coping with isolation and loneliness [ ]. The ICU staff considered VR logistically and operationally feasible [ ]. Fourth, remote reminiscence training conducted during the first months of the COVID-19 pandemic using immersive VR diminished state anxiety without serious side effects (ie, nausea, dizziness, and headache) in patients with mild cognitive impairment [ ]. Fifth, immersive VR exposure therapy showing scenes related to COVID-19 was effective in diminishing anxiety symptoms among patients with chief complaints of fear of COVID-19 [ ], and in reducing levels of posttraumatic stress disorder (PTSD), anxiety, and depression among patients treated in the ICU due to COVID-19 [ ]. The use of VR in the ICU was considered by patients to be feasible, and improved their satisfaction with and ratings of the ICU after care [ ]. Sixth, immersive VR showing 360° YouTube videos reduced depressive symptoms in an adult with mild depressive disorder [ ]. Seventh, 360° virtual tourism experiences diminished perceived stress among healthy young adults [ ].
VR has also been used as a distraction tool for pain management during the COVID-19 pandemic. A self-administered daily at-home immersive VR-based program (ie, EaseVRx) helped reduce pain intensity and interference with activity, stress, mood, and sleep over time among young adults experiencing chronic low back pain .
Four studies reported the efficacy of VR for promoting physical activities during the COVID-19 crisis. First, a desktop VR video game developed ad hoc and used at home improved movement performance and physical activity intensity among patients with cerebral palsy . Second, commercial off-the-shelf VR video games (ie, “Box VR” and “NVIDIA VR Fun House”) fostered physical exercise in 4 older people without adverse effects [ ]. Participants reported that using such games was “positive and relaxing, and motivates you to exercise” [ ]. Third, VR recreational use significantly increased during the COVID-19 lockdown period, and users expressed overwhelmingly positive opinions on the impact of VR activities on their mental and physical health. The self-reported intensity of physical activity was considerably more strenuous in VR users than in console users [ ]. Fourth, VR fitness impacted individuals’ physical and mental health, playing a substantial role in improving the overall quality of life and positively influencing the behavior and attitude of users [ ].
Two studies provided early indications that Spanish-speaking (ie, Spain and Latin America) therapists have begun using VR to help their patients during social isolation due to COVID-19 [, ], and 20% of the therapists indicated that they had used VR technology for remote psychological support [ ].
Education and Training
Six studies revealed that VR represented a helpful learning and training tool during the COVID-19 pandemic in several areas. First, watching 360° videos on desktop displays effectively taught emergency medicine during the COVID-19 crisis . Second, immersive VR showed relatively better outcomes regarding skills acquired, learning speed, and information retention rates than classroom training in a sample of frontline health workers [ ]. Third, pediatric residents experienced a desktop VR-based training program as immersive, feasible, and realistic in terms of the clinic setting, and as a safe space to practice and learn new skills [ ]. Fourth, VR-based pregraduation medical training was considered realistic with regard to the initial clinical assessment and diagnostic activity by medical students [ ], even if a nonnegligible proportion of the students experienced difficulties in online access to the VR platform [ ]. Fifth, an immersive social VR platform was reported as easy to use, helpful, and better than tele and video conferencing for remote multidisciplinary heart team meetings [ ]. Sixth, applicants of a radiology residency reported positive attitudes toward a nonimmersive social VR platform [ ].
Six studies successfully adopted VR for teaching various medical topics during the COVID-19 pandemic, and there were several findings. First, immersive VR was useful for teaching brain and spinal cord neuroanatomy and for practicing neurorehabilitation exercises . Second, desktop VR–based training was used successfully for teaching nursing content [ , , ] and urology [ ]. Third, immersive VR experiences helped simulate pediatric ICU clinical scenarios, with some specific critiques regarding limited realism in some mechanical aspects of the simulation [ ].
Six studies evaluated this technology to teach COVID-19–related skills to doctors and nurses, and as an educational tool for increasing public understanding of the COVID-19 crisis, and there were several findings. First, an immersive VR simulation (ie, COVID-19 VR Strikes Back) was at least as effective as traditional learning methods for training medical students regarding COVID-19–related skills . Second, 2 immersive VR experiences, one involving wearing and stripping personal protective equipment [ ] and the other involving the management of patients with respiratory infectious diseases due to COVID-19 [ ], provided an effective and safe alternative to training nurses during the first year of the COVID-19 pandemic. Third, a virtual simulation of 2 patients (ie, COVID-19 and surgical trauma), which tried using desktop displays, helped nursing students bridge gaps in teaching and learning processes [ ]. Fourth, a VR intervention using HMDs effectively increased COVID-19 vaccination intentions among unvaccinated young adults [ ]. Fifth, VR, using desktop displays and HMDs, provided an effective educational tool for COVID-19 pandemic fundamental knowledge, increasing public understanding of the spread of the crisis [ ].
Four studies investigated the adoption rate and the perception of VR in medicine during the COVID-19 pandemic. First, pediatric health care providers reported frequent modifications to existing simulation programs during the first months of the COVID-19 pandemic, including VR training . Second, medical students mostly agreed that VR and online teaching compensated for the suspension of face-to-face medical education and reported that these technologies are the best alternatives to physical learning [ ]. Third, the potential of VR for future teaching was rated low in a sample of medical students and lecturers, probably due to a lack of practical experience [ ]. Fourth, high-fidelity immersive VR and specialized profession-specific resources were used heavily in medical education and training during the first year of the COVID-19 pandemic [ ].
Considering the entirety of the studies, the quantitative descriptive was the design of 19 studies (ie, 12 surveys and 7 case report studies). Ten studies adopted a quantitative nonrandomized design, 6 adopted a mixed methods design, and 4 adopted an RCT design ().
The number of participants ranged from 1 [, ] to 4300 [ ]. The study samples’ characteristics are described in .
|Characteristic||References||Value (N=39), n (%)|
|Treatment||[, , , , - , , , - , - ]||17 (44)|
|Education and training||[, , , , , - , - , - , , - ]||22 (56)|
|Medical students||[, , , , , , , ]||8 (21)|
|Nursing students||[, , , ]||4 (10)|
|Doctors||[, , , , ]||5 (13)|
|Nurses||[, ]||2 (5)|
|COVID-19 frontline workers||[, , ]||3 (8)|
|Mental health professionals||[, ]||2 (5)|
|Patient health condition|
|Recovered in the ICUa due to COVID-19||[, , ]||3 (8)|
|Chronic back pain||||1 (2)|
|Cerebral palsy||||1 (2)|
|Mild cognitive impairment||||1 (2)|
|Chief complaints of fear of COVID-19||||1 (2)|
|Healthy individuals||[, , , , , , ]||7 (18)|
|Under 12 years old||N/Ab||0 (0)|
|12-18 years old||||1 (2)|
|18-35 years old||[, , , , , , , , , , , , , ]||14 (36)|
|36-55 years old||[, , , , , ]||6 (15)|
|Over 55 years old||[, , , ]||4 (10)|
|Unspecified||[, , , , , - , , , , , , ]||14 (36)|
|Both male and female||[- , , , - , , , , , , , , , , , , ]||24 (62)|
|Male only||[, , ]||3 (8)|
|Female only||N/A||0 (0)|
|Unspecified||[, , , , , , , , , , , ]||12 (31)|
|Europe||[, , , , , , , , , , , , - , - ]||18 (46)|
|North America||[, , , , , , , , ]||9 (23)|
|South America||||1 (2)|
|Asia||[- , , , , ]||8 (20)|
|Africa||[, ]||2 (5)|
|Self-report questionnaire||All||39 (100)|
|Performance task||[, , ]||3 (8)|
|Semistructured interview||[, , ]||3 (8)|
|User session data||[, ]||2 (5)|
|Focus group||||1 (2)|
|Knowledge task||[, , - , ]||6 (15)|
|Physiological data||||1 (2)|
aICU: intensive care unit.
bN/A: not applicable.
All the studies used self-reported questionnaires. Twenty-three studies adopted only this type of measure, while 16 studies also used other instruments, including performance tasks, semistructured interviews, knowledge tasks, users’ session data, focus groups, and physiological data (seefor details).
Of the 39 studies included, 31 tested the efficacy of specific VR systems (), while 8 investigated the general use of VR in different samples (eg, mental health professionals and doctors) ( ) [ - , , , , ].
|Characteristic||References||Value (N=31), n (%)|
|High||[, - , , , - , , , , , , , , , ]||19 (61)|
|Low||[, , , , , , , , ]||9 (29)|
|Both high and low||[, , ]||3 (10)|
|Type of HMDb|
|PC-based||[, , , , , , , ]||8 (36)|
|Mobile||[, , ]||3 (14)|
|Standalone||[, , , , , , , , ]||9 (41)|
|Unspecified||[, ]||2 (9)|
|Virtual environments||[, , , - , , ]||8 (26)|
|360° videos or photos||[, , , , , , , , , , ]||11 (35)|
|Embodied virtual agents||[, , , , ]||5 (16)|
|VRc video games||[, ]||2 (6)|
|Hybrid||[, , , ]||4 (13)|
|Single user||[, , - , , , - , , , - , ]||29 (93)|
|Multiuser||[, ]||2 (7)|
|Site of use|
|Face-to-face||[, , , , , - , - , , , , , , , , ]||19 (61)|
|Remotely||[, , , , , , , , , ]||10 (32)|
|Time of use (in total)|
|<10 minutes||[, , - , ]||6 (19)|
|11-60 minutes||[, , - , , , ]||9 (29)|
|61-180 minutes||[, , , ]||4 (13)|
|>180 minutes||[, ]||2 (6)|
|Unspecified||[, , , , , , , , , ]||10 (32)|
aN/A: not applicable.
bHMD: head-mounted display.
cVR: virtual reality.
Level of Immersion
Nineteen studies used a VR system with high immersion (ie, immersive VR), while 9 tested a system with low immersion (ie, desktop VR). Three studies adopted both immersive and desktop VR systems [, , ]. No study used semi-immersive VR systems ( ).
Types of HMDs
Of the 22 studies that used an HMD, standalone systems were the most popular, with 9 studies using them. Eight studies adopted PC-based VR systems, 3 studies used mobile VR systems [, , ], and 2 studies did not specify the model of HMD [ , ] ( ).
The most adopted VR content was 360° videos or photos, used in 11 studies. Eight studies used virtual environments, 2 studies adopted VR video games [, ], 5 studies used embodied agents [ , , , , ], 4 studies used hybrid content (ie, a mix of 360° videos or photo and virtual environments or VR video games) [ , , , ], and 1 study did not specify the content [ ] ( ).
The virtual content was of the single-user type in 29 studies and was of the multiuser type in 2 studies [, ] ( ).
Site of Use
Nineteen studies used VR through face-to-face experiments, 10 used it remotely, and 1 used it both face-to-face and remotely . Finally, 1 study did not specify the site of use [ ] ( ).
Time of Use
In 21 studies reporting the number of sessions using VR, the mean number of sessions was 4.1, ranging from 1 (eg, [, , ]) to 56 sessions [ ]. The number of minutes spent using VR differed among studies from about 5 minutes (eg, [ , ]) to up to 4 hours [ , ]. Ten studies did not indicate the exact time of use of VR ( ).
This systematic review examined studies conducted on VR applications in medicine during the COVID-19 pandemic, with a focus on using VR to treat health conditions and in medical education and training. After applying the inclusion criteria, 39 papers were included and analyzed.
From this systematic review, VR was found to be useful during the COVID-19 pandemic for reducing stress (eg, [, , ]), anxiety [ ], depression [ ], and pain [ ], and promoting physical activity [ , ]. This technology has been successfully used in healthy people (eg, [ , , , ]); COVID-19 ICU health care workers [ , ]; patients with various diseases, including mild cognitive impairment [ ], mild depressive disorders [ ], and cerebral palsy [ ]; chief complaints of fear of COVID-19 [ ]; and individuals treated in the ICU due to COVID-19 [ , , ].
VR has been successfully applied for decades to diminish stress, anxiety, depression, and pain [, , - ]. Besides, this technology, especially immersive video games [ , ], stimulates physical activity [ ], with long-term positive outcomes on mental and physical health [ , ]. What is new from the past, as noted in this review, is that VR is no longer used only in the specialist’s office or at the hospital but is also used remotely. The release of current standalone and low-cost mobile VR systems, thanks to the high ease of use and limited costs, has made this technology feasible for everyday in-home use [ ]. Studies from this review showed that VR during the COVID-19 pandemic has begun to be adopted into mainstream clinical practice for remote psychological support [ , ]. This fact appears relevant since VR home-based training represents new promising interventions for remote support [ , , ].
From this review, not only VR environments developed ad hoc, but also commercial contents (eg, VR video games downloaded from the Steam platform, and 360° videos and photos available on YouTube) were useful for diminishing stress , anxiety [ ], and depression [ ], and for promoting physical activity [ ]. These findings seem interesting since using commercial off-the-shelf VR video games and experiences for promoting mental health and physical activity could have several advantages, including their low cost and ready-to-use format, their advanced graphic quality, and the possibility to reach millions of individuals worldwide [ , ].
The second main result of this systematic review is that during the COVID-19 pandemic, VR was a helpful learning tool for medical education and training in several areas, including emergency medicine, nursing, pediatrics, radiology, and cardiology (eg, [- ]). This technology has been successfully used for teaching various topics, including neuroanatomy and clinical anatomy (eg, [ , ]). VR has also been adopted to train COVID-19–related skills among doctors and nurses [ , ]. Studies that emerged from this systematic review showed that, where available, high-fidelity immersive VR was adopted heavily after the outbreak of the COVID-19 pandemic [ , ].
Several previous studies proved the usefulness of VR for the training of medical students and nurses  and for teaching many medical topics and procedures, such as clinical anatomy [ , ] and radiation oncology [ ]. This technology is considered a superior educational modality in comparison to passive teaching methods, such as learning from a textbook, slideshow, or lecture, thanks especially to the greater involvement of individuals in the learning process [ - ].
This review, with regard to treatment, showed that during the COVID-19 pandemic, VR started to be used for medical education and training in face-to-face sessions and remotely. This flexibility of access allowed the adoption of this technology into medical education curricula for both in-person and home-based activities . Medical students mostly agreed that online teaching using VR compensated for the suspension of face-to-face medical education and reported this technology as the best alternative to physical learning [ ].
Some studies that emerged in this review used content in multiuser VR, including social VR platforms . Such VR environments are an emerging diverse group of multiuser online applications creating new opportunities for remote simulation and communication, and facilitating and extending the existing communication channels of the physical world [ , ]. Although its exploration has been limited so far, education using social VR will be increasingly relevant in the next few years. Indeed, major technology companies are making huge investments in the so-called “metaverse” (ie, a simulated digital environment that incorporates VR and other technologies, including augmented reality and artificial intelligence, to build spaces where users can interact as in the actual world) [ ]. “Metaversities” could become the universities of the future [ ].
In this systematic review, VR was also an effective educational tool for increasing public understanding and knowledge of the COVID-19 pandemic  and COVID-19 vaccination intentions [ ]. This result, in line with previous literature [ , ], stresses the potential of VR as a useful and innovative tool for promoting scientific and medical knowledge among the general population. VR can improve health awareness and assist in health-related decision-making for the general population [ ], increasing the overall motivation and engagement and leading to a more effective transfer of medical knowledge and comprehension of information [ , ]. In turn, this could improve individuals’ overall health, decrease hospitalization rates, and save long-term physician consultation costs [ ].
Moreover, studies from this systematic review showed the usefulness of VR for treating health problems and for medical education and training in both the high (ie, immersive VR) and low immersion (ie, desktop VR) formats. This result appears important as these VR systems have very different characteristics and costs [- ]. On the one hand, as emerged from studies included in this review (eg, [ , ]), immersive VR offers advantages over desktop VR [ , ]. However, desktop VR has the main advantages of being more readily usable and accessible, and having a lower cost.
Various types of HMDs (ie, PC-based, mobile, and standalone) and contents (ie, 360° videos and photos, virtual environments, VR games, and embodied virtual agents) have been used with positive results in medicine during the COVID-19 crisis (eg, [, , ]). This underscores how nowadays it is possible to choose among different VR hardware and software. To select the potentially most effective ones, it is critical to reason about the specific goal of VR use [ ]. For example, in the case of VR for relaxation, it may be sufficient to include a low-cost mobile system and 360° videos or commercial off-the-shelf VR games. The same is true when VR is adopted to offer doctors and nurses “soft skills” (eg, problem-solving, communication, and interpersonal skills) training. On the other hand, in the case where VR is used for complex operations, simulation training, a PC-based system, haptic devices, and specially created virtual content should be used. In fact, in this case, the graphical and sensory fidelity of VR are crucial [ , ].
Barriers for Using VR in the Health Care Sector
In addition to offering data in favor of the usefulness of VR in the health care sector, this review also raises some critical reflections on the possible limits of using this technology for these aims.
First, some health care workers experienced difficulties in using VR . Due to a lack of knowledge of this technology, many individuals may find it challenging to use VR. To overcome the mentioned obstacle, governments and other societal bodies (eg, medical societies, medical schools, and residency training programs) should provide information about VR (eg, through training courses dedicated to health care practitioners and mental health professionals). They should offer clear guidelines for correctly using this technology in mental health, and medical education and training [ ].
Second, when the graphic quality of the virtual experience was perceived as too low, users had difficulties comprehending a target scenario [, ]. To prevent graphical or technical problems (eg, breakdown or problems with interaction) [ , ], with potential detrimental effects on treatment and learning outcomes [ , ], the principles and practices of user-centered design are recommended [ , ]. Many VR-based treatment, and medical education and training programs are currently taking a technology approach rather than a human-centered approach, which can lead to limited impact of VR in these fields. Due to the close bond between the user and the system within virtual environments, it may be impossible to segregate human factors from design issues when striving to achieve the potential of VR [ ]. For this reason, it appears critical that a psychologist with expertise in human-computer interaction always be included in the team developing virtual experiences for the medical field.
Third, a primary issue in using this technology in the mental health panorama is related to its costs. The price of HMDs range from US $300 to US $1500. In the case of VR content developed ad hoc, the costs for implementation are high. One way to overcome the economic barrier of VR could be to use standalone or low-cost mobile systems [, ] and, whenever possible, commercially available content [ , ]. Another solution could be providing hospitals with a certain number of VR systems to be available for their patients and staff for free [ ].
Recommendations for Future Research
More studies should be undertaken to expand the limited literature on home-based VR for treatment, and medical education and training. Multiuser VR platforms could be studied more deeply in education and as tools to implement future interventions for treating mental and physical health conditions. More research should be conducted to test the efficacy of commercial off-the-shelf VR experiences for treatment and in educational settings. Limitations and possibilities of VR systems with high versus low immersion require further investigation, for example, through RCT studies comparing their effectiveness for clinical or learning outcomes. It might be worth investigating the longer-term effects of VR on such outcomes. Finally, the number of desired or needed VR sessions and the specific time to spend using VR remain matters of debate.
This review summarizes health care-targeted applications of VR during the COVID-19 pandemic based on specific keywords used in the search string, the databases included, and the review’s time period. Therefore, certain articles could have been missed. Second, only articles written in English and peer reviewed were included. Hence, preprints and gray literature were left out, which may have introduced some biases. Third, a meta-analysis was not possible due to the heterogeneity of the included studies. Fourth, the quality assessment performed using the MMAT suggests that even if much of the research in this area is of high quality, methodological concerns are significant issues for many studies.
VR has been applied frequently during the COVID-19 pandemic in medicine, with positive effects for treating several health conditions and for medical education and training. Some barriers need to be overcome for a broader adoption of VR in the health care panorama. Based on these findings, it is possible to offer certain VR-based programs even remotely for therapeutic or educational purposes, and not only VR environments developed ad hoc, but also commercial content can be helpful in clinical or educational support. Moreover, VR-based interventions have the potential to be used effectively for the treatment of several mental and physical conditions, as well as for medical education and training in both immersive and desktop systems. Various VR contents are helpful in treating health problems and for medical education and training, including 360° videos and photos, VR games, and embodied virtual agents.
FP conceived the work and wrote the first draft of the manuscript. All the authors contributed to manuscript revision, and read and approved the submitted version of the manuscript.
Conflicts of Interest
Mixed Methods Appraisal Tool evaluation.DOCX File , 38 KB
Study characteristics.DOCX File , 59 KB
Virtual reality characteristics.DOCX File , 30 KB
- Cucinotta D, Vanelli M. WHO Declares COVID-19 a Pandemic. Acta Biomed 2020 Mar 19;91(1):157-160 [FREE Full text] [CrossRef] [Medline]
- Di Renzo L, Gualtieri P, Pivari F, Soldati L, Attinà A, Cinelli G, et al. Eating habits and lifestyle changes during COVID-19 lockdown: an Italian survey. J Transl Med 2020 Jun 08;18(1):229 [FREE Full text] [CrossRef] [Medline]
- Giuntella O, Hyde K, Saccardo S, Sadoff S. Lifestyle and mental health disruptions during COVID-19. Proc Natl Acad Sci U S A 2021 Mar 02;118(9):e2016632118 [FREE Full text] [CrossRef] [Medline]
- Park K, Kim A, Yang M, Lim S, Park J. Impact of the COVID-19 pandemic on the lifestyle, mental health, and quality of life of adults in South Korea. PLoS One 2021;16(2):e0247970 [FREE Full text] [CrossRef] [Medline]
- Kaye A, Okeagu C, Pham A, Silva R, Hurley J, Arron B, et al. Economic impact of COVID-19 pandemic on healthcare facilities and systems: International perspectives. Best Pract Res Clin Anaesthesiol 2021 Oct;35(3):293-306 [FREE Full text] [CrossRef] [Medline]
- Chudasama YV, Gillies CL, Zaccardi F, Coles B, Davies MJ, Seidu S, et al. Impact of COVID-19 on routine care for chronic diseases: A global survey of views from healthcare professionals. Diabetes Metab Syndr 2020;14(5):965-967 [FREE Full text] [CrossRef] [Medline]
- Marasco G, Nardone O, Maida M, Boskoski I, Pastorelli L, Scaldaferri F, Italian Association of Young Gastroenterologist and Endoscopist (AGGEI). Impact of COVID-19 outbreak on clinical practice and training of young gastroenterologists: A European survey. Dig Liver Dis 2020 Dec;52(12):1396-1402 [FREE Full text] [CrossRef] [Medline]
- Hall AK, Nousiainen MT, Campisi P, Dagnone JD, Frank JR, Kroeker KI, et al. Training disrupted: Practical tips for supporting competency-based medical education during the COVID-19 pandemic. Med Teach 2020 Jul;42(7):756-761. [CrossRef] [Medline]
- Ahmed H, Allaf M, Elghazaly H. COVID-19 and medical education. Lancet Infect Dis 2020 Jul;20(7):777-778 [FREE Full text] [CrossRef] [Medline]
- Rose S. Medical Student Education in the Time of COVID-19. JAMA 2020 Jun 02;323(21):2131-2132. [CrossRef] [Medline]
- Pimentel D, Foxman M, Davis DZ, Markowitz DM. Virtually Real, But Not Quite There: Social and Economic Barriers to Meeting Virtual Reality’s True Potential for Mental Health. Front. Virtual Real 2021 Feb 17;2:627059. [CrossRef]
- Matamala-Gomez M, Bottiroli S, Realdon O, Riva G, Galvagni L, Platz T, et al. Telemedicine and Virtual Reality at Time of COVID-19 Pandemic: An Overview for Future Perspectives in Neurorehabilitation. Front Neurol 2021;12:646902 [FREE Full text] [CrossRef] [Medline]
- Woolliscroft JO. Innovation in Response to the COVID-19 Pandemic Crisis. Acad Med 2020 Aug;95(8):1140-1142 [FREE Full text] [CrossRef] [Medline]
- Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked 2021;24:100579 [FREE Full text] [CrossRef] [Medline]
- Imperatori C, Dakanalis A, Farina B, Pallavicini F, Colmegna F, Mantovani F, et al. Global Storm of Stress-Related Psychopathological Symptoms: A Brief Overview on the Usefulness of Virtual Reality in Facing the Mental Health Impact of COVID-19. Cyberpsychol Behav Soc Netw 2020 Nov 01;23(11):782-788. [CrossRef] [Medline]
- Pallavicini F, Chicchi Giglioli I, Kim G, Alcañiz M, Rizzo A. Editorial: Virtual Reality, Augmented Reality and Video Games for Addressing the Impact of COVID-19 on Mental Health. Front. Virtual Real 2021 Jul 14;2:719358. [CrossRef]
- Singh RP, Javaid M, Kataria R, Tyagi M, Haleem A, Suman R. Significant applications of virtual reality for COVID-19 pandemic. Diabetes Metab Syndr 2020;14(4):661-664 [FREE Full text] [CrossRef] [Medline]
- Miller HL, Bugnariu NL. Level of Immersion in Virtual Environments Impacts the Ability to Assess and Teach Social Skills in Autism Spectrum Disorder. Cyberpsychol Behav Soc Netw 2016 Apr;19(4):246-256 [FREE Full text] [CrossRef] [Medline]
- Kardong-Edgren S, Farra SL, Alinier G, Young HM. A Call to Unify Definitions of Virtual Reality. Clinical Simulation in Nursing 2019 Jun;31:28-34. [CrossRef]
- Rebelo F, Noriega P, Duarte E, Soares M. Using virtual reality to assess user experience. Hum Factors 2012 Dec;54(6):964-982. [CrossRef] [Medline]
- Parsons TD. Virtual Reality for Enhanced Ecological Validity and Experimental Control in the Clinical, Affective and Social Neurosciences. Front Hum Neurosci 2015;9:660 [FREE Full text] [CrossRef] [Medline]
- Izard SG, Juanes JA, García Peñalvo FJ, Estella JMG, Ledesma MJS, Ruisoto P. Virtual Reality as an Educational and Training Tool for Medicine. J Med Syst 2018 Feb 01;42(3):50. [CrossRef] [Medline]
- Riva G, Wiederhold BK, Mantovani F. Neuroscience of Virtual Reality: From Virtual Exposure to Embodied Medicine. Cyberpsychol Behav Soc Netw 2019 Jan;22(1):82-96 [FREE Full text] [CrossRef] [Medline]
- Riener R, Harders M. Virtual Reality in Medicine. London: Springer; 2012.
- Li L, Yu F, Shi D, Shi J, Tian Z, Yang J, et al. Application of virtual reality technology in clinical medicine. Am J Transl Res 2017;9(9):3867-3880 [FREE Full text] [Medline]
- Kyaw BM, Saxena N, Posadzki P, Vseteckova J, Nikolaou CK, George PP, et al. Virtual Reality for Health Professions Education: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration. J Med Internet Res 2019 Jan 22;21(1):e12959 [FREE Full text] [CrossRef] [Medline]
- Hilty DM, Randhawa K, Maheu MM, McKean AJS, Pantera R, Mishkind MC, et al. A Review of Telepresence, Virtual Reality, and Augmented Reality Applied to Clinical Care. J Technol Behav Sci 2020 Feb 20;5(2):178-205. [CrossRef]
- Dascal J, Reid M, IsHak WW, Spiegel B, Recacho J, Rosen B, et al. Virtual Reality and Medical Inpatients: A Systematic Review of Randomized, Controlled Trials. Innov Clin Neurosci 2017;14(1-2):14-21 [FREE Full text] [Medline]
- Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, et al. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med 2017 Oct;47(14):2393-2400 [FREE Full text] [CrossRef] [Medline]
- Kim S, Kim E. The Use of Virtual Reality in Psychiatry: A Review. Soa Chongsonyon Chongsin Uihak 2020 Jan 01;31(1):26-32 [FREE Full text] [CrossRef] [Medline]
- Maples-Keller JL, Bunnell BE, Kim S, Rothbaum BO. The Use of Virtual Reality Technology in the Treatment of Anxiety and Other Psychiatric Disorders. Harv Rev Psychiatry 2017;25(3):103-113 [FREE Full text] [CrossRef] [Medline]
- Oing T, Prescott J. Implementations of Virtual Reality for Anxiety-Related Disorders: Systematic Review. JMIR Serious Games 2018 Nov 07;6(4):e10965 [FREE Full text] [CrossRef] [Medline]
- Wechsler TF, Kümpers F, Mühlberger A. Inferiority or Even Superiority of Virtual Reality Exposure Therapy in Phobias?-A Systematic Review and Quantitative Meta-Analysis on Randomized Controlled Trials Specifically Comparing the Efficacy of Virtual Reality Exposure to Gold Standard Exposure in Agoraphobia, Specific Phobia, and Social Phobia. Front Psychol 2019 Sep 10;10:1758 [FREE Full text] [CrossRef] [Medline]
- Fodor LA, Coteț CD, Cuijpers P, Szamoskozi Ș, David D, Cristea IA. The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: A meta-analysis. Sci Rep 2018 Jul 09;8(1):10323 [FREE Full text] [CrossRef] [Medline]
- Dulau E, Botha-Ravyse C, Luimula M. Virtual reality for physical rehabilitation: A Pilot study How will virtual reality change physical therapy? 2019 Presented at: 10th IEEE International Conference on Cognitive Infocommunications (CogInfoCom); October 23-25, 2019; Naples, Italy. [CrossRef]
- Cho GH, Hwangbo G, Shin HS. The Effects of Virtual Reality-based Balance Training on Balance of the Elderly. J Phys Ther Sci 2014 Apr;26(4):615-617 [FREE Full text] [CrossRef] [Medline]
- Ahmadpour N, Randall H, Choksi H, Gao A, Vaughan C, Poronnik P. Virtual Reality interventions for acute and chronic pain management. Int J Biochem Cell Biol 2019 Sep;114:105568. [CrossRef] [Medline]
- Smith V, Warty RR, Sursas JA, Payne O, Nair A, Krishnan S, et al. The Effectiveness of Virtual Reality in Managing Acute Pain and Anxiety for Medical Inpatients: Systematic Review. J Med Internet Res 2020 Nov 02;22(11):e17980 [FREE Full text] [CrossRef] [Medline]
- Li A, Montaño Z, Chen VJ, Gold JI. Virtual reality and pain management: current trends and future directions. Pain Manag 2011 Mar;1(2):147-157 [FREE Full text] [CrossRef] [Medline]
- Hoffman HG, Patterson DR, Carrougher GJ. Use of virtual reality for adjunctive treatment of adult burn pain during physical therapy: a controlled study. Clin J Pain 2000 Sep;16(3):244-250. [CrossRef] [Medline]
- Stepan K, Zeiger J, Hanchuk S, Del Signore A, Shrivastava R, Govindaraj S, et al. Immersive virtual reality as a teaching tool for neuroanatomy. Int Forum Allergy Rhinol 2017 Oct;7(10):1006-1013. [CrossRef] [Medline]
- Taubert M, Webber L, Hamilton T, Carr M, Harvey M. Virtual reality videos used in undergraduate palliative and oncology medical teaching: results of a pilot study. BMJ Support Palliat Care 2019 Sep;9(3):281-285 [FREE Full text] [CrossRef] [Medline]
- Barteit S, Lanfermann L, Bärnighausen T, Neuhann F, Beiersmann C. Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review. JMIR Serious Games 2021 Jul 08;9(3):e29080 [FREE Full text] [CrossRef] [Medline]
- Erolin C, Reid L, McDougall S. Using virtual reality to complement and enhance anatomy education. J Vis Commun Med 2019 Jul 06;42(3):93-101. [CrossRef] [Medline]
- Maresky HS, Oikonomou A, Ali I, Ditkofsky N, Pakkal M, Ballyk B. Virtual reality and cardiac anatomy: Exploring immersive three-dimensional cardiac imaging, a pilot study in undergraduate medical anatomy education. Clin Anat 2019 Mar;32(2):238-243. [CrossRef] [Medline]
- Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ 2020 Apr 25;20(1):127 [FREE Full text] [CrossRef] [Medline]
- Hardcastle T, Wood A. The utility of virtual reality surgical simulation in the undergraduate otorhinolaryngology curriculum. J. Laryngol. Otol 2018 Nov 20;132(12):1072-1076. [CrossRef]
- De Ponti R, Marazzato J, Maresca A, Rovera F, Carcano G, Ferrario M. Pre-graduation medical training including virtual reality during COVID-19 pandemic: a report on students' perception. BMC Med Educ 2020 Sep 25;20(1):332 [FREE Full text] [CrossRef] [Medline]
- Kolbe L, Jaywant A, Gupta A, Vanderlind WM, Jabbour G. Use of virtual reality in the inpatient rehabilitation of COVID-19 patients. Gen Hosp Psychiatry 2021;71:76-81 [FREE Full text] [CrossRef] [Medline]
- Atli K, Selman W, Ray A. A Comprehensive Multicomponent Neurosurgical Course with use of Virtual Reality: Modernizing the Medical Classroom. J Surg Educ 2021;78(4):1350-1356. [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
- Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. EFI 2018 Dec 18;34(4):285-291. [CrossRef]
- Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract 2018 Jun 21;24(3):459-467. [CrossRef] [Medline]
- Cohen J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 2016 Jul 02;20(1):37-46. [CrossRef]
- Petry NM. A comparison of young, middle-aged, and older adult treatment-seeking pathological gamblers. Gerontologist 2002 Feb;42(1):92-99. [CrossRef] [Medline]
- Pigozzi LB, Pereira DD, Pattussi MP, Moret-Tatay C, Irigaray TQ, Weber JBB, et al. Quality of life in young and middle age adult temporomandibular disorders patients and asymptomatic subjects: a systematic review and meta-analysis. Health Qual Life Outcomes 2021 Mar 10;19(1):83 [FREE Full text] [CrossRef] [Medline]
- Jaworska N, MacQueen G. Adolescence as a unique developmental period. J Psychiatry Neurosci 2015 Sep;40(5):291-293 [FREE Full text] [CrossRef] [Medline]
- Slater M, Linakis V, Usoh M, Kooper R. Immersion, presence and performance in virtual environments: an experiment with tri-dimensional chess. In: VRST '96: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. 1996 Presented at: VRST96: Virtual Reality Software and Technology; July 1-4, 1996; Hong Kong. [CrossRef]
- Schultheis MT, Rizzo AA. The application of virtual reality technology in rehabilitation. Rehabilitation Psychology 2001 Aug;46(3):296-311. [CrossRef]
- Lampropoulos G, Barkoukis V, Burden K, Anastasiadis T. 360-degree video in education: An overview and a comparative social media data analysis of the last decade. Smart Learn. Environ 2021 Sep 26;8(1):A. [CrossRef]
- Riva G, Serino S. Virtual Reality in the Assessment, Understanding and Treatment of Mental Health Disorders. J Clin Med 2020 Oct 26;9(11):3434 [FREE Full text] [CrossRef] [Medline]
- Pallavicini F, Pepe A, Minissi M. Gaming in Virtual Reality: What Changes in Terms of Usability, Emotional Response and Sense of Presence Compared to Non-Immersive Video Games? Simulation & Gaming 2019 Mar 05;50(2):136-159. [CrossRef]
- Colder Carras M, Van Rooij AJ, Spruijt-Metz D, Kvedar J, Griffiths MD, Carabas Y, et al. Commercial Video Games As Therapy: A New Research Agenda to Unlock the Potential of a Global Pastime. Front Psychiatry 2017 Jan 22;8:300 [FREE Full text] [CrossRef] [Medline]
- Zyda M. From visual simulation to virtual reality to games. Computer 2005 Sep;38(9):25-32. [CrossRef]
- Maloney D, Freeman G. Falling Asleep Together: What Makes Activities in Social Virtual Reality Meaningful to Users. In: CHI PLAY '20: Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 2020 Presented at: Annual Symposium on Computer-Human Interaction in Play; November 2-4, 2020; Virtual Event (Canada) p. 510-521. [CrossRef]
- Sykownik P, Graf L, Zils C, Masuch M. 2021 Presented at: IEEE Virtual Reality and 3D User Interfaces (VR); March 27, 2021-April 01, 2021; Lisboa, Portugal. [CrossRef]
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977 Mar;33(1):159-174. [Medline]
- Birrenbach T, Zbinden J, Papagiannakis G, Exadaktylos AK, Müller M, Hautz WE, et al. Effectiveness and Utility of Virtual Reality Simulation as an Educational Tool for Safe Performance of COVID-19 Diagnostics: Prospective, Randomized Pilot Trial. JMIR Serious Games 2021 Oct 08;9(4):e29586 [FREE Full text] [CrossRef] [Medline]
- Mottelson A, Vandeweerdt C, Atchapero M, Luong T, Holz C, Böhm R, et al. A self-administered virtual reality intervention increases COVID-19 vaccination intention. Vaccine 2021 Nov 05;39(46):6746-6753 [FREE Full text] [CrossRef] [Medline]
- Garcia LM, Birckhead BJ, Krishnamurthy P, Sackman J, Mackey IG, Louis RG, et al. An 8-Week Self-Administered At-Home Behavioral Skills-Based Virtual Reality Program for Chronic Low Back Pain: Double-Blind, Randomized, Placebo-Controlled Trial Conducted During COVID-19. J Med Internet Res 2021 Feb 22;23(2):e26292 [FREE Full text] [CrossRef] [Medline]
- Vlake JH, van Bommel J, Wils E, Bienvenu J, Hellemons ME, Korevaar TI, et al. Intensive Care Unit-Specific Virtual Reality for Critically Ill Patients With COVID-19: Multicenter Randomized Controlled Trial. J Med Internet Res 2022 Jan 31;24(1):e32368 [FREE Full text] [CrossRef] [Medline]
- Liu Y, Butzlaff A. Where's the germs? The effects of using virtual reality on nursing students' hospital infection prevention during the COVID-19 pandemic. J Comput Assist Learn 2021 Dec 25;37(6):1622-1628 [FREE Full text] [CrossRef] [Medline]
- Cecil J, Kauffman S, Gupta A, McKinney V, Miguel Pirela-Cruz MD. Design of a Human Centered Computing (HCC) based Virtual Reality Simulator to train First Responders Involved in the COVID-19 Pandemic. 2021 Presented at: IEEE International Systems Conference (SysCon); April 15, 2021-May 15, 2021; Vancouver, BC, Canada p. 1-7. [CrossRef]
- Riva G, Bernardelli L, Castelnuovo G, Di Lernia D, Tuena C, Clementi A, et al. A Virtual Reality-Based Self-Help Intervention for Dealing with the Psychological Distress Associated with the COVID-19 Lockdown: An Effectiveness Study with a Two-Week Follow-Up. Int J Environ Res Public Health 2021 Aug 02;18(15):8188 [FREE Full text] [CrossRef] [Medline]
- Zhang D, Liao H, Jia Y, Yang W, He P, Wang D, et al. Effect of virtual reality simulation training on the response capability of public health emergency reserve nurses in China: a quasiexperimental study. BMJ Open 2021 Sep 22;11(9):e048611 [FREE Full text] [CrossRef] [Medline]
- Xing Y, Liang Z, Fahy C, Shell J, Guan K, Liu Y, et al. Virtual Reality Research: Design Virtual Education System for Epidemic (COVID-19) Knowledge to Public. Applied Sciences 2021 Nov 10;11(22):10586. [CrossRef]
- Jeong Y, Lee H, Han J. Development and evaluation of virtual reality simulation education based on coronavirus disease 2019 scenario for nursing students: A pilot study. Nurs Open 2022 Mar;9(2):1066-1076 [FREE Full text] [CrossRef] [Medline]
- Kang K, Kim S, Lee M, Kim M, Kim S. Comparison of Learning Effects of Virtual Reality Simulation on Nursing Students Caring for Children with Asthma. Int J Environ Res Public Health 2020 Nov 13;17(22):8417 [FREE Full text] [CrossRef] [Medline]
- da Silva TD, da Silva PL, Valenzuela EDJ, Dias ED, Simcsik AO, de Carvalho MG, et al. Serious Game Platform as a Possibility for Home-Based Telerehabilitation for Individuals With Cerebral Palsy During COVID-19 Quarantine - A Cross-Sectional Pilot Study. Front Psychol 2021;12:622678 [FREE Full text] [CrossRef] [Medline]
- Beverly E, Hommema L, Coates K, Duncan G, Gable B, Gutman T, et al. A tranquil virtual reality experience to reduce subjective stress among COVID-19 frontline healthcare workers. PLoS One 2022;17(2):e0262703 [FREE Full text] [CrossRef] [Medline]
- Nijland JWHM, Veling W, Lestestuiver BP, Van Driel CMG. Virtual Reality Relaxation for Reducing Perceived Stress of Intensive Care Nurses During the COVID-19 Pandemic. Front Psychol 2021 Sep 29;12:706527 [FREE Full text] [CrossRef] [Medline]
- Sadeghi AH, Wahadat AR, Dereci A, Budde RPJ, Tanis W, Roos-Hesselink JW, et al. Remote multidisciplinary heart team meetings in immersive virtual reality: a first experience during the COVID-19 pandemic. BMJ Innov 2021 Apr;7(2):311-315 [FREE Full text] [CrossRef] [Medline]
- Guichet PL, Huang J, Zhan C, Millet A, Kulkarni K, Chhor C, et al. Incorporation of a Social Virtual Reality Platform into the Residency Recruitment Season. Acad Radiol 2022 Jun;29(6):935-942 [FREE Full text] [CrossRef] [Medline]
- Leung LY, Malthouse T, Chong J, Down C, Crawford R, Coker C, et al. PD02-11 Utilising 360 Live Streaming to Deliver Immersive Urology Education in the COVID-19 Era with Virtual Reality in Medicine and Surgery (VRIMS). Journal of Urology 2021 Sep;206(Supplement 3):e41-e42. [CrossRef]
- Zhang W, Paudel D, Shi R, Liang J, Liu J, Zeng X, et al. Virtual Reality Exposure Therapy (VRET) for Anxiety Due to Fear of COVID-19 Infection: A Case Series. Neuropsychiatr Dis Treat 2020;16:2669-2675 [FREE Full text] [CrossRef] [Medline]
- Siani A, Marley SA. Impact of the recreational use of virtual reality on physical and mental wellbeing during the Covid-19 lockdown. Health Technol (Berl) 2021 Feb 14;11(2):425-435 [FREE Full text] [CrossRef] [Medline]
- Speidel R, Schneider A, Körner J, Grab-Kroll C, Öchsner W. Did video kill the XR star? Digital trends in medical education before and after the COVID-19 outbreak from the perspective of students and lecturers from the faculty of medicine at the University of Ulm. GMS J Med Educ 2021;38(6):Doc101 [FREE Full text] [CrossRef] [Medline]
- Wagner M, Jaki C, Löllgen RM, Mileder L, Eibensteiner F, Ritschl V, et al. Readiness for and Response to Coronavirus Disease 2019 Among Pediatric Healthcare Providers: The Role of Simulation for Pandemics and Other Disasters. Pediatr Crit Care Med 2021 Jun 01;22(6):e333-e338 [FREE Full text] [CrossRef] [Medline]
- Tsekhmister YV, Konovalova T, Tsekhmister BY, Agrawal A, Ghosh D. Evaluation of Virtual Reality Technology and Online Teaching System for Medical Students in Ukraine During COVID-19 Pandemic. Int. J. Emerg. Technol. Learn 2021 Dec 08;16(23):127-139. [CrossRef]
- Campo-Prieto P, Rodríguez-Fuentes G, Cancela-Carral JM. Immersive Virtual Reality Exergame Promotes the Practice of Physical Activity in Older People: An Opportunity during COVID-19. MTI 2021 Sep 01;5(9):52. [CrossRef]
- Yahara M, Niki K, Ueno K, Okamoto M, Okuda T, Tanaka H, et al. Remote Reminiscence Using Immersive Virtual Reality May Be Efficacious for Reducing Anxiety in Patients with Mild Cognitive Impairment Even in COVID-19 Pandemic: A Case Report. Biol Pharm Bull 2021;44(7):1019-1023 [FREE Full text] [CrossRef] [Medline]
- Yang T, Lai IKW, Fan ZB, Mo QM. The impact of a 360° virtual tour on the reduction of psychological stress caused by COVID-19. Technol Soc 2021 Feb;64:101514 [FREE Full text] [CrossRef] [Medline]
- Oulefki A, Agaian S, Trongtirakul T, Benbelkacem S, Aouam D, Zenati-Henda N, et al. Virtual Reality visualization for computerized COVID-19 lesion segmentation and interpretation. Biomed Signal Process Control 2022 Mar;73:103371 [FREE Full text] [CrossRef] [Medline]
- Yang J, Menhas R, Dai J, Younas T, Anwar U, Iqbal W, et al. Virtual Reality Fitness (VRF) for Behavior Management During the COVID-19 Pandemic: A Mediation Analysis Approach. PRBM 2022 Jan;Volume 15:171-182. [CrossRef]
- Paul M, Bullock K, Bailenson J. Virtual Reality Behavioral Activation as an Intervention for Major Depressive Disorder: Case Report. JMIR Ment Health 2020 Nov 03;7(11):e24331 [FREE Full text] [CrossRef] [Medline]
- Vlake JH, van Bommel J, Hellemons ME, Wils E, Gommers D, van Genderen ME. Intensive Care Unit-Specific Virtual Reality for Psychological Recovery After ICU Treatment for COVID-19; A Brief Case Report. Front Med (Lausanne) 2020;7:629086 [FREE Full text] [CrossRef] [Medline]
- Sampaio M, Navarro Haro MV, Wilks C, De Sousa B, Garcia-Palacios A, Hoffman H. Spanish-Speaking Therapists Increasingly Switch to Telepsychology During COVID-19: Networked Virtual Reality May Be Next. Telemed J E Health 2021 Aug;27(8):919-928 [FREE Full text] [CrossRef] [Medline]
- Sampaio M, Haro MVN, De Sousa B, Melo WV, Hoffman HG. Therapists Make the Switch to Telepsychology to Safely Continue Treating Their Patients During the COVID-19 Pandemic. Virtual Reality Telepsychology May Be Next. Front Virtual Real 2021 Jan 15;1:576421 [FREE Full text] [CrossRef] [Medline]
- Ralston B, Willett R, Namperumal S, Brown N, Walsh H, Muñoz RA, et al. Use of Virtual Reality for Pediatric Cardiac Critical Care Simulation. Cureus 2021 Jun;13(6):e15856 [FREE Full text] [CrossRef] [Medline]
- Buyego P, Katwesigye E, Kebirungi G, Nsubuga M, Nakyejwe S, Cruz P, et al. Feasibility of Virtual Reality based Training for Optimising COVID-19 Case Handling in Uganda. Res Sq 2021 Oct 04:rs.3.rs-882147 [FREE Full text] [CrossRef] [Medline]
- Herbst R, Rybak T, Meisman A, Whitehead M, Rosen B, Crosby LE, et al. A Virtual Reality Resident Training Curriculum on Behavioral Health Anticipatory Guidance: Development and Usability Study. JMIR Pediatr Parent 2021 Jun 29;4(2):e29518 [FREE Full text] [CrossRef] [Medline]
- Petrica A, Lungeanu D, Ciuta A, Marza AM, Botea M, Mederle OA. Using 360-degree video for teaching emergency medicine during and beyond the COVID-19 pandemic. Ann Med 2021 Dec;53(1):1520-1530 [FREE Full text] [CrossRef] [Medline]
- Bridge P, Shiner N, Bolderston A, Gunn T, Hazell L, Johnson R, et al. International audit of simulation use in pre-registration medical radiation science training. Radiography (Lond) 2021 Nov;27(4):1172-1178. [CrossRef] [Medline]
- Flo J, Byermoen KR, Egilsdottir H, Eide H, Heyn LG. Nursing students' experiences of virtual simulation when using a video conferencing system - a mixed methods study. Int J Nurs Educ Scholarsh 2021 Sep 10;18(1):0056. [CrossRef] [Medline]
- Barreda-Ángeles M, Hartmann T. Psychological benefits of using social virtual reality platforms during the covid-19 pandemic: The role of social and spatial presence. Comput Human Behav 2022 Feb;127:107047 [FREE Full text] [CrossRef] [Medline]
- Anderson-Hanley C, Maloney M, Barcelos N, Striegnitz K, Kramer A. Neuropsychological Benefits of Neuro-Exergaming for Older Adults: A Pilot Study of an Interactive Physical and Cognitive Exercise System (iPACES). J Aging Phys Act 2017 Jan;25(1):73-83. [CrossRef] [Medline]
- Lindner P, Miloff A, Hamilton W, Carlbring P. The Potential of Consumer-Targeted Virtual Reality Relaxation Applications: Descriptive Usage, Uptake and Application Performance Statistics for a First-Generation Application. Front Psychol 2019 Feb 4;10:132 [FREE Full text] [CrossRef] [Medline]
- Pallavicini F, Pepe A. Virtual Reality Games and the Role of Body Involvement in Enhancing Positive Emotions and Decreasing Anxiety: Within-Subjects Pilot Study. JMIR Serious Games 2020 Jun 17;8(2):e15635 [FREE Full text] [CrossRef] [Medline]
- Indovina P, Barone D, Gallo L, Chirico A, De Pietro G, Giordano A. Virtual Reality as a Distraction Intervention to Relieve Pain and Distress During Medical Procedures: A Comprehensive Literature Review. Clin J Pain 2018 Sep;34(9):858-877. [CrossRef] [Medline]
- Spiegel B, Fuller G, Lopez M, Dupuy T, Noah B, Howard A, et al. Virtual reality for management of pain in hospitalized patients: A randomized comparative effectiveness trial. PLoS One 2019;14(8):e0219115 [FREE Full text] [CrossRef] [Medline]
- Qian J, McDonough DJ, Gao Z. The Effectiveness of Virtual Reality Exercise on Individual's Physiological, Psychological and Rehabilitative Outcomes: A Systematic Review. Int J Environ Res Public Health 2020 Jun 10;17(11):4133 [FREE Full text] [CrossRef] [Medline]
- Pallavicini F, Pepe A. Comparing Player Experience in Video Games Played in Virtual Reality or on Desktop Displays: Immersion, Flow, and Positive Emotions. 2019 Presented at: Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts; October 22-25, 2019; Barcelona, Spain p. 195-210. [CrossRef]
- Li J, Erdt M, Chen L, Cao Y, Lee S, Theng Y. The Social Effects of Exergames on Older Adults: Systematic Review and Metric Analysis. J Med Internet Res 2018 Jun 28;20(6):e10486 [FREE Full text] [CrossRef] [Medline]
- Marques A, Peralta M, Martins J, Catunda R, Matos MGD, Saboga Nunes L. Associations between physical activity and self-rated wellbeing in European adults: A population-based, cross-sectional study. Prev Med 2016 Oct;91:18-23. [CrossRef] [Medline]
- Birckhead B, Eberlein S, Alvarez G, Gale R, Dupuy T, Makaroff K, et al. Home-based virtual reality for chronic pain: protocol for an NIH-supported randomised-controlled trial. BMJ Open 2021 Jun 15;11(6):e050545 [FREE Full text] [CrossRef] [Medline]
- Pedram S, Palmisano S, Perez P, Mursic R, Farrelly M. Examining the potential of virtual reality to deliver remote rehabilitation. Computers in Human Behavior 2020 Apr;105:106223. [CrossRef]
- Pallavicini F, Orena E, di Santo S, Greci L, Caragnano C, Ranieri P, et al. A virtual reality home-based training for the management of stress and anxiety among healthcare workers during the COVID-19 pandemic: study protocol for a randomized controlled trial. Trials 2022 Jun 02;23(1):451 [FREE Full text] [CrossRef] [Medline]
- Pallavicini F, Pepe A, Mantovani F. Commercial Off-The-Shelf Video Games for Reducing Stress and Anxiety: Systematic Review. JMIR Ment Health 2021 Aug 16;8(8):e28150 [FREE Full text] [CrossRef] [Medline]
- Parong J, Mayer RE. Learning science in immersive virtual reality. Journal of Educational Psychology 2018 Aug;110(6):785-797. [CrossRef]
- Webster R. Declarative knowledge acquisition in immersive virtual learning environments. Interactive Learning Environments 2015 Jan 08;24(6):1319-1333. [CrossRef]
- De Freitas FS, Rebolledo-Mendez G, Liarokapis F, Magoulas G, Poulovassilis A. Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. British Journal of Educational Technology 2010:69-85. [CrossRef]
- Pottle J. Virtual reality and the transformation of medical education. Future Healthc J 2019 Oct;6(3):181-185 [FREE Full text] [CrossRef] [Medline]
- Triandafilou KM, Tsoupikova D, Barry AJ, Thielbar KN, Stoykov N, Kamper DG. Development of a 3D, networked multi-user virtual reality environment for home therapy after stroke. J Neuroeng Rehabil 2018 Oct 05;15(1):88 [FREE Full text] [CrossRef] [Medline]
- Schild J, Misztal S, Roth B, Flock L, Luiz T, Lerner D, et al. Applying Multi-User Virtual Reality to Collaborative Medical Training. 2018 Presented at: IEEE Conference on Virtual Reality and 3D User Interfaces (VR); March 18-22, 2018; Tuebingen/Reutlingen, Germany p. 775-776. [CrossRef]
- Riva G, Wiederhold BK. What the Metaverse Is (Really) and Why We Need to Know About It. Cyberpsychol Behav Soc Netw 2022 Jun;25(6):355-359. [CrossRef] [Medline]
- Asiedu KG. Meta’s next big bet: The ‘metaversity’. Protocol. URL: https://www.protocol.com/enterprise/metaverse-in-education-morehouse-meta [accessed 2022-10-16]
- Jimenez YA, Cumming S, Wang W, Stuart K, Thwaites DI, Lewis SJ. Patient education using virtual reality increases knowledge and positive experience for breast cancer patients undergoing radiation therapy. Support Care Cancer 2018 Aug 13;26(8):2879-2888. [CrossRef] [Medline]
- Daricello L, Leonardi L, Maggio A, Orlando S, Pillitteri I, Bocchino F, et al. Virtual and Augmented Reality for increasing the awareness of current scientific research. 2020 Presented at: Europlanet Science Congress; September 21-October 9, 2020; Virtual meeting. [CrossRef]
- Palanica A, Docktor MJ, Lee A, Fossat Y. Using mobile virtual reality to enhance medical comprehension and satisfaction in patients and their families. Perspect Med Educ 2019 Apr;8(2):123-127 [FREE Full text] [CrossRef] [Medline]
- Pandrangi VC, Gaston B, Appelbaum NP, Albuquerque FC, Levy MM, Larson RA. The Application of Virtual Reality in Patient Education. Ann Vasc Surg 2019 Aug;59:184-189. [CrossRef] [Medline]
- Chang S, Kuo M, Lin Y, Chen S, Chen C, Yang Y, et al. Virtual reality-based preprocedural education increases preparedness and satisfaction of patients about the catheter ablation of atrial fibrillation. J Chin Med Assoc 2021 Jul 01;84(7):690-697. [CrossRef] [Medline]
- Jerdan SW, Grindle M, van Woerden HC, Kamel Boulos MN. Head-Mounted Virtual Reality and Mental Health: Critical Review of Current Research. JMIR Serious Games 2018 Jul 06;6(3):e14 [FREE Full text] [CrossRef] [Medline]
- Concannon BJ, Esmail S, Roduta Roberts M. Head-Mounted Display Virtual Reality in Post-secondary Education and Skill Training. Front. Educ 2019 Aug 14;4:80. [CrossRef]
- Bevilacqua R, Maranesi E, Riccardi GR, Donna VD, Pelliccioni P, Luzi R, et al. Non-Immersive Virtual Reality for Rehabilitation of the Older People: A Systematic Review into Efficacy and Effectiveness. J Clin Med 2019 Nov 05;8(11):1882 [FREE Full text] [CrossRef] [Medline]
- Hamilton D, McKechnie J, Edgerton E, Wilson C. Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. J. Comput. Educ 2020 Jul 11;8(1):1-32. [CrossRef]
- Gladden M. A Phenomenological Framework of Architectural Paradigms for the User-Centered Design of Virtual Environments. MTI 2018 Nov 30;2(4):80. [CrossRef]
- Jensen L, Konradsen F. A review of the use of virtual reality head-mounted displays in education and training. Educ Inf Technol 2017 Nov 25;23(4):1515-1529. [CrossRef]
- Cooper N, Milella F, Pinto C, Cant I, White M, Meyer G. The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment. PLoS One 2018;13(2):e0191846 [FREE Full text] [CrossRef] [Medline]
- Marsh T, Wright P, Smith S. Evaluation for the design of experience in virtual environments: modeling breakdown of interaction and illusion. Cyberpsychol Behav 2001 Apr;4(2):225-238. [CrossRef] [Medline]
- Pallavicini F, Cipresso P, Raspelli S, Grassi A, Serino S, Vigna C, et al. Is virtual reality always an effective stressors for exposure treatments? Some insights from a controlled trial. BMC Psychiatry 2013 Feb 11;13:52 [FREE Full text] [CrossRef] [Medline]
- Lindner P, Hamilton W, Miloff A, Carlbring P. How to Treat Depression With Low-Intensity Virtual Reality Interventions: Perspectives on Translating Cognitive Behavioral Techniques Into the Virtual Reality Modality and How to Make Anti-Depressive Use of Virtual Reality-Unique Experiences. Front Psychiatry 2019;10:792 [FREE Full text] [CrossRef] [Medline]
- Gabbard J, Hix D, Swan J. User-centered design and evaluation of virtual environments. IEEE Comput. Grap. Appl 1999;19(6):51-59. [CrossRef]
- Pizzoli SFM, Mazzocco K, Triberti S, Monzani D, Alcañiz Raya ML, Pravettoni G. User-Centered Virtual Reality for Promoting Relaxation: An Innovative Approach. Front Psychol 2019;10:479 [FREE Full text] [CrossRef] [Medline]
- Pallavicini F, Orena E, di Santo S, Greci L, Caragnano C, Ranieri P, et al. MIND-VR: Design and Evaluation Protocol of a Virtual Reality Psychoeducational Experience on Stress and Anxiety for the Psychological Support of Healthcare Workers Involved in the COVID-19 Pandemic. Front. Virtual Real 2021 Feb 2;2:620225. [CrossRef]
- Angelov V, Petkov E, Shipkovenski G, Kalushkov T. Modern Virtual Reality Headsets. 2020 Presented at: International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA); June 26-28, 2020; Ankara, Turkey p. 1-5. [CrossRef]
|HMD: head-mounted display|
|ICT: information and communication technology|
|ICU: intensive care unit|
|MMAT: Mixed Methods Appraisal Tool|
|RCT: randomized controlled trial|
|VR: virtual reality|
Edited by N Zary; submitted 16.11.21; peer-reviewed by P Lindner, J Vlake, T Birrenbach; comments to author 17.01.22; revised version received 18.04.22; accepted 04.08.22; published 25.10.22Copyright
©Federica Pallavicini, Alessandro Pepe, Massimo Clerici, Fabrizia Mantovani. Originally published in JMIR Serious Games (https://games.jmir.org), 25.10.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.