Background: Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by impairments in social interaction and repetitive patterns of behavior, which can lead to deficits in adaptive behavior. In this study, a serious game was developed to train individuals with ASD for an important type of outdoor activity, which is the use of buses as a means of transportation.
Objective: The aim of this study was to develop a serious game that defines a “safe environment” where the players became familiar with the process of taking a bus and to validate if it could be used effectively to teach bus-taking routines and adaptive procedures to individuals with ASD.
Methods: In the game, players were placed in a three-dimensional city and were submitted to a set of tasks that involved taking buses to reach specific destinations. Participants with ASD (n=10) underwent between 1 to 3 training sessions. Participants with typical development (n=10) were also included in this study for comparison purposes and received 1 control session.
Results: We found a statistically significant increase in the measures of knowledge of the process of riding a bus, a reduction in the electrodermal activity (a metric of anxiety) measured inside the bus environments, and a high success rate of their application within the game (93.8%).
Conclusions: The developed game proved to be potentially useful in the context of emerging immersive virtual reality technologies, of which use in therapies and serious games is still in its infancy. Our findings suggest that serious games, using these technologies, can be used effectively in helping people with ASD become more independent in outdoor activities, specifically regarding the use of buses for transportation.
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder responsible for impairments in social communication and interaction and restricted repetitive patterns of behavior, interests, or activities. Current ASD prevalence is estimated to be around 1% of the population, worldwide . Given that a cure is yet to be found, individuals rely on interventional approaches to improve and overcome their impairments. Further increasing the relevance of rehabilitation driven strategies is the fact that still only a minority of individuals with ASD are able to live independently in adulthood [ ].
Virtual reality (VR) consists of artificial, 3D, computer-generated environments which the user can explore and interact with . VR usually takes advantage of different types of devices such as head-mounted displays and controllers to deliver the stimulation and provide means of interaction that lead to immersive experiences [ ]. The immersiveness of an experience is measured by the level of fidelity, concerning all sensory modalities, that a VR system can provide. Thus, immersion is objective, measurable, and depends only on the technology used by the VR system. Presence, on the other hand, is the human reaction to immersion, the feeling of actually being in the virtual environment and behaving as such [ , , ]. In fact, the immersive VR technologies available nowadays are able to present users with experiences realistic enough to trick the mind and create a feeling of presence within the environment [ ]. These technologies have already been used and proven effective in therapies for posttraumatic stress disorder [ ], phobias [ ], as well as ASD [ ]. There are several reasons that justify the use of VR in those approaches. Its capacity to provide safe, realistic, and controlled environments [ ] make therapy possible for people who, due to physical (eg paralysis) or psychological (eg, anxiety) reasons, cannot undergo exposure in real life situations. Moreover, this creates the possibility of applying therapies at ease without the need to go to a specific location where real exposure occurs. Finally, even when using low-budget hardware and software, it has been proven that VR therapies can be as effective as exposure in real life [ ].
Serious games have also been proven effective in ASD therapies, not only for including game design techniques to keep the players motivated, but also because individuals with ASD are often interested in computer-based activities . However, according to Zakari et al [ ], most of the serious games developed for ASD rehabilitation between 2004 and 2014 are delivered through nonimmersive VR (eg, desktops, tablets) and focus mainly on communication and social skill development [ ]. This highlights both the importance of focusing on other relevant ASD impairments, including executive function in outdoor situations, as well as exploring the potential of these new immersive technologies.
The aim of this study was to develop and validate a serious game which uses a VR headset (Oculus Rift) as a proof of concept for rehabilitation of individuals with ASD. Teaching a person with ASD to use public transportation requires parents or therapists to practice with them until they are ready and comfortable enough to perform this task alone (basically, the same process used with people with typical development, but with all the obstacles set by ASD particularities). In fact, being able to engage in outdoor activities such as the efficient use of public transport can be particularly challenging for people with ASD due to deficits in adaptive behavior  and anxiety [ ]. This project intended to ease this process by creating and validating a game that prepares the players to use, in this case, buses for transportation. This included not only teaching the required skills, but also making them comfortable with the involved procedures and environments. To our knowledge, this is the first study to use VR training for teaching the process of bus-taking for people with ASD.
In this section, we describe the game, the experimental setup, the participants, and the validation procedure.
The game was developed in-house with the aim of preparing individuals with ASD to use buses for transportation. To achieve that, it places the user in a three-dimensional city and sets a task that is completed by riding buses to reach a specific destination.shows different screenshots of the city and the buses.
There are several different buses driving in 4 pre-defined routes within the city.shows the map of the city with the 4 bus lines available. The player can enter any of these buses, validate his/her ticket, choose a place to sit, and press the STOP button, requesting the bus to stop, and leave the bus. Before starting the game, it is possible to choose from 7 different tasks, 4 of them labelled by complexity as simple (the player only needs to take 1 bus to reach the destination) and 3 labelled as complex (the player needs to take 2 buses to reach the destination). Each task has 2 levels of difficulty: an easy and a hard mode. The easy mode leads the player, step by step, to the destination, while in the hard mode, the player is only told the place he/she must go to. At the end of each task, a scoring system evaluates the performance of the player on 2 different components: “Actions” (the capacity of the player to memorize and execute bus norms, eg, validating the ticket or sitting on unreserved seats) and “Route” (the capacity of the player to plan a route to the destination, eg, if the player took the right buses in the right bus stops).
The game includes several objects/elements, such as people, traffic, and dogs, which might cause anxiety, depending on the player. For that reason, a biofeedback system was implemented to ensure that, while the player becomes familiarized with the environment, it never becomes hostile to him/her. This is achieved by assessing the anxiety felt by the player through the analysis of the electrodermal activity (EDA) and reducing the stimulus clutter the player is exposed to, in real time, in case high anxiety levels are reached (eg, reducing the amount of noise in the environment).schematizes the biofeedback loop.
For this project, 2 groups were selected: a clinical and a control group. For the clinical group, 10 participants with ASD, whose diagnosis followed the criteria established on the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition , were recruited from Associação Portuguesa para as Perturbações do Desenvolvimento e Autismo de Viseu (APPDA-Viseu). For the clinical group, 9 males and 1 female were recruited, with a mean age of 18.8 (SD 4.5) years. In the same group, 8 were without intellectual disability (IQ equal to 70 or higher) and 2 were with mild intellectual disability (IQ between 50 and 69 [ ]). For the control group, 10 individuals with typical development (TD) were also recruited; 4 males and 6 females, with a mean age of 21.9 (SD 3.56) years. The groups were matched by age (t(18)=−1.633, P=.12). Participants gave oral consent, and a written informed consent was obtained from their parents/guardians or themselves if they were adults with sufficient autonomy.
The ASD group underwent an intervention of 3 sessions of increasing complexity and difficulty (see), with a duration between 20 and 40 minutes each. Of the 10 participants recruited with ASD, only 6 performed all the intervention sessions of the study due to scheduling issues. During the recruitment, participants completed a questionnaire to assess their experience in using buses for transportation. With an exception for 1 of the participants, all the participants were unable to take buses autonomously at the beginning of the study.
The control group was submitted to a single session (corresponding to the first of the patients). This group was used to provide a task baseline and to assess the capacity of the serious game to identify differences between groups.
The sessions took place in APPDA-Viseu for the ASD participants and in the Institute of Biomedical Imaging and Life Sciences for the TD participants. In all of them, the same research staff was present. This study and all of the procedures were reviewed and approved by the Ethics Commission of our Faculty of Medicine from University of Coimbra and were conducted in accordance with the declaration of Helsinki.
In each session, the players received a tutorial (to learn or review the game controls) and a task. The task difficulty and complexity changed from session to session, as shown in. At the end of every session, participants were asked to describe the process of riding a bus, from the moment they arrived at the bus stop, until they reached their destination, but never received feedback on the answer given. Their responses were recorded in a checklist containing all the steps existing in a bus trip ( ).
In every session, players sat on a swivel chair and wore a bracelet for wireless EDA recording (Biopac Bionomadix BN-PPGED and MP150 amplifier). All tasks were run on a laptop computer (Windows 8.1, 16.0 GB RAM and an IntelCore i7 2.50 GHz processor). The head-mounted display used was Oculus Rift Development Kit 2, firmware version 2.12, and a gamepad was used for input. Three participants with ASD and 1 with TD received the tasks without the Oculus Rift, due to vision impairments.illustrates the setup used.
Checklist with steps existing in a bus trip.
- Wait for the bus
- Enter in the bus
- Validate ticket
- Avoid reserved seats
- Wait until getting close to the destination
- Press stop button
- Leave the bus
Metrics and Outcome Measures
Because the main objective of the serious game was to teach how to take the bus, we defined two main outcome measures to evaluate the knowledge of the process of riding the bus. One is measured automatically by the game and the other is measured using the debriefing. Both consist of the percentage of the checklist () performed correctly.
The game identified every action of the participant (entering the bus, ticket validation, etc) and calculated the actions accuracy based on the equation: number of correct actions/number of expected actions ().
After the game, we asked the participants to describe the step-by-step process of riding a bus and calculated an accuracy based on the same equation of the actions accuracy.
Duration, in minutes, of the time taken to complete the task in each session. Since the tasks changes in complexity and difficulty, this metric is not directly comparable between sessions. Nevertheless, it is useful to analyze intersubject variability and to perform intergroup comparisons for the first session.
For every metric, normality was assessed using histograms, Q-Q plots and the Kolmogorov-Smirnov test. Normality test results were used to choose between parametric and nonparametric test, accordingly.
Clinical vs Control Group Analysis:
The metrics actions accuracy, debriefing accuracy, and task duration were not normally distributed. Therefore, we used the Mann-Whitney U test for the between-group comparison of those metrics. Anxiety Level data was nevertheless normally distributed. Thus, we used two-sample t test for those between-groups comparisons.
Intervention Analysis (Intragroup Analysis for the Clinical Group):
As expected from the intergroup analysis, Actions Accuracy, Debriefing Accuracy, and Task Duration were not normally distributed. Paired data for last and first session were therefore compared using a Wilcoxon Sign-rank test. Regarding Anxiety Level, a one-sample t test was used for the paired data (last and first session). Additionally, we generated heat maps of locations where consistent peaks of anxiety where identified.
The results are organized in two sections, one for the intergroup analysis and another for the within-subject analysis of the clinical group. In the figures presented, the error bars always represent the standard error of the mean, and the horizontal bar represents the median when considering nonparametric data (actions and debriefing accuracies, as well as task durations) and the mean when considering parametric data (EDA measures).
Clinical vs Control Group Results
Regarding the actions accuracy (percentage of correct actions performed during the task), we found a statistically significant difference between groups. The Mann-Whitney U test indicated that the actions accuracy of the control group (median=100%) was greater than for the clinical group (median=88%), U=75.0, P=.01. When analyzing the knowledge retained by the debriefing, the control group (median=100%) has greater accuracy than the clinical group (median=62.5%), as shown by the Mann-Whitney U test, U=80.5, P=.02 ().
In terms of Task Duration,shows the between-group differences. A Mann-Whitney U test showed that the task duration of the ASD group (median=3.76) was statistically significant higher than for the control group (median=3.19), U=18.0, P=.02.
Regarding the anxiety level of each group, we see a trend for higher values for the clinical group for all the scenarios (global EDA, inside the bus and outdoors, in the street).shows the mean EDA fluctuations of each group in each condition. However, two-sample t tests showed no statistically significant differences (possibly because of the biofeedback implementation) for either of the conditions (global EDA: t(18)=−0.60, P=.56; bus condition: t(18)=−0.99, P=.33; streets condition: t(18)=−0.48, P=.63).
We then compared the main outcome measures pre- and postintervention for the 6 participants who completed the 3 sessions. The in-game measure (actions accuracy) evolved positively throughout the sessions (preintervention median accuracy=75.0%, postintervention median accuracy=93.8%; see), with a Wilcoxon signed-rank test showing a small trend for significant differences (Z=1.63, P=.10). Even though the differences were not statistically significant when using “in-game” measures, the same test applied to the Debriefing Accuracy show a significant increase (Z=2.22, P=.03) from session 1 (median=68.8%) to session 3 (median=100.0%). illustrates the evolution of the debriefing checklist accuracy throughout the sessions.
Regarding the task duration, since the tasks increased in complexity and difficulty along the intervention, the task duration is not directly comparable. However, it is important to verify that the time to successfully complete the task did not statistically increase, even with exposure to harder levels (). In session 1, the task was performed in the “easy” mode, in which players are guided step by step, from the starting position until the final location. Tasks in “easy” mode do not require as much planning as the ones in “hard” mode, but slowly introduce the player to this concept and demonstrate how buses can be used to travel between bus stations. The “hard” mode tasks, on the other hand, only tell the player where they must go to complete the task (eg, “go to the hospital”). In these tasks, players have to analyze the map to discover which bus or buses they can take to reach the final destination. Furthermore, the tasks received in sessions 1 and 2 required the player to take 1 bus, while the task from session 3 required the player to take 2 buses. A Wilcoxon signed-rank test show no difference between the easiest and simpler (first session) task and the most complex and difficult task (last session), Z=0.105, P=.92.
The anxiety levels decreased from the first to the last session, for both overall EDA and for the bus and streets conditions.shows the mean differences and their error. However, only the bus condition shows a strong tendency to significance (t(5)=−2.36, P=.07). The general EDA values showed a weak tendency towards an effect (t(5)=−1.93, P=.11) and the EDA values on the streets had the smaller decrease with the intervention (t(5)=−1.48, P=.20).
We created heatmaps using the peaks of anxiety of each participant and separated the conditions inside the bus and outside.shows the streets scenario for each of the sessions, and shows the same metrics but for the inside the bus condition.
According to, players felt most anxious in 3 different situations ( ).
Situations in which players felt anxious.
- When waiting for the bus at the bus stops (red areas near the bus stop signs)
- When planning the trip or looking for the correct bus stop in the starting areas (ie, green areas near the restaurant in session 1, the fire station in stage 2, one of the bus stops in session 3)
- When reaching or looking for the destination in the finishing areas (ie, green areas near the police station in session 1, the hospital in session 2, and the fire station in session 3).
This project aimed to assess the potential of the developed serious game and here presents it as a tool for rehabilitation. The game is, to our knowledge, the first application specifically developed to teach ASD individuals to use public transportation systems. In addition, we were able to measure psychophysiological markers of anxiety, paving the way for future biofeedback applications. With only three sessions, it was possible to improve the knowledge of the participants regarding the norms of the bus-taking process and to reduce the anxiety levels felt by the participants during that process.
The impacts of a learning tool with this purpose are broad since it trains executive functions and might increase the autonomy of the users, providing them with a new way of moving through a city. It is also a way to make cities more inclusive, providing people with special needs ways to successfully use this type of public service.
Some studies have been conducted using VR training for ASD, usually focusing on training other skills. Most interventional approaches target social performance training [, ] or job interviewing [ ]. Gaming platforms [ ] and brain-computer interfaces [ ] were also suggested in the literature for autism training, but without validation with patients. Although these are important targets of intervention, our work focuses on a more specific task of executive function that is relevant for the needs of daily life. Our pilot validation study aimed to assess not only the efficacy of the application, but also the acceptance of the solution with this specific clinical population. Few studies perform fully immersive interventions, and the difficulties of combining them with biofeedback create a technology apparatus that could potentially be disruptive to the participants. The 4 drop-outs during the intervention occurred exclusively due to scheduling issues. No patient dropped the study due to discomfort or raised any difficulty in using the setup. There were specific cases where the Oculus device was not used, but all of those were related to vision impairments, not to lack of tolerance from the user.
The baseline comparison with the control group was clear in identifying the impairments in the clinical group. Both the debriefing of the procedure for taking the bus and the in-game actions showed statistically different results between groups, proving the validity of the rehabilitation target and confirming the capacity of the game to, by itself, identify the deficits of target participants in the process of taking the bus. Additionally, the time needed to complete the task was increased for the clinical group. Despite having mean anxiety levels above the control group, this difference was not significant, perhaps due to the small statistical power resulting from a small group of participants, as well as the implementation of biofeedback, which decreases differences between groups.
The intervention was successful in increasing the accuracy of the process description during the debriefing, showing a statistically significant improvement in the theoretical knowledge of the process, which was the main outcome measure. When evaluated inside the game by the user actions, the increase was not statistically significant, but showed a tendency that we believe additional sessions or a larger intervention group would further confirm. It was also successful in decreasing the anxiety felt by participants, especially inside the bus. By using heat maps to represent the anxiety peaks recorded, it was possible to understand that participants with ASD felt more anxious in bus stops and near the starting and finishing areas. This led to the conclusion that, when outside the buses, players felt most anxious when planning the trip, when looking for the bus stop, when waiting for the bus, and when looking for the final destination. Inside the bus, we observed a desensitization to stress throughout the sessions, with a final session showing fewer peaks of EDA activity.
Despite the increase of task complexity and difficulty across sessions, the time duration to complete the task did not increase, suggesting a learning effect and adaptation to the serious game.
By using the game as a therapeutic intervention tool, in just three sessions it was possible to improve the general efficiency of participants and expose them to peculiar scenarios in which they could train their planning skills. More importantly, it was possible to nearly extinguish the anxiety felt in bus environments and teach the bus-taking norms necessary for the autonomous use of buses for transportation, both in theoretical and practical contexts. Future studies should conduct randomized controlled trials, with larger intervention groups, to replicate the findings and extend them to other clinical populations with executive function deficits and lack of autonomy.
This study was supported by the AAC SI/2011/HomeTech/QREN Compete, cofinanced by FEDER, the Portuguese Foundation for Science and Technology, the European Projects BRAINTRAIN (FP7-HEALTH-2013-Innovation-1-602186BrainTrain), H2020-STIPED Project number: 731827, FCT (Fundação para a Ciência e Tecnologia) UID/NEU/04539/2013, POCI-01-0145-FEDER-007440, and PhD grant SFRH/BD/77044/2011. The authors would like to thank APPDA-Viseu and all the participants and parents who collaborated in this study.
Conflicts of Interest
- American PA. Diagnostic and Statistical Manual of Mental Disorders. 5th edition. Arlington: American Psychiatric Publishing; 2013.
- Gutierrez M, Vexo F, Thalmann D. Stepping into virtual reality, 1st edition. London: Springer-Verlag; 2008.
- Kim G. Designing Virtual Reality Systems, 1st edition. London: Springer-Verlag; 2005.
- Slater M. A Note on Presence Terminology. Presence-Connect 2003;3(3):1-5.
- Slater M, Wilbur S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 1997 Dec;6(6):603-616. [CrossRef]
- Sanchez-Vives MV, Slater M. From presence to consciousness through virtual reality. Nat Rev Neurosci 2005 Apr;6(4):332-339. [CrossRef] [Medline]
- Bimber O. Thinking Virtual. Computer 2014 Jul;47(7):22-23. [CrossRef]
- Rizzo A, Hartholt A, Grimani M, Leeds A, Liewer M. Virtual Reality Exposure Therapy for Combat-Related Posttraumatic Stress Disorder. Computer 2014 Jul;47(7):31-37. [CrossRef]
- Emmelkamp PMG, Krijn M, Hulsbosch AM, de VS, Schuemie MJ, van DMCAPG. Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia. Behav Res Ther 2002 May;40(5):509-516. [Medline]
- Strickland D, Marcus LM, Mesibov GB, Hogan K. Brief report: two case studies using virtual reality as a learning tool for autistic children. J Autism Dev Disord 1996 Dec;26(6):651-659. [Medline]
- Parsons S, Cobb S. State-of-the-art of virtual reality technologies for children on the autism spectrum. European Journal of Special Needs Education 2011 Aug;26(3):355-366. [CrossRef]
- Chen SH, Bernard-Opitz V. Comparison of personal and computer-assisted instruction for children with autism. Ment Retard 1993 Dec;31(6):368-376. [Medline]
- Zakari H, Ma M, Simmons D. A Review of Serious Games for Children with Autism Spectrum Disorders (ASD). In: Ma M., Oliveira MF, Baalsrud Hauge J. editors. Serious Games Development and Applications. SGDA 2014. Lecture Notes in Computer Science, vol 8778. Cham: Springer; 2014:93-106.
- Mouga S, Almeida J, Café C, Duque F, Oliveira G. Adaptive profiles in autism and other neurodevelopmental disorders. J Autism Dev Disord 2015 Apr;45(4):1001-1012. [CrossRef] [Medline]
- White S, Oswald D, Ollendick T, Scahill L. Anxiety in children and adolescents with autism spectrum disorders. Clin Psychol Rev 2009;29(3):216-229.
- ICD 1. Classifications of Mental and Behavioural Disorder. In: Clinical Descriptions and Disgnostic Guidelines, 1st edition. Geneva: World Health Organisation; 1992.
- Wied P. js - Dynamic Heatmaps for the Web. heatmap URL: https://www.patrick-wied.at/static/heatmapjs/ [accessed 2017-07-11] [WebCite Cache]
- Stichter JP, Laffey J, Galyen K, Herzog M. iSocial: delivering the Social Competence Intervention for Adolescents (SCI-A) in a 3D virtual learning environment for youth with high functioning autism. J Autism Dev Disord 2014 Feb;44(2):417-430. [CrossRef] [Medline]
- Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB. Virtual reality social cognition training for young adults with high-functioning autism. J Autism Dev Disord 2013 Jan;43(1):34-44 [FREE Full text] [CrossRef] [Medline]
- Smith MJ, Ginger EJ, Wright K, Wright MA, Taylor JL, Humm LB, et al. Virtual reality job interview training in adults with autism spectrum disorder. J Autism Dev Disord 2014 Oct;44(10):2450-2463 [FREE Full text] [CrossRef] [Medline]
- Simões M, Mouga S, Pedrosa F, Carvalho P, Oliveira G, Branco MC. Neurohab: A Platform for Virtual Training of Daily Living Skills in Autism Spectrum Disorder. Procedia Technology 2014;16:1417-1423. [CrossRef]
- Simões M, Carvalho P, Castelo-Branco M. Virtual reality and brain-computer interface for joint-attention training in autism. 2012 Presented at: Proceedings of the 9th Intl Conf. on Disability, Virtual Reality and Assoc. Technologies; Sep; Laval, France p. 10-12.
|ASD: Autism Spectrum Disorder
|EDA: electrodermal activity
|HMD: head-mounted display
|IQ: Intelligence Quotient
|TD: typical development
|VR: virtual reality
Edited by G Eysenbach; submitted 13.07.17; peer-reviewed by B Bie; comments to author 17.11.17; revised version received 20.11.17; accepted 22.11.17; published 20.03.18Copyright
©Marco Simões, Miguel Bernardes, Fernando Barros, Miguel Castelo-Branco. Originally published in JMIR Serious Games (http://games.jmir.org), 20.03.2018.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.