This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by impairments in social interaction and repetitive patterns of behavior, which can lead to deficits in adaptive behavior. In this study, a serious game was developed to train individuals with ASD for an important type of outdoor activity, which is the use of buses as a means of transportation.
The aim of this study was to develop a serious game that defines a “safe environment” where the players became familiar with the process of taking a bus and to validate if it could be used effectively to teach bus-taking routines and adaptive procedures to individuals with ASD.
In the game, players were placed in a three-dimensional city and were submitted to a set of tasks that involved taking buses to reach specific destinations. Participants with ASD (n=10) underwent between 1 to 3 training sessions. Participants with typical development (n=10) were also included in this study for comparison purposes and received 1 control session.
We found a statistically significant increase in the measures of knowledge of the process of riding a bus, a reduction in the electrodermal activity (a metric of anxiety) measured inside the bus environments, and a high success rate of their application within the game (93.8%).
The developed game proved to be potentially useful in the context of emerging immersive virtual reality technologies, of which use in therapies and serious games is still in its infancy. Our findings suggest that serious games, using these technologies, can be used effectively in helping people with ASD become more independent in outdoor activities, specifically regarding the use of buses for transportation.
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder responsible for impairments in social communication and interaction and restricted repetitive patterns of behavior, interests, or activities. Current ASD prevalence is estimated to be around 1% of the population, worldwide [
Virtual reality (VR) consists of artificial, 3D, computer-generated environments which the user can explore and interact with [
Serious games have also been proven effective in ASD therapies, not only for including game design techniques to keep the players motivated, but also because individuals with ASD are often interested in computer-based activities [
The aim of this study was to develop and validate a serious game which uses a VR headset (Oculus Rift) as a proof of concept for rehabilitation of individuals with ASD. Teaching a person with ASD to use public transportation requires parents or therapists to practice with them until they are ready and comfortable enough to perform this task alone (basically, the same process used with people with typical development, but with all the obstacles set by ASD particularities). In fact, being able to engage in outdoor activities such as the efficient use of public transport can be particularly challenging for people with ASD due to deficits in adaptive behavior [
In this section, we describe the game, the experimental setup, the participants, and the validation procedure.
The game was developed in-house with the aim of preparing individuals with ASD to use buses for transportation. To achieve that, it places the user in a three-dimensional city and sets a task that is completed by riding buses to reach a specific destination.
There are several different buses driving in 4 pre-defined routes within the city.
The game includes several objects/elements, such as people, traffic, and dogs, which might cause anxiety, depending on the player. For that reason, a biofeedback system was implemented to ensure that, while the player becomes familiarized with the environment, it never becomes hostile to him/her. This is achieved by assessing the anxiety felt by the player through the analysis of the electrodermal activity (EDA) and reducing the stimulus clutter the player is exposed to, in real time, in case high anxiety levels are reached (eg, reducing the amount of noise in the environment).
Screenshot from the virtual environment, showing two views from the bus stop perspective on the top, and two views from inside the bus on the bottom. On the top left corner, we can see one bus stop, with other people waiting for the bus, and the map to be used by the participant on the wall. On the top right, we see a bus with its designated number signaled in red, and some traffic on the street. The bottom images show two perspectives from inside of the bus.
City map showing the bus lines, stops, and important places like the hospital, church, restaurant, and others.
Biofeedback loop diagram. The level of electrodermal activity is measured from the participant by the game. If it detects a peak of activity, it decreases the level of stimuli and noise in the scene.
For this project, 2 groups were selected: a clinical and a control group. For the clinical group, 10 participants with ASD, whose diagnosis followed the criteria established on the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition [
The ASD group underwent an intervention of 3 sessions of increasing complexity and difficulty (see
The control group was submitted to a single session (corresponding to the first of the patients). This group was used to provide a task baseline and to assess the capacity of the serious game to identify differences between groups.
The sessions took place in APPDA-Viseu for the ASD participants and in the Institute of Biomedical Imaging and Life Sciences for the TD participants. In all of them, the same research staff was present. This study and all of the procedures were reviewed and approved by the Ethics Commission of our Faculty of Medicine from University of Coimbra and were conducted in accordance with the declaration of Helsinki.
In each session, the players received a tutorial (to learn or review the game controls) and a task. The task difficulty and complexity changed from session to session, as shown in
In every session, players sat on a swivel chair and wore a bracelet for wireless EDA recording (Biopac Bionomadix BN-PPGED and MP150 amplifier). All tasks were run on a laptop computer (Windows 8.1, 16.0 GB RAM and an IntelCore i7 2.50 GHz processor). The head-mounted display used was Oculus Rift Development Kit 2, firmware version 2.12, and a gamepad was used for input. Three participants with ASD and 1 with TD received the tasks without the Oculus Rift, due to vision impairments.
Task complexity and difficulty per session.
Metric | Session 1 | Session 2 | Session 3 |
Difficulty | Easy | Hard | Hard |
Complexity | Simple | Simple | Complex |
Wait for the bus
Enter in the bus
Validate ticket
Avoid reserved seats
Sit
Wait until getting close to the destination
Press stop button
Leave the bus
Diagram of the setup used during the sessions, including the virtual reality headset, game controller, biosignal recorder, and the main computer.
Because the main objective of the serious game was to teach how to take the bus, we defined two main outcome measures to evaluate the knowledge of the process of riding the bus. One is measured automatically by the game and the other is measured using the debriefing. Both consist of the percentage of the checklist (
The game identified every action of the participant (entering the bus, ticket validation, etc) and calculated the actions accuracy based on the equation: number of correct actions/number of expected actions (
After the game, we asked the participants to describe the step-by-step process of riding a bus and calculated an accuracy based on the same equation of the actions accuracy.
Duration, in minutes, of the time taken to complete the task in each session. Since the tasks changes in complexity and difficulty, this metric is not directly comparable between sessions. Nevertheless, it is useful to analyze intersubject variability and to perform intergroup comparisons for the first session.
The variation of EDA values during the session was used as a measure of anxiety. To calculate it, we detrended the signal, subtracting it to its best fit to a straight-line (for details, see detrend implementation in MathWorks Matlab), then calculated the standard deviations of the detrended EDA values. We then created heat maps of anxiety peaks in the “virtual city”, where it is possible to highlight the game locations where EDA peaks occurred, corresponding to anxiety events felt by the players. If the locations of anxiety peaks are the same between subjects and sessions, those locations will become red, but if they are sparse, their representations are green and blue. The heat maps were created using heatmap.js [
For every metric, normality was assessed using histograms, Q-Q plots and the Kolmogorov-Smirnov test. Normality test results were used to choose between parametric and nonparametric test, accordingly.
The metrics actions accuracy, debriefing accuracy, and task duration were not normally distributed. Therefore, we used the Mann-Whitney U test for the between-group comparison of those metrics. Anxiety Level data was nevertheless normally distributed. Thus, we used two-sample
As expected from the intergroup analysis, Actions Accuracy, Debriefing Accuracy, and Task Duration were not normally distributed. Paired data for last and first session were therefore compared using a Wilcoxon Sign-rank test. Regarding Anxiety Level, a one-sample
The results are organized in two sections, one for the intergroup analysis and another for the within-subject analysis of the clinical group. In the figures presented, the error bars always represent the standard error of the mean, and the horizontal bar represents the median when considering nonparametric data (actions and debriefing accuracies, as well as task durations) and the mean when considering parametric data (EDA measures).
Regarding the actions accuracy (percentage of correct actions performed during the task), we found a statistically significant difference between groups. The Mann-Whitney U test indicated that the actions accuracy of the control group (median=100%) was greater than for the clinical group (median=88%), U=75.0,
In terms of Task Duration,
Regarding the anxiety level of each group, we see a trend for higher values for the clinical group for all the scenarios (global EDA, inside the bus and outdoors, in the street).
Left: Actions accuracy for both groups in the first session. The control group (typical development) had a perfect performance, while participants in the clinical group missed some actions. Right: Debriefing accuracy for both groups. Higher, but not perfect, accuracy was found in the control group.
Task duration for session 1 from each group.
Anxiety levels for each group (mean and standard error) for the overall task and two subconditions: inside the bus and outside (streets). The clinical group presents higher values for all the settings, although without statistical significance. EDA: electrodermal activity.
We then compared the main outcome measures pre- and postintervention for the 6 participants who completed the 3 sessions. The in-game measure (actions accuracy) evolved positively throughout the sessions (preintervention median accuracy=75.0%, postintervention median accuracy=93.8%; see
Regarding the task duration, since the tasks increased in complexity and difficulty along the intervention, the task duration is not directly comparable. However, it is important to verify that the time to successfully complete the task did not statistically increase, even with exposure to harder levels (
The anxiety levels decreased from the first to the last session, for both overall EDA and for the bus and streets conditions.
We created heatmaps using the peaks of anxiety of each participant and separated the conditions inside the bus and outside.
According to
Actions accuracy for the clinical group, measured “inside the game” throughout the intervention sessions.
Debriefing accuracy of the intervention group for each session.
Time taken by each participant in the clinical group to complete the task in each intervention session.
Decrease observed in anxiety levels, measured by electrodermal activity variability, between the last session and the first. EDA: electrodermal activity.
Anxiety peaks heat map from session 1 (left) to session 3 (right) for the times the participant was not inside of the bus environment. Most of the locations represent bus stops, where participants need to make the decision of what bus to take and wait for it to arrive.
Anxiety peaks heat map from session 1 (left) to session 3 (right) for the times the participant was inside of the bus environment. The locations are much more dispersed through the route than in the outside the bus scenario. There is a visible decrease in frequency of anxiety peaks from the first to the last session.
When waiting for the bus at the bus stops (red areas near the bus stop signs)
When planning the trip or looking for the correct bus stop in the starting areas (ie, green areas near the restaurant in session 1, the fire station in stage 2, one of the bus stops in session 3)
When reaching or looking for the destination in the finishing areas (ie, green areas near the police station in session 1, the hospital in session 2, and the fire station in session 3).
This project aimed to assess the potential of the developed serious game and here presents it as a tool for rehabilitation. The game is, to our knowledge, the first application specifically developed to teach ASD individuals to use public transportation systems. In addition, we were able to measure psychophysiological markers of anxiety, paving the way for future biofeedback applications. With only three sessions, it was possible to improve the knowledge of the participants regarding the norms of the bus-taking process and to reduce the anxiety levels felt by the participants during that process.
The impacts of a learning tool with this purpose are broad since it trains executive functions and might increase the autonomy of the users, providing them with a new way of moving through a city. It is also a way to make cities more inclusive, providing people with special needs ways to successfully use this type of public service.
Some studies have been conducted using VR training for ASD, usually focusing on training other skills. Most interventional approaches target social performance training [
The baseline comparison with the control group was clear in identifying the impairments in the clinical group. Both the debriefing of the procedure for taking the bus and the in-game actions showed statistically different results between groups, proving the validity of the rehabilitation target and confirming the capacity of the game to, by itself, identify the deficits of target participants in the process of taking the bus. Additionally, the time needed to complete the task was increased for the clinical group. Despite having mean anxiety levels above the control group, this difference was not significant, perhaps due to the small statistical power resulting from a small group of participants, as well as the implementation of biofeedback, which decreases differences between groups.
The intervention was successful in increasing the accuracy of the process description during the debriefing, showing a statistically significant improvement in the theoretical knowledge of the process, which was the main outcome measure. When evaluated inside the game by the user actions, the increase was not statistically significant, but showed a tendency that we believe additional sessions or a larger intervention group would further confirm. It was also successful in decreasing the anxiety felt by participants, especially inside the bus. By using heat maps to represent the anxiety peaks recorded, it was possible to understand that participants with ASD felt more anxious in bus stops and near the starting and finishing areas. This led to the conclusion that, when outside the buses, players felt most anxious when planning the trip, when looking for the bus stop, when waiting for the bus, and when looking for the final destination. Inside the bus, we observed a desensitization to stress throughout the sessions, with a final session showing fewer peaks of EDA activity.
Despite the increase of task complexity and difficulty across sessions, the time duration to complete the task did not increase, suggesting a learning effect and adaptation to the serious game.
By using the game as a therapeutic intervention tool, in just three sessions it was possible to improve the general efficiency of participants and expose them to peculiar scenarios in which they could train their planning skills. More importantly, it was possible to nearly extinguish the anxiety felt in bus environments and teach the bus-taking norms necessary for the autonomous use of buses for transportation, both in theoretical and practical contexts. Future studies should conduct randomized controlled trials, with larger intervention groups, to replicate the findings and extend them to other clinical populations with executive function deficits and lack of autonomy.
Autism Spectrum Disorder
electrodermal activity
head-mounted display
Intelligence Quotient
typical development
virtual reality
This study was supported by the AAC SI/2011/HomeTech/QREN Compete, cofinanced by FEDER, the Portuguese Foundation for Science and Technology, the European Projects BRAINTRAIN (FP7-HEALTH-2013-Innovation-1-602186BrainTrain), H2020-STIPED Project number: 731827, FCT (Fundação para a Ciência e Tecnologia) UID/NEU/04539/2013, POCI-01-0145-FEDER-007440, and PhD grant SFRH/BD/77044/2011. The authors would like to thank APPDA-Viseu and all the participants and parents who collaborated in this study.
None declared.