This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license information must be included.
A serious game–based cellphone augmented reality system (CARS) was developed for rehabilitation of stroke survivors, which is portable, convenient, and suitable for self-training.
This study aims to examine the effectiveness of CARS in improving upper limb motor function and cognitive function of stroke survivors via conducting a long-term randomized controlled trial and analyze the patient’s acceptance of the proposed system.
A double-blind randomized controlled trial was performed with 30 poststroke, subacute phase patients. All patients in both the experimental group (n=15) and the control group (n=15) performed a 1-hour session of therapy each day, 5 days per week for 2 weeks. Patients in the experimental group received 30 minutes of rehabilitation training with CARS and 30 minutes of conventional occupational therapy (OT) each session, while patients in the control group received conventional OT for the full 1 hour each session. The Fugl-Meyer Assessment of Upper Extremity (FMA-UE) subscale, Action Research Arm Test (ARAT), manual muscle test and Brunnstrom stage were used to assess motor function; the Mini-Mental State Examination, Add VS Sub, and Stroop Game were used to assess cognitive function; and the Barthel index was used to assess activities of daily living before and after the 2-week treatment period. In addition, the User Satisfaction Evaluation Questionnaire was used to reflect the patients’ adoption of the system in the experimental group after the final intervention.
All the assessment scores of the experimental group and control group were significantly improved after intervention. After the intervention. The experimental group’s FMA-UE and ARAT scores increased by 11.47 and 5.86, respectively, and were both significantly higher than the increase of the control group. Similarly, the score of the Add VS Sub and Stroop Game in the experimental group increased by 7.53 and 6.83, respectively, after the intervention, which also represented a higher increase than that in the control group. The evaluation of the adoption of this system had 3 sub-dimensions. In terms of accessibility, the patients reported a mean score of 4.27 (SD 0.704) for the enjoyment of their experience with the system, a mean 4.33 (SD 0.816) for success in using the system, and a mean 4.67 (SD 0.617) for the ability to control the system. In terms of comfort, the patients reported a mean 4.40 (SD 0.737) for the clarity of information provided by the system and a mean 4.40 (SD 0.632) for comfort. In terms of acceptability, the patients reported a mean 4.27 (SD 0.884) for usefulness in their rehabilitation and a mean 4.67 (0.617) in agreeing that CARS is a suitable tool for home-based rehabilitation.
The rehabilitation based on combined CARS and conventional OT was more effective in improving both upper limb motor function and cognitive function than was conventional OT. Due to the low cost and ease of use, CARS is also potentially suitable for home-based rehabilitation.
Chinese Clinical Trial Registry ChiCTR1800017568; https://tinyurl.com/xbkkyfyz
Stroke is the leading cause of mortality and permanent disability in adults worldwide [
The primary mechanism of functional recovery after stroke is synaptic reorganization and neurological functional recovery [
To address these issues, augmented reality (AR) technology has been introduced to the field of stroke rehabilitation. The AR system is a useful new technology that blends virtual objects with real scenes in real time [
Based on a review of the relevant literature, we have developed a serious game-based cellphone augmented reality rehabilitation system (CARS). In a previous pilot study performed in a clinical setting, we found that CARS motivated individuals with stroke to perform task-oriented games (eg, Pyramid Reach) during a 30-minute intervention. Most patients who used CARS also reported that the exercise was more motivating than conventional occupational therapy (OT) [
To study these questions, a double-blind randomized controlled trial that compared combined CARS and conventional OT rehabilitation with conventional OT alone was performed. The objective of this paper is twofold: first, to study the long-term effectiveness of the system in improving upper limb function and cognitive state in survivors of subacute stroke; and second, to determine the acceptance of this intervention for home-based rehabilitation of poststroke survivors. We hypothesized that the participants using the combined CARS and conventional OT rehabilitation would receive at least equivalent results to those using conventional OT. We also hypothesized that CARS would be a suitable option for home-based rehabilitation.
This was a multicenter, double-blind, 2-group randomized controlled trial comparing combined CARS and conventional OT rehabilitation with matched control conventional OT in patients with upper limb dysfunction in the subacute phase of stroke. The study protocol was approved by the Institutional Review Board of Huashan Hospital, Fudan University (no. KY2018-248) and registered at the Chinese Clinical Trial Registry (ChiCTR1800017568) (
CARS was developed based on the ARKit toolbox run on an iPhone XR cellphone (Apple Inc) with an iOS 13 operating system. Stroke survivors may not be able to pick up the phone due to the weakness of their affected side. Therefore, a fingerless glove cellphone case was designed, which could easily fix the cellphone to the patient’s affected hand. Patients could move their affected upper limb to use the cellphone and interact with 3D virtual targets generated on the cellphone screen (
Hardware for the cellphone augmented reality rehabilitation system. (Left) Phone: iPhone XR, iOS 13. (Center) Cellphone cases and gloves. (Right) Method of wearing.
Three serious games were developed based on CARS for improving both motor function and cognition function of stroke survivors (
Three augmented reality–based serious games for rehabilitation of upper limb motor function and cognitive function. (Left) Pyramid Reach. (Center) Add VS Sub. (Right) Stroop Game.
The proposed system can record the scores of each round and the trajectory of user’s upper limb distal during the training, which enables clinicians to track the change in the range of motion of the patient’s affected side and the progress of the patient’s recovery. This function can help clinicians give further diagnosis and rehabilitation guidance to outpatients.
Thirty participants were recruited in the study between August 2020 and March 2021. The inclusion criteria were the following: age ≥20 and <70 years; first incidence of a stroke with unilateral hemiparesis; chronicity ≥ 7 and<180 days, Mini-Mental State Examination (MMSE) score ≥20, Brunnstrom stage for upper limb ≥3, ability to give informed consent and operate a mobile phone, and the visual and mental ability to actively participate in the protocol. Exclusion criteria were the following: history of epilepsy orthopedic alteration or pain syndrome of the upper limb, severe aphasia or other psychiatric illnesses that limit the ability to participate or give consent, visual disturbance such as visuospatial neglect, and poor sitting balance.
PASS software (NCSS Statistical Software) was used to calculate the required sample size. The Fugl-Meyer Assessment for Upper Extremity (FMA-UE) subscale was considered significant when the change value was more than 5 points [
All participants were invited for an initial assessment to confirm that they met the inclusion criteria. An independent researcher not involved in the study created a blocked randomization sequence using a computerized program (Microsoft Excel). Block randomization ensured equal numbers of participants for the experimental and the control group. Allocation assignments were placed in sequentially numbered, opaque, and sealed envelopes by an offsite officer not involved in the study. Patients were not blind regarding the intervention received. Once the participant passed the screening process and completed the baseline assessment, an independent researcher would open an envelope and reveal the group allocation. In addition, outcome evaluators were blinded to the group assignment.
After giving informed consent, eligible participants were allocated to 1 of 2 groups. A previous study by Saposnik et al [
For the experimental group, we provided 2 methods for patients to use the system (
Two training methods. (Left) The participant training individually with the affected side. (Right) The participant using the unaffected side to assist the affected side for training.
Four effective clinical-based assessments were selected and performed to evaluate the upper limb motor function of patients in both groups before and after the intervention. Primary outcomes were the FMA-UE and the Action Research Arm Test (ARAT). The FMA-UE can objectively measure arm impairment and the degree of muscle synergies present. Performance is rated on a 3-point ordinal scale from 0 to 2 with a maximum score of 66. A higher score indicates minimal or no impairment [
In addition, the manual muscle test (MMT) and Brunnstrom stage (BS) were selected as second motor outcomes. The MMT is a procedure for the evaluation of muscle strength, based on the effective performance of a movement in relation to the forces of gravity or manual resistance through the available range of motion. The grading of MMT ranges from 0 (no visible or palpable contraction) to 5 (full range of motion against gravity, maximal resistance) [
To assess the change of cognitive function for patients in both the experimental and control groups, 1 clinical-based assessment and 2 cognition evaluations designed based on the proposed serious games were performed additionally before and after the intervention.
MMSE is a widely used test of cognitive function for stroke survivors and includes tests of orientation, attention, memory, language, and visual-spatial skills [
Game 2 (AVS) and game 3 (SG) can train the patient's ability of calculation and comprehension. The scores of the 2 games trained by patients each day were recorded by CARS, and these scores were derived and analyzed. We further designed a paper version of AVS and SG to test the change of cognitive ability in patients. The patients were asked to answer the questions from the paper version as quickly as possible within 1 minute. The number of correct answers was the score of the patients.
The Barthel index (BI) is used to measure performance in ADLs. Ten variables describing ADL and mobility are scored, with a higher number being a reflection of greater ability to function independently following hospital discharge. Scores of 0 to 20 indicate “total” dependency, 21 to 60 indicate “severe” dependency, 61 to 90 indicate “moderate” dependency, and 91 to 99 indicate “slight” dependency [
Referring to the User Satisfaction Evaluation Questionnaire [
Statistical analysis was performed in SPSS version 26 (IBM Corp). Data were confirmed to have a normal distribution according to the Shapiro-Wilk normality test since the sample size was small. Two-tailed
The flowchart for participants is presented in
Flowchart for participant selection and assignment. AR: augmented reality.
The patients’ baseline demographic and clinical characteristics are shown in
Baseline demographic and clinical characteristics of the patients.
Variable | Experimental group (N=15) | Control group (N=15) | |||||
Age (years), median (IQR) | 62 (24) | 57 (32) | 0.062a | .96 | |||
|
0.144b | .70 | |||||
|
Male | 10 (66) | 9 (60) |
|
|
||
|
Female | 5 (33) | 6 (40) |
|
|
||
Time from onset (days), mean (SD) | 78.2 (40) | 69.2 (51) | 0.530c | 0.60 | |||
|
0.133b | .71 |
|||||
|
Left | 8 (53) | 7 (46) |
|
|
||
|
Right | 7 (46) | 8 (53) |
|
|
||
MMSEd, median (IQR) | 26 (4) | 25 (3) | 0.168a | .87 | |||
FMA-UEe, median (IQR) | 30 (18) | 25 (23) | 0.583a | .56 | |||
ARATf, median (IQR) | 14 (17) | 12 (22) | 0.417a | .68 | |||
BS-Ug, median (IQR) | 3 (1) | 3 (1) | 0.024a | .99 | |||
BS-Hg, median (IQR) | 3 (2) | 4 (3) | 0.085a | .93 | |||
MMTh shoulder, median (IQR) | 3 (0) | 3 (0) | 0.338a | .81 | |||
MMT elbow, median (IQR) | 3 (0) | 3 (1) | 0.898a | .53 | |||
MMT (wrist), median (IQR) | 1 (3) | 3 (1) | 1.346a | .20 | |||
BIi, mean (SD) | 64.67 (12) | 63 (13) | 0.354c | .72 | |||
AVSj, mean (SD) | 17.8 (4) | 19.47 (6) | 0.841c | .41 | |||
SGk, mean (SD) | 11.67 (5) | 13 (6) | 0.625c | .53 |
aWilcoxon rank sum test.
bChi-square test.
cTwo-tailed
dMMSE: Mini-Mental State Examination.
eFMA-UE: Fugl-Meyer Assessment of the Upper Extremity.
fARAT: Action Research Arm Test.
gBS: Brunnstrom stage (U: upper extremity; H: hand).
hMMT: manual muscle test.
iBI: Barthel index.
jAVS: Add VS Sub.
kSG: Stroop Game.
After the intervention, both groups showed significant improvement in the FMA-UE, ARAT, BS, and MMT scores over time (
Comparison of outcomes in the experimental group and control group.
Outcomes | Experimental group (N=15) | Control group (N=15) | |||||
|
Pretest | Posttest | Pretest | Posttest | |||
FMA-UEa, median (IQR) | 30 (18) | 41 (21) | .001 | 25 (23) | 33 (26) | .001 | |
ARATb, median (IQR) | 14 (17) | 21 (23) | .001 | 12 (22) | 16 (24) | .001 | |
MMSEc, median (IQR) | 26 (4) | 27 (2) | .005 | 25 (5) | 25 (4) | .004 | |
BS-Ud, median (IQR) | 3 (1) | 4 (1) | ﹤.001 | 3 (1) | 4 (2) | .008 | |
BS-Hd, median (IQR) | 3 (2) | 4 (3) | .007 | 4 (3) | 4 (3) | .034 | |
MMTe shoulder, median (IQR) | 3 (0) | 4 (1) | .001 | 3 (0) | 4 (0) | .002 | |
MMT elbow, median (IQR) | 3 (0) | 4 (0) | ﹤.001 | 3 (1) | 4 (0) | .005 | |
MMT wrist, median (IQR) | 1 (3) | 3 (2) | .001 | 3 (1) | 3 (2) | .014 | |
BIf, median (IQR) | 65 (25) | 75 (15) | .002 | 60 (25) | 65 (20) | .01 | |
AVSg, median (IQR) | 17 (4) | 25 (5) | .001 | 18 (10) | 23 (10) | .001 | |
SGh, median (IQR) | 10 (6) | 18 (10) | .001 | 12 (7) | 15 (8) | .001 |
aFMA-UE: Fugl-Meyer Assessment of the Upper Extremity.
bARAT: Action Research Arm Test.
cMMSE: Mini-Mental State Examination.
dBS: Brunnstrom stage (U: upper extremity; H: hand).
eMMT: manual muscle test.
fBI: Barthel index.
gAVS: Add VS Sub.
hSG: Stroop Game.
Longitudinal changes in motor outcomes with the experimental group showing significantly greater improvements than the control group in FMA-UE and ARAT. ARAT: Action Research Arm Test; BI: Barthel index; BS: Brunnstrom stage (U: upper extremity; H: hand); FMA-UE: Fugl-Meyer Assessment of the Upper Extremity; MMT: manual muscle test (S: shoulder; E: elbow; W: wrist).
Significant group interaction effects were observed for AVS and SG scores (
Longitudinal changes in Add VS Sub (AVS), Stroop Game (SG), and Mini-Mental State Examination (MMSE) with the experimental group showing significantly greater improvements than the control group in AVS and SG. AVS: Add VS Sub; MMSE: Mini-Mental State Examination; SG: Stroop Game.
We derived all patients’ training scores recorded on the mobile phone. After processing and analyzing the scores, we created a line chart of each patient (
Line charts of 15 patients’ daily scores for 3 serious games. The average score of 5 trails represents a session, and each color represents a patient.
A user satisfaction questionnaire was administered for patients at the end of their final intervention in the experimental group. The 5-point rating was used to assess responses to the questions (1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree). In terms of accessibility, the patients reported a mean score of 4.27 (SD 0.704) for the enjoyment of their experience with the system, a mean 4.33 (SD 0.816) for success in using the system, and a mean 4.67 (SD 0.617) for the ability to control the system. In terms of comfort, the patients reported a mean 4.40 (SD 0.737) for the clarity of information provided by the system and a mean 4.40 (SD 0.632) for comfort. In terms of acceptability, the patients reported a mean 4.27 (SD 0.884) for usefulness in their rehabilitation (
Here, we report patients’ suggestions for the system. Some patients felt that the cellphone screen was too small to see clearly. Some patients thought these 3 AR serious games should be set to a different level, so that they can be applied to patients in different stages.
Results of the acceptability questionnairea.
Questions | Score, mean (SD) |
Q1. Did you enjoy your experience with the system? | 4.27 (0.704) |
Q2. Were you successful using the system? | 4.33 (0.816) |
Q3. Were you able to control the system? | 4.67 (0.617) |
Q4. Is the information provided by the system clear? | 4.40 (0.737) |
Q5. Did you feel comfortable during your experience with the system? | 4.40 (0.632) |
Q6. Do you think that this system will be helpful for your rehabilitation? | 4.27 (0.884) |
Q7. Do you think this system can be used for home-based rehabilitation? | 4.67 (0.617) |
aThe questionnaire includes 7 questions, each with a score of 1 to 5 (1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree).
No significant adverse events related to CARS occurred during the clinical research. Reported adverse events were pain and fatigue in the shoulder (n=3) and elbow (n=2), which were foreseen and already mentioned to the patients before therapy began. The use of this cellphone game-based AR system was safe and acceptable for all participants in the study.
We have developed a serious game–based cellphone augmented reality rehabilitation system for upper limb recovery and improving cognitive function after stroke. Compared with conventional OT, combined CARS and conventional OT rehabilitation proved to be more effective in improving both upper limb function and cognitive function. As a new rehabilitation technique, CARS can replace part of traditional OT and be used as a supplementary treatment method in the hospital to decrease the consumption of medical resources and reduce the burden of patients. In addition, this system can be a suitable option for self-oriented or home-based rehabilitation.
Our findings showed that CARS effectively enhanced upper extremity recovery in patients with stroke. In the comparison between CARS and conventional OT, intervention using CARS showed greater improvement in the FMA-UE and ARAT scores. Comparable improvements between groups were shown in the BS, MMT, and BI. Our results showed that compared with the control group, the experiment group using CARS experienced effectively improved calculation and color-matching ability. In addition, patients in the experimental group completed the 2-week intervention using CARS without severe adverse effects, and they were satisfied with this system for self-oriented or home-based rehabilitation. We speculated that the therapeutic effectiveness of combined CARS and conventional OT rehabilitation would be equal to or greater than that of conventional OT alone, and the results confirmed our hypothesis. The results that CARS can effectively improve the upper limb function and cognitive state of patients may be related to the following factors. Studies conclude that serious games seem to be a safe rehabilitation modality for patients recovering from stroke [
Although some studies have attempted to use AR-based rehabilitation systems for patients with stroke, few studies have included a control group. Assis et al [
Compared with other systems, the proposed CARS we developed has the following advantages. First, our system requires the least amount of equipment and is also the cheapest and most convenient. CARS was developed for users with a mobile phone because almost everyone has one. The system is portable and very easy to use regardless of a person’s location. Other AR systems often require independent 3D tracking systems, monitors, and interactive systems, and we integrate all 3 into a single cellphone [
There are a few limitations in this study. First, although compensatory movements were restricted during the intervention, they were not controlled during the assessment, which could have influenced the performance in the scales and tests. Second, although the physical therapist who assessed the participants’ condition did not know the protocol, the therapists who administered and controlled the intervention were not blind. Third, we use gloves of uniform size, but each patient had a different hand size, and thus it might have been difficult for some patients to wear the gloves. Fourth, none of the patients who participated in the study had any experience with AR rehabilitation. Therefore, they could not compare our games with other similar games. Finally, the sample of the study (N=30) can be considered small, which may limit the degree to which the results can be extrapolated.
In future research, we will further improve our devices, which will involve developing an Android version to reduce installation costs, preparing different specifications of gloves for patients with different hand sizes, and increasing the variety and difficulty settings of the game. In addition, future work will include more engaging serious games to increase the variety of therapy solutions and adaptability to patient abilities so that a therapist or patient can match the degree of challenge necessary to keep the rehabilitation advancing. In the trial design, large-sample, controlled, follow-up clinical studies will be conducted in the future to verify the long-term efficacy of the system for home-based rehabilitation.
At the behavioral level, there was additional benefit received from CARS. Combined CARS and conventional OT rehabilitation was more effective in improving both upper limb function and cognitive function compared with conventional OT alone. The results of our study indicated that the proposed CARS can replace the one-on-one conventional OT delivered by an occupational therapist and that the system can be used as an assistant therapeutic tool in the hospital. In addition, CARS is convenient, low-cost, and user-friendly, which indicates that this system is also suitable for home-based rehabilitation. Future studies with a longer intervention time and a follow-up of patients for home-based rehabilitation are needed to explore the effectiveness of the system.
Consolidated Standards of Reporting Trials (CONSORT) eHealth checklist (v 1.6.1).
activity of daily living
Action Research Arm Test
Add VS Sub
Barthel index
Brunnstrom stage
cellphone augmented reality system
Fugl-Meyer Assessment Upper Extremity
Mini-Mental State Examination
manual muscle test
Stroop Game
The authors wish to thank all the therapists and patients for their participation in this study. This work was supported by the Key National Research and Development Program (#2018YFC2002300), the National Nature Innovation Research Group Project (#82021002), the National Nature Integration Project (#91948302), and the National Natural Science Foundation of China (#51950410602).
CL and XS conceived the study design. JH, CW, YZ, ZY, and SX recruited the patients and performed the assessments. CL and SC completed the data collection. XS developed the CARS system. CL and XS performed the data analysis and drafted the manuscript. JJ and PS were involved in the interpretation of the results and provided feedback on the manuscript. All authors read and approved the final manuscript.
None declared.