A Web-Based Serious Game on Delirium as an Educational Intervention for Medical Students: Randomized Controlled Trial

Background Adequate delirium recognition and management are important to reduce the incidence and severity of delirium. To improve delirium recognition and management, training of medical staff and students is needed. Objective In this study, we aimed to gain insight into whether the serious game, Delirium Experience, is suited as an educational intervention. Methods We conducted a three-arm randomized controlled trial. We enrolled 156 students in the third year of their Bachelor of Medical Sciences degree at the University Medical Centre Groningen. The Game group of this study played Delirium Experience. The Control D group watched a video with explanations on delirium and a patient’s experience of delirious episodes. The Control A group watched a video on healthy aging. To investigate students’ skills, we used a video of a delirious patient for which students had to give care recommendations and complete the Delirium Observations Screening Scale and Delirium Rating Scale R-98. Furthermore, students completed the Delirium Attitude Scale, the Learning Motivation and Engagement Questionnaire, and self-reported knowledge on delirium. Results In total, 156 students participated in this study (Game group, n=51; Control D group, n=51; Control A group, n=55). The Game group scored higher with a median (interquartile range) of 6 (4-8) for given recommendations and learning motivation and engagement compared with the Control D (1, 1-4) and A (0, 0-3) groups (P<.001). Furthermore, the Game group scored higher (7, 6-8) on self-reported knowledge compared with the Control A group (6, 5-6; P<.001). We did not find differences between the groups regarding delirium screening (P=.07) and rating (P=.45) skills or attitude toward delirious patients (P=.55). Conclusions The serious game, Delirium Experience, is suitable as an educational intervention to teach delirium care to medical students and has added value in addition to a lecture.


INTRODUCTION 2a) In INTRODUCTION: Scientific background and explanation of rationale
1b-iv) RESULTS section in abstract must contain use data Does your paper address subitem 1b-iv? "In total, 156 students participated in this study, 51 in the Game Group, 51in the Control D Group and 55 in the Control A Group." 1b-v) CONCLUSIONS/DISCUSSION in abstract for negative trials Does your paper address subitem 1b-v? 2a-i) Problem and the type of system/solution Does your paper address subitem 2a-i? * "Current educational interventions focus merely on increasing knowledge and skills in recognition of delirium, but do not seem to be effective enough. [6,7] It was suggested that educational interventions on delirium should have a broader scope in order to target the attitude of the medical staff and students towards delirious patients, the understanding of the patient's needs, and the translation of this knowledge into the practice of offering good health care to delirious patients. [7], [8] Future educational interventions on delirium should not only have a broader scope, but also focus on interactive teaching methods and supportive technologies with sufficient feedback loops. [6, 7]" 2b) In INTRODUCTION: Specific objectives or hypotheses METHODS 3a) Description of trial design (such as parallel, factorial) including allocation ratio 2a-ii) Scientific background, rationale: What is known about the (type of) system Does your paper address subitem 2a-ii? * "Serious games might be an opportunity to meet this demand for new educational interventions. Serious games are games developed and intended to provide playful learning experiences, which can be transferable to or applicable in real-life settings.
[9] Compared to regular healthcare educational interventions [10] or assessments [11] in general, serious games are often more effective.
However, there is a lack of effect studies [12] and assessment [13] of good quality on serious games.
The Delirium Experience is a recently designed serious game that makes use of interactive video simulation, [14] which can be used to train and educate medical students on how to take better care of delirious patients. As both serious games [15] and simulation-based learning [16,17] provide learning spaces in which learners can safely practice, Delirium Experience might serve as a new educational intervention by addressing the need for a focus on attitude and applying knowledge of caring for delirious patients." Does your paper address CONSORT subitem 2b? * "In this study, we aim to gain insight in whether Delirium Experience is suited as an educational intervention for medical students regarding skills in advising care for delirious patients, skills in screening and rating of delirium symptoms, and improving the attitude towards delirious patients. Additionally, we aim to gain insight in possible effects of Delirium Experience on learning motivation and engagement, as well as self-reported knowledge on delirium." Does your paper address CONSORT subitem 3a? * "We performed a three-arm randomized controlled trial." "We used IBM SPSS statistics version 23 for stratified block randomisation (block size of 6) to allocate participants into one of the research groups. [18] Learning communities represented the four different strata used. All participants that signed up for the practical were randomly allocated to one of the groups. Subsequently, they received an e-mail in which classroom they were expected." pagina 8 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons 4a) Eligibility criteria for participants Does your paper address CONSORT subitem 3b? * Not applicable 3b-i) Bug fixes, Downtimes, Content Changes Does your paper address subitem 3b-i? Does your paper address CONSORT subitem 4a? * "The study population consisted of undergraduate medical students at the University Medical Centre Groningen (UMCG). To be included in this study, participants had to be in in the third year of the preclinical education in December 2016, sign up for the practical on delirium, and sign informed consent. The UMCG offers an undergraduate programme of six years; three years preclinical and three years of clinical education. Preclinical medical students at the UMCG select one out of four different learning communities with an extra, different focus during their medical education (global health, sustainable care, intramural care, and molecular medicine). Students started with a conventional lecture on delirium. Thereafter, students could voluntarily sign in for the practical on delirium, in which the study conditions took place.
The practicals were given in three separate classrooms of the University of Groningen. All students had the opportunity to join the practical on delirium, also students that did not want to participate in the study. All data was collected and analysed anonymously." 4a-i) Computer / Internet literacy Does your paper address subitem 4a-i?

5)
The interventions for each group with sufficient details to allow replication, including how and when they were actually administered 4b-i) Report if outcomes were (self-)assessed through online questionnaires Does your paper address subitem 4b-i? * Yes: see "outcome measurements" section 4b-ii) Report how institutional affiliations are displayed Does your paper address subitem 4b-ii? 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners Does your paper address subitem 5-i? 5-ii) Describe the history/development process pagina 11 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... Does your paper address subitem 5-ii?

5-vii) Access
Does your paper address subitem 5-vii? * "To be included in this study, participants had to be in in the third year of the preclinical education in December 2016, sign up for the practical on delirium, and sign informed consent. " 5-viii) Mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework Does your paper address subitem 5-viii? * "The Delirium Experience is a serious game focussing on delirium both from the patient and caregiver perspective (watch the trailer in Multimedia Appendix 1: Trailer of Delirium Experience). Throughout the game, your actions as a caregiver influence the severity of the delirium that the patient experiences."

5-ix) Describe use parameters
Does your paper address subitem 5-ix? pagina 13 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... 6a) Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed 5-x) Clarify the level of human involvement Does your paper address subitem 5-x? 5-xi) Report any prompts/reminders used Does your paper address subitem 5-xi? * Not applicable. No prompts used 5-xii) Describe any co-interventions (incl. training/support) Does your paper address subitem 5-xii? * "Students started with a conventional lecture on delirium." pagina 14 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... Does your paper address CONSORT subitem 6a? * "The primary outcome of this research was the skills obtained by students in advising care for delirious patients. To measure skills in advising care, all participants observed an interview of a delirious patient and were asked to give three text-based recommendations for the care of this patient. A pre-defined rubric-form was used to asses all given recommendations as rubric-forms can enhance the reliability of scoring.
[19] The rubric-form was based upon the Dutch delirium guidelines.
[20] Recommendations were assessed independently by two researchers, and a weighted kappa was calculated. Each recommendation could receive 0 (incorrect/not mentioned), 1 (topic mentioned), 2 (nonspecific recommendation) or 3 (specific recommendation) points from the ten different domains of the Dutch delirium guidelines[20] (range 0-9 points).
Subsequently, several secondary outcomes were measured. Firstly, use of screening and rating instruments for delirium was measured. Participants that is a 9-item 5-point Likert scale (range 9-45 points). Examples of statements used in this questionnaire are: "It was challenging to perform well in this practical" and "I liked this way of learning." Finally, participants were asked to self-report their knowledge on delirium (range 0-10 points)." 6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed Does your paper address subitem 6a-i? 6a-ii) Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored Does your paper address subitem 6a-ii?
pagina 15 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... 6b) Any changes to trial outcomes after the trial commenced, with reasons 7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed 7b) When applicable, explanation of any interim analyses and stopping guidelines 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained Does your paper address subitem 6a-iii?
Does your paper address CONSORT subitem 6b? * No changes 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Does your paper address subitem 7a-i? Does your paper address CONSORT subitem 7b? * No pagina 16 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... 8b) Type of randomisation; details of any restriction (such as blocking and block size) 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 8a? * "We used IBM SPSS statistics version 23 for stratified block randomisation (block size of 6) to allocate participants into one of the research groups. [18] Learning communities represented the four different strata used. All participants that signed up for the practical were randomly allocated to one of the groups. Subsequently, they received an e-mail in which classroom they were expected." Does your paper address CONSORT subitem 8b? * "We used IBM SPSS statistics version 23 for stratified block randomisation (block size of 6) to allocate participants into one of the research groups. [18] Learning communities represented the four different strata used. All participants that signed up for the practical were randomly allocated to one of the groups. Subsequently, they received an e-mail in which classroom they were expected." Does your paper address CONSORT subitem 9? * "We used IBM SPSS statistics version 23 for stratified block randomisation (block size of 6) to allocate participants into one of the research groups.
[18] Learning communities represented the four different strata used. All participants that signed up for the practical were randomly allocated to one of the groups. Subsequently, they received an e-mail in which classroom they were expected." Does your paper address CONSORT subitem 10? * "We used IBM SPSS statistics version 23 for stratified block randomisation (block size of 6) to allocate participants into one of the research groups.
[18] Learning communities represented the four different strata used. All participants that signed up for the practical were randomly allocated to one of the groups. Subsequently, they received an e-mail in which classroom they were expected." 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment 11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed 11a-i) Specify who was blinded, and who wasn't Does your paper address subitem 11a-i? * Students didn't know the other conditions 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator"  Does your paper address CONSORT subitem 12a? * "We checked data for normality by judging histograms, skewness, and kurtosis.
We analysed discrete variables by use of a Chi-squared test. Furthermore, continuous variables were analysed with One-way ANOVA in case of normal distribution, and Kruskal-Wallis in case of a non-normal distribution. P-values <.05 were considered statistically significant results of the Chi-squared and Oneway ANOVA or Kruskal-Wallis test. In case of significant results regarding outcome measurements, specific post hoc tests or Mann-Whitney tests were conducted to investigate differences between the (i) Game Group and Control D Group or (ii) Game Group and Control A Group. A Bonferroni correction for two tests was used for the Mann-Whitney, therefore P-values <.025 were considered statistically significant for the results of the Mann-Whitney test. " 12a-i) Imputation techniques to deal with attrition / missing values Does your paper address subitem 12a-i? * Not applicable in this controlled environment Does your paper address CONSORT subitem 12b? * "We checked data for normality by judging histograms, skewness, and kurtosis.
We analysed discrete variables by use of a Chi-squared test. Furthermore, continuous variables were analysed with One-way ANOVA in case of normal distribution, and Kruskal-Wallis in case of a non-normal distribution. P-values <.05 were considered statistically significant results of the Chi-squared and Oneway ANOVA or Kruskal-Wallis test. In case of significant results regarding outcome measurements, specific post hoc tests or Mann-Whitney tests were conducted to investigate differences between the (i) Game Group and Control D Group or (ii) Game Group and Control A Group. A Bonferroni correction for two tests was used for the Mann-Whitney, therefore P-values <.025 were considered statistically significant for the results of the Mann-Whitney test. " X26-i) Comment on ethics committee approval pagina 19 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT...

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center 13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address subitem X26-i? Not applicable: voluntary medical students x26-ii) Outline informed consent procedures Does your paper address subitem X26-ii? "To be included in this study, participants had to be in in the third year of the preclinical education in December 2016, sign up for the practical on delirium, and sign informed consent. ... All students had the opportunity to join the practical on delirium, also students that did not want to participate in the study. All data was collected and analysed anonymously."

X26-iii) Safety and security procedures
Does your paper address subitem X26-iii?
All data was collected an analysed anonymously.

19) All important harms or unintended effects in each group
(for specific guidance see CONSORT for harms)

22) Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence
NPT: In addition, take into account the choice of the comparator, lack of or partial blinding, and unequal expertise of care providers or centers in each group Does your paper address CONSORT subitem 19? * No unintended harms were described or found 19-i) Include privacy breaches, technical problems Does your paper address subitem 19-i? 19-ii) Include qualitative feedback from participants or observations from staff/researchers Does your paper address subitem 19-ii? 22-i) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use) 20) Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses Does your paper address subitem 22-i? * "In this study, we investigated the effects of a serious game, Delirium Experience, as a new educational intervention. We compared playing a serious game to watching a video with delirium explanation in combination with a patient experience video, or a video on healthy ageing. Our results show that the serious game had a positive effect on students' skills in advising care for delirious patients, learning motivation and engagement, and self-reported knowledge on delirium. However, the serious game did not influence skills in screening and in rating the severity of delirium. In addition, it did not have an effect on attitude towards delirious patients." 22-ii) Highlight unanswered new questions, suggest future research Does your paper address subitem 22-ii?
"Further research should be performed on whether it is possible to improve attitudes towards delirious patients with Delirium Experience. If the change in attitude can be established by more emotional and intense patient scenarios, Delirium Experience might improve attitudes when students are allowed to play Delirium Experience several times, including a 'dark play' in Delirium Experience.
In a dark play situation, players try to perform as worse as possible in the game, which results in adverse events and scenarios with extremely frightened patients in Delirium Experience. Subsequently, it would be interesting to investigate the effect of dark play on learning outcomes such as advising care for delirious patients. Furthermore, player characteristics might influence the effectiveness and use of games, and should be taken into account in future studies.
[30] Additionally, future studies should also take into account other healthcare professionals and trainees in order to generalise the results and use of interdisciplinary games such as Delirium Experience. Finally, it is important to look at long-term effects of a serious game and whether it can influence strain of care in experienced healthcare professionals working with delirious patients."

20-i) Typical limitations in ehealth trials
pagina 25 van 30 CONSORT-EHEALTH (V 1.6.1) -Submission/Publication Form 23-1-2018 https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrT... Does your paper address subitem 20-i? * "This study has also a number of limitations. Firstly, skills in advising care for delirious patients were measured with a video of a delirious patient and textbased instead of a real clinical situation, which would involve both the responsibility of caring for a delirious patient and the demonstration of the correct skills. However, simulation-based assessment seems to be a suitable tool for predicting clinical performances.
[29] Secondly, there might have been selection bias in the recruitment of students, as higher motivated students might have signed up for the practical more often than less motivated students.
However, due to the design of this randomised study this could not have influenced the differences between the research groups."  About the CONSORT EHEALTH checklist Does your paper address CONSORT subitem 23? * No, as our participants were volunteering medical students Does your paper address CONSORT subitem 24? * Study protocol is described in the method section completely Does your paper address CONSORT subitem 25? * The serious game 'Delirium Experience' was developed by IJsfontein and is owned by Stichting Effectieve Ouderenzorg (a Dutch foundation for improving elderly care by research and education). E. Hoogendoorn is an employee of IJsfontein. S. de Rooij is an unpaid member of the supervisory board of Stichting Effectieve Ouderenzorg, which waived the licensing fee required for use of Intellectual Property for the purposes of this research. The game is currently commercialized, but the revenues are solely used to improve current elderly care by gamification. X27-i) State the relation of the study team towards the system being evaluated Does your paper address subitem X27-i?

21-i) Generalizability to other populations
The serious game 'Delirium Experience' was developed by IJsfontein and is owned by Stichting Effectieve Ouderenzorg (a Dutch foundation for improving elderly care by research and education). E. Hoogendoorn is an employee of IJsfontein. S. de Rooij is an unpaid member of the supervisory board of Stichting Effectieve Ouderenzorg, which waived the licensing fee required for use of Intellectual Property for the purposes of this research. The game is currently commercialized, but the revenues are solely used to improve current elderly care by gamification.

STOP -Save this form as PDF before you click submit
To generate a record that you filled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it.
When you submit your (revised) paper to JMIR, please upload the PDF as supplementary file.
Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you! Final step: Click submit ! Click submit so we have your answers in our database! As a result of using this checklist, did you make changes in your manuscript? * What were the most important changes you made as a result of using this checklist?
More detailed information on processes How much time did you spend on going through the checklist INCLUDING making changes in your manuscript * 45 minutes As a result of using this checklist, do you think your manuscript has improved? * Would you like to become involved in the CONSORT EHEALTH group? depends on invested time Any other comments or questions on CONSORT EHEALTH VERZENDEN