Published on 27.03.19 in Vol 7, No 1 (2019): Jan-Mar
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11861, first published Aug 23, 2018.
Creating a Theoretically Grounded Gaming App to Increase Adherence to Pre-Exposure Prophylaxis: Lessons From the Development of the Viral Combat Mobile Phone Game
Background: In the United States, young minority men who have sex with men (MSM) are most likely to become infected with HIV. The use of antiretroviral medications to reduce the risk of acquiring HIV infection (pre-exposure prophylaxis, PrEP) is an efficacious and promising prevention strategy. There have been significant advances regarding PrEP, including the definitive demonstration that PrEP reduces HIV acquisition and the development of clinical prescribing guidelines. Despite these promising events, the practical implementation of PrEP can be challenging. Data show that PrEP’s safety and effectiveness could be greatly compromised by suboptimal adherence to treatment, and there is concern about the potential for an increase in HIV risk behavior among PrEP users. Due to these challenges, the prescribing of PrEP should be accompanied by behavioral interventions to promote adherence.
Objective: This study aimed to develop an immersive, action-oriented iPhone gaming intervention to improve motivation for adherence to PrEP.
Methods: Game development was guided by social learning theory, taking into consideration the perspectives of young adult MSM who are taking PrEP. A total of 20 young men who have sex with men (YMSM; aged 18-35 years) were recruited from a sexually transmitted infection (STI), HIV testing, and PrEP care clinic in Jackson, Mississippi, between October 2016 and June 2017. They participated in qualitative interviews guided by the information-motivation-behavioral skills (IMB) model of behavior change. The mean age of participants was 26 years, and all the participants identified as male. Acceptability of the game was assessed with the Client Service Questionnaire and session evaluation form.
Results: A number of themes emerged that informed game development. YMSM taking PrEP desired informational game content that included new and comprehensive details about the effectiveness of PrEP, details about PrEP as it relates to doctors’ visits, and general information about STIs other than HIV. Motivational themes that emerged were the desire for enhancement of future orientation; reinforcement of positive influences from partners, parents, and friends; collaboration with health care providers; decreasing stigma; and a focus on personal relevance of PrEP-related medical care. Behavioral skills themes centered around self-efficacy and strategies for adherence to PrEP and self-care.
Conclusions: We utilized youth feedback, IMB, and agile software development to create a multilevel, immersive, action-oriented iPhone gaming intervention to improve motivation for adherence to PrEP. There is a dearth of gaming interventions for persons on PrEP. This study is a significant step in working toward the development and testing of an iPhone gaming intervention to decrease HIV risk and promote adherence to PrEP for YMSM.
JMIR Serious Games 2019;7(1):e11861
The primary prevention of HIV infection remains a crucial priority. In 2016, there were 39,782 new HIV diagnoses in the United States, and 67% of these new diagnoses were of gay and bisexual men . HIV also disproportionately affects communities of color. Despite only comprising 12.6% of the US population, nearly 45% of all new HIV infections occur among African Americans. The use of antiretroviral medications to reduce the risk of acquiring HIV infection (pre-exposure prophylaxis, PrEP) is an efficacious and promising new prevention strategy [ , ]. There have been significant advances regarding PrEP including the definitive demonstration that PrEP reduces HIV acquisition. Despite promising data, the practical implementation of PrEP is challenging [ - ]. Unfortunately, individuals who are often at the highest risk of HIV infection and eligible for PrEP come from populations that historically have been underserved by health care [ , ]. Therefore, engaging diverse persons on PrEP in care is challenging, and reinforcement and support for patients and doctors is required [ , - ]. Behavioral Interventions promoting adherence to comprehensive PrEP treatment and healthy sexual behaviors will need to be tailored to diverse, underserved, and at-risk populations and will need to reinforce the clinician-patient relationship.
In addition, the behavioral interventions accompanying PrEP need to be scalable, cost-effective, and easily integrated into clinical settings [, ]. Without these necessary components, integration of behavioral interventions into clinical settings cannot be realistically sustained. Utilizing gaming technology to deliver behavioral interventions for young men who have sex with men (YMSM) can improve intrinsic motivation and information and build behavioral skills for adherence. The use of an intervention that utilizes gaming technology is particularly compelling for use with younger adults, as this age group is at the highest risk of acquiring HIV and this age group has most actively inquired about PrEP in clinical settings. In addition, 72% of young male adults in the United States say they play video games often or sometimes [ ]. Gaming technology is also popular among minority men who have sex with men (MSM), the subgroup most at risk for acquiring HIV. In the past, gaming was mistakenly identified as a primarily adolescent and heterosexually dominated activity, but current data support that gaming is actually quite diverse [ ]. MSM gamers, referred to as gaymers in pop culture and mainstream articles, are highly represented on the web and increasingly shape the market [ - ]. In addition, within the United States, African Americans aged 18 to 35 years represent the most active and fastest growing user group of mobile phones [ , ], and more than half of the reported video gaming occurs on portable devices [ , , ]. The widespread appeal and use of mobile phones and video game playing creates a unique opportunity to deliver health education to diverse YMSM on PrEP during leisure time, outside of the clinic, and in a manner that is cost-effective and easily scalable [ - ].
Interactive game play has been shown to enhance players’ motivation to improve health behaviors and self-care in a variety of clinical settings and populations. Games can attract and maintain attention, a key component for effective behavior change. Compelling interactive phone-based games can expose players to essential health-related content thousands of times and also give players unlimited opportunities to rehearse new skills and receive personalized feedback on health choices made within the game [- ]. Games have been shown to be efficacious in promoting fitness, improving weight management, and improving safer sex skills [ , , , ]. For example, an HIV and AIDS prevention computer game called Life Challenge was developed by the New York State Department of Health to enhance safer sex negotiation by adolescents and young adults. The game showed significant improvement in self-efficacy for partner negotiation and condom skills for those who started with the least self-efficacy [ ]. Two pregnancy prevention games, The Baby Game and Romance, designed for sexually active young adults, showed trends in improving knowledge and attitudes about parenting and unprotected sexual behaviors [ ]. Video games have also been applied to improve self-management skills and healthy behaviors in those living with asthma, diabetes, and cancer [ - ]. For example, a video game named Re-Mission (tested with adolescents and young adults aged 13-29 years) was designed as an action-adventure in which the main character shoots cancer-causing agents in the bloodstream. In a randomized control study, 375 male and female participants who played Re-Mission had significantly improved adherence to trimethoprim-sulfamethoxazole (TMP/SMX; P=.01) and 6-mercaptopurine (P=.002) after an average of only 10.7 hours of play. Adherence to TMP/SMX by those playing Re-Mission was 19% greater than those in the control group. Self-efficacy (P=.01) and knowledge (P=.03) also increased [ , ]. Thus, appealing interactive games can target information, motivation, and skills for medical care and have led to a broad spectrum of desirable health outcomes including increases in knowledge, attitude changes, and increased medication adherence [ , , - ].
Gaps in Literature
Despite the promise and popularity of digital games, there is a paucity of interventions or publications related to gaming for persons on PrEP . Relatedly, there are games in development for YMSM either living with HIV or at risk for HIV [ - ]. For example, our research group has developed a gaming intervention to improve antiretroviral treatment (ART) adherence for youth and young adults living with HIV that informs the development of this game [ ]. LeGrand et al have published a description of the development phase of a game entitled Epic Allies. Epic Allies is designed to improve ART uptake, engagement in care, and adherence among HIV positive YMSM and transgender women who have sex with men. While playing Epic Allies, users can earn medals and tokens for taking medications and reading health-related studies. [ ]. Another intervention called PlayForward aims to reduce the risk for HIV among at-risk, ethnic, and racial minority adolescents. This tablet-based game provides an interactive world using an avatar where players face challenges such as peer pressure to drink alcohol or engage in other risky sexual behaviors [ ]. A mobile phone–optimized intervention entitled healthMpowerment is designed to reduce sexual risk behaviors among YMSM. In this intervention, YMSM can acquire reputation points through reading information about HIV, playing sexually transmitted infection (STI)–related games, and through positive interactions with other users. Despite the promise and popularity of digital games, there are no published or presented abstracts related to gaming for persons on PrEP that we could find.
We developed a gaming intervention to improve adherence to PrEP for YMSM aged 18 to 35 years. This intervention integrates a smart pill bottle cap (that measures adherence) with an iPhone game. The gaming intervention was informed by the information-motivation-behavioral skills (IMB) theory of learning. The IMB model, consistent with social learning theory, is broadly applicable and can be used to guide game development and create theoretically consistent gaming content [- ]. The iPhone gaming app was designed for participants to experience absorbing action-oriented adventures that increase information about their health (eg, knowledge about PrEP and HIV prevention and adherence), improve motivation (eg, action figures experience health benefits of adherence to PrEP), and build skills (action figures interact with clinicians at appointments, take medications as prescribed, and practice safe sex; see - ). Adherence (measured by the smart pill bottle cap) and game-related text messages are integrated into the gaming intervention. Multiple reviews have demonstrated that behavioral interventions shown to be most efficacious are those tailored for the target population and preceded by formative research to inform intervention development [ , ]. The aim of this study was to describe the development of this iPhone gaming app entitled ViralCombat.
Gaming App Development
Development of ViralCombat was accomplished using iterative and collaborative procedures to integrate the clinical experiences of YMSM taking PrEP, academic researchers, and technology partners. Game development was guided by qualitative interviews with a group of MSM aged between 18 and 35 years taking PrEP. Guided by the principles of agile software development , the qualitative interviews and the game and app programming were synergistic. Agile software development aims for continuous design testing and adaptation based on continuous feedback [ ].
Sample and Recruitment
Males aged between 18 and 35 years were eligible for enrollment in the study according to the following criteria: (1) English-speaking, (2) currently taking PrEP, (3) reported having sex with other men, and (4) able to give consent and not impaired by cognitive or medical limitations as per clinical assessment. Those who did not meet the abovementioned inclusion criteria were excluded. We recruited 20 YMSM for qualitative interviews to guide game development after institutional review board’s (IRB) approval. Subjects were from a convenience sample recruited from a PrEP clinic in Jackson, Mississippi, between October 2016 and June 2017. Subjects were approached by research staff with an IRB-approved flyer, and written consent was obtained upon meeting with study staff for the qualitative interview. Overall, 20 subjects were approached over the course of the interviews, and all of them consented and completed the interview. Subjects were recruited until data saturation was achieved, and a relative balance in the sample was achieved based on age greater than 26 years versus younger. The mean age of participants was 26 years (range: 18-35 years; 11 out of the 20 were younger than 26 years). A majority of the participants (17/20, 85%) identified as African American, 100% (20/20) identified as MSM, 70% (13/20) completed 12th grade, 75% (15/20) had been taking PrEP longer than 6 months, 75% (15/20) reported missing a dose of PrEP in the previous week, 45% (9/20) reported missing 3 doses or more in the previous week, and 75% (5/20) had been taking PrEP for 6 months or longer.
The study was approved by the hospital’s IRB, and participant consent and interviews were conducted in a private room located in a PrEP clinic in Jackson, Mississippi. Interviews were conducted by 1 of the 2 MDs (psychiatrists) with support from a trained research assistant, and subjects were reimbursed US $50 per survey (for a total of US $150) for their time. The psychiatrists who conducted the interviews did not provide medical or clinical services in the HIV clinic. Interviews lasted between 45 and 60 min and were digitally recorded. As we adapted our gaming intervention from games that were already developed (Dr. Nano X and BattleViro), the system and the framework (eg, code, database, and design) were already in place at the beginning of the project. Adaptations to the game occurred as themes emerged from the interviews . Biweekly meetings were held with the programmers to discuss all adaptations, including changes to content, game graphics framework, and game messaging.
Adaptation of the Information-Motivation-Behavioral Skills Adherence Gaming Intervention
The IMB gaming app for persons on PrEP, entitled ViralCombat, was adapted from the popular Mission Critical Studios game entitled Dr. Nano X as well as our previously developed game for persons on ART entitled BattleViro [, ]. Dr. Nano X is a 5-star-rated mobile game (the highest rating possible) in the iTunes app store and is available on both Android and iPhone. The ART adherence game, named BattleViro, has been previously described in the Journal of Medical Internet Research [ ]. We worked directly with the development team at Mission Critical Studios to develop ViralCombat using Dr. Nano X and BattleViro as a basic framework. Adapting our game to promote PrEP adherence from already existing games greatly decreased the cost of the project. Characters, actions, and IMB messaging about PrEP were built specifically for ViralCombat; however, we were able to reuse backgrounds, mechanisms of game play and controls, and many sound effects from Dr. Nano X and BattleViro [ , ]. ViralCombat game levels are distinct from those in BattleViro, and they include organ systems such as the penis, anus, and mouth.
We worked iteratively with youth living with HIV and Mission Critical Studios to create the app. The game starts with a 45-second narrative movie that explains the storyline and game objectives. The player is told by the narrator, in a deep and dramatic voice, “you have been chosen, due to your smart decisions and healthy choices, to be cloned and shrunken in order to enter your own body to destroy attacking viruses and infections.” The narrator emphasizes that “in order to acquire ammunition and strength, you must take you medication in real life and also pick up pills during play.” Players are given a tutorial on game actions and can also design their individual and diverse characters. As players successfully battle HIV, engage with providers, and take medication, they move to new, distinctive levels (arterial system, penis, anus, and mouth). Messages from the doctors, nurses, and friends encourage and provide clues during difficult twists and turns in the battle. Answering quiz questions from clinician avatars allows each player to earn strength and points; wrong answers are corrected and explained. Players find medication, strength, and points by acting on positive suggestions. During each mission, the player’s score (pill count and health) is shown. All character control and gaming is done by touch screen technology on the phone; no additional accessories are needed for play. Throughout the game, the terms “HIV” and “PrEP” are seen on the screen, but they are never part of an audio feature, so stigmatizing information cannot be overheard by another person. The game-related text messages have gaming graphics, similar to other apps and games, and the app uses push notifications to engage gamers in play. If players were less than 90% adherent during the week, phrases such as “Missing you in Combat” and “Get back in the game” are texted to their phones. Congratulatory short message service (SMS) text messages such as “Great job in battle” and “You are fighting off virus well” are sent for greater than 90% adherence (see). Throughout gaming, the mission stays the same: kill the virus and build strength through taking medicine, learning information, improving motivation, and engaging with healthy characters to improve adherence to PrEP and build HIV preventative skills.
Interviews with YMSM taking PrEP-guided game development were conducted. A total of 11 participants were shown the storyboard of the game and completed interviews. When the iPhone version became available, 9 participants were given the game on an iPhone, and their feedback was elicited through an interview. Therefore, qualitative feedback was received from all 20 participants. The interview guide consisted of focused but open-ended questions aimed at maximizing participant responses (see). Participants were asked about information relevant to medication adherence, motivating factors to adherence, and behavioral skills needed for adherence. The IMB model guided interview content, and participants were also queried about their general gaming experience and their reactions to the storyboard and gaming content.
Information Needed for Adherence
Participants were asked about knowledge and information that influences their adherence to PrEP and engagement in medical appointments. Questions included:
What type of information from doctors or friends makes it easier to take PrEP?
What information makes it easier to come to PrEP related appointments?
This part of the interview aimed to understand the specific knowledge about HIV and PrEP that promotes adherence behaviors. For example, some probes focused on how side effects and other health-related information can influence adherence to medication and care (for more examples, see).
Motivation for Adherence
Participants were queried about motivational issues related to adherence to PrEP with probes such as:
I would like to hear about what you think the serious issues are surrounding PrEP and taking medications to prevent HIV,
What are the things that make it hard to take PrEP?
This part of the interview was dedicated to understanding both personal and social motivations for adherence to PrEP. Queries were focused on the positive and negative attitudes toward taking PrEP, perceived negative effects of nonadherence to PrEP, and the individual’s perceptions of social support from significant others, family, friends, and medical care providers (for more examples, see).
Behavioral Skills for Adherence
Participants were also asked about their ability to perform necessary adherence and HIV preventative tasks and their perceived self-efficacy for these tasks. Questions included:
What are the ways that you stay safe from HIV?
What are the ways that you remember to take PrEP and remember your PrEP related appointments?
What events in your life make it harder to remember to take PrEP? Or remember your appointments?
We also asked participants about strategies for self-reinforcement for adherence over time and across different situations. We asked questions such as:
Do you consciously think about your PrEP medication schedule on a long-term basis?
What strategies have you used or developed to remember to take PrEP or to go to your appointments based on your activities?
This part of the interview aimed to assess perceived abilities and strategies to store, obtain, and self-cue the use of medications despite challenges and across situations (for more examples, see).
General Gaming Attitudes
Participants were also asked about their general attitudes and experiences with games. Participants were asked questions such as:
What games do you, or people you know, play on the cellphone?
What types of graphics, avatars, and rewards do you like? And what do you not like?
How are games useful? Do you develop any skills when you play games?
These queries elicited descriptions of popular game activities and attitudes about gaming. The responses were used to make the format and game mechanics of ViralCombat engaging and immersive (for more examples, see).
ViralCombat Storyboard and iPhone Game
Participants were asked for feedback about the storyboard or the iPhone game (once the mobile game was ready) with the probes such as:
What was the main point of this activity?
What could you learn from this activity?
Would you recommend this type of game to your friends?
What is your reaction to having some HIV and PrEP related information and skills in an iPhone game?
After the first version of the game was developed on the iPhone, participants were asked additional and modified probes such as:
Is the game easy to navigate and easy to understand?
Did any part of the game not work?
Are there other topics that the game should cover that it does not?
Answers to these questions guided the iterative development of the game levels, actions, characters, and graphics (for more examples, see).
Medication Adherence Monitoring Tracking and Game-Related Text Messages
Participants were also asked about the electronic pill monitoring organizers and adherence-related text messages. We queried participants about the smart pill bottle cap that can electronically monitor, measure, and securely relay adherence pill bottle openings to our research team. Participants were asked about the use of reminder messages with a game-related graphic. Feedback was elicited about game-related messages if players missed a dose using texted phrases such as “Missing you in Combat” and “Get in the game.” Feedback was also received about SMS text messages if doses are taken on time such as “Great job in battle” and “You are fighting off virus well!” (see).
Quantitative Feasibility and Acceptability Data
The game was developed iteratively. A total of 11 participants were shown the storyboard of the game for approximately 30 min; qualitative feedback was elicited, and game development on the iPhone occurred. After the development of the iPhone version of ViralCombat, 9 of the 20 participants played the game for 45 to 50 min with the interviewer in the room (see- ). These 9 participants were shown each of the game levels on the phone by the interviewer and played each level. After playing the game, these 9 participants completed qualitative interviews, and then, written or quantitative feedback was obtained. Quantitative feedback was collected from these 9 participants using versions of the Client Service Questionnaire (CSQ) and the session evaluation form (SEF) [ , ]. Both these instruments were developed to measure client satisfaction and perspectives on intervention aspects. The CSQ consists of 8 items that assess general satisfaction with the game. An example query from the CSQ is “In an overall, general sense, how satisfied are you with the game?” (for which the response options are 4=“Very satisfied”; 3=“Mostly satisfied”; 2=“Indifferent or mildly dissatisfied”; and 1=“Quite dissatisfied”) [ ]. We did not calculate an alpha coefficient for this sample because of the small number of participants completing the CSQ (n=9). In other studies, the internal consistency of the CSQ-8 is high, with alpha coefficients ranging from .84 to .93 for the CSQ [ ]. The SEF contains 13 items that assess the feasibility and perceived utility of activities in the game. For example, the SEF states: “I will be able to apply what I learned from this game in my life” (for which the response options are 1=“Strongly agree”; 2=“Agree”; 3=“Disagree”; and 4=“Strongly disagree”) [ ]. As the SEF asks about specific the utility of separate elements of the intervention, and not about singular constructs, alpha coefficients for the SEF are not commonly calculated.
Trained research assistants transcribed verbatim the digital audio recordings of each interview. Then, the MD- or PhD-level research team member reviewed the transcripts with the digital recording for accuracy. Qualitative data analysis followed the tenets of thematic analysis, which consisted of sequential steps [, ], and interviews continued until data saturation was achieved. The research team familiarized themselves with the data, reviewing each transcription. Next, the research team met weekly and generated a list of codes as they emerged. The team generated a thematic table of the analyses and checked the extent to which the emerging themes reflected the coded data [ , ]. The team grouped the themes under the general categories of the interview guide (ART information, ART motivation, ART behavioral skills, general game attitudes, and reactions to ViralCombat). Themes were examined in their relation to perceived utility of the game and for factors that would improve or detract from the game’s impact. Team discussion and interviews continued until discrepancies were resolved.
Participant responses on the CSQ and SEF were entered into an Excel file, and responses were verified with a second entry. Categorical response frequencies were calculated for each item of both scales. General acceptability of the intervention is illustrated using individual items from the scales. CSQ items are reported using the proportion of participants endorsing satisfaction with the intervention (response options “Very satisfied” and “Mostly satisfied” were combined). SEF items are reported using the proportion endorsing “agreement” with feasibility and utility of the game (response options “Strongly agree” and “Agree” were combined).
Reactions to ViralCombat Storyboard and iPhone Game
Interviews from both the storyboard and iPhone game revealed a number of themes that guided game development. Participants desired informational game content that included new and comprehensive details about PrEP, details about PrEP as it relates to doctors’ visits, and general health information. Motivational themes that emerged were the desire for enhancement of future orientation; reinforcement of positive influences from peers, partners, and friends; collaboration with health care providers; decreasing stigma; and increasing personal relevance of HIV prevention. Behavioral skills themes centered around self-efficacy and strategies for PrEP adherence and medical care (seefor qualitative interview themes and resulting game adaptations based on the IMB model).
highlights the barriers and facilitators to adherence expressed by our participants and the corresponding gaming app action or message that was adapted to enhance facilitators or challenge barriers. also includes general gaming attitudes that influenced the development of ViralCombat and specific reactions to the ViralCombat storyboard and iPhone game. In addition to the themes in , participants who played the game on the phone said that important gaming characteristics included directly destroying and fighting off HIV in game play, earning health points by taking pills that looked like PrEP, a prologue or introduction with a dramatic voice-over, and images about HIV and STIs that were relevant. Participants wanted levels that become increasingly difficult (for a sense of accomplishment). Participants did not want HIV or PrEP in the title of the game because of concerns about privacy and stigma but wanted to fight graphics labeled “HIV” and wanted to see the word PrEP during gaming. A 29-year-old black male participant said:
I like that the organ systems are the penis and the anus, it’s funny, but also realistic. It helped me learn about how infection with HIV happens.
A 30-year-old white male participant said:
It was awesome to shoot HIV virus before it enters the body, and I like that HIV looked like it would under a microscope.
An 18-year-old black male stated:
I liked taking pills that look exactly like PrEP; it’s like my real-life.
A 21-year-old white male stated (see):
It was cool that I am playing a game about preventing, that it was like tailored to me and my friends.
I think my other friends who think about HIV would like playing this.
The music and sound is really awesome.
The game was iteratively changed as comments were received that indicated a need for alteration. For example, facts about HIV prevention and the benefits of adherence to PrEP were made more sophisticated when multiple participants gave feedback such that they knew most of the information given in the game and that they wanted more detailed information about side effects. Many participants also asked for information about substance use. A representative comment was from a 19-year-old black male who said:
I think there should be facts in the game about other health stuff, about smoking and drinking on PrEP.
Many participants also wanted more guidance through the levels. For example, a 29-year-old white male participant stated:
I would like better hints on each level on how to beat the level.
An 18-year-old black male said (see):
Clues or hints when it gets hard would make this better.
Monitoring Pill Bottle Opening and Game-Related Text Messages
We asked participants about the SMS text messages with gaming graphics and the use of a smart pill cap that measured adherence. During the interviews, we demonstrated how openings of the pill bottle were measured wirelessly, and we showed participants sample adherence-informed SMS text messages. When looking at the smart pill cap, an 18-year-old black male participant stated:
It’s cool how it links with game.
It’s awesome that there is a bottle that knows what you are doing.
The feedback about the cap was extremely positive; however, a 19-year-old black male participant stated:
The pill cap is OK, I don’t love how big it is. People can see it in my bookbag more.
A 26-year-old white male described:
It would be better if it was a bit smaller, but I like that it can record when I take pills, that helps me.
Although 2 participants had negative remarks about the size of the pill bottle, 18 out of 20 participants responded they liked the cap design and did not think the size of the cap was an impediment to use. During interviews, participants were also shown SMS text messages that corresponded to adherence data from the smart pill cap. Participants liked the proposed SMS text messages, and an 18-year-old black male described:
These texts cue me to take my meds.
A 30-year-old black male stated:
I like the game pictures.
...the texts made me smile and also I felt like they were just for me.
A 35-year-old black male participant described:
The texts got me in the mood to play the game again.
Of the participants, 3 described that they wanted more variation in the game-related texts:
Seeing the same graphics again and again is boring.
[35-year-old black male and a 19-year-old black male]
An 18-year-old black male stated:
I would like more variety in the game texts.
Acceptability and Feasibility
CSQ and SEF scores were available from participants who played the game on the iPhone for 45 to 50 min. The responses were generally quite positive, 88% (8/9) of the participants were satisfied with the activities in the game, 77% (7/9) learned a lot from this game, 88% (8/9) thought the game was well organized, 77% (7/9) felt game topics were interesting, 100% (9/9) felt they would recommend the game to a friend, 77% (7/9) felt game topics stimulated their interest in the material, 100% (9/9) felt that game topics were relevant to their lives, and 66% (6/9) felt they were able to do the activities in the game (see). The gaming intervention was improved based on the above acceptability and feasibility feedback from the CSQ and SEF and also on the feedback from the qualitative interviews (see ). Specifically, game play was made easier with written messages and hints throughout each level on how to move forward. We also improved narrated instructions at the beginning of each level to assist players. General health facts about smoking, side effects of PrEP, and how substances such as drugs and alcohol can increase risky behavior were incorporated into the game to increase perceived relevance. We also incorporated more detailed information about HIV and how PrEP works to improve learning and interest. We also included a condom quiz game and information about other STIs (see , , and ). To improve SMS text messages, we included emojis and designed 5 different game-related SMS text messages utilizing a larger variety of game graphics based on participant feedback.
|Questionnaire or form||Participants endorsing statement, n (%)|
|Client Service Questionnaire|
|How would you rate the quality of the game?||8 (88)a|
|Did you get desired information from the game?||6 (66)a|
|To what extent does the game’s content meet your needs?||7 (77)a|
|If a friend were interested in a similar program, would you recommend our game to him or her?||9 (100)a|
|How satisfied are you with the amount of information you have received in the game?||7 (77)a|
|Has the information you received helped you to deal more effectively with issues important to you?||8 (88)a|
|In an overall, general sense, how satisfied are you with the game?||8 (88)a|
|Would you come back to the game again?||9 (100)a|
|Session evaluation form|
|I learned a lot from the game.||7 (77)b|
|I will be able to apply what I learned from the game in my life.||6 (66)b|
|I was able to do the activities in the game.||8 (88)b|
|The game was well organized.||7 (77)b|
|The topic of the game was interesting.||7 (77)b|
|The presentation of the information stimulated my interest in the material.||9 (100)b|
|The topics of the game were relevant to my life.||8 (88)b|
|The game was enjoyable.||8 (88)b|
|I would recommend the game to others.||9 (100)b|
|I felt comfortable during game play.||9 (100)b|
aProportion of participants that endorsed “Very satisfied” and “Mostly satisfied.”
bProportion of participants that endorsed “Strongly agree” and “Agree.”
In this project, we utilized in-depth interviewing, focused by social learning theory (IMB), to create an iPhone gaming intervention to measure and improve treatment PrEP adherence and decrease HIV risk for YMSM [- , , ]. A number of themes emerged through qualitative interviews with young men that informed game development. We found that YMSM desired informational game content that included comprehensive details about how HIV is transmitted, PrEP-related doctors’ visits, side effects of PrEP, and general health information. Motivational themes or findings that emerged were the desire for enhancement of future orientation; the need for reinforcement of positive influences from peers, partners, and friends; and the promotion of collaboration with health care providers. Motivational themes also included decreasing stigma and increasing personal relevance of HIV prevention. Behavioral skills themes or findings centered around self-efficacy and strategies for PrEP medication and appointment adherence including regular HIV and STI testing.
Using the IMB theory in the development of this game ensured that the intervention was informed by decades of prevention research. This study demonstrates that qualitative assessment, social learning theory, and agile software development can complement each other and are important components to the development of a tailored and clinically relevant app. Participant data were used throughout the development of the game and informed the informational, motivational, and behavioral skill-building components of the game. Using a storyboard provided the research team with opportunities to share concept models with participants early on in the design process and gather feedback with respect to necessary modifications. Sharing the iPhone game with participants as it was developed also allowed for necessary, incremental improvements. YMSM at risk of HIV infection provided key qualitative insights with respect to the content and design and process of the game. Tailored games that are informed by those who will use them have more potential for effective integration and uptake in clinical settings .
Although iPhone games are pervasive in popular culture, no other gaming apps have been developed for YMSM on PrEP. Findings of this study highlight several important barriers and facilitators to PREP adherence for YMSM. Mobile interventions have the potential to reinforce skills learned in the clinic and require fewer resources to deliver patient-centered, evidence-based interventions . Furthermore, apps and mobile phone games have the potential to engage YMSM in interventions, who otherwise may not be willing or able to participate in intervention programs.
Gaming and mobile apps also have the potential to advance the delivery of information and promote healthy decision-making in disproportionately affected populations, including minority men who often have less access to medical care and support . National data from the Pew Research Center indicate that younger, ethnic and racial minority populations use mobile phones frequently [ ]. The adolescents and young adults in this study repeatedly expressed having access to, and familiarity with, iPhones. This widespread use of iPhones facilitates the uptake of gaming apps in clinical populations. Therefore, mobile technologies such as mobile phone games and apps have a great potential to enhance medical care and prevention interventions for populations who are disproportionately affected by HIV and other STIs.
Findings should be interpreted in light of study limitations. First, our participants were recruited from a PrEP clinic in Jackson, Mississippi. This clinic may not be representative of all PrEP clinics in the United States or internationally. Therefore, the generalizability of the data collected to inform the development of the app is unknown and may be limited. Second, this study focused on individual PrEP user perspectives. It may be equally important to integrate the perspectives of clinicians, friends, or partners into the game. In the future, including friends and social networks into gaming prevention programs could be novel and effective. Perspectives of partners, friends, and clinicians could also lead to a more robust understanding of barriers and facilitators to adherence to PrEP and PrEP-related medical care. Therefore, future research could examine the utility of integrating feedback from clinicians and friends into the gaming app. Finally, this app was developed for the iPhone. Development of the app for Android devices could allow for greater availability of the game and could be a forthcoming step in the future phases of research.
This study is a significant step in the development and testing of an iPhone gaming app to promote adherence to PrEP and PrEP-related medical care. The long-term goal of this research program is to test ViralCombat in a randomized trial and examine if the game is effective in changing PrEP-related attitudes and adherence-related behaviors. If the results appear promising, the research team can distribute the technology procedures for the intervention to relevant and interested clinics, community-based organizations, the Centers for AIDS Research, and AIDS Trials Networks. The app could also be made available on iTunes and Google Play (Android) for a nominal fee.
There are many advantages to using newer interactive technology to improve adherence rather than traditional face-to-face counseling, including scalability, efficiency, and cost-effectiveness. As electronic games are highly appealing to young men , they are a natural opportunity to deliver health education during leisure time and outside of the clinic [ - , ]. Games can attract and maintain attention, which is a key component for effective behavior change. Compelling interactive games can expose players to essential health-related content thousands of times and also give players unlimited opportunities to rehearse new skills and receive personalized feedback on health choices made within the game [ , ]. We are not aware of other adherence interventions that integrate medication adherence monitoring technology, SMS text messaging, and a theoretically informed game to improve information, motivation, and behavioral skills for PrEP adherence. An intervention with these components may empower and engage YMSM, aid clinics, and result in improvements in health.
This publication was made possible with help from the Providence-Boston Center for AIDS Research (P30 AI 042853). The project described was supported by grant number R34MH104068 (MPI: LW, LKB) from the National Institute of Child Health and Human Development. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health.
Conflicts of Interest
LM is a member of the Gilead Sciences Advisory Board and receives compensation for consulting from Gilead Sciences, Inc. No competing financial interests exist for the remaining authors.
Multimedia Appendix 1
ViralCombat main menu graphic.PDF File (Adobe PDF File), 66KB
Multimedia Appendix 2
Example of a short message service text message to the participants.PDF File (Adobe PDF File), 62KB
Multimedia Appendix 3
ViralCombat game menu.PDF File (Adobe PDF File), 37KB
Multimedia Appendix 4
Short narrative movie at the beginning of the game.PDF File (Adobe PDF File), 57KB
Multimedia Appendix 5
Players can design and individualize their game character.PDF File (Adobe PDF File), 47KB
Multimedia Appendix 6
Players are shrunken down to be able to enter the body to fight off HIV.PDF File (Adobe PDF File), 29KB
Multimedia Appendix 7
Answering questions with allied doctors and building knowledge help each player successfully move to the next level or area of the body. Example of questions answered incorrectly.PDF File (Adobe PDF File), 46KB
Multimedia Appendix 8
Answering questions with allied doctors and building knowledge help each player successfully move to the next level or area of the body. Example of questions answered correctly.PDF File (Adobe PDF File), 42KB
Multimedia Appendix 9
Example of a player gaining strength in the anus level by collecting health pills and condoms.PDF File (Adobe PDF File), 23KB
Multimedia Appendix 10
As players travel through the bloodstream, they must fight off viruses and gain health pills.PDF File (Adobe PDF File), 51KB
Multimedia Appendix 11
Throughout ViralCombat, players engage in various games in order to move on to the next level. In the Condom Sequence game, players must go through the correct sequence of steps for correctly using a condom.PDF File (Adobe PDF File), 59KB
Multimedia Appendix 12
Players prevent HIV and other sexually transmitted infections in the penis by killing viruses and other bacteria.PDF File (Adobe PDF File), 54KB
Multimedia Appendix 13
Summary of points earned at the end of each level.PDF File (Adobe PDF File), 53KB
- Centers for Disease Control and Prevention. Centers for Disease Control and Prevention. 2018. HIV in the United States: At a Glance URL: https://www.cdc.gov/hiv/pdf/statistics/overview/cdc-hiv-us-ataglance.pdf [accessed 2018-08-10] [WebCite Cache]
- Underhill K, Operario D, Skeer M, Mimiaga M, Mayer K. Packaging PrEP to prevent HIV: an integrated framework to plan for Pre-Exposure Prophylaxis implementation in clinical practice. J Acquir Immune Defic Syndr 2010 Sep;55(1):8-13 [FREE Full text] [CrossRef] [Medline]
- Padian NS, Buvé A, Balkus J, Serwadda D, Cates W. Biomedical interventions to prevent HIV infection: evidence, challenges, and way forward. Lancet 2008 Aug 16;372(9638):585-599. [CrossRef] [Medline]
- Paxton LA, Hope T, Jaffe HW. Pre-exposure prophylaxis for HIV infection: what if it works? Lancet 2007 Jul 7;370(9581):89-93. [CrossRef] [Medline]
- Karim SS, Baxter C. Antiretroviral prophylaxis for the prevention of HIV infection: future implementation challenges. HIV Ther 2009 Jan;3(1):3-6. [CrossRef]
- Rotheram-Borus MJ, Swendeman D, Chovnick G. The past, present, and future of HIV prevention: integrating behavioral, biomedical, and structural intervention strategies for the next generation of HIV prevention. Annu Rev Clin Psychol 2009;5:143-167 [FREE Full text] [CrossRef] [Medline]
- Liu AY, Grant RM, Buchbinder SP. Preexposure prophylaxis for HIV: unproven promise and potential pitfalls. J Am Med Assoc 2006 Aug 16;296(7):863-865. [CrossRef] [Medline]
- Mayer KH, Wheeler DP, Bekker L, Grinsztejn B, Remien RH, Sandfort TGM, et al. Overcoming biological, behavioral, and structural vulnerabilities: new directions in research to decrease HIV transmission in men who have sex with men. J Acquir Immune Defic Syndr 2013 Jul;63 Suppl 2:S161-S167 [FREE Full text] [CrossRef] [Medline]
- Centers for Disease Control and Prevention (CDC). Interim guidance: preexposure prophylaxis for the prevention of HIV infection in men who have sex with men. MMWR Morb Mortal Wkly Rep 2011 Jan 28;60(3):65-68 [FREE Full text] [CrossRef] [Medline]
- Paltiel AD, Freedberg KA, Scott CA, Schackman BR, Losina E, Wang B, et al. HIV preexposure prophylaxis in the United States: impact on lifetime infection risk, clinical outcomes, and cost-effectiveness. Clin Infect Dis 2009 Mar 15;48(6):806-815 [FREE Full text] [CrossRef] [Medline]
- Cahill S, Taylor SW, Elsesser SA, Mena L, Hickson D, Mayer KH. Stigma, medical mistrust, and perceived racism may affect PrEP awareness and uptake in black compared to white gay and bisexual men in Jackson, Mississippi and Boston, Massachusetts. AIDS Care 2017 Nov;29(11):1351-1358 [FREE Full text] [CrossRef] [Medline]
- Arnold T, Brinkley-Rubinstein L, Chan PA, Perez-Brumer A, Bologna ES, Beauchamps L, et al. Social, structural, behavioral and clinical factors influencing retention in Pre-Exposure Prophylaxis (PrEP) care in Mississippi. PLoS One 2017;12(2):e0172354 [FREE Full text] [CrossRef] [Medline]
- Abbas UL, Anderson RM, Mellors JW. Potential impact of antiretroviral chemoprophylaxis on HIV-1 transmission in resource-limited settings. PLoS One 2007 Sep 19;2(9):e875 [FREE Full text] [CrossRef] [Medline]
- Desai K, Sansom SL, Ackers ML, Stewart SR, Hall HI, Hu DJ, et al. Modeling the impact of HIV chemoprophylaxis strategies among men who have sex with men in the United States: HIV infections prevented and cost-effectiveness. AIDS 2008 Sep 12;22(14):1829-1839. [CrossRef] [Medline]
- Golub SA, Kowalczyk W, Weinberger CL, Parsons JT. Preexposure prophylaxis and predicted condom use among high-risk men who have sex with men. J Acquir Immune Defic Syndr 2010 Aug;54(5):548-555 [FREE Full text] [CrossRef] [Medline]
- Amico KR, Mansoor LE, Corneli A, Torjesen K, van der Straten A. Adherence support approaches in biomedical HIV prevention trials: experiences, insights and future directions from four multisite prevention trials. AIDS Behav 2013 Jul;17(6):2143-2155 [FREE Full text] [CrossRef] [Medline]
- Brown A. Pew Research Center. 2017 Sep 11. Younger men play video games, but so do a diverse group of other Americans URL: https://tinyurl.com/y8msj4wu [accessed 2018-05-23] [WebCite Cache]
- Lenhart A, Jones S, Macgill A. Pew Research Center. 2008 Dec 7. Asults and Video Games URL: http://www.pewinternet.org/2008/12/07/adults-and-video-games/ [accessed 2018-05-23] [WebCite Cache]
- Shaw A. Putting the gay in games: cultural production and GLBT content in video games. Games Cult 2009 Jun 23;4(3):228-253. [CrossRef]
- Lumpkin B. MTV News. 2007 Apr 12. Gay gamers push for more player choice, like same-sex smooching in 'Bully' URL: http://www.mtv.com/news/1557022/gay-gamers-push-for-more-player-choice-like-same-sex-smooching-in-bully/ [accessed 2018-05-23] [WebCite Cache]
- Community Marketing & Insights. 2012 LGBT Community Survey URL: http://www.communitymarketinginc.com/documents/CMI_6th_LGBT_Community_Survey_USv1.pdf [accessed 2018-05-23] [WebCite Cache]
- Rainie L. Pew Research Center. 2012 Sep 11. Two-thirds of young adults and those with higher income are smartphone owners URL: http://www.pewinternet.org/wp-content/uploads/sites/9/media/Files/Reports/2012/PIP_Smartphones_Sept12-9-10-12.pdf [accessed 2018-05-23] [WebCite Cache]
- Pew Research Center. 2008 Mar 5. Mobile access to data and information URL: http://www.pewinternet.org/2008/03/05/mobile-access-to-data-and-information-3/ [WebCite Cache]
- Swendeman D, Rotheram-Borus MJ. Innovation in sexually transmitted disease and HIV prevention: internet and mobile phone delivery vehicles for global diffusion. Curr Opin Psychiatry 2010 Mar;23(2):139-144 [FREE Full text] [CrossRef] [Medline]
- Raney AA, Smith JK, Baker K. Adolescents and the appeal of video games. In: Voderer P, Bryant J, editors. Playing Video Games: Motives, Responses and Consequences. Mahwah, NJ: Lawrence Erlbaum; 2006:165-180.
- Lee KM, Peng W. What do we know about social and psychological effects of computer games? A comprehensive review of the current literature. In: Vorderer P, Bryant J, editors. Playing Video Games: Motives, Responses and Consequences. Mahwah, NJ: Lawrence Erlbaum; 2006:325-346.
- Lieberman DA. Interactive video games for health promotion: effects on knowledge, self-efficacy, social support, and health. In: Street RL, Gold WR, Manning T, editors. Health Promotion and Interactive Technology: Theoretical Applications and Future Directions. Mahwah, NJ: Lawrence Erlbaum; 1997:103-120.
- Lieberman DA. What can we learn from playing interactive games? In: Vorderer P, Bryant J, editors. Playing Video Games: Motives, Responses and Consequences. Mahwah, NJ: Lawrence Erlbaum; 2006:379-398.
- Paperny DM, Starn JR. Adolescent pregnancy prevention by health education computer games: computer-assisted instruction of knowledge and attitudes. Pediatrics 1989 May;83(5):742-752. [Medline]
- Thomas R, Cahill J, Santilli L. Using an interactive computer game to increase skill and self-efficacy regarding safer sex negotiation: field test results. Health Educ Behav 1997 Feb;24(1):71-86. [CrossRef] [Medline]
- Baranowski T, Buday R, Thompson DI, Baranowski J. Playing for real: video games and stories for health-related behavior change. Am J Prev Med 2008 Jan;34(1):74-82 [FREE Full text] [CrossRef] [Medline]
- Homer C, Susskind O, Alpert HR, Owusu MS, Schneider L, Rappaport LA, et al. An evaluation of an innovative multimedia educational software program for asthma management: report of a randomized, controlled trial. Pediatrics 2000 Jul;106(1 Pt 2):210-215. [Medline]
- Brown SJ, Lieberman DA, Gemeny BA, Fan YC, Wilson DM, Pasta DJ. Educational video game for juvenile diabetes: results of a controlled trial. Med Inform (London) 1997;22(1):77-89. [CrossRef] [Medline]
- Shames RS, Sharek P, Mayer M, Robinson TN, Hoyte EG, Gonzalez-Hensley F, et al. Effectiveness of a multicomponent self-management program in at-risk, school-aged children with asthma. Ann Allergy Asthma Immunol 2004 Jun;92(6):611-618. [CrossRef] [Medline]
- Cole S, Kato P, Marin-Bowling V, Dahl G, Pollock B. Clinical trial of Re-Mission: a video game for young people with cancer. Cyberpsychol Behav 2006 Dec;9(6):665-666.
- Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 2008 Aug;122(2):e305-e317 [FREE Full text] [CrossRef] [Medline]
- Thompson D, Baranowski T, Buday R, Baranowski J, Thompson V, Jago R, et al. Serious video games for health: how behavioral science guided the development of a serious video game. Simul Gaming 2010 Aug 1;41(4):587-606 [FREE Full text] [CrossRef] [Medline]
- Wideman HH, Owston RD, Brown C, Kushniruk A, Ho F, Pitts KC. York University.: Simulation & Gaming; 2016 Aug 18. Unpacking the potential of educational gaming: A new tool for gaming research URL: http://www.yorku.ca/rowston/unpacking.pdf [accessed 2019-01-21] [WebCite Cache]
- Stokes B. Benjamin Stokes.: The Development Education Journal; 2005 Jun. Video games have changed: Time to consider 'serious games'? URL: https://tinyurl.com/y2tktoxp [accessed 2018-05-23] [WebCite Cache]
- Hightow-Weidman LB, Muessig KE, Bauermeister JA, LeGrand S, Fiellin LE. The future of digital games for HIV prevention and care. Curr Opin HIV AIDS 2017 Sep;12(5):501-507. [CrossRef] [Medline]
- Whiteley L, Brown L, Lally M, Heck N, van den Berg JJ. A mobile gaming intervention to increase adherence to antiretroviral treatment for youth living with HIV: development guided by the information, motivation, and behavioral skills model. JMIR Mhealth Uhealth 2018 Apr 23;6(4):e96 [FREE Full text] [CrossRef] [Medline]
- LeGrand S, Muessig KE, McNulty T, Soni K, Knudtson K, Lemann A, et al. Epic Allies: development of a gaming app to improve antiretroviral therapy adherence among young HIV-positive men who have sex with men. JMIR Serious Games 2016 May 13;4(1):e6 [FREE Full text] [CrossRef] [Medline]
- Fiellin LE, Kyriakides TC, Hieftje KD, Pendergrass TM, Duncan LR, Dziura JD, et al. The design and implementation of a randomized controlled trial of a risk reduction and human immunodeficiency virus prevention videogame intervention in minority adolescents: PlayForward: Elm City Stories. Clin Trials 2016 Aug;13(4):400-408 [FREE Full text] [CrossRef] [Medline]
- Lukhele BW, Musumari P, El-Saaidi C, Techasrivichien T, Suguimoto SP, Kihara MO, et al. Efficacy of mobile serious games in increasing HIV risk perception in Swaziland: a randomized control trial (SGprev Trial) research protocol. JMIR Res Protoc 2016 Nov 22;5(4):e224 [FREE Full text] [CrossRef] [Medline]
- Baltierra NB, Muessig KE, Pike EC, LeGrand S, Bull SS, Hightow-Weidman LB. More than just tracking time: complex measures of user engagement with an internet-based health promotion intervention. J Biomed Inform 2016 Feb;59:299-307. [CrossRef] [Medline]
- Fisher WA, Williams SS, Fisher JD, Malloy TE. Understanding AIDS risk behavior among sexually active urban adolescents: an empirical test of the information-motivation-behavioral skills model. AIDS Behav 1999 Mar;3(1):13-23. [CrossRef]
- Fisher JD, Fisher WA, Misovich SJ, Kimble DL, Malloy TE. Changing AIDS risk behavior: effects of an intervention emphasizing AIDS risk reduction information, motivation, and behavioral skills in a college student population. Health Psychol 1996 Mar;15(2):114-123. [CrossRef] [Medline]
- Fisher JD, Fisher WA, Amico KR, Harman JJ. An information-motivation-behavioral skills model of adherence to antiretroviral therapy. Health Psychol 2006 Jul;25(4):462-473. [CrossRef] [Medline]
- Simoni JM, Montgomery A, Martin E, New M, Demas PA, Rana S. Adherence to antiretroviral therapy for pediatric HIV infection: a qualitative systematic review with recommendations for research and clinical management. Pediatrics 2007 Jun;119(6):e1371-e1383 [FREE Full text] [CrossRef] [Medline]
- Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice Hall; 1986.
- Romero LM, Galbraith JS, Wilson-Williams L, Gloppen KM. HIV prevention among African American youth: how well have evidence-based interventions addressed key theoretical constructs? AIDS Behav 2011 Jul;15(5):976-991. [CrossRef] [Medline]
- Resnicow K, DiIorio CK, Davis R. Culture and development of HIV prevention and treatment programs. In: Edgar T, Noar S, Freimuth V, editors. Communication Perspectives on HIV/AIDS for the 21st Century. New York, NY: Lawrence Erlbaum; 2008:193-220.
- Dybå T, Dingsøyr T. Empirical studies of agile software development: a systematic review. Inform Software Tech 2008 Aug;50(9-10):833-859. [CrossRef]
- Mission Critical Studios. YouTube. 2010 Dec 1. Dr Nano X: Incredible Voyage Inside The Body URL: https://www.youtube.com/watch?v=lyHzSZFzU1Q [accessed 2017-06-06] [WebCite Cache]
- Mission Critical Studios. iTunes. Dr Nano X: Incredible Voyage - Lite URL: https://itunes.apple.com/us/app/dr-nano-x-incredible-voyage-lite/id398377065?mt=8 [accessed 2017-06-06] [WebCite Cache]
- Larsen DL, Attkisson C, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Eval Program Plann 1979 Jan;2(3):197-207. [CrossRef] [Medline]
- Harper GW, Riplinger AJ. HIV prevention interventions for adolescents and young adults: what about the needs of gay and bisexual males? AIDS Behav 2013 Mar;17(3):1082-1095. [CrossRef] [Medline]
- Gaston L, Sabourin S. Client satisfaction and social desirability in psychotherapy. Eval Program Plann 1992 Jul;15(3):227-231. [CrossRef]
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [CrossRef]
- Arya M, Kumar D, Patel S, Street RL, Giordano TP, Viswanath K. Mitigating HIV health disparities: the promise of mobile health for a patient-initiated solution. Am J Public Health 2014 Dec;104(12):2251-2255. [CrossRef] [Medline]
- Guest G, MacQueen C, Namey EE. Applied Thematic Analysis. Thousand Oaks, CA: Sage Publications, Inc; 2012.
- Starace F, Massa A, Amico KR, Fisher JD. Adherence to antiretroviral therapy: an empirical test of the information-motivation-behavioral skills model. Health Psychol 2006 Mar;25(2):153-162. [CrossRef] [Medline]
- Gerbert B, Berg-Smith S, Mancuso M, Caspers N, McPhee S, Null D, et al. Using innovative video doctor technology in primary care to deliver brief smoking and alcohol intervention. Health Promot Pract 2003 Jul;4(3):249-261. [CrossRef] [Medline]
- Smith A, McGeeney K, Duggan M, Rainie L, Keeter S. Pew Research Center. 2015. US Smartphone Use in 2015 URL: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/ [accessed 2019-01-21] [WebCite Cache]
|ART: antiretroviral treatment|
|CSQ: Client Service Questionnaire|
|IMB: information-motivation-behavioral skills|
|IRB: institutional review board|
|MSM: men who have sex with men|
|PrEP: pre-exposure prophylaxis|
|SEF: session evaluation form|
|SMS: short message service|
|STI: sexually transmitted infection|
|YMSM: young men who have sex with men|
Edited by G Eysenbach; submitted 23.08.18; peer-reviewed by J Vandenberg, L Hightow-Weidman; comments to author 12.10.18; revised version received 05.12.18; accepted 09.12.18; published 27.03.19.
©Laura Whiteley, Leandro Mena, Lacey K Craker, Meredith Garver Healy, Larry K Brown. Originally published in JMIR Serious Games (http://games.jmir.org), 27.03.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as well as this copyright and license information must be included.