The local ethics committee of the Radboud University Medical Center (2020-6645) reviewed the study and judged it not to fall within the remit of the Medical Research Involving Human Subjects Act. Therefore, the ethics committee approved the study based on the Dutch Code of Conduct for Health Research and Responsible Use, the Dutch Personal Data Protection Act, and the Medical Treatment Agreement Act. Participants provided informed consent in-app before data collection. No compensation was provided to participants, and data was securely stored in a database compliant with European laws (SO 9001, 27001 and NEN 7510), ensuring privacy and confidentiality according to the Dutch Personal Data Protection Act.

The research aimed to streamline the recruitment and inclusion process and identify ways to enhance efficiency in health-related gaming studies. This study did not involve specific sample size calculations for the recruitment aim, although our original study focused on measuring social interaction through gaming. For details on the sample size calculation of the original study, see Multimedia Appendix 1 [32-34].

We recruited participants via newsletters, studies, workshops, project websites, and all project partners’ networks and social media channels (mainly through Facebook campaigns). Recruitment through the social media campaign lasted from August 2020 to February 2021 (ie, 6 months). The app remained available after the campaign ended. Recruitment efforts mainly targeted potential players aged 65 and older. However, younger players were also targeted to allow younger players to invite, assist, and play with older players. For example, we organized a gaming cocreation workshop with children and included one of the games they created in the app.

No exclusion criteria were applied to prevent in-game social network formation restrictions, allowing everyone to download the app. Participants were divided into 2 groups for analysis: older adults are those aged 65 years and older (“65+”) and younger adults are those younger than 65 years (“65–”) in accordance with the United Nations’ definition [35]. We chose 65+ as the cutoff age because this period is characterized by retirement (the average age at retirement is 65 years and 11 months [36]) and good health, often referred to as “the third age” [37]. This group is also considered well-suited for preventative measures against loneliness later in life [38].

After downloading the app, agreement to the terms of conditions was mandatory to use the app, which included sharing backend data (eg, message type and timestamp and game session information). Participants were then provided information, asked for informed consent, and administered questionnaires by a chatbot. Finally, Dutch-speaking participants 65 years or older could enroll in poststudy interviews assessing user experience regarding game and study design. For more information on participant demographics, see Multimedia Appendix 2.

We collected data through semistructured interviews and the app. The app collected data through a questionnaire administering chatbot, helping to boost completion rates as participants can stay in the app to answer questionnaires in a conversational format. In addition, it provides game data by storing backend information regarding gameplay.

Among the 155 participants who provided informed consent, the average age for all participants was 48 (SD 16) years. The “65+” group had an average age of 70 (SD 5) years, while the “65–” group had an average age of 42 (SD 12) years. Both groups exhibited characteristics of being predominantly healthy, highly educated, married, female, living independently with others, and having children (Multimedia Appendix 2). In total, 57 out of 126 (45.2%) participants who completed the DJG questionnaire at baseline reported at least moderate levels of loneliness. The subgroup of 13 out of 27 (48.1%) participants aged 65 and older, experienced moderate loneliness, while 8 out of those 27 (30%) reported severe loneliness. Twenty-five participants gave consent to be contacted by the researchers, of whom 12 (48%; female: n=12, 83%) consented to be interviewed.

This study investigated the recruitment and inclusion process of a serious game app to reduce loneliness in the older population. Due to the high dropout during the recruitment and onboarding process, substantial overshooting of the intended sample size was required. Of the 193,361 potential participants reached through social media, plus an unknown number through other channels, only 0.08% of participants provided informed consent, indicating significant challenges in recruitment. The A/B testing revealed that the campaign focusing on distanced playful interaction had a significantly higher CTR (1.98%) than the campaign focusing on research participation (CTR=0.51%). In-app activity analysis indicated that 16.4% of participants (n=91) remained active after the first week, but only 5.8% (n=32) stayed active for over 90 days. Interviews highlighted barriers such as excessive chatbot questions and a mismatch between the app’s design and the target group’s preferences. These findings underscore the difficulties in recruitment and retention.

Even though these low participation rates are considered typical in digital epidemiological research [50,51], especially in studies where enrollment is completely web-based [52], evidence for higher participation rates among older adults (though inconclusive) increased expectations [51]. Due to the nature and goal of our app, it is also possible that the stigma on loneliness [15] played a role in the low recruitment rate. Furthermore, the study’s timing during the COVID-19 quarantining period made in-person recruitment strategies (eg, workshops and talks) more challenging, if not impossible.

Most growth came from social media, recruiting more than 200 people in 1 month through Facebook, suggesting participants can be reached. However, few participants completed the entire study. Additionally, when aiming to reach new participants through existing ones, the game did not result in players inviting others once they logged into the app. To that end, improvements to increase game satisfaction must be implemented. This mouth-to-mouth recruitment is considered powerful and should be prioritized [53]. One of the possibilities could be introducing a feature to recommend the game to a Facebook friend and receive in-game rewards. Another way to improve recruitment is to optimize organic user acquisition by, for example, increasing visibility at the top of distributor channel lists like the Apple App Store and Google Play Store, updating metadata often and using metadata keywords based on seasonal contexts [53-55].

After the first week post download, 16.4% of participants remained active; 5.8% remained active after the first 90 days. The questionnaire retention rate remained around 10% compared to the baseline. These retention rates comply with gaming industry standards where a 7-day retention rate of 8.6% and 30-day retention rate of 3.5% in European countries are regular among highly rated gaming apps [56]. Internet-based interventions also experience low retention rates compared to traditional interventions, especially in older populations [57-59]. However, intervention research attrition usually needs to be much lower than that of the gaming industry; attrition of 45% is already described as high [58-61]. Furthermore, player types and participation reasons play a role in retention [62], as differences could exist between players joining for gameplay versus those joining to participate in research. This mismatch between gaming and research standards might imply that initial sample sizes must be much higher than anticipated.

Data collection procedures (ie, backend data collection and chatbot) were considered pleasurable and straightforward. However, language use needs to be adapted per age category as older participants found the chatbot speaking too childlike. The chatbot’s tone is essential as negative emotions regarding the chatbot might affect interaction willingness [63], possibly resulting in communication breakdown and increasing the number of missing values [64]. Furthermore, the questionnaires were considered too long and complicated. However, as the attrition rates after 1 month of play were similar between older adults and the younger group, it seems that questionnaire duration is not the only limiting factor, as the younger group did not need to answer all questionnaires. Additionally, even though interviewees preferred answering questionnaires in the app or on the computer, research shows that most older adults prefer answering questionnaires on paper or, if digitally, on the computer [65]. Therefore, our method of administering questionnaires could have been a barrier.

The games were designed to facilitate inter-generational contact, and participants confirmed they were most suited for children and older adults playing together. However, we found that most people played within their age stratum (ie, good friends and partners), corresponding with the literature suggesting that people of different ages do not uniformly intermingle [66]. This age segregation is not only a matter of preferences [66-69], but also a matter of institutional, spatial, and cultural segregation based on age [67,70-72]. Indicating that the design choices toward intergenerational gameplay did not have their intended effect and possibly even had an adverse impact as older users felt the app was too childish. However, the app is possibly only perceived as “childish” as participants did not interact with it within the context of its intergenerational intention.

The first strength of this study is the use of A/B testing as it helped understand recruitment through social media by providing insights into the discrepancy between the extensive reach needed and app downloads realized. Furthermore, it allowed for adjusting recruitment strategies and optimizing recruitment. A second strength is that we provide a detailed qualitative and quantitative evaluation of an innovation built with stakeholders (ie, older persons and game innovation companies), allowing for iterative analysis and improvement in a controlled and verifiable manner. A third strength regards the data collection during the COVID-19 pandemic. Loneliness has increased during COVID-19 [73,74] and many people, especially older adults, had less in-person social contact [75]. Therefore, the need for digital inclusion of older adults increased [76,77]. Furthermore, research shows that older adults use digital technology to improve social connectedness [78,79] and change how they use and adopt new technology during the COVID-19 pandemic [79]. The PlayingTogether app has the potential to fill the need for digital inclusion as it aims to make digital communication easy and accessible, possibly positively influencing some players to engage more with the game.

A first limitation is the timing of data collection during the COVID-19 pandemic as it made in-person recruitment strategies more difficult and therefore reduced inclusion numbers and lessons learned about the effectiveness of in-person recruitment strategies (eg, workshops and presentations). Consequently, recruitment shifted toward digital strategies. Though this made calculating the advertisement reach possible, it possibly caused the substantial filtering of 99.02% which is still a lower estimate as reach increases when adding unknown contributions through different channels. A second limitation is the representativeness of the target group in the in-game sample. Participants were few (ie, N=155), relatively young, highly educated, and not lonely, so the participant pool did not reflect the target group well. By campaigning toward a broad audience, we believed younger players would invite older potential participants; as lonely, more senior adults are hard to reach [80]. Though we successfully recruited a younger target group, this indirect recruitment did not occur, highlighting the difficulty of reaching a diverse group which hampers the game design process and assessment as the target group is underrepresented. A third limitation is that the app was in iterations of active development during the study, including an update on UI. This is also the case in most other eHealth innovations, as the iterations of innovation cannot wait until the end of a (slow) trial evaluation with long-term follow-up. However, making direct comparisons between players became more challenging with several app versions. Nevertheless, the interviewees’ experiences did not differ between versions, indicating that the underlying problems were more profound than those covered by the updates. Finally, our intervention was designed too broad, with the intention to attract both older adults and their grandchildren. We tested game mechanics with older adults and grandchildren, but the intersection of games and questionnaires was not tested. Furthermore, the new UI, although it received the most votes in a digital campaign, was not rigorously tested by older adults and the younger generations. Therefore, the app might have felt “childish” for older adults, and too simple for younger generations, which made the app unsuccessful in connecting the different generations. This might have been overcome by more rigorous pilots and repeated tests within this intended context.

Based on the data and insights gained in this study, we deem the following matters important to address before further (large-scale) evaluation or implementation of social games in alleviating loneliness in older adults. First, game experiences must provide clear guidance to start and continue gameplay. One option could be to introduce volunteers scouted to keep communication energetic. They could catalyze communication and “break the ice” while monitoring participant activity so necessary changes can promptly be implemented. Second, the intervention needs a sharply defined target group that thoroughly tests the app and the intervention procedures. Our approach of using a mixed target group, comprising both older and younger generations, posed challenges in effectively engaging either demographic. Third, substantial initial recruitment investments and continuous advertisement strategies are essential for long-lasting digital effectiveness trials. Current efforts are insufficient for scaling up, as shown by a 99.92% drop from exposure to download and a 90% dropout rate after 1 month. In-person recruitment could enhance engagement and reduce dropout rates.

There is increasing literature on digital gaming interventions and loneliness [81] and web-based interventions containing a gaming feature and loneliness [82]. However, we believe that further large-scale evaluation only becomes viable once these concerns are addressed. This study adds insights into the gap between the reach of potential participants and the actual recruitment outcome. Furthermore, we have shown how to design, develop, recruit for, and improve games for older adults and where these aspects might need to be improved for future endeavors.

We describe the development process and participant funnel from advertisement to app usage to provide context and guidance for future work. These stages often remain unreported even when crucial for acceptance and, thus, effectiveness. As such, this study underlines the difficulty of recruiting and retaining older adults in a social gaming intervention focused on loneliness. We showed that combining the target group with a hybrid solution to attract different generations leads to difficulty in providing a user experience where all users understand the app’s intent and feel appropriately addressed. Furthermore, we have shown the discrepancy between advertisement reach and the number of included participants. This filtering process shows that casting a wide net does not necessarily result in high participation rates, indicating that expectations and recruitment strategies should be realistically assessed when initiating such interventions. More focused small-scale, in-person recruitment strategies might therefore be more effective in increasing participation and decreasing attrition.