User Experience and Usability of Neumorphism and Gamification User Interface Designs in an HIV Self-Test Referral Program for Men Who Have Sex With Men: Prospective Open-Label Parallel-Group Randomized Controlled Trial

Introduction

Digital technology has been leveraged as an innovative approach to improve health in different settings, especially in HIV care and prevention. It was adopted for promoting HIV testing through the use of specially designed apps targeting the men who have sex with men (MSM) community [1], monitoring adherence of medication intake [2], and providing HIV prevention materials to the community [3]. A qualitative study showed that ease of operation is an important factor for MSM to use and keep using such apps [4]. This is in line with concepts of the theoretical framework in the technology acceptance model stating that perceived ease of use and usefulness contribute to one’s intention to use technology [5]. Perception of ease of use is underpinned by user interface (UI) design, such that poor visual design could negatively affect the user experience (UX) and create usage challenges [6]. Therefore, UI design is one of the key elements influencing the usability of a digital intervention [7].

There are many models and styles in UI design, and a popular one is gamification. Gamification is the application of video game elements in nongaming contexts to enhance UX [8]. It has been applied for improving drug adherence in rheumatoid arthritis patients [9] and for supporting clinical decision-making [10]. The effectiveness of gamification has been shown in the promotion of health among MSM. A previous study found that a gamified mobile app could improve HIV knowledge and confidence in adherence to HIV medications among these individuals [11]. The positive effect of gamification was markedly higher in regular users [12]. With rewards like badges, credit points, and actual prizes, and social elements, such as leader boards, the elements in the gamified experience were not just acceptable to MSM, but could also motivate them to engage in promotional activities [13]. On the other hand, a new UI design style, neumorphism or soft UI, emerged in late 2019, which was introduced by Alexander Plyuto [14]. It is a simplistic version of skeuomorphism, which was widely used in the past to imitate a real-world physical object in the UI. Neumorphism simplifies skeuomorphism by making use of light and dark shadows to accentuate the UI element from its background with the same color while capturing the realistic form of the object [15]. The combination of realistic element and the simplistic design can offer an interactive and tactile UX [16].

The 2 UI design styles of gamification and neumorphism were visually distinctive, leading us to hypothesize that different groups of MSM may be in favor of either design. Against these backgrounds, we aimed to assess the differential impact of gamification and neumorphism UI design styles on UX and usability in a web application developed for promoting HIV self-testing among MSM.

Methods

Study Design and Participants

This was a prospective parallel-group open-label randomized controlled study. The study was conducted in Hong Kong, where MSM accounted for a majority of new HIV infections in the past decade [17]. Through a social network approach, enrolled eligible participants were men who ever had sex with other men, who were aged 18 years or above, and who were either referred by another participant in the study or recruited by the research team as seeds. The exclusion criterion was inability to communicate in written English or Chinese. Block randomization was adopted with a block size of 4 and an allocation ratio of 1:1, using a computer-generated list concealed in server-side codes in the web application. With a sample size of 400, a mean difference in the acceptance score of 1% could be detected at an assumed standard deviation of 10% between 2 arms.

Intervention

The participants were randomized into either the neumorphism or gamification UI arm, with the unified objective of undertaking a free HIV self-test using oral fluid or blood at their will. The differences in the 2 arms were related to the UI only (primarily visual). In the neumorphism arm, in terms of form elements, the buttons and frames were in the neumorphism style, while checkboxes and radio buttons remained browser default to balance readability and identification of form elements (Multimedia Appendix 1). A monochromatic color scheme was adopted in compliance with the neumorphism design style. To enrich a gamified UX, the process table in the neumorphic design was replaced by a badge board, and a story background was created (Figure 1). Each of the pages in the gamification arm had a unique backdrop simulating different places in a role-playing game. The profile and questionnaire pages appeared as a forest in daytime and at night, respectively. The test kit request and result upload pages depicted a sword to be picked from the soil and items to be devoted in a shrine, respectively. While training was illustrated as a magical book, the backdrop on the referral page was a tavern. The incentive redemption page was assimilated as a reward bestowed by the “crown.” The study procedures remained the same between the 2 arms, namely the questionnaire, test kit request, test result upload, online training, and peer referral.

‎View this figure

Data Collection

The questionnaire contained questions on demographics; sex networking characteristics, including self-perceived body image types and preferences of sex partners’ body image types [18,19], history of sexual behavior, sex networking, and HIV testing; preferences of self-tests; and 17 questions on HIV-related knowledge. The primary outcome of the study was the usability of the web application measured by the System Usability Scale (SUS) [20] and the Single Ease Question (SEQ) [21], as well as the satisfaction of the UI designs of the 2 styles. The SEQ was used after each task throughout the study, whereas the SUS was only displayed after test result upload and peer referral. Participants were free to fill out these scales without obstruction to the main study procedure. Akin to the SUS, an additional question, “I am satisfied with the system interface design,” with a scale ranging from 1 (strongly disagree) to 5 (strongly agree), was used to assess participants’ acceptance of the UI (UI score). For individual SUS items, a score of at least 4 out of 5 was considered a positive response, whereas for the SEQ, a score of 5 or higher on the scale of 7 was regarded as positive. Overall, the SUS score was calculated according to the standard by inversing responses in even number questions and then multiplying the scores by 2.5. Previous research has shown that an SUS score of 68 is average [22], 71 is acceptable [23] or “good” [24], and 80 indicates a user is willing to promote the product or system to a friend [21]. In the analysis, we adopted the latter 2 thresholds as the definitions of acceptance and participation as a promoter of the system, respectively. Two subscale scores were derived from the SUS by separating items 4 and 10 from the remainder to form the learnability and usability scores, respectively, and the same threshold was used.

Analysis

The scores of the UI, individual SUS items, SUS main scale and subscales, and SEQ at different stages were compared between the 2 arms, and other participant attributes and sex networking characteristics were compared using the Mann-Whitney U test. With an overarching aim to examine the willingness of referring peers to undergo self-testing, the SUS scores were dichotomized by the promoter-level threshold, and associated factors were identified per arm by univariable and multivariable logistic regression models. The analyses and reporting of the results followed the CONSORT-eHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) guidelines [25] (Multimedia Appendix 2). This study has been registered at ClinicalTrials.gov (ID: NCT04379206). All analyses were conducted in R software (R Project for Statistical Computing).

Ethics Approval

This study was approved by the Joint Chinese University of Hong Kong – New Territories East Cluster Clinical Research Ethics Committee and the Ethics Committee of the Department of Health (CREC reference number 2020.087).

Results

Overall, 463 MSM registered in the study between March 1, 2021, and May 12, 2021, of whom, 232 and 231 were assigned to the gamification and neumorphism arms, respectively (Figure 2). After excluding 21 subjects who had not completed the questionnaire and 8 who failed to request a self-test kit, data from 434 participants were analyzed. Overall, the participants’ median age was 28 years (IQR 24-33 years) (Table 1). Moreover, 76.5% (332/434) and 43.3% (188/434) of participants had used location-based MSM social networking apps and social media apps, respectively, to seek male sex partners in the past year, whereas patronizing physical venues, such as bars (33/434, 7.6%), saunas (56/434, 12.9%), and sex parties (26/434, 6.0%), for sex networking amidst the COVID-19 pandemic when social distancing policies were in force was rare. In terms of HIV-related knowledge, 59.0% (256/434) of participants got 11 correct answers and 21.0% (91/434) gave at the most 3 wrong answers out of 17 question items. Overall, 340 (78.3%) had tested for HIV before, of whom, 139 (40.9%) never used an HIV self-test product. More participants were confident to collect sufficient oral fluid (269/434, 62.0%) than blood (197/434, 45.4%) for the self-test. In addition, 66.8% (290/434) were confident in performing self-tests and 70.3% (305/434) were confident in interpreting the test results. A similar proportion (282/434, 65.0%) did not prefer assistance in self-tests.

‎View this figure

The median SUS score given at test result upload was 80.0 (IQR 70.0-90.0), with median learnability and usability scores of 87.5 (IQR 62.5-100.0) and 78.1 (IQR 68.8-90.6), respectively. Higher learnability was scored by MSM who used social media for sex networking (P=.02), preferred sex partners to be of the “cute” body image type (P=.009), never had sex with a female individual (P=.04), and gave more correct answers in HIV-related knowledge (P=.005) (Table 2). They also gave higher confidence scores in sample collection for both oral fluid (P=.002) and blood (P=.004), performance of the self-test (P<.001), and result interpretation (P<.001). A higher usability score was given by participants who used social media for sex networking (P=.002), preferred sex partners with the body image type of “cute” (P=.03) or “decent” (P=.02) but not “mature” (P=.04), were confident in using the self-test (P<.01), and returned the self-test result within 24 hours upon receiving the self-test kit (P=.04) (Table 3).

Promoters giving at least 80 points in the SUS scale at test result upload (184/351, 52.4%) were more likely to have passed the online training, made a peer referral, used social media for sex networking, used pre-exposure prophylaxis (PrEP) for HIV prevention when having condomless anal intercourse (CLAI) with known male sex partners, and given a higher UI score (P<.05). They were also more confident in using the self-test (P<.001), while preferring no assistance for the self-test (P=.001), and had requested an oral fluid self-test (P=.04) (Table 4). While promoters did not give more correct answers in HIV-related knowledge, nonpromoters were more likely to wrongfully consider blood donation an appropriate way for HIV screening in the HIV-related knowledge assessment (odds ratio [OR] 2.57, 95% CI 1.33-4.95; P=.01). They also gave a higher SEQ score after the test kit request, result upload, online training, and peer referral (P<.01).

Participants in the neumorphism arm gave a higher SUS score than those in the gamification arm (median 82.5 vs 77.5; “Excellent” vs “Good” using the adjective rating scale [24]; P<.001), and more participants in the neumorphism arm had an acceptance level (P<.001) and a promoter level (P=.002) (Table 5). The learnability score was similar between the 2 arms; therefore, the differences in the scores of SUS items 4 and 10 were insignificant. On the other hand, the usability score was higher in the neumorphism arm (P<.001). The proportions of participants who were satisfied with the UI (P=.01) and agreed to all positively worded statements in the SUS, except for item 9 on confidence in system use, were higher in the neumorphism arm (P<.01).

To identify the distinct characteristics associated with a promoter-level SUS score with gamified and neumorphic designs, subgroup analyses were performed (Figure 3). Promoters in the gamification arm were more likely to prefer sex partners to be of “meaty,” “hairy,” or “bear” body image types, request for an oral fluid test, and give a higher SEQ score at peer referral (P<.05). They were also less likely to give a wrong answer in some HIV-related knowledge questions (P<.05) and more likely to use PrEP when having CLAI with known sex partners (P=.009) and pass the online training (P=.04). Neumorphism arm promoters were more likely to give a higher SEQ score at test kit request (P<.001) and be referred from channels other than instant messaging apps (P=.02). The multivariable logistic regression model showed that, other than high UI (P<.001) and SEQ (P=.02) scores, promoters in the gamification arm were more likely to request for an oral fluid test (P=.001), prefer a “bear” body image type partner (P=.02), and be confident in collecting blood samples (P=.003), while those in the neumorphism arm gave a higher SEQ score at result upload (P<.001) and were confident in performing the self-test (P<.001).

‎View this figure

Discussion

The social network–based HIV self-test web application was well-accepted by MSM, with an overall median SUS score of 80. Higher system usability was associated with confidence in using the self-test kit and completing the self-test quicker, indicating that this application had linked the user experiences of online activities with offline ones. MSM encountering issues in learning to use the system were less knowledgeable about HIV and more likely to be bisexual. This echoed the results from a previous study showing that bisexual men had lower HIV-related knowledge, who had a higher level of stigma or discrimination associated with the infection [26]. Online social networking tools could be leveraged to promote health behaviors [27], and users gave higher usability and learnability scores. Together with the participants’ distinct preferences in sex partners’ characteristics, our results informed the need of addressing bisexual MSM and the choice of key opinion leaders of different body image types in future HIV-related promotion campaigns through social media.

The study results confirmed that participants who were satisfied with the system and with a high perceived ease of use throughout the engagement, were more likely to promote it to their peers [21]. Promoters were confident in using the self-test without assistance. They tended to have used PrEP for HIV prevention and possessed adequate HIV knowledge; therefore, they were competent to share their positive self-test experiences and knowledge in HIV, and access to HIV prevention services with their peers, which could in turn invite them to get self-tested. On the other hand, knowledge deficit may have hindered some from referring their peers to engage in prevention programs. This showed that a satisfactory UX, adequate HIV knowledge, and peer referral and support could facilitate HIV testing promotion programs [28].

Participants in the 2 arms were satisfied with the system and the interface design. Compared with the findings in the gamification arm, higher SUS and UI scores were observed among participants assigned to the neumorphism arm, with over half giving a promoter-level score. The impact of UI on UX was only reflected in the usability facet, but not the learnability facet. In reference to the 10 usability heuristics of UI design [29], “esthetic and minimalist design” was found to be of particular importance in contrasting the 2 designs in this study. Neumorphism adopts a simplistic visual design conforming with the heuristics, whereas the colorful graphics in the gamification interface may have distracted users from the primary goal and subsequently weakened its usability. Nevertheless, no matter which arm participants were assigned to, almost all users considered the system consistent and agreed that they did not need to learn a lot of things or ask for support to use the system. These were underpinned by the fourth and sixth heuristics, which were “consistency and standards” and “recognition rather than recall,” respectively.

Preferences in UI design style may be related to one’s personality, age, gender, and education [30,31]. Shared characteristics among promoters in the 2 arms included higher SEQ and UI scores at the same time when the promoter-level SUS score was given, and confidence in performing the self-test without assistance. No personal attributes were associated with promoters in the neumorphism arm, but those in the gamification arm were featured with partner body image type preference, PrEP use, and HIV-related knowledge. This highlighted that the gamified UI was favored by selected MSM, whereas the neumorphic design was more universally accepted. Our finding provides an insight into the future selection of art styles for designing promotion materials. Although physical appearance traits may differ contextually, a previous study in the United States has suggested that the bear community could be a unique group characterized by body image type in the MSM population, which may be at higher HIV risk than others [32]. If a campaign is launched to target them and their potential partners, a gamified approach could be of use. A previous study also showed that gamification could make HIV and sexually transmitted infection testing less stressful [33]. On the contrary, a neumorphic style would be more suitable to reach the general MSM community.

This study has several limitations. First, the use of sensitive sexual behavioral questions might have given rise to social desirability bias in responders, which we have minimized by adopting a self-administered approach. We also used a short recall period to reduce recall bias. Second, in a randomized parallel-group design, we were unable to capture participants’ preferences by presenting them with both UI designs. Instead, we compared the scores between 2 arms and conducted subgroup analyses to identify associating factors with scores reaching the promoter level. The SUS scores used in this study were collected at test result upload; therefore, all participants who did not upload their results were excluded. As the primary goal of the study was to have participants refer their peers for self-testing, uploading a test result was a necessary step, and thus, only those who had uploaded a test result were considered to have completed the study procedure and were included in the analyses. Third, HIV knowledge was not assessed with a conventional validated scale, as such a scale may have been outdated with the recent development of HIV medicine and PrEP. The knowledge questionnaire we designed could however only serve the purpose of identifying a knowledge gap in the community, which was relevant to the theme of the study. Fourth, generalization to other settings with different cultural backgrounds could be limited, particularly for body image types. Finally, neumorphism has an inherent weakness in accessibility due to the lack of contrast in color. We did not enquire about users’ color vision; hence, we were unable to consider the impact of color vision deficiency on the usability of the 2 UI designs.

In conclusion, we demonstrated the effective use of a web application that could link users from online activities to offline engagement for the purpose of promoting HIV testing through the collection of self-test kits. The experience of the offline activity could impact subsequent online engagements, which involve peer referral for extending the network of MSM who could go for HIV self-testing as a health promotion strategy. We also identified MSM’s preferences in and possible implementation scenarios of gamified and neumorphic UI designs, laying the scientific foundation for future UI and UX designs for internet interventions targeting the MSM community.