Engagement is a multidimensional construct encompassing mental, behavioral, and emotional components [20]. Sensor-based assessment is promising for children with ASD, who may have difficulty self-evaluating and expressing their experiences [21,22], while multimodal sensing can improve engagement prediction [4,23]. Accordingly, InteractiSense integrates diverse physio-behavioral and peripheral modalities to examine their individual use for engagement assessment.

The sensor metrics used are as follows: galvanic skin response (GSR), which measures skin conductance, detects affective and attentional states associated with engagement [24,25]. Heightened GSR serves as an indicator of increased engagement during interactions, making it a promising metric for identifying the engagement status [26]. Heart rate (HR), measured through changes in beats per minute, showed potential for detecting engagement in nonautistic students during structured writing activities [27]. Skin temperature (ST), a thermal response indicator, can also be used to assess arousal and attentional states [28,29]. Inertial measurements, comprising accelerometers and gyroscopes, quantify physical involvement and serve as an indicator of engagement by capturing behavioral patterns, activity levels, and synchronization with others [30-32]. In addition, physical involvement, such as grasping and progressing actions, is associated with engagement, as tasks and given interactions like applying pressure [12,33,34] and providing user input through buttons connect fundamental sensory and motor activity [35,36]. Therefore, pressure and button inputs were included for their high relevance in assessing engagement. Audio amplitude generated by children with ASD correlates with their engagement and attention during social activities [37,38], offering a noninvasive method for monitoring behavioral states as a measure of perceived loudness [28]. In this study, we integrated these multidimensional sensors into InteractiSense, which we propose to enable engagement estimation while supporting interactions for children with ASD.

Video games were among the earliest digital interventions used to entertain and encourage participation among children with ASD. Their associated console controllers have since become familiar in daily life and have been linked to behavioral regulation and reductions in stress and stereotyped behaviors [39,40]. However, familiar devices such as the Xbox controller may activate leisure-oriented associations [41], potentially constraining openness to SGs for educational, therapeutic, skill-building, or structured collaborative purposes [42,43]. Their familiar form may therefore constrain interpretive flexibility, motivating alternative interaction modalities [44,45]. Such alternatives should provide usability comparable to or better than that of console controllers [46] without imposing greater task-related demands [47], making the Xbox controller an appropriate reference interface [48].

Recent systems have combined interaction support with social engagement monitoring for children with ASD. Ward et al [32] used accelerometers to examine interpersonal synchrony in theatrical interactions, while Lanini et al [14] used a commercial wireless sensorized toy to assess ASD-related behaviors. Using a commercial toy rather than a wearable or camera-based system also offers a less invasive approach intended to improve generalizability and promote spontaneous engagement [9]. Varma et al [49] used smartphone-based gaze measures to identify social engagement during semistructured parent-child gameplay. However, these systems often rely on single modalities [14,32], limiting their ability to capture the multifaceted nature of engagement. Although multimodal sensing can improve engagement prediction [20], it remains underexplored in tangible prototypes [49]. The transitional wearable companions of Ozcan et al [8] similarly emphasize soft, interactive, and tangible designs intended to support social-skill development [11,12], child-initiated interaction, and multimodal feedback. Although tangible and spontaneous interactions can benefit peer engagement [9], systems combining them with multimodal engagement sensing remain scarce [12]. This gap motivates dual-purpose prototypes that support both interaction and sensing in SG contexts, providing a basis for future research on targeted interventions.

The InteractiSense adopts a spherical form, as its round and soft design minimizes the risk of injury for children with ASD during play [15]. This design also aligns with the natural communicative and collaborative mechanisms of socialization [50], providing a familiar and safe structure that fosters spontaneous interactions in both their daily life and therapeutic settings [9,51]. Our design process considered prior concepts from the TangToys workshop, soft animal-shaped dolls, a torus, a cube, and a sphere, which aimed to support communication and well-being [15]. The torus was excluded due to constraints in sensor deployment, and the cube due to concerns about edge sharpness [51]. A spherical structure was selected not only for its physical safety but also due to its capacity to accommodate various forms of interaction (eg, holding, pressing, and rotating) [9,51].

Designed with an outer diameter of 18 cm to support 2-handed manipulation for a typical 10-year-old (whose average hand length is approximately 15 cm), the body was fabricated using 3D printing to ensure precise placement and integration of 7 sensor positions (Figure 1). The 18 cm measurement refers to the overall spherical enclosure rather than the required grip span. To enhance its affordance and tactile experience, the handle of InteractiSense features finger grooves for a secure and comfortable grip, while its surface is covered with yellow-colored [51,52] and felt fabric, a soft textile and comforting color that reflects the characteristics of transitional objects (Figure 1) [53,54]. Furthermore, a hook-and-loop hand strap is attached to ensure a stable mount, promoting self-directed interaction by the child. The prototype (Multimedia Appendix 1) was also presented in an extended abstract for a public demonstration [55]. In this study, InteractiSense is evaluated more broadly through a mixed methods feasibility approach, including preliminary ML-based engagement classification.

InteractiSense integrates 7 physio-behavioral sensors aimed at monitoring engagement status (Figure 1 and Multimedia Appendix 2). Sensor data are wirelessly transmitted via low-energy Bluetooth at 2.1 Mbps with a 10-meter range (≤314 m² circular range), ensuring real-time multimodal transmission without data loss in spacious environments. The power uses a 1200 mAh lithium polymer battery, accounting for total power consumption and supporting continuous operation (≤10 hours), allowing for use in home settings and extended interaction periods.

We conducted expert interviews and a pilot test for iterative design with 3 teachers who work in special school (>17-year experience). The experts were given an overview of the study and participated in a pilot test of InteractiSense and the SG, assuming a child with moderate to severe ASD. We then requested a quantitative evaluation of InteractiSense through questionnaires on workload NASA Task Load Index (NASA-TLX) [47] and System Usability Scale (SUS) [46], as well as qualitative feedback on the prototype and SG, including insights on how they were perceived and the specific contexts in which the prototype could be applied in real classroom settings.

The need for enhancing the tutorial was identified by 3 experts to better support tutorial and explicit visual guidance, fostering gameplay confidence in children with ASD. We incorporated visual feedback through button-pressing animations that map the player’s input interaction to each device (Figure 2 [Tutorial]). In the initial version of the game, a small spiderweb was displayed to the player waiting for the other’s turn. This was later improved by enlarging the spiderweb and adding a colored arrow to indicate the player’s turn. The importance of emphasizing success feedback was raised, leading to enhancements, confetti effects, success reflections on in-game characters (embracing effects), and reinforced cheering sounds. Incidentally, experts noted that soft, ball-like objects are often used in therapeutic settings to promote gentle, tactile engagement, which informed our decision to adopt a spherical design. Quantitative evaluation revealed that experts’ workload and usability scores were positioned as moderate-to-high workload (mean 57, SD 17) and excellent usability (mean 84, SD 11.8), with both showing appropriate difficulty and ease of application.

Following an iterative expert review, the study involving children with ASD was conducted at the Dream Tree Children Education Center in Gwangju Metropolitan City, Republic of Korea, from January 16 to 22, 2025. The center is a community-based institution that provides educational and support services for children with special educational needs, including programs addressing basic motor function, physical fitness, and physical and psychological health. Participants with ASD were recruited from children attending the center. Recruitment began approximately 1 week before the experimental sessions. During this period, study information was posted at the center and provided to parents and caregivers of potentially eligible children. Parents and caregivers who were interested in the study were invited to indicate their willingness for their child to participate. Potential participants subsequently underwent preliminary screening for eligibility, and the final study sample was selected according to the predefined inclusion and exclusion criteria.

The experimental setup is illustrated in Figure 4. Paired children were seated side by side in front of a 55-inch display connected to an Apple MacBook Pro with an M4 Pro chip running the Unity3D-based SG. InteractiSense and the Xbox controller were connected to the MacBook Pro via Bluetooth for gameplay input, with InteractiSense additionally transmitting its embedded sensor data to the computer. Each session was recorded from front, side, and rear viewpoints using 3 cameras, providing complementary views for subsequent behavioral coding and review.

Accompanying parents participated as proxy respondents to provide assessments on behalf of children with ASD, or, when parents were unavailable, caregivers attended in their place [21,22]. They were informed about the experiment and completed consent and demographic forms, while a tutorial was conducted to familiarize participants with the experiment (Figure 3). Paired children were assigned 2 conditions (C1: InteractiSense, C2: Xbox controller) that alternated the order for each pair, following a within-subjects design. Given its widespread familiarity and established role in gaming, the Xbox controller was selected as the baseline input device, as its form [66] and usage are already well-integrated into the experiences of children with ASD [48], a decision further supported by expert feedback indicating its common use in regionally implemented educational and recreational purposes. For the Xbox controller (C2), the interaction was specifically mapped to the A and B buttons, the most accessible for right-hand operation, while all other buttons were deactivated to ensure a consistent control assessment (Figure 3). Using this recognized standard allowed us to evaluate InteractiSense as a promising option for practical applications. After each condition, proxy users completed interim questionnaires. During the experiment, 2 therapists and proxy users observed the children and managed unexpected actions. After completing all assigned experimental conditions, the accompanying parent, or a caregiver when the parent was unavailable, participated in an approximately 10-minute face-to-face semistructured interview. The postsession semistructured interviews were conducted by one of the researchers, who had no prior relationship with the study participants. Children with ASD who were able and willing to respond were also invited to share their views, which were incorporated into the qualitative findings. With consent, the interviews were audio-recorded, transcribed, and analyzed thematically to identify recurring themes. During the analysis, plausible alternative interpretations were considered, and no substantively new themes emerged from the later interviews, indicating that thematic saturation had been reached. The complete interview guide is provided in Multimedia Appendix 3.

This cross-sectional study focused on evaluating the interaction and sensing feasibility of InteractiSense. The planned sample size was 25 children, and the final sample comprised 15 eligible children. Recruitment was restricted to a single special education center to maintain observational consistency. Data from 8 participants were retained for the preliminary ML analysis. Uncertainty in the interaction-related estimates was summarized using 95% CIs for the preference odds ratio (OR) and device-specific mean usability and workload scores, together with a 95% credible interval for the Bayesian estimate of preference probability.

The measures evaluated in this study addressed 2 dimensions of feasibility: interaction feasibility and sensing feasibility. Device condition (InteractiSense vs Xbox controller) was the within-subjects experimental factor for the interaction-feasibility comparisons. Interaction feasibility was assessed through device preference, usability using SUS [46], and workload using NASA-TLX [47,67,68]. Postsession semistructured interviews further assessed perceived enjoyment, task performance and assistance needs, device-related facilitators and barriers, participation in the shared task, potential home use, and suggested improvements.

Sensing feasibility was assessed through signal-to-noise ratio (SNR) analysis of the analyzable sensor modalities and preliminary ML-based engagement classification. Sensor-derived features from the retained modalities served as predictors, while video-coded binary engagement states (engaged vs not engaged) served as the classification target. Balanced accuracy was the primary classification metric, supplemented by precision, recall, F₁-score, and confusion matrices. Participant characteristics were reported descriptively in Table 1. No covariate adjustment was applied to the device-condition comparisons. To reduce potential order effects, game-module order was counterbalanced using a balanced Latin square, and device-condition order was alternated across pairs.

Researcher-coded engagement labels for children with ASD were derived from video recordings. Each session was recorded using 3 cameras, and the ELAN software was used to annotate engagement status. The start and end times of each behavioral event were marked with second-level granularity. Each event was assigned to 1 of 2 labels (0: not engaged, 1: engaged) based on the specific criteria (Table 2). Two independent coders (WK and CL) performed the annotations, and interrater agreement was evaluated following [69], yielding an agreement score of 0.94. Only events where both coders assigned the same label were retained as the ground truth for subsequent analyses. These annotations served as the ground truths for training and evaluating the classification models.

Given the limited number of available datasets, ML methods were used, particularly those suited for imbalanced distributions and small-sample contexts (Figure 5). Input data were segmented into 25-sample windows with a step size of 0.5 seconds. To mitigate the scarcity of minority-class windows that would result from a strict majority rule, each window was labeled as 0 if the proportion of label-0 frames was greater than or equal to 0.2, and as 1 otherwise. This threshold corresponds to at least 5 label-0 frames (=0.17 seconds within a 25-sample window) and was empirically chosen to preserve the minority class while maintaining temporal consistency with observed engagement dynamics.

Feature vectors concatenated the raw samples with 12 summary statistics (mean, SD, minimum, maximum, range, median, IQR, slope, energy, root-mean-square, zero-crossing rate, and dominant frequency ratio) computed for each sensing channel. The sensing modalities used as ML inputs included inertial measurements (orientation: Yaw, Pitch, Roll; ACC: ACC_X, ACC_Y, ACC_Z; gyroscope (Gyro): Gyro_X, Gyro_Y, Gyro_Z), physiological signals (HR, GSR, and ST), and audio amplitude data. Pressure input was excluded because its sensing configuration required a low activation threshold to register brief and localized contacts, which led to a high proportion of missing values and rendered the data unsuitable for regression-based analysis and window-level labeling. Data were collected from 15 children with ASD, and samples from 8 participants were retained for ML analysis due to therapist-assisted sessions (n=4), missing data from connectivity issues or gameplay interruptions initiated at the child’s request (n=2), and annotation infeasibility due to incomplete video coverage of the child’s participation (n=1). The final dataset comprised 5745 windows after segmentation and labeling, with an imbalanced distribution (class-0: 139, 2.4%, class-1: 5606, 97.6%, ≈1:40 ratio). Model training was conducted on 4596 windows (class-0: 111, class-1: 4485).

Extreme gradient boosting (XGBoost) was selected for its capacity to handle structured sequential inputs, generate probabilistic outputs for threshold tuning, and incorporate class-imbalance handling [70]. Class imbalance was handled at two levels: (1) class-prior correction using scale_pos_weight = N₀/N₁, where N₀ and N₁ are the counts of labels 0 and 1 in the training split, and (2) sample-level weights to give higher weight to minority-class and borderline windows near the labeling threshold. For comparison, 3 baselines were also implemented: majority, which predicts the most frequent class in the training split; and random (uniform), which samples labels with equal probability; and logistic regression, a linear classifier included as a stronger baseline. Performance was assessed using balanced accuracy (primary), precision, recall, F₁-score, and confusion matrices to characterize class-wise behavior.

Model evaluation used Group 8-fold cross-validation to prevent data from the same participant appearing in both training and validation folds. Hyperparameters were tuned via random search over standard XGBoost ranges for tree depth, minimum child weight, subsample ratio, column subsample ratio, 𝛾, and L₁/L₂ penalties (Multimedia Appendix 4). Early stopping monitored validation log-loss within each fold. The decision threshold τ was selected to maximize balanced accuracy using the validation data only and was then fixed before application to the test set; no test-set information was used for threshold selection. Predictions from 3 models trained with different random seeds were averaged. All stochastic processes, including data splitting, parameter search, and model training, were initialized with fixed random seeds to ensure reproducibility.

Quantitative user preferences were analyzed using the McNemar test and ORs for discordant pairs because the preference responses were paired within participants across the 2 device conditions, thereby accounting for the within-participant dependency of the observations. The preference analysis was supplemented by a Bayesian analysis using a uniform Beta(1,1) prior, representing no a priori preference for either device. Proxy user-reported usability and workload scores, measured using the SUS and NASA-TLX, respectively, were compared using 2-tailed paired t tests, with the proxy user as the unit of analysis (n=15). To validate physiological data fidelity, the SNR was calculated via an autocorrelation function with a second-order polynomial fit.

Qualitative interview data on InteractiSense and the SG were analyzed using thematic analysis. Quantitative findings and interview themes were integrated during interpretation in the Discussion, where caregivers’ accounts of device characteristics and familiarization were used to contextualize preference and usability findings.

During the qualitative analysis, plausible alternative interpretations were considered. No substantively new themes emerged from the later interviews, which was interpreted as evidence of thematic saturation within the dataset. Findings concerning InteractiSense and the SG were grounded in interview accounts and illustrated with quotations from caregivers and children who contributed comments.

In this short-term feasibility study, preference was included as an exploratory indicator of initial receptivity to the unfamiliar interaction concept and form factor, rather than as a proxy for usability or therapeutic effectiveness. InteractiSense tended to be preferred over the conventional Xbox controller, which served as the comparison baseline in this study, although remarkable differences were not confirmed; Bayesian analysis suggested a high probability that InteractiSense was favored, indicating favorable initial reception despite its unfamiliarity. In terms of usability, InteractiSense performed slightly below the conventional Xbox controller, whereas on workload it performed marginally above; however, neither measure indicated statistically significant differences. These outcomes suggest that InteractiSense achieved a level of performance comparable to the baseline. Considering that the Xbox controller represents a familiar and widely accepted interface for children with ASD [48], achieving functional parity in these short-term usability and workload measures supports the interpretation that the prototype is usable and holds potential for practical deployment in ASD-oriented play contexts.

The tendency for InteractiSense to be preferred over the Xbox controller was echoed in postinterview accounts. Caregivers emphasized that the simplified 2-button configuration reduced the effort required to map game actions, and that the prototype’s tangible qualities, its round shape, soft texture, and bright appearance, contributed to comfort and engagement, consistent with prior work showing that tangible designs with enriched sensory cues can promote attention and collaborative play among children with ASD [51]. These characteristics helped explain why children were reported to show enthusiasm for our device despite its unfamiliarity and their tendency to favor familiarity [36]. Some caregivers also noted that once children adapted, the streamlined design could offer practical advantages, although improvements in size and flexibility would be desirable. Together, the quantitative pattern and interview accounts converge to suggest that while InteractiSense achieves functional parity with familiar interfaces, its positive reception is more accurately attributed to its ability to become an object of interest for the children, a finding that aligns with prior work underscoring the centrality of user interest [72].

Given that the Xbox controller represented a robust baseline grounded in longer-term familiarity [36,48], whereas InteractiSense was evaluated after short-term exposure, the comparable pattern observed here may be interpreted as encouraging. However, this interpretation should be considered in light of the limited sample and the absence of repeated exposure over time, highlighting the need for further validation in a larger cohort.

The integration of SNR analysis with the preliminary ML implementation indicated that selected InteractiSense modalities exhibited sufficient stability for engagement modeling, while signal quality varied across sensors. Among the sensing modalities, ST yielded consistently robust signals, even during movement, while HR and GSR provided valid and analyzable data, although somewhat less stable due to their susceptibility to detachment and motion artifacts [73,74]. The reduced stability observed for GSR is consistent with prior ambulatory GSR research showing that movement-related artifacts can compromise signal quality [75]. In the present setting, this contrast suggests a relative advantage of noncontact sensing in dynamic play contexts, while also highlighting the need for careful handling of contact-based modalities. Overall, these results indicate that InteractiSense can function not only as an interaction tool but also as a preliminary sensing platform that provides multimodal inputs for engagement modeling.

Building on this stability, preliminary ML indicated that InteractiSense holds potential to support binary classification of engagement states, with XGBoost performance exceeding baselines (eg, random guess, majority baseline, and logistic regression). Classification outcomes further indicated balanced accuracy above 0.8 across both classes, with disengagement classified with higher accuracy. However, given the inclusion of only 8 participants, the temporal dependence among windows, and the extreme class imbalance, the aggregate balanced accuracy should be interpreted cautiously and only as preliminary feasibility evidence. The model may also have captured movement- or sensor contact-related patterns associated with the annotated labels rather than engagement itself.

Valid sensing and preliminary classification together indicate that InteractiSense extends beyond its role as an interaction device, demonstrating a dual role as both a participatory social interaction medium and a sensing platform for engagement inference. In doing so, this work realizes the conceptual goals for such “TangToys” [15], while extending the capabilities of earlier sensorized toys [14] from single-modal behavioral observation to complex, multimodal engagement inference. The findings underscore its potential to guide monitoring approaches that may evolve into adaptive support strategies that, in line with the growing demand for interventions [4,76], can adjust the assistance provided based on a child’s inferred state of engagement [20]. This serves as a proof-of-concept for embedding engagement-aware sensing into tangible play environments, providing methodological direction for subsequent system development.

The integrated social and collaborative SG revealed further advantages beyond the device itself. Caregivers observed that the SG encouraged paired interaction among children who often prefer solitary play [77], with instances of turn-taking, peer encouragement, and calmer responses to setbacks. Such observations suggest that the game context provided a medium for practicing collaboration and building peer relationships, which are often difficult to nurture in children with ASD [59,78]. This finding aligns with a body of research suggesting that the clear rules and shared goals inherent in digital SGs can provide a crucial scaffold for children with ASD, simplifying the complex social demands of unstructured play [59,79]. Caregivers also expressed interest in adopting this type of content for home use, highlighting its potential to extend engagement beyond structured settings [80]. In sum, these accounts indicate that collaborative SGs integrated with novel interaction tools can amplify social affordances and point toward future directions for designing social play that fosters collaboration in children with ASD [59,78].