Authors: Ali Ebrahimi, Jakob Bo Reinevald Eriksen, David Krogh Kølbæk, Jonas Mohr Pedersen, Ebbe Vincent Just Christensen, Søren Skovbakke, Ole Skov, Susanne Schmidt Pedersen, Amir Sorayaie Azar, Uffe Kock Wiil
Categories: Research, Psychological distress, Anxiety, ICD, Patient-reported outcomes, Machine learning, Ensemble model, Explainable AI, SHAP
Source: BMC Medical Informatics and Decision Making
Authors: Ali Ebrahimi, Jakob Bo Reinevald Eriksen, David Krogh Kølbæk, Jonas Mohr Pedersen, Ebbe Vincent Just Christensen, Søren Skovbakke, Ole Skov, Susanne Schmidt Pedersen, Amir Sorayaie Azar, Uffe Kock Wiil
Cardiac patients with Implantable Cardioverter Defibrillators (ICDs) are at elevated risk of experiencing anxiety, depression, or general psychological distress. Nevertheless, routine psychological screening in cardiology care is limited, and early identification of vulnerable individuals remains a challenge, particularly in settings transitioning to remote monitoring. Scalable, data-driven tools are urgently needed to support timely detection and intervention.
We developed Machine Learning (ML) models to predict anxiety and psychological distress using patient-reported outcomes from the ACQUIRE-ICD dataset (N = 478). The methodology included imputing values, selecting features strategically, and tuning hyperparameters. Nine ML algorithms and two ensemble approaches (stacking and voting) were evaluated. The best-performing models were interpreted using SHAP (SHapley Additive exPlanations) to enhance clinical transparency.
The stacking classifier achieved an F1-macro score of 0.93 for anxiety detection, while the light gradient boosting machine reached 0.93 for distress prediction. Feature importance analyses revealed that Mental Health Scores (MCS), loneliness scores, depression and anxiety scales, and quality-of-life indices were the most predictive features. Balancing methods such as SMOTE-ENN did not improve model performance and were excluded from final configurations.
Our findings demonstrate that combining clinical insight, structured preprocessing, and ensemble learning leads to highly accurate and explainable models for detecting psychological distress in ICDs patients. The integration of SHAP enables interpretability and supports clinical trust, paving the way for scalable decision-support tools in cardiology. Future work should explore external validation and real-world deployment.
Not applicable
The online version contains supplementary material available at 10.1186/s12911-025-03315-x.
Heart failure represents a complex and progressive condition that often culminates as the final stage of various cardiac diseases. For many patients, the implantation of an Implantable Cardioverter Defibrillator (ICD) is the standard intervention to mitigate the risk of sudden cardiac death, particularly in those at high risk of life-threatening arrhythmias [1, 2]. In recent years, the follow-up care of ICD recipients has shifted from traditional in-person consultations to remote monitoring, which offers notable clinical and economic advantages [3]. However, this digital transformation has reduced direct patient-clinician contact, complicating the early identification of vulnerable individuals who may require additional psychosocial support [4].
In Denmark and many other healthcare systems, psychological screening is not routinely embedded in cardiology care pathways, and access to psychological interventions within hospital settings remains limited [5]. This oversight is concerning, given the elevated burden of mental health issues in cardiac populations. For instance, individuals recovering from myocardial infarction are nearly three times more likely to experience psychological distress compared to the general population, with 15–20% meeting criteria for major depressive disorder and many more reporting subclinical symptoms of depression [6]. While the majority of ICD recipients adjust well post-implantation [7], approximately 20% face difficulties adapting, presenting with clinically significant anxiety, depression, or both, collectively referred to as psychological distress alongside reduced quality of life [8]. These symptoms have been linked to adverse outcomes, including increased all-cause mortality and a higher incidence of ventricular arrhythmias, even when advanced ICD technologies are in place [9, 10].
Timely identification and treatment of psychological distress is critical for improving mental health, physical well-being, and overall quality of life in this population [11, 12]. Nevertheless, current diagnostic practices rely heavily on resource-intensive clinical assessments or standardized questionnaires administered by healthcare professionals [13, 14]. These tools, while effective, are time-consuming and may not be feasible for large-scale or routine use. As such, there is a pressing need for scalable, data-driven approaches to support early detection and management of psychological distress in ICD patients.
With the increasing prevalence of mental illnesses and the increasing pressure on the healthcare system in general, there is a need for more effective procedures and automation. As described in an article by et al. [15], many practitioners struggle to accurately and efficiently diagnose patients. Nonetheless, machine learning can be applied as a decision-support tool to ease practitioners’ task of diagnosing patients. The main ideas and takeaways from incorporating machine learning into the health care system are improving the quality of the care patients receive while reducing clinicians’ workload through the analytical and predictive power of machine learning. However, the machine learning algorithms used will also need to be transparent and explainable for the benefit of both clinicians and patients [15].
Priya, et al. [16] investigated anxiety, depression, and stress prediction using machine learning. The study included 348 participants (aged 20–60) and employed several machine learning models. Labels were derived from the DASS-21 questionnaire. Among the developed algorithms, Naive Bayes achieved the highest F1-score for depression (0.836), random forest for stress (0.711), and K-nearest neighbor for anxiety (0.527), indicating suboptimal performance for anxiety prediction. In another study, Sau and Bhakta [17] explored machine learning for screening for anxiety and depression among seafarers. The study included 470 participants, categorized as “Anxiety and/or Depression” (51.3%) or “No Anxiety – No Depression” (48.7%) based on HAM-A and HAM-D scores. They employed several machine learning models, and CatBoost achieved the highest performance with 89.3% accuracy and 89.0% precision on the test set, while other models had accuracies ranging from 78.6% to 87.5% and precisions ranging from 76.9% to 84.0%.
Nemesure, et al. [18] used electronic health records and machine learning to predict depression and anxiety among undergraduate students (N = 4,184), with 12% diagnosed with Major Depressive Disorder (MDD) and 8% with Generalized Anxiety Disorder (GAD) based on screening questionnaires and DSM-IV criteria. They extracted 59 biomedical and demographic features from a general health survey, excluding highly interdependent MDD-related variables. The best-performing machine learning model achieved an AUC of 0.67 on the test set, with MDD showing 0.55 sensitivity and 20% PPV, and GAD showing 0.70 sensitivity and 16% PPV. They employed SHAP to present feature importance analysis. In another study, Sau and Bhakta [19] applied machine learning to predict anxiety and depression in elderly patients (N = 520). Labels were assigned based on the Hospital Anxiety and Depression Scale (HADS), categorizing participants as “Anxiety and/or Depression Present” or “Absent.” Features included personal, socio-demographic, economic, and medical attributes. Feature selection employed five filter-based methods, Correlation Feature Selection, OneR, PCA, Gain Ratio, and Symmetrical Uncertainty, reducing features from 20 to 10. Models were evaluated on an independent test set (N = 110, class 53 anxiety/depression, 57 neither), with random forest achieving the highest performance (91% accuracy, 92% TP-rate, 10% FP-rate, 89% precision, 90% specificity, and 90% F-score).
Andersen, et al. [20] proposed a transformer-based approach for detecting psychological distress in cardiac patients with ICDs using Danish text messages. Their study utilized data from 168 participants in the ACQUIRE-ICD trial [21], labeling messages as “Distressed” or “Non-Distressed” based on questionnaire responses. They trained multiple transformer models, including mBERT, XLM-RoBERTa, ÆLÆCTRA, and RøBERTa, and employed a hard voting ensemble technique to improve classification performance. To mitigate data scarcity and class imbalance, the study incorporated data augmentation techniques. The ensemble model achieved the highest classification performance with 80% precision, 67% recall, a 73% F1-score, and 75% accuracy, outperforming individual transformers. Their findings highlight the potential of natural langugage processing based approaches in identifying at-risk cardiac patients and supporting early psychological interventions in clinical settings.
Given the limitations of existing screening approaches and the need for scalable, timely identification of psychological distress among ICD patients, this study aims to develop and evaluate advanced machine learning models capable of accurately detecting anxiety and distressed in this vulnerable population. Specifically, we use structured patient-reported outcome measures (PROMs) from the ACQUIRE-ICD dataset and implement a rigorous, transparent pipeline that integrates feature selection, imputation, and scaling within a reproducible machine learning framework.
Unlike prior ICD-related machine learning frameworks that have primarily utilized unstructured textual or physiological sensor data for distressed detection [20, 21], the present study introduces a novel integration of structured PROMs with interpretable ensemble learning. This approach enables both robust predictive accuracy and direct clinical interpretability, as the derived models can transparently explain how individual psychosocial and health-related variables contribute to anxiety and distressed predictions. The use of PROMs not only ensures reproducibility and scalability across clinical settings but also facilitates future integration into digital health platforms and routine follow-up care. By combining explainability with methodological rigor, this study contributes a new paradigm for psychologically informed risk stratification in ICD populations.
The contributions of our study can be summarized as follows*Improving the performance of psychological distress predictive models for ICD * We developed optimized machine learning models by systematically integrating SimpleImputer to address missing values, Forward Greedy Search (FGS) for strategic feature selection, and ensemble learning techniques (stacking classifiers and light gradient boosting machine) to achieve superior predictive accuracy. The models were trained on the ACQUIRE-ICD dataset and demonstrated robustness despite the dataset’s highly imbalanced nature.*Analyzing model performance and comparison with current state-of-the-art predictive * We rigorously evaluated our developed models against baseline classifiers, applying a structured preprocessing pipeline (scaling, imputation, feature selection) and hyperparameter optimization. Our models were benchmarked against existing research employing similar machine learning approaches. Statistical evaluation and t-test analysis not only confirmed the superior performance and generalization capability of our best models compared to base models but also achieved promising results compared to previous studies.*Identifying clinically meaningful * Key clinical and psychosocial factors associated with anxiety and distressed among ICD patients were identified using advanced feature selection methods and explainable AI (SHAP analysis). These clinically relevant factors included patient-reported outcomes such as mental component summary scores, loneliness indices, depression and anxiety scores, and perceived health-related quality of life.*Enhancing interpretability for clinical * To facilitate real-world clinical adoption, we employed explainable AI methods (SHAP) to provide transparent and interpretable insights into model predictions. This approach is designed to build clinical trust and enable actionable psychosocial interventions, thereby making the developed models practical decision-support tools within cardiology care pathways.
As illustrated in Fig. 1, the overall methodology adopted in this study encompasses four primary 1) data collection, involving the integration and preprocessing of the ACQUIRE-ICD dataset; 2) preprocessing including imputation, feature selection, data scaling, and balancing; 3) model development, involving rigorous evaluation of multiple classifiers and ensemble learning strategies (stacking and voting methods); and 4) model evaluation, incorporating performance metrics and explainable AI (SHAP) to ensure clinical interpretability. Clinical experts were actively involved throughout all stages, including critical decisions on labeling thresholds for anxiety and distressed based on validated clinical scales (PHQ-9 and GAD-7), as well as in-depth discussions to ensure the medical relevance of selected features.Fig. 1Overall method
The datasets utilized in this study were provided by six national ICD implanting centers in Denmark [21]. Following a comprehensive assessment, the ACQUIRE dataset was selected as the primary data source for its integration of multiple data points from additional datasets. Specifically, the ACQUIRE dataset contained 1,346 records and initially comprised 2,108 features, a scale necessitating careful data preprocessing to ensure analytical manageability and relevance. Initial data preparation steps included removing redundancy and creating new derived metrics, producing a refined dataset tailored for machine learning applications. To align with the study’s objective of proactively identifying patients at increased risk of anxiety or depression, the refined dataset was segmented into four distinct subsets corresponding to anxiety and depression predictions at 6-month and 12-month intervals.
The complexity and volume of the features necessitated strategic feature selection informed by expert clinical consultation. Features were primarily discarded based on their irrelevance to anxiety and depression prediction, as indicated by categorical prefixes such as ‘inclusion procedures’ and ‘dropout analyses’. Remaining features, carefully selected with clinical guidance, included socioeconomic factors, general health indicators, and specific PROMs such as PHQ-9, GAD-7, EQ-5D-5 L, and SF-12. The periodic nature of data collection via patient-completed PROMs resulted in intermittent missing values across the baseline, 6-month, and 12-month intervals. To address this, records were strategically backfilled before duplicates were removed, thereby preserving critical longitudinal data.
Labeling for anxiety and depression utilized validated the GAD-7 for anxiety and the PHQ-9 for depression. At first, a threshold of 14 was set for moderate severity; however, due to significant class imbalance, clinicians advised lowering it to 9 (mild severity) to improve class balance. This adjustment improved the dataset’s suitability for training predictive models to distinguish between anxiety and depression cases. Ultimately, the processed dataset (ACQUIRE-ICD) used for analysis contained 478 records and 142 features, encompassing demographic variables (e.g., BMI, age) and summarized patient-reported measures derived from the original data sources.
The labels from both the anxiety and the depression datasets were combined into a new dataset called distressed. If the patient has either anxiety or depression, they would be labelled as distressed. This adjusts the classification task to focus on classifying a disturbance of the mind rather than a specific disease. This was done to improve classification performance by increasing the number of positive samples.
The preprocessing pipeline was designed to handle common challenges in clinical datasets, including missing values, high-dimensional features, and class imbalance. The full process included three core imputation, feature selection, and balancing, with performance evaluated using five-fold cross-validation and F1-macro as the primary metric. To prevent data leakage during cross-validation, imputation, scaling, and feature selection were performed separately within each training fold. The fitted transformations were subsequently applied to the corresponding test fold, ensuring that the test data remained unseen during model training and preserving evaluation integrity. To accommodate limited sample sizes and maintain robustness, the pipeline was applied using the entire dataset without a separate train-test split during preprocessing. This allowed consistent exploration across all preprocessing variations before final model evaluation.
A total of 9 model candidates were tested using this pipeline, and for each, all combinations of preprocessing steps were evaluated, yielding 3,312 unique F1-macro scores across the experimental space. Furthermore, detailed explanations of preprocessing methods that are utilized in this study are described in Appendix A. Notably, for Feature Selection (FS), to evaluate different feature selection strategies robustly, we implemented an ensemble-based FS framework adapted from Ebrahimi, et al. [22]. This framework integrates multiple selection techniques across three methodological filter methods, wrapper methods, and embedded methods, organized into a two-level ensemble that are thoroughly described in Appendix A.
All methods were applied to both anxiety and distressed datasets for preproccesing steps, and also each balanced version of the dataset was evaluated using five-fold stratified cross-validation with F1-macro as the primary metric. Nonetheless, across all experiments, oversampling-based approaches consistently achieved lower predictive performance than models trained on the original unbalanced dataset. This finding aligns with prior research showing that when the minority class is extremely small or heterogeneous, SMOTE-based synthetic interpolation may generate ambiguous samples or artificially smooth complex feature interactions [23–25]. In our case, anxiety and distressed outcomes originate from multifactorial psychosocial indicators, making them difficult to represent with synthetic oversampling.
Based on empirical evidence, the final models were therefore trained on the original, unbalanced dataset, yielding more stable and clinically meaningful performance. Retaining the natural distribution helped avoid artificial oversampling noise, preserved the structure of PROM-based psychosocial data, and resulted in the highest cross-validation performance across both prediction tasks.
Although balancing methods were comprehensively explored for completeness, their performance limitations justify our decision to rely on the original class distribution for the final classifier development.
To systematically explore the impact of different preprocessing strategies on classification performance, we constructed a large-scale combinatorial pipeline. This process involved applying multiple imputation, feature selection, and balancing techniques in a layered fashion for each candidate model.
The exploration followed a multi-stage pipeline expansion, as outlined belowA.Dataset and Imputation Layer
We worked with two distinct datasets (anxiety and distressed detection). For each, four imputation strategies were applied, resulting 2 datasets × 4 imputers = 8 imputed dataset variants.B.Feature Selection Layer
Each imputed dataset was processed using nine distinct feature selection strategies, including both base and ensemble methods. This expanded the pipeline to 8 imputed datasets × 9 feature subsets = 72 unique feature-selected variants.C.Balancing Layer
To address class imbalance, four oversampling methods were applied to each of the 72 feature-selected variants, yielding 72 feature-selected datasets × 4 balancing techniques = 288 fully preprocessed datasets.
At each of these three layers (imputation, feature selection, and balancing), classification performance was independently evaluated using 5-fold cross-validation and F1-macro scoring. This design allowed us to trace the performance impact of each preprocessing step, independently and in combination.
For every developed model, the following number of scores were Imputation 8 variantsFeature selection 72 variantsBalancing 288 variantsTotal preprocessing evaluations per 368
With 9 model candidates, the cumulative number of evaluations
368 × 9 = 3,312 unique F1-macro scores
This extensive evaluation framework enabled a detailed understanding of how each preprocessing decision affected model performance and allowed us to identify the most effective pipeline configurations per model.
Hyperparameter tuning is essential for optimizing model performance and preventing under- or overfitting. In this study, we applied two a sequential approach combining Randomized Search followed by Grid Search, and Bayesian Optimization using BayesSearch. Grid Search exhaustively evaluates all parameter combinations within a predefined space, ensuring optimal results at the cost of computation time. In contrast, Randomized Search samples a limited number of random combinations, making it more efficient for high-dimensional spaces. BayesSearch uses previous evaluations to model the search space and prioritize promising areas, enabling faster convergence toward optimal settings. All tuning procedures incorporated cross-validation and used F1-macro as the scoring metric.
This section outlines the candidate machine learning models selected to evaluate classification performance on the ACQUIRE-ICD datasets. The following classifiers were included in the Random Forest (RF), Extreme Gradient Boosting (XGB), Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Multi-Layer Perceptron (MLP), Naive Bayes (NB), Light Gradient Boosting Machine (LGBM), Decision Tree (DT), and Logistic Regression (LGR). These models were chosen to reflect a diverse set of algorithmic principles, including tree-based ensembles, distance-based methods, neural networks, probabilistic models, and linear classifiers. The rationale for this diversity was to ensure robustness across the heterogeneous and imbalanced characteristics of the dataset, where no single modeling paradigm could be presumed optimal a priori. A summary of each classifier’s core mechanism is provided in Table A1 in Appendix A.
To maximize predictive performance, we implemented two ensemble learning strategies [26]: stacking and voting, using the top models identified during hyperparameter tuning. These ensemble approaches aim to leverage the strengths of multiple well-performing base models and mitigate their individual weaknesses by aggregating their predictions.
In the stacking framework, the five base models were combined with three different meta-learners: Random Forest, Logistic Regression, and Support Vector Machine. Each combination of base learners and meta-learners was systematically evaluated using the F1-macro score to identify the most effective pairing. The best-performing meta-learner configuration was then further fine-tuned through hyperparameter optimization. To prevent data leakage and ensure robust meta-learner training, we employed k-fold cross-validation to generate level-1 predictions from the base classifiers. These predictions were then used as input features for the meta-classifier, creating a second layer that learns how to combine the base predictions into a final, more accurate output.
In parallel, voting ensembles were also explored. Both hard voting (majority class prediction) and soft voting (averaging predicted class probabilities) were implemented using the same five tuned models. Various weighting schemes were applied to adjust the influence of each model within the ensemble, and performance was again evaluated using F1-macro. While hard voting emphasizes model agreement, soft voting leverages model confidence, which can lead to improved results when individual models differ in certainty. These ensemble methods provided a robust foundation for comparison and final model selection.
The SE method differs from previous ensemble learning techniques in that it employs meta-learning to combine multiple machine learning algorithms. SE is used in a two-level structure where the level-1 meta learner combines the outputs of the level-0 base learners. The first stage involves training the base classifiers, namely K-Nearest Neighbors, Decision Trees, Random Forests, and XGBoost. The second stage involves aggregating the output detection (feature vector) outputs of the base classifiers to generate a new, reorganized training set. Finally, in the third stage, the Logistic Regression algorithm is utilized to train the meta-classifier using the new training set, resulting in the development of SE. Detailed descriptions of the developed machine learning algorithms are provided in Table A1 in Appendix A.
To assess the detection performance of the created classifiers, several metrics were employed, including the Receiver Operating Characteristics (ROC) curve, Area Under the ROC curve (AUC), Precision, Recall, Precision-Recall (PR) curve, F1-Score, and Detection Accuracy. These performance metrics are determined by counting the values for true positive (TP), false positive (FP), false negative (FN), and true negative (TN). Detailed descriptions of the evaluation metrics used are presented in Table 1.Table 1Description of performance metrics and their formulaMetricDescriptionPrecisionPrecision, or Positive Predictive Value (PPV), quantifies the proportion of predicted positive cases that are correctly identified as true positives. It is particularly relevant in scenarios where minimizing false positives is critical, offering insight into the model’s reliability in predicting the positive class.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Precision = {\text{ }}\frac{{TP}}{{TP + FP}}
\usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Recall{\text{ }}\left( {Sensitivity} \right) = {\text{ }}\frac{{TP}}{{TP + FN}} $$\end{document}F1-ScoreThe F1-Score is the harmonic mean of Precision and Recall, balancing both false positives and false negatives in a single metric. It is especially informative in the presence of class imbalance, where traditional accuracy can be misleading, and emphasizes the model’s performance on the minority (positive) class.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ F1 - Score = 2 \times {\text{ }}\frac{{Precision \times Recall}}{{Precision + Recall}} $$\end{document}Detection AccuracyAccuracy reflects the overall proportion of correctly classified instances across both positive and negative classes. While it provides a broad indication of model performance, it may mask poor sensitivity or precision in imbalanced datasets, thus requiring careful interpretation alongside other metrics.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Accuracy = {\text{ }}\frac{{TP + TN}}{{TP + TN + FP + FN}} $$\end{document}AU-ROCAU-ROC summarizes the diagnostic ability of a classifier by measuring the area under the curve plotting TPR against False Positive Rate (FPR) across various decision thresholds. It assumes equal cost of false positives and false negatives, which may not hold in medical applications. Nevertheless, AU-ROC is widely used to evaluate the trade-off between sensitivity and specificity.PRC-CurvePRC-Curve is a graphical representation that illustrates the trade-off between precision and recall for a classification model at different threshold settings. Precision measures the proportion of accurate positive predictions among all positive predictions, while recall measures the proportion of true positives identified among all actual positives. A higher area under the PRC indicates better model performance, reflecting both high precision and high recall. ### Model explanation and statistical analysis Explainable Artificial Intelligence (XAI) techniques help explain the decisions made by machine learning models so humans can understand them. Ensuring that clinical staff and end users trust a machine learning model’s decisions requires making its reasoning process clear and comprehensible [27]. To enhance the interpretability of the final models, we employed SHAP (SHapley Additive exPlanations) [28], a method rooted in cooperative game theory that quantifies each feature’s contribution to a specific prediction. SHAP assigns each feature a Shapley value, which reflects its average marginal impact across all possible combinations of feature subsets. This approach accounts for feature interactions and ensures fair, consistent attribution of influence, regardless of the order in which features are introduced into the model. By doing so, SHAP enables us to understand better why a model reaches a particular decision, offering insight beyond raw performance metrics. Interpretability is critical in clinical applications, where model outputs influence decisions with direct consequences for patient care. Transparent models foster trust among clinicians and support the responsible use of AI in healthcare settings. Moreover, by examining the relative influence and directionality of features across predictions and individual cases, SHAP provides insight into the underlying data patterns and can help identify new potential clinical markers. These interpretability tools not only validate the model’s reasoning but also promote further domain understanding and hypothesis generation. To assess whether the performance differences between models were statistically meaningful, we applied the paired sample t-test on F1-macro scores [29]. This test evaluates whether each related sets of performance scores, such as results obtained from different models evaluated on the cross-validation folds, exhibit a statistically significant difference in their mean values [29]. This method is appropriate for our study because all models were evaluated under identical conditions using 5-fold cross-validation, in which the dataset was split into five folds, and in each iteration, 382 patients were considered for the training set while the remaining patients were used for testing. For all experiments, random seed 42 was applied to ensure reproducibility across all methods. For each model comparison, the corresponding *p*-value was calculated, with a *p*-value below 0.05 considered indicative of a statistically meaningful difference. ## Results ### Overview of the experimental dataset and setup The dataset used in this study, derived from the ACQUIRE-ICD project, consists of 478 patient records and 39 features capturing a range of demographic, clinical, and patient-reported outcomes. These variables include both continuous and discrete numerical measures (e.g., age, BMI, anxiety scores) and binary categorical variables (e.g., gender, clinical indication). A detailed description of each feature is provided in Table 2.Table 2Summary of variablesVariable nameCountMeanStdMinMaxDescriptionage47859.611.5823.084.0Age specified in yearsImplWeight47889.6519.2910.0188.0Weight specified in kilosImplHeight478177.969.28100.0200.0Height specified in centimetersage_of_implant47859.611.5823.084.0Age when ICD was implantedBMI47828.275.9210.979.0Body Mass Index of the patientPCS_base47840.3410.5210.0167.88Aggregation of multiple questionnaires - Likert scaleMCS_base47848.9410.756.9868.89Aggregation of multiple questionnaires - Likert scalePCS_t636244.4410.676.6370.08Aggregation of multiple questionnaires - Likert scaleMCS_t636252.569.2715.9568.67Aggregation of multiple questionnaires - Likert scaleeq5d5l_index_baseline4780.820.19−0.081.0Quality of life index at baselineeq5d5l_index_t63610.880.18−0.331.0Quality of life index at follow-upsummary_score_kccq_base4780.30.240.00.92Patient-reported KCCQ score at baselinesummary_score_kccq_t64780.170.220.00.92Patient-reported KCCQ score at follow-upeq5d5l_vas2_den_dan_t047864.4621.162.0100.0VAS scale, T0eq5d5l_vas2_den_dan_t636174.5720.3110.0100.0VAS scale, T6depression_base_score4783.574.180.021.0PHQ-9 depression scoreanxiety_base_score4784.554.430.023.0GAD-7 anxiety scoreucla_base_sum4784.431.893.012.0Loneliness score (baseline)ucla_t6_sum3584.582.073.012.0Loneliness score (T6)icdc_base_sum4787.536.390.031.0Concerns about ICD (baseline)empower_icd_base_sum47842.717.1315.056.0ICD knowledge (baseline)empower_icd_t6_sum36544.299.790.056.0ICD knowledge (T6)postive_appraisal_base47816.312.934.020.0Positive appraisal (baseline)postive_appraisal_t647812.727.770.020.0Positive appraisal (T6)device_related_distress_base47810.064.075.024.0ICD distress score (baseline)Return_to_function_base47814.983.124.020.0Readiness to resume activity (baseline)Return_to_function_t647811.83.420.020.0Readiness to resume activity (T6)Body_image_concern_base4784.062.192.010.0Body image concern (baseline)Body_image_concern_t64782.662.380.010.0Body image concern (T6)bipq_sum35039.8313.128.076.0Illness perception scoredevice_related_distress_t64786.524.970.023.0ICD distress score (T6)icdc_t6_sum3693.984.940.032.0Concerns about ICD (T6) To assess data quality and distributional properties, boxplots of all numerical variables are shown in Fig. A1 in Appendix A. Many features display asymmetric distributions and contain extreme values. Nevertheless, these may not necessarily be outliers, as several variables are constructed from summed Likert-scale questionnaire responses, which inherently yield skewed distributions with valid high or low extremes. For example, ucla_t6_sum and device_related_distress_base show extended whiskers consistent with skewed self-reported outcomes. Histograms in Fig. A2 in Appendix A confirm that most variables deviate from normality, with the majority showing right- or left-skew. Features such as anxiety_base_score and depression_base_score are notably right-skewed, reflecting low symptom burden for most participants, but with a minority scoring high. This non-normality is further validated using Quantile-Quantile plots as shown in Fig. A3 in Appendix A, where systematic deviation from the diagonal reference line, especially in the tails, suggests that many variables exhibit heavy-tailed distributions. Notably, only ImplHeight approximately follows a normal distribution, whereas most patient-reported features exhibit curved or segmented QQ patterns. Taken together, these findings underscore the importance of applying data transformations, appropriate imputation techniques, and robust feature selection methods to address non-normality, potential outliers, and data sparsity prior to model development. Eight of 39 variables in the ACQUIRE-ICD dataset contain missing values, primarily among PROMs collected at the six-month follow-up (T6), including MCS_t6, eq5d5l_index_t6, and ucla_t6_sum, with missing counts ranging from 109 to 128 per variable (Fig. A4 in Appendix A). The consistent timing of these missing values suggests a follow-up nonresponse pattern, while the exception bipq_sum indicates an additional mechanism unrelated to T6 data collection. This distribution implies a combination of missing completely at random (MCAR) and missing at random (MAR) mechanisms, warranting the use of multiple imputation strategies to retain data and minimize bias. In addition, both target variables are the anxiety dataset includes 35 positive cases and 443 negative cases (7.3% positive), while the distress dataset includes 39 positive and 439 negative cases (8.2% positive), as shown in Fig. A5 in Appendix A. These ~ 9:1 class ratio pose a risk of model bias toward the negative class, potentially reducing sensitivity to detecting at-risk patients. These findings highlight the need for comprehensive preprocessing, including imputation, scaling, feature selection, and class balancing, to address non-normality, outliers, missing data, and imbalanced class distributions prior to model development. ### Baseline models To establish a baseline for model comparison, we evaluated the performance of ten standard classifiers trained on the unprocessed ACQUIRE-ICD dataset, without applying imputation, feature selection, or class balancing. This step provides a reference point for assessing the incremental benefit of subsequent preprocessing. Because some classifiers, including SVM, KNN, MLP, NB, and LGR, cannot natively handle missing values, rows with missing values were removed from those models. In contrast, models capable of handling missing values (e.g., RF, XGB, LGBM) were trained on the complete dataset. Also, all results for these models were obtained using 5-fold cross-validation, with F1-macro as the primary metric and its standard deviation. Although imputation enabled the inclusion of all samples, it likely reduced performance in specific models by smoothing feature distributions, potentially obscuring subtle predictive patterns essential for accurate classification, particularly in small or heterogeneous datasets. The baseline performance across classifiers is presented in Table A2 in Appendix A. For the anxiety prediction task, RF and KNN achieved the highest F1-scores (0.84), followed closely by MLP (0.83) and XGB (0.81). SVM, in contrast, yielded the lowest F1-score (0.47), with both precision and recall below 0.5. For the distress detection, the best-performing classifiers were LGR (F1 = 0.86), LGBM (0.83), and MLP (0.80), while SVM again lagged significantly (F1 = 0.47). Overall, RF performed consistently well across both tasks, with high accuracy, precision, and recall. Despite low performance from models such as SVM and NB on unprocessed data, all classifiers were retained for further evaluation after preprocessing. This decision allows us to explore whether appropriate data handling, such as imputation and feature engineering, can significantly improve underperforming models. In sum, these findings emphasize two key first, tree-based ensemble models (RF, LGBM, XGB) consistently outperform simpler classifiers on raw data; and second, failure to handle missing values substantially degrades model performance, underscoring the importance of preprocessing in real-world clinical datasets. ### Preprocessing results #### Scaling Although scaling is often considered a routine step, its effects were examined in detail, given the wide range of feature magnitudes. Three scalers were MinMaxScaler, StandardScaler, and RobustScaler. The results revealed that the impact of scaling varied significantly across models. For most models, MinMaxScaler yielded the best performance improvements. However, in the final anxiety pipeline, RobustScaler was chosen due to a marginal gain in F1-macro compared to MinMaxScaler. The influence of scaling was most notable for models sensitive to input distributions, such as SVM, MLP, and KNN, and less important for tree-based models, which are invariant to feature scaling. Detailed results are presented in Table A3 in Appendix 1. #### Imputation To address missing values identified in the ACQUIRE-ICD datasets, four imputation strategies were Simple Imputer, KNN Imputer, Iterative Imputer, and Iterative Imputer with Random Forest Regressor. Each method was applied to the candidate models, and performance was measured using stratified 5-fold cross-validation and the F1-macro score. Results are presented in Table A4 in Appendix A. For the anxiety dataset, the Simple Imputer yielded the best performance across most models, including RF, XGB, LGBM, MLP, KNN, DCT, SVM, and NB, while the LGR Imputer performed slightly better with KNN. In some cases (e.g., RF and LGBM), the baseline score without any imputation still outperformed imputed versions, suggesting that these models are remarkably robust to missing data. For the distress dataset, the results were more varied. The Simple Imputer again outperformed others for most classifiers, including RF, DCT, LGBM, SVM, MLP, and NB. However, XGB and LGR performed best with the KNN Imputer, and the KNN classifier achieved its highest F1-macro using the more complex Iterative Imputer with Random Forest. Overall, the Simple Imputer was the most consistently effective strategy, offering a good balance between performance and computational simplicity across both tasks. #### Feature selection Feature selection was carried out using an ensemble of Mutual Information, Chi-squared, Fisher Score, RFECV, and FGS. These were combined into both homogeneous and heterogeneous ensembles to construct nine distinct feature subsets for evaluation. Detailed results are shown in Table A5 in Appendix A. For the anxiety dataset, FGS was the most effective method for five RF, SVM, KNN, XGB, and LGBM. The selected features included *ucla t6 sum*, s*ummary score kccq base, eq5d5l index t6, MCS t6, ImplWeight*, and *personality type D*. Other models performed best with individual methods like Fisher Score, RFECV, or Mutual Information. In the distress dataset, feature selection patterns differed. While FGS remained effective for some models, others (e.g., SVM and LGBM) showed improved performance with alternative methods. The optimal feature subset for distress included *ucla t6 sum, eq5d5l index t6, depression base score, icdc t6 sum, MCS t6, return to function t6, age*, and *anxiety base score*. Compared to the anxiety task, models generally performed better with more features. Across both datasets, FGS emerged as the most frequently selected method, confirming its ability to construct compact and informative feature subsets tailored to specific models and targets. #### Data balancing Four class-balancing techniques were evaluated to address the imbalance between the positive and negative Random Undersampling, Random Oversampling, SMOTEENN, and SMOTETomek. Although these techniques are theoretically helpful for improving minority-class representation, their application in our datasets generally led to a decline in model performance. Full comparative results are provided in Table A6 in Appendix A. Across most models, the F1-macro scores decreased after applying the balancing methods, suggesting that the resampling procedures may have introduced synthetic noise or contributed to overfitting, an effect particularly plausible given the relatively small number of positive-class samples. Only DCT, LGR, and LGBM showed minor performance improvements in isolated cases, but these gains were not consistent enough to justify integrating balancing into the final preprocessing workflow. Given these findings, all four balancing methods were excluded from the final modeling pipelines. The optimal preprocessing configurations instead relied on scaling, imputation, and feature selection, which collectively yielded more stable and reliable performance. ### Model development #### Best individual pipeline Each candidate model was evaluated with all possible combinations of imputation, feature selection, and balancing. The best-performing pipeline per model was identified for both datasets and is presented in detail in Table A6 in Appendix A. For each model, F1-macro scores were recorded for the baseline, post-imputation, post-feature selection, and post-balancing. For anxiety, the RF classifier again emerged as the best model, achieving an F1-macro score of 0.91 with a pipeline that included SimpleImputer, RobustScaler, and FGS, using six selected features. For distress, RF also achieved the highest F1-macro score of 0.91, but only with MinMaxScaler and eight features. In both cases, performance dropped slightly when balancing methods were applied. Other models, such as SVM, KNN, MLP, and XGB, also performed competitively, with modest differences in preprocessing preferences. The best-performing pipeline for each task was selected as the reference configuration for all models, as shown in Table A7 in Appendix A. To ensure a fair and standardized comparison between candidate models, the best-performing preprocessing pipeline, as shown in Table A7 in Appendix A, was applied uniformly across all models. This approach enabled evaluation of model performance under optimized, consistent data conditions, removing variation introduced by different preprocessing configurations. The results of applying the best pipeline to all 10 models are shown in Table A8 in Appendix A. For anxiety, three models achieved an F1-macro score above 0.90, including RF (0.91), SVM (0.90), and KNN (0.90). Two additional models, XGB and LGBM, followed closely with scores above 0.89. For distress, the top-performing model was again RF 0.91. While only one model exceeded the 0.90 threshold, five other models, including SVM, MLP, XGB, KNN, and LGBM, achieved F1-macro scores above 0.86. This suggests a slightly more challenging classification task for the distressed label compared to anxiety. Overall, applying the best pipeline improved performance and reduced variation across folds. It also provided a robust basis for selecting top-performing models for ensemble development. #### Hyperparameter Tuning To enhance classification performance and reduce model variance, hyperparameter tuning was conducted for the top five models in each dataset using a grid search with stratified 5-fold cross-validation. The F1-macro score served as the optimization criterion. Table 3 presents the F1-macro scores before and after tuning for each model.Table 3Impact of hyperparameter tuning on individual modelsModelF1-macroStdF1-macroStdF1-macroStdF1-macroStdDefaultRandomSearchGridSearchBayesian optimization**Anxiety**RF0.910.070.920.050.920.050.920.07KNN0.900.060.910.080.910.080.910.08SVM0.900.080.910.080.910.080.910.08XGB0.890.080.910.080.920.090.910.09LGBM0.890.070.910.070.910.070.900.07**Distressed**RF0.910.080.920.080.920.080.910.05KNN0.880.080.870.080.890.080.890.08SVM0.860.100.890.110.890.110.890.11XGB0.870.050.900.070.900.070.890.07LGBM0.880.03**0.92****0.05****0.93****0.05**0.910.03 Across both the anxiety and distress datasets, tuning consistently improved predictive performance. However, its effect on model stability (as measured by fold standard deviation) was mixed. For anxiety, tuning reduced variance for RF and SVM but slightly increased it for KNN, XGB, and LGBM. In contrast, tuning led to greater variability for most distress models, except for KNN, which showed a marginal reduction. Model-specific tuning strategies included adjustments to tree depth and number of estimators for RF, neighborhood size and distance metrics for KNN, kernel and regularization parameters for SVM, and learning rate, estimators, and boosting parameters for XGB and LGBM. Full details of the selected parameter configurations are presented in Table A9 in Appendix A. Final tuned configurations were retained and served as the foundation for subsequent ensemble model development. #### Ensemble modeling Following the identification of the top-performing models from the preprocessing and tuning phases, ensemble methods were explored to further enhance classification performance for the detection of anxiety and distress. Two ensemble strategies were stacking and voting, utilizing the five best-performing models for each dataset. Table A10 in Appendix A shows the details of the development of the stacking classifier for the detection of anxiety. The meta-learner SVM demonstrated the highest F1-macro score among all meta-learners, achieving optimal performance when combined with four or five base models. In contrast, this pattern was not observed in the distress dataset, where the best SVM-based stacking performance was achieved with only two base learners, KNN and LGBM. For the anxiety dataset, the stacking classifier consistently outperformed the voting ensemble. As shown in Table A10 in Appendix A, the stacking model (RF + SVM + KNN + XGB) achieved an F1-macro score of 0.93, surpassing the best voting ensemble (0.92). Conversely, for the distress dataset, both soft and hard voting classifiers reached an F1-macro score of 0.92, quite similar to the stacking classifier’s peak score of 0.91. Notably, even the best ensemble performance for distressed did not exceed the score previously achieved by the standalone LGBM classifier. These findings demonstrate that ensemble configurations need to be tailored to each dataset. While stacking was more effective for detecting anxiety, voting achieved stronger performance on the distress dataset, as shown in Table A11 in Appendix A. To further refine ensemble performance, the meta-learners used in the stacking classifiers were tuned. However, no improvements in F1-macro score were observed beyond the default configurations. Consequently, the final stacking classifiers employed default parameters for the SVM meta-learners. ### Best selected model result #### Best selected model overview The final models for both anxiety and distress present detection were selected based on overall performance across cross-validation folds, with an emphasis on generalizability and class balance. For the anxiety dataset, the best model was the stacking classifier, achieving an F1-macro score of 0.93 with a standard deviation of 0.072, as shown in Table A10 in Appendix A. For the distress dataset, the LGBM classifier delivered the strongest performance, with an F1-macro score of 0.93 and a standard deviation of 0.05 (see Table 3). Details regarding the preprocessing pipeline for each final model, including imputation, scaling, and FS for the detection of anxiety and distressed, are summarized in Table 4. These models were chosen not only for their high accuracy but also for their consistency across folds, supporting their potential clinical utility.Table 4The best selected model for anxiety and distressed detectionAnxiety Detection**Model****Base-learners****Meta-learner****Scaler****Imputer****Feature selection method****Selected features****F1-macro/std**Stacking ClassifierRF + SVM + KNN + XGBSVMRobustScalerSimpleImputerForward Greedy Searchucla t6 sum, summary score kccq base, eq5d5l index t6, MCS t6, ImplWeight, personality type D0.93/0.07**Distressed Detection****Model****Scaler****Imputer****Feature selection method****Selected features****F1-macro/std****Model****Scaler**LGBMMinMaxScalerSimpleImputerForward Greedy Searchucla t6 sum, eq5d5l index t6, depression base score, icdc t6 sum, MCS t6, Return to function t6, age, anxiety base score0.93/0.05LGBMMinMaxScaler According to the obtained results of the paired sample t-test, the proposed approach successfully identified statistically meaningful best-performing model configurations for both target outcomes. For anxiety prediction, the ensemble stacking model based on the stacking classifier with the SVM-metalearner achieved the most significant improvement, supported by a *p*-value of 0.00086, confirming meaningful superiority over alternative configurations. For distressed detection, the best-performing configuration was identified as the LGBM model optimized with Bayesian optimization, which also achieved leading predictive performance. The t-test analysis yielded a *p*-value of 0.0412, indicating that its advantage is statistically meaningful compared to the other evaluated models. These statistical results validate that the selected configurations are not only empirically strong but also supported by inferential statistical evidence. The detailed *p*-values of all models are presented in Table A12 and Table A13, respectively, in Appendix A. #### Classifier performance by class To evaluate performance beyond global metrics, per-class results were calculated using stratified 5-fold cross-validation. These metrics are presented in Table 5. For the anxiety dataset, the stacking classifier showed stronger performance for the negative class, with a relatively high standard deviation for the positive class (recall: SD = 0.19 vs. 0.004 for the negative class). For the distress dataset, the LGBM model showed more balanced class performance and lower fold-to-fold variability.Table 5Performance metrics for the selected best models for detecting anxiety and distressed**Anxiety**5-fold stratified CVStacking ClassifierRecallPrecisionF1ROC-AUCPRC-AUCAccuracyClassMSMSMSMSMSMSNo-Anxiety0.990.0040.980.010.990.010.890.170.850.190.970.01Anxiety0.80.190.970.060.860.14macro0.90.090.980.030.930.07**Anxiety**5-fold stratified CVRandom Forest ClassifierRecallPrecisionF1ROC-AUCPRC-AUCAccuracyClassMSMSMSMSMSMSNo-Anxiety0.990.0040.990.010.990.010.970.040.890.090.980.01Anxiety0.890.170.840.040.850.09macro0.940.080.910.020.920.05**Distressed**5-fold stratified CVLGBM ClassifierRecallPrecisionF1ROC-AUCPRC-AUCAccuracyClassMSMSMSMSMSMSNo-Distressed0.990.010.990.010.990.010.980.020.860.110.980.01Distressed0.840.10.890.10.870.1macro0.920.050.940.050.930.05The abbreviation” M” stands for the mean value of the 5 folds, and” S” is the standard deviation. This table also reports the Recall, Precision, and F1 score for both classes (0 and 1), as well as a macro score. ROC-AUC and PRC-AUC are also added for each classifier Given the imbalance in class-specific performance of the stacking classifier on the anxiety dataset, an additional analysis was conducted using the second-best model, RF, which had a slightly lower F1-macro but better recall for the positive class and lower standard deviation. The RF classifier’s ROC-AUC and PRC-AUC scores were 0.98 and 0.89, respectively, both of which outperformed the stacking classifier. These results, summarized in Table 5, provide evidence that simpler models may offer advantages in recall and interpretability, particularly for underrepresented classes. #### Learning curve analysis Learning curves were analyzed to assess model behavior as a function of training set size. As shown in Fig. 2, the stacking classifier on the anxiety dataset exhibited a consistent upward trend in cross-validation scores as the sample size increased, suggesting potential for further performance improvement with additional data. The training and validation curves converged gradually, indicating good generalization and limited overfitting.Fig. 2Learning curves for the final models For the RF classifier, the curve showed steady performance gains until a slight drop toward the end, possibly due to variance in the final fold. The LGBM model for the distress dataset exhibited a sharp performance increase in the final segments of the learning curve, further supporting the benefit of increased sample sizes. Across all classifiers, stacking exhibited the narrowest gap between training and validation curves, reinforcing its robustness. #### Precision-recall curves PRC was used to visualize the trade-off between precision and recall across thresholds. As shown in Fig. 3, the stacking classifier for anxiety exhibited high initial precision that dropped sharply as recall approached 80%, indicating conservative predictions with few false positives. The RF classifier presented a more balanced trade-off, with higher recall but lower precision at high recall thresholds. The LGBM classifier for the distress dataset showed a less pronounced drop in precision and improved recall in the 60–83% range. PRC-AUC scores for the classifiers were 88.59% (RF), 85.73% (stacking), and 86.27% (LGBM), confirming their strong performance in imbalanced settings and offering nuanced insights for threshold optimization in clinical deployment.Fig. 3PRC-curves for the final models ### XAI to explain detection model decisions To enhance the transparency and clinical trustworthiness of our predictive models, we applied SHAP, a model-agnostic method rooted in cooperative game theory. SHAP assigns each feature an important value for individual predictions, thereby enabling both global and local interpretability of model behavior. We focused our SHAP analysis on the two best-performing the stacking classifier for anxiety detection and the LGBM for distressed detection. Since SHAP requires a fixed dataset, we selected one of the cross-validation folds to generate explanations, avoiding data leakage. #### Global model explanations Global SHAP summary plots (Fig. 4) revealed the most influential features driving model predictions. For anxiety detection using the stacking classifier, the top contributors were the EQ-5D-5 L index, Mental Component Summary (MCS) score, and the UCLA loneliness scale. Features like the KCCQ summary score, implant weight, and personality type D contributed less prominently. The corresponding Beeswarm plot (Fig. 5) illustrated how feature values affect SHAP values in the detection of low EQ-5D-5 L and MCS values were associated with higher predicted risk, whereas higher loneliness scores increased the likelihood of anxiety classification. For distressed detection via the LGBM classifier, MCS and EQ-5D-5 L again emerged as dominant features, followed by anxiety and depression baseline scores, age, and functional indices, as shown in Fig. 4. The Beeswarm plot shown in Fig. 5 confirmed that low MCS and EQ-5D-5 L scores elevated the probability of distressed detection, with notable variation across other features.Fig. 4SHAP summary 1) The Stacking model on anxiety detection, 2) The LGBM model on distressed detectionFig. 5SHAP beeswarm 1) The Stacking model on anxiety detection, 2) The LGBM model on distressed detection #### Local model explanations To further explore model decision boundaries, SHAP force plots were generated for individual predictions. These visualizations explain the direction and magnitude of each feature’s contribution to the final output. For anxiety detection, examples are shown as true positives and true negatives in Fig. 6. Similarly, for distressed detection, local force plots in Fig. 7 illustrate both true-positive and true-negative samples across different probability thresholds. SHAP force plots illustrate individual feature contributions for each prediction. In true positive cases, higher MCS and EQ-5D-5 L scores strongly predicted anxiety or distressed, whereas in true negative cases, the absence of these key risk factors drove correct predictions. This detailed interpretation enhances clinical understanding of the model’s decision-making process.Fig. 6Force-plots with two true positive and two true negative detections from the stacking classifier on the anxietyFig. 7Force-plots with two true positive and two true negative detections from the LGBM classifier on the distressed ## Discussion ### Summary of findings This study developed and evaluated multiple ML models to detect symptoms of anxiety and psychological distress among cardiac patients with ICDs using patient-reported outcomes from the ACQUIRE-ICD dataset. The highest-performing models achieved F1-macro scores of 0.93 for anxiety detection with a stacking ensemble model (RF + SVM + KNN + XGB) and 0.93 for distressed detection with an LGBM model. These results highlight the effectiveness of ensemble learning in capturing the complex patterns embedded in multidimensional psychosocial and health-related variables. Among the three stacking configurations, the RF + SVM + KNN + XGB combination was selected for downstream analysis due to its computational efficiency and interpretability while maintaining comparable predictive performance for anxiety detection. Furthermore, the LGBM model’s superior performance is likely due to its ability to capture complex feature interactions and its built-in regularization mechanisms, which reduce overfitting in distressed detection. Nevertheless, stacking ensembles may introduce additional variance when combining base learners of varying strengths, resulting in slightly lower predictive performance. The statistical outcomes obtained from the paired sample t-test reinforce the robustness of our modeling framework. In addition to developing models with optimized hyperparameters, incorporating 5-fold cross-validation, applying feature selection strategies, and evaluating multiple ensemble and individual algorithms, the significance analysis further confirms the reliability of the selected best models. For both anxiety and distressed prediction tasks, the *p*-values associated with the top-performing models were below the 0.05 threshold, indicating that the improvements achieved by our best models are statistically meaningful rather than the result of random variation. Specifically, the anxiety prediction task identified the best stacking ensemble model with the SVM-metalearner as statistically superior, while the distressed prediction task identified the Bayesian-optimized LGBM model as a significantly stronger configuration. These findings demonstrate that the proposed system not only attains high predictive accuracy but also provides reproducible and statistically confirmed improvements in performance. This study advances prior ICD-related machine learning research by integrating structured PROMs with explainable ensemble modeling, thereby offering a transparent, clinically interpretable framework for detecting psychological distress. Unlike earlier investigations that focused mainly on unstructured data, such as free-text clinical notes or wearable sensor data [21, 27], the present work leverages standardized PROM-based assessments that can be directly incorporated into existing clinical workflows. The inclusion of SHAP-based interpretability further enhances the clinical value of the proposed framework by clarifying how individual features, such as depression, loneliness, or mental health composite scores, contribute to the predicted outcomes of anxiety and distressed. These findings demonstrate that structured PROM data, when analyzed through ensemble learning, can achieve comparable or even superior accuracy while preserving interpretability, an advantage that deep learning approaches often lack. Beyond methodological innovation, the proposed framework has translational potential for integration into clinical decision support systems and remote monitoring infrastructures, thereby improving the scalability and timeliness of psychological assessment for ICD patients. Importantly, the predictive features identified through feature selection and explainability analyses reflect clinically relevant constructs. Variables such as the MCS t6 from the SF, EQ-5D-5 L index scores, and UCLA loneliness scores emerged consistently among the most influential features, aligning with known predictors of mental health outcomes in cardiology patients. Together, these insights not only reinforce the clinical interpretability of the proposed models but also underscore the novelty of integrating structured PROM data with explainable ensemble learning to enhance psychosocial care in cardiac populations. These findings suggest that routinely collected PROMs can be leveraged effectively to detect psychological comorbidities early in cardiac populations using ML-based decision-support tools. Notably, the proposed ML framework could be integrated into remote patient monitoring workflows, enabling automated assessment of psychological distressed and anxiety in ICD patients. Routinely collected PROMs could generate real-time risk scores, alerting clinicians to patients who require timely psychosocial interventions. This integration may enhance clinical efficiency, enabling healthcare providers to prioritize high-risk patients, optimize resource allocation, and support personalized care strategies. ### Preprocessing impact The preprocessing pipeline substantially improved model performance across both classification tasks. In particular, the combination of scaling, imputation, and feature selection substantially improved F1-macro scores relative to baseline models trained on raw data. Among the evaluated scaling techniques, the MinMaxScaler and RobustScaler consistently outperformed others, depending on the distribution of input features and the specific model architecture. SimpleImputer emerged as the most effective imputation method, yielding the highest performance across most classifiers. This may be attributed to the relatively structured nature of the missing data and the limited dataset size, where complex imputation techniques such as iterative regression and KNN imputation did not provide additional benefits and, in some cases, introduced variability. Feature selection had the most significant impact on performance, with the FGS algorithm outperforming other techniques across both anxiety and distressed detection tasks. By reducing dimensionality and improving the signal-to-noise ratio, this method improved generalization and reduced overfitting. The most consistently selected features aligned with psychological and functional health domains, reinforcing their relevance for clinical risk profiling; nonetheless, the FGS algorithm is not a clinical decision rule. FGS was chosen after a systematic comparison with other selection techniques because it consistently produced feature subsets that optimized predictive performance in cross-validation. In contrast, applying class-balancing methods, including Random Oversampling, Random Undersampling, SMOTEENN, and SMOTETomek, did not yield consistent performance improvements. On the contrary, their use often led to a slight degradation of model performance, particularly for tree-based classifiers such as RF, XGB, and LGBM. This effect is likely attributable to the small number of minority-class samples in the ACQUIRE-ICD dataset, which limits the diversity of synthetic samples generated through oversampling. Moreover, borderline or noisy instances in the minority class may have been amplified, introducing additional variance and reducing the models’ generalizability. Similar observations have been reported in clinical machine learning studies, where oversampling can occasionally increase overfitting or distort the underlying feature distribution [25]. These findings highlight that, while balancing techniques are theoretically useful, their application must be carefully considered in small, highly imbalanced clinical datasets. Consequently, our final pipelines relied on feature selection, scaling, and imputation without additional class-balancing steps, yielding more stable and reliable performance while preserving clinically meaningful patterns. ### Model optimization and evaluation Hyperparameter tuning substantially improved the classification performance of all top candidate models. Using a combination of Randomized Search followed by Grid Search or Bayesian Optimization, each model’s configuration was systematically adjusted to maximize the F1-macro score while mitigating overfitting. This approach enabled the models to capture underlying patterns in the data better and contributed to overall performance gains in both the anxiety and distressed detection tasks. The tuning process was particularly effective for the RF and SVM classifiers in the anxiety detection task, where both models exhibited reduced standard deviation across validation folds following optimization, indicating improved model stability. In contrast, tuning led to increased variability in cross-validation scores for models such as KNN, XGB, and LGBM, highlighting a trade-off between performance and consistency in specific classifiers. For the distressed detection, the standard deviation increased for most tuned models, except KNN, which showed a slight improvement in stability. These results suggest that while tuning enhances predictive accuracy, it may also increase models’ sensitivity to data variation, particularly in small, imbalanced datasets. Nonetheless, all tuned models outperformed their default counterparts, justifying their inclusion in subsequent ensemble strategies. The optimized hyperparameters, ranging from the number of estimators and tree depth in ensemble methods to kernel and regularization settings in SVM, formed the foundation for the final stacking and voting classifiers. This refined model architecture enabled the construction of robust ensembles that outperformed individual models in terms of generalizability and overall classification accuracy. ### Explainability and clinical relevance A key strength of this study is the incorporation of XAI techniques, specifically SHAP, to enhance the interpretability of model predictions. In clinical settings, where transparency and trust are paramount, understanding how individual features contribute to a prediction is critical for clinical uptake. The SHAP analyses consistently highlighted clinically meaningful features such as the MCS t6, UCLA loneliness scores, EQ-5D-5 L index values, and baseline anxiety and depression scores as key drivers of model predictions, supporting their interpretability and clinical relevance [30, 31]. Notably, some of the highest-ranking features were patient-reported constructs not routinely used in clinical decision-making, suggesting that machine learning can uncover underutilized but informative markers of psychological burden. Furthermore, the features identified by FGS, which we then examined with SHAP for interpretability, including the MCS score, loneliness indices, and health-related quality-of-life measures, are strongly supported by clinical research. For example, a study in cardiac patients demonstrated that lower SF MCS scores reliably indicate depressive or anxiety symptoms [32, 33]. Similarly, loneliness has been longitudinally associated with poorer mental health–related quality of life in patients with coronary heart disease [34, 35]. These findings show substantial concordance between the variables selected by FGS and clinically validated predictors, increasing our confidence that FGS not only optimizes model performance but also identifies medically meaningful constructs. We then used SHAP-based explainability to contextualize how these features influence model predictions, ensuring that our models remain both high-performing and clinically transparent. From a practical standpoint, the ability to visualize feature contributions for individual patients supports personalized care strategies. For instance, a clinician can explore which domains, such as social support, perceived control, or symptom burden, are most strongly associated with a patient’s elevated risk of anxiety or distressed. This level of insight may guide more targeted psychosocial interventions, supporting the movement toward precision mental health in cardiology. Moreover, the SHAP findings lend credibility to the machine learning models by reinforcing the face validity of the predictions. The alignment between SHAP-identified important features and known psychological constructs enhances trust in the model’s outputs and strengthens the case for future clinical implementation. ### Comparison to related work Few studies have applied machine learning to predict psychological outcomes in patients with ICD, and even fewer have used structured patient-reported outcomes. Existing research in mental health classification has often relied on self-administered tools such as the HADS, GAD-7, or clinician-assessed psychiatric diagnoses. Compared to these studies, the present work distinguishes itself by integrating a broader set of patient-reported outcomes capturing physical, mental, and psychosocial domains and systematically evaluating model performance across a comprehensive suite of preprocessing and modeling strategies. Our findings demonstrate significant improvements in predicting psychological distress among ICD patients compared to existing research. Previous studies employing machine learning for mental health detection have often reported moderate predictive performance, typically achieving F1-scores between 0.70 and 0.85, even with relatively balanced datasets. For instance, Priya, et al. [16] achieved an F1-score of 0.836 for depression prediction but only moderate performance for anxiety (F1-score: 0.527), while Nemesure, et al. [18] reported an AUC of 0.67 for depression and anxiety detection using electronic health records. Andersen, et al. [20], who specifically investigated psychological distress in ICD patients, achieved a precision of 80% and a recall of 67%, leading to an F1-score of 73%. In comparison, our optimized ensemble models yielded substantially higher F1-macro scores of 0.93 for both anxiety and distressed detection, demonstrating improved sensitivity and specificity even under conditions of significant class imbalance. Our approach differed notably from prior studies by systematically integrating a broader range of patient-reported outcomes and by employing rigorous preprocessing strategies, including feature selection and advanced imputation methods, which effectively addressed issues such as missing data and feature redundancy. Additionally, the use of SHAP for model explanation provided a deeper level of clinical interpretability than typically found in related work, facilitating trust and potential clinical integration. Thus, this study significantly extends the state of the art by providing not only superior predictive accuracy but also enhanced clinical relevance and interpretability to support decision-making in cardiology care pathways. ### Limitations and future work Despite promising results, several limitations should be considered. The dataset was relatively small (N = 478) and highly imbalanced (~9–10% positive cases), limiting model exposure to rare outcomes. Resampling techniques were explored but excluded due to reduced performance, and this imbalance may affect generalizability to broader populations. The data were derived from a single cohort of patients with ICDs, limiting their applicability to other clinical contexts. External validation on larger, more diverse datasets is needed to confirm robustness. Currently, our study utilized the ACQUIRE-ICD dataset from Danish centers; in future work, we plan to evaluate the models on additional datasets from other regions of Denmark and, if feasible, on datasets from other countries. This approach will allow assessment of geographic and demographic biases and enable recalibration of the models for broader applicability. Furthermore, while the SHAP-enhanced model improves transparency, it does not fully address potential biases from unmeasured or underrepresented demographic factors. Future work should examine fairness-aware modeling approaches and incorporate external validation as a critical step for translational deployment. While developed models in this study could be considered a remote monitoring integration system that offers practical benefits, ethical considerations must be carefully addressed, including patient data privacy and ensuring clinician oversight in decision-making. Future studies should explore embedding these models into electronic health records and telemedicine platforms and evaluate real-world workflows to ensure the responsible and effective deployment of AI-assisted screening for psychological distress in ICD patients. The cross-sectional design precludes causal interpretation, and longitudinal studies are needed to understand how psychological symptoms evolve. Expanding models with multimodal data, such as clinical history and device metrics, could improve predictive power and support individualized care. Finally, for clinical implementation, models must be integrated into electronic health records and evaluated in real-world settings to assess their practical utility in early detection and intervention. ## Conclusion In this study, we successfully developed and evaluated advanced machine learning models to accurately detect anxiety and psychological distress among cardiac patients with ICDs using structured patient-reported outcomes from the ACQUIRE-ICD dataset. The optimized ensemble models, most notably the stacking classifier for anxiety and the Bayesian-optimized LightGBM for distressed, demonstrated robust predictive performance, each achieving an F1-macro score of 0.93 despite severe class imbalance. The incorporation of comprehensive preprocessing procedures, including feature selection, scaling, and imputation, further contributed to improved model stability and generalization. Notably, the statistical validation performed using a t-test on F1-macro scores confirmed that the improvements obtained by the best-performing models were statistically meaningful (p < 0.05) when compared with alternative configurations. These findings highlight that the predictive gains are not attributable to random variation but reflect the actual superiority of the selected modeling strategies. Explainability analysis using SHAP revealed clinically relevant predictors such as mental health composite scores, loneliness measures, and health-related quality-of-life indices, thereby enhancing model transparency and supporting clinical confidence. Together, these contributions underscore the potential for scalable, data-driven decision support systems capable of enabling early detection and targeted psychosocial interventions for ICD patients. Future research should focus on external validation using multi-center cohorts, prospective deployment in real-world clinical workflows, and integration with digital health platforms to confirm the robustness, usability, and practical utility of the proposed framework. Such advancements may facilitate more timely mental-health assessment, improve patient outcomes, and enhance personalized care strategies within cardiac populations. ## Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1