Authors: Amir Sorayaie Azar, Tahereh Samimi, Ghanbar Tavassoli, Amin Naemi, Bahlol Rahimi, Zahra Hadianfard, Uffe Kock Wiil, Surena Nazarbaghi, Jamshid Bagherzadeh Mohasefi, Hadi Lotfnezhad Afshar
Categories: Research, Interpretable machine learning, Machine learning, Prediction, Stroke severity
Source: European Journal of Medical Research
Stroke is a significant global health concern, ranking as the second leading cause of death and placing a substantial financial burden on healthcare systems, particularly in low- and middle-income countries. Timely evaluation of stroke severity is crucial for predicting clinical outcomes, with standard assessment tools being the Rapid Arterial Occlusion Evaluation (RACE) and the National Institutes of Health Stroke Scale (NIHSS). This study aims to utilize Machine Learning (ML) algorithms to predict stroke severity using these two distinct scales.
We conducted this study using two datasets collected from hospitals in Urmia, Iran, corresponding to stroke severity assessments based on RACE and NIHSS. Seven ML algorithms were applied, including K-Nearest Neighbor (KNN), Decision Tree (DT), Random Forest (RF), Adaptive Boosting (AdaBoost), Extreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), and Artificial Neural Network (ANN). Hyperparameter tuning was performed using grid search to optimize model performance, and SHapley Additive Explanations (SHAP) were used to interpret the contribution of individual features.
Among the models, the RF achieved the highest performance, with accuracies of 92.68% for the RACE dataset and 91.19% for the NIHSS dataset. The Area Under the Curve (AUC) was 92.02% and 97.86% for the RACE and NIHSS datasets, respectively. The SHAP analysis identified triglyceride levels, length of hospital stay, and age as critical predictors of stroke severity.
This study is the first to apply ML models to the RACE and NIHSS scales for predicting stroke severity. The use of SHAP enhances the interpretability of the models, increasing clinicians’ trust in these ML algorithms. The best-performing ML model can be a valuable tool for assisting medical professionals in predicting stroke severity in clinical settings.
The online version contains supplementary material available at 10.1186/s40001-024-02147-1.
Keywords: Stroke severity, Prediction, Machine learning, Interpretable machine learning
Stroke, or cerebrovascular accident (CVA), occurs in two primary ischemic stroke, caused by blocked blood flow, and hemorrhagic stroke, due to blood vessel rupture [1–3]. Ischemic strokes disrupt oxygen delivery, leading to cerebral infarction and neuronal loss [4]. Timely diagnosis and treatment are critical to restoring blood flow and enabling nerve recovery [5]. Delays can result in irreversible damage [4]. Hemorrhagic strokes pose significant mortality risks, with rates of 10–20% in developed countries and up to 50% in developing nations [6].
Stroke is a considerable health concern [7, 8], standing as the second most prevalent cause of mortality and the third most prevalent cause of both mortality and disability [9]. Moreover, it places considerable financial strains, demanding costly, protracted, and intricate medical interventions [10]. Consequently, there is a pressing need for swift and effective preventive measures, with a particular emphasis on addressing the primary risk factors [11].
A systematic review has highlighted the top ten risk factors associated with stroke, including high systolic blood pressure, elevated body mass index, increased fasting glucose levels, exposure to particulate matter pollution, smoking, a diet low in fruits, kidney dysfunction, elevated Low-Density Lipoprotein (LDL) levels, household air pollution from solid fuel use, and a sodium-rich diet [9]. Notably, stroke prevalence is significantly higher in the elderly population, particularly among individuals aged over 65, underscoring age as a significant risk factor for CVA [12, 13]. Moreover, there is evidence suggesting that a country's economic and income conditions influence stroke incidence, with a documented rise in stroke cases, especially in low- and middle-income nations [9, 11, 14].
Early consideration of stroke severity is crucial when assessing future clinical outcomes. Stroke severity is a vital indicator for various significant outcomes, including mortality, duration of hospitalization, discharge destination, and functional recovery [15]. Stroke assessment scales fall into two main diagnostic and impairment scales [16].
The Rapid Arterial Occlusion Evaluation (RACE) and National Institutes of Health Stroke Scale (NIHSS) are two diagnostic scales renowned for their exceptional accuracy in identifying Large Vessel Occlusion (LVO) cases [17]. Specifically, the RACE scale, tailored for prehospital emergencies, is the pioneering validated tool for diagnosing acute stroke and LVO [18, 19]. It comprises a 10-value range, where "0" signifies a normal state and "9" indicates severe obstruction. Notably, it exhibits predictive capabilities for LVO likelihood assessment. Scores meeting or exceeding "5" provide an 85% sensitivity and 69% specificity in detecting LVO cases, while scores below "5" maintain an 89% sensitivity, albeit with reduced specificity at 55%. On the other hand, the NIHSS scale, part of a comprehensive set for measuring stroke-related impairments, is adept at evaluating stroke effects in acute settings, albeit primarily designed for research and clinical trial applications rather than widespread bedside assessments [16, 20, 21]. It is essential to recognize that each scale possesses its strengths and limitations, with no universally acknowledged gold standard among them in research studies [16].
In recent years, Artificial Intelligence (AI), including Machine Learning (ML) and Deep Learning (DL), has gained significant traction in healthcare, particularly for stroke prediction and diagnosis [22]. ML methods such as Decision Tree (DT), Support Vector Machine (SVM), and Random Forest (RF) have become popular due to their ability to handle structured data, ease of interpretability, and relatively fast processing times [23]. Traditional ML models' main advantage is their simplicity and explainability, making them suitable for clinical applications. However, they may require manual feature engineering and often struggle with large unstructured datasets [24].
DL methods, especially Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have revolutionized the field of medical imaging and time-series data analysis [25]. CNNs have shown exceptional performance in analyzing complex medical images like Magnetic Resonance Imaging (MRI) and Computer Tomography (CT) scans, where feature extraction is automated. RNNs, particularly in analyzing temporal data such as electronic health records, can learn sequential patterns that are difficult to capture with traditional models [23]. The main advantage of DL models is their ability to handle large volumes of high-dimensional data and detect complex patterns. However, these models require vast training data, are computationally expensive, and are often criticized for their black-box nature, making them difficult to interpret for clinical decision-making [26]. In the medical field, transparency and interpretability are critical, which is why explainable AI (XAI) methods like SHapley Additive Explanations (SHAP) are now being integrated with ML and DL models to ensure that clinicians can understand the basis of a model's predictions [27, 28].
Given these considerations, the use of AI in cerebrovascular disorders, such as stroke, holds promise for early detection and severity prediction [27]. However, for AI models to be clinically relevant, they must address several challenges, such as class imbalances in datasets, generalizability across healthcare systems, and the need for model interpretability [24, 29, 30]. Generalization issues can arise when AI models trained on one dataset perform poorly on other datasets due to differences in population, data collection practices, or healthcare systems [31]. Handling imbalanced datasets, common in stroke prediction where certain severity levels may be underrepresented, requires advanced techniques like resampling or specialized algorithms to ensure accurate model performance [32]. Furthermore, clinician trust is paramount, and models perceived as "black boxes" are less likely to be adopted in practice [33]. Addressing these issues is essential for AI's reliable integration into clinical workflows [29].
The following are the main contributions of this study to address some of the research mentioned earlier
ASA, AN, and JBM were involved in the conception and design of this study. TS, GT, BR, ZH, SN, and HLA prepared datasets, and ASA performed the analysis. ASA, AN, UKW, SN, JBM, and HLA interpreted the results. ASA, TS, and GT drafted the manuscript, and all authors (ASA, TS, GT, AN, BR, ZH, UKW, SN, JBM, and HLA) contributed to writing the final draft of the manuscript.
In recent years, ML techniques have emerged as powerful tools in predicting stroke outcomes, with numerous studies highlighting their efficacy. Stroke prediction models have become increasingly sophisticated, driven by advancements in ML algorithms and data availability.
Su et al. [24] developed ML models such as SVM, RF, and Light Gradient Boosting Machine (LGBM) using stroke registry data from the Chang Gung Healthcare System. Their study focused on predicting modified Ranking Scale (mRS) outcomes and in-hospital deterioration. Notably, RF excelled in both predictions, whereas deep neural networks (NNs) outperformed other models in predicting in-hospital deterioration, particularly when resampling was not applied. This highlights the potential of Deep Learning (DL) in highly complex data characteristics.
Wu et al. [30] took a different approach, focusing on predicting stroke risk among elderly individuals using imbalanced datasets. Applying balancing techniques, such as the Synthetic Minority Oversampling Technique (SMOTE), significantly improved the performance of ML models like RF, SVM, and Ridge Logistic Regression (RLR). Their results emphasized the importance of addressing data imbalance, a critical challenge in medical datasets, to achieve robust model predictions.
Similarly, Zhang et al. [31] explored stroke prediction in elderly surgical patients, developing models incorporating data balancing and imputation techniques. Among the seven ML models evaluated, Extreme Gradient Boosting (XGBoost) emerged as the top performer, benefitting from data balancing techniques that optimized model performance. The study illustrated the significance of preprocessing techniques, mainly when working with medical datasets prone to missing values and class imbalances.
Kogan et al. [32] extended the application of ML to Electronic Health Records (EHR), where they aimed to estimate NIHSS scores for stroke patients using Natural Language Processing (NLP). Their RF model demonstrated a strong correlation between NLP-extracted NIHSS scores and clinical assessments, emphasizing the potential of integrating ML with unstructured clinical data for accurate stroke severity prediction.
Cui et al. [33] further validated the use of ML for predicting acute ischemic stroke and neurological impairment severity in patients with Anterior Circulation Large Vessel Occlusions (AC-LVO). This study compared four ML models (RF, SVM, RLR, and Logistic Regression (LR)), with SVM and RLR outperforming the others. Their work underscores the utility of combining different models to achieve optimal predictive performance for acute stroke conditions.
Regarding imaging-based stroke severity prediction, Faust et al. [34] developed a classification system using post-stroke MRI data. They applied various SVM classifiers and found that the SVM with a Radial Basis Function (RBF) kernel achieved superior accuracy, specificity, and sensitivity. This work highlights the importance of selecting appropriate kernels and parameters when applying ML models to imaging data, particularly in stroke.
Further, Yu et al. [35] focused on real-time stroke severity classification using NIHSS features, employing the C4.5 DT algorithm. This model demonstrated the highest recall and precision, suggesting that DTs can offer high interpretability alongside predictive accuracy, which is crucial for clinical application.
Someeh et al. [36] took a different angle by utilizing a Multilayer Perceptron (MLP) on a decade-long dataset, demonstrating high accuracy rates (81–85%) in stroke prediction. Their study pointed out that MLP models can be particularly effective when trained on longitudinal data, providing insights into long-term stroke risks.
Zhu et al. [37] focused on mortality prediction in stroke patients using a dataset of over 7,000 individuals. Their ML models achieved the highest reported accuracy for this task, pinpointing demographic and clinical factors, such as age, BMI, and marital status, as key predictors of mortality. This work emphasized the role of patient demographics in predicting stroke outcomes, a critical aspect for improving tailored care strategies.
Kokkotis et al. [38] employed a dataset of over 43,000 subjects to investigate ten key stroke risk factors. Their comparative analysis of ML classifiers revealed that the MLP performed best in reducing false negatives, essential for minimizing misdiagnoses in clinical settings. The study underscores the importance of balancing predictive performance with the need for low false-negative rates in critical health conditions like stroke.
Moreover, Dritsas and Trigka [28] advanced stroke prediction by proposing a stacking classifier framework, achieving an impressive Area Under the Curve (AUC) of 98.9%. Their work highlighted the potential of ensemble learning techniques to enhance predictive accuracy, a key consideration for future stroke prediction models.
JM and P [39] emphasized stroke's growing risk in younger populations due to unhealthy diets and highlighted the need for early detection. They developed an ML model to enhance stroke prediction, applying feature selection techniques like gradient boosting and RF. Their evaluation of classifiers, including DT, SVM, LR, GB, RF, K-Nearest Neighbor (KNN), and XGBoost, showed RF achieving the highest accuracy at 98%, demonstrating its strong predictive ability.
Finally, Hassan et al. [40] tackled the issue of imbalanced datasets, using three imputation techniques and SMOTE to enhance model performance. Their Dense Stacking Ensemble (DSE) model achieved over 96% accuracy, illustrating the potential of ensemble methods for handling complex stroke datasets. They identified age, BMI, and glucose levels as crucial for early stroke detection.
A summary of the reviewed literature from these studies are provided in Table A1 in Appendix A. While many studies have demonstrated the utility of ML for stroke prediction, few have focused explicitly on predicting stroke severity using RACE or NIHSS scales. The work of Su et al. [24] and Kogan et al. [32] highlights the promise of ML in predicting patient outcomes and stroke severity; however, a clear gap exists in multi-center studies that combine these two widely recognized scales. Our study addresses this gap by being the first to develop a stroke severity prediction model using both RACE and NIHSS scales, applying seven ML algorithms (KNN, DT, RF, AdaBoost, XGBoost, SVM, and Artificial Neural Network (ANN) across two different hospitals. Furthermore, SHAP enhanced model interpretability, a crucial step towards increasing clinician trust in ML-driven decision-making.
A block diagram of this study's methodology is shown in Fig. 1, illustrating the detailed steps from data preprocessing to model evaluation.
Fig. 1 Diagram of this study methodology
For transparency and interpretability, this study followed the Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklist [41]. In this study, two datasets with two different scales from Imam Khomeini Teaching Hospital (IKTH) with the RACE scale and Imam Reza Hospital (IRH) with the NIHSS scale were considered to predict the severity of stroke in patients [42].
These datasets containing information on stroke severity were systematically collected from 2011 to 2017. A clinical neurology expert conducted the inclusion of patient information and relevant features aimed at predicting stroke severity. Patients under 17 years or those lacking a definitive diagnosis of either ischemic or hemorrhagic stroke, as determined by ICD-10 codes, were excluded from the datasets.
All patients provided explicit written and verbal consent for using their data, and an anonymization process was rigorously applied to safeguard confidentiality and privacy.
Nine classes were considered for IKTH based on the stroke severity of patients (classes 0 to 8). In addition, six classes were considered for IRH based on patients' stroke severity (classes 0 to 5).
A retrospective registry examination methodology was employed in the study. Medical records from the neurology departments were used to collect data, and their description is presented in Table 1 [42]. International Statistical Classification of Diseases and Related Health Problems (ICD-10) codes were used to review the medical records.
All features, except age and Length of Stay (LOS), were categorical. Based on consultation with neurology specialists, cholesterol and triglycerides were converted from numerical to categorical (Table 2). Hospital-acquired complications (HACs) represented deep vein thrombosis, bedsores, lung infections, and urinary tract infections.
As outliers and missing data may impact how knowledge is interpreted, they should be identified and addressed with appropriate techniques. The missing ratio for IKTH and IRH was 9.91% and 8.15%, respectively (Table 1). Based on the literature [43], a feature with a more than 50% missing ratio cannot be used for analysis; hence, BMI for the IKTH dataset, with a 73% missing ratio, was removed from this study analysis. Table 3 outlines the techniques for imputing missing values in two datasets using the MICE library in R.
To normalize the values of the dataset samples, the minimum–maximum method from the Sklearn library in Python was used, so the values are mapped into the 0 to 1 range [44]. In this technique, according to Eq. (1), Xmin shows the minimum value and Xmax shows the maximum value in each feature of the
Nevertheless, in the IKTH dataset, 13 features, and in the IRH dataset, 14 features are preprocessed as input for ML algorithms. Each sample's features and the corresponding class label were transformed to an interval between 0 and 1. These preprocessed datasets serve as inputs for the proposed ML algorithms. The specific features selected as input for these algorithms are outlined in Table A2 in Appendix A.
Pearson's correlation coefficient is used to identify the relationship between the features of the datasets. This correlation coefficient specifies the type and degree of relevancy between features [44]. This coefficient has values between − 1 and + 1.
The SHAP method was used in this study for the interpretability of the proposed model [44, 45]. By using the SHAP method, this study aims to increase clinicians' trust in using ML models to diagnose the severity of the stroke and highlight the most important features that influence the decision-making process of the developed models.
Data imbalance is one of the main challenges in most datasets, especially in medical datasets [45]. One of the methods for measuring data skewness is Pearson's second coefficient skewness test. This method has been used to find the skewness of the dataset. Equation (2) shows the method of this test, where m is the median, x¯ is the mean, and s is the standard
The skewness of data could impact the training phase of ML models [45]. To address the imbalanced data issue in this study, the Synthetic Minority Oversampling Technique (SMOTE) was used to increase the number of class samples to balance the datasets [45]. By balancing the datasets, IKTH eventually had 1,179 samples of each class and a total of 10,611 samples for all classes. In addition, IRH, after balancing, had 96 samples of each class and a total of 576 samples for all classes. Moreover, Table 1 in Appendix A displays the final number of samples for each dataset after preprocessing and balancing, serving as the input values for the ML models.
The KNN algorithm is an ML algorithm. This algorithm is straightforward, and in the classification approach, it predicts the class based on the majority vote of the nearest neighbors [44, 45].
DT is one of the most widely used ML classification algorithms. This algorithm consists of leaves, roots, and branches. DT algorithm uses the Gini or entropy criterion to determine which strategy will most likely solve the problem successfully [44, 45].
RF is one of the standard algorithms with group decision-making strategies in ML. This algorithm creates DTs on sample data. Then, each tree makes its prediction, and in the end, the best possible prediction is selected by voting from all trees [44, 45].
AdaBoost algorithm is a member of the boosting algorithms family. In this algorithm, a problem is predicted by several classifiers, usually DT, and the final result for the problem is obtained based on the combination of the previous results [45].
XGBoost is one of the most powerful algorithms in ML. This algorithm is designed based on DT and gradient boosting, which is swift and highly efficient [44, 45].
SVM represents a supervised ML algorithm specifically formulated for classification and regression tasks. In a formal sense, SVM functions by creating a hyperplane within a multidimensional space to delineate distinct classes of data points. The primary goal is to optimize the margin, defined as the spatial separation between the hyperplane and the closest data points belonging to each class [44].
ANN serves as a computational model influenced by the architectural principles of the human brain. Comprising interconnected nodes arranged in layers (input, hidden, and output), this model employs weighted connections and activation functions to facilitate information processing. The network iteratively adjusts weights throughout training to minimize the disparity between predicted and actual outputs [46].
Furthermore, this study employed a fivefold cross-validation technique to minimize errors and reduce the risk of model overfitting [44, 45]. In this methodology, the dataset was divided into five subsets. In each iteration, 80% of the dataset was designated for training the classifier, allowing the model to learn patterns and relationships inherent in the data. The remaining 20% of samples in each fold were reserved for evaluating the classifier's performance, thus assessing its ability to generalize to new, unseen data. This iterative process was repeated five times with different subsets, ensuring a robust assessment of the classifier's effectiveness and generalizability across the entire dataset.
In addition to cross-validation, each dataset (IKTH and IRH) retained 20% of the samples as an independent test set. This independent test set was explicitly designated for unbiased validation of the trained models. After training and validating the models using the fivefold cross-validation method on the remaining 80% of the dataset, we assessed their performance on the reserved 20% independent test set. This dedicated test set is a benchmark for evaluating the models' generalizability to entirely new and unseen data.
The Grid Search method has been applied to find the optimal hyperparameter values of each model. Further descriptions and details about these hyperparameters can be found on the scikit-learn website [47]. The best hyperparameter values for IKTH and IRH are shown in Tables 4 and 5, respectively.
Confusion matrices serve as a valuable tool for assessing the performance of classification algorithms, offering a detailed breakdown of the model's efficacy in terms of TP, TN, FP, and FN predictions. Where TP, TN, FP, and FN indicate True Positive, True Negative, False Positive, and False Negative, respectively [44, 45]. These values form the basis for deriving various performance metrics.
Accuracy (Eq. 3) evaluates the overall correctness of the model by calculating the ratio of correctly predicted instances (both positive and negative) to the total instances [45]:
Kappa (Eq. 4) assesses the level of agreement between the model's predictions and the actual outcomes, accounting for the possibility of chance agreement. It is computed as
where Pagree means the proportion of trials in which judges agree and PChance means the proportion of trials in which agreement would be expected due to chance [48].
Precision (Eq. 5), also known as Positive Predictive Value (PPV), measures the proportion of positive predictions that were actually correct [45]:
Sensitivity or Recall (Eq. 6) quantifies the model's ability to identify positive instances [45]
Specificity (Eq. 7) calculates the proportion of negative instances correctly identified by the model [45]:
F1-Score (Eq. 8) is the harmonic mean of Precision and Recall, providing a single metric that balances the trade-off between these two measures [45]:
AUC measures class separability and is particularly useful for imbalanced datasets [44, 45]. These metrics, shown in Eqs. (3–8) were applied to evaluate the performance of the developed models in this study.
Pearson's second correlation coefficient was used to investigate the relevancies between the features of the datasets in this study. Figures 2 and 3 illustrate the associations among the features in the IKTH and IRH datasets, respectively. In addition, skewness tests were performed on these datasets. Skewness values exceeding 1 or falling below − 1 indicate significant skewness. Before balancing, the skewness value in the IKTH dataset was 0.69, which shifted to -0.41 after balancing. Similarly, for the IRH dataset, the skewness value was initially 0.84 and shifted to -0.78 after balancing. The closer proximity of these skewness values to 0 after balancing suggests improved dataset distribution.
Fig. 2 Pearson correlation coefficient map for IKTH dataset
Fig. 3 Pearson correlation coefficient map for IKTH dataset
Tables 6 and 7 present the average performance metrics across all classes and folds for the IKTH and IRH datasets, respectively. Based on these results, the RF model consistently outperformed the other models across most evaluation metrics. For a more comprehensive assessment of the RF model, Tables 8 and 9 present the detailed performance results for the RF model on all folds in both datasets. In addition, Table 10 outlines the computational time required by the RF model for each dataset.
A more thorough assessment of the RF model is shown through AUC and Precision–Recall (PR) diagrams for the IKTH and IRH datasets in Fig. 4, Fig. 5 , and Fig. 6 through Fig. 7. Furthermore, confusion matrices of the RF model across all folds for the IKTH and IRH datasets are displayed in Figs. 8 and 9, respectively.
Fig. 4 AUC diagram of RF for IKTH
Fig. 5 AUC diagram of RF for IRH
Fig. 6 PR diagram of RF for IKTH
Fig. 7 PR diagram of RF for IRH
Fig. 8 Confusion matrices of RF for IKTH
Fig. 9 Confusion matrices of RF for IRH
The proposed ML models were also evaluated on independent test sets for both datasets. Notably, the RF model once again demonstrated superior performance compared to the other models. In the independent test set for the IKTH dataset, the RF model achieved the following accuracy 92.89%, kappa 64.01%, precision 66.46%, sensitivity 67.73%, specificity 96.00%, and F1-Score 67.01%. Similarly, in the independent test set for the IRH dataset, the RF model achieved an accuracy of 93.39%, kappa of 75.97%, precision of 75.84%, sensitivity of 77.43%, specificity of 96.12%, and F1-Score of 76.07%. AUC, PR curves, and confusion matrix results for the independent test set of the IKTH dataset are shown in Fig. 10, Fig. 11 through Fig. 12, while those for the IRH dataset are displayed in Fig. 13, Fig. 14 through Fig. 15.
Fig. 10 AUC diagram of RF for the independent test set of IKTH
Fig. 11 PR diagram of RF for the independent test set of IKTH
Fig. 12 Confusion matrix of RF for the independent test set of IKTH
Fig. 13 AUC diagram of RF for the independent test set of IRH
Fig. 14 PR diagram of RF for the independent test set of IRH
Fig. 15 Confusion matrix of RF for the independent test set of IRH
Finally, Figs. 16 and 17 depict the most important features influencing the RF model's decision-making in the IKTH and IRH datasets. For the IKTH dataset, the most influential features include LOS, age, and triglycerides category (tri.cat), while for the IRH dataset, the key features were tri.cat, age, and LOS. Due to the tree-based structure of the RF model, we also provide the 100th rule set of the DT with a depth of 4 for the IKTH and IRH datasets in Figs. A1 and A2 in Appendix A, respectively.
Fig. 16 Feature importance in the decision-making of RF for IKTH
Fig. 17 Feature importance in the decision-making of RF for IRH
By considering both fivefold cross-validation and the independent test set as an external validation set, this study ensures a thorough evaluation of the classifiers' performance, generalization capability, and reliability across distinct datasets (IKTH and IRH). The primary research question we addressed in this study is how effectively proposed ML models, precisely the RF algorithm, can predict stroke severity in patients using the RACE and NIHSS scales. Among all developed ML models, the RF model exhibited the highest performance.
A thorough examination of the confusion matrices has smoothed a comprehensive analysis of different performance criteria, explaining their underlying concepts. As shown in Figs. 8, 9, 12, and 15, representing the confusion matrices of the best model in this study for each dataset, the model has consistently demonstrated proficiency in accurately determining patients' actual positive class stroke severity. Notably, there was a minimal margin of error in false negative cases, indicating instances where the model incorrectly identified the severity of stroke in patients.
The proposed RF model achieved an accuracy of 92.86% and an AUC of 92.02% for IKTH. Similarly, for the IRH dataset, the RF model had an accuracy of 91.19% and an AUC of 97.86%. Furthermore, the RF model achieved an accuracy of 92.89% and an AUC of 91.72% in the independent test set of IKTH. Likewise, the RF model demonstrated an accuracy of 93.39% and an AUC of 96.00% in the independent test set of IRH. Moreover, the RF model had less computational time during the training phase for both datasets. Therefore, based on the high efficiency of the best model of this study, which was obtained in a short time, it can be expected that this proposed model can accurately predict the severity of brain stroke in patients even by adding new samples. Our initial hypothesis posited that applying ML algorithms, particularly the RF model, would yield superior accuracy in predicting stroke severity compared to traditional clinical assessment methods. This hypothesis was confirmed by the results obtained.
Numerous studies have used ML algorithms for stroke diagnosis [24, 30, 31]. Su et al. achieved AUCs of 0.829 and 0.71 for the RF model using balanced and unbalanced datasets, respectively [24]. Wu et al. obtained AUC = 0.72 using the RLR algorithm [30]. Zhang et al. reached an AUC of 0.78 for the SVM model with RBF kernel [31]. Based on the mentioned results, considering the AUC metric, the proposed RF model outperformed these studies with similar methodologies [24, 30, 31]. The novelty of our work lies in its dual approach of utilizing the RACE and NIHSS scales for stroke severity prediction, addressing gaps in previous research, and contributing to the advancement of ML applications in clinical settings.
Although, some studies have previously predicted brain stroke severity [19–21, 24, 25, 27–40]. Nevertheless, these studies had gaps, which this research addresses. First, this study employed two standard scales, namely, RACE and NIHSS, to predict brain stroke severity in patients using ML models. Second, the top-performing model in this study produced favorable outcomes and demonstrated efficient computational performance, rendering it suitable for real-time clinical applications in promptly identifying stroke severity in patients. Third, the study emphasized interpretability and transparency in the decision-making process of the best-proposed model. To achieve this, SHAP is utilized to elucidate the model's key influential features, and rules were extracted from the RF model to enhance comprehension and transparency in predicting brain stroke severity for patients.
Furthermore, the Coronavirus Disease 2019 (COVID-19) pandemic has strained healthcare systems [49–53], making the development of ML-driven clinical support crucial for effective stroke management during these challenging times [53]. The increasing incidence of COVID-19-causing strokes highlights the urgent need for advanced clinical support systems [28, 49, 50]. ML models can accurately predict stroke severity [30], aiding healthcare decisions and improving patient outcomes.
Using SHAP for analysis for the IKTH dataset highlighted that LOS, age, and tri.cat were the three most influential features in predicting stroke severity. Similarly, in the IRH dataset, tri.cat, age, and LOS emerged as the most important features in the decision-making process of the proposed RF model. The shared identification of these three features in both datasets emphasizes their paramount importance in medicine for predicting the severity of brain strokes.
In line with this study's findings, a systematic review conducted by Yang et al. [54] revealed a correlation between elevated triglyceride and glucose (TyG) index levels and an increased risk of ischemic stroke in the broader population. While this study did not directly investigate the relationship between triglycerides and stroke severity, the obtained results indicate the importance of triglycerides as a significant factor associated with stroke [54]. Furthermore, variables such as LOS and age, recognized as significant in this study, have been identified as important predictors of stroke severity in previous research. For example, Okere's study [55] demonstrated a significant correlation between ischemic stroke severity and LOS. Although Okere's study primarily focused on LOS as the dependent variable, it implies that LOS is vital in stroke severity research [54]. Another study by Mineian et al. [56] found that a shorter LOS in the emergency department correlated with worse 90-day functional outcomes for ischemic stroke patients. In general, by examining the findings of previous studies [54–56], it is clear that the results obtained and the importance of the essential features identified in this study are in line with them, which can increase the confidence of clinicians in using the best model of this study as an auxiliary decision-making tool.
Highlighting the clinical significance of our findings, the proposed RF model can provide healthcare practitioners with a robust tool for assessing stroke severity, facilitating timely intervention, and improving patient outcomes. This capability is crucial in emergency medical settings, where rapid and accurate assessment can significantly affect treatment decisions.
The primary audience for this study comprises ML researchers and clinicians with a keen interest in a precise and comprehensible examination of stroke severity in patients. The study centers on the formulation of decision rules and the enhancement of interpretability.
The outcomes derived from this study have significant implications for managing emergency cases involving the severity of stroke patients' conditions. Identifying the critical factors influencing stroke severity, as demonstrated by the best predictive model, can inform the development or revision of treatment protocols and clinical guidelines.
The study shows numerous strengths. It uses two datasets and developed multiple ML techniques for stroke severity prediction. In addition, proper preprocessing techniques are utilized to ensure fair and reliable outcomes. Furthermore, the interpretability of model results is enhanced by applying SHAP. Nevertheless, certain limitations exist in this study. One of these datasets contained a limited number of samples, which limited the chance of developing DL models. The duration of illnesses and medication adherence are often the primary determining factors of CVA. The significance of these factors cannot be understated, and their absence in this study constitutes a limitation. Furthermore, prospective analysis and external validation are imperative to authenticate the study's outcomes further.
Stroke is among the primary global causes of both mortality and disability, entailing significant long-term management expenses. This study aimed to concurrently use two distinct scales, RACE and NIHSS, to predict stroke severity in patients from Urmia hospitals in West Azerbaijan province, Iran. This investigation explores the application of ML algorithms for predicting stroke severity in hospitalized patients.
Among developed ML models, the proposed RF model demonstrates superior performance. The best model in this study claims interpretability and explainability owing to its tree-based structure. In addition, decision rules for the proposed model are presented. Furthermore, the SHAP technique provides interpretation and clarification for this study's top model, enhancing transparency in the decision-making process. These promising outcomes show the potential of using the developed model as an auxiliary tool for clinicians.
As future work, we plan to validate the best model externally by applying it to additional datasets from diverse regions and hospitals to ensure its generalizability. We aim to develop a clinical protocol for real-time stroke severity prediction. Future research should also investigate additional factors influencing CVA, such as medication adherence and lifestyle, as key features in predictive models. Implementing DL algorithms, particularly with larger datasets, may enhance prediction accuracy and capture complex patterns. In addition, we see potential in developing a predictive scale that integrates cutoff points for age, LOS, triglyceride levels, and NIHSS and RACE severity scores, serving as an auxiliary tool to aid clinicians in effectively assessing stroke risk and severity.
We thank the Machine Learning Lab of Urmia University for providing the resources needed to implement this study. In addition, we thank Imam Khomeini Teaching Hospital and Imam Reza Hospital for rendering datasets.
ASA, AN, and JBM were involved in the conception and design of this study. TS, GT, BR, ZH, SN, and HLA prepared datasets, and ASA performed the analysis. ASA, AN, UKW, SN, JBM, and HLA interpreted the results. ASA, TS, and GT drafted the manuscript, and all authors (ASA, TS, GT, AN, BR, ZH, UKW, SN, JBM, and HLA) contributed to writing the final draft of the manuscript.
This work was supported by the Urmia University of Medical Sciences, Urmia, Iran.
The dataset used in this study is not publicly available due to privacy restrictions. Researchers interested in accessing the data can request it from the corresponding authors with valid justification.
This study gained ethics approval from the ethics committee of Urmia University of Medical Sciences (IR.UMSU.REC.1402.090). The data for this study were stored and retrieved from an ethics-approved prospective database. All methods were carried out in compliance with pertinent guidelines and regulations. Participants' data was become anonymized and Informed consent to participate was obtained from all participants.
Not applicable.
The authors report no conflict of interest.
Jamshid Bagherzadeh Mohasefi, Email: j.bagherzadeh@urmia.ac.ir, Email: jmoh@mmmi.sdu.dk.
Hadi Lotfnezhad Afshar, Email: lotfnezhadafshar.h@umsu.ac.ir, Email: hadi.afshar@gmail.com.
The dataset used in this study is not publicly available due to privacy restrictions. Researchers interested in accessing the data can request it from the corresponding authors with valid justification.