Authors: Linfeng Xi, Han Kang, Mei Deng, Wenqing Xu, Feiya Xu, Qian Gao, Wanmu Xie, Rongguo Zhang, Min Liu, Zhenguo Zhai, Chen Wang
Categories: Original Article, Acute pulmonary embolism, Machine learning, Revised Geneva score, Wells score, Years algorithm
Source: Chinese Medical Journal
A machine learning model for diagnosing acute pulmonary embolism and comparison with Wells score, revised Geneva score, and Years algorithm
Authors: Linfeng Xi, Han Kang, Mei Deng, Wenqing Xu, Feiya Xu, Qian Gao, Wanmu Xie, Rongguo Zhang, Min Liu, Zhenguo Zhai, Chen Wang
Abstract
Background:
Acute pulmonary embolism (APE) is a fatal cardiovascular disease, yet missed diagnosis and misdiagnosis often occur due to non-specific symptoms and signs. A simple, objective technique will help clinicians make a quick and precise diagnosis. In population studies, machine learning (ML) plays a critical role in characterizing cardiovascular risks, predicting outcomes, and identifying biomarkers. This work sought to develop an ML model for helping APE diagnosis and compare it against current clinical probability assessment models.
Methods:
This is a single-center retrospective study. Patients with suspected APE were continuously enrolled and randomly divided into two groups including training and testing sets. A total of 8 ML models, including random forest (RF), Naïve Bayes, decision tree, K-nearest neighbors, logistic regression, multi-layer perceptron, support vector machine, and gradient boosting decision tree were developed based on the training set to diagnose APE. Thereafter, the model with the best diagnostic performance was selected and evaluated against the current clinical assessment strategies, including the Wells score, revised Geneva score, and Years algorithm. Eventually, the ML model was internally validated to assess the diagnostic performance using receiver operating characteristic (ROC) analysis.
Results:
The ML models were constructed using eight clinical features, including D-dimer, cardiac troponin T (cTNT), arterial oxygen saturation, heart rate, chest pain, lower limb pain, hemoptysis, and chronic heart failure. Among eight ML models, the RF model achieved the best performance with the highest area under the curve (AUC) (AUC = 0.774). Compared to the current clinical assessment strategies, the RF model outperformed the Wells score (P = 0.030) and was not inferior to any other clinical probability assessment strategy. The AUC of the RF model for diagnosing APE onset in internal validation set was 0.726.
Conclusions:
Based on RF algorithm, a novel prediction model was finally constructed for APE diagnosis. When compared to the current clinical assessment strategies, the RF model achieved better diagnostic efficacy and accuracy. Therefore, the ML algorithm can be a useful tool in assisting with the diagnosis of APE.
Introduction
Acute pulmonary embolism (APE) is the 3rd most prevalent cardiovascular disease globally, after ischemic heart disease and stroke.^[1]^ Early detection and accurate diagnosis of APE are crucial for subsequent evaluation and treatment. Nonetheless, the clinical manifestations of APE are non-specific, making it easier to be missed or misdiagnosed. The current optimal diagnostic strategy for suspected APE is controversial. The European guidelines recommend a diagnostic strategy based on clinical probability assessment, including the Wells score,^[2]^ revised Geneva score,^[3]^ and Years algorithm.^[4]^ However, one survey suggested that most of the investigated hospitals did not use the currently recommended approach when evaluating patients diagnosed with suspected APE.^[5]^ Meanwhile, other scholars reiterate that the current clinical assessment may not be used to detect APE among critically ill patients.^[6,7]^ What's more, implementing the assessment protocol is a labor-intensive and time-consuming task, making it inapplicable to the routine diagnosis of APE.^[8]^ Therefore, there is a need for a simple and objective assessment protocol for diagnosing APE.
Data-driven machine learning (ML) techniques improve risk prediction performance by discovering novel risk predictors and understanding the complex interactions between them. ML is increasingly being used to diagnose a wide range of medical conditions, particularly cardiovascular diseases.^[9]^ It improves the accuracy of cardiovascular event prediction in asymptomatic population^[10]^ and stroke prediction in hypertensive populations.^[11]^ A new venous thromboembolism (VTE) prediction model based on ML has been recently developed for young hospitalized patients.^[12]^ which has been shown to quickly and accurately detect VTE, thus contributing to the standardization of disease prevention and management. Therefore, this work developed an ML-based model for diagnosing APE, and then compared its performance with the Wells score, revised Geneva score, and Years algorithm in diagnosing APE.
Methods
Patient enrollment and data collection
This study complied with the Declaration of Helsinki and was approved by the Ethics Committee of China-Japan Friendship Hospital (IRB No. 2022-KY-048). Informed consent was waived due to its retrospective design. Patients with suspected APE were retrospectively enrolled between January 2019 and December 2019 in China-Japan Friendship Hospital. The study enrolled patients older than 18 years and underwent computed tomography pulmonary angiography (CTPA) in our hospital. Exclusion criteria (1) diagnosed with Takayasu arteritis, chronic thromboembolic pulmonary disease, pulmonary artery sarcoma, fibrosing mediastinitis, and pulmonary vascular malformation; (2) readmission for definite APE; (3) poor imaging quality of CTPA, which failed to diagnose; (3) loss to follow-up within 3 months; and (4) incomplete clinical information. We collected the basic demographic information, clinical symptoms, vital signs at onset, and laboratory data from the electronic medical records. APE was finally diagnosed by two experienced physicians according to clinical information and CTPA images. All patients were followed up for 3 months by outpatient visits and telephone visits to confirm or exclude the diagnosis of APE.
Feature selection and dimensionality reduction
According to the current clinical assessment of APE.^[8]^ 27 clinical characteristics were collected from the electronic medical record system including gender, age, previous APE/deep venous thrombosis (DVT), surgery or fracture history within the past month, tachycardia, chronic heart failure, active malignancy, chest pain, dyspnea, hemoptysis, syncope, unconsciousness, chest tightness, fever, lower limb pain, lower limb weakness, lower limb edema, hemodynamic stability, systolic blood pressure, blood diastolic pressure, heart rate, arterial oxygen saturation, oxygenation index, biomarkers of myocardial injury (creatine kinase-MB [CK-MB], cardiac troponin T [cTnT], N-terminal pro-B-type natriuretic peptide [NT pro-BNP]), and D-dimer.
Dimensionality reduction was performed before modeling given that a high-dimensional feature space may cause model overfitting. We adopted a two-step flow for feature dimensionality reduction. Initially, the statistical difference between APE and non-APE patients for all features was compared and features with P <0.05 were retained for subsequent analysis. The Pearson correlation coefficient (r) was then computed between each pair of the remained features. If one pair of features exhibited an absolute value of r over 0.85, then the feature with the smaller P-value was kept for modeling.
ML model development
The ML model was constructed based on the selected features. For unbiased estimates of model accuracy, the collected data of enrolled patients were randomly divided into the training and testing sets by a 1 ratio. The training set trained the ML model and optimized the model parameters. A total of eight independent ML algorithms were selected to develop the model, including Naive Bayes, logistic regression, K-nearest neighbors, random forest (RF), decision tree, gradient boosting decision tree (GBDT), support vector machine, and multi-layer perceptron. Five-fold cross-validation was applied in the training set to obtain the best parameters for each model. The receiver operating characteristic (ROC) analysis and the area under the curve (AUC) were evaluated to assess the diagnostic performance of each model. Eventually, the average AUC values from the cross-validation were obtained to describe the discriminatory power of certain trained models incorporating specific parameters.
ML model test and validation
The ML model with optimal outcomes on the training set was applied to the testing set, in order to further assess the diagnostic performance of the model. Additionally, current clinical assessment strategies, including the Wells score, revised Geneva score, and Years algorithm were used for APE diagnosis based on the testing dataset. The ML model was then compared with the current clinical assessment strategies incorporating the D-dimer levels to comprehensively assess the diagnostic performance.
Patients suspected of APE between January and August 2018 in the China-Japan Friendship Hospital were enrolled as an internal validation set. After constructing and testing the model, the final model with comparable performance and diagnostic stability was determined for internal validation. Figure 1 demonstrates a flowchart showing the overall scheme of the study.
Statistical analysis
The baseline characteristics were summarized as median with interquartile range or n (%) appropriately. Differences in variables were compared by the Mann–Whitney U test for measurement variables and by χ^2^ test or Fisher's exact test for categorical variables. To test the diagnostic efficacy of different models, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were computed using the ROC curve and AUC analysis. The Delong test was used to compare the difference between diagnostic models. A value of P <0.05 was considered statistically significant. Statistical analyses were performed using SPSS version 26.0 (SPSS Inc; Chicago, Illinois, USA) and Python software version 3.7 (https://www.python.org/).
Results
Study population
A total of 454 patients (241 males, median age 65 years) were randomly grouped into training (n = 340, 187 males, median age 65 years) and testing set (n = 114, 54 males, median age 66 years). In the training and testing datasets (n = 454), 113 (24.9%) cases were eventually diagnosed with APE [Supplementary Table 1, http://links.lww.com/CM9/B703]. A total of 204 cases (107 males, median age 66 years) were enrolled in the internal validation set; 59 (28.9%) of them were finally diagnosed with APE [Supplementary Table 1, http://links.lww.com/CM9/B703]. The baseline characteristics of the training and testing datasets were comparable [Supplementary Table 2, http://links.lww.com/CM9/B703].
Feature selection and ML model development
According to the differential analysis results, 10 out of 27 features (including surgery or fracture within the past month, tachycardia, chronic heart failure, chest pain, hemoptysis, lower limb pain, heart rate, D-dimer, cTnT, and arterial oxygen saturation) displayed significant differences between APE and non-APE groups based on training dataset [Table 1].
After calculating the Pearson correlation coefficient and filtering features, eight clinical features, including D-dimer, cTNT, arterial oxygen saturation, heart rate, chest pain, lower limb pain, hemoptysis, and chronic heart failure were identified to construct ML models. These selected features were then placed into the training dataset to cross-validate the candidate models and obtain the warranted model with the best diagnostic performance.
In total, eight candidate models based on different algorithms were constructed, and Table 2 shows their diagnostic performance on the training set. The RF model achieved the best discriminatory performance (AUC = 0.774), which was applied to the testing set for further validation of the model stability. Figure 2 shows the weight of the selected eight metrics in the trained RF. The weight of the D-dimer was 0.35, contributing the most to the model. In the testing set, the RF model achieved an excellent unbiased estimate of the discriminant capacity (AUC = 0.813, 95% confidence interval [CI]: 0.729, 0.880).
Diagnostic performance and internal validation
The diagnostic capacity of the RF model was preferable among established models. However, it was necessary to compare it with the current clinical strategies in terms of APE diagnosis. Therefore, we computed the AUC of the RF model and three usual clinical assessment strategies using testing dataset [Table 3]. Since D-dimer is an important enrolled parameter that contributed the most in the RF model and evidence-based guidelines advise D-dimer detection in APE diagnosis, the clinical assessment strategies combined with D-dimer were used to compare with the RF model. Table 3 shows that the RF model achieved the best diagnostic performance and the highest NPV. The AUC of the RF model was 0.813 (95% CI: 0.729, 0.880), followed by the Years algorithm (0.719, 95% CI: 0.627, 0.799). The revised Geneva score with D-dimer displayed the best diagnostic specificity for APE (specificity = 0.744 [95% CI: 0.639, 0.832]). Figure 3 shows the ROC curves of different models evaluated on the testing set. The diagnostic efficacy of the RF model was not inferior to any other clinical assessment strategies. Specifically, the RF model outperformed Wells score combined with D-dimer significantly (P = 0.030) [Figure 3]. Finally, we verified the established RF model in the internal validation dataset. As shown in Figure 4, the AUC of the RF model in the validation dataset reached 0.726 (95% CI: 0.652, 0.801).
Discussion
In the present study, we developed and validated the ML model based on clinical and laboratory data for diagnosing APE and further compared it with the current clinical assessment strategies. In contrast with the Wells score combined with D-dimer, the revised Geneva score with D-dimer, and the Years algorithm, the RF model demonstrated the best performance in diagnosing APE.
Many previous studies have appreciated the importance of clinical assessment in the diagnosis of APE^[8,13–15]^; however, a standard assessment criterion is currently lacking. Notably, using clinical assessment to its full potential could help in the safe exclusion of APE without the need for CTPA examination. Van der Hulle et al^[4]^ discovered that the Years algorithm, which applies to all patients, reduced the number of CTPA examinations by 14%. However, the Wells score does not apply to inpatients when it comes to other clinical prediction rules.^[16]^ The Wells score and the revised Geneva score are not fully applicable to APE diagnosis in critically ill patients.^[6]^ These rules are complex, thereby limiting their use in diagnosing APE.^[17]^ One study confirmed that only 50–60% of clinicians adhered to standardized evaluation.^[18]^ The Years algorithm is a novel and relatively simple clinical assessment strategy, hence convenient for clinical use. However, its simplicity makes it easy to misdiagnose. Therefore, the Years algorithm should be used with caution.^[19]^ Importantly, the Years algorithm has a subjective item that is highly dependent on the experience of the clinician.
ML has been used in numerous medical conditions, which can reduce physician workload, and accelerate diagnosis.^[20]^ ML has an advantage in handling massive, complex, and mixed data types, allowing it to automatically detect APE.^[21]^ Previous studies have used ML to predict APE, focusing on specific patients, including young patients or those diagnosed with coronavirus pneumonia, etc.^[12,22,23]^ Banerjee et al^[24]^ proposed a customized risk score system incorporating clinical parameters for patients suspected of APE to assist clinical decisions; the results revealed that the AUC of the scoring system to diagnose APE was 0.90 (0.87–0.91). Meanwhile, the neural network model has the potential to predict the APE development and outperforms current clinical scores (the AUC of the neural network model, Wells score, and revised Geneva score were 0.73–0.81, 0.48, and 0.53, respectively). Villacorta et al^[25]^ used an elastic net model to develop and internally validate a logistic regression based on four clinical variables and D-dimer levels. They discovered that the ML model outperformed traditional scores in the emergency department for risk stratification of APE. Hou et al^[26]^ used the GBDT model to diagnose APE in hospitalized patients and found that it exhibits more superiority in APE diagnosis (AUC = 0.799). This implies that it can more accurately identify APE patients than traditional clinical scores, and improve prognosis by diagnosing APE at an earlier stage, thus promoting timely intervention.
The ML model, specifically RF, showed the best performance in diagnosing APE compared to the available clinical assessment strategies. With the existing clinical data, we sought to establish whether the novel model based on ML could improve the diagnostic performance of APE, rather than searching for novel diagnosing parameters. Of note, the RF model has been used in diagnosing various diseases. It combines multiple regression trees comprising bootstrap aggregation and randomization of predictors to achieve high predictive accuracy.^[27]^ Research on cardiovascular disease revealed that the RF model is superior to other ML models in improving the accuracy of cardiovascular disease risk prediction.^[28]^ A retrospective cohort study on patients with asthma exacerbation developed an RF model with an 88% accuracy in distinguishing the APE status (AUC = 0.899), which assisted in evaluating the effect of APE on outcomes.^[29]^ Emir et al^[30]^ also found that the RF model could screen patients who may need more comprehensive examinations for fibromyalgia. Moreover, they found that the RF model could quantify the probability of fibromyalgia, enabling more precise diagnosis and individualized management. Similarly, our findings revealed that the RF model is superior to other ML models, providing an accurate and comprehensive tool for APE diagnosis.
The ML model in the present study is superior to traditional clinical assessment strategies in APE diagnosis, but it needs a combination of more relevant data underlying the patient condition for better disease management. Multi-modal ML, a form of artificial intelligence, constructs models that could cope with related data associated with multiple modalities.^[31]^ In routine practice, physicians rely on different data types to make clinical decisions, including textual data like medical history, laboratory, and imaging data. Therefore, a multi-modal fusion model is necessary to offer an appropriate method for integrating multiple information. One study developed a multi-modal fusion model to detect APE by integrating a medical record system with CT imaging and found that the multi-modal model outperforms the single-modal model.^[32]^ Similarly, the ML model in our study can be combined with CT imaging data to further improve the diagnostic accuracy of APE; however, this needs further research.
This study has several limitations. First, it is a retrospective study in a single center, which may cause bias. Secondly, considering the inaccessible property of ML algorithms, the RF model in this work is unable to compare with other published ML models in terms of diagnostic efficacy. Thirdly, the metrics in the current model are less specific since the model aims to quickly diagnose patients with highly suspicious APE. Furthermore, the data in this work were manually extracted rather than automatically. If the model is integrated with the natural language processing system in the future, it will automatically detect potential APE in patients for clinicians and standardize the diagnosis of APE.
In conclusion, the RF model constructed based on clinical data on D-dimer, cTNT, arterial oxygen saturation, heart rate, chest pain, lower limb pain, hemoptysis, and chronic heart failure has better diagnostic efficacy and accuracy than the current clinical assessment models in APE diagnosis. This model may help in accurately diagnosing the APE earlier.
Funding
This work was supported by grants from the Chinese Academy of Medical Sciences Innovation Fund for Medical Sciences (No. 2021-I2M-1-049), the Elite Medical Professionals Project of China-Japan Friendship Hospital (No. ZRJY2021-BJ02), and the National High Level Hospital Clinical Research Funding (No. 2022-NHLHCRF-LX-01).
Conflicts of interest
Han Kang and Rongguo Zhang are employed by Institute of Advanced Research, Infervision Medical Technology Co., Ltd. All authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.