Authors: Chanyoung Rhee (Republic of Korea), Ki Jeong Hong (Republic of Korea), Ki Hong Kim (Republic of Korea), Jin Mo Goo (Republic of Korea), Eui Jin Hwang (Republic of Korea)
Categories: Thoracic Imaging, Artificial intelligence, Chest radiography, Emergency department, Triage, Korean
Source: Korean Journal of Radiology
Authors: Chanyoung Rhee, Ki Jeong Hong, Ki Hong Kim, Jin Mo Goo, Eui Jin Hwang
In this study, we investigated whether artificial intelligence (AI) analysis of chest radiographs (CXRs) can predict major adverse clinical events in patients visiting the emergency department (ED) with acute cardiopulmonary symptoms.
This secondary analysis of a previous clinical trial included patients who visited the ED with symptoms suggestive of acute cardiopulmonary disease and underwent chest radiography between June 2020 and December 2021. All patients underwent triage upon arrival at ED according to the Korean Triage and Acuity Scale (KTAS). The CXRs were retrospectively analyzed using a commercial AI (Lunit INSIGHT CXR, version 3.1.4.1) capable of detecting seven abnormalities on a single frontal CXR. The predictive performance of the AI analysis for major adverse cardiopulmonary events (any among hospitalization, ED revisits, and death in the ED due to acute cardiopulmonary disease) was compared with that of the KTAS using the area under the receiver operating characteristic curve (AUC). Multivariable (the AI analysis result and KTAS level) logistic regression analysis was conducted to investigate whether the AI analysis result was an independent predictor of the events and whether the combination of the AI analysis and KTAS has additional merit.
Among 3576 patients (1966 males; mean age, 64 years), 1148 (32.1%) experienced major adverse cardiopulmonary events. AI analysis of CXRs outperformed the KTAS in predicting these events (AUC, 0.795 vs. 0.610; P < 0.001). The AI analysis result was an independent predictor of these events after adjusting for the KTAS level (adjusted odd ratios of 1.032 and 6.913 for every 1% increase and ≥15%, respectively, in the AI probability score; P < 0.001). The combination of the AI analysis and KTAS showed an AUC that was higher than that of the KTAS alone (0.799; P < 0.001) and in-par with that of the AI analysis only (P = 0.187).
AI analysis of CXRs showed greater accuracy than the KTAS did in predicting major adverse cardiopulmonary events in patients visiting the ED with acute cardiopulmonary symptoms. AI analysis may enhance the efficacy of patient triage in the ED.
Symptoms related to acute cardiopulmonary diseases are a major reason for visiting the emergency department (ED) [12]. In the United States, chest pain, dyspnea, and cough were the second, third, and fourth most common symptoms among patients in the ED in 2021, comprising 5.6%, 4.2%, and 3.3% of the complaints during ED visits, respectively [3]. Given the broad spectrum of potential diagnoses and varying urgency of conditions, effective triage is critical for patients with these symptoms. The current triage system based on patients’ symptoms and signs, the Korean Triage and Acuity Scale (KTAS), is a useful tool for assessing patients’ acuity in the ED [4]. However, its accuracy and inter- and intra-rater variability pose significant limitations [5678]. Chest radiographs (CXRs) are routinely used for the initial evaluation of acute cardiopulmonary diseases and are readily available in the ED at a low cost, with a short acquisition time [91011]. As a CXR can reveal the presence and severity of various cardiopulmonary diseases, analyzing it for relevant abnormalities may help identify patients with a higher risk of adverse clinical events (e.g., revisiting the ED, death in the ED) or those requiring urgent management.
Nevertheless, immediate CXR interpretation by a radiologist in the ED is often impractical, which limits its potential as a triage tool. Artificial intelligence (AI) tools have demonstrated excellent performance in identifying various abnormalities on CXRs and are currently used in clinical practice to assist physicians in interpreting CXR findings [1112131415161718192021222324]. As AI tools can automatically analyze CXRs immediately after acquisition, they may aid in triaging patients with acute cardiopulmonary symptoms in the ED [25]. However, few studies have assessed the potential of commercial AI tools for CXRs as triage aids in the ED setting.
In this study, we aimed to investigate whether the analysis of CXRs using AI can predict major adverse clinical events in patients visiting the ED with acute cardiopulmonary symptoms.
This secondary analysis of a previously conducted clinical trial was approved by the Institutional Review Board of the Seoul National University Hospital (IRB No. 2308-145-1459). The requirement for obtaining informed consent was waived by the Institutional Review Board.
All patients included in the present study were selected from among the participants of a previously reported randomized clinical trial [26]. No additional inclusion or exclusion criteria were applied beyond those of the original trial. The objective of the clinical trial was to compare the accuracy of conventional and AI-assisted CXR interpretations in patients visiting the ED with acute cardiopulmonary symptoms. The trial included adult patients (aged ≥19 years) who visited the ED with chief complaints of fever, dyspnea, chest pain, cough, hemoptysis, sputum, or chills and underwent chest radiography. Patients requiring immediate resuscitation (KTAS level 1) and those presenting with trauma to the ED were excluded.
For this secondary analysis, data were obtained from the original trial, including the 1) age and sex, 2) chief complaint upon arrival at the ED, 3) triage results based on the KTAS (Table 1) [4], 4) CXRs and interpretations by the duty radiologist, 5) clinical diagnosis based on medical record review, 6) patient disposition after ED management (i.e., discharge to home, transfer to another institution for hospitalization, hospitalization, or death), and 7) revisits to the ED within 30 days for the same condition.
Each patient’s clinical diagnosis was determined by reviewing their medical records for at least 30 days after the ED visit. This review was conducted during the original clinical trial by a thoracic radiologist (E.J.H., 3 years of experience as an attending thoracic radiologist) [26].
CXRs were acquired using a fixed scanner (Multix FD; Siemens Healthineers, Erlangen, Germany) with posteroanterior projection in the erect position or a portable scanner (AccE GM85; Samsung Healthcare, Seoul, Korea) with anteroposterior projection in the supine position, depending on the patient’s condition.
All CXRs were analyzed using a commercial AI tool (Lunit INSIGHT CXR, version 3.1.4.1; Lunit, Seoul, Korea) that has been approved for clinical use as an assistive tool for physician interpretation of CXRs. The AI tool detects 10 types of abnormalities (atelectasis, calcification, cardiomegaly, consolidation, fibrosis, mediastinal widening, nodules, pleural effusion, pneumoperitoneum, and pneumothorax) on a single frontal CXR [14]. The tool assigns probability scores (0%–100%). In this study, the AI tool was used to identify seven target abnormalities, excluding atelectasis, calcification, and fibrosis. The highest probability score among these seven abnormalities was regarded as the final CXR score, and CXRs with a final probability score ≥15% were defined as positive. Although all CXRs were obtained from the previous clinical trial [26], the AI analyses for this study were conducted separately from the original trial.
To evaluate the potential of AI analysis of CXR as a triage tool, we evaluated its performance in predicting the occurrence of major adverse events. We defined the major adverse event as the occurrence of any of the following adverse events in the 1) hospitalization, 2) transfer to another institution for hospitalization, 3) revisit to the ED within 30 days with the same problem, and 4) death in the ED (death of the patient in the ED before discharge, hospitalization, or transfer). The primary events of interest in our study were major adverse events in patients diagnosed with acute cardiopulmonary diseases. We also investigated the occurrence of major adverse events regardless of the clinical diagnosis.
We compared the predictive accuracy of AI-assisted CXR analysis for the occurrence of major adverse events with those of initial triage results using the KTAS and CXR interpretation by duty radiologists in the ED. During the original clinical trial, the initial triage upon arrival at the ED was conducted by trained nurses using the KTAS, whereas all CXRs were interpreted by duty radiologists in the ED (third-year residents) using a standardized report format, which included the presence or absence of any findings suggesting acute thoracic diseases [26].
We conducted subgroup analyses to investigate the predictive accuracy of the AI results and their added value to the KTAS in subgroups of patients undergoing chest radiography with fixed versus portable scanners and patients with different chief complaints (i.e., fever, dyspnea, chest pain, and other symptoms).
To evaluate the accuracy of the continuous probability score assigned in the AI analysis and the KTAS levels for predicting of adverse events, we evaluated the area under the receiver operating characteristic curve (AUC). In addition, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were determined by defining a probability score ≥15% and a KTAS level of 2 or 3 as positive results. The AUCs were compared using DeLong’s test [27]. Sensitivities and specificities were compared using McNemar’s test, whereas PPVs and NPVs were compared using the permutation method suggested by Park et al. [28].
We conducted logistic regression analyses to investigate whether AI analysis of CXR has additional merit in predicting adverse events. The continuous probability score assigned in the AI analysis and binary result at a threshold probability score of 15% were utilized as independent variables in the logistic regression model for the prediction of adverse events, whereas the KTAS level was regarded as a covariate. We also investigated the AUC of the logistic model using the continuous probability score assigned in the AI analysis and KTAS level (hereafter referred to as the combination model) to predict adverse events. A P-value <0.05 was considered statistically significant. All statistical analyses were performed using Python (version 3.11; Python Software Foundation, Wilmington, DE, USA) with relevant libraries, including scikit-learn (version 1.1.3), statsmodels (version 0.14.0), and matplotlib (version 3.6.2).
A total of 3576 participants (1966 males; mean age, 64 years) were included in the study (Table 2). The most common chief complaint during the ED visit was fever (1561 participants; 43.7%), followed by dyspnea (1250 participants; 35.0%) and chest pain (486 participants; 13.6%). In total, 1153 (32.2%), 1933 (54.1%), 444 (12.4%), and 46 (1.3%) participants, respectively, were classified with the KTAS levels of 2, 3, 4, and 5.
Acute cardiopulmonary diseases were diagnosed in 1519 participants (42.5%). Pneumonia was the most frequent clinical diagnosis (643 participants, 18.0%), followed by acute decompensated heart failure (140 participants, 3.9%) and acute coronary syndrome (97 participants, 2.7%). Among 2057 (57.5%) participants without acute cardiopulmonary diseases, no specific disease could explain the acute symptoms in 1102 (30.8%) participants. Urinary tract infection (208 participants, 5.8%) was the most common among diagnosis other than acute cardiopulmonary diseases, followed by acute cholangitis (138 participants, 3.9%) and liver abscess (40 participants, 1.1%).
Regarding adverse events, hospitalization, transfer to another institution for hospitalization, revisit of the ED within 30 days for the same problem, and in-hospital deaths were noted in 1400 (39.1%), 635 (17.8%), 238 (6.7%), and 22 (0.6%) participants, respectively. After the exclusion of duplicate events (e.g., revisiting the ED after hospitalization), major adverse events were noted in 2190 participants (61.2%). Among the 1519 participants with acute cardiopulmonary diseases, major adverse events occurred in 1148 participants (32.1% of all participants) (Table 3).
The median abnormality score assigned in the AI analysis was 73 (interquartile range, 12–95; Fig. 1). At a threshold of 15%, 2557 (71.5%) CXRs were deemed positive by the AI analysis. Consolidation was the most commonly identified abnormality according to the AI (2269, 63.5%), followed by nodules (1443, 40.4%), cardiomegaly (1205, 33.7%), and pleural effusion (982, 27.5%). The distribution of the AI results for each abnormal finding is shown in Supplementary Table 1. Patients with major adverse events had significantly higher AI scores than did those without (median, 87 vs. 32; P < 0.001).
For predicting major adverse events due to acute cardiopulmonary diseases, the AI analysis exhibited an AUC of 0.795 (95% confidence interval [CI], 0.779–0.812), which was significantly higher than that of the KTAS (0.610; 95% CI, 0.595–0.627; P < 0.001) (Fig. 2). At a threshold of 15%, the AI analysis demonstrated a sensitivity of 92.4% (1061/1148) and a specificity of 38.4% (932/2428) (Table 4). Compared with a KTAS level of 2 or 3 (sensitivity, 94.9% [1089/1148]; specificity, 17.8% [431/2428]), the AI results showed lower sensitivity (P = 0.021) but higher specificity (P < 0.001). Compared with the interpretations by the duty radiologist (sensitivity, 62.9% [722/1148]; specificity, 83.0% [2015/2428]), the AI analysis demonstrated higher sensitivity (P < 0.001) but lower specificity (P < 0.001).
Both the continuous probability score (adjusted odds ratio [OR], 1.032 per 1% increase; 95% CI, 1.029–1.035; P < 0.001) and binary result at a threshold of 15% (adjusted OR, 6.913; 95% CI, 5.464–8.747; P < 0.001) were independent predictors of major adverse events due to acute cardiopulmonary diseases after adjusting for the KTAS level. The combination model (logistic regression model using the continuous probability scores assigned by AI and the KTAS level) showed an AUC of 0.799 (95% CI, 0.785–0.815), which was significantly higher than that of the KTAS (P < 0.001) and similar to that of the AI analysis (P = 0.187).
For predicting major adverse events due to any diagnosis, the AI analysis achieved an AUC of 0.674 (95% CI, 0.655–0.691), which was significantly higher than that of the KTAS (0.638; 95% CI, 0.619–0.656; P = 0.002) (Fig. 2). At a threshold of 15%, the AI analysis yielded a sensitivity of 80.6% (1765/2190) and specificity of 42.9% (594/1386) (Table 4). Compared with a KTAS level of 2 or 3 (sensitivity, 92.6% [2029/2190]; specificity, 23.7% [329/1386]), the AI results exhibited significantly lower sensitivity (P < 0.001) but higher specificity (P < 0.001). Compared with the interpretations by the duty radiologist (sensitivity, 39.3% [861/2190]; specificity, 80.2% [1112/1386]), the AI results demonstrated higher sensitivity (P < 0.001) but lower specificity (P < 0.001).
Both the continuous probability score (adjusted OR, 1.013 per 1% increase; 95% CI, 1.012–1.015; P < 0.001) and binary result at a threshold of 15% (adjusted OR, 2.727; 95% CI, 2.335–3.185; P < 0.001) were independent predictors of major adverse events due to any diagnosis after adjusting for the KTAS level. The combination model showed an AUC of 0.705 (95% CI, 0.688–0.723), which was significantly higher than those of the KTAS and AI analysis (P < 0.001).
The performance of the AI analysis at different threshold scores is presented in Supplementary Table 2. Representative images of patients are shown in Figures 3 and 4.
In the subgroup analyses according to the type of radiographic scanner (fixed versus portable scanner), compared with the KTAS, the AI analysis showed higher AUCs for predicting major adverse events in both subgroups. The AI results (both the continuous score and binary results) were independent predictors of both major adverse events, and the combination model showed higher AUCs than the KTAS did in both subgroups (Supplementary Table 3).
In the subgroup analyses according to the chief complaint of the patients, the AI analysis showed higher AUCs than the KTAS did for major adverse events in patients with fever and those with dyspnea, whereas no significant difference was observed in patients with chest pain and those with other chief complaints. The AI results were independent predictors of major adverse events, and the combination model showed higher AUCs than the KTAS did in all subgroups (Supplementary Table 4).
In this study, the result of analysis of CXRs of patients visiting the ED with acute cardiopulmonary symptoms obtained using a commercialized AI tool was an independent predictor of major adverse clinical events (hospitalization, ED revisits, and death in the ED). The AI tool exhibited better performance in predicting major adverse events, compared with the symptom- and sign-based triage scale.
Thus far, many studies have focused on the efficacy and value of AI tools targeting CXRs as assistive tools for radiologists or physicians interpreting CXRs [131415161718192021222324]. Focusing on the ED setting, a previous study reported that an AI tool could accurately identify relevant abnormalities in the CXRs of patients visiting the ED and enhance the sensitivity of interpretations by trainee radiologists [25]. However, the impact of AI as an assistive tool for CXR interpretation on patient management and outcomes may be limited because of the need for the confirmation of AI results by a human reader, relatively narrow margin of accuracy improvement, and limited contribution of a single CXR interpretation to patient management. In a previous randomized controlled trial involving patients visiting the ED with acute cardiopulmonary symptoms [26], the accuracy of CXR interpretation and subsequent patient management did not differ between interpretations with and without AI assistance, underscoring the complexity of effectively integrating AI tools into clinical practice. In this regard, we explored how AI could be more effectively integrated into ED clinical workflows, beyond merely serving as an interpretation assistant. Specifically, using AI analysis to help differentiate between patients who require urgent management and those whose care can be safely deferred could represent an attractive case of use.
In our study, AI analysis of CXRs effectively identified patients with major adverse events due to acute cardiopulmonary diseases, demonstrating higher accuracy than the conventional symptom- and sign-based triage scale. These findings highlight the potential of AI as a triage tool for patients with acute cardiopulmonary symptoms visiting the ED.
A substantial proportion of patients with acute cardiopulmonary symptoms may have diagnoses other than acute cardiopulmonary diseases. In our study, only 42.5% of patients had a clinical diagnosis of acute cardiopulmonary diseases. The predictive accuracy of AI analysis of CXRs for major adverse events due to any clinical diagnosis (AUC, 0.674) was lower than that for major adverse events due to acute cardiopulmonary diseases (AUC, 0.795). However, the AI analysis of CXRs demonstrated higher accuracy than the conventional triage scale did for major adverse events due to any clinical diagnosis. These findings suggest that the AI tool can assist in triaging patients, regardless of the presence of acute cardiopulmonary diseases.
Despite its promising performance in predicting major adverse events, AI analysis of CXRs cannot replace the conventional triage system in the ED. This is because a significant proportion of diseases presenting with acute cardiopulmonary symptoms requiring urgent management may not be detectable on a CXR (e.g., acute coronary artery diseases). Instead, it may serve as a supplementary tool to the conventional triage system. Therefore, we investigated whether the AI analysis result was an independent predictor of adverse events when the KTAS level was adjusted for, and whether the combination of the AI result and KTAS level had a synergetic effect on the prediction. We found that the AI result was an independent predictor of major adverse events regardless of the diagnosis of acute cardiopulmonary diseases. For major adverse events due to acute cardiopulmonary diseases, the combination of the AI analysis and KTAS showed a performance that was better than that of the KTAS but similar to that of the AI analysis alone. Meanwhile, for the prediction of major adverse events due to any diagnosis, the combination model outperformed both the KTAS and AI analysis. Given that CXRs can be obtained relatively easily in the ED, the AI analysis of CXRs can be an effective supplementary tool for triaging patients shortly after their arrival.
In the subgroup analysis by the type of radiographic scanner, the AI analysis showed consistent performance for the prediction of major adverse events, outperforming the KTAS, suggesting that the AI analysis would be effective both in relatively stable patients undergoing chest radiography using fixed scanners and in relatively unstable patients undergoing chest radiography using portable scanners. Meanwhile, in the subgroup analysis according to the chief complaint of patients, the AI analysis outperformed the KTAS in patients with fever or dyspnea, whereas there was no significant difference in performance between these methods in patients with chest pain or other chief complaints. We assume that the reason for this difference might be the relative difficulty in identifying diseases presenting with chest pain (e.g., acute coronary syndrome) on CXRs compared with those presenting with fever or dyspnea (e.g., pneumonia or congestive heart failure).
Our study has some limitations. First, the CXR analysis was conducted retrospectively, independent of actual clinical practice. Second, the diagnosis of acute cardiopulmonary diseases was based on a retrospective review of medical records involving subjective judgment. Third, as this was a single-center study, the generalizability and reproducibility of our results remain uncertain. Finally, patients classified as having a KTAS level of 1 who required immediate resuscitation were excluded from the study.
In conclusion, the analysis of CXRs of patients visiting the ED with acute cardiopulmonary symptoms using a commercial AI tool effectively identified patients who later had major adverse clinical outcomes with a higher accuracy, compared with the symptom- and sign-based triage scale. Further prospective studies are necessary to validate the practical feasibility and value of AI-assisted triage.