Authors: Nathan Chaclas, David VanEenenaam, Vineet Desai, Joey Logan, Scott J. Mahon, John M. Flynn, Apurva S. Shah
Categories: Original Research, Pediatric distal radius fracture, Physeal fracture, Convolutional neural networks, Fracture classification
Source: Journal of the Pediatric Orthopaedic Society of North America
Authors: Nathan Chaclas, David VanEenenaam, Vineet Desai, Joey Logan, Scott J. Mahon, John M. Flynn, Apurva S. Shah
Distal radial fractures (DRFs) are some of the most common pediatric injuries, often involving the physis. Diagnostic accuracy can be challenging for physicians unfamiliar with managing pediatric injuries, with misdiagnosis rates reported as high as 46% [1–3]. These patients frequently do not present to a pediatric orthopaedic surgeon or musculoskeletal radiologist initially. Therefore, the development of tools to classify DRFs by physeal involvement could improve diagnostic accuracy and hence enhance patient safety. We propose to fine-tune convolutional neural networks (CNNs), a class of artificial intelligence models that are highly effective at analyzing and interpreting visual data like radiographs, to classify skeletally immature DRF radiographs by physeal involvement.
DRF radiographs from skeletally immature patients aged 4 to 18 years were identified from a database at a single tertiary-care children’s hospital. All fractures were classified regarding physeal involvement by a pediatric orthopaedic surgeon. We fine-tuned a particularly powerful family of CNNs (EfficientNet) on the dataset and calculated the accuracy, precision, recall, and F1-score of the classification. To interpret the model’s predictions, we used gradient-weighted class activation mapping (Grad-CAM), a technique that visualizes the gradient-weighted class activation maps of the convolutional layers, to highlight regions of the radiographs that the model learned to focus on for classification.
2,103 radiographs comprised of anteroposterior and lateral views from 1,082 patients (655 [60.5%] male, mean 10 ± 5.7 years) were included. There were 425 views of 203 physeal fractures and 1,678 views of 879 nonphyseal fractures. The best model was EfficientNet-B2, which achieved a validation accuracy of 84% and a test accuracy of 84% in identifying physeal fractures. The model had high precision (81%), recall (89%), and F1-score (0.86) on the test set.
Fine-tuning an EfficientNet model can achieve high accuracy in identifying physeal fractures. Furthermore, Grad-CAM provides interpretability and transparency for the model’s predictions, highlighting regions of interest and potential sources of error. Results suggest that CNNs can be a powerful tool for fracture classification and that EfficientNet is a promising architecture for this task. Future work should include expanding the dataset with more diverse and balanced samples and testing the model on different views and fracture patterns.
(1)Patients with physeal injuries often do not present to orthopaedists initially.(2)Tools to assist urgent care and emergency room physicians should be aimed at identifying which fractures need orthopaedic follow-up.(3)Convolutional neural networks are a worthwhile option for this task.(4)Historically, these networks have had difficulty differentiating a fracture from a physis.(5)Fine tuning EfficientNet in this study demonstrates an ability to separate physeal fractures from nonphyseal fractures, which could assist “front line” evaluators in determining the necessity of orthopaedic follow-up.
III, retrospective cohort study
Distal radial fractures (DRFs) are some of the most common injuries in children and adolescents, accounting for about 20% to 30% of all pediatric fractures [1]. The involvement of the physis in these fractures has important implications for the prognosis and treatment of the patients as it may affect the future growth and alignment of the radius [2]. While more than 20% of these fractures involve the physis, the accurate and consistent recognition of physeal involvement is challenging, as displacement or disruption of the growth plate can be difficult to assess, and these patients often do not present to a pediatric orthopaedic surgeon or a pediatric musculoskeletal radiologist for the initial evaluation [5], [6], [7]. Misdiagnosis rates for DRFs have been reported from 8% up to as high as 46% [[3], [4], [5], [6], [7], [8], [9], [10]]. Therefore, the development of tools to assist in the classification of DRFs by physeal involvement should be explored as they have immense potential to improve diagnostic accuracy and patient safety and as misdiagnosis can lead to growth arrest, increasing the risk for ulnocarpal impaction syndrome and injury to the triangular fibrocartilage complex (TFCC) [8,[11], [12], [13]]. Correcting these deformities often requires complex surgical procedures later in life, such as physeal bar resection, corrective osteotomies, or epiphysiodesis, all of which carry their own risks and costs [14].
Recent advances in deep learning and computer vision have enabled the development of convolutional neural networks (CNNs), a class of artificial intelligence (AI) models that are highly effective at analyzing and interpreting visual data like radiographs. These models can perform image recognition and classification tasks with high accuracy and efficiency [15]. Among the various CNN architectures, the EfficientNet family of models are a state-of-the-art type of CNN that have been shown to achieve excellent performance on image datasets while using fewer computational resources [15]. The EfficientNet family is based on a compound scaling method that optimizes the network depth, width, and resolution simultaneously [16]. In this paper, we propose to fine-tune the EfficientNet family to classify skeletally immature DRF radiographs by involvement of the physis.
We hypothesized that fine-tuning EfficientNet would result in high accuracy and sensitivity in classifying skeletally immature DRF radiographs by involvement of the physis. Our study aimed to show the potential of CNNs as a novel and effective tool to help diagnose and manage DRFs in children and adolescents.
After approval from our institutional review board (IRB #24-022240), we conducted a retrospective study using DRF radiographs identified from a database collected within a single large tertiary care children’s hospital. The database included skeletally immature patients between the ages of 4 and 18 years who presented with DRFs between June 2018 and December 2019. We excluded patients who had incomplete or poor-quality radiographs.
The entire database was queried, and all fractures were classified with respect to involvement of the physis. Physeal fractures were defined as those that involved the growth plate of the distal radius, while nonphyseal fractures consisted of bicortical, torus, and greenstick fractures of the distal radial metaphysis. The assignment was based on the ground truth diagnosis in the database made by pediatric orthopaedic surgeons (database managed by A.S.) which served as the gold standard for this study.
Our research team used the EfficientNet CNN architecture to classify the radiographs by physeal involvement. The models range from EfficientNet-B0 to EfficientNet-B7, increasing in complexity as they advance. We used the pretrained weights of the models on the ImageNet dataset as the initial parameters and fine-tuned them on our radiograph dataset. We unfroze all the layers of the models and used the Hyperband algorithm in Keras Tuner to determine the following optimal hyperparameters for each epochs, momentum, learning rate, dropout rate, and L1 and L2 regularization. An epoch is one complete pass through the entire training dataset. Momentum is a technique that allows the model to access previous iterations when training. Learning rate determines how quickly the network weights adjust. Dropout removes a fraction of the inputs during each training cycle. L1 and L2 regularization helps to select features and stabilize the solution. Hyperband selects the top-performing hyperparameter variants out of a preset range and iterates until it finds the most optimal combination [17]. Optimal hyperparameter selection was determined by validation accuracy.
Our research team used a binary cross-entropy loss function and a sigmoid activation function for the output layer. We also applied image augmentation techniques, such as rotation, zoom, shear, and flip, to increase the diversity and robustness of the training data. We used TensorFlow as the deep learning framework and JupyterLab notebooks in Anaconda as the development environment.
The research team elected to split the radiograph dataset into training, validation, and test sets, with a ratio of 20:20. This ratio is a standard convention in machine learning that allows for robust model training while reserving independent datasets for model tuning (validation) and final, unbiased performance evaluation (test). We used the training set to train the models, the validation set to monitor the training process and select the best model, and the test set to evaluate the performance of the models. We compared the performance of the EfficientNet models from B0 to B7 and selected the best model based on the highest F1 score. The F1-score is a measure of a model’s accuracy that considers both precision (the proportion of positive identifications that were actually correct) and recall (the proportion of actual positives that were correctly identified), providing a single score that balances the two. A score closer to 1.0 indicates better overall performance. We then tested the best model with the most optimal hyperparameters on the test set and calculated the accuracy, sensitivity, specificity, precision, recall, and F1-score of the classification. We also plotted the receiver operating characteristic (ROC) curve and computed the area under the curve (AUC) of the model.
To interpret the model’s predictions, we used gradient-weighted class activation mapping (Grad-CAM), a technique that visualizes the gradient-weighted class activation maps of the convolutional layers, to highlight the regions of the radiographs that the model learned to focus on for the classification task.
The database returned 2,103 radiographs comprised of anteroposterior and lateral views from 1,082 patients (655 [60.5%] male) at a mean age of 10 ± 5.7 years after exclusion criteria were applied. This consisted of 425 views of 203 physeal fractures and 1,678 views of 879 nonphyseal fractures.
The results of the classification of the radiographs by physeal involvement using the EfficientNet models are shown in Table 1.Table 1Performance of the EfficientNet models on the validation and test sets.Table 1EfficientNet modelValidation accuracy (%)Validation lossTest accuracy (%)Test lossPrecision (%)Recall (%)F1-scoreB0800.67780.7476810.79B1860.49830.5979880.84B2840.48840.548189****0.86B3870.39840.6478950.86B4900.40830.9174990.85B5890.36840.6577980.86B6900.35780.94701000.82B7900.43840.6481890.86
The best model was EfficientNet-B2 (bold above), which achieved a validation accuracy of 84% and a test accuracy of 84% in distinguishing physeal fractures from extraphyseal fractures. The model had a high precision (81%), recall (89%), and F1-score (0.86) on the test set. The ROC curve of the model is shown in Fig. 1, with an AUC of 0.93, indicating a high discriminative ability of the model.Figure 1ROC curve of the EfficientNet-B2 model on the test set. ROC, receiver operating characteristic*.*Figure 1
The confusion matrix, a table that summarizes the performance of a classification model by showing the counts of correct and incorrect predictions for each class for the best model on the test set is shown in Fig. 2, illustrating the breakdown of true and false positives and negatives in relation to ground truth.Figure 2Confusion matrix of EfficientNet-B2 model on the test set.Figure 2
The Grad-CAM heatmap of the model for a sample radiograph is shown in Fig. 3, which illustrates the regions of the radiograph that the model focused on for the classification task.Figure 3Grad-CAM heatmap of the EfficientNet-B2 model for a sample radiograph. Grad-CAM, gradient-weighted class activation mapping.Figure 3
The blue regions are those which the model determined to be the least important when assigning a class in the final convolutional the left lateral border of the radiograph, the right inferolateral border of the radiograph, and the phalanges. The green regions are those which the model determined to be of intermediate the forearm, the carpal bones, and the right superolateral border of the radiograph. The red regions are those the model determined to be of the highest the distal radial and ulnar region and the distal ulnarmost metacarpals.
In this study, we aimed to fine-tune the EfficientNet family of models to classify DRF radiographs by physeal involvement. This is a clinically relevant task as physeal fractures require different management and follow-up than nonphyseal fractures. For example, a nondisplaced or minimally displaced distal radius torus fracture can be safely managed in a removable splint, but a distal radial physeal fracture should be casted to minimize the risk for loss of reduction. Previous studies have shown that CNNs can achieve high accuracy in identifying and classifying fractures from radiographs, but few to date have focused on the specific problem of determining whether there is physeal involvement [15]. We used a dataset of 2,103 radiographs from 1,082 patients with DRFs and applied Grad-CAM to visualize and interpret the model’s predictions.
We found that our model achieved an overall accuracy of 84%, outperforming all seven other models in the class. While direct F1-score comparisons for human radiologists are not commonly reported in literature, our model’s high accuracy (84%) and AUC (0.93) are comparable to the reported diagnostic accuracies for clinicians in similar tasks [3,4,15]. Furthermore, we showed that Grad-CAM could highlight the regions of interest that the model used for classification, providing some insights into the “black-box phenomenon” often associated with CNNs.
One of the main contributions of our study is the use of EfficientNet, a family of CNNs that scale up the network depth, width, and resolution in a balanced way. EfficientNet has been shown to achieve state-of-the-art performance in various image classification tasks, such as ImageNet and CIFAR-10 [16]. However, to the best of our knowledge, this is the first study that applies EfficientNet to classification by physeal involvement. Compared to other CNN architectures, EfficientNet has fewer parameters and requires fewer computational resources, making it more suitable for clinical applications [18]. Moreover, EfficientNet has a built-in mechanism to adjust the input image resolution according to the available resources, which can improve the model’s robustness and generalization ability.
Work in this area to date has focused on demonstrating the utility of various models to detect DRFs in adults [15]. Fracture detection results here have achieved accuracy above 99% [15]. A recent paper by Zech et al. focused on pediatric wrist fracture detection performed with 88% accuracy [11]. The presence of physes may be a contributing factor for the discrepancy in detection accuracy. Another paper by Aryasomayajula et al. achieved an 85% accuracy with the same detection task in pediatric wrist fractures [19]. Most recently, a step forward for fracture classification occurred in the adult population as Min et al. published their model differentiating intra-articular vs extra-articular DRFs in adults with 81% accuracy [20]. Classification may be more difficult for these models than detection, and classification in skeletally immature patients with open physes may be even more so.
While pediatric orthopaedic surgeons and pediatric musculoskeletal radiologists are uniquely familiar with physeal fracture evaluation, they are often not responsible for the initial evaluation of these injuries. Therefore, these results suggest that our model may be most beneficial as a decision support tool in urgent care and emergency room settings where physeal fractures necessitate an orthopaedic referral and nonphyseal fractures may not. A potential clinical implementation would involve integrating the model into an institution’s picture archiving and communication system (PACS). The AI would automatically analyze pediatric wrist radiographs and flag potential physeal fractures for the reviewing physician, helping to standardize care. Such a tool could help mitigate diagnostic errors, reducing both the number of true physeal fractures that are not referred and the number of false physeal fractures that are, thereby having potential to improve patient safety and reduce unnecessary medical costs [[11], [12], [13],21]. While similar AI applications are being adopted for fractures in general, this specific use case remains to be perfected as even Federal Drug Administration (FDA)-approved models have difficulty with more subtle fracture patterns, such as radial condyle fractures, and little exists to evaluate their physeal performance [13,22,23].
A further contribution of our study is the application of Grad-CAM to address the “black-box phenomenon” inherent to complex AI models. This phenomenon refers to the difficulty in understanding the internal reasoning of a CNN; we can see the input (image) and output (diagnosis), but the decision-making process itself is not transparent. Grad-CAM helps to open this black box by producing a heatmap that highlights the specific pixels in the image the model found most important for its classification. In our study, we used Grad-CAM to generate a representative heatmap for these radiographs. Our research team found that Grad-CAM could accurately localize the physeal region in most cases and that the model’s confidence was correlated with the intensity of the heatmap. Grad-CAM also revealed some cases where the model made incorrect predictions due to confounding factors, such as overlapping bones, or poor image quality. These cases can provide valuable feedback for improving the model’s performance and reliability.
Our study has some limitations that should be acknowledged. First, our dataset was collected from a single institution, which may limit the generalizability of our results to other settings and populations. While we did assign class weights to mitigate the imbalance of our dataset, it was imbalanced as there were more nonphyseal fractures than physeal fractures, which may introduce some bias in the model’s learning process. We treated each radiograph in the model as a unique fracture, which is not entirely representative of the way these injuries are evaluated in the clinical setting. Additionally, while techniques like Grad-CAM visualization help to mitigate the “black-box phenomenon” inherent to CNNs, the risk of overfitting remains. Overfitting is when a machine learning model learns its training data too well, memorizing noise and random details instead of the underlying pattern. CNNs are ideally trained on exceptionally large datasets, with thousands to tens of thousands of images recommended for the most robust results. While image augmentation and inclusion of extra views did help maximize the size of the training dataset, this was still a relatively small pool for the model to learn from. Smaller datasets with significantly complex models such as EfficientNet increase the risk of overfitting, as it is easier for the model to memorize patterns in the training dataset that are outside the intended focus area. While we ensured no data leakage from the training to the test dataset and tracked validation loss and accuracy across models relative to test loss and accuracy, there likely exists some degree of overfitting in the model which is inherent to the nature of work in this area. This is likely what we see happening to some degree with the widened validation-test gaps and recall drift across increasing model complexities. The results of the Grad-CAM heatmap visualization indicate the model has a focus at the distal radius, offering potential reassurance for the high scores on the test set, while the focus at the distal ulnarmost metacarpals likely represents some degree of overfitting preventing the model from reaching perfect scores.
In conclusion, we have demonstrated that fine-tuning an EfficientNet model can achieve high accuracy in classifying DRF radiographs by physeal involvement. We have also shown that Grad-CAM can provide some interpretability and transparency for the model’s predictions, highlighting the regions of interest and the potential sources of error. Our study suggests that CNNs can be a powerful tool for fracture detection and classification and that EfficientNet is a promising architecture for this task. Future work could include expanding the dataset with more diverse and balanced samples, testing the model on different views and modalities of the wrist, incorporating uncertainty estimation and confidence intervals into the model’s output, and evaluating the model’s impact on clinical outcomes and patient satisfaction.
•Bone & Joint Open: An increasing number of convolutional neural networks for fracture recognition and classification in orthopaedics - are these externally validated and ready for clinical application?•Scientific Reports: An automated hip fracture detection, classification system on pelvic radiographs and comparison with 35 clinicians•SICOT-J: Artificial intelligence for fracture diagnosis in orthopaedic X-rays: current developments and future potential
Nathan Chaclas: Writing – review & editing, Writing – original draft, Visualization, Formal analysis, Data curation, Conceptualization. David VanEenenaam: Writing – review & editing, Visualization, Conceptualization. Vineet Desai: Writing – review & editing, Visualization, Conceptualization. Joey Logan: Writing – review & editing, Visualization, Validation, Formal analysis, Conceptualization. Scott J. Mahon: Writing – review & editing, Visualization, Project administration, Data curation, Conceptualization. John M. Flynn: Writing – review & editing, Validation, Conceptualization. Apurva S. Shah: Writing – review & editing, Validation, Supervision, Conceptualization.
The author(s) declare that no patient consent was necessary as no images or identifying information is included in the article.
No funding was received for this work.
All authors have no financial, consultant, institutional, and other relationships that might lead to bias or a conflict of interest.