Authors: Joon Yul Choi, Doo Eun Kim, Sung Jin Kim, Hannuy Choi, Tae Keun Yoo
Categories: Brief Communication, Corneal diseases, Data processing
Source: NPJ Digital Medicine
Authors: Joon Yul Choi, Doo Eun Kim, Sung Jin Kim, Hannuy Choi, Tae Keun Yoo
This study demonstrates the potential of multimodal large language models in calculating safety indicators and predicting contraindications for laser vision correction. ChatGPT-4 effectively analyzed ocular data, calculated key indicators, generated calculator codes, and outperformed traditional machine learning models and indicators in handling unstructured data and corneal topography. Its modality-independent system enabled efficient and accurate data analysis. Despite longer processing times, ChatGPT-4’s performance highlights its potential as a decision-support tool, offering advancements in improving safety.
The application of generative artificial intelligence (AI) has progressed far beyond simple data processing, now powering software that can augment and convert data into user-specified formats while also creating conversational system^1,2^. In the field of ophthalmology, data analysis methods based on large language models have recently been introduced^3^. Currently, ChatGPT-4, an advanced AI system utilizing a large language model (LLM), can assist in planning retinal surgeries and provide information on basic eye diseases^4,5^. Unlike traditional machine learning (ML) methods, which are limited to analyzing structured data, multimodal AI systems can flexibly process diverse data types by drawing on existing knowledge, offering significant adaptability in utilizing unstructured data.
Several indicators have been developed to assess the safety of laser vision correction procedures such as laser in situ keratomileusis (LASIK) and photorefractive keratectomy (PRK)^6^. Many studies have focused on using these indicators to detect keratoconus, a known contraindication to corneal refractive surgery^7^. However, the manual calculation of these safety indicators can be time-consuming and labor-intensive, making it challenging for clinicians to assess surgical risk by considering factors like age, corneal thickness, ablation depth, and surgical conditions^8^. Ignoring these factors could compromise the safety of the surgery, particularly in LASIK cases where flap creation and high myopia are involved. While AI technologies have been explored for use in vision correction^9,10^, current AI tools remain difficult for clinicians to incorporate into routine surgical evaluations due to their complexity and lack of user-friendliness.
This study investigated the potential of multimodal LLMs to assist clinicians in a real-world setting by analyzing ocular measurement data for laser vision correction (Fig. 1). Specifically, we assessed ChatGPT-4’s capability to automatically calculate key safety indicators, including residual stromal bed thickness (RSB), percentage of tissue ablated (PTA), and the Randleman Ectasia Risk Score System (ERSS)^6^. Additionally, we investigated ChatGPT-4’s ability to autonomously generate a calculator for these indicators without requiring coding, enabling rapid and accurate calculations in clinical practice. Furthermore, we examined whether data from corneal measurement devices and refractive information could be effectively utilized to identify patients contraindicated for laser vision correction surgery. The findings demonstrated that the multimodal LLMs, equipped with optical character recognition (OCR), outperformed traditional ML models in screening for contraindications in vision correction surgery (Fig. 2).Fig. 1Overview of this study.We investigated the ability of ChatGPT-4 to automatically calculate key safety indicators in laser vision correction, including residual stromal bed thickness (RSB), percentage of tissue ablated (PTA), and Randleman Ectasia Risk Score System (ERSS). The system automates decision-support tool development and facilitates clinical implementation by providing detailed analysis and efficient, multimodal data integration.Fig. 2Various corneal measurement modalities and comparison of data processing methods between ChatGPT-4 and traditional machine learning software.The figure illustrates modality-free, OCR-based automatic data entry enabled by ChatGPT-4, allowing seamless analysis across different devices, versus modality-dependent, manual data entry required for traditional machine learning models, which necessitate device-specific training and input handling.
Ocular data from 136 eyes of 68 eligible surgery patients (44 LASIK and 24 PRK) and 64 eyes of 32 contraindicated patients were analyzed. Patients’ manifest refraction, corneal thickness, and corneal shape were entered into ChatGPT-4 using prompts without requiring specific alignment (Supplementary Table 1). As depicted in Fig. 3, ChatGPT-4 successfully processed ocular data for all patients, accurately inferring units for each parameter without prior information. The calculated values for ablation depth, RSB, PTA, and ERSS matched those obtained through manual expert calculations (Table 1). No significant statistical differences were observed among manual calculations, ChatGPT-4, and Gemini Advance for both eligible and contraindicated groups. Minor discrepancies arose from data rounding methods, but no calculation errors or hallucinations were detected. In contrast, LLAMA-3 frequently made errors in numerical operations, resulting in substantial discrepancies across all calculated indices when compared to manual calculations (Supplementary Fig. 1). ChatGPT-4 demonstrated versatility by accurately processing data in various formats, including images, tables, and text, while also providing detailed explanations for its calculations. However, a notable limitation was the extended time required to respond to each query, often exceeding 10 seconds (Supplementary Movie 1).Fig. 3Screenshots of data input, questions, and answers in ChatGPT-4 for laser vision correction.a Input data. b Ablation depth and residual stromal bed (RSB) thickness calculations. c Randleman Ectasia Risk Score System (RESS) calculations. d Percentage of tissue ablated (PTA) calculations.Table 1Comparison of indicator values related to laser vision correction calculated manually and using ChatGPT-4Manual calculation in a clinicChatGPT-4 calculationGemini AdvanceLlama-3P value for ChatGPT-4P* value for Gemini AdvanceP* value for Llama-3Eyes eligible for surgery Ablation depth (μm)62.45 ± 20.7162.46 ± 20.6662.44 ± 20.6633.98 ± 24.840.7190.264<0.001 RSB for LASIK (μm)370.83 ± 32.36370.83 ± 32.32370.84 ± 32.33399.31 ± 36.470.8210.353<0.001 RSB for PRK (μm)435.83 ± 32.37435.83 ± 32.32435.84 ± 32.32463.68 ± 34.810.7280.445<0.001 PTA for LASIK (%)33.03 ± 3.8833.03 ± 3.8733.03 ± 3.8828.03 ± 4.270.6330.394<0.001 PTA for PRK (%)11.29 ± 3.7311.29 ± 3.7211.30 ± 3.726.06 ± 4.430.6250.981<0.001 ERSS (points)2.05 ± 1.682.05 ± 1.682.03 ± 1.691.74 ± 1.051.0001.000<0.001Surgical contraindications Ablation depth (μm)93.93 ± 24.4394.02 ± 24.4493.92 ± 24.4445.86 ± 30.720.4590.459<0.001 RSB for LASIK (μm)305.86 ± 35.00305.77 ± 35.03305.85 ± 34.99353.91 ± 41.190.5430.216<0.001 RSB for PRK (μm)370.85 ± 35.00370.77 ± 35.02370.83 ± 35.00418.92 ± 41.190.5110.497<0.001 PTA for LASIK (%)41.25 ± 5.1641.26 ± 5.1741.25 ± 5.1732.01 ± 6.380.5870.610<0.001 PTA for PRK (%)18.11 ± 4.8518.13 ± 4.8418.12 ± 4.858.87 ± 6.100.7470.340<0.001 ERSS (points)3.63 ± 2.123.63 ± 2.113.62 ± 2.132.90 ± 1.780.9100.992<0.001ERSS* Randleman Ectasia Risk Score System, PTA percentage of tissue ablated, RSB residual stromal thickness.* The comparison analysis was conducted using the Wilcoxon signed-rank test to evaluate differences between the values and manual calculations.The calculations were done using data obtained from 136 eyes of 68 myopic patients and 64 eyes of 32 contraindicated patients.
ChatGPT-4 demonstrated the capability to generate a custom calculator for refractive surgery indicators based on prompt instructions (Fig. 4). The calculator, developed in Hypertext Markup Language (HTML), is universally compatible with all operating systems. ChatGPT-4 successfully produced the calculator HTML code in response to straightforward prompts, and the complete code is provided in Supplementary Note 1 (accessible at https://taekeuntoo.github.io/LVC_calc/). This task was also possible using an image-based ERSS table (Supplementary Table 2). This calculator can be further customized by individual vision correction centers through tailored prompts to meet specific clinical needs. However, Gemini Advance exhibited recurrent hallucinations in the ERSS formula, leading to the generation of inaccurate calculators.Fig. 4Creating a safety indicator calculator using ChatGPT-4.a Prompt and response screen instructing to create a calculator. b Screenshot of the resulting calculator developed using HTML, which can be executed in a web browser (available at https://taekeuntoo.github.io/LVC_calc/).
We compared the performance of the HTML-based ERSS calculator, a ML ensemble model (Supplementary Fig. 2), and ChatGPT-4 (multimodal analysis incorporating corneal measurements) in predicting contraindications for corneal laser surgery. Prior to data entry, ChatGPT-4 was provided with the center’s initial screening criteria for refractive surgery, including defined thresholds for refraction and corneal thickness (Supplementary Table 3). As illustrated in Fig. 5, ChatGPT-4 effectively analyzed corneal topography images, accurately detecting early-stage keratoconus or subclinical ectasia. It delivered clear and explainable insights by identifying key diagnostic features from Pentacam images and providing a detailed rationale for its conclusions. Additionally, ChatGPT-4 effectively distinguished cases where LASIK surgery was feasible (Supplementary Fig. 3) from those where it was contraindicated (Supplementary Fig. 4), utilizing Pentacam imaging and optometric data to deliver comprehensive explanations that differentiated normal findings from pathological abnormalities.Fig. 5Example of ChatGPT-4 analysis of Pentacam results for detecting contraindications in corneal laser surgery.a Subclinical ectasia. b Keratoconus. The green lines highlight the features used by ChatGPT-4 for the analysis, while the red lines represent its conclusion.
As shown in Fig. 6, ChatGPT-4 demonstrated superior diagnostic performance for screening LASIK candidates, with an area under the curve (AUC) of 0.977, compared to the ML ensemble (AUC = 0.930), ERSS (AUC = 0.788), and PTA (AUC = 0.897). The ROC curve further illustrates ChatGPT-4’s superior sensitivity (98.4%) and specificity (97.1%) (Table 2). ChatGPT-4’s modality-free, OCR-based approach enabled fast, accurate analysis of corneal topography images without manual data entry. Data entry time was significantly shorter for ChatGPT-4 compared to the ML ensemble model (P < 0.001). While ChatGPT-4 had a longer mean calculation time (27.02 seconds) compared to the ML ensemble (1.83 seconds) and ERSS (instant), its higher accuracy and detailed result interpretation make it a viable tool for clinical decision-making regarding corneal surgery contraindications. Statistical analysis revealed significant differences between ERSS and ChatGPT-4 (P < 0.001 for most metrics), while no significant differences were found between the ML ensemble and ChatGPT-4 in terms of accuracy (P = 0.573), sensitivity (P = 0.355), or specificity (P = 0.951). To confirm ChatGPT-4’s modality-free performance, we validated it using images from external literature^11–13^. ChatGPT-4 successfully analyzed corneal measurements from devices such as Pentacam, CASIA2, NIDEK OPD III, RTVue, and Galilei G4, accurately diagnosing keratoconus, a contraindication to surgery (Supplementary Fig. 5).Fig. 6ROC curve comparison between ChatGPT-4, ML ensemble, ERSS, and PTA for detecting contraindications in corneal laser surgery.AUC, area under curve; CI, confidence interval; ERSS, Randleman Ectasia Risk Score System; ML machine learning; PTA, percentage of tissue ablated; SE, standard error.Table 2Comparison of accuracy, sensitivity, specificity, data entry time, and calculation time between ERSS calculator, machine learning ensemble model, and ChatGPT-4 to predict corneal laser surgery contraindicationsERSS calculator (Web app)ML ensemble with manual data entryChatGPT-4 using corneal topography imagesP value^a^P value^b^Accuracy (%, 95% CI)78.5 (72.2 – 83.9)92.0 (87.3 – 95.4)97.5 (94.3 – 99.2)0.0430.573Sensitivity (%, 95% CI)54.7 (41.7 – 67.2)82.8 (71.3 – 91.1)98.4 (91.6 – 99.9)0.0050.355Specificity (%, 95% CI)89.7 (83.3 – 94.3)96.3 (91.6 – 98.8)97.1 (92.6 – 99.2)0.5310.951Mean data entry time (seconds)12.23 ± 1.1854.20 ± 4.2114.42 ± 1.01<0.001<0.001Mean calculation time (seconds)0.0 ± 0.01.83 ± 0.2727.02 ± 2.83 (Including result text output time)<0.001<0.001ProsSimple and fast.Relatively fast calculation time and high accuracy.High accuracy. Analyzes topography result images based on OCR and deep learning, interprets and explains results, and is not limited to specific corneal measurement equipment.--ConsLow accuracy.Optimized only for Pentacam equipment, requires manual data entry, and lacks result explanations.Longer calculation and result output time.--CI confidence interval, ERSS Randleman Ectasia Risk Score System, MLmachine learning, OCR optical character recognition, PTA percentage of tissue ablated^a^A comparison analysis between ERSS calculator and ChatGPT-4.^b^A comparison analysis between ML ensemble model and ChatGPT-4.
This study demonstrated that a multimodal LLM, ChatGPT-4, can assist clinicians in calculating key safety indicators for laser vision correction. Gemini Advance also showed strong computational performance for these indicators. However, Llama-3, a smaller-scale model, exhibited frequent calculation errors, significantly limiting its clinical applicability. The complexity of calculating the ERSS makes it particularly suited for automation, and the LLM used in this study facilitated this process efficiently. Utilizing corneal measurement images, optical character recognition (OCR), and inference based on pre-existing knowledge, ChatGPT-4 demonstrated superior accuracy in analyzing surgical candidates compared to traditional ML systems, even when handling unstructured data. While data processing and result output required additional time, the workflow remained efficient and user-friendly. Moreover, ChatGPT-4 offered interpretable results, providing detailed explanations that enhanced the clarity and utility of the analysis. Generally, clinicians can rely on this study’s findings, especially the recommendation that an ERSS score of 4 or higher and a PTA of 40% or greater are high-risk indicators requiring caution^14,15^. ChatGPT-4’s calculations were as accurate as those performed by experts, and it even generated a functional calculator for clinical use without the need for coding, representing a significant advance in AI application in medicine.
ChatGPT-4’s ability to calculate various laser refractive surgery indicators based on simple prompt instructions illustrates its potential as a versatile AI tool for clinicians. Additionally, its capability to customize formulas through commands allows vision correction centers to tailor calculations to their specific requirements, accommodating variations in laser and measurement equipment across institutions. This adaptability addresses the challenge of developing a unified AI system, which has been hindered by institutional biases and differing surgical approaches. Although previous attempts have been made to address such biases in ML applications, a standardized protocol has yet to be established^16^. With ChatGPT-4, institutions can utilize their own data and expertise to create new indices and develop customizable calculators for clinical use. Importantly, this study showed that corneal and data analysis is possible regardless of the measurement equipment used, thanks to the multimodal capabilities of ChatGPT-4. This flexibility overcomes the limitations of existing ML systems, which often require standardized data and are restricted to specific equipment, marking a significant breakthrough in AI’s role in medical practice. While the explainability of traditional ML techniques is often limited to identifying relevant factors^17^, ChatGPT-4 goes further by providing specific rationales for its results, offering users an opportunity to review and understand the outcomes more thoroughly.
ChatGPT-4 also holds promise for educational purposes, as it provides detailed, user-friendly information on foundational knowledge and the analysis process^18^ It is also accessible on both mobile and desktop platforms. In real-time clinical settings, ChatGPT-4 could be utilized as an interactive decision-support tool, allowing clinicians to engage with the model throughout the patient evaluation process. For example, clinicians could ask follow-up questions, refine input data, and receive personalized guidance based on evolving clinical information. This would enable more comprehensive, ongoing interactions that go beyond static calculations, ultimately enhancing clinical decision-making in more complex or changing situations. Future work should focus on integrating ChatGPT-4 into dynamic clinical workflows to fully leverage its capabilities in providing real-time, context-specific support. Additionally, the HTML-based calculator created by ChatGPT-4 is universally accessible, making it highly convenient for clinicians. This tool addresses limitations of existing ML methods, which often require complex configurations, manual data entry, and significant setup time, further enhancing its practical value in clinical settings.
It is essential to recognize that the indicator values calculated by ChatGPT-4 are for reference purposes only, and ChatGPT-4 cannot make full medical decisions. While it offers detailed domain knowledge, it cannot confirm clinical outcomes or decisions, as it emphasized in conversations that its results are only reference points. The final evaluation of risks associated with refractive surgery must be performed by a qualified expert. Its ability to calculate various indices for vision correction surgery makes it a valuable decision-support tool for novice clinicians. However, it is important to clarify that ChatGPT-4 is intended to assist clinicians rather than act as a primary decision-making tool, particularly for those new to the field. Final clinical decisions must always be made by qualified healthcare professionals, ensuring compliance with legal and regulatory standards. Additionally, caution is required, as hallucinations may occur if incorrect information is provided in the prompt. For example, there are different versions of ERSS, and if ChatGPT-4 is not clearly instructed, it may combine data from various versions, leading to inaccurate outputs. However, no other hallucinations occurred during this study.
In this study, we evaluated other LLMs, including LLAMA-3 and Gemini Advance, to benchmark ChatGPT-4’s performance. LLAMA-3 exhibited significant calculation errors, while Gemini Advance struggled with formula accuracy and occasional hallucinations. Neither matched ChatGPT-4’s diagnostic performance, explainability, or ability to handle multimodal inputs, highlighting ChatGPT-4’s unique advantages and the need for further optimization in competing models.
While ChatGPT-4 demonstrated robust performance in calculating safety indicators and predicting contraindications for laser vision correction, it is important to recognize its limitations as a general-purpose LLM not specifically designed for medical applications. Its training does not prioritize medical accuracy or domain-specific knowledge, which may pose challenges in specialized fields like ophthalmology, where precise calculations and interpretations are critical. Additionally, both ChatGPT-4 and Gemini Advance are constrained by their commercial nature and associated usage fees, potentially limiting accessibility, particularly in low-resource settings where such tools could have the greatest impact. This cost barrier is especially significant given the study’s aim to support less experienced clinicians who may benefit most from these decision-support systems. To address these limitations, future research should explore the use of open-source or lightweight LLMs as cost-effective alternatives and consider fine-tuning models like ChatGPT-4 with medical datasets to improve their reliability and applicability in clinical contexts^19^.
This study has certain limitations, including its retrospective design. Future prospective studies will be necessary to assess the clinical advantages of using ChatGPT-4 in vision correction clinics. Additionally, the proposed approaches were not validated across multiple centers, which may limit the generalizability of the results. However, since the methods for calculating safety indicators are standardized across institutions, external validation of the calculation process may not be essential. Furthermore, we conducted external validation using corneal measurement data from open access literature, confirming that ChatGPT-4 successfully analyzed results from various devices. This suggests robustness in ChatGPT-4’s application to refractive surgery data across different clinical settings.
In conclusion, ChatGPT-4 offers clinicians a highly convenient tool for calculating the safety indicators of laser vision correction, facilitating more comprehensive evaluations. It demonstrated superior performance compared to existing ML systems in screening contraindications, offering greater convenience, interpretability, and accuracy. With its capability to handle unstructured data and generate calculators without the need for prior training or coding, ChatGPT-4 has the potential for widespread adoption across vision correction institutions. Additionally, it can be used for both decision support and educational purposes, offering clear explanations and insights into the calculation process. Future work should focus on conducting prospective, comparative studies involving multimodal LLMs to further validate and expand these findings. Ultimately, we anticipate that AI tools like ChatGPT-4 will significantly contribute to improving the safety and precision of laser vision correction for patients. As AI systems evolve, ensuring compliance with data security standards and transparent data handling processes will be critical for clinical adoption.
We studied a retrospective data analysis which examined the ocular measurement data from B&VIIT Eye Center to screen the candidates for corneal laser refractive surgeries^9^. This study included patients who underwent LASIK or PRK for myopia correction, as well as those deemed unsuitable for surgery due to contraindications, between April and June 2022. This period was selected due to the availability of complete, well-organized electronic charts for all surgeries and relevant indicators. All patients underwent preoperative measurements of manifest refraction, corneal tomography, and central corneal thickness using a Pentacam Scheimpflug device (Oculus GmbH, Wetzlar, Germany). PTA and ERSS were calculated by expert ophthalmologists to evaluate the suitability of surgery^9^. Patients with incomplete examinations or missing data were excluded. The study protocol was approved by the Ethics Committee of the Korea National Institute for Bioethics Policy (P01-202212-01-029) and followed the Declaration of Helsinki. Individual consent was not required as this was secondary data analysis with anonymized patient information in data. All data input into ChatGPT-4 was anonymized and personal information was removed.
The prompts used in this study are detailed in Supplementary Table 1. Case descriptions, including age, preoperative manifest refraction, corneal thickness, corneal shape (topography), optical zone, and flap thickness, were provided to ChatGPT-4. We then requested calculations for ablation depth and postoperative RSB for both LASIK and PRK using the Munnerlyn Formula. Next, we asked ChatGPT-4 to calculate PTA for both LASIK and PRK and ERSS for LASIK. Importantly, the Munnerlyn Formula, PTA, and ERSS calculation methods were not provided to ChatGPT-4 in advance; these were inferred from the prompts. Given the variation in ERSS criteria across the literature, specific criteria were outlined in the prompt. Finally, ChatGPT-4 was instructed to generate an HTML-based calculator using the calculated values. The calculation formulas were not explicitly provided, allowing the calculator to be created contextually.
To validate the results, we compared the values calculated by ChatGPT-4 with those manually computed by experts, as well as with outputs from the Gemini Advance system and LLAMA-3 (Llama-3.1-70B-Instruct)^20^. Gemini Advance was accessed via its official service homepage, while LLAMA-3 was utilized through the Hugging Face platform at https://huggingface.co/chat/. A new conversation was created for each case, and ablation depth, RSB (LASIK and PRK), PTA (LASIK and PRK), and ERSS values were recorded. Manual calculations were performed by skilled professionals using medical records. Paired t-tests were conducted to compare the average and bias of the calculated values.
ChatGPT-4 was instructed to develop a calculator for safety indicators using HTML. The instructions included calculating ablation depth, RSB for LASIK and PRK, PTA for LASIK and PRK, and ERSS, following the same formulas used earlier. The HTML code allows the calculator to function as a webpage, executable in any web browser. Specific input, output, and design details were provided in the prompts, with the final HTML file designed for ease of use in a clinical setting. To improve user interface convenience, an additional prompt was added to specify the calculator’s design. The code generated by ChatGPT-4 was saved as a single HTML file and executed in a web browser.
As shown in Fig. 2, we compared ChatGPT-4’s multimodal analysis capabilities with a traditional ML model for screening LASIK candidates. The ML model, developed in a previously published study^9^, is an ensemble-based algorithm with a user interface detailed in Supplementary Fig. 2. For the ML model, ocular biometric data and Pentacam measurements were manually entered to standardize the dataset. In contrast, ChatGPT-4 processed corneal topography images by dragging the Pentacam images into the chat window, along with additional refractive information provided through prompts. The center’s predefined safety screening criteria, which specify topography classification, a postoperative corneal thickness of at least 380 μm, and a RSB of at least 280 μm, were entered into ChatGPT-4 prior to evaluating individual cases (Supplementary Table 3). Given the interactive nature of LLM system, the inference process may vary for each case, influenced by the preceding conversation or analysis. Therefore, each case was analyzed by starting a new session and performing a pre-information task following the prompt protocol.
We conducted a comparative analysis of three approaches—ERSS calculator (Web app), the ML ensemble model with manual data entry, and ChatGPT-4 using corneal topography images—for predicting contraindications in corneal laser surgery. Each method was evaluated for accuracy, sensitivity, specificity, data entry time, and calculation time. Two researchers (JYC and TKY) participated in the experiment, entered the required data, and measured execution times. While ERSS and ML ensemble models required manual data entry, ChatGPT-4 utilized an OCR-based, modality-free system to automatically extract and analyze corneal topography images. Diagnostic performance was assessed through ROC curves, with the AUC calculated for each model. Statistical significance between methods was determined using chi-square tests for accuracy and pairwise comparisons for time data, comparing ERSS with ChatGPT-4 and ML ensemble with ChatGPT-4. Mean data entry and calculation times were also recorded to evaluate workflow efficiency.
To further validate ChatGPT-4’s modality-independent analysis capabilities, we conducted additional tests using data from publicly available literature without restrictions on analysis. Specifically, we assessed whether ChatGPT-4 could accurately detect keratoconus, a key contraindication for laser vision correction surgery^11–13^. This analysis was conducted on a test set of corneal topography images from multiple devices, ensuring that ChatGPT-4’s evaluation was modality-agnostic.
Supplementary Materials Supplementary Movie 1