Authors: Ankur Kayastha, Kirthika Lakshmanan, Michael J. Valentine, Anh Nguyen, Kaushal Dholakia, Daniel Wang
Categories: Clinical Studies, Artificial intelligence, ChatGPT, Clinical guidelines, Lumbar disc herniation, Radiculopathy, Sciatica, Spine
Source: North American Spine Society Journal
ChatGPT is an advanced language AI able to generate responses to clinical questions regarding lumbar disc herniation with radiculopathy. Artificial intelligence (AI) tools are increasingly being considered to assist clinicians in decision-making. This study compared ChatGPT-3.5 and ChatGPT-4.0 responses to established NASS clinical guidelines and evaluated concordance.
ChatGPT-3.5 and ChatGPT-4.0 were prompted with fifteen questions from The 2012 NASS Clinical Guidelines for the diagnosis and treatment of lumbar disc herniation with radiculopathy. Clinical questions organized into categories were directly entered as unmodified queries into ChatGPT. Language output was assessed by two independent authors on September 26, 2023 based on operationally-defined parameters of accuracy, over-conclusiveness, supplementary, and incompleteness. ChatGPT-3.5 and ChatGPT-4.0 performance was compared via chi-square analyses.
Among the fifteen responses produced by ChatGPT-3.5, 7 (47%) were accurate, 7 (47%) were over-conclusive, fifteen (100%) were supplementary, and 6 (40%) were incomplete. For ChatGPT-4.0, ten (67%) were accurate, 5 (33%) were over-conclusive, 10 (67%) were supplementary, and 6 (40%) were incomplete. There was a statistically significant difference in supplementary information (100% vs. 67%; p=.014) between ChatGPT-3.5 and ChatGPT-4.0. Accuracy (47% vs. 67%; p=.269), over-conclusiveness (47% vs. 33%; p=.456), and incompleteness (40% vs. 40%; p=1.000) did not show significant differences between ChatGPT-3.5 and ChatGPT-4.0. ChatGPT-3.5 and ChatGPT-4.0 both yielded 100% accuracy for definition and history and physical examination categories. Diagnostic testing yielded 0% accuracy for ChatGPT-3.5 and 100% accuracy for ChatGPT-4.0. Nonsurgical interventions had 50% accuracy for ChatGPT-3.5 and 63% accuracy for ChatGPT-4.0. Surgical interventions resulted in 0% accuracy for ChatGPT-3.5 and 33% accuracy for ChatGPT-4.0.
ChatGPT-4.0 provided less supplementary information and overall higher accuracy in question categories than ChatGPT-3.5. ChatGPT showed reasonable concordance to NASS guidelines, but clinicians should caution use of ChatGPT in its current state as it fails to safeguard against misinformation.
Keywords: Artificial intelligence, ChatGPT, Clinical guidelines, Lumbar disc herniation, Radiculopathy, Sciatica, Spine
Lumbar disc herniation with radiculopathy is a common and debilitating spinal condition that is complex in its presentation, diagnosis, and treatment. Lumbar disc herniations are most prevalent in the third to fifth decade in life with a male to female ratio of 1 [1]. The incidence of low back pain is estimated to be between 13% and 31%, but with the addition of radicular symptoms, the incidence ranges from 12% to 40% for all etiologies [2]. Clinical management of this condition warrants multidisciplinary care and consideration of various prognostic factors, often necessitating the use of established clinical guidelines.
As defined by the North American Spine Society (NASS), lumbar disc herniation with radiculopathy is characterized by localized displacement of disc material beyond the normal margins of the intervertebral disc space resulting in pain, weakness, or numbness in a myotome or dermatome distribution [3]. Isolated lumbar disc herniation refers to the condition where the soft inner core of an intervertebral disc, known as the nucleus pulposus, herniates through the outer fibrous ring of the disc, known as the annulus fibrosus. This pathophysiology ultimately results in disc degeneration, arthritic changes, and subsequent lower back pain due to associated inflammation [1]. The sequela of radiculopathy occurs when herniated disc material compresses or irritates spinal nerve roots exiting intervertebral foramina. This results in potential sensory and motor deficits, paresthesias, and radiating pain. The presentation of symptoms and their severity depends on several factors including location of the herniated disc, extent of nerve compression, and degenerative changes to adjacent anatomy.
Clinical symptoms include low back pain, leg pain, numbness and tingling, muscle weakness, and in more severe cases, bladder and bowel dysfunction [1]. Physical examination may reveal exacerbation of pain in certain positions or weakness along the distribution of lumbosacral nerve roots. Magnetic resonance imaging (MRI) is the gold standard diagnostic modality in confirming lumbar disc herniation [1]. Treatment plans are complex, but they generally span from conservative pain management and physical therapy in mild cases to discectomy or laminotomy in severe cases.
The diagnosis and treatment of lumbar disc herniation with radiculopathy involves comprehensive decision-making processes optimized for individual patients. Clinical guideline references are available for clinician use along with a plethora of streamlined internet-based resources. In the modern era of medicine, artificial intelligence (AI) tools are increasingly being considered to guide clinical decisions due to rapid response rate and ease of access. Many regulatory bodies disseminate clinical guidelines, but there is a fundamental lack of consolidation of this information. AI may help to mitigate this problem.
Chat generative pre-trained transformer (ChatGPT), developed by OpenAI, is one example of a publicly available advanced language model that uses deep learning techniques and large volumes of textual information to produce human-like responses to language inputs [4]. ChatGPT was trained using vast libraries and databases of information, allowing it to predict an answer to a user's question using complex algorithms. AI models have the capability to transform the healthcare landscape through their profound knowledge bases and functional versatility. In an impressive feat, a recent study showed that ChatGPT performed at or near the passing threshold for all 3 United States Medical Licensing Exams (USMLE) without any specialized prompting [5]. ChatGPT demonstrates reasonable accuracy in its general medical knowledge. When presented with clinical vignettes, it can generate differential diagnoses, suggest diagnostic tests, provide final diagnoses, and recommend patient management strategies based on age, gender, and case acuity [6]. The multitude of uses for ChatGPT remains to be explored, but these achievements speak to its potential applications in the medical field.
ChatGPT may play an assistive role in clinical decision-making regarding lumbar disc herniation with radiculopathy. Analyses comparing ChatGPT against NASS guidelines have been conducted for other spine-related conditions, including thromboembolic prophylaxis in spine surgery [7] and lumbar spinal stenosis [8]. In this analysis, the accuracy of ChatGPT responses was compared to The 2012 NASS Clinical Guidelines for the diagnosis and treatment of lumbar disc herniation with radiculopathy, a set of recommendations derived from extensive medical research and expert consensus.
The main objective of this observational study was to assess accuracy and performance of ChatGPT when compared to NASS guidelines. Institutional review board (IRB) approval was not required for this study, as ChatGPT is a publicly available resource, and no patients were involved. The methodology aimed to compare the responses generated by two versions of an AI model, ChatGPT-3.5 and ChatGPT-4.0, to a published set of guidelines called The 2012 NASS Clinical Guidelines for the diagnosis and treatment of lumbar disc herniation with radiculopathy. The original clinical guidelines by NASS were formatted as clinical questions to which answers were supported by evidence-based research. A systematic process was employed to identify relevant clinical guidelines, categorize them appropriately, and evaluate AI-generated responses to these unmodified clinical questions. Responses from ChatGPT were compared to NASS recommendations and graded on operationally-defined accuracy, over-conclusiveness, supplementary, and incompleteness. To highlight studies referenced in the creation of NASS guidelines, a brief narrative review was conducted for reader supplementation and discussion of ChatGPT concordance.
In conceptualizing this project, AI models considered for analysis included Google Bard, Microsoft Bing, and ChatGPT. ChatGPT was ultimately selected for a number of reasons. First, a majority of Americans reported being aware of ChatGPT as a new publicly available AI tool. Approximately 6-in-10 adults, or roughly 58%, of US adults are familiar with ChatGPT [9]. Second, ChatGPT has demonstrated superior efficacy in interpreting and solving case physiology case vignettes when compared to Google Bard and Microsoft Bing [10,11]. This AI is relevant in the current medical literature and has shown promise in being assistive during clinical workflow [[5], [6], [7], [8],10,11]. Third, the selection of ChatGPT in this study allows for two versions, ChatGPT-3.5 and ChatGPT-4.0, to be directly compared and scored. Comparing an updated version of AI to its predecessor may provide insights on the pace of technological advancements that contribute to increased accuracy and utility of AI responses over time.
A total of twenty-nine clinical guidelines from NASS were initially considered for analysis. Qualitative content analyses were used to generate content categories for the clinical guideline questions. Two authors independently analyzed and assessed the clinical guideline questions to generate content categories. A total of 5 content categories were generated including definition and history, physical findings, diagnostic testing, nonsurgical interventions, and surgical interventions. These categories were adapted from a previous study [8].
To ensure a meaningful comparison between ChatGPT and NASS, a set of exclusion criteria was applied. The questions that were nonconforming to the discussed categories were excluded. For example, questions involving patient functional outcomes or cost-effectiveness of treatment protocols were excluded. Eleven guidelines nonadherent to categories were removed as a result. ChatGPT does not have access to real-time information [4]. This constraint may negatively affect its responses to highly specific and complex questions. Therefore, guideline questions containing multiple subcomponents, comparisons between entities, or references to highly specific or complicated procedures were also excluded. Namely, questions assessing effectiveness of contrast-enhanced fluoroscopic-guided interventional procedures were considered outside the scope of this study. Three additional clinical questions were removed as a result of these criteria. Fifteen clinical guideline questions were included for final data analysis (Fig. 1).
Fig. 1 PRISMA flow diagram illustrating the selection of included NASS guidelines. Exclusion criteria were applied when considering guidelines for analysis.
ChatGPT responses were evaluated for concordance with NASS guidelines using 4 outcome measures. ChatGPT responses were assessed for accuracy, over-conclusiveness, supplementary, and incompleteness [7]. These measures were adopted from a previous study [7]. Accuracy reflected the overall AI response, while the other measures targeted individual subcomponents of responses. Scoring for outcome measures are further described
Fifteen clinical guideline questions from The 2012 NASS Clinical Guidelines for the diagnosis and treatment of lumbar disc herniation with radiculopathy were inputted into ChatGPT-3.5 and ChatGPT-4.0 on September 26, 2023. For each guideline question, the prompt was inputted without modifications into a distinct ChatGPT session to prevent biases and ensure no influence from previous inquiries (Fig. 2). If questions did not generate a relevant response, ChatGPT was prompted again with minimal context added to the question (i.e. specifying that the question pertains to lumbar disc herniation with radiculopathy or requesting an answer in paragraph format without bullet points). ChatGPT-generated responses were then recorded verbatim into Microsoft Word alongside NASS recommendations. Transitional phrasing, duplicate information, and unnecessary verbiage in the AI responses were removed. Each ChatGPT-3.5 and ChatGPT-4.0 response was then evaluated by 2 independent reviewers for accuracy, over-conclusiveness, supplementary, and incompleteness compared to NASS guidelines. Discrepancies in scoring were resolved by a third reviewer. Interobserver reliability was calculated using Cohen's kappa coefficient.
Fig. 2 Example of ChatGPT-4.0 response to a clinical guideline question retrieved from NASS.
This study primarily assessed and compared the language output of ChatGPT to already established information disseminated by NASS. Data were then compiled, analyzed, and visualized in Microsoft Word and Microsoft Excel. ChatGPT-3.5 and ChatGPT-4.0 performance was compared using chi-square tests for the outcome measures. Each chi-square test was conducted using an alpha value of 0.05.
A total of fifteen NASS Clinical Guidelines for the diagnosis and treatment of lumbar disc herniation with radiculopathy were compared against responses generated by both ChatGPT-3.5 and ChatGPT-4.0. Outcome measure scoring for ChatGPT-3.5 and ChatGPT-4.0 responses were detailed (Table 1). A comprehensive list of the NASS guidelines alongside responses from both versions of ChatGPT were recorded (Supplementary Table 1). Among the fifteen responses produced by ChatGPT-3.5, 7 (47%) were accurate, aligning with NASS guidelines without contradictions. Seven (47%) responses were over-conclusive, responding with definitive guidelines where NASS had determined insufficient evidence on the topic. Supplementary information was noted in all fifteen (100%) of its responses, providing additional information beyond NASS guidelines. Six (40%) responses were incomplete, failing to provide relevant details suggested in NASS guidelines. Among the fifteen ChatGPT-4.0 generated responses, 10 (67%) were accurate, 5 (33%) were over-conclusive, 10 (67%) were supplementary, and 6 (40%) were incomplete. The difference in supplementary information (100% vs. 67%; p=.014) between ChatGPT-3.5 and ChatGPT-4.0 was statistically significant. However, accuracy (47% vs. 67%; p=.269), over-conclusiveness (47% vs. 33%; p=.456), and incompleteness (40% vs. 40%; p=1.000) did not show significant differences between ChatGPT-3.5 and ChatGPT-4.0 (Fig. 3). Cohen's kappa coefficient was calculated at a value of 0.966 for interobserver reliability.
Fig. 3 Accuracy, over-conclusiveness, supplementary, and incompleteness of ChatGPT-3.5 and ChatGPT-4.0 recommendations compared to NASS clinical guidelines.
The accuracy of responses from both ChatGPT-3.5 and ChatGPT-4.0 across question categories was also analyzed (Fig. 4). ChatGPT-3.5 and ChatGPT-4.0 yielded 100% accuracy for definition and history (two questions) and 100% accuracy for physical examination (1 question) categories. Diagnostic testing (1 question) yielded 0% accuracy for ChatGPT-3.5 and 100% accuracy for ChatGPT-4.0. Non-surgical interventions (8 questions) had 50% accuracy for ChatGPT-3.5 and 63% accuracy for ChatGPT-4.0. Surgical interventions (3 questions) resulted in 0% accuracy for ChatGPT-3.5 and 33% accuracy for ChatGPT-4.0.
Fig. 4 Accuracy of ChatGPT-3.5 and ChatGPT-4.0 recommendations compared to NASS guidelines stratified by question category.
The results of the comparative analysis showed that ChatGPT-3.5 and ChatGPT-4.0 exhibit differences in performance. ChatGPT-4.0 appeared to be better equipped in accurately responding to questions than ChatGPT-3.5, apparent with higher accuracy percentage in most question categories. These findings align with earlier research on the performance of AI in the medical field. ChatGPT-4.0 has been shown to outperform ChatGPT-3.5 on assessments originally designed for medical students, such as the Medical Knowledge Self-Assessment Program [12]. This difference can be attributed to ChatGPT-4.0 being trained with a substantially larger number of parameters than ChatGPT-3.5, allowing it to comprehend a wider range of queries [13,14]. ChatGPT-4.0 also has larger context windows and a broader base of knowledge due to its updated training data in 2023 as opposed to 2022.
Despite advancements in ChatGPT-4.0, a third of the responses in the data obtained in this comparative analysis still contained inaccurate information. ChatGPT-4.0 outperformed ChatGPT-3.5 in a statistically significant fashion only in terms of one out of four outcome supplementary information. ChatGPT-3.5 added supplemental information to all tested clinical guidelines, whereas ChatGPT-4.0s responses were more conservative. Both models were vulnerable to producing unsupported or irrelevant details, a phenomenon termed “AI hallucination” in the current literature [15]. ChatGPT sometimes generates fabricated data to provide the user with an immediate response, regardless of the content's factual integrity [13,16]. Despite these limitations, ChatGPT may still have potential to be a supplemental source for medical professionals pending future updates and ethical considerations.
A brief narrative review of The 2012 NASS Clinical Guidelines for the diagnosis and treatment of lumbar disc herniation with radiculopathy was conducted to assess ChatGPT performance in the 5 guideline categories for reader supplementation (Table 2).
Both versions of ChatGPT provided accurate answers for the 2 questions that fell under the category of definition and history for lumbar disc herniation with radiculopathy. The AI models were able to accurately describe the condition, specifying that the intervertebral disc herniates beyond anatomical boundaries and elicits radiating numbness or pain along the path of the affected nerve. Although it did not explicitly specify radiation along myotomal or dermatomal distributions as outlined by NASS, it conveyed that radiation often extends into the leg. Nomenclature and classification of lumbar disc pathology informing these NASS guidelines were based on a literature review conducted in 2001 and revised in 2014.
Herniation was defined as a localized or focal displacement of disc material beyond the limits of intervertebral disc space. Adjacent anatomy includes the lateral recess as the portion of the subarticular zone within the vertebra defined by the medial wall of the pedicle where the same numbered nerve root traverses [17]. L4, L5, and S1 radiculopathies can occur as a result of pressure exerted by the herniated disc onto the thecal sac or lumbar nerve roots. Such pressure may result in nerve root ischemia, inflammation, and characteristic radicular pain down the leg. These symptoms and their correlation with the condition may have shaped ChatGPT's response. Additionally, ChatGPT was correct in describing the natural course of the condition, as 85% to 90% of cases are short-lived and regress over time [1]. Both versions of ChatGPT performed best in this category, perhaps due to the lack of clinical judgment required to answer definition-based questions.
One clinical guideline question conformed to the diagnostic testing category. NASS describes that there is relative paucity in high quality studies on advanced imaging. However, the NASS workgroup opinion was that in patients with a history and physical examination findings consistent with lumbar disc herniation with radiculopathy, magnetic resonance imaging (MRI) is considered the most appropriate test for diagnosis [3]. MRI compares favorably to other imaging techniques due to noninvasiveness, lack of ionizing radiation, and high image quality in sagittal and axial planes [18]. Computed tomography (CT) scans may also be appropriate in certain cases, but there is no mention of X-ray utilization within NASS guidelines. ChatGPT-3.5 asserted that X-ray may be used for preliminary imaging, which is incorrect, but ChatGPT-4.0 correctly mentioned its lack of utility in aiding diagnosis.
NASS also posits that while somatosensory evoked potentials (SEPs) can complement cross-sectional imaging, they lack specificity at the level of nerve roots. Pape et al examined validity of sensory nerve SEPs as a diagnostic tool for sensory radiculopathies [19]. The results indicated that the true-positive rate of sensory nerve SEP is higher in patients when sciatic symptoms are present rather than absent [19]. Both versions of ChatGPT failed to mention electrodiagnostics in their responses, leading to a grading of incompleteness.
Physical findings, consisting of one question, was a category that included both history and physical examination findings. ChatGPT-4.0 was much more conservative in its response than ChatGPT-3.5. NASS guidelines emphasized use of Lasegue's sign, straight leg raise, and muscle/sensory testing as recommendations for diagnosis, whereas ChatGPT also mentioned potential diagnostic validity of lumbar range of motion and reflex strength. NASS indicated insufficient evidence for lumbar range of motion and reflex strength. Concurrent with NASS, ChatGPT also reflected a sense of ambiguity. Both versions of ChatGPT did not make definitive claims and were therefore graded as accurate. Vucetic et al employed a stepwise discriminant analysis revealing Lasegue sign and lumbar range of motion as two signs capable of discriminating between ruptured and intact annuli, ultimately impacting choice of invasive therapy [20]. Slump and straight leg raise tests also showed diagnostic validity in patients demonstrating signs of herniations [21]. ChatGPT-3.5 discussed various physical symptoms and imaging modalities irrelevant to the guideline question and NASS recommendations, whereas ChatGPT-4.0 provided a focused response discussing importance of muscle strength and sensory symptoms in concordance with NASS.
The most extensive category was nonsurgical interventions which comprised eight questions. According to NASS, all answers within this category contained some form of insufficient evidence. Both versions of ChatGPT answered approximately half of these questions accurately. NASS recommendations discussed use of TNF-alpha inhibitors, exercise, spinal manipulation, and epidural steroid injections (ESIs) as forms of nonsurgical interventions with varying evidence grades. Most recommendations provided were based on randomized controlled trials consisting of patients with clinical or imaging-diagnosed lumbar disc herniation. Exercise was recommended in the form of workgroup opinion. In one NASS-referenced study, it was found that exercise led to significant pain relief and improved performance in activities of daily living [22]. In terms of medication therapy, a short course of adalimumab (TNF-alpha inhibitor) was shown to decrease leg pain and back pain in patients affected by sciatica [23]. Owing to inconsistent results, spinal traction could not be recommended by NASS [24]. Finally, 1 study showed transforaminal epidural steroid injections were considered superior to both caudal and interlaminar routes. This superiority was attributed to higher levels of complete or partial pain relief in comparison to the other routes [25]. Of note, this injection route was not substantiated as superior by NASS as a whole [3].
Both versions of ChatGPT often made incorrect and definitive claims in this category, recommending use of nonsteroidal inflammatory drugs (NSAIDs), muscle relaxants, and oral corticosteroids as pharmacologic interventions. ChatGPT described physical therapy and exercise as essential components to treatment, perhaps due to the importance of activity as a lifestyle modification in many other spinal conditions. However, ChatGPT was able to appropriately capture the controversial nature of evidence in spinal manipulation and traction in scientific literature. Both versions of ChatGPT also correctly identified the efficacy of epidural steroid injections but utilized unsubstantiated claims to differentiate risks and effectiveness between various injection routes.
Surgical interventions were the last category addressed, containing 3 questions. Surgical outcomes and optimal surgical timing were discussed as topics within related guidelines. ChatGPT performed worst in this category, with ChatGPT-3.5 being incorrect on all questions and ChatGPT-4.0 answering only one correctly. NASS asserts that earlier surgery (within 1 year) was associated with favorable surgical outcomes due to faster rates of pain relief and perceived recovery [26]. Additionally, while Lasegue sign, depression, and sensory pain descriptions were adequate predictors of surgical outcome, pain cognition, and coping strategies were not [27]. ChatGPT incorrectly asserted that pain, functional impairment, failure of conservative treatments, and MRI-confirmed nerve compression predict favorable surgical outcomes. The AI also stated that there is no universal optimal timing for surgery, directly contradicting NASS. Notably, ChatGPT did provide a caveat and suggested evaluation by healthcare providers for accurate prognosis. Although this was a responsible addition to the overall response, this was not considered in grading due to lack of utility in clinical scenarios.
ChatGPT performance was scrutinized appropriately during grading. Although impressive in its knowledge base and fast rate of response, both versions of ChatGPT often made uncorroborated claims that were not evidence-based. ChatGPT is currently unable to access databases of medical literature, which may partially explain its inaccuracies. In its current state, ChatGPT cannot be recommended as an assistive resource in clinical workflow. If ChatGPT is updated to prevent fabrication and enable controlled access to current recommendations disseminated by regulatory bodies, it may have potential in assisting clinical decision-making.
Prior studies have explored the efficacy of ChatGPT in clinical workflow and its ability to make recommendations regarding other spinal conditions. Duey et al conducted a study on ChatGPT's effectiveness at providing recommendations for thromboembolic prophylaxis in spine surgery [7]. ChatGPT proved to be reasonably accurate when responses were compared to NASS guidelines with ChatGPT-4.0 demonstrating an accuracy of 92%, while ChatGPT-3.5 performed with an accuracy of 33%. The AI models were observed to exhibit several limitations regarding vagueness in response, lack of completeness in clinical planning, erroneous citation of sources, and providing recommendations when NASS guidelines deemed evidence to be inconclusive [7].
Another study conducted by Rajjoub et al explored ChatGPT capabilities in diagnosis and treatment of lumbar stenosis [8]. The authors concluded that the AI provided more specific results when compared to NASS guidelines. For instance, ChatGPT provided specific physical therapy exercises and commented on the effects of ancillary treatments on lumbar stenosis. Additionally, it spoke on aspects of bracing, electrical stimulation, and acupuncture, even expanding on pharmacological treatments while NASS guidelines made no such recommendations [8]. It is important to note that the latter study only conducted data analysis utilizing ChatGPT-3.5, whereas the former study conducted their analysis on both ChatGPT-3.5 and ChatGPT-4.0 to provide comparison between versions of AI.
Continued research on ChatGPT's abilities and limitations is necessary to develop a better understanding of its clinical utility before any consideration is given to its incorporation into clinical workflow. Rao et al addressed this notion by analyzing ChatGPT responses to 36 different clinical vignettes [6]. The authors concluded that ChatGPT improves proportionally in clinical decision-making as more clinical information is provided. In other words, the AI was most effective at clinical planning with respect to final diagnosis and specific clinical presentation as opposed to initial or differential diagnoses [6,28].
A full discussion of ethical considerations is beyond the scope of this article; however, it is vital to account for the ethics of integrating AI into healthcare. The widespread nature of AI requires strong privacy protections to keep patient information safe, making the transparency of AI usage in clinical settings paramount [29,30]. Additionally, AI algorithms cannot account for many multifactorial components associated with health, and to prevent disparities in patient care, it is crucial to rectify any inherent biases within AI algorithms [30]. Accountability mechanisms must be established as AI should not serve as a replacement for physician care [29]. If AI is to be effectively employed as a supplementary tool in healthcare settings, it is essential that such models are not only trained extensively in medical knowledge but are also calibrated with stringent ethical and privacy standards. The development of AI systems that adhere to these principles represents a promising direction for future exploration.
This study contains limitations that warrant consideration. ChatGPT performance in generating responses was altered by specificity of the questions posed to it and its adaptation to individual user questions at a timepoint. This made it difficult to assess the consistency of AI responses. While the NASS guidelines provided a broad framework, more precise, and specific inquiries tended to yield improved responses [31]. Of note, ChatGPT was graded in all four outcome measures based on a reference document which has not been revised since 2012. ChatGPT-3.5 and ChatGPT-4.0 were trained with information until 2022 or 2023 respectively, and therefore may have led to discrepancies when compared to the NASS guidelines [4,32]. Scoring of outcome measures may have been influenced by a limited number of observers and observer bias. Finally, clinical guideline categories were formulated subjectively and contained a small sample size which may have affected statistical significance and generalizability of the results.
ChatGPT was able to provide recommendations regarding lumbar disc herniation with radiculopathy with reasonable concordance to NASS guidelines. ChatGPT-4.0 demonstrated overall higher accuracy than ChatGPT-3.5 in question categories. ChatGPT-4.0 provided less supplementary information than ChatGPT-3.5, a statistically significant finding. ChatGPT shows potential as an adjunctive resource to clinical workflow, but more validation research and AI updates are necessary before implementation into the healthcare landscape can be recommended. The processes of AI hallucination and fabrication must be addressed to protect patient safety, and the ethical concerns of utilizing AI within clinical settings must also be reconciled. Once these steps are taken, future iterations of ChatGPT and other forms of advanced AI may have the potential to be a clinical resource when assessing recommendations for various spinal conditions.
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.