Patient-Centered Research Through Artificial Intelligence to Identify Priorities in Cancer Care

Authors: Jiyeong Kim, Michael L. Chen, Shawheen J. Rezaei, Mariana Ramirez-Posada, Jennifer L. Caswell-Jin, Allison W. Kurian, Fauzia Riaz, Kavita Y. Sarin, Jean Y. Tang, Steven M. Asch, Eleni Linos

Abstract

This case series uses artificial intelligence and natural language processing to analyze a large dataset of patient messages to inform patient-centered research, counseling, and quality improvement initiatives in cancer care.

Introduction

Patient-centered research is critical for bridging the gap between research and patient care, ensuring clinically meaningful outcomes.^1^ Despite long-standing efforts to include patient perspectives in health research, traditional methods face considerable barriers, including high resource demands and inconsistent patient engagement.^2,3^ Over the past decade, patient portal messaging within electronic health record (EHR) systems has become a primary means for patients to communicate their clinical concerns to clinicians.^4^ The COVID-19 pandemic has further amplified the use of these EHR-based communications, doubling the volume.^5,6^

Collecting extensive patient input is challenging and resource intensive, but patient portal messages offer a valuable and underused resource for real-time identification of patient concerns. Natural language processing (NLP), a probability-based language model, can extract key information from vast text datasets.^7,8,9,10^ Advances in artificial intelligence (AI) have propelled the use of AI-based NLP in research, enabling the analysis of large volumes of patient-generated data, such as social media and patient forums.^11,12,13,14^ Large language models (LLMs) have demonstrated impressive performance in various research tasks, including providing peer-review article feedback and assisting in patient selection for clinical trials.^15,16^

Using a large database of secure messages from patients with cancer, we leveraged a framework of AI-enabled NLP to prioritize patient-discussed issues and generate patient-centered research topics. We further validated these topics with domain experts. This pilot case series demonstrates how AI-enhanced NLP can prioritize patient concerns, thereby informing patient-centered research, counseling, and quality improvement initiatives in cancer care.

Methods

Data Source and Study Design

We obtained deidentified patient portal messages from individuals with breast or skin cancer (melanoma, basal cell carcinoma, or squamous cell carcinoma) defined using International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, codes from Stanford Health Care and 22 affiliated centers over July 2013 to April 2024. Only messages labeled as a patient medical advice request (PMAR) routed to oncology or dermatology were included. Patient characteristics were also collected from the EHR system, including sex and race and ethnicity (Asian, Black, Hispanic, Native American or Pacific Islander, White, race other than those listed, and unknown). Race and ethnicity were reported to present the distribution of patient demographics.

The institutional review board at Stanford University approved the study and granted waiver of informed consent owing to retrospective use of deidentified data. We followed Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines.

Two-Staged Topic Modeling

To identify patients’ clinical concerns, we analyzed the secure messages using a 2-staged unsupervised NLP topic model. This process used BERT (bidirectional encoder representations from transformers) and BIRCH (balanced iterative reducing and clustering using hierarchies) techniques.^17,18^ Preprocessed messages were converted into sentences with precalculated embedding and simplified.^19^ Similar topics were categorized via zero-shot clustering using cosine similarity scores. We further refined these clusters with a new BERTopic model, applying BIRCH algorithm, principal component analysis, and incremental fitting techniques for process efficiency (eMethods 1 in Supplement 1).^20^ This analysis highlighted the top 5 clinical concerns systematically selected based on exclusion criteria for both breast and skin cancer groups, excluding administrative issues (eTable 1 in Supplement 1).

AI to Generate Research Topics

Using an LLM (ChatGPT-4o [OpenAI]; April 2024), we generated research topics to address patient concerns. We used multiple prompt-engineering strategies to provide contextual background, role prompting, directive commanding, expertise emulation, and zero-shot chain of thought.^21,22^ The AI performed multilevel tasks, including knowledge interpretation and summarization (eg, interpreting NLP-defined topics regarding patients’ clinical issues), generation (eg, generating research ideas corresponding to patients’ issues), self-reflection (eg, reflecting suggestions), and self-reassurance (eg, reassuring the meaningfulness and novelty after the article search) (eMethods 2 and eTable 2 in Supplement 1).

Evaluation of AI-Generated Research Topics

Three breast oncologists (J.L.C., A.W.K., F.R) and 3 dermatologists (K.Y.S, J.Y.T., E.L.) assessed their agreement if LLM-interpreted topics were representative of patient concerns using a 5-point Likert scale (1 representing agree to 5 representing disagree) and the meaningfulness and novelty of the AI-generated research topics using a 5-point Likert scale (1 representing exceptional to 5 representing poor), adapted from grant-review scoring. These experts had between 10 and 30 years of clinical practice and extensive research experience. The meaningfulness and novelty scores were averaged, and standard deviations were calculated to gauge assessor agreement.^23^ Additionally, 2 researchers (S.J.R. and M.R.P.) independently conducted literature searches to confirm the novelty, and a third researcher (J.K.) confirmed. The search concluded if the terms ceased to appear relevant or on reaching the tenth page of Google Scholar (eTable 3 in Supplement 1).

Statistical Analysis

We assessed the correlation between the 2 novelty measures using Spearman rank correlation test. AI-drafted research topics and their evaluations are summarized in eTable 4 in Supplement 1. We computed 2-sided *P *values and determined the statistical significance at P < .05 using Python, version 3.10 (Python Software Foundation).

Results

Study Population Characteristics

Among the 25 549 patients analyzed (10 665 with breast cancer and 14 884 with skin cancer), 11.7% were Asian, 1.0% were Black, 0.7% were Native American or Pacific Islander, 77.2% were White, 6.8% were a race other than those listed, and 2.6% were unknown, with 5.0% being of Hispanic ethnicity. Patients with breast cancer were predominantly White (61.1%) and female (98.6%), while patients with skin cancer were largely White (88.7%) with a balanced sex distribution (51.0% male and 49.0% female) (eTable 5 in Supplement 1).

Patient Message Characteristics

A total of 44 984 615 unique message threads from 2013 to 2024 were identified across 14 672 401 patients (PMAR, 32.6%). This study focused on 614 464 PMARs: 474 194 from patients with breast cancer and 140 270 from patients with skin cancer (Figure).

Patients’ Clinical Concerns

The Box presents the primary clinical concerns of patients with cancer interpreted by AI using NLP-generated key words. Domain experts mostly agreed with LLM’s interpretation of patient concerns (breast mean [SD], 1.00 [0.00]; skin mean [SD], 1.33 [0.94]).

Assessment of AI-Generated Research Topics

The overall meaningfulness score was lower than the novelty score (lower score means higher quality) in both breast cancer (meaningfulness: mean [SD], 3.00 [0.50]; mean [SD], 3.29 [0.74]) and skin cancer (meaningfulness: mean [SD], 2.67 [0.45]; mean [SD], 3.09 [0.68]) (Table). For breast cancer, meaningful and novel topics (when score was lower than mean) (1) interdisciplinary approach to managing dental health in breast cancer care (meaningfulness: mean [SD] score, 2.33 [1.15]; mean [SD] score, 2.33 [0.58]) and (2) evaluating the efficacy of hepatoprotective agents in preventing liver damage during breast cancer treatment (meaningfulness: mean [SD] score, 2.33 [1.15]; mean [SD] score, 3.00 [1.00]). For skin cancer, topics included (1) development and evaluation of a patient-centered digital tool for postsurgical wound care (meaningfulness: mean [SD] score, 2.33 [1.15]; mean [SD] score, 2.33 [0.58]) and (2) impact of patient education on Efudex (fluorouracil) treatment adherence and outcomes (meaningfulness: mean [SD] score, 1.67 [0.58]; mean [SD] score, 2.33 [1.53]).

One-third of the AI-suggested research topics were highly meaningful and novel when both scores were lower than the average (5 of 15 for breast cancer and 6 of 15 for skin cancer). Two-thirds of AI-generated topics were novel (10 of 15 for breast cancer and 11 of 15 for skin cancer). Expert novelty scores had a positive correlation with the absence of existing literature, though not conclusively (ρ, 0.28; 95% CI, −0.37 to 0.36; P = .13).

Discussion

This study evaluated an LLM’s capability to generate patient-centered research topics and assessed the meaningfulness and novelty of these AI-generated topics. Approximately one-third of the proposed research topics were highly meaningful and novel, and two-thirds were novel for both breast and skin cancer. These findings suggest that using AI/NLP for creating research questions is a promising method to advance patient-centered research by addressing patients’ priorities.

Collecting patient perspectives on a large scale is challenging due to the resource-intensive nature of qualitative interviews. Additionally, patient priorities evolve, complicating the identification of important patient-reported concerns when designing new research studies. This AI-enabled NLP pilot study offers a quantitative and repeatable method to identify key patient concerns, thereby bridging patient issues with health research. By analyzing 614 464 unique messages, we could define the top clinical issues, highlighting current knowledge gaps and generating scientifically meaningful and novel research topics informed by patient input.

Limitations

This study has limitations, including generating 30 research questions in 2 specialties, which may not generalize to other conditions or populations. We focused on clinical issues, excluding administrative support needs. Experts were from a single institution, which may introduce bias. Further studies should include larger samples, diverse specialties, and broader assessors, and consider exploring using representative messages or more recent issues. Evaluating patient perspectives on AI-generated topics and fostering collaboration with investigators and funding agencies is essential for identifying truly important issues.

Conclusions

This case series demonstrates that an AI/NLP–based framework can systematically prioritize patient concerns to guide patient-centered research despite the inconsistent quality by topic. Overall, we observed that AI-generated research topics could meaningfully guide future health research directions.