Authors: M. Samie Tootooni (1Department of Health Informatics and Data Science, Loyola University Chicago, Maywood, IL, United States of America), Erin F. Barreto (2Department of Pharmacy, Mayo Clinic, Rochester, MN, United States of America), Phichet Wutthisirisart (3Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States of America), Kianoush B. Kashani (4Division of Nephrology and Hypertension, Division of Pulmonary and Critical Care Medicine, Department of Medicine, Mayo Clinic, Rochester, MN, United States of America), Kalyan S. Pasupathy (5Department of Biomedical and Health Information Sciences, University of Illinois Chicago, Chicago, IL, United States of America)
Categories: Article
Source: Journal of critical care
Authors: M. Samie Tootooni, Erin F. Barreto, Phichet Wutthisirisart, Kianoush B. Kashani, Kalyan S. Pasupathy
Vancomycin is a renally eliminated, nephrotoxic, glycopeptide antibiotic with a narrow therapeutic window, widely used in intensive care units (ICU). We aimed to predict the risk of inappropriate vancomycin trough levels and appropriate dosing for each ICU patient.
Observed vancomycin trough levels were categorized into sub-therapeutic, therapeutic, and supra-therapeutic levels to train and compare different classification models. We included adult ICU patients (≥ 18 years) with at least one vancomycin concentration measurement during hospitalization at Mayo Clinic, Rochester, MN, from January 2007 to December 2017.
The final cohort consisted of 5,337 vancomycin courses. The XGBoost models outperformed other machine learning models with the AUC-ROC of 0.85 and 0.83, specificity of 53% and 47%, and sensitivity of 94% and 94% for sub- and supra-therapeutic categories, respectively. Kinetic estimated glomerular filtration rate and other creatinine-based measurements, vancomycin regimen (dose and interval), comorbidities, body mass index, age, sex, and blood pressure were among the most important variables in the models.
We developed models to assess the risk of sub- and supra-therapeutic vancomycin trough levels to improve the accuracy of drug dosing in critically ill patients. 
Appropriate drug dosing is a critical issue in hospital settings, particularly for renally eliminated and nephrotoxic therapies. Vancomycin is one of the most widely used antibiotics in intensive care units (ICU), with a narrow therapeutic window [1, 2]. Considering the rise in resistant gram-positive infections, the use of vancomycin continues to rise [3]. Insufficient drug dosing leads to subtherapeutic serum concentrations, which increases the risk of clinical and microbiologic failure and the development of antibiotic resistance [4]. On the other hand, supratherapeutic concentrations can lead to dose-dependent drug-associated toxicity [5–7]. Indeed, nephrotoxin-associated acute kidney injury (AKI) occurs in 5–20% of vancomycin-treated individuals [8–15]. In turn, AKI is associated with a longer ICU length of stay, higher mortality rate, increased need for mechanical ventilation, and many other adverse outcomes among ICU patients [16, 17]. Therefore, identifying and mitigating factors associated with vancomycin nephrotoxicity may decrease the incidence, severity, and complications [18]. Selecting a proper individualized dosing regimen for vancomycin to maintain the drug concentrations within the target trough level range is essential for optimizing its safe and effective use [19–22].
With the growing utilization of electronic health records (EHR), there is an incredible opportunity to employ data science methodologies in online monitoring and predictive and prescriptive analytics to deliver precise drug dosing in clinical settings, such as ICU [23]. Recently, Sutherland et al. reviewed the existing approaches in predicting nephrotoxin-associated AKI and recommended applying big data analytics to develop more complex data-driven models [24]. Also, Dorajoo et al. suggested considering the essential clinical needs in practice when developing data-driven prediction models [25]. Therefore, successful development and implementation of an AI-based dosing decision support models focusing on clinical workflow could help identify those patients with a higher risk for sub- and supra-therapeutic range and allow the providers to take individualized, timely preventive actions at the right time, such as replacing or dose-adjusting the drug [26, 27].
This study aimed to predict the steady-state vancomycin trough concentration and identify the main contributory patient factors to facilitate dosing decisions. To begin with, we 1) developed and validated ensemble machine learning models to predict the drug level threshold, namely, sub-therapeutic (<10 mg/L), therapeutic (10–20 mg/L), and supra-therapeutic (>20 mg/L) for ICU adult patients and estimated the risk of inappropriate vancomycin trough levels; 2) we identified the predictive factors for the vancomycin steady-state trough level and ascertain their relative contribution; and 3) predicted the vancomycin steady-state trough level for each ICU patient.
The treatment course was defined as a set of consecutive administrations within pre-specified time intervals. It included a one-time administration of a loading dose followed by multiple maintenance doses. Our final cohort consisted of treatment courses, and our targets were the first measured vancomycin trough level. Figure 1 shows a theoretical timeline for a course and highlights our prediction horizon. Details about the consideration of courses are discussed further in Supplementary File 1.
This study predicted the steady-state trough level before each administration so that it could be used for clinical decision support. Trough concentration was measured immediately before an administration, typically after three or more administrations. At each decision point (DP) before the next administration, the prediction model identified the class the steady-state trough concentration would fall into (i.e., sub-therapeutic, therapeutic, or supra-therapeutic). The result of the prediction models at each DP identified the risk of under or over-dosing. The DP immediately before the last dose was named DP-1. The decision points before DP1 were named ordinally, e.g., the second last DP, the third last DP as DP-3, etc. Figure 1 shows the defined decision-making points.
This retrospective study was reviewed and approved by the Mayo Clinic Institutional Board Review (IRB # 07–001380). Informed consent was waived due to the minimal-risk nature of the study. Our cohort consisted of adult patients admitted to the intensive care unit (ICU) at Mayo Clinic, Rochester, MN. All ICU patients from January 1, 2007, through December 31, 2017, with a vancomycin concentration measured were included (n=38,200).
Those with inadequate information related to drug administration, concentration measurement, and baseline information were excluded. More details about the vancomycin course and exclusions are provided in Supplementary File 1.
The first dose of vancomycin was considered a loading dose independent of renal clearance. The determinants of loading dose for vancomycin depend on the target level for the infection being treated and the volume of distribution. At Mayo Clinic, all loading doses are calculated as 20 to 30 mg/kg based on actual body weight and rounded to the nearest 250 mg increment [28]. For patients with obesity with a BMI >40 kg/m^2^ or a weight >120 kg, adjusted body weight was used for dose calculations [28]. The estimated loading dose could not exceed 3000 mg.
To mitigate the risks of rapid vancomycin infusion, the adult/peds/neonate infusion rates were ≤1 gm over 60 min, 1.001–1.749 gm over 90 min, and 1.750–2 gm over 2 hours. Doses of >2 gm were infused at the rate of 1 g/hr. In the case of the development of infusion reactions, slower infusion rates were used. Our standard product concentration was approximately 4–7 mg/mL, with concentrations up to 10mg/mL in fluid-restricted patients, e.g., a 1000 mg dose would go in 250mL for a concentration of 4mg/mL in the product, and that would be administered over 60 minutes at a rate of 16.7 mg/min.
A list of static and dynamic variables, either extracted or calculated, used in the development of the models are shown in Table 1. The static values were considered constant during one vancomycin course, whereas the dynamic data elements were subject to changes over time and differed at each decision point. Among the dynamic variables, we included the contribution of the kinetics of the estimated GFR (Kinetic eGFR or KeGFR), mean arterial pressure (MAP), and hourly urine output. Particularly for critically ill patients with unstable kidney function, KeGFR may be a valuable biomarker that estimates the velocity of GFR fluctuations between two consecutive serum creatinine measurements [29–32]. Indeed, due to rapid changes in GFR during acute illnesses, the Cockroft-Gault formula to calculate eGFR may not provide a reliable estimation of kidney function [33]. Detailed descriptions of the determination of KeGFR, MAP, hourly urine output, and measurement and administration delay are provided in Supplementary File 2.
To include features with missing values in our model, we developed logistic and linear regression models using the MICE package in R [34] to impute missing data for categorical and linear variables. Some data elements, such as the Acute Physiology And Chronic Health Evaluation (APACHE) score, were unavailable during the first 24 hours of ICU admission. Consequently, most of its values were missing, especially at DP-2 and DP-3. To reduce the effect of large-scale imputation for such values, we replaced them with median values.
The final cohort was randomly split into training (80%) and test (20%) sets. Five-fold cross-validation was used for the training and validation of the models. We aimed to have two binary classification models at each DP to predict the steady-state trough concentrations’ therapeutic level accurately. We trained and compared six types of models selected based on maximum dissimilarity between model structures (i.e., Recursive Partitioning and Regression Trees (rpart) [35], Random Forest (rf), Stochastic Gradient Boosting (GBM), Adaptive Boosting with Statistical framework (LogitBoost) [36], eXtreme Gradient Boosting (xgbTree) [37], and Neural Networks [with a principal component step] (pcaNNet)). In addition to the steady-state therapeutic class, the two finalized classification models at each DP provided an individualized risk of being under or over-dosed using the administered dosing regimen. At each DP, Hosmer and Lemeshow goodness of fit tests were applied to ensure the estimated probabilities could be interpreted as the true risk of sub- or supra-therapeutic levels. All coding and analyses were performed in JMP^®^ (version 14.0.0, SAS Institute Inc., Cary, NC) and R (Version 3.6.1; Vienna, Austria).
We developed continuous (regression) prediction models to estimate the steady-state trough concentration to fulfill our third objective. We developed three different models at each DP (i.e., Elastic Net, GBM, and Random Forest). We calculated Root Mean Square Error (RMSE), Mean Average Error (MAE), and R^2^ to compare developed models’ performance and choose the best-performing model at each decision point.
Our final cohort included 5,337 vancomycin courses (4,270 used for training and 1,067 used for testing) from 2007 to 2017 administered at Mayo Clinic, Rochester, MN (Figure 2). Table 2 shows the descriptive statistics of the studied cohort following cleaning and performing exclusions. More details about the final dataset used to develop the models, including the shared anonymized dataset and detailed descriptive statistics, are discussed in Supplementary File 3. Among those included, 2,638 (49%) patients had a steady-state trough outside the target range. Figure 3 shows a comparison between six selected model performances in predicting steady-state trough concentration of vancomycin based on AUC, sensitivity, and specificity.
The eXtreme Gradient Boosting, an ensemble/hybrid machine learning algorithm, was chosen as our final model due to its overall superior performance. The selected models were internally validated via the test set in predicting sub-therapeutic (ROC: 0.85, Specificity: 0.53, and Sensitivity: 0.94) and supra-therapeutic (ROC: 0.83, Specificity: 0.47, and Sensitivity: 0.94) categories, respectively. Figure 4A shows the ROC curves for predicting sub- and supra-therapeutic classes at DP-1, and Figure 4B plots the output probabilities for our patients in the test set and compares them with their actual therapeutic class.
Table 3 provides prediction accuracy with the 95% confidence interval, other classification performance measurements on the test set, and the p-values for the Hosmer Lemeshow (HL) calibration test at each DP. The P-values each exceeded 0.05, suggesting that the xgbTree models’ output probabilities embodied the actual risk of the defined therapeutic categories without additional calibration steps.
The variable importance within the models were calculated based on individual variable ROC analysis, which showed that KeGFR and other serum creatinine-related kidney function measurements, vancomycin prior dosages and frequencies, Charlson comorbidity index, body mass index (BMI), age, gender, and MAP were among the essential variables in predicting the vancomycin therapeutic class. We ranked the predictors based on their overall feature importance for detecting both sub- and supra-therapeutic dosing regimens. Figure 5 shows the most effective variables in detecting sub- and supra-therapeutic dosing regimens at DP-1. The detailed structure of the developed models and the contribution of all variables at each decision point are provided in Supplementary File 3.
In addition to predicting therapeutic class, we also estimated the final trough concentration at each decision point. While the Elastic Net model at DP-1 outperformed the other models (MAE: 4.44, R^2^: 0.43, and RMSE: 6.18), at DP-2 and DP-3, the best performance belonged to the GBM Regression models. Figure 6 compares the performance of the three regression models based on their RMSE, MAE, and R^2^.
In a sensitivity analysis, we stratified patients based on their BMI (<30 vs. ≥30). We noted while the model performance, i.e., AUC, was slightly higher in obese patients, the model showed slightly more errors among this group as well (Supplementary Figure 1).
We developed and validated two binary xgbTree classification models at each decision point with 0.79 ≤ ROC ≤ 0.85 to provide an individualized dosing scheme for vancomycin therapy. We demonstrated that the output probability provided by these models represented the individualized risk of vancomycin under and over-dosing for ICU patients. Among the variables evaluated, KeGFR and other serum creatinine-related kidney function measurements, vancomycin prior dosages and frequencies, Charlson comorbidity index, BMI, age, gender, and MAP were among the essential variables in predicting the vancomycin therapeutic class. Finally, we developed Elastic Net and GBM regression models at each DP to estimate the steady-state trough concentration.
National guidelines [3, 38, 39] recommend the ratio of the area under the vancomycin concentration-time curve at 24 hours to the minimum inhibitory concentration (MIC) (AUC: MIC ratio) of at least 400 mg∙h/L [39, 40]. For practical reasons, collecting multiple levels to calculate a patient-specific AUC accurately is not yet mainstream [41, 42]. Hence, using the steady-state trough concentration to approximate the area under the vancomycin concentration-time curve remains a standard alternative approach. The steady-state target trough range for adult infections is 10–20 mg/L (10–15mg/L for most patients and 15–20mg/L for severe and difficult-to-treat infections).
Furthermore, we utilized a retrospective dataset, focusing on the vancomycin trough-level data due to its availability. It also reflects the common clinical practice of using these measurements for dosing decisions at our institution during the study period. This choice underscores our goal to evaluate how AI could enhance clinical decisions using data that are readily available and routinely collected. We also recognize the pharmacokinetic advantages of AUC-guided dosing for vancomycin therapy. Unfortunately, we could not train our models on vancomycin AUCs due to a lack of data. We intended to highlight the relevance of our findings to current practices and inspire further exploration into the efficacy of trough versus AUC-guided dosing approaches in real-world clinical settings.
Maintaining the vancomycin trough concentration within the recommended range is essential. The most common algorithms to guide vancomycin dosage and interval selection are based on body weight [43], which is approximated by the Cockcroft-Gault (CG) formula. While customary for drug dosing, such estimations have known limitations, including altered creatinine production in the setting of decreased muscle mass, sepsis, liver disease, a lag-time from the onset of kidney damage by as much as 24–48 hours, and dilution in patients with significant positive fluid balance [33, 44]. Collectively, for renally-eliminated drugs, this renders vancomycin dosing difficult and often inaccurate among ICU patients. With this strategy, we showed that only 30% of critically ill patients achieve an initial trough level within a goal [45]. Calculating the KeGFR can address these issues to some extent [29–32]; however, it is still expected that both inter- and intra-individual vancomycin concentrations observed in treating patients will be highly variable. [7, 20, 46–52]. Despite several efforts to optimize dosing methods [38–40, 53–56], achieving a trough concentration within the goal may still take upwards of three days [57]. The substantial delay in adequate antimicrobial therapy could negatively affect patient outcomes.
Artificial intelligence (AI) has emerged recently to inform vancomycin dosing decisions [58–61]. The existing machine learning models aimed to assist providers by suggesting a ‘proper’ next dose. Since there is no existing gold standard as the “appropriate” dose for each patient in the EHR data, labeling the dataset in such approaches requires additional efforts, such as calculating daily-per-day instead of dose-per-time [59, 61], using expert opinion [60], or only including doses that resulted in the therapeutic trough range [58]. Despite promising results, labeling the dataset and finding the appropriate dose using any of the efforts mentioned above introduces a new source of inaccuracy to the reported result. Our approach fundamentally differed from the existing literature since we aimed to inform the providers of the risk of the current dosing regimen instead of suggesting a dose. We considered the measured trough levels standard, determining the class of the current dosing regimen.
From a practical standpoint, we envision a clinical question before each planned vancomycin administration or decision “Is this the right dose?” Our models are designed to answer this question in real time by providing each patient’s individualized sub-therapeutic, therapeutic, or supra-therapeutic risk. Therefore, our model could fit into a practical clinical decision-support tool without increasing the clinicians’ additional decision-making burden. In addition, we kept our model blind from any previous information on vancomycin through measurements. This resembles a common situation in practice where trough levels are often measured right before administering the third or higher maintenance dose.
We reported the effect of a variety of patient-related factors on the vancomycin trough level. We highlighted that often unexploited patient factors such as KeGFR or MAP are among the most significant contributors in the determination of vancomycin trough levels (Supplementary File 2). The mechanisms related to the effects of MAP on vancomycin through levels are not well known, but it could be extrapolated from the observations that during critical illnesses, lower MAP (thus lower tissue perfusion) does not allow adequate circulation of blood to the muscles and therefore creatinine and urea sequestration within the muscle tissues. In addition, unlike Matsuzaki et al. [60], we found that patient gender is important in identifying supra-therapeutic courses. This may be due to differences in muscle mass between sexes that lead to variability in model performance, considering the dependency of serum creatinine on muscle mass. We suggest separate multi-institutional studies to further study the underlying sociodemographic factors affecting vancomycin-dosing decisions.
Recent studies indicate that continuous infusion of vancomycin (CVI) offers several advantages over the traditional intermittent infusion (IVI), including improved target concentration achievement, reduced risk of nephrotoxicity, and more stable serum levels, which may simplify monitoring and reduce costs[62, 63]. Despite this, our models, developed using the IVI method reflecting Mayo Clinic’s prevalent practice during the study period, achieved a high degree of accuracy in predicting steady-state levels. This approach allowed a comparison between AI model outcomes and historical clinical decisions concerning dosing and timing of vancomycin administration. While CVI remains an emerging practice in many hospitals, our research paves the way for future studies to develop models based on CVI, especially as more related data become available in electronic health records (EHRs).
We assumed a constant maintenance dose and interval for each course. The ineligible cases 1) already had more than one maintenance dose administered, and 2) either the interval or the administered maintenance dose was modified. We were aware that modifications in the maintenance doses occur frequently in clinical workflow. Most such modifications (interventions) for vancomycin are due to significant changes in serum creatinine [or eGFR] or urine output recognized by clinicians. On the other hand, our focus was to optimize vancomycin dosing decisions for all patients. Our results indicate that even in this carefully constructed cohort of patients with unchanged courses, 49% of patients still had vancomycin trough levels outside of their target levels, either low or high, when dosing decisions were done clinically.
The developed xgbTree models provided a promising base for future development of future dosing recommendation engines. However, their performance may still be improved by advancing our knowledge about pharmacokinetics and pharmacodynamics, improving model structure, increasing the number of records, and considering other patient factors, including comorbidities, medications, vital signs, and microbiologic culture data. For instance, we did not consider the multiplicity of nephrotoxic drugs and their interactions, which, per sé, may contribute to changes in GFR and, therefore, drug dosing scheme.
Despite the rapid evolution of using AI in vancomycin dosing in recent years, challenges still need to be recognized and addressed before routine implementation to individualize therapy [25]. In addition, our developed models were trained based on an assumption of a constant therapeutic (target) range (e.g., 10–20 mg/L in this study). Although such a range may apply to most infections, patient scenarios may warrant higher (e.g., meningitis) or lower targets (e.g., skin and soft tissue infection). The proposed models also did not assess the patient’s target based on the source of their infection. Instead, the surrogate marker of pharmacokinetic/pharmacodynamic target attainment was utilized.
Other shortcomings of this study include a lack of scalability and translational assessments. Our cohort was predominately white and collected from a single hospital. The existence of such demographic variables may introduce inherent bias and limit the scalability of the proposed models [64]. Our next step is to assess the models’ scalability from different populations. As the features needed to predict vancomycin through levels were entirely different among patients who received renal replacement therapies, our models are not generalizable to this group of patients. Finally, although the developed models were retrospectively validated, we still need to study the impact of such models in clinical practice through prospective investigations. In addition, issues such as data availability in real-time, incorporating the tools with the current clinical workflow, and acceptability of the outcome of AI models among clinicians need to be addressed [65].
Using artificial intelligence models to improve intravenous vancomycin dosing strategy for adult ICU patients is feasible. By minimizing the risk of sub- or supratherapeutic levels, our models have the potential to offer the optimal dosing regimen. The next steps include assessing the model’s scalability in a multi-institutional study, developing a clinical dosing decision support algorithm, and analyzing the clinicians’ workflow to implement these models to assess their clinical impacts.
Although we focused on optimizing vancomycin dosing, this study was designed with a broader goal to improve the accuracy of dosing nephrotoxic drugs for critically ill patients. We specifically used vancomycin as our drug prototype due to its high frequency in administration and trough-level collection (used as our gold standard) in the ICU setting. Therefore, we consider this study as an initial step toward improving the accuracy of drug dosing in critically ill patients.