Authors: Brian E. Malley, Joel M. Levin, Jeremy M. Kahn, Leigh A. Bukowski, David T. Huang
Categories: Article, Bayesian statistics, clinical trials, critical care, frequentist statistics, intensive care, implementation science, sepsis, survey methods
Source: CHEST critical care
Authors: Brian E. Malley, Joel M. Levin, Jeremy M. Kahn, Leigh A. Bukowski, David T. Huang
Randomized clinical trials are a mainstay of medical research, but have drawbacks including time and cost. Bayesian adaptive trials can improve the speed and efficiency of clinical trials and are an increasingly common trial design in critical care. However, the complexity of Bayesian adaptive trials may prevent clinicians from easily interpreting their results.
Do intensivist physicians perceive, understand, and accept results from Bayesian adaptive clinical trials differently than results from traditional frequentist trials when presented with otherwise identical data?
We surveyed US intensivists from March through April 2022 about their perceptions of Bayesian adaptive trials. Within the survey, participants were randomized to read an abstract for a hypothetical new sepsis drug trial that used either Bayesian adaptive methods or traditional frequentist methods, with both abstracts based on the same simulated trial data. Participants then were asked about perceived understanding, validity, and general acceptance of the trial’s methods and results. Survey responses were compared between experimental groups using Mann-Whitney U tests and ordinal logistic regressions.
We received complete results from 273 of 592 eligible physicians (response rate, 46.1%). Participants in the Bayesian group reported lesser understanding of the methods (mean rating in frequentist vs Bayesian groups, 3.18 vs 2.85; P < .001) and the results of the trial (mean rating, 3.36 vs 3.00; P < .001) compared with the frequentist group. Participants in the Bayesian group also expressed greater concerns about trial validity (eg, mean rating about the risk of type 1 error, 2.70 vs 2.99; P = .001) compared with the frequentist group. Participants in both groups reported similar beliefs about acceptance of trial results (eg, mean rating in frequentist vs Bayesian group about the effectiveness of the study drug, 3.87 vs 3.86; P = .47).
Our results show that despite lower perceived understanding of Bayesian trials and skepticism about elements of their validity, no substantive differences were found in intensivist physician acceptance of Bayesian trial results compared with frequentist trial results. CHEST Critical Care 2026; 4(2):100264
The randomized clinical trial is a mainstay of clinical research and the gold standard for obtaining causal inference about treatment effects in health care.^1^ Traditionally, randomized trials evaluate 1 or 2 interventions at a time, with the results analyzed at the end of the trial using frequentist statistical methodology.^2^ Such trials have been instrumental in advancing clinical practice,^3–5^ but have several drawbacks. Most notably, frequentist trials are costly and take a long time to complete, slowing the pace of scientific discovery and straining the budgets of research funders.^6–8^
Bayesian adaptive trials are a relatively recent methodologic innovation in clinical trial design intended, in part, to address these concerns.^9–11^ Bayesian trials use experimental data and prior information on treatment effects to estimate the probability of a given conclusion being true (eg, drug A is better than drug B),^12^ and adaptive trial designs allow for addition or removal of study arms in the middle of a trial, as well as the ability to weight randomization toward higher-performing arms.^13^ These innovations can allow trials to generate more information in less time and at lower costs. Bayesian adaptive trials also can pose less risk to trial participants because the act of weighting randomization toward higher-performing arms allows more participants to see the potential benefits of new drugs and for negative trials to be stopped earlier compared with frequentist trials.^14,15^ Perhaps because of these strengths, Bayesian adaptive trials increasingly are common in intensive care, with several recent high-profile trials using Bayesian methodology.^16–19^
Despite their merits, Bayesian adaptive trials are more methodologically complex than frequentist trials and may be difficult for practicing clinicians to understand. Indeed, prior work suggests that even many trialists are not comfortable interpreting Bayesian analyses,^20^ a metric that is likely higher among clinicians who are not trialists. These observations raise a critical issue for implementation of trial results. Clinicians’ understanding and acceptance of the evidence are considered essential to the implementation process.^21^ To the degree to which clinicians do not accept the results of Bayesian adaptive trials, even informative trials may fail to change clinician behavior, further widening the gap between evidence and practice. More broadly, if gains in trial efficiency are offset by losses in the likelihood of implementation, Bayesian adaptive trials will fail to achieve their promise.
To understand better the impact of trial design on physicians’ beliefs and perceptions, we conducted a randomized experiment embedded in a survey of practicing intensivist physicians. Our goal was to determine if differences exists in physicians’ self-perceived understanding, perceptions of validity, and acceptance of results between Bayesian adaptive trials and conventional frequentist trials, offering insight into how the increasing prevalence of Bayesian adaptive trials may affect the critical care research landscape.
We surveyed attending US intensivist physicians from March through April 2022 as part of a larger longitudinal study of how physicians develop and update their beliefs about treatments of unknown effectiveness in response to new evidence.^22,23^ We focused on intensivist physicians because the parent study primarily examined treatments for severe COVID-19 pneumonia; these individuals generally have more expertise on this topic relative to other physician groups. In addition, focusing on intensivist physicians allowed us to isolate better the effects of evidence on physician decision-making as opposed to patient decision-making, because most critically ill patients are incapacitated and therefore unable to participate in shared decision-making about drug treatments. Complete details on the survey sample are available elsewhere.^22^ The sample initially was derived from board-certified intensivist physicians listed in the American Medical Association Physician Master file and surveyed in April 2020. Physicians from the initial survey that opted to receive additional email communication were included in this survey and therefore were a smaller subset than the originally surveyed group. The study was approved by the University of Pittsburgh Human Research Protection Office (Identifier: 19100282).
We randomly assigned participants 1 to 1 of 2 experimental groups. Both groups were asked to read a hypothetical research abstract describing a prospective, randomized, placebo-controlled trial of an investigational new drug on 28-day mortality in septic shock. The first group read an abstract that described the study results in the form of a Bayesian adaptive trial (henceforth called the Bayesian group). The second group read an abstract that described the results in the form of a traditional frequentist clinical trial (henceforth called the frequentist group). Both abstracts described the hypothetical study drug as a new immunomodulator for vasopressor-dependent septic shock called AB-37, and both were titled “AB-37 for Vasopressor Dependent Septic Shock: A Randomized Controlled Trial.” These descriptors were chosen to be intentionally vague so as to avoid introducing bias from participants’ prior beliefs about any specific treatments.
To ensure parity across the 2 abstracts, we simulated the 2 clinical trials using a single synthetic data set in which all patients’ outcomes were specified in advance. We intentionally specified a moderate positive treatment effect so that the results would require some interpretation on the part of the physician as to how best to apply the study findings in clinical practice. We then separately ran Bayesian and frequentist statistical analyses on the simulated trial data set. For the Bayesian abstract, we chose neutral priors to ensure further similarity between the abstracts. The 2 abstracts otherwise were similar in length, format, and content. Neither abstract included an author list or an identifying journal. The simulation and trial analyses were performed using the UPG package for Bayesian regressions in R (R Foundation for Statistical Computing).^24^ The 2 abstracts are shown in Table 1.
We then developed a series of de novo survey items related to 3 domains related to physicians’ assessment of the perceived understanding (2 items), perceived validity (3 items), and acceptance (2 items). For perceived understanding, participants were asked to rate their understanding of the trial’s methods and its results. For perceived validity, participants were asked to evaluate 3 1 statement about whether the design negatively impacted the random assignment, 1 statement about whether the investigators chose to stop the trial early, and 1 statement about the general risk of false-positive results. For acceptance, participants were asked whether the experimental drug was more effective than placebo and to rate the quality of evidence from the trial. We also asked all participants the same 1-item question about their general beliefs about Bayesian adaptive clinical trials. All items solicited responses along a 4-point or 5-point Likert-type scale, depending on whether a neutral response was potentially of interest.
The survey instrument was developed by the authors and reviewed for content validity with researchers experienced in running large-scale Bayesian and frequentist trials. The instrument then was piloted with an independent sample of intensivist physicians, with revisions for clarity made based on their feedback. The final instrument is shown in e-Table 1.
We administered the survey in March and April 2022 using a commercially available online survey tool (Qualtrics XM, Qualtrics LLC). Randomization was embedded in the survey, such that participants were randomized after survey access, but before seeing any survey questions. After the initial contact email, nonresponders were sent up to 2 reminder emails. Participants were offered a $75 payment card as incentive for survey completion.
In line with a prespecified analysis plan for a companion survey,^23^ we excluded incomplete surveys and calculated the response rate as the number of complete surveys received divided by the number of surveys sent. We used standard summary statistics to compare demographics between the 2 groups and between responders and nonresponders.
To analyze the survey results, we first made the a priori decision to analyze each survey item independently, rather than combine items into summary scales. We made this decision because we considered each individual item of interest, independent of the other items. We compared responses on individual items between groups using 2-sample Mann-Whitney U tests. For this analysis, we did not adjust for multiple comparisons because all tests reflected a single conceptual model for the relationship between study group and outcomes. To help provide insight into whether our effect sizes are clinically meaningful, we also calculated Cohen’s d for each item.^25^ Traditional interpretations of Cohen’s d are small effect size (> 0.02 and < 0.5), medium effect size (0.5–0.8), and large effect sizes (> 0.08).^25^
In secondary analyses, we fit a series of ordinal logistic regression models which allowed us to investigate the impact of demographics and professional characteristics on our estimates. In all regressions, the dependent variable was the response to each survey item, treated as ordinal variables, and the independent variable was experimental group (frequentist = 1 vs Bayesian = 0). To understand the magnitude of any confounding, we fit both adjusted and unadjusted models. For adjusted models, covariates include age, sex, proportion of time spent working in a clinical capacity, base specialty, and practice setting (eg, academic hospital, community hospital, etc), all defined at the participant level. The ORs from these models can be interpreted as the odds of a 1-unit increase in the response scale in the frequentist group relative to the Bayesian group. We fit all ordinal regression models using the polr function from the MASS package in R.^26^ To assess the statistical significance of the predictors, we calculated z values by dividing each estimated coefficient by its SE. P values then were computed using 2-tailed tests based on the standard normal distribution.^27^ For the secondary analyses, we adjusted for multiple comparisons using the Bonferroni method. All statistical analyses were performed in R version 4.4.1, and a P value of ≤ .05 was considered statistically significant.
We surveyed 592 physicians and received 273 complete responses, resulting in a response rate of 46.1%. Respondents and nonrespondents generally were well balanced on observable characteristics (e-Table 2). The 2 experimental groups also generally were well balanced on observable characteristics (Table 2). Descriptive results for all outcomes are shown in Figure 1.
In the perceived understanding domain, a higher score reflects lower perceived understanding. Participants in the Bayesian group reported less understanding of both the methods (mean rating in frequentist vs Bayesian group, 3.18 vs 2.85; P < .001; Cohen’s d = 0.50) and the results of the trial (mean rating in frequentist vs Bayesian group, 3.36 vs 3.00; P < .001; Cohen’s d = 0.58), compared to the frequentist group.
In the perceived validity domain, a lower score indicates greater perceived validity. Participants in the Bayesian group were more likely to believe that the investigators chose to stop the trial early (mean rating in frequentist vs Bayesian group, 1.81 vs 2.42; P < .001; Cohen’s d = 0.64) and were more concerned about the risk of type I error (mean rating in frequentist vs Bayesian group, 2.70 vs 2.99; P = .001; Cohen’s d = 0.37) compared with the frequentist group. We did not find evidence of differences regarding beliefs about bias in randomization (mean rating in frequentist vs Bayesian group, 2.45 vs 2.61; P = .08; Cohen’s d = 0.18), although the direction was consistent with the preceding items, with participants expressing directionally more skepticism about the randomization process in the Bayesian group.
In the acceptance domain, we did not find differences between groups on either item. On the effectiveness of the study drug, participants in both groups reported nearly identical beliefs (mean rating in frequentist vs Bayesian group, 3.87 vs 3.86; P = .47; Cohen’s d = 0.01). Participants in the Bayesian group demonstrated a nonsignificant trend toward perceiving the evidence in the abstract to be of slightly lower quality, but these results were not conclusive (mean rating in frequentist vs Bayesian 3.68 vs 3.53; P = .10; Cohen’s d = 0.18).
Finally, in the survey item assessing overall beliefs, respondents generally were supportive of the use of Bayesian adaptive trials in critical care, although a notable minority were unsupportive. Overall, 135 of 273 participants (49.5%) reported that the use of Bayesian trials is probably good or definitely good, whereas 31 of 273 participants (11.4%) reported their opinion as probably bad or definitely bad. The remaining 107 of 273 participants (39.2%) reported their opinion as neither good nor bad. These results did not differ by study group (mean rating in frequentist vs Bayesian group, 3.39 vs 3.49; P = .25; Cohen’s d = 0.13).
Results of the secondary regression analyses both with and without demographic and professional control variables are shown in Table 3. No changes were found in direction or significance of results between the primary analyses and the proportional odds logistic regression adjusting for participant demographics.
The overall impact of clinical research on patient outcomes depends not only on the efficiency by which scientists produce evidence, but also on the degree to which that evidence is adopted by practicing clinicians. Relative to traditional clinical trials, Bayesian trials can improve the efficiency of evidence production, increasing the evidentiary value of each research dollar and speeding up the timeline of clinical discovery. However, to the degree that the methodologic complexity of Bayesian trials poses a barrier to implementation, any efficiency gains in evidence generation will be offset by losses in the degree to which that evidence shapes clinical practice.
In an experiment designed to understand these tradeoffs better, we found a mixed picture. On the one hand, physicians who read an abstract describing a Bayesian adaptive trial similarly were likely to report general acceptance of the trial’s findings compared with physicians who read an abstract that described a traditional frequentist trial. On the other hand, physicians who read the Bayesian abstract were significantly less likely to understand the trial’s methods or results.
In one sense these results are reassuring, because many physicians seem ready to accept the findings from novel trial designs, despite a relative lack of understanding. These could be because physicians are primed to accept the findings of large multicenter trials or because the hypothetical results of our trial were relatively compelling. It also could be that physicians overstated their willingness to accept the trial’s finding to appear more knowledgeable, a concept known as social desirability bias. The fact that our survey was anonymous should have mitigated any social desirability bias, but perhaps not completely.
In another sense, however, these results are concerning because they demonstrate a potentially important gap in physicians’ understanding of these new trial designs. Although empirical data that statistical understanding directly impacts the implementation process remain incomplete, strong theoretical reasons for this to be true exist,^21,28^ and doctors themselves identify statistical education as an important part of the medical curriculum.^29,30^ Future research should build on our work by directly examining this relationship. More broadly, it is notable that even physicians randomized to read the frequentist abstract demonstrated a less than complete understanding of the trial’s methods and results. This finding suggests the need for better statistical education in the medical curriculum about all clinical trials, not only advanced clinical trial designs.
The implementation process is complex and multifaceted, relying not only the beliefs of individual clinicians, but also the social, environmental, and clinical contexts in which the treatments take place.^31^
Nonetheless, an extensive body of literature shows that clinician beliefs about effectiveness are integral to the implementation of new evidence.^32–35^ If clinicians do not find evidence from Bayesian adaptive trials to be trustworthy, then they may be less likely to implement the results in practice. Against this backdrop, our study suggests a need for additional education surrounding complex trial designs to ensure that clinicians understand the methods and results of Bayesian adaptive trials. At the same time, our study does provide some reassurance, because overall acceptance did not differ between groups and most clinicians supported the use of these trial designs.
Our findings extend those of a previous study of medical researchers which found significantly lower self-reported knowledge of Bayesian trial methods compared with frequentist trial methods.^20^ In that study, only 8.4% of researchers reported comfort interpreting Bayesian statistics. The researchers also reported insufficient knowledge as the largest barrier to greater uptake of Bayesian methods. Our study showed similar important gaps among practicing clinicians. Our findings also extend prior research showing that statistical methods in medical research have increased in complexity over time.^36–38^ Overall, these trends create a body of biomedical research that is increasingly opaque to front-line clinicians.
Our study has several limitations. First, our findings may not generalize to other physician specialties. However, intensivists come from multiple primary specialty backgrounds and the field recently saw multiple large-scale Bayesian adaptive trials, so they are in many ways representative of physicians in general.^16–19^ Second, we examined only a single hypothetical study, and that study showed a positive treatment effect. We may have observed different results for a trial with a negative treatment effect or a trial in which the drug showed tradeoffs between benefits and harms. We intentionally chose a simple trial with modestly positive results so as not to overly complicate our findings, but it is possible varying the hypothetical trial findings could have impacted participants’ acceptance or understanding. Third, because no validated scales exist to measure acceptance and understanding of Bayesian trials, we developed a de novo survey instrument. In recognition of this inherent limitation, our survey questions were tested for face validity and were piloted on intensivists outside the study team. Fourth, our study may have been underpowered to detect meaningful differences between groups. More broadly, it is unclear how large an effect size might be meaningful clinically in this setting, because no way exists to link our findings directly to clinical practice patterns or patient outcomes. Although the Cohen’s d calculations suggest many effect sizes in the moderate range, these are not directly transferable to the clinical setting. As such, the implications of this work should be considered more theoretical than practical. Fifth, to keep our study relatively simple, we only tested 1 aspect of physician understanding, Bayesian adaptive trials. More work is needed to understand the impact of other aspects on physician understanding, such as the ability to examine multiple arms simultaneously and the platform nature of these trials. Overall, these limitations are offset by the strengths of our study, which include the use of a randomized experimental design to strengthen causal inference about the impact of trial methods on clinicians understanding and acceptance.
We found no difference in physicians’ acceptance of Bayesian adaptive trials compared with frequentist trials, despite lower perceived understanding and greater concerns about the validity of Bayesian adaptive trials. These findings provide reassurance that acceptance is not a barrier to uptake of evidence from Bayesian trials, but also highlight the need for ongoing education to support the eventual implementation of the results from these increasingly common trials.
**Additional ** The e-Tables are available online under “Supplementary Data.”