Factors associated problems of using exploratory multivariable regression to identify causal risk factors

Authors: Dan Lewer, Thomas Brothers, Elizabeth O’Nions, John Pickavance

Introduction

Many medical and epidemiological studies use multivariable regression to test whether several independent variables (exposures) are causal determinants of a health outcome. Where mutually adjusted regression coefficients are significant, the exposures are labelled as risk factors for the outcome. We call this study design “factors associated with.” In this article, we argue that this method is flawed due to a lack of reasoning about which variables are treated as confounders, multiple statistical testing, and post hoc interpretation of the results. In some cases, researchers use algorithmic or stepwise approaches to select exposure variables, which further exacerbates these problems. Although the results of factors associated with studies often seem reasonable, these problems mean that the method can also produce implausible results, such as dementia reducing the risk of death in patients admitted to hospital for trauma,^1^ diabetes reducing the risk of venous thromboembolism in the general population,^2^ and lack of food reducing the risk of post-traumatic stress disorder in refugees.^3^ Many of these studies are published every year, including in well respected journals. We argue that these studies are misleading and contribute to research waste, and the “factors associated with” method should be abandoned.

The “factors associated with” study design

What factors are associated with death and disease? What are the most important determinants of health? Can we identify risk factors that might be modified? These questions are often asked in observational health research. For example, researchers have investigated risk factors associated with heart attacks,^4^ strokes,^5^ serious covid-19,69 complications of cystic fibrosis,^10^ suicide among veterans,^11^ and pain after breast cancer surgery,^12^ with the aim of informing preventive interventions or identifying high risk groups. These studies are not testing a theory or hypothesis about the role of a specific risk factor, but instead simultaneously screen multiple potential risk factors for significant associations with the outcome (box 1).

In a typical “factors associated with” study, the researcher uses a regression model where the dependent variable is the outcome of interest and the independent variables are exposures such as personal characteristics, health behaviours, clinical factors, living and working conditions, and other socioeconomic and environmental factors. Where the mutually adjusted regression coefficients are statistically significant, the exposures are identified as factors associated with the outcome.

Although the results often look reasonable, we argue that the findings provide little scientific value and can be misleading. “Factors associated with” studies combine multiple poor research practices, including the table 2 fallacy, P-hacking, fishing expeditions, data dredging, HARKing (hypothesising after the results are known), and the Texas sharpshooter fallacy.^13 14^

The “factors associated with” design was mainstream by the 1970s^15 16^ and has become increasingly common. In 2024, more than 4000 articles with the phrase “factors associated with” in the title were added to the PubMed database, a 10-fold increase from 2004. Reasons for the increased use of this method might include the widespread availability of multivariable regression in desktop statistical software, ease of doing these studies when data are already collected, lack of requirement for theoretical development, and apparent efficiency of studying several risk factors simultaneously. Although many epidemiologists might be aware of the limitations of this study design, increasing numbers of studies continue to be published, including in the highest ranked journals.

In this article, we do not consider other forms of exploratory causal research, such as genome-wide association studies,^17^ which typically adjust the association between each candidate genetic variant and the outcome for a limited set of confounding variables, rather than adjusting all of the genetic variables for each other.

Problems with “factors associated with” studies

Consider these six unexpected findings published in high quality journals, each of which was based on analysis of multiple candidate exposures using a regression model.

• A study of primary care patients in England found that tobacco smoking reduced the risk of death due to covid-19.^7^ The authors suggested that inclusion of chronic respiratory disease in the regression model might explain this finding, because smoking causes chronic respiratory disease, which in turn increases vulnerability to covid-19. Therefore, adjusting for chronic respiratory disease could mask the effect of smoking on death due to covid-19. This problem is known as adjusting for a mediator.
• A study of people testing for covid-19 found that tobacco smoking reduced the risk of a positive test result.^18^ This finding may be explained by collider bias. Two reasons why people might access a covid-19 test are that they have symptoms of covid-19 or they smoke tobacco and have a cough related to smoking. Those with a smoking related cough are less likely to have a positive test result within the sample of people testing for covid-19. This problem is known as collider bias, which can occur when an analysis adjusts, stratifies, or selects participants based on a variable (in this case, covid-19 testing) that is affected by both the exposure (tobacco smoking) and the outcome (covid-19).^19^ Collider bias can mean that the observed association between the exposure and outcome is very different from the causal effect.
• A study of refugees from Guatemala found that lacking sufficient food almost eliminated the risk of post-traumatic stress disorder.^3^ The authors suggested that sharing of food and collective cooking might have created a sense of wellbeing. Again, collider bias might be an alternative explanation. Here, refugee status is the collider, which can arise from famine or food scarcity (the exposure) and war (which is a cause of post-traumatic stress disorder, the outcome). Among refugees, those displaced by famine may be less likely to have experienced war, producing a misleading negative association between food scarcity and post-traumatic stress disorder.
• A study of patients admitted to hospital due to trauma found that dementia reduced the risk of death.^1^ This finding might be explained by selection bias, where patients with dementia have a lower threshold for hospital admission or activation of a trauma team response.
• A study of US enlisted marines found that relationship counselling increased the risk of suicide, whereas post-traumatic stress disorder, self-harm, and adverse childhood experiences did not.^20^ The authors suggested that relationship problems might be underlying both counselling and suicide, a scenario sometimes known as confounding by a common cause.
• A study of participants in the UK Biobank found that diabetes reduced the risk of venous thromboembolism.^2^ The authors could not explain the apparent protective effect of diabetes and acknowledged that further evidence would be needed to support this finding. More plausible findings included increased risk of venous thromboembolism associated with older age, male sex, tobacco smoking, and higher body mass index.

Although we offered some possible explanations for these unexpected findings, we can only speculate, and many unknown mechanisms and unmeasured variables likely contribed to the observed associations (ie, a kind of spaghetti of causation). Apparently plausible results generated by the same method should be treated with the same caution as these strange findings. For example, in the study of venous thromboembolism,^2^ tobacco smoking was associated with an increased risk, which is likely true, but the effect size may be biased by other elements of the study design.

Key problems of the “factors associated with” studies no prespecified primary exposure and no rationale for which variables are used in statistical adjustment; use of multiple statistical tests; and creating post hoc hypotheses.

No strategy for statistical the table 2 fallacy

In a focused observational study of cause and effect, one variable is examined as a potential risk factor (exposure) on a health outcome, and the other variables are included in the model to adjust for confounding. In a “factors associated with” study, all variables are treated simultaneously as exposures and confounding variables for each other. The lack of rationale for statistical adjustment means that the quantities estimated by the regression model (eg, odds ratios) do not represent the effect of each variable on the outcome.

Westreich and Greenland called this problem the table 2 fallacy,^13^ because many studies include a table 1 describing the distribution of exposure variables in a study sample and a table 2 showing multivariable associations between the exposures and an outcome. The table 2 fallacy occurs when all values in table 2 are interpreted as causal effects on the outcome, rather than only the value estimated for the primary exposure.

Imagine having survey data from the general population, including whether participants recently had an injury or fall as a pedestrian (the outcome), their age (an exposure), and whether participants had balance problems (a second exposure). Table 2 might have two rows showing the mutually adjusted results for age and balance problems, based on a multivariable regression model. To estimate the effect of balance problems on pedestrian injuries and falls, an adjustment for age might be made, because age affects both balance and the risk of injuries, and hence is a confounder. Therefore, the value in table 2 might be a useful estimate. To estimate the effect of age on injuries, however, the effect would likely be left unadjusted in relation to balance problems, because balance is probably best understood as a mechanism or mediator for the effect of age on injuries.

Exposures can affect each other in many ways, and some are easier to understand than others. Some common relations between relevant variables and an exposure of interest a confounder, which affects the probability of both the primary exposure and the outcome, and may therefore mean that the exposure is correlated with the outcome even if it does not cause the outcome; a mediator, which lies on the causal pathway between the exposure and outcome, and can be considered a mechanism for a causal effect; and a collider, which is an event that is caused by both the exposure and the outcome, and can introduce bias if controlled in the analysis. These relations can be represented diagrammatically with directed acyclic graphs, which can help in the design of statistical adjustment strategies to estimate the causal effect of an exposure.^21^ When estimating causal effects, in most cases, confounders but not other variables should be controlled. The effect of controlling other variables is unpredictable, but in some cases, controlling for a mediator can create the false impression of no relation between a real exposure and an outcome (a false negative result) whereas controlling for a collider can create the false appearance of a relation between the exposure and outcome (a false positive result). Figure 1 shows an example of a confounder, mediator, and collider in a hypothetical study of risk factors for an injury or fall as a pedestrian.

These causal pathways are difficult to understand in a study that was not designed to quantify these pathways. Using a different set of variables to adjust each the effect of each variable is unlikely to resolve these problems, because unmeasured variables and the choice of study sample may also affect observed associations. For example, collider bias can result from the choice of study sample or selection bias during recruitment.^22^ In the examples above, where studies were conducted in samples of people testing for covid-19 or in Guatemalan refugees, collider bias could result from the fact that people not testing for covid-19 and non-refugees were excluded. Collider variables could affect each exposure in a “factors associated with” study differently, and may not appear in the list of exposure variables or in the description of the sample frame.

Given these complexities, it is generally not useful to try to unpick the reasons for an apparent multivariable association related to an exposure that is not the primary exposure. Unfortunately, in “factors associated with” studies, plausible findings are often taken to be true, whereas implausible findings are ignored or dismissed, even though these findings were generated with the same method.

Multiple statistical fishing expeditions, data dredging, and P-hacking

By design, “factors associated with” studies involve multiple statistical tests. Each independent variable has its own P value, which is purportedly a probability that the effect size or something greater might be observed if there was actually no real effect. When none of the variables under investigation are true risk factors and all are unrelated to each other and to the outcome, on average one in 20 will yield a P value below 0.05 — a false positive finding (type I error). The probability of obtaining a false positive finding increases with the number of statistical tests performed and depends on whether any true associations exist — which, in practice, are never known. For instance, when testing 10 independent candidate risk factors, there is roughly a 40% probability of obtaining at least one false positive finding, irrespective of sample size (figure 2, top). If the study includes one true risk factor and a sample size of 250 participants, the probability that an observed statistically significant association reflects a real effect is only about 50% (figure 2, bottom). Researchers might assume a 95% probability that a significant risk factor is real.

Researchers might include all of their candidate risk factors into one model, or use algorithmic or stepwise methods to reduce the number of variables. Algorithmic or stepwise methods involve iteratively adding and removing independent variables based on the significance of their associations with the outcome, or the impact on model residuals (the model fit).

Stepwise variable selection exacerbates the problems of multiple statistical testing. By comparing combinations of risk factors and selecting those with significant associations, this procedure maximises false positive results,^23 24^ and can be viewed as a type of automated P-hacking. P-hacking has been defined as “trying out several statistical analyses and/or data eligibility specifications and then selectively reporting those that produce significant results.”^25^ Researchers might use P-hacking to deliberately maximise their scientific publications at the cost of misleading results. More often, well intentioned researchers are unaware that the stepwise algorithms are a type of P-hacking. These problems are also known as data dredging^26^ or fishing expeditions because the researcher looks for associations in a dataset rather than testing a hypothesis or theory based on previous knowledge.

Post hoc Texas sharpshooter fallacy and HARKing

The Texas sharpshooter fallacy is an analogy to describe hypothesising about associations after these associations have been observed. Imagine that someone is doing target practice when no one is watching. They fire a gun at a blank wall and then draw a target around the tightest cluster of bullet holes. They then invite people to observe the accuracy of their aim. This practice is also known as Hypothesising After the Results are Known (HARKing).^27^

“Factors associated with” studies are a special case of the Texas sharpshooter fallacy and HARKing. Classically, these practices involve pretending that the hypothesis (or target) was already present before the research was started, and therefore have an element of dishonesty. Many “factors associated with” studies do not claim they had prior hypotheses. The conclusions, however, often suggest mechanisms for the observed risk factors as if these proposed mechanisms were being tested. In fact, these mechanisms are being suggested post hoc.

In summary, these three problems mean that the results of “factors associated with” studies do not provide useful insights into causal relations between exposures and outcomes. Methods that estimate causal effects in observational research must be guided by counterfactual theories about what might have happened to people in unobserved exposure statuses. Introductions to these problems and tools such as directed acyclic graphs are available elsewhere,^28 29^ and recommendations for avoiding some common pitfalls are provided in box 2.

Defences of “factors associated with” studies

Results are exploratory or hypothesis generating

Researchers might accept that limitations exist in a “factors associated with” analysis, but argue that the results give a useful initial indication of potential risk factors. Researchers might say that the results are hypothesis generating and can inform confirmatory studies with a more focused study design. But how often has an important hypothesis been generated for the first time from a “factors associated with” study, and then confirmed to be true? We have yet to identify an example. If readers can identify examples, which we would like to hear about, we wonder if these justify the waste from the large number of “factors associated with studies that are done each year.

In some cases, “factors associated with” studies are not only wasteful, but potentially harmful. Two of the examples above^7 18^ suggested that tobacco smoking might prevent serious covid-19, which could undermine public health efforts to reduce the prevalence of smoking. In another example, researchers examined the risk factors associated with recent high intensity physical activity in patients with hypertrophic cardiomyopathy who died during physical activity of any intensity.^30^ The results suggested that younger age was associated with “high intensity physical activity related sudden cardiac death”. The researchers concluded that younger patients (but not older patients) should be advised against high intensity physical activity, which could undermine clinical guidance and evidence from randomised trials.^31 32^

A “factors associated with” study might identify an important risk factor by chance, but at the cost of misleading findings and substantial research resources.

Results show associations rather than causal effects

Some researchers argue that an analysis of observational data is not attempting to quantify the effect of a risk factor, and therefore the rigour required to measure a causal effect is not needed. Instead, researchers might claim that the risk factor is “independently associated” with the outcome, and that estimating the size of an association is different from estimating a causal effect.

We have argued that the findings of a factors associated with study can reflect the arbitrary design of a regression model rather than processes of substantive importance in the real world. Moreover, if the findings are interpreted in ways that imply the exposure should be modified or might contribute to the risk of a disease, then the inference is inherently causal. The term independently associated, if not implying causation, does not mean anything. This language can obscure methodological challenges or downplay the need for a clear research question and careful causal reasoning. Even if the discussion section of the article highlights the limitations of the method, the results may still be highlighted and promoted by journalists or policy makers.

Results can help prioritise groups for more support

Based on the results of “factors associated with” studies, researchers might conclude that subgroups should be prioritised for effective interventions. In the example of US enlisted marines,^20^ one might argue that relationship counselling probably did not cause suicide, but nonetheless marines who receive relationship counselling are at higher risk. Therefore, those receiving counselling could benefit from interventions to prevent suicide. This is a question about prediction rather than causation. If the question is whether marines should be prioritised for interventions to prevent suicide according to whether they received relationship counselling, then the relevant value is the univariable association between relationship counselling and suicide, rather than this association after adjustment for other variables, such as deployment to war zones and traumatic brain injury. If the multivariable model is used to prioritise interventions to prevent suicide, then the relevant value would be the predicted risk for each marine based on the full model, rather than coefficients for individual exposures. Neither approach would involve inferring a causal relation based on the significant association between receiving counselling and suicide, after adjustment for the other available variables.

Multivariable regression models can estimate causal effects (what is the effect of an exposure on a disease?) or predict outcomes (who is most at risk?) but cannot do both. “Factors associated with” studies often confuse these different aims, and use language and methods related to both paradigms in the same research study.

Conclusions

We recommend that researchers should not use the “factors associated with” method and scientific journals should not publish these analyses. Many examples of “factors associated with” studies have results that seem reasonable. We are concerned, however, that these studies add little beyond common sense, and may be misleading and harmful. To our knowledge, no “factors associated with” studies have led to important scientific progress, despite the publication of many studies each year. Hence we believe that the “factors associated with” study design should be abandoned.