Authors: Felix B. Muniz (1)Center for Indigenous Health, Johns Hopkins University), D. P. MacKinnon (2)Department of Psychology, Arizona State University)
Categories: Article, suppression, mediation, regression
Source: Multivariate behavioral research
Authors: Felix B. Muniz, D. P. MacKinnon
Suppression effects are important for theoretical and applied research because these effects occur when there is an unexpected increase in an effect when it is adjusted for a third variable. This paper investigates three approaches to testing for statistical suppression. The first test was proposed by Velicer (1978) and is based on the relationship between the zero-order and semi-partial correlations. The second test comes from a condition that is necessary for suppression proposed by Sharpe and Roberts (1996). The third test is an extension of the test for the inconsistent mediated effect (MacKinnon, Krull, & Lockwood, 2000). We derive standard errors for the Velicer, and Sharpe and Roberts tests, conduct a statistical simulation study, and apply all three tests to two real data sets and several published correlation matrices. In the simulation study, the test based on inconsistent mediation had the best properties overall. For the data examples, when raw data were available, we constructed bootstrap confidence intervals to assess significance, and for correlations, we compared each test statistic to the normal distribution to assess statistical significance. Each test gave consistent results when applied to the example data sets. Analytical work demonstrated conditions where each test gave conflicting results. The mediation test of suppression based on the sign of the product of the mediated effect and the direct effect had the best overall performance.
According to Horst (1941, p. 434), a variable that is unrelated to the outcome could increase the predictive ability of another predictor variable when it is added to the regression equation. Horst called this variable a suppressor variable because it suppresses criterion-irrelevant noise in the other predictor. Horst described what is now called classical suppression, that is when the suppressor variable is uncorrelated (or weakly correlated) with the dependent variable. In 1966, Horst described an example of a study involving mechanical ability predicting pilot performance. A third variable, verbal ability, was not thought to be associated with pilot performance but when included in the model, the overall prediction of mechanical ability increased. Verbal ability was required to read the test for mechanical ability and therefore explained some of the residual variance in mechanical ability, making mechanical ability a stronger predictor. In general, we consider a suppressor variable to be one that increases the predictive ability of another predictor variable after the suppressor is added to the equation.
The unique nature of suppressor variables has captured the attention of psychological and methodological researchers for many years (Beckstead, 2012; Conger, 1974; Horst, 1941; Kim, 2019; MacKinnon, Krull, & Lockwood, 2000; Velicer, 1978). Suppression effects can manifest in different ways depending on the model and analysis used. For example, in mediation analysis, suppression is present when the direct and indirect effects have opposing signs and when adjustment for a confounder increases the relation between two variables, called negative confounding (MacKinnon et al., 2000). In terms of correlations, suppression is present when the zero-order correlation is less than the semi-partial correlation (Velicer, 1978). Kim (2019) explored suppression within the causal discovery framework as the causal relation between three variables and proposed that suppression is another form of an instrumental variable. Beckstead (2012) stated that “Suppression is viewed as the result of criterion-irrelevant variance operating among predictors” p. 224. Suppression effects are interesting because they are counterintuitive in that adding a suppressor variable increases the predictive ability of other variables rather than reducing it.
In a real study, MacKinnon et al. (2000) found a suppression or inconsistent mediation effect for a program designed to reduce intentions to use anabolic steroids in high school football players. One of the mediators studied was reasons for using steroids. The intervention (X) increased knowledge of reasons to use steroids (M) and reasons to use steroids increased intentions to use steroids (Y) when compared to a randomized comparison group. Overall, the intervention significantly reduced intentions to use steroids. The magnitude of c^′, the direct effect was, larger than c^, the program effect without adjustment for the mediator consistent with suppression. In this example, the mediated effect was positive, and the direct effect of the intervention was negative. MacKinnon et al. proposed a test for suppression that tests the statistical significance of the indirect effect and if the sign of the indirect effect differs from the direct effect, then suppression is present. The method performed well in a large simulation study included in supplemental material for MacKinnon et al. (2000). Cheung and Lau (2008) showed that the product of coefficients test for suppression based on the signs of the direct and indirect effect, described in MacKinnon et al., worked well in a large simulation study that varied measurement error.
In another study, Gaylord-Harden, Holmbeck, Cunningham, and Grant (2010) sought to replicate hypothesized suppressor effects in separate samples. Working within the structural equation modeling (SEM) framework, they created latent variables with multiple indicators for each measure to reduce measurement error and considered sign changes and increases in magnitude of path coefficients as evidence of suppression. They hypothesized and found evidence that support-active coping and support-seeking coping acted as suppressors of the relationship between avoidant coping and internalizing symptoms. Substantively, they used the suppression conclusions as evidence to support their hypothesis that coping strategies are interdependent, and as an explanation of how suppression can enhance the understanding of how these coping strategies work together. They took steps to correct for measurement error by using a latent variable model and still found significant suppression effects based on the mediation test for change in regression coefficients described by MacKinnon et al. (2000).
Velicer (1978) proposed that suppression can be identified by an inequality derived from standardized regression coefficients and semi-partial correlations. Velicer proposed that if the squared correlation between the predictor and outcome is smaller than the squared correlation between the predictor and the outcome with the effect of the third variable partialled out, then there is evidence of suppression. He generated the estimator based on Conger’s (1974) definition of suppression as occurring when the regression coefficient of the predictor increases after the third variable is added to the regression equation. Velicer applied the suppression definition to data for classical (suppressor has little or no relation to the outcome), reciprocal (predictors are suppressors of each other), and negative (inclusion of the suppressor creates a negative relationship between predictor and outcome) suppression using data from earlier papers on suppression (Table 1, p. 957).
Based on Velicer’s (1978) work, Gignac (2018) investigated socially desirable responding (SDR) as a suppressor of self-reported intelligence (SRI) in its relationship with task-based intelligence (TBI), used as a proxy of actual intelligence. The zero-order correlation of SRI and TBI was compared to the semi-partial correlation between the same two variables and partialling out the effect of SDR. A numerical increase from the zero-order to semi-partial correlation was taken as evidence of suppression. Gignac used a 1-2% increase in r2 to suggest moderate to large suppression effects based on a simulation study from Paunonen and Label (2012). Gignac (2018) noted that the difference of the zero-order and semi-partial correlations had not been tested for statistical significance because the standard error formula for Velicer’s test had not yet been derived.
Suppression is a complex phenomenon with a meandering history of definitions and literature. This paper adds to this literature in hopes of providing a starting point for statistical analyses in addition to the theoretical discussion that exists. For a more comprehensive review of suppression see Maassen and Bakker (2001). In this paper we examined three approaches to suppression, two based on correlations and semi-partial correlations, while the third is based on regression coefficients. Since there are direct transformations between correlations and regression coefficients, we operationalize suppression using a common three variable model, the mediation model. The simple, elegant, and familiar model allows for each approach to suppression to be visualized and conceptualized for better understanding of this complex topic.
To clarify our discussion of suppression we present three equations for a model with one suppressor variable (M), one independent variable (X), and one dependent variable (Y). Figure 1 shows the relationships between X and Y before and after the suppressor is added to the model. Equations (1), (2), and (3) show the regression equations for these models. Note that Figure 1 is equivalent to the single mediator model (MacKinnon et al., 2000) and we describe suppression in the context of the widely applied mediation model.
For this paper, it assumed that X, M, and Y are continuous variables and there are linear relations between the variables. M is the suppressor variable. The c coefficient represents the relation between X and Y (see Figure 1); the c’ coefficient represents the relation between X and Y, adjusted for M; the b coefficient represents the relation between M and Y adjusted for X; and the a coefficient represents the relation between X and M. The regression residuals are e1, e2, and e3, and the intercepts are i1, i2, and i3. In a sample, a^, b^, c^, and c^′ are estimators of a, b, c, and c’, respectively.
MacKinnon et al. (2000) showed that mediation, confounding, and suppression effects were statistically equivalent in the single mediator model. For the single mediator model with linear regression, they note the two ways to estimate the mediated effect, ab and c-c’, and that the difference in coefficients, c-c’, offers a conceptualization of suppression that is consistent with earlier definitions of suppression. They considered Conger’s definition of suppression, which is when the regression coefficient between two variables increases in magnitude (absolute value of the regression coefficient) when adjusted for the suppressor. They proposed a more specific definition for suppression in the mediation model as when the direct, c’, and indirect effects, ab, having opposing signs.
Four situations presented in table 1 demonstrate suppression effects in mediation regression equations in terms of the estimates of ab, c, and c’ from MacKinnon, Warsi, and Dwyer (1995). These conc00lusions are based on work from MacKinnon et al. (1995) proving that ab is equivalent to c-c’ for continuous M and Y. With this equivalence in mind, table 1 demonstrates four situations in which suppression effects can manifest in the single mediator model. Implicit in this conceptualization is the idea from Conger that the magnitude of the coefficient of the relationship between X and Y gets larger after the suppressor is added ∣c′∣>∣c∣.
The first suppression situation occurs when the indirect effect (a∗b) is negative and the direct effects (c and c’) are positive (Fig. 1). The first row in table 1 shows that the magnitude of c’ is larger than the magnitude of c since c-c’ is negative. The second situation occurs when the indirect effect (a∗b) is positive and the direct effects (c and c’) are negative. In the second row of table 1 the magnitude of c’ is larger than the magnitude of c since c-c’ is positive, which shows an increase in the direct effect coefficient after the third variable is added to the model. The third a fourth rows demonstrate the last suppression situation. This situation occurs when the sign of the direct effect changes after the third variable is introduced into the model. For example, c will be positive before the third variable is introduced and negative after the third variable is introduced (c’) and the indirect effect will be positive. However, in this situation it is not analytically possible to tell if c or c’ is larger in magnitude, similarly vice versa, because of the change in sign of c and c’. Thus, the definition that requires the magnitude of c’ to be larger than c is not satisfied for suppression in this situation. This situation may not be as common as the first two suppression situations, but it is an example of suppression even if the magnitude of c’ versus c does not increase. This last situation shows that although the magnitude did not increase, there is a change in the sign of coefficients.
In this study we examined suppression situations that arise in the mediation context as outlined in the first two rows of table 1. In terms of previously outlined suppression situations, we examined reciprocal suppression which occurs when the coefficients increase for both predictors after either is added to the regression analysis. This situation does not explicitly identify which predictor is a suppressor, only that suppression is present.
Although there has been considerable research and methodological attention to suppression, there have been few tests developed specifically for suppression. There are four goals of this paper. The first goal is to describe three approaches to testing for suppression based on prior research. The second goal is to derive standard errors for those tests that do not already have them and describe resampling methods for building confidence intervals. The third goal is to apply these tests to real data and compare their performance. The fourth goal is to investigate these tests using a Monte Carlo simulation. Each goal is addressed in order below.
As demonstrated in the previous section, suppression can be approached from multiple perspectives including, but not limited to, correlation, regression/mediation, and measurement. Within each perspective there is no consensus on how this statistical phenomenon behaves and manifests. Given these limitations, we do our best to present each of the three tests using similar language and labels while acknowledging that they are not exactly aligned in how they are testing for suppression. Rather, these are three different approaches to testing for suppression.
Velicer (1978) defined suppression based on correlations. He uses an argument proposed by Conger (1974) as the starting point for his definition. Conger’s argument “the question of interest under what conditions…will the multiple regression contribution of the first predictor exceed its zero-order contribution.” This argument is operationalized by considering the situation in which a multiple regression coefficient is defined in terms of zero-order squared correlations shown in equation 12.
He noted that in a typical situation the sum of the correlations on the right side of equation 12 is greater than the left side. He then proposed that the case where the left side of equation 12 is greater than the right side is indicative of suppression. With some substitution and algebra, Velicer proposed that if the semi-partial correlation is greater than the zero-order correlation then there is a suppression effect, shown in equation 13. Note that in a two-predictor equation, this relationship between correlations is reciprocal in that it does not distinguish which variable is the suppressor, only that suppression is present.
The Velicer estimator is equal to the Rmed2 mediation effect size measure for the variance accounted for by the mediated effect (Fairchild, MacKinnon, Taborga, and Taylor, 2009; Taborga, 2000). A proof of the equivalence of the Velicer test and Rmed2 is shown below.
The logic behind the Rmed2 effect size was that by isolating the variance accounted for in Y by each predictor, a measure of the size of the mediated effect could be demonstrated. The authors noted that Rmed2 may take on negative values since it does not square the differences of each element in the equation. In (13), we show that when Rmed2 / Velicer is negative, it indicates suppression.
The Velicer approach to suppression builds off work originally conducted by Horst in 1941 and Conger in 1974 and is considered by many as classical suppression. This approach to suppression is based on correlations which are familiar to most researchers and readily available in many published studies.
Based on Velicer’s and Fairchild et al.’s formula, we derived a standard error for (13) as shown below in equation 14 (the full derivation based on the multivariate delta method is provided in Appendix A).
Where: (15)sVelicer=Var(Velicer)
So, for the Velicer test for suppression we (16)PointEstimator=rmy2−rmy.x2 (17)CriticalValueTest=rmy2−rmy.x2sVelicer (18)Suppressioncriterion=ifrmy2−rmy.x2<0
Sharpe and Roberts (1997) examined sums of squares and correlation coefficients to gain a better understanding of suppression effects. Their work led to a necessary and sufficient condition for suppression which they investigated algebraically and graphically. They created a ratio of correlations called γ, which is equal to the correlation between the outcome (Y) and the predictor that is entered into the equation first (X), divided by the correlation between the outcome and the predictor added into the equation second, or the suppressor (M), (∕rmyrxy). After some simplification and algebra, they come up with a condition that holds when suppression is present, shown in equation 19.
This inequality is reversed for the case where rxm<0; and when rxm=0 suppression cannot occur. They posit that suppression can be identified directly by putting elements of the correlation matrix into the inequality shown in equation 19. They also claim that suppression is dominated by γ since the minimum R2 value is attained when rxm=1∕γ (Mitra, 1988). This approach is more complex than the Velicer approach, however it frames suppression in terms of bivariate correlations which are commonly understood and more accessible.
Similar to the Velicer test for suppression, a formula for the standard error is needed. Following the same logic as the derivation for the Velicer standard error, the multivariate delta method was used (full derivation in Appendix B) and the formula is shown below.
So, for the Sharpe and Roberts test for suppression we (22)PointEstimator=rxm−2γ1+γ2,whereγ=rxyrmy (23)CriticalValueTesst=rxm−2γ1+γ2SSR (24)Suppressioncriterion=rxm−2γ1+γ2>0whenrxm>0orrxm−2γ1+γ2<0whenrxm<0
This test was based on the inconsistent mediation test from the MacKinnon et al. (2000) paper described previously. This test takes the product of the mediated effect (commonly referred to as a∗b) and the direct effect after adjusting for the mediator (c’) This approach is from a regression/mediation framework, which are common concepts and analyses in psychology. This test utilizes commonly reported estimates (a∗b) and one that requires a simple calculation based on readily available estimates (c’=c-a∗b). The framing of suppression in this approach is intuitive to researchers utilizing regression and mediation models and offers interpretations in line with interpretations in published studies.
A standard error for this test was derived based on methods for the standard error of three independent coefficients (Taylor, MacKinnon, & Tein, 2008). The standard error described here includes a covariance between the b and c’ coefficients.
In summary, the purpose of this research was to develop and apply tests for suppression. In addition to testing suppression by dividing the estimates of suppression by their standard error, we also create confidence limits using the bootstrap method. Bootstrap resampling has been shown to be more accurate for mediation and related analyses because it more accurately models the non-normal distributions involved in the suppression estimators (MacKinnon, Lockwood, & Williams, 2004).
A simulation study was conducted using SAS 9.4 to assess bias, type-I error rates, power, coverage of the suppression estimators, and the accuracy of the derived standard errors in three no suppression or mediation (all correlations set to zero), complete mediation (c’=0, other parameter values vary, and direct and indirect effects have the same sign), and suppression (parameter values vary and the direct and indirect effects have opposing signs). Parameter values were varied between 0, .14, .39, .59, and .89 to reflect null, small, medium, large, and very large approximate effect sizes (MacKinnon et al., 2002). We evaluated these conditions for sample sizes of 50, 100, 200, 500, and 1,000. A total of 1,000 replications were used for each sample size and condition. In total there were 72 conditions across parameter values and sample sizes. Table 3 shows example conditions for the three situations we simulated. We calculated standardized bias for the standard errors and the estimators by comparing the average value of the derived standard errors (ω^) across 1,000 replications to the average standard deviation (ω) of the estimator across 1,000 replications.
For the simulation study, normal theory tests for suppression were created by dividing each estimator by its derived standard error and comparing it to the Z-distribution. For the bootstrap method, statistical significance was assessed by whether zero was in the confidence interval or not. Type I error rates were set at α=.05, as is common in psychological studies and empirical Type I error rates were considered acceptable if they ranged between .025 and .075. Bootstrap confidence intervals, type-I error rates, power, and coverage were obtained for each condition and sample size in the simulation study.
The tests for suppression were applied to two raw data sets and two published correlation matrices. The first raw data set comes from a study by Gignac (2018) in which socially desirable responding was hypothesized to suppress self-reported IQ when predicting task-based IQ (actual IQ) in a sample of 253 first year undergraduate students at a large university in Australia. The second data set comes from a study by Goldberg et al. (1996) where a program aimed at preventing anabolic steroid use was delivered to 15 out of 31 high school football teams. The study found that while there was a significant reduction in intentions to use steroids between intervention and control groups, there were also targeted mediators that unintentionally increased intentions to use steroids which reflect suppression effects. The third application is to a correlation matrix published in a study conducted by Gaylord-Harden et al. (2010) where active coping and support seeking coping were hypothesized suppressors of the relationship between avoidant coping and internalizing symptoms with 497 participants. The fourth application is to a correlation matrix published in a study that investigated shame and guilt and their relationship to alcohol-related outcomes through mechanisms of impulsivity with 419 participants (Patock-Peckham, Canning, & Leeman, 2018). Although this study did not have suppression hypotheses, shame and guilt have been shown to be mutual suppressors (Paulhus et al., 2004; Tangney & Dearing, 2002).
The three tests were also conducted on a table of correlations from Velicer (1978, Table 3) as well as three sets of correlations that described three types of suppression based on prior articles that addressed tests for suppression (Table 4). The first type is classical suppression from McNemar (1945) where the suppressor variable has no relationship with the outcome. The second type is reciprocal suppression from Conger (1974) where both predictors are suppressors of each other. The last type is negative suppression where an original negative relation between X and Y becomes positive after adjustment for the suppressor. Velicer used these data to demonstrate that his method gave evidence for suppression for the three different types of suppression.
For the raw and correlation data sets, the suppression tests were conducted by computing the point estimates and standard errors and then dividing the point estimate of each test by its standard error and comparing the obtained Z value to the standard normal distribution. Confidence intervals were created for the data sets that contained the raw data by using the bootstrap technique (10,000 draws) to address possible non-normality in the point estimates.
Standardized bias of the suppression estimators was considered acceptable when its magnitude was less than 0.1. For the Sharpe and Roberts estimator, it was negatively biased in the suppression condition, positively biased in the no suppression condition, and reached acceptable standardized bias levels in the complete mediation condition at a sample size of 200. For the Velicer estimator, it had acceptable standardized bias in the no suppression and complete mediation conditions at all sample sizes and had acceptable standardized bias in the suppression condition at a sample size of 200. In aggregate, the Velicer estimator reached acceptable levels after sample size of 200 and the Sharpe and Roberts estimator reached acceptable levels after sample size of 500. The standardized bias of the a∗b∗c’ estimator was acceptable in all conditions and sample sizes except for the complete mediation condition at sample size of 50.
The accuracy of the derived standard errors was assessed by comparing the average value obtained from the formula to the average empirical standard deviation of the estimator across 1,000 replications and standardized bias. Standardized bias equaled the derived standard error divided by the empirical standard deviation of the derived standard error, providing a measure of standardized bias appropriate for all parameter combinations in the simulation. Standardized bias was averaged across the 1,000 replications for each parameter combination. The standardized bias for the Velicer standard error was greater than 0.10 only in the no mediation condition in the 50 and 500 sample sizes. In both the complete mediation and suppression conditions, standardized bias was below 0.10. For the Sharp and Roberts standard error, the complete mediation and suppression conditions exhibited acceptable standardized bias across all sample sizes. The no mediation condition exhibited acceptable standardized bias at a sample size of 500 and higher.
The a∗b∗c’ and Velicer tests had type-I error rates below 0.05 for all sample sizes. The Sharpe and Roberts test had the lowest type-I error rate at the smallest sample size and continued to rise as sample size increased, eventually reaching 0.266, an unacceptably high Type I error rate.
Across all nonzero suppression conditions, for the Velicer estimator, power of 0.80 was reached between a sample size of 500 and 1000. Power for the Velicer estimator was as low as 0.193 with a sample size of 50 and was 0.460 with a sample size of 200 (Table 7). For the Sharpe and Roberts estimator, power of 0.80 was reached between samples sizes 500 and 1000. For the Sharpe and Roberts estimator, power was as low as 0.412 with a sample size of 50 and 0.631 when the sample size was 200 (Table 7). The a∗b∗c’ test for suppression reached .8 power at sample size of 500.
The a∗b∗c’ estimator had adequate coverage beginning at sample size of 100 and had the best coverage at sample size of 50. The coverage for the Velicer and Sharpe and Roberts estimators decreased as sample size increased.
For the bootstrap analysis, the Velicer estimator had acceptable type-I error rates across all sample sizes. The Sharpe and Roberts estimator had acceptable type-I error rates for sample sizes of 50 and 100. However, the type-I error rate for the Sharpe and Roberts estimator increased as sample size increased and reached 0.248 with a sample size of 1000, as did the normal theory tests. For the Velicer estimator, the bootstrap power reached 0.80 between sample sizes of 500 and 1000. The lowest level of power was 0.184 at a sample size of 50 for the Velicer estimator. For the Sharpe and Roberts estimator, the bootstrap power reached 0.80 between sample sizes 500 and 1000. The lowest level of power was 0.302 at sample size of 50 for the Sharpe and Roberts estimator. The a∗b∗c’ estimator had type-I error rates below 0.05 for all sample sizes. It reached 0.8 power around a sample size of 500 and had adequate coverage for all sample sizes.
In the original Gignac (2018) study, socially desirable responding was hypothesized to be a suppressor of the relationship between self-reported IQ and task-based IQ. Our analysis found that the suppression criteria for each test did indicate that suppression was present. That is, a∗b∗c’ was negative, the Velicer estimate was negative, and the SR estimate was the same sign as the correlation between SDR and SRIQ (positive). However, a∗b∗c’, Velicer, or Sharpe and Roberts were not statistically significant (a∗b∗c’=−0.22, bootstrap C.I. [−1.351, 0.072], Velicer = −0.004, bootstrap C.I. [−0.018, 0.005], SR = 0.889, bootstrap C.I. [−0.6311, 0.355]). The evidence for suppression in these data was consistent with chance.
For the Goldberg et al. (1996) study, all three tests indicated a statistically significant suppression effect (a∗b∗c’=−0.027, bootstrap C.I. [−0.061, −0.005], Velicer = −0.0075, bootstrap C.I. [−0.015, −0.001], SR = 0.678, bootstrap C.I. [0.174, 1.038]). In the Goldberg et al. (1996) study, the treatment program decreased intention to use steroids, but the reasons to use steroids mediator increased intentions to use steroids. This demonstrates a positive mediated effect, and a negative direct effect which is consistent with the mediation definition of suppression as a triple product with a negative sign. All three tests conducted supported the original findings of a suppression effect.
Gaylord-Harden et al. (2010) hypothesized a suppression relationship and tested it by testing the increase in coefficients after the suppressor was included in the analysis. They found evidence to support their hypothesis and concluded that there was a significant suppression effect in two separate samples. All three tests, using their correlation matrix, supported the original findings that there was a significant suppression effect (a∗b∗c’=−0.013, Z = −2.99, p < 0.01, Velicer = −0.012, Z = −1.67, p < 0.05 SR = 1.51, Z = 4.87, p < 0.001).
Although there were no suppression hypotheses in the Patock-Peckham et al. (2018) study, we were able to support that shame and guilt were mutual suppressors of each other, proposed originally by Tangney & Dearing (2002). All three tests for suppression were statistically significant (a∗b∗c’=−0.026, Z = 3.66, p < 0.001, Velicer = −0.030, Z = −2.78, p < 0.01 SR = 1.10, Z = 3.41, p < 0.001).
Since the table in the Velicer (1978) article did not include the sample size of the studies that produced the correlations, we specified sample sizes of 100, 200, and 500. In the classical suppression condition, the Sharpe and Roberts test is undefined because of the zero correlation between M and Y, which is in the denominator of γ in the estimator. For each condition and sample size, besides the condition mentioned previously, each test produced a significant suppression effect (see Table 8). Each test led to the accurate conclusion regarding suppression for the three types of suppression outlined by Velicer (1978). The change in standard error as sample size is increased is also shown in the table.
We compared the different tests of suppression analytically for many different parameter values to locate situations where the tests lead to different answers. Because we know the analytical formulas for each method, the true covariance matrix for three variables, X, M, and Y (MacKinnon, 2008, p. 88), and the correlations among the three variables, we can obtain the true values for all suppression estimators and standard errors for any combination of parameter values, correlation, and sample size. Assuming that the critical ratio of the true effect divided by the true standard error is normally distributed, we can also compare the statistical power of each method. However, there is a limitation of the power calculations because a normal distribution is assumed for the ratio of the true test coefficient and its standard error, which may not be accurate for some values, particularly at small samples. We calculated each test for all combinations of a, b, and c’ from −0.9 to 0.9 by increments of 0.1 for sample sizes of 50, 100, 200, 500, and 1000. The program in Appendix C, computes the suppression tests for any correlation matrix of three variables.
According to our analysis, there were two situations when the three tests for suppression did not agree. The first situation occurred when the magnitude increases of c to c’ was less than 0.1. This indicates that there was little change in the relationship between X and Y after the suppressor was included in the analysis. The a∗b∗c’ test still indicates that suppression is present since it is negative. However, in this case both the Velicer and Sharpe and Roberts tests did not meet their respective conditions of suppression (i.e., Velicer < 0; Sharpe and Roberts had the same sign as the correlation between X and M). The indication or detection of suppression is necessary in order to assess the significance. The second situation where the test conclusions differed was when two of the three correlations were either very large (> 0.7) with the third being very small (< 0.3) or two of the three correlations were very small, and the other correlation was very large. The a∗b∗c’ test allows for this situation to occur and does not affect its calculation. These situations may be extreme cases and may not reflect common situations in psychology when investigating suppression effects.
The multivariate delta method was used to derive an equation for approximate standard errors for the Velicer and Sharpe and Roberts estimators. Although the multivariate delta method is a large sample approximation, we saw that it was generally accurate even at the smaller sample sizes in our simulation study.
The standardized bias of the Velicer estimator was acceptable in the no mediation/suppression and complete mediation conditions (Fig. 2). However, it did not reach acceptable levels in the suppression condition until a sample size of 200 and was positively biased in all three conditions. Since negative values indicate suppression, being positively biased means that the estimator underestimates the population value when suppression is present. The standardized bias of the Sharpe and Roberts estimator reached acceptable levels in the complete mediation condition after reaching a sample size of 100 (Fig. 2). In the suppression condition, it was negatively biased. Because this estimator can take on both negative and positive values, on average it underestimates positive and overestimates negative population parameters. The Sharpe and Roberts estimator became less biased as sample size increased (Fig. 2).
The Sharpe and Roberts estimator had inflated type-I error rates and poor coverage and typically would not be recommended for use. However, as seen in the application to real data, it gave similar results to the other two estimators. It is important to note that the original Sharpe and Roberts paper did not intend for their work to lead to an estimator, rather it was deemed a necessary and sufficient condition for suppression. We formed a significance test based on the formulation. Perhaps it should be used only as a necessary and sufficient condition for suppression and not an estimator or test of significance.
The Velicer estimator had poor coverage and was biased. These two issues may be related in that the bias of the estimator may have been large enough such that the width of the confidence interval was unable to capture the true value (Table 11). This is exacerbated as the width of the interval shrinks as sample size goes up. The Velicer estimator is positively biased which means that it is closer to zero than the true value since only negative values indicate suppression. However, as with the Sharpe and Roberts estimator, problems with the Velicer estimator may not be detrimental in many real data sets.
None of the estimators distinguish which predictor is the suppressor, only that suppression is present. That is, you would get the same estimates whether you identify X or M as the suppressor for the calculation.
The complete mediation condition had several important results for testing for suppression. First, we must consider whether suppression is possible in the complete mediation condition. According to the definition of suppression in the mediation model, suppression is not possible in this condition since c’ is zero and zero is neither positive nor negative. So, the requirement for a∗b∗c’ to be negative is not possible if the population parameter is zero and would appear to be negative by chance roughly 50% of the time. That is, random error will cause the sample parameter values to be positive or negative roughly half the time which leads to false suppression detection (Table 12). However, false detection only seems to be problematic at a sample size of 50, beyond that, all three accurately conclude a non-significant suppression effect in the complete mediation condition at an acceptable rate.
The Sharpe and Roberts and Velicer estimators presented problems that might be too difficult to overcome in practice. For the Sharpe and Roberts estimator, the type-I error rate behavior was particularly concerning. As sample size increased, the type-I error rate increased, and it did not have adequate power until beyond a sample size of 500. So, if a researcher had enough data to have enough power to detect an effect, the risk of making a type-I error was simultaneously greater. It also had unacceptable standardized bias in the suppression condition, which is typically the only time it would be used. Depending on the sign of the estimator, it would be under or over estimated. For instance, if the population value was negative, the Sharpe and Roberts estimator would overestimate it since it is negatively biased. As mentioned earlier, it may be wise to calculate the Sharpe and Roberts estimator as a necessary and sufficient condition for suppression, as it was intended. In practice this would mean that if the correlation of X and M (the suppressor) was positive and greater than 2γ1+γ2 then one could conclude that suppression was present, but not whether it was statistically significant or not.
For the Velicer estimator, the bias and coverage were concerning and possibly related. The Velicer estimator was positively biased and since it is negative when suppression is present, it underestimates the population value. It requires a sample size of 500 or more for adequate power with the parameters presented in this study but as sample size goes up, coverage gets worse.
The a∗b∗c’ test for suppression performed the best and has a clear statistical hypothesis based on observed regression coefficients (a∗b∗c’<0). The type-I error rate and coverage were acceptable, and the power reached 0.80 at a sample size of 500. The a∗b∗c’ estimator did not detect suppression in the complete mediation condition, which is consistent with its own criteria for suppression but not with the Velicer definition for classical suppression in which there is no relationship between the suppressor and the outcome. Although the classical suppression and complete mediation situations are not the same theoretically, mathematically they are identical since there is no distinction between which predictor variable is the suppressor.
This project examined and created possible statistical tests for suppression in hopes of providing researchers with a way that would help them identify and assess the significance of suppression in research. Suppression is a complex phenomenon that requires solid causal theory and statistical analysis to assess its effects properly. This project provides three possible ways to investigate statistical suppression with a∗b∗c’ being the preferred test. In these analytical results, the suppression test based on a∗b∗c’ gives different answers than the other two tests in specific cases showing limitations of the Velicer and Sharpe and Roberts tests.
These ways to test for suppression differ in ease of computation. The test a∗b∗c’ is a straightforward test and has the most intuitive hypothesis for suppression (a∗b∗c’<0). A researcher can apply each test and expect them all to lead to the same conclusions if suppression is present. We advocate that researchers should test whether suppression is consistent with chance by constructing confidence intervals because chance could lead to results consistent with a suppression effect. It is important to note that each method can assess whether suppression is due to chance but there is more to do after that to explain the substantive meaning of the results. What is the causal model for the three variables? With mediation it is inconsistent mediation. With confounding it is negative confounding. With colliders it is negative collision. In each case suppression is observed when there is evidence that c’ is larger in magnitude than c or that the product of a, b, and c’ is negative.
Some researchers discuss suppression as a manifestation of a violation of the multicollinearity assumption in linear regression (Beckstead, 2012; Friedman & Wall, 2005). Farrar & Glauber (1967) discuss an idea they termed “harmful multicollinearity” which essentially is describing negative suppression, where the sign of the coefficient of one of the predictors changes when the other is added to the regression equation. Beckstead (2012) was able to decompose the covariance matrix of X, Y, and the suppressor to remove the suppression effects. In this context, suppression was seen as a nuisance and was not theoretically relevant for the analysis. That paper also concluded that suppression effects were dependent on correlations among the predictors and the bivariate correlations between the predictors and the outcome. So, if you have a set of predictors that are correlated, suppression effects would be present depending on the outcome. In other words, the outcome plays a role in whether suppression effects are present as well as the correlations among the predictors.
As described by Gaylord-Harden, Holmbeck, Cunningham, and Grant (2010), suppression may also result from measurement error in either the suppressor, predictor, or both variables. By chance, the measurement error in a predictor could be associated with the measurement error in another variable. For example, say there is an unreliable measure of shame with a substantial amount of residual or error variance, and that the shame measure is used to predict depression. If a second predictor were added, for example guilt, which explained some of the residual variance in the shame measure, then a suppression effect would be observed. However, if the shame measure was highly reliable and had little residual variance then it would be difficult to have a guilt measure explain any of the residual variance in the shame measure and thus a suppression effect would not be observed.
The tests conducted in this paper were for three-variable models and it may be helpful to extend these tests for models involving more than three variables. The a∗b∗c’ test for suppression can intuitively extend, theoretically, to more complicated models however, more work is needed to explore its performance in those models. Velicer (1978) notes that the general approach of his method can also be used for multiple predictors, again in the framework of comparing an estimate before and after adjustment for one or more other variables. Additionally, the current study constructed tests based on the assumption that the estimators were normally distributed which may not be true. Confidence intervals were constructed using standard errors derived based on normal theory and bootstrap confidence intervals were constructed to potentially address non-normality. However, there are other methods out there that are worthy of investigation such as Bayesian credible intervals and Monte Carlo confidence intervals.
We acknowledge that the null distribution for suppression effects is not fixed but instead varies as a function of the parameter values themselves. This presents a conceptual challenge when interpreting p-values, as different null distributions can yield different conclusions for the same observed suppression statistic. In other words, the parameter that is hypothesized to be zero in those two scenarios drives what potential null distributions look like. Liu et al. (2023) proposed a general approach to Bayesian hypothesis testing for mediation that could be useful in addressing this issue of infinitely many null distributions for a∗b∗c’=0.
The theoretical conceptualization of suppression is a critical consideration that must precede any empirical testing framework. The operationalization of suppression should be guided by theoretical considerations rather than the performance of associated significance tests. In this study, we aimed to evaluate the performance of novel statistical methods for testing suppression effects once they were operationalized, rather than proposing a new conceptual definition of suppression. Future studies on the topic would benefit from making this distinction and possibly examining both scenarios in which parameter combinations are different.
The underlying theory for suppression provides many interesting questions and this study proposed a statistical test for suppression in a three-variable model by multiplying the three regression coefficients. Overall, the a∗b∗c’ test performed well with sample sizes of 500 or more. The test provides a way to investigate the unusual case where adjustment for a third variable increases the predictive power of two variables. The three tests proposed statistically explore suppression from three different theoretical conceptualizations of suppression. The a∗b∗c’ test takes a mediation approach to suppression. The Velicer test views suppression from a correlation and partial correlation lens. The Sharpe and Roberts test was creating by operationalizing a necessary and sufficient condition for suppression. Ultimately, the theoretical ideology on suppression is what should drive the examination of it. This study is a crucial first step in creating statistical tests for the complex statistical phenomenon that is suppression.