Are alternative variables in a set differently associated with a target variable? Statistical tests and practical advice for dealing with dependent correlations

Authors: Miguel A. García‐Pérez

Abstract

The analysis of multiple bivariate correlations is often carried out by conducting simple tests to check whether each of them is significantly different from zero. In addition, pairwise differences are often judged by eye or by comparing the p‐values of the individual tests of significance despite the existence of statistical tests for differences between correlations. This paper uses simulation methods to assess the accuracy (empirical Type I error rate), power, and robustness of 10 tests designed to check the significance of the difference between two dependent correlations with overlapping variables (i.e., the correlation between X 1 and Y and the correlation between X 2 and Y). Five of the tests turned out to be inadvisable because their empirical Type I error rates under normality differ greatly from the nominal alpha level of .05 either across the board or within certain sub‐ranges of the parameter space. The remaining five tests were acceptable and their merits were similar in terms of all comparison criteria, although none of them was robust across all forms of non‐normality explored in the study. Practical recommendations are given for the choice of a statistical test to compare dependent correlations with overlapping variables.

INTRODUCTION

The statistical analysis of correlations among variables has not changed much over decades of psychological research, nor has it been essentially altered by an increasing concern about matching statistical methods to the intended inferences. The most common approach to dealing with large numbers of correlations (e.g., intercorrelations among all pairs of variables in a set or correlations between each variable in a set and each variable in another set) consists of reporting all of them accompanied by a description of which of them, considered in isolation, are significantly different from zero. In addition, inferences about differences in correlation are usually made by comparing the p‐values of the respective isolated tests of significance (for a relatively old but representative example, see Thorson & Powell, 1993). The inadequacy of this strategy for comparing correlations is easy to understand on considering that the boundary for a test that a correlation is significantly different from zero is sharp. For instance, with sample sizes of 120 observations and a two‐sided test with α = .05, a correlation of .180 in one group will be significantly different from zero whereas a correlation of .179 in another group will not. While these separate claims about each correlation are unquestionable, they cannot sustain a claim that the strength of correlation between the two variables of concern is different in each group, a claim that seemingly calls for a direct test that the two population correlations differ from one another. Similar considerations hold when separate tests reject the null that the correlation is zero or when they do not reject it, whether with similar or with different accompanying p‐values.

Empirical scenarios requiring tests of equality of two correlations may have a number of distinct characteristics. The case mentioned in the previous paragraph involves equality of correlation between the same two variables in different populations, a case usually termed the statistical comparison of independent correlations because each correlation is computed using data from a different sample of bivariate observations. A variant thereof involves equality of the correlation between two variables and the correlation between two other variables, all four of them measured in a sample from the same population. This scenario pertains to what is generally termed the statistical comparison of two dependent correlations with non‐overlapping variables, where the term captures the fact that all observations come from the same multivariate sample but none of the observations is used twice (e.g., for the computation of more than one of the correlations of concern). A somewhat different scenario also involving a single multivariate sample is referred to as the comparison of dependent correlations with overlapping variables, because now some observations are used more than once for the computation of all relevant correlations. A prototypical example involves testing whether two variables, X 1 and X 2, are equally correlated with a third variable Y (see, for example, Blake & Palmisano, 2021; Guo et al., 2022). In such cases, observations in Y are used for computation of its correlation with X 1 and, once again, for computation of its correlation with X 2. Scenarios involving a comparison of independent or dependent correlations can both involve omnibus tests of equality of more than two correlations.

The study reported in this paper focuses on statistical tests for the comparison of two dependent correlations with overlapping variables. Cases in which the comparisons of concern involve more than two correlations will be discussed in Section 6. A large number of statistical tests have been proposed for this purpose, 10 of which are presented in Section 2. In principle, all of them are interchangeable although some technicalities of their computation may affect their performance in terms of accuracy (i.e., Type I error rates), power, robustness to violations of distributional assumptions, or even the feasibility of their computation given the values of the sample statistics on hand. Earlier research on the comparative performance of these tests has been fragmentary and somewhat limited in scope. Some studies have compared a subset of these tests according to their accuracy and power with data drawn from trivariate normal distributions, although none of the studies has considered a broad range of values for the population correlations and a broad range of sample sizes (see, for example, Boyer et al., 1983; Cohen, 1989; Dunn & Clark, 1971; Hittner et al., 2003; May & Hittner, 1997a; Neill & Dunn, 1975; Wilcox & Tian, 2008). In addition, some studies have reported fractional results on robustness under violation of trivariate normality, always using identical non‐normal (univariate) distributions for X 1, X 2, and Y. For instance, Boyer et al. (1983) used lognormal distributions, Cohen (1989) used chi‐square distributions with two and four degrees of freedom, and May and Hittner (1997a); see also (Hittner et al., 2003; Wilcox & Tian, 2008) used uniform and exponential distributions. Results reported in the studies just mentioned have shown a relatively small deterioration in the accuracy and power of the tests under violation of normality, but the generalizability of this conclusion to the empirically plausible case of arbitrarily different non‐normal univariate distributions for X 1, X 2, and Y is uncertain. A further aspect that has not previously been studied is the comparative applicability of the tests, all of which involve computations that may render the test statistic undefined for certain combinations of the values of sample statistics.

The motivation for conducting a comprehensive comparative study of accuracy, power, robustness, and applicability of alternative tests for equality of two dependent correlations is twofold. A first component comes from the foreseeable replacement of current malpractice (i.e., conducting isolated tests of significance of each of the implied correlations and comparing their outcomes) with the use of proper tests of significance of the difference between dependent correlations. If this is about to come, the choice of statistical test for this purpose must be based on evidence of their comparative performance. The second component comes from the availability of computer programs and web applications that carry out and report the outcomes of a number of alternative tests with no accompanying information regarding which test is more dependable than which other, thus leaving users uncertain about which result should be pulled out from the list (see, for example, Diedenhofen & Musch, 2015; Silver et al., 2006; Silver & Merino‐Soto, 2016). To investigate the usage of the various tests considered in this paper, on 22 December 2023, we used Google Scholar to find papers published in 2023 in journals containing ‘psychology’ or ‘psychological’ in their title and that cited cocor (Diedenhofen & Musch, 2015), an R package (https://cran.r‐project.org/web/packages/cocor) and web application (http://comparingcorrelations.org) to conduct tests of equality of dependent correlations with overlapping variables among other comparisons of correlations. The search retrieved 52 papers, but nearly half of them reported using cocor to conduct tests of equality of independent correlations or equality of dependent correlations with non‐overlapping variables, whereas others cited the target paper for reasons not related to actual usage of the tests. Twenty‐seven papers remained that reported using cocor for comparing dependent correlations with overlapping variables. Table A1 in Appendix A identifies each of these papers and indicates which test was reportedly used. Ten of the papers did not even report which test had been used, one other reported that all tests gave the same results, and the remaining 16 papers varied in their choice of test.

The plan of this paper is as follows. Section 2 describes the 10 tests whose performance will be compared. Section 3 describes the criteria for comparison, which cover the four aspects mentioned above, namely, Type I error rates, power, robustness, and empirical applicability. Section 4 describes the simulation method used to achieve these goals, including a description of the extensive set of scenarios (population correlations, sample sizes, and univariate distributions) under which the performance of the tests is compared. Section 5 reports the results in terms of each of the comparison criteria. Section 6 presents some empirical examples with data from recent papers in which dependent correlations were conventionally compared (i.e., via the p‐value of their respective tests of significance against zero). Finally, Section 7 summarizes and discusses the results, also offering practical recommendations regarding the optimal choice of statistical test.

GENERAL NOTATION AND TEST STATISTICS

Let X 1 and X 2 be the two variables whose individual correlations with variable Y are to be compared. We initially assume that all variables have standard normal distributions and a trivariate normal distribution with covariance and correlation matrixΡ=1ρX1X2ρX1YρX1X21ρX2YρX1YρX2Y1.

We will simplify notation via ρ1Y :=ρX1Y, ρ2Y :=ρX2Y, and ρ12 :=ρX1X2. This section describes briefly a number of methods that have been proposed to test the null hypothesis H0: ρ ~1Y ~ = ρ ~2Y ~ using n triplets of observations from which sample correlations r ~1Y ~, r ~2Y ~, and r 12 are computed. We name the tests as in Diedenhofen and Musch (2015), where a full presentation is given. Computation of all test statistics involves some square root whose radicand is determined by sample correlations and, thus, it is not guaranteed to be positive‐valued. Whenever the radicand turns up negative for the sample of concern, the test statistic is undefined.

Pearson–Filon

This statistic (Pearson & Filon, 1898) has a standard normal distribution and is computed as(1)zPF=r1Y−r2Yn1−r1Y22+1−r2Y22−2r121−r1Y2−r2Y2+r1Yr2Y1−r1Y2−r2Y2−r122.

Olkin

This statistic (Olkin, 1967; with the correct formula in Hendrickson & Collins, 1970) has a standard normal distribution and is computed as(2)zO=r1Y−r2Yn1−r1Y22+1−r2Y22−2r123−2r12−r1Yr2Y1−r1Y2−r2Y2−r122.

Hotelling

This statistic (Hotelling, 1940) has a Student's t distribution with n − 3 degrees of freedom and is computed as(3)tH=r1Y−r2Yn−31+r122Rwith(4)R=1−r1Y2−r2Y2−r122+2r1Yr2Yr12.

Standard Williams

This statistic (Williams, 1959) has a Student's t distribution with n − 3 degrees of freedom and is computed as(5)tW=r1Y−r2Yn−31+r122R+r¯2n−3n−11−r123with R from Equation (4) and(6)r¯=r1Y+r2Y2.

Hendrickson–Stanley–Hills

This statistic (Hendrickson et al., 1970) has a Student's t distribution with n − 3 degrees of freedom and is computed as(7)tHSH=r1Y−r2Yn−31+r122R+r1Y−r2Y21−r1234n−1with R from Equation (4).

Dunn–Clark

This statistic (Dunn & Clark, 1969) has a standard normal distribution and is computed as(8)zDC=Z1Y−Z2Yn−32−2cDCwith Z ~1Y ~ and Z ~2Y ~ given by Fisher's transformation by whichZxy=12ln1+rxy1−rxyand(9)cDC=r121−r1Y2−r2Y2−12r1Yr2Y1−r1Y2−r2Y2−r1221−r1Y21−r2Y2.

Steiger

This statistic (Steiger, 1980) has a standard normal distribution and is computed as(10)zS=Z1Y−Z2Yn−32−2cSwith Z ~1Y ~ and Z ~2Y ~ given by Fisher's transformation and(11)cS=r121−2r¯2−12r¯21−2r¯2−r1221−r¯22,with r¯ from Equation (6).

Hittner–May–Silver

This statistic (Hittner et al., 2003) has a standard normal distribution and is computed as(12)zHMS=Z1Y−Z2Yn−32−2cHMSwith Z ~1Y ~ and Z ~2Y ~ given by Fisher's transformation and(13)cHMS=r121−2r¯z2−12r¯z21−2r¯z2−r1221−r¯z22,in whichr¯z=exp2Z¯−1exp2Z¯+1andZ¯=Z1Y+Z2Y2.

Meng–Rosenthal–Rubin

This statistic (Meng et al., 1992) has a standard normal distribution and is computed as(14)zMRR=Z1Y−Z2Yn−321−r12hwith Z ~1Y ~ and Z ~2Y ~ given by Fisher's transformation and(15)h=1−fr2¯1−r2¯withr2¯=r1Y2+r2Y22andf=1−r1221−r2¯

Meng et al. (1992) stated that f = 1 should be used whenever the computation of f using sample statistics results in f > 1, but the justification for this replacement is unclear. At first glance, the only apparent requirement is that the radicand in Equation (14) is positive‐valued. This implies h > 0 or, by Equation (15), f<1/r2¯, which can sometimes result in f > 1. This study will use this test without replacing any f > 1 with f = 1, with the consequence that the test statistic is undefined when sample statistics result in f≥1/r2¯.

Zou

This test (Zou, 2007) is based on a confidence interval. The lower (L) and upper (U) limits of a 100 (1 − α)% confidence interval for ρ ~1Y ~ − ρ ~2Y ~ are(16)L=r1Y−r2Y−r1Y−l12+u2−r2Y2−2cr1Y−l1u2−r2Y, (17)U=r1Y−r2Y+u1−r1Y2+r2Y−l22−2cZu1−r1Yr2Y−l2,with(18)li=exp2li*−1exp2li*+1,i∈1,2, (19)ui=exp2ui*−1exp2ui*+1,i∈1,2, (20)li*=ZiY−z1−α/2n−3,i∈1,2, (21)ui*=ZiY+z1−α/2n−3,i∈1,2,where Z ~ iY ~ is given by Fisher's transformation of r ~ iY ~, z 1−α/2 is the 100(1 − α/2)% standard normal quantile, and(22)cZ=r12−12r1Yr2Y1−r1Y2−r2Y2−r122+r1231−r1Y21−r2Y2.

Given that L < U by construction, application of this method rejects the null hypothesis of equality of correlations whenever L > 0 or U < 0.

CRITERIA FOR COMPARISON

Before presenting the criteria for comparison of alternative tests for H0: ρ ~1Y ~ = ρ ~2Y ~, it should be noted that the identity stated in the null can only hold for a restricted range of values determined by the population correlation ρ 12. The reason is that a non‐degenerate trivariate normal distribution can only exist if Ρ=1−ρ1Y2−ρ2Y2−ρ122+2ρ1Yρ2Yρ12>0. Algebraic manipulation of this inequality indicates that the possible range for ρ ~2Y ~ given arbitrary values for ρ 12 and ρ ~1Y ~ is(23)ρ12ρ1Y−1−ρ1221−ρ1Y2<ρ2Y<ρ12ρ1Y+1−ρ1221−ρ1Y2;note that the range is not empty for any pair of values for ρ 12 and ρ ~1Y ~. The shaded areas in Figure 1 show cross‐sectional slices of the region of feasibility of ρ 1Y ~ and ρ 2Y ~ at selected values for ρ 12. The intersection of these regions with the diagonal identity line describes the range of values for which a trivariate normal distribution may exist where H0 is true. A comparison of the accuracy of tests is thus restricted to feasible true nulls (i.e., ρ ~1Y ~ = ρ ~2Y ~) for each value of ρ 12, whereas a comparison of their power is restricted to feasible false nulls (i.e., ρ ~1Y ~ ≠ ρ ~2Y ~) for each value of ρ 12.

The first criterion for comparison is the region in the three‐dimensional space of sample correlations r ~1Y ~, r ~2Y ~, and r 12 for which a test statistic is real‐valued. Although the region of existence of a trivariate normal distribution in the analogous three‐dimensional space of population correlations is well defined and illustrated in Figure 1, it is not clear that all test statistics can accommodate any sample correlations in this region. Consider the Hotelling test statistic in Equation (3), where the radicand (Equation (4)) simply sets on sample correlations the same condition set on population correlations for the existence of a trivariate normal distribution, thus ensuring coverage of the sample space of trivariate correlations. In contrast, the standard Williams statistic in Equation (5) and the Hendrickson–Stanley–Hills statistic in Equation (7) both incorporate into the radicand additive and positive‐valued terms that broaden the applicability of these statistics. It is not immediately obvious how the relatively more complex radicands in the remaining test statistics affect their corresponding regions of applicability.

The second and third criteria for comparison involve the accuracy of the tests (i.e., the match between nominal and actual Type I error rates) and their power, both assessed when observations are drawn from trivariate normal distributions and from non‐normal distributions. Comparisons involving these criteria were carried out via simulations described in the next section.

SIMULATION METHOD

Empirical Type I error rates were determined for each test for values of ρ 12 between −.9 and .9 in steps of .15 and, in each case, within the admissible range of values for ρ ~1Y ~ = ρ ~2Y ~ in steps of .02 (i.e., along the diagonal identity line in the panels of Figure 1 within the region indicated by the shaded area). For each combination of ρ 12 and ρ ~1Y ~ = ρ ~2Y ~, we conducted N = 500,000 replications each consisting of n triplets of standard normal observations X 1, X 2, and Y drawn from the corresponding trivariate normal distribution. In separate simulation runs, sample sizes n were 20, 50, 100, and 200. The 10 test statistics were computed for each sample in each replication, and the empirical Type I error rate for each test was computed as the proportion of replications (out of N) in which the null was rejected in a two‐sided test at α = .05, keeping separate counts of rejection in each tail.

Power was determined for each test in a similar manner with 200,000 replications per condition and for the same set of values of ρ 12, drawing triplets from trivariate normal distributions with ρ ~1Y ~ ≠ ρ ~2Y ~ in pairings described by the lattice points in Figure 2. Lattice points are located along lines perpendicular to the identity line and with respect to reference points on the diagonal ranging from ρ ~1Y ~ = ρ ~2Y ~ = −.96 to ρ ~1Y ~ = ρ ~2Y ~ = .96 in steps of .04. Successive lattice points along off‐diagonal directions are created by progressively subtracting .04 from the reference value of ρ ~1Y ~ and adding .04 to the reference value of ρ ~2Y ~. Power at a given lattice point was determined only if a trivariate normal distribution exists under the applicable value of ρ 12, for example, for the lattice points within the region of existence illustrated in Figure 2 for ρ 12 = −.60 (red ellipse) or for ρ 12 = .90 (blue ellipse). No lattice points were defined below the diagonal in Figure 2 because of the symmetry of the tests with respect to the sign of the difference between ρ ~1Y ~ and ρ ~2Y ~. The off‐diagonal lines on which lattice points are placed allow assessment of whether power varies as a simple function of the absolute value of the difference ρ ~1Y ~ − ρ ~2Y ~ (which is constant for all lattice points falling on the same imaginary line parallel to the identity line) or differs also with the point on the diagonal relative to which |ρ ~1Y ~ − ρ ~2Y ~| is placed, that is, with the location of the intersection of the off‐diagonal line and the identity line. The location of this intersection is the point at which both ρ ~1Y ~ and ρ ~2Y ~ equal their average (ρ ~1Y ~ + ρ ~2Y ~)/2. In other words, different off‐diagonal lines define lattice points of increasing value of |ρ ~1Y ~ − ρ ~2Y ~| at fixed values of (ρ ~1Y ~ + ρ ~2Y ~)/2.

Triplets of observations drawn from non‐normal distributions were used to assess accuracy and power under violation of the assumption of trivariate normality. Many forms of non‐normality have been described in psychological data at both the univariate and multivariate levels (Cain et al., 2017). Hence, a thorough assessment of robustness to arbitrary violations of trivariate normality is impractical. Our goal here is thus limited to describing results for a number of plausible cases that nevertheless shed light on the robustness (or lack thereof) of the tests under analysis. This was done in three different ways. For the first, data from beta distributions with a = 1 and b = 1 (i.e., uniform) or a = 2 and b = 5 (positively skewed) were generated by first drawing triplets of observations in Z 1, Z 2, and Z ~ y ~ from a trivariate standard normal distribution with correlation matrix Ρ. Sample values were then transformed via X1=F−1ΦZ1a1b1, X2=F−1ΦZ2a2b2, and Y=F−1ΦZyayby, where Φ is the standard normal distribution function and F ^−1^ is the beta quantile function with parameters a and b. Note that beta parameters are defined separately for each variable, thus allowing any combination of forms for the univariate distributions of X 1, X 2, and Y. For the second way, triplets Z 1, Z 2, and Z ~ y ~ generated as described above were transformed via X 1 = exp(Z 1), X 2 = exp(Z 2) and Y = exp(Z ~ y ~), which produces data with univariate Lognormal(0, 1) distributions. Finally, normal mixtures were generated whose first component (with probability .9) was the trivariate standard normal distribution with correlation matrix Ρ and whose second component (with probability .1) was a trivariate normal distribution with the same correlation matrix but larger variances (2, 4, and 10) for all variables.

Generation of non‐normal variables via transformation (or mixture) of normal variables ensured that the former matched the target correlations ρ 12, ρ 1Y , and ρ 2Y ~ included in analogous simulations with normally distributed data. Specifically, for our mixtures of normal variables with the same correlation matrix and differing only in a common variance, the correlation matrix of the mixture is that of the original variables and, then, the components were generated with the applicable target correlations. When standard normal variables X and Y with correlation ρ ~ xy ~ are exponentially transformed into lognormal variables X and Y, the correlation between the latter is given by(24)ρxy~=expρxy−1exp1−1(De Veaux, 1976). Then the correlation matrix for the originating normal variables was set by inverting this relation so that the resulting lognormal variables had the required target correlations. This, in turn, reduces the breadth of target correlations that can be explored because the lowest possible correlation between lognormal variables is –exp(−1) ≈ −.3679 (De Veaux, 1976). Finally, beta variables generated via the inverse transform method do not generally keep the correlation of the originating normal variables and there is no closed‐form expression to compensate for this fact in the correlation matrix of the generating normal variables. Yet, Appendix B shows that the original correlations are almost exactly preserved with the beta parameters of our choice. The skewness and kurtosis of each of the resulting non‐normal univariate distributions are listed in Table 1; note that heteroscedasticity generally holds for bivariate distributions with non‐normal marginals.

The study includes cases in which all variables had the same non‐normal univariate distribution and a number of other cases in which form of distribution varied across variables while keeping the same distribution for X 1 and X 2 to ensure that the null hypothesis remains true or false as needed. In all cases, data were generated under all of the applicable conditions described earlier for similar analyses under trivariate normality, except that we used N = 250,000 replications in studies of accuracy and N = 100,000 replications in studies of power.

Because of the extensive and time‐consuming computations required for completion of the study, Fortran code was written to carry out all the simulations.

RESULTS

Region of empirical applicability

Obtaining the region of empirical applicability amounts to finding out which triplets (r ~1Y ~, r ~2Y ~, r 12) on the cube [−1, 1] × [−1, 1] × [−1, 1] result in a positive‐valued radicand on computation of each test statistic. A minimum requirement is that the statistic is real‐valued when sample correlations take any values compatible with a trivariate normal distribution, and we have already discussed that the Hotelling statistic in Equation (3) has this exact characteristic. The standard Williams and Hendrickson–Stanley–Hills statistics (both with n = 100; see the denominators in Equations (5) and (7)) are similar in this respect and their empirical region of applicability does not differ much from the theoretical region of existence of trivariate normal distributions (see the second and third columns from the left in Figure 3).

At the other end, the Hittner–May–Silver test is applicable for all triplets on the cube and it is thus not displayed in Figure 3. The remaining statistics vary in their region of applicability, as seen in Figure 3. All tests are applicable with any triplet (r ~1Y ~, r ~2Y ~, r 12) compatible with a trivariate normal distribution, except the Meng–Rosenthal–Rubin test without the imposition f ≤ 1 discussed in Section 2.9 above. As seen in the rightmost column of Figure 3, not replacing an f≥1/r2¯ with f = 1 prevents computation of the statistic in some areas of the region of existence of trivariate normal distributions, although only when the sample value of r 12 is negative. The consequences of setting versus not setting a bound on f will be described later.

Accuracy with trivariate normal observations

For a preliminary illustration, Figure 4 shows the empirical sampling distribution of each of the nine test statistics in the simulation condition with (ρ ~1Y ~, ρ ~2Y ~, ρ 12) = (.80, .80, .75) and n = 50 (recall that the Zou test does not come in the form of a test statistic whose sampling distribution can be inspected). The continuous curve in each panel is the applicable asymptotic distribution (i.e., standard normal or Student's t with n − 3 degrees of freedom), to which empirical distributions generally match except for the Hittner–May–Silver test. Empirical Type I error rates at the nominal size−.05 test are also shown for each tail in the panels of Figure 4.

The similarity between empirical and nominal rejection rates at each tail indicates an acceptable accuracy for most tests, although some of them seem a little conservative (e.g., Pearson–Filon and Olkin) while others seem a little liberal (e.g., Hotelling and Hendrickson–Stanley–Hills), leaving aside the unacceptably liberal Hittner–May–Silver test. Empirical sampling distributions are also approximately symmetric in Figure 4, as was the case for the all other simulation conditions. For this reason, subsequent graphical results display Type I error rates aggregated over tails.

Figure 5 displays summary results (at n = 200) for the three tests whose accuracy proved unacceptable but whose Type I error rates turned out to be essentially invariant with sample size n. The Hittner–May–Silver test (Figure 5a) is accurate only when ρ ~1Y ~ = ρ ~2Y ~ is very near zero, something that is generally the case when ρ 12 is negative (see the top row in Figure 1). Yet, as the positive value of ρ 12 increases, the test becomes increasingly liberal as the common value of ρ ~1Y ~ and ρ ~2Y ~ (under the null) moves away from zero. On the other hand, the Hotelling and Hendrickson–Stanley–Hills tests (Figure 5b) are very (and identically) inaccurate when ρ 12 is negative unless ρ ~1Y ~ = ρ ~2Y ~ is virtually zero, whereas their accuracy increases across the range of values that ρ ~1Y ~ = ρ ~2Y ~ can actually take as the positive value of ρ 12 increases.

Results for the seven other tests are displayed in Figure 6 in another format, where the vertical axis in each panel also spans a narrower range (from .03 to .07) than that in Figure 5. The Pearson–Filon and Olkin tests (black curves in Figure 6) have identical (in)accuracy that varies with sample size n also identically. Although these two tests are much less inaccurate than those for which results were displayed in Figure 5, they are both outperformed by the five other tests displayed in Figure 6 (coloured curves), whose accuracies are similar at large sample sizes (bottom row in Figure 6). Yet, with small sample sizes (top row in Figure 6) and positive values of ρ 12, differences can be observed that allow the tests to be ranked in increasing order of accuracy down the list in the legend to Figure 6. At one extreme, the Zou test (blue curves) displays a wavy pattern that departs the most from the .05 line when ρ 12 = .9; at the other extreme, the standard Williams test (light green curves) stays closest overall to the .05 line. Full graphical results for all tests at all sample sizes are presented in Section A of the Appendix S1.

Based on these results on accuracy, only five tests are advisable (standard Williams, Dunn–Clark, Steiger, Meng–Rosenthal–Rubin, and Zou) although the latter is inferior with small samples and large ρ 12. Yet, it is worth looking at the region within which sample values of r ~1Y ~ and r ~2Y ~ result in non‐rejection of the null by each of these five tests, a region that is in turn confined within the region where each test can actually be used by the requirement of a positive radicand. Figure 7 shows these regions (green areas) for the tests of concern at selected sample values of r 12 (rows) when n = 50.

All tests appear virtually identical at any given r 12 within the region of feasible trivariate distributions (ellipse in each panel). It is also worth pointing out that the Dunn–Clark test (second column from the left in Figure 7) displays an anomaly at the top‐left and bottom‐right corners of the panel (i.e., when r ~1Y ~ and r ~2Y ~ are both large in absolute value but have opposite signs), an anomaly that spreads out to a larger region as r 12 increases and that progressively disappears as the sample size n increases. Admittedly, the applicable combinations of r 12, r ~1Y ~, and r ~2Y ~ in this area are empirically impossible, but structural non‐rejection of the null in such cases defies logic.

This (inconsequential) anomaly of the Dunn–Clark test raises the question of whether some anomaly may also obtain when the Meng–Rosenthal–Rubin test is applied under the admonition to use f = 1 in Equation (15) whenever computation of f using sample statistics renders a value greater than unity. This replacement has two consequences. One is that it turns an otherwise undefined statistic into a real‐valued one whose magnitude might result in inadequate rejections or non‐rejections; the other is that it alters the value of a well‐defined statistic when 1<f<1/r2¯.

Results for this (original) version of Meng–Rosenthal–Rubin test are displayed in the fifth column from the right of Figure 7, where it is immediately obvious that the test becomes computable at all points, thus (unnecessarily) extending its applicability. A second feature not apparent in Figure 7 is that the region of non‐rejection is minimally larger for the original form within the region of existence of trivariate normal distributions. This is more apparent with small samples and negative ρ 12 and it is caused by making f = 1 when 1<f<1/r2¯ and, thus, even when the test statistic is well defined. The desirability of an artificially larger non‐rejection area may be contentious, but we see little reason for the admonition to define f as the smaller of unity and its computed value via Equation (15).

In sum, as far as accuracy and applicability are concerned, the list of recommendable tests in order of preference are standard Williams, Dunn–Clark, Steiger, Meng–Rosenthal–Rubin (without setting a bound for f at unity), and Zou (see accuracies for these tests in Figure 6).

Power with trivariate normal observations

Power results are reported here for the five tests just mentioned, but full results for all tests are presented in Section B of the Appendix S1. For a quick glimpse, Figure 8 shows the power of each test (rows) as a function of the absolute difference |ρ ~1Y ~ − ρ ~2Y ~| at selected values of ρ 12 (columns) along each of the applicable off‐diagonal lines in Figure 2 for n = 50 (red curves) and n = 200 (blue curves). The most apparent feature is that all tests appear identically powered, as there are no discernible differences across rows. Two other unsurprising characteristics are that power increases with sample size (blue versus red curves in the panels of Figure 8) and with the magnitude of the absolute difference between ρ ~1Y ~ and ρ ~2Y ~ (horizontal axis in the panels of Figure 8). Two further characteristics are also apparent. One is that the power to detect an absolute difference of some magnitude between ρ ~1Y ~ and ρ ~2Y ~ (i.e., at any arbitrary point along the horizontal axis) increases with increasing ρ 12 (i.e., rightward each row in Figure 8). The other is that the power curves for different values of (ρ ~1Y ~ + ρ ~2Y ~)/2 (set of identically coloured curves in each panel) do not superimpose, the less so as ρ 12 approaches zero from either side. Altogether, this means that power is not a simple function of |ρ ~1Y ~ − ρ ~2Y ~| but that it also depends heavily on (ρ ~1Y ~ + ρ ~2Y ~)/2 and ρ 12.

This dependence is best appreciated in perspective plots of the power surface defined on the two‐dimensional space of Figure 2 at different values of ρ 12. Figure 9 displays these plots for the standard Williams test; plots for other tests were virtually identical. Note that the identity line in Figure 2 runs slightly tilted from left to right in each panel of Figure 9 and that the off‐diagonal lines in Figure 2, which represent locations with increasing value of |ρ ~1Y ~ − ρ ~2Y ~| at a fixed value for (ρ ~1Y ~ + ρ ~2Y ~)/2, run perpendicular to the identity line in the panels of Figure 9, whereas lines parallel to the identity line describe power at locations of constant |ρ ~1Y ~ − ρ ~2Y ~| for different values for (ρ ~1Y ~ + ρ ~2Y ~)/2. The U‐shaped form of the latter lines within the region where power increases from 0 to 1 indicates that, for any fixed value of |ρ ~1Y ~ − ρ ~2Y ~|, power increases as (ρ ~1Y ~ + ρ ~2Y ~)/2 moves away from 0.

Robustness

Non‐normality deteriorated accuracy similarly for all tests, on top of the fact that tests already differed in accuracy under trivariate normality (see Figures 5 and 6). The magnitude and form of deterioration differed greatly across the cases under study. To illustrate the mildest deterioration, Figure 10 shows aggregated Type I error rates for the standard Williams test when all variables had uniform distributions (Figure 10a), Beta(2, 5) distributions (Figure 10b), or mixed uniform and Beta(2, 5) distributions (Figure 10c,d). In these cases, the deterioration was largely invariant with sample size (compare the red and blue curves in each panel of Figure 10) and was milder when the univariate distribution of Y differed from the common univariate distribution of X 1 and X 2 (Figure 10c,d) than it was when the three variables had the same univariate distribution (Figure 10a,b). In comparison to the corresponding plots in Figure 6 for accuracy of the same test (light green curves) under trivariate normality, uniform distributions for all variables (Figure 10a) do not affect accuracy when ρ ~1Y ~ = ρ ~2Y ~ are very close to zero but they make the test increasingly liberal as ρ ~1Y ~ = ρ ~2Y ~ move away from zero within their possible range of values (which broadens as ρ 12 increases). Beta(2, 5) distributions for all variables (Figure 10b) also do not affect the accuracy of the test when ρ ~1Y ~ = ρ ~2Y ~ are virtually zero but, in contrast, they make the test increasingly more conservative (liberal) as ρ ~1Y ~ = ρ ~2Y ~ become increasingly negative (positive) within their possible range of values. Mixed forms of distribution across variables (Figure 10c,d) clearly reduced the deterioration, a result that is likely limited to the particular distributions considered here. Non‐normality also resulted in symmetric sampling distributions of test statistics that were also centred on zero (i.e., similar to what was shown in Figure 4 under normality) and, thus, Type I error rates in each of the tails were virtually identical. The remaining tests with acceptable accuracy under normality (see Figure 6) displayed a deterioration under non‐normality that was similar to that shown in Figure 10 for the standard Williams test.

The lack of generality of the robustness results just discussed is corroborated by the results obtained for the remaining forms of distribution listed in Table 1, which are also illustrated in Figure 11 in a different form and with a broader range of Type I error rates (between 0 and .4, compared to between 0 and .1 in Figure 10) for the standard Williams test. The lognormal distribution (blue curves in Figure 11) resulted in the most dramatic and nonlinear deterioration. On the other hand, normal mixtures (green, red, and pink curves in Figure 11) resulted in a deterioration whose magnitude was constant across the board but increased with the variance difference between the two components of the mixture. In addition, the magnitude of deterioration increased with sample size (top versus bottom rows in Figure 11). Full results for all tests are presented in Sections C, E, and G–J of the Appendix S1.

Given the preceding results, it comes as no surprise that power curves for the cases considered in Figure 10 had the same features displayed in Figures 8 and 9 for trivariate normal data. Power curves for the cases considered in Figure 11 were similarly sigmoidal, although they increased from a lower asymptote at the inappropriate Type I error rates depicted in Figure 11. Graphical presentation of these results is deferred to Sections D and F of the Appendix S1.

EMPIRICAL EXAMPLES

Correlational studies are the natural context of application of the tests considered in this paper, but tests are also needed in other types of study in which dependent correlations are compared for whatever reason (e.g., Blake & Palmisano, 2021; Ćoso et al., 2023). A research area in which these tests are frequently needed is the search for cognitive or behavioural correlates of the 2D–4D digit ratio (i.e., the ratio of the lengths of the index and ring fingers), and this is the area from which the coming examples were drawn. Another area of application is research on correlates of state versus trait anxiety (e.g., Guo et al., 2022; Menculini et al., 2023), but no examples will be given from this context. It should be noted that the following examples do not come from papers in which the core of the analysis was a comparison of correlations, and the only criterion guiding the choice of papers was availability of the value of r 12 in the paper itself, a correlation that is rarely reported when the nominal correlations of concern are only r ~1Y ~ and r ~2Y ~.

Since the 2D–4D digit ratio can be measured on each of the hands, a common question of interest is whether the target variable Y correlates equally with the right‐hand ratio (X 1) and the left‐hand ratio (X 2). For instance, Brosnan et al. (2011) reported that the correlation between right‐hand digit ratio and Java grade in a sample of 73 first‐year students taking a course in Java programming reached statistical significance (r ~1Y ~ = −.22; p = .03) but not so for the left‐hand ratio (r ~2Y ~ = −.13; n.s.), whereas the left‐ and right‐hand ratios were significantly correlated (r 12 = .46; p < .01). Use of the standard Williams test yields t W = −0.7420 with p = .461, so that there is no basis for supporting the notion that digit ratios in the right versus the left hand are differently correlated to Java grades, despite the anecdotal result that one of the correlations is significantly different from zero and the other is not.

In another paper from the same research field, Uzun and Tok (2023) reported correlational analyses of right‐hand and left‐hand digit ratios (variables X 1 and X 2) with attention gathering skill (variable Y) in 112 children aged 60–72 months. The correlation was significant for the right hand (r ~1Y ~ = .209; p < .05) but not for the left hand (r ~2Y ~ = .086; n.s.), while the left‐ and right‐hand ratios correlated r 12 = .114 (which is not significantly different from zero). Use of the standard Williams test yields t W = 0.9846 with p = .327, so that there is no basis for supporting the notion that digit ratios in the right versus the left hand are differently correlated to attention‐gathering skill, despite the anecdotal result that only one of the correlations is significantly different from zero (Incidentally, table 7 in Uzun and Tok (2023) gives r ~2Y ~ = −.086; if this were the actual value, the standard Williams test would yield t W = 2.3795 with p = .019, rejecting the null that attention‐gathering skill holds the same relation to the right‐ and left‐hand digit ratio).

Also of interest in this area is the search for associations between digit ratio in one of the hands and personality factors, cognitive performance in various tasks, or behavioural or physiological measures. In such cases, digit ratio plays the role of variable Y whereas the other measures define a set of J variables X ~ j ~ (with 1 ≤ j ≤ J and J > 2). At first sight, this situation may seem to imply the null H0: ρ ~1Y ~ = ρ 2Y ~ = … = ρ ~ JY , for which statistical tests have been developed (see, for example, Choi, 1977; Cohen, 1989; Olkin & Finn, 1990). Yet, the context for such a null typically implies that the X ~ j ~ are measures of the same variable at different time points in a longitudinal study and the goal is to check out whether the correlation between X and Y across the J occasions remains stable. In the context of our examples here, there is no theoretical expectation that digit ratio should hold the same relation with, say, all dimensions of personality. The actual research question is instead whether digit ratio is more strongly associated with some personality dimension than it is with others (see, for example, Fink et al., 2004) and, thus, it requires pairwise analyses with multiple tests of the simple null H0: ρ ~ jY ~ = ρ ~ kY ~ for all j ≠ k. Although each of these multiple tests leads to an individual claim (and, hence, there is no risk of inflation of Type I error rates that might demand alpha adjustments; see García‐Pérez, 2023; Rubin, 2021, 2024), a researcher may still want to set the alpha level different from the usual .05 for other reasons (Maier & Lakens, 2022).

The handling of comparisons of correlations in a case like this one is illustrated here with data from Del Giudice and Angeleri (2016), a study in which 135 boys and 150 girls contributed measures of right‐hand digit ratio (variable Y) and three measures of attachment avoidance (variable X 1), preoccupation (variable X 2), and felt security (variable X 3). Intercorrelations among these variables were reported in their Table 1 separately for the samples of boys and girls. Results of our pairwise analyses of equality of correlations with the standard Williams test in the sample of girls are presented in Table 2, with the outcome that equality of correlation with digit ratio is not rejected for any pair of attachment styles. It should be stressed that an analogous and independent set of tests for the sample of boys could be conducted but, depending on the goal of the analysis, it might be more fitting to use the ANOVA‐like strategy developed by Bilker et al. (2004) for that purpose.

DISCUSSION

Summary of results

Our comparative analysis of the performance of the 10 tests can be summarized as follows. When variables have a trivariate normal all tests are applicable under any possible combination of the values that sample correlations may take. This holds for the Meng–Rosenthal–Rubin test only when administered with the original admonition to replace any occurrence of f > 1 in Equation (15) with f = 1.Type I error rates stay at their nominal level across all possible combinations of population correlations for only five of the tests (in order of preference, standard Williams, Dunn–Clark, Steiger, Meng–Rosenthal–Rubin, and Zou), whose Type I error rates are also barely affected by sample size.power curves are similar for the five tests just mentioned and, across the board, power increases with sample size and with effect size defined as the absolute value of the difference between ρ ~1Y ~ and ρ ~2Y ~. Two additional factors that modulate power are the average of the two population correlations ρ ~1Y ~ and ρ ~2Y ~ under comparison (with power increasing faster with effect size as this average correlation moves away from zero) and the value of the population correlation ρ 12 (with power also increasing faster with effect size as ρ 12 increases). The former feature was anecdotally noted by May and Hittner (1997b). The presence of these modulating factors raises a warning flag on the results of power analyses that only consider effect size defined as a difference in correlation (see Faul et al., 2009).When variables are instead non‐normally distributed in the limited set of forms explored here, accuracy of the five tests deteriorates to varying extents from negligibly in some cases (see Figure 10) to unacceptably in others (see Figure 11). Further comments on non‐normal distributions are deferred to the next subsection.

The results just summarized modulate those reported in previous papers that addressed the analysis for a smaller number of test statistics, under a narrower set of population correlations, with smaller samples, and with fewer replications to estimate Type I error rates and power. By exploring combinations of population correlations more thoroughly than was done in previous studies and by using substantially more replications to estimate Type I error rates and power, this study has identified inadequate performance of some tests whose deficiencies only show within certain ranges of the parameter space (see, for example, accuracy results in Figure 5). Analogously, a presumed inaccuracy of the standard Williams test with trivariate normal observations when ρ ~1Y ~ = ρ ~2Y ~ = .7 and ρ 12 = .1 for n ∈ {20, 50, 100, 300} and Type I error rates estimated with 2000 replications (see, for example, May & Hittner, 1997a; see also Hittner et al., 2003) was not replicated in our results, which do not show any sign of a singularity at or near this region (see the light green curves in Figure 6).

Non‐normal distributions

The non‐normal distributions included in our study offer a suitable account of the diverse performance of the tests under plausible scenarios, but no study can include all of the non‐normal distributions that can be found in empirical data (see Cain et al., 2017). For instance, some variables are subject to order restrictions such that X ~ i ~ < Y ~ i ~ for every paired observation i in the sample (e.g., years of marriage and age), a feature that incorporates irrelevant structural components into the magnitude of sample correlations (see García‐Pérez & Núñez‐Antón, 2013). Our results show that non‐normality per se is not necessarily a threat to the accuracy of tests for equality of dependent correlations with overlapping variables (compare Figures 10 and 11), although it seems impossible to determine in advance whether the tests will be robust to the particular (and unknown) form of non‐normality of the distributions of any data on hand.

One might think of checking for form of distribution to decide whether any of these tests is safe to use under the circumstances, perhaps leading to preliminary simulations of accuracy under the particular forms of distribution that have been identified. It should nevertheless be noted that such an identification is impossible in strict sense, as one can only test (and then simply reject or fail to reject) a null hypothesis about some particular form of distribution. For instance, with typical small samples, normality may often be incorrectly not rejected when the data actually come from contaminated normal distributions such as those considered here (see Table 1). And this preliminary testing for normality also brings its own inflation of Type I error rates due to the conditional approach (see García‐Pérez, 2012).

One could also think of using alternative tests that are robust to violations of normality, none of which have been explored in this paper for reasons discussed next. Wilcox and Tian (2008) considered six such alternative tests in a comparison of accuracy and power with those of the standard Williams test under normality and non‐normality (uniform or exponential distributions for all variables). They discarded four of the tests (designated B1, B2, B3, and B4) for various reasons, and their simulations (which were somewhat limited in scope) showed that the two other tests (designated D1 and D2) slightly outperformed the standard Williams test when data were drawn from non‐normal distributions, but only by the criterion that Type I error rates do not exceed the nominal α level. Yet, neither of these two tests fared well in comparison to the standard Williams test when data were instead drawn from normal distributions. Given the difficulty of ascertaining whether the data on hand come or do not come from normal distributions, using these alternative tests cannot be recommended in general. Alternative tests do not seem to be available that are robust to violations of normality and remain accurate under normality.

Another option is to address the hypothesis of equality of dependent correlations with overlapping variables via replacement of Pearson's correlation with an alternative correlation coefficient for which tests have been developed that are robust to violation of normality. Some such tests have been developed with mixed results in a narrow set of simulation conditions (see, for example, Wilcox, 2016). Nevertheless, it should be noted that the statistical analysis of data must be guided by the research question that a study addresses; replacing Pearson's correlation with an alternative coefficient may not then be appropriate to address the research question.

We must also stress that our study only included data generated to have specific univariate distributions whose higher‐order moments are naturally predetermined. Alternatively, non‐normal data can be generated to have predefined higher‐order moments (skewness and kurtosis) without specification of the form of the distribution. This may seem to capture more realistically the features of psychologically relevant data distributions (see, for example, Cain et al., 2017), which immensely broadens the set of scenarios for an analysis of the robustness of the statistical tests of equality of dependent correlations. In addition, Fairchild et al. (2024) have shown that different algorithms that nominally generate non‐normal data with identical levels of skewness and kurtosis produce distributions with distinctly different univariate shapes, with potential implications for the resulting robustness of the tests under study. In consequence, the generalizability of the results of any study on robustness to violations of normality is always uncertain.

Another aspect that must be kept in mind when using these tests in practical applications with (potentially) non‐normally distributed variables is that the mere possibility that the null is true is subject to serious structural threats. Consider the hypothetical case in which X 1 has a Lognormal(0, 1) distribution, X 2 has a Beta(2, 5) distribution, and Y has a chi‐square distribution with 10 degrees of freedom. In these circumstances, Xiang (2019) showed that the ranges of possible correlations vary greatly across pairs of ρ ~1Y ~ ⊂ (−.640, .862), ρ ~2Y ~ ⊂ (−.928, .996), and ρ 12 ⊂ (−.646, .815). Thus, structurally, there is a very limited range of true nulls on consideration that positive‐definiteness is additionally imposed on the triplets of intercorrelations. There may actually be occasions when ρ ~1Y ~ and ρ ~2Y ~ will both be near their maximum (or minimum) values and, yet, their magnitude will differ. General recommendations may not be given for handling these situations in practice although, in a sense, the utility of a correlation lies in its practical implications and not so much in the extent to which it is free of a structural consequence of the forms of distribution of the variables of concern.

Testing for equality of strength of correlation, regardless of sign

What matters in a number of practical situations is whether the unsigned strength of the association of two alternative variables with a target variable differs. For instance, for predictive purposes, the sign of correlation is irrelevant and the relative merits of two candidate predictors depend on whether the absolute value of their correlations with the target variable differ. This entails the null hypothesis H0: |ρ ~1Y ~| = |ρ 2Y |, for which a test does not seem to have been developed. Yet, this null hypothesis branches out into two tractable nulls, namely, H0: ρ 1Y ~ = ρ 2Y ~ (for testing that both correlations are equal in value and sign, which is the case considered thus far in this paper) and H0: ρ ~1Y ~ = −ρ ~2Y ~ (for testing that both correlations differ only in sign). As discussed by Boyer et al. (1983) in cases in which r 1Y ~ and r 2Y ~ differ in sign, this difference in sign can be circumvented by testing instead the null H0: ρ ~1Y ~ = ρ ~−2Y ~, where ρ ~−2Y ~ stands for the correlation between –X 2 and Y. Thus, this test of equality of unsigned strength of correlation is applied identically to the previous one but using instead the correlation between X 1 and –X 2 and the correlation between –X 2 and Y. An analysis of Type I error rates and power analogous to that presented above was conducted for the same 10 tests considered in this paper and the results were identical. For illustration, Figure 12 shows accuracy results under trivariate normality, which are analogous to those in Figure 6. The only natural difference is that results for negative and positive values of ρ 12 are swapped because true nulls now lie on the off‐diagonal in the panels of Figure 1, where ρ ~1Y ~ = −ρ ~2Y ~.

In the absence of a proper test of the null H0: |ρ ~1Y ~| = |ρ 2Y |, it should be stressed that the choice between testing H0: ρ 1Y ~ = ρ 2Y ~ and testing H0: ρ ~1Y ~ = −ρ ~2Y ~ must be based (and justified) on a priori theoretical or practical considerations. The result of using one or the other will generally differ, a natural consequence of the fact that each of the nulls captures a different reality. For instance, consider a situation in which r 1Y ~ = −.22, r 2Y ~ = .17, and r 12 = −.35 with a sample size of 100. Use of the standard Williams test for H0: ρ 1Y ~ = ρ 2Y ~ yields t W = −2.4077 with p = .018; in apparent contrast, use of the same test for H0: ρ ~1Y ~ = −ρ ~2Y ~ yields t W = 0.4437 with p = .658. Although the test comes out significant in the former case and not in the latter, one has to realize that the first test rejects the null that the two correlations have the same magnitude and sign whereas the second test does not reject the null that the two correlations have the same magnitude regardless of sign. It is incumbent on the researcher to decide which of the two tests captures the statistical aspects of the research question that motivated the comparison of correlations. Nevertheless, a pattern of sample correlations similar to that in the preceding example (namely, r ~1Y ~ and r ~2Y ~ with different signs along with negative and relatively large r 12) is suggestive of a negative relation between X 1 and X 2 that naturally results in correlations between X 1 and Y and between X 2 and Y that differ in sign. In contrast, a different pattern in which r 12 is relatively large and positive will naturally result in correlations between X 1 and Y and between X 2 and Y that have the same sign.

Extensions

A natural extension of the study reported in this paper consists of conducting analogous analyses under the remaining scenarios described in the Introduction, namely, equality of independent correlations and equality of dependent correlations with non‐overlapping variables. Indeed, there are a number of alternative tests for each of these scenarios (see Diedenhofen & Musch, 2015) but a study along those lines is beyond the scope of the present paper.

Practical recommendations

Our results advise against the use of Pearson–Filon, Olkin, Hotelling, Hendrickson–Stanley–Hills, and Hittner–May–Silver tests of equality of dependent correlations with overlapping variables. The five remaining tests (standard Williams, Dunn–Clark, Steiger, Meng–Rosenthal–Rubin, and Zou) are all dependable and virtually identical as regards their accuracy, power, and applicability under normality, but they are robust to violation of the normality assumption only under some forms of non‐normality that are impossible to identify for actual data.

Needless to say, the core recommendation is to compare dependent correlations with overlapping variables by using one of these five dependable tests. Given their uncertain robustness, use of these tests in practical applications should perhaps be accompanied by tests of normality for each of the variables of concern, not with the purpose of validating the tests of equality of correlations themselves but with the purpose of modulating their interpretation. Specifically, the statistical validity of the results of one of the above‐mentioned tests of equality of dependent correlations with overlapping variables is less questionable if compatibility with normal distributions is not rejected by normality tests within their own limitations of accuracy and power.

The simple practice of comparing correlations by eye or via the p‐values of isolated tests of significance with respect to zero should also be avoided. The prevalence of such practices is hard to investigate because of the absence of an identifiable characteristic of the papers reporting them that could lead to database searches. Yet, reporting practices involving proper use of tests is easier to investigate. The Introduction mentioned a search for papers that cited cocor (Diedenhofen & Musch, 2015) and reported the use of tests of equality of dependent correlations, with the results listed in Table A1 in Appendix A. Note that 10 of the papers did not report which test had been used, one reported that all tests gave the same result, and seven reported using the defective Pearson–Filon or Hittner–May–Silver tests. Only nine papers reported using the dependable Dunn–Clark, Steiger, or Meng–Rosenthal–Rubin tests. In the interest of transparency and best reporting practices, the particular test that was chosen in a given study should be reported and the choice should be among the tests that have proven dependable under normality, with a preference for the standard Williams test over all others.

AUTHOR CONTRIBUTIONS

Miguel A. García‐Pérez: Conceptualization; methodology; software; data curation; investigation; validation; formal analysis; supervision; resources; project administration; visualization; writing – review and editing; funding acquisition; writing – original draft.

DISCLOSURE OF ARTIFICIAL INTELLIGENCE‐GENERATED CONTENT (AIGC) TOOLS

The author did not use generative AI or any other similar tools at any step of this research or for any purpose.

CONFLICTS OF INTEREST STATEMENT

The author declares no conflicts of interest.

Supporting information