Authors: Christie Newton, Justin Feeney, Gordon Pennycook
Categories: Articles, analytic thinking, cognitive reflection, dual-process theories, intuitive thinking, thinking styles
Source: Personality & Social Psychology Bulletin
Many measures have been developed to index intuitive versus analytic thinking. Yet it remains an open question whether people primarily vary along a single dimension or if there are genuinely different types of thinking styles. We distinguish between four distinct types of thinking Actively Open-minded Thinking, Close-Minded Thinking, Preference for Intuitive Thinking, and Preference for Effortful Thinking. We discovered strong predictive validity across several outcome measures (e.g., epistemically suspect beliefs, bullshit receptivity, empathy, moral judgments), with some subscales having stronger predictive validity for some outcomes but not others. Furthermore, Actively Open-minded Thinking, in particular, strongly outperformed the Cognitive Reflection Test in predicting misperceptions about COVID-19 and the ability to discern between vaccination-related true and false news. Our results indicate that people do, in fact, differ along multiple dimensions of intuitive-analytic thinking styles and that these dimensions have consequences for understanding a wide range of beliefs and behaviors.
Keywords: analytic thinking, cognitive reflection, dual-process theories, intuitive thinking, thinking styles
It has long been theorized that people differ in how much they prefer to engage in effortful and deliberative thought (Asch, 1952; Cacioppo & Petty, 1982; Cohen et al., 1955; Maslow, 1943) and, relatedly, that some people may rely more on their intuitions and gut feelings when making judgments (Epstein, 1983; Epstein et al., 1996; Jung, 1953). Scores of empirical studies have shown that individual differences in thinking styles (broadly construed) have apparent widespread consequences, from predicting differences in the judgments and decisions that people make (e.g., Dewberry et al., 2013; Epstein et al., 1996; Haran et al., 2013; Juanchich et al., 2016; Petty et al., 2009; Stanovich & West, 1998, 2000; Toplak et al., 2011), to variation in people’s beliefs and values (e.g., Baron et al., 2015; Lindeman, 2011; Pennycook et al., 2012, 2015a; Pennycook & Rand, 2019a; Svedholm-Häkkinen & Lindeman, 2013), to academic performance, happiness, health and longevity (Grossmann et al., 2013; Petty et al., 2009; Stevenson & Hicks, 2016; von Stumm et al., 2011). However, the popularity of research on thinking styles is a double-edged There is strong evidence that thinking styles are important, but very little parsimony in the literature.
In particular, there is a lack of theoretical consensus on the nature of the underlying cognitive processes that determine whether someone is relatively “intuitive” or “analytic.” For example, there are meaningful conceptual differences between someone having a “need for cognition,” which is described as enjoying effortful thinking, and someone having an “actively open-minded thinking style,” which is described as someone who is disposed to deeply questioning their prior beliefs and intuitions according to evidence. Furthermore, it is even unclear if people primarily differ in terms of whether they are intuitive or analytic, or if there are genuinely various different types of intuitive-analytic thinking styles. Relatedly, there are a large number of different measures that purport to index thinking styles (even when defined fairly narrowly). Indeed, a review of the literature on thinking styles reveals at least 15 measures that claim to assess variation in how people think—albeit with a broad definition of “thinking”—that include elements related to intuition and reflection. These scales are sometimes related to specific thinking processes, such as whether one trusts their intuitive hunches, but many have items that also relate to information integration and whether one values rationality and so on. Nonetheless, they all relate to higher-order cognition in a broad sense and relate to the way that people form beliefs and make decisions.
Although a great deal of research has focused on the implications of individual differences in thinking styles, there is no consensus about the nature of these individual differences. Do people primarily differ along a single dimension (from intuitive to analytic) or are there different types of intuitive-analytic thinking styles? And, if so, does this matter? And which types are the most important for the outcomes that have been a major focus of recent research (e.g., epistemically suspect beliefs, vaccination attitudes, etc.). Here we report the findings of two studies that support the idea that there are, in fact, different types of intuitive-analytic thinking styles and that this distinction is important.
Dual-process (or dual-system) theory connects the various thinking style measures in a broad sense. Dual-process theory states that human cognition can be separated into two fundamentally different types of intuitive processes that are automatic and generally faster and not effortful (“Type 1” or “System 1”) and analytic or reflective processes that are generally slower and more effortful (“Type 2” or “System 2”; Chaiken & Trope, 1999; Evans, 2008; Evans & Stanovich, 2013; Kahneman, 2011). Many of the thinking style measures explicitly mention concepts that are related to dual-process theory like intuition/emotion versus reflection (Preference for Intuition and Deliberation; Betsch, 2004), rationality versus intuition (Rational-Experiential Inventory; Pacini & Epstein, 1999), and the importance of rationality (Ståhl et al., 2016) and questioning (Intolerance of Ambiguity; Budner, 1962). Even measures with a less obvious connection, such as the Comprehensive Intellectual Humility Scale (Krumrei-Mancuso & Rouse, 2016) or the Self-Righteousness Scale (Falbo & Belk, 1985), ask questions that are related to this broad perspective such as “I am willing to change my opinions on the basis of compelling reason” and “To me, things don’t seem black and white, they’re, mostly shades of gray.” Thus, the general dual-process perspective appears pervasive in a large number of scales (as outlined in Table 1).
Importantly, however, there is a growing appreciation among dual-process theorists that the theory should move away from simply characterizing the distinction between two different types of processes (e.g., by listing features of one versus the other) and toward forming models of how intuitive and analytic processing operate and interact (De Neys, 2017; Pennycook, 2022; Pennycook et al., 2015b; Thompson, 2009). This has consequences for the measurement of thinking styles because it implies that there may be other characteristics of how people engage in intuitive versus analytic thinking that are separate from the mere preference for one or the other. Interestingly, the Need for Cognition scale is perhaps the most common thinking style scale across literatures (e.g., the original has been cited more than 7,500 times according to Google Scholar) but it is essentially a measure of the “tendency for an individual to engage in and enjoy thinking” (Cacioppo & Petty, 1982, p. 116).
A central focus of more modern dual-process theories of reasoning is determining the factors that lead to the engagement and nature of analytic thinking (Bago & De Neys, 2017; Pennycook, 2022; Pennycook et al., 2015b; Thompson et al., 2011). This perspective implies that analytic thinking is (sometimes but not always) centrally focused on overriding intuitions (Pennycook, 2023)—an aspect that is featured in some but not all of the above-cited thinking styles measures (and, most notably, not a strong feature of “Need for Cognition”). One specific measure that does directly relate to this category of dual-process theories is the Actively Open-minded Thinking (AOT) scale, which measures the disposition toward reflectivity, the willingness to consider evidence that goes against held beliefs, and the willingness to consider alternative opinions and explanations (Stanovich & Toplak, 2019). In other words, the AOT scale assesses the tendency to engage in Type 2 processing specifically as it pertains to suppressing and overriding incorrect intuitive (Type 1) responses.
Another common strategy is to assess intuitive-analytic thinking using behavioral measures. By far, the most popular measure for this is the Cognitive Reflection Test (CRT; Frederick, 2005; cited over 5,500 times according to Google Scholar). The CRT consists of problems that trigger an intuitive response that is incorrect; for “A bat and ball cost 1.00 more than the ball. How much does the ball cost?” The most common intuitive response is “10 cents,” however, the correct response is “5 cents.” People who do well on the CRT are thought to be more reflective because they stop and question their intuition on the problems (Campitelli & Gerrans, 2014; Travers et al., 2016). Suitably, the CRT is associated with other types of intuitive judgments (Campitelli & Labollita, 2010; Koehler & James, 2014; Pennycook et al., 2015a; Toplak et al., 2011) as well as thinking styles as measured by self-report scales (such as those discussed above; Pennycook, Cheyne, et al., 2016; Thomson & Oppenheimer, 2016; Toplak et al., 2011). However, the CRT does not capture differences in aspects or types of thinking styles—rather, it can only capture variation along a single dimension. Sole reliance on the CRT may, therefore, cause oversimplification in the precise nature of people’s thinking styles and how (or whether) these individual differences are consequential.
Here we set out to investigate potential differences in predictive validity between different types of intuitive-analytic thinking styles. For example, is the mere enjoyment of effortful thinking the reason why deliberative people tend to (for example) be less religious (Pennycook, Ross, et al., 2016), or is the specific tendency to question your beliefs in an actively open-minded way more important (Pennycook, Cheyne, et al., 2020)? Investigating these different types of thinking styles is difficult, however, given the sheer number of measures outlined in Table 1. We, therefore, constrained the conceptual space by narrowing down the list of potential scale items that measure individual differences in intuitive-analytic thinking styles (the details of these studies are fully described in Online Appendix B).
For this, we took the 265 unique items from the 15 scales outlined in Table 1 and completed an initial study where participants (N = 774) were given a randomly selected subset of the items along with the CRT. We then narrowed down the list empirically by taking only those items that had at least a moderate correlation (r = .21, following the benchmark in Gignac & Szodorai, 2016) with the CRT. Then, across five additional studies with 1,196 participants, we explored the factor structure of the remaining 50 items and narrowed them down into four unique factors. The scale that emerged from this, the 4-Component Thinking Style Questionnaire, includes 24 items (6 items per subscale). The items are listed in Table 2.
Interestingly, despite the number of scales that contributed items to the first and second stage of this validation (12 out of 15 scales provided items to the resulting set^ 1 ^), three of the factors correspond conceptually to scales that are already in (1) Preference for Intuitive Thinking (PIT; i.e., “Faith in Intuition” and “Experiential Inventory”), (2) Preference for Effortful Thinking (PET; i.e., “Need for Cognition” and “Rationality Inventory”), and (3) AOT. The final factor, which we refer to as Close-Minded Thinking (CMT), came out of a mix of other scales and relates to the extent to which people see the truth in black-and-white. It should be noted, however, that CMT and AOT (or the PIT and PET, for that matter) should not be considered parallel scales that operate on a continuum. In fact, the scales were derived specifically to represent distinct types of thinking styles. Indeed, each of the four subscales was shown to be reliable and confirmatory factor analyses consistently supported a four-factor structure despite the subscales being moderately intercorrelated.
Finally, it should also be noted that every item in each subscale is in the same direction (or “key”)—this is because positively and negatively worded items tend to form distinct factors (Kam & Meyer, 2015, 2022; Kam & Sun, 2022; McLarnon et al., 2016), as was the case in our own validation (see Online Appendix B). Thus, to maintain strong reliability within the subscales, all of the items have the same direction. Furthermore, every item across subscales is in the same direction; that is, a higher score corresponds to what has been considered in past work as greater intuitiveness and a lower score corresponds to greater analyticity. As a consequence, the AOT and PET subscales are both entirely reverse-scored.
In Study 1, we focused on seven measures that have been shown in previous studies to relate to intuitive-analytic thinking styles. One group of these measures relates to people’s beliefs. For example, the disposition to think analytically has been associated with religious disbelief (Pennycook et al., 2012; Shenhav et al., 2012; for a meta-analysis, Pennycook, Ross, et al., 2016). This is a central finding because it indicates that the tendency to reflect on and question one’s prior beliefs (as indexed by measures such as the CRT) is even linked to skepticism about beliefs that are core to one’s identity. Critically, this perspective on the association between religious disbelief and analytic thinking implies that AOT may be a particularly salient aspect of thinking styles that is driving the causal association (Pennycook, Cheyne, et al., 2020); however, other theories posit that the association is driven more by the “intuitiveness” of religious beliefs (Shenhav et al., 2012), which suggests that PIT should be a stronger predictor.
Related work has also shown that paranormal belief is both associated with stronger intuitiveness (based on a self-report measure similar to the PIT; Aarnio & Lindeman, 2005) as well as a lower willingness to engage in analytic thinking (based on lower performance on the CRT; Pennycook et al., 2012). At the same time, paranormal belief has been linked to openness (Eudell & Campbell, 2007) and is generally not predicted by dogmatism, indicating that CMT may not be a particularly strong predictor. This contrasts with conspiratorial belief, which is indicative of dogmatic tendencies and, potentially, CMT (Prooijen & van Krouwel, 2016; Richey, 2017; conspiracy beliefs have also been associated more broadly with decreased analytic thinking; Binnendyk & Pennycook, 2022; Swami et al., 2014; Yelbuz et al., 2022). Thus, the 4-Component Thinking Styles Questionnaire (4-CTSQ) may help distinguish between the underlying mechanisms that produce variation in different types of epistemically suspect beliefs (c/o Lobato et al., 2014) beyond the mere observation that analytic thinking is, in a broad sense, associated with skepticism about such beliefs (Pennycook, 2023).
We also included measures that are focused on judgments that have been linked to intuitive-analytic thinking. A classic and highly influential dual-process model—the social-intuitionist model (Haidt, 2001)—argues that intuitions typically dominate reasoning to such an extent that thinking analytically is unlikely to be predictive in cases where a highly salient intuition is cued. To illustrate this point, Haidt (2001; see also Haidt, 2012) used disgust-based moral dilemmas where people are given a narrative that contains a taboo (and disgusting) act, such as incest between siblings, but that emphasizes that no harm is done. People are then argued to rely on a disgust-based intuition when judging whether the act is morally wrong. Interestingly, however, prior work has shown that people who are more reflective are less likely to rely on disgust when making moral judgments (Pennycook et al., 2014b). One interpretation of this, which is consistent with the social-intuitionist model, is that this occurs because people who are more intuitive (i.e., those with a stronger PIT) are more strongly compelled by the dilemma (and hence have a stronger disgust-based response). However, an alternative account is that people who are more willing to question their gut feelings and intuitions (i.e., those higher on AOT) are less likely to be influenced by their disgust-based intuitions (Pennycook, 2023; Pennycook et al., 2014b).
Another judgment task that we included is the bullshit receptivity scale, where participants are asked to indicate how profound (or otherwise) they consider random sentences that are constructed using abstract buzzwords (e.g., “Imagination is inside exponential space time events”; Pennycook et al., 2015). People who are more analytic tend to rate the sentences as less profound, and hence are thought to be stronger at detecting bullshit (Pennycook et al., 2015; Pennycook & Rand, 2019b). However, similar to the moral dilemma case, it may be that this trend (evidenced using measures that do not distinguish between different types of intuitive-analytic thinking styles) is driven simply by greater PIT.
Finally, as more exploratory measures that have high practical relevance, we included subjective happiness (which has been associated with intuitiveness in past work; Stevenson & Hicks, 2016) and empathizing (which has been associated with decreased deliberation; Jack et al., 2016; Svedholm-Häkkinen & Lindeman, 2017).
We report all manipulations, measures, and exclusions in these studies.
American participants (n = 509) were recruited from Prolific. Data from seven participants who did not complete the entire study were discarded before all analyses. Fifteen participants who reported random responding and 15 who failed the attention check were excluded from further analysis. Data from 59 participants who did not respond “strongly disagree” to “I have been to every country in the world” were excluded from further analyses. Four hundred and thirteen participants (N = 413) were retained for further analyses (Mean age = 33.5, 222 male, 184 female, 7 who chose another category or did not respond). These exclusion criteria were preregistered (see https://osf.io/7kf5b/). We did not complete a power analysis prior to data collection; however, correlations in personality psychology tend to stabilize at an N of 250 (Schönbrodt & Perugini, 2013) and a sample of 413 provides 80% power to detect effect sizes as small as r = .14.
We used the 24-item 4-CTSQ as validated in Online Appendix B. The scale has four subscales created by using average scores and all had good (and roughly similar) AOT (α = .89, M = 4.29, SD = .99), CMT (α = .83, M = 2.82, SD = .96), PIT (α = .92, M = 3.59, SD = .95), and PET (α = .88, M = 4.55, SD = .94). AOT and PET were recoded prior to running the analyses (see Appendix).
Participants were given the 8-item Religious Belief Questionnaire (Pennycook et al., 2014a) that assessed eight commonly held religious afterlife, angels, demons, heaven, hell, miracles, Satan, soul. The measure had excellent reliability (α = .96). To index categorical religious belief change, we asked two further questions. First, participants were asked to indicate which of four possible categories best represents their current stance toward God (or gods): “I believe in God” (theism; N = 153), “I don’t really take a stance on God” (apathetic; N = 40), “I don’t really know if God exists or not” (agnostic; N = 120), and “I don’t believe in God” (atheist; N = 100). These were chosen because they represent distinct and presumably memorable stances toward God. Critically, we then asked participants a follow-up question where they indicated whether they, at any point in their past, fell into one (or more) of the other categories. We then coded cases where participants indicated a prior stance that was different from their present stance as indicating categorical religious belief change (47.7% of the sample indicated change based on this measure). This was included as an exploratory (non-preregistered) variable.
Participants completed a slightly revised, 10-item version of the Paranormal Belief Scale without any traditional religious belief or witchcraft items (Pennycook et al., 2012; Tobacyk, 2004). Items covered four domains of supernatural Psi (“Mind reading is possible”), Superstition (“If you break a mirror, you will have bad luck”), Spiritualism (“Reincarnation does occur”), Extraordinary life forms (“There is life on other planets that has reached Earth”), and Precognition (“The horoscope accurately tells a person’s future.”). The measure had strong reliability (α = .89).
Participants completed a slightly revised, 10-item version of the Belief in Conspiracy Theory Inventory (Swami et al., 2011). An example of these items “Area 51 in Nevada, United States, is a secretive military base that contains hidden alien spacecraft and/or alien bodies.” The measure had good reliability (α = .87).
Participants read two disgust-based moral dilemmas by Haidt et al. (1993). One dilemma describes an incestuous relationship between siblings and the other is about sexual relations with a dead chicken. Two follow-up questions were “Is this morally wrong?” (1 = not morally wrong; 7 = extremely morally wrong), and “Is this disgusting?” (1 = not disgusting; 7 = extremely disgusting). Average scores were created for each scale with higher scores reflecting a stronger view that the dilemma is morally wrong (α = .74) and disgusting (α = .54).
Participants completed a shortened 5-item version of the Bullshit Receptivity Scale (Pennycook et al., 2015), which measures the extent to which people find profundity in randomly generated sentences filled with buzzwords. One example of an item is “Imagination is inside exponential space time events.” The measure had good reliability (α = .87).
Participants were given the 22-item version of the Empathy Quotient (EQ-Short; Wakabayashi et al., 2006). One example of an item is “I can pick up quickly if someone says one thing but means another.” The measure had good reliability (α = .89).
The 4-item Subjective Happiness Scale (Lyubomirsky & Lepper, 1999) was presented to assess how happy participants rate themselves compared with others. Responses are rated on a 7-point Likert-type scale with varying labels for items; for example, one item, “In general, I consider myself”: is rated from 1 (not a very happy person) to 7 (a very happy person), whereas other items are rated 1 (less happy) to 7 (more happy) and 1 (not at all) to 7 (a great deal). The measure had good reliability (α = .90).
After consent, participants were either given the 24-item 4-CTSQ first or following the various outcome measures (i.e., it was counterbalanced). Participants then completed measures of religious, paranormal, conspiratorial beliefs, moral dilemma, bullshit receptivity, empathizing quotient, and subjective happiness (the order of which was randomized). Finally, participants completed a demographics questionnaire.
In our preregistration (available on OSF), we predicted that our correlated four-factor 4-CTSQ model would fit the data better than alternative factor structures.^ 2 ^ We tested the factor structure using confirmatory factor analysis and used full information maximum likelihood estimation to handle missing data within Mplus 8.4 (Muthén & Muthén, 2017), which is less biased than listwise or pairwise deletion (Newman, 2014). Four measurement models were (a) a one-factor model of general 4-CTSQ; (b) an orthogonal (uncorrelated) four-factor model; (c) an oblique (correlated) four-factor model; (d) Model C with a second-order 4-CTSQ factor. We compared models by examining whether changes in chi-square exceeded the critical value and whether changes in the comparative fit index (CFI) model fit exceeded the recommended cut-point of .002 (Meade et al., 2008). The CFA results confirmed that the best-fitting model comprised four correlated factors (Model C), supporting our hypothesis (see Table 3). Factor loadings were very similar to prior validation in Online Appendix B and are reported in Supplementary Materials.
As preregistered, bivariate Pearson (r) correlations were computed with the four subscales and various outcome measures (see Table 4). The correlation between the subscales was generally moderate, with the strongest correlation (r = −.46) being between AOT and PIT. To disentangle the intercorrelations between our subscales and the various dependent variables, multiple regression analyses were conducted for each of the belief measures with the four subscales as predictors (note that these analyses were not preregistered). A summary of the results can be seen in Table 5.
The results indicate strong predictive validity across the subscales. All four subscales were (largely) significantly correlated with most of the outcome measures. There were some distinct patterns that are informative for theory, however. First, AOT was a much stronger predictor of religious belief (r = −.49) than PIT (r = .25; this difference in effect sizes was significant using a post hoc Fisher r-to-z transformation, z = −9.2, p < .001). Similarly, in a multiple regression predicting religious belief (Table 5), AOT (β = −.47) was significantly predictive but PIT (β = .02) was not. Interestingly, the mere PET was similarly non-predictive of religious belief relative to AOT (and to a lesser extent, CMT), indicating that thinking in an actively open-minded way, as opposed to simply enjoying thinking, is a more central element of why thinking style is associated with skepticism about religious claims. This was supported by our exploratory analysis of categorical religious belief change. Specifically, in a binary logistic regression with belief change (or not) as the dependent variable (and controlling for religious belief), AOT (B = .33, Wald = 5.63, p = .018) and CMT (B = −.32, Wald = 7.18, p = .007) were unique predictors of belief change, but PIT (B = −.08, Wald = .38, p = .537) and PET (B = −.21, Wald = 2.95, p = .086) were not (and, in fact, PET was in the opposite direction as would be expected).
Second, the results for paranormal and conspiratorial belief offer an interesting contrast. In particular, whereas CMT is associated with conspiracy belief (r = .18), it was not associated with paranormal belief (r = −.01); this difference in effect sizes was significant using a post hoc Fisher r-to-z transformation, z = −3.7, p < .001. In fact, in a multiple regression analysis predicting paranormal belief (Table 5), CMT was actually negatively predictive of paranormal belief (β = −.11) once the other subscales were taken into account. However, it is also noteworthy that AOT, and not CMT, was the strongest predictor of conspiracy beliefs.
Moving on to the judgment tasks, we found that disgustingness ratings for the moral dilemmas was only significantly correlated with PIT (in both the zero-order correlations, Table 4, and the regression analysis, Table 5). In contrast, moral judgments were roughly equally predicted by PIT, AOT, and CMT (but not PET). This indicates that intuitiveness does seem to be an important element of moral judgment when there are strong disgust-based intuitions, consistent with the social-intuitionist model (Haidt, 2001); however, one’s willingness to question their intuitions also seems to be roughly as important even though it does not predict how disgusting one perceives the dilemmas to be, thereby supporting more “default-interventionist” models of moral judgment (e.g., Pennycook, 2023).
A similar conclusion is evidenced in the results for the bullshit receptivity measure. Specifically, both AOT and PIT were strongly and independently associated with profundity ratings for random pseudo-profound bullshit sentences (in opposite directions, of course, and roughly equally so). Thus, both one’s willingness to reflect and one’s intuitiveness are linked to detecting pseudo-profound bullshit. Being close-minded or not enjoying effortful thought appears to be less important for this.
Finally, empathizing and subjective happiness (our more exploratory measures) are perhaps surprisingly most strongly associated with the PET. Specifically, people who have a stronger preference are more likely to rate themselves as happy and are more likely to empathize with others—a pattern that is actually the opposite (albeit weakly so) for people who are higher in AOT. This illustrates that the disposition to enjoy effortful thought is, in fact, a unique element of intuitive-analytic thinking styles that has important consequences that differ meaningfully from other elements of thinking styles.
Study 1 established that the 4-CTSQ has strong predictive validity for a wide range of important psychological outcomes. We also completed a study, reported in the Supplementary Materials (“Study 3”), that confirms that the 4-CTSQ is also strongly predictive of actual performance on behavioral tasks that are thought to be influenced by individual differences in analytic thinking. This latter result is not particularly surprising, of course, since the initial winnowing down of items in Study 1 of Online Appendix B was done by finding only those items that are both thought to measure intuitive-analytic thinking styles and that had at least a moderate correlation with performance on the CRT. A key question, then, is whether the 4-CTSQ is a stronger measure than the CRT.
To investigate this, we turn to an issue of major present COVID-19 misperceptions and vaccination hesitancy. Numerous reports have documented the apparent spread and negative consequences of COVID-19 misinformation (Enders et al., 2020; Loomba et al., 2021; Montagni et al., 2021). Critically, research shows that analytic thinkers are less likely to believe fake news and are better at distinguishing between true and false news, both in the context of political news (Bago et al., 2020; Pennycook & Rand, 2021; Ross et al., 2021; for a meta-analysis, see Pennycook & Rand, 2021) and news about COVID-19 (Pennycook, McPhetres, et al., 2020). More broadly, misperceptions about COVID-19 and vaccination hesitancy have been associated with decreased analytic thinking as well (Pennycook, 2023). However, these prior studies have relied on behavioral measures of intuitive-analytic thinking styles that vary on a single dimension (such as the CRT)—thus, we ask whether distinguishing between different aspects of thinking styles may improve predictive validity. Following the results of Study 1, it is expected that AOT, in particular, should be a key element (see also Roozenbeek et al., 2023).
In Study 2, we also set out to replicate the exploratory religious belief change analysis from Study 1 with a much larger and more representative sample. For this, we administered the categorical God-belief question and the follow-up asking about whether one held any other position on the question of God in the past.
We report all manipulations, measures, and exclusions in these studies.
Nationally representative samples were recruited via YouGov (who fielded the survey) in the United States (n = 1,090; Mean age = 49, 429 male, 620 female, 39 who chose another category or did not respond) and Canada (n = 1,001; Mean age = 51, 448 male, 529 female, 22 who chose another category or did not respond) in August 2020. The surveys were identical, apart from some minor differences in how the demographics were recorded in each country. We did not complete a power analysis prior to data collection.
We used a 6-item version of the CRT that included reworded versions of the original 3-item CRT (Frederick, 2005; Shenhav et al., 2012) and 3 items from the non-numeric CRT (excluding the “dirt in a hole” item; Thomson & Oppenheimer, 2016). The measure had good reliability in both samples (Canada α = .70, M = .40, SD = .30; United States α = .70, M = .40, SD = .30).
We used the 24-item 4-CTSQ as specified in Study 1. The subscales had good reliability in both AOT (Canada α = .88, M = 3.37, SD = 1.33; United States α = .89, M = 3.48, SD = 1.34), CMT (Canada α = .78, M = 3.84, SD = 1.23; United States α = .80, M = 3.93, SD = 1.24), PIT (α = .89, M = 4.53, SD = 1.11; United States α = .90, M = 4.49, SD = 1.12), and PET (Canada α = .85, M = 3.77, SD = 1.19; United States α = .86, M = 3.81, SD = 1.25).
We asked seven questions relating to COVID-19 vaccinations (Pennycook et al., 2021). Three of the questions related to vaccination intentions (note that this study was completed in August 2020 when the COVID-19 vaccines were not yet approved or available): “I am, in general, willing to get vaccinated for COVID-19,” “I would vaccinate my children for COVID-19, given the opportunity,” and “If a COVID-19 vaccination were available for free, would you get vaccinated?.” The first two items were rated on a 7-point Likert-type scale (1 = strongly disagree; 7 = strongly agree) and the third item was rated on a 5-point scale from “definitely not get it” to “definitely get it.” To create one COVID-19 vaccination intention scale, we z-scored the three questions and computed the mean. We also asked four questions about vaccination “We can rely on a SARS-CoV-2 vaccine to stop the spread of COVID-19,” “The COVID-19 vaccine is not likely to cause unforeseen problems in children,” “Authorities will promote the COVID-19 vaccination for financial gain, not for people’s health” (reverse-scored), and “Being exposed to COVID-19 naturally is safer for the immune system than being exposed through vaccination” (these were all rated the same 7-point Likert-type scale as above). Although the vaccination attitudes measure had excellent reliability in both countries (Canada α = .87, United States α = .87), the measure of vaccination intentions had relatively poor reliability (Canada α = .59, United States α = .62), and therefore should be interpreted with caution.
A 12-item measure of popular misperceptions about COVID-19 was used (Pennycook et al., 2021). It included items like “The seasonal flu is just as dangerous as COVID-19,” and “The coronavirus that causes COVID-19 was engineered as a bioweapon in a Chinese lab.” The measure had excellent reliability in both countries (Canada α = .84, M = 2.73, SD = 1.18; United States α = .85, M = 2.98, SD = 1.29)
A 20-item measure was used which consists of 10 fake (false) and 10 real (true) news headlines about COVID-19 vaccinations in the format of a Facebook post (as in Pennycook, McPhetres, et al., 2020). The false headlines were chosen by fact-checking websites such as snopes.com and factcheck.org. The true headlines were from mainstream media sources and reported accurately on true events (see Pennycook et al., 2021). The headlines were randomized and presented one at a time (see OSF for the headlines). Each headline was paired with the following “To the best of your knowledge, how accurate is the claim in the above headline?” which was rated on a 4-point scale (1 = not at all accurate; 4 = very accurate; Pennycook & Rand, 2019a). An average score was created for accuracy on the fake headlines and the true headlines, respectively. We will focus on “news discernment,” which is the overall accuracy of people’s judgments (for a discussion focusing on news discernment instead of mere belief in fake news, see Pennycook & Rand, 2021) and is calculated by subtracting the average scores for fake headlines from the average score for true headlines.
We used the same categorical belief change questions that were administered in Study 1. The breakdown for the theism question is as “I believe in God” (theism; NCanada = 533, NUnited States = 685), “I don’t really take a stance on God” (apathetic; NCanada = 125, NUnited States = 101), “I don’t really know if God exists or not” (agnostic; NCanada = 150, NUnited States = 175), and “I don’t believe in God” (atheist; NCanada = 190, NUnited States = 127). Given the large combined sample size, we were able to also investigate change within each category (albeit by combining data from Canada and the United States). In total, 29.4% of Canadians and 29.7% of Americans in the sample indicated change based on this measure—this is smaller than in Study 1 owing to the higher level of religious belief overall; that is, across countries, theists reported belief change in only 13.6% of cases whereas non-theists reported change in 51.8% of cases.
Participants in both countries were asked about both social and economic political ideology and rated their political ideology on the following 5-point (1) strongly liberal, (2) liberal, (3) moderate, (4) conservative, (5) strongly conservative. For simplicity, we combined the social and economic ideology questions (which correlated very highly in both r = .78 in Canada, r = .82 in United States) into a single conservatism measure. We also created a categorical ideology measure by combining all of those who rated themselves as lower than 3 as liberal, as higher than 3 as conservative, and at 3 as moderate.
The survey began with basic demographics (location, age, gender). The first measure was the vaccination attitude/intention questions, followed by the misperceptions measure, and finally the news discernment measure. The CRT and 4-CTSQ always came after, and the CRT was always completed before the 4-CTSQ. There were also some measures included in the survey (intermixed among the aforementioned measures) that are outside the scope of the present investigation. These include questions about general flu vaccinations, perceptions of risks and other attitudes relating to COVID-19, a basic science knowledge test, bullshit receptivity, media trust questions, and a public service announcement image regarding COVID-19 vaccines, and various demographic and political questions. A full list of the materials in the order that they were presented for both countries is available on OSF.
We calculated bivariate Pearson (r) correlations between the four subscales and four key dependent measures (Table 6). The four subscales were all significantly correlated with COVID-19 vaccination attitudes, COVID-19 vaccination behaviors, misperceptions about COVID-19, and news discernment. Furthermore, most correlations were in the medium to large range for effect sizes and, critically, many were larger than the correlations between the CRT and the outcome measures. Specifically, the 4-CTSQ subscales significantly outperformed the CRT for 15 out of 32 correlations (using Fisher’s r-to-z transformation; the correlations that are significantly stronger for a 4-CTSQ subscale than for the CRT are in bold in Table 6; see Supplementary Materials for full statistics). However, this was not equally distributed across subscales. Indeed, the AOT significantly out-predicted the CRT in six out of eight cases and PIT out-predicted the CRT in four out of eight cases. CMT (two out of eight) and PET (three out of eight) were not quite as strong for these outcomes. In contrast, the CRT only outperformed the 4-CTSQ in 3 out of 32 cases.
Turning to the pattern across measures, AOT emerged as the strongest predictor of misperceptions about COVID-19 and news discernment (with very robust correlations in both countries, r’s > .50 for misperceptions and r’s > .40 for discernment)—a result that offers a strong parallel to Study 1. AOT was also one of the strongest predictors of COVID-19 vaccination attitudes and intentions, although in this case, PIT had similar correlations. In contrast, PET was not a strong predictor of attitudes and behavior even though it was a strong predictor of misperceptions and discernment. These results are paralleled in the regression analyses (Table 7).
Finally, turning to the issue of political ideology, previous work found that thinking disposition measures were less predictive of several outcome measures for conservatives relative to liberals in a U.S. sample (Pennycook, Cheyne, et al., 2020). Furthermore, attitudes about COVID-19 (including vaccinations) are strongly associated with political ideology in both the United States and Canada (Calvillo et al., 2020; Havey, 2020; Pennycook, 2023; Shepherd et al., 2020). Thus, we investigated the predictive validity of the 4-CTSQ separately for liberals (NCanada = 398; NUnited States = 351), moderates (NCanada = 317; NUnited States = 338), and conservatives (NCanada = 286; NUnited States = 401) as another robustness test. These results are displayed in Table 8, which support several key conclusions. First, in the case of misperceptions and news discernment, the CRT and 4-CTSQ subscales are all significantly predictive regardless of political ideology or country. Furthermore, in most cases, AOT, in particular, emerges as a stronger predictor than the CRT for misperceptions and news discernment. Although the measures are generally more predictive for liberals than conservatives, the results are quite robust even for conservatives (e.g., AOT correlates with misperceptions at r = −.40 for Canadian conservatives and r = −.38 for American conservatives).
The results for vaccine attitudes and intentions among moderates and conservatives are notably weaker. For example, although AOT and PIT predicted vaccination attitudes consistently for liberals and moderates in both countries, this is not the case for American conservatives (AOT r = .072, p = .148, PIT r = −.064, p = .202). Interestingly, CMT is the only significant predictor of vaccine attitudes among U.S. participants (r = −.139, p = .005). A similar pattern emerged for vaccination intentions, where PIT (but not AOT) predicts intentions among conservatives in Canada, but not conservatives in the United States or moderates in either country. This serves as a good case study for the value of measuring thinking dispositions in a more comprehensive Based solely on the CRT, it appears that analytic thinking disposition is not relevant for COVID-19 vaccination attitudes or intentions among American conservatives. However, the 4-CTSQ reveals that a specific aspect of thinking disposition, namely CMT, is predictive of vaccination attitudes and intentions among conservatives.
Turning now to categorical religious belief change, our results replicated what we found in Study 1 and, more importantly, indicate that AOT and CMT are stronger predictors of belief change (or a lack of change, for CMT) than performance on the CRT. To assess this, we entered belief change as the dependent variable in a binary logistic regression with our key measures as independent variables (and controlling for country and one’s stance on God). As in Study 1, AOT (B = .18, Wald = 11.79, p = .001) and CMT (B = −.18, Wald = 15.62, p < .001) were unique predictors of belief change. However, there was no significant correlation for PIT (B = .06, Wald = 1.20, p = .274), PET (B = .01, Wald = .05, p = .814), and performance on the CRT (B = .17, Wald = 2.72, p = .099).
To provide a better sense of the pattern of results, Figure 1 shows the mean level of AOT as a function of one’s stance toward God and whether one reported having taken a different stance than their present stance at some point in their life. The data clearly show two main Actively Open-Minded Thinkers are both stronger in their disbelief in God and (across categories) more likely to have changed belief at some point in their life (indeed, there is no interaction between belief and belief change for Canada, the United States, or when the data is combined, all F’s < 2, all p’s < .122).
Figure 1. Actively Open-minded Thinking as a Function of One’s Stance Toward God (Theist, Apathetic, Agnostic, Atheist) and Whether One Reported Having Taken a Different Stance Than Their Present Stance at Some Point in Their Life (No Change, Change).Note. Error bars are 95% confidence intervals. Theist No Change N = 1,052, Theist Change N = 166, Apathetic No Change N = 100, Apathetic Change N = 126, Agnostic No Change N = 151, Agnostic Change N = 174, Atheist No Change N = 167, Atheist Change N = 150.
People differ not only in what they think but how they think. Importantly, this variation is predictive of a wide range of important psychological outcomes. Accordingly, individual differences in intuitive-analytic thinking styles have been used to support various broad theories of human social and cognitive psychology. We investigated whether there are important differences between different types of thinking styles using a 4-Component Thinking Styles Questionnaire (4-CTSQ; see Appendix). Study 1 provided evidence that the 4-CTSQ—and each of the subscales—were broadly predictive of a number of outcomes, although this varied depending on the type of thinking style in question. Study 2 then demonstrated that different types of thinking styles can out-predict the CRT) in the context of COVID-19 misperceptions, fake news discernment, and vaccination attitudes/intentions.
The present studies support the idea that people vary across at least four different elements of intuitive-analytic thinking styles and that these elements differentially predict different outcomes in ways that can inform theories about why, exactly, intuitive-analytic thinking dispositions have consequences for our beliefs and behaviors. For example, there was a clear distinction in the predictors of paranormal versus religious belief. Whereas AOT was a particularly robust predictor of religious belief, PIT was a relatively more robust predictor of paranormal belief. Previous work using behavioral measures of intuitive-analytic thinking found similar correlations for religious and paranormal belief (Pennycook et al., 2012), which was taken to imply that the underlying mechanism for each was similar. This is of particular consequence because earlier explanations of why intuitive responding on the CRT was correlated with religious belief argued that religious beliefs evolved to be particularly intuitive (Shenhav et al., 2012). In conflict with this, our data indicate that intuitiveness (PIT) is actually more strongly correlated with paranormal belief than religious belief, whereas questioning one’s prior beliefs by engaging with evidence (AOT) is relatively more important for religious belief. We also found that AOT was more strongly associated with prior categorical changes in religious belief than PIT.
In further contrast, CMT was associated positively with religious belief but negatively with paranormal belief in our regression models, adding to the literature on critical differences between these related forms of belief (Aarnio & Lindeman, 2007).
Another clear contrast in the 4-CTSQ is the distinction between the PIT versus PET. Our validation studies support earlier work on the Rational-Experiential Inventory in which “Faith in Intuition” and “Need for Cognition” emerged as separate thinking style factors (Epstein et al., 1996; Pacini & Epstein, 1999). Indeed, our PIT items came largely (four out of six items) from the Faith in Intuition measure and our PET items came entirely from the Need for Cognition measure. Still, our measure has strong psychometric properties and predictive validity despite only consisting of six items for each subscale.^ 3 ^
Interestingly, the Need for Cognition scale is likely the most commonly used thinking style scale. Despite this, the PET scale (which is parallel to Need for Cognition) had generally weaker predictive validity than the other scales in our studies, and most starkly the AOT scale. This resonates with the developments in the literature around dual-process theory that were discussed in the Having a specific tendency to question one’s intuitions and evaluate new evidence is apparently more impactful than the mere preference for thinking in an effortful way (at least in terms of the outcomes we focus on here). For example, in Study 1, AOT was strongly associated with skepticism about religious,^ 4 ^ paranormal, and conspiratorial claims as well as lower bullshit receptivity, but PET did not significantly predict these factors in the regression model. Similarly, in Study 2, PET did not predict vaccination attitudes or intentions (and was generally a weaker predictor of COVID misperceptions and fake news discernment). Although having a PET may be important for some things (and, in fact, it did consistently correlate with decision-making performance in the Supplemental Study 3 and was most strongly associated with subjective happiness), it is evidently not characteristic of critical thinking and skepticism. Therefore, studies that rely on a single-dimension measure to index thinking styles, such as the Need for Cognition scale, may fail to detect cases where intuitive-analytic thinking styles are playing an important role.
There are some limitations of this work that would benefit from further investigation. Most notably, our samples are focused only on U.S. and Canadian participants. Although there has been working on thinking styles across the globe using past measures (Bahçekapili & Yilmaz, 2017; Browne et al., 2014; Lindeman et al., 2019; Stagnaro et al., 2019; Yilmaz & Saribay, 2016), the present research needs to be validated cross-culturally. It would be informative, in particular, to administer the 4-CTSQ to an Eastern sample as Asian cultures tend to be more collectivistic compared with Western individualistic Americans (Nisbett et al., 2001). For example, an Asian sample may score lower on the close-minded and intuitive thinking subscales as they assess thinking based on personal feelings and maintaining personal (individual) beliefs and convictions. In addition, there is some evidence that Western cultures tend to think linearly, whereas Easterners think more dialectically (Nisbett et al., 2001). Since dialectical thinking assesses how an individual interacts with contradictions to form a comprehensive opinion, Eastern samples may score higher on actively open-minded and effortful thinking and lower on close-minded and intuitive thinking.
Another limitation of the 4-CTSQ comes from the way that it was created. Our goal was to add breadth to the way that intuitive-analytic thinking styles are assessed and found four consistent factors that, apparently, have strong predictive validity. However, each subscale only consists of six items and each item is in the same “direction” (i.e., either positively or negatively worded, depending on the name of the scale). Thus, the subscales do not provide much depth in measurement for the underlying types of thinking styles. Future studies that expand on the four factors discovered here may find that they actually consist of sub-factors themselves. Our research indicates that intuitive-analytic thinking styles are not as unidimensional as past research would lead one to believe. Future work will hopefully clarify this potential diversity in thinking styles.
Also related to our scale validation, we initially selected items for further validation based how strongly they correlated with the CRT (see Online Appendix B), a broadly used measure to index the propensity to override an incorrect intuitive response. This selection criterion limits the use of the scale to conceptualizations of dual-process theory that align with the CRT. As our data highlight, there are good reasons to believe that people vary across several types of thinking styles. Our 4-CTSQ scale cannot reasonably capture every underlying thinking style. Rather, the 4-CTSQ can be used to capture multiple distinct thinking styles, unlike commonly used measures that are limited to one domain.
Although the outcome measures used to validate the 4-CTSQ were selected based on theoretical relevance indicated by prior research, we expect that there are more outcomes that may differently relate to each subscale that were not included in our study. Furthermore, our studies included self-report measures which are subject to biases, despite our best efforts to reduce them. Future studies could use behavioral outcome measures to further support the predictive validity of the 4-CTSQ.
Finally, further work is required to more systematically investigate different types of thinking styles. Although we included a relatively large number of unique measures, the literature on thinking styles is vast and there may be many areas of research where some aspects of intuitive-analytic thinking styles just are not relevant. For example, in financial decision-making, it may be that the mere PET is more important than taking an actively open-minded stance to evidence. Or, in terms of creativity where prior work has shown that people who are more analytic (in general terms) tend to be better at complex creative tasks (Barr et al., 2015), it may be that a combination of low CMT and high PIT are the most critical. Future work may also help clarify whether factors like AOT and CMT should be considered “thinking” styles, or rather if they are better thought of as values or tendencies that relate to how information is approached. Past research on thinking styles has approached this issue with a relatively inclusive definition of what is considered “thinking” and subsequent scale validation may help add further conceptual clarity to the field.
In the present research, we employed a scale that distinguishes between four correlated but independent intuitive-analytic thinking styles. The data indicate that some different types of thinking styles are relevant for some different outcomes and therefore that classifying people as “intuitive” or “reflective” (absent further specification) is an oversimplification. Furthermore, using single-dimension measures, such as the Need for Cognition scale or the CRT, as a proxy for intuitive-analytic thinking styles may lead to incorrect conclusions or, at the very least, an underestimation of the importance of intuitive-analytic thinking styles. Such investigations would benefit theoretically from taking other types of thinking styles into account such as with the 4-Component Thinking Styles Questionnaire.
Strongly disagree (1), disagree (2), somewhat disagree (3), somewhat agree (4), agree (5), strongly agree (6).
Calculate a mean score for each subscale. The items indicated by (R) are reversed scored.
Actively Open-minded Thinking (Items 1-6), Close-Minded Thinking (Items 7-12), Preference for Intuitive Thinking (Items 13-18), Preference for Effortful Thinking (Items 19-24)
Please indicate the extent to which you agree or disagree with the following statements.