Authors: Matthew W. Southward (1Department of Psychology, The Ohio State University), Madeline L. Kushner (2Department of Psychology, University of Kentucky), Nicole E. Stumpp (2Department of Psychology, University of Kentucky), Sarah E. Cecil (2Department of Psychology, University of Kentucky), Caden J. Maynard (2Department of Psychology, University of Kentucky), Alexandra K. Barnhill (2Department of Psychology, University of Kentucky), Vincent J. Buchenberger (2Department of Psychology, University of Kentucky), Shannon Sauer-Zavala (2Department of Psychology, University of Kentucky)
Categories: Article, personality, self-report, repeated measures, reliability, validity
Source: Psychological assessment
Doi: 10.1037/pas0001440
Authors: Matthew W. Southward, Madeline L. Kushner, Nicole E. Stumpp, Sarah E. Cecil, Caden J. Maynard, Alexandra K. Barnhill, Vincent J. Buchenberger, Shannon Sauer-Zavala
Because most validated personality measures were designed to capture relatively general and stable characteristics, they do not specify a particular timeframe for respondents to consider. It is thus unknown how these measures perform when administered repeatedly or how this performance compares to the same measures with instructions and items adapted to the repeated timeframe of interest. We randomly assigned undergraduate participants (N = 257; Mage = 20.4; 79% female; 77% white; 77% heterosexual) to complete measures of personality (NEO-FFI-3, LPFS-BF-2.0, PID-5-BF, FFBI-SF) with validated general instructions and items or measures with instructions and items pertaining to the previous week once per week for six weeks. Compared to measures with general instructions, measures with weekly instructions demonstrated greater within-person internal consistency (weekly ωs: .42-.83; general ωs: .44-.72), lower rank-order stability (weekly average one-week r: .72; general average one-week r: .86), greater variability (ds: .08-.94), lower average mean scores across time (ds: –.96 – .25), and stronger associations with measures of anxiety and depression, well-being, and functioning but similar between-person internal consistencies (weekly ωs: .79-.99; general ωs: .79-.99) and measurement invariance. Researchers assessing personality weekly may thus be able to capture more variability and stronger associations with relevant constructs while still maintaining reliable individual differences and construct validity using personality measures referencing participants’ past week. However, nuances such as lower average scores when referencing the past week should be kept in mind when comparing results between studies using different reference time frames.
Personality, defined as a person’s characteristic way of thinking, feeling, and behaving, can be assessed in many ways, including self-report, observer ratings, and open-ended responses (Funder, 2010; Robins et al., 2007). Self-report is the most widely used way of measuring personality, as these measures are practical and readily interpreted (Paulhus & Vazire, 2007). However, the validity of self-reported personality measures can vary based on contextual factors (e.g., the time allotted to complete a measure), individual differences (e.g., motivation or self-awareness to complete a measure accurately), and aspects of the measures themselves.
Although some researchers have explored the effects of time frames on self-report broadly, and psychological symptoms specifically, few have examined how instructed time frames impact personality measures. Most well-validated measures of personality ask how people are generally (Goldberg et al., 2006) or do not specify a time frame (Ben-Porath & Tellegen, 2020; Gosling et al., 2003; Krueger et al., 2013; McRae & Costa, 2010; Morey, 2007; Soto & John, 2017). Assessing personality this way is in line with the common understanding of personality as a fairly stable trait especially over relatively longer time spans. At the same time, many researchers acknowledge that personality may also be more state-like and malleable in shorter time frames or specific contexts. Beck and Jackson (2020), for instance, described how personality can develop as a dynamic, within-person system in the context of environmental pressures that may more specifically point to how personality influences particular behaviors. To then link these dynamic patterns back to individual differences, Fleeson (2001) showed how aggregating across repeated responses on personality measures adapted to refer to relatively brief time frames can accurately represent individual differences in personality traits.
However, it may still be important to use trait measures when assessing personality over a relatively brief period. Psychometrically, researchers may wish to assess the test-retest reliability of a measure (e.g., Soto & John, 2017) or enhance the precision of their estimates of participants’ trait scores through repeated measures. These issues of measurement, specifically at a weekly level, are important to research on interventions for personality. In their systematic review of personality change during interventions using general, trait-based personality measures, Roberts et al. (2017) noted that personality changes followed an exponential trajectory, with nearly all small-to-medium sized change occurring in 4–8 weeks. Similarly, across studies of individual personality change interventions using general, trait-based measures administered weekly, participants reported significant improvements in neuroticism and extraversion, with more mixed results for conscientiousness, agreeableness, and openness among community participants, participants with substance use disorders, and participants with borderline personality disorder (Hudson & Fraley, 2015; Sauer-Zavala et al., 2023a; Stieger et al., 2021, 2022).
By contrast, when administering measures adapted to refer only to the prior week (or month) once per week (or month) in two- and four-week studies of digital interventions to improve conscientiousness and openness to experience and group-based interventions to improve flirting behaviors, community participants reported significant improvements in each personality dimension (Allemand et al., 2022; Stieger et al., 2020). Personality changes varied widely from small- to large-sized effects depending on how directly each dimension was targeted and how narrow the dimension was, with more specific dimensions showing larger changes.
The use of personality measures written to capture general, trait-like experiences and more time-limited, state-like experiences makes it difficult to compare these results. For instance, if participants responding to trait-like measures are using peak-end heuristics to inform their reports, they may be less consistent from one timepoint to the next than those using more state-like measures who may be more likely to rely on episodic memory. There may be further differences in mean scores and variability since trait-like measures encompass a larger set of experiences than state-like measures. Finally, although state-like measures can be aggregated into general trait scores, it is unclear if these measures are capturing the same latent constructs.
Understanding how the framing of measures influences self-reported personality traits over weekly time scales can improve our knowledge of personality assessment reliability and validity, since measures with high reliability ensure that observed changes in individual scores can be attributed to genuine shifts in personality, rather than fluctuating questionnaire responses. If people do respond differently based on the frame of reference given, it suggests that the item phrasing may play a vital role in how they interpret and report their personality characteristics. In essence, gaining insight into how time affects personality self-reporting ultimately allows researchers to refine existing assessment tools to most adequately capture changes in personality.
In the current study, we compared the psychometric properties of four commonly used personality measures administered once per week for up to six weeks with instructions that either asked participants to complete the measure with reference to the prior week or did not ask participants to complete the measure with a specific time period in mind. These personality measures were selected to cover normal personality functioning, Criteria A and B of the DSM-5’s Alternative Model for Personality Disorders (American Psychiatric Association [APA], 2022), and borderline personality disorder, given its relevance to both personality and psychopathology more broadly (Ringwald et al., 2023; Southward et al., 2023). We randomly assigned participants to complete personality measures with instructions to consider the prior week or consider their personality in general. We compared each of these instructions conditions on the between- and within-person internal consistencies of the items in each personality measure (Aim 1), the multilevel measurement invariance of each personality measure (Aim 2), the rank-order stability of each personality measure (Aim 3), the person-specific standard deviations of each personality measure (Aim 4), and the person-specific means of each personality measure (Aim 5). To better contextualize and benchmark these results, we compared the psychometric properties of these personality measures with a set of commonly used mood, well-being, and functioning measures that all asked participants to consider the prior week. Because these measures were delivered after the personality measures, we were able to explore any downstream impacts of instruction condition on these measures. Finally, we explored if instruction condition moderated the strength of the within-person associations among personality measures and mood, well-being, and functioning measures (Aim 6). To our knowledge, this is the only study to date to directly compare the psychometric properties of repeatedly administered personality measures with the same content but instructions focusing on different timeframes.
Participants were 289 undergraduate students recruited from the University of Kentucky’s psychology department recruitment pool who participated for class credit. Of these 289 participants, we excluded 31 due to suspected careless responses defined by any combination of 0 variance on the 60-item NEO-Five Factor Inventory-3 (NEO-FFI-3; McCrae & Costa, 2010), completing the 25-item Personality Inventory for DSM-5-Brief Form (PID-5-BF; Krueger et al., 2013) in < 1 minute, completing the 48-item Five Factor Borderline Inventory-Short Form (FFBI-SF; DeShong et al., 2016) in 1 minute or less, or completing two or more weekly surveys out of order and in rapid succession before the end of semester deadline. When participants only completed one survey out of order (k = 9) or less than two days apart (k = 6), we removed those specific surveys from our analyses but retained all others. Finally, one participant was excluded from analyses because of a randomization failure (i.e., receiving established instructions surveys each week despite being randomized at baseline to the weekly instructions condition).
Of the 257 remaining participants, 227 (88.3%) enrolled in the study by selecting it from a list of study options. The remaining 30 (11.7%) participants were invited to participate via email because they reported elevated neuroticism, antagonism (i.e., low agreeableness), or disinhibition (i.e., low conscientiousness) at the pre-screening stage to oversample clinical presentations. Elevations were defined as scores >1 SD above the normed means from the NEO-FFI-3 on one or more of the three personality dimensions above (McCrae & Costa, 2010). To note, elevations in these areas were not required for participation.
Participants were 20.37 years old (SD = 4.98) on average, and a majority identified as women (n = 204; 79.4%), followed by men (n = 47; 18.3%) and non-binary (n = 5; 1.9%). Most participants identified as white (n = 198; 77.0%), followed by African-American (n = 29; 11.3%), Latinx (n = 11; 4.3%), East Asian (n = 9; 3.5%), South Asian (n = 5; 1.9%), Native American (n = 3; 1.2%), Arab/Middle Eastern American (n = 2; 0.8%), and Native Hawaiian/Pacific Islander (n = 1; 0.4%). Most participants identified as heterosexual (n = 199; 77.4%), followed by bisexual (n = 23; 9.0%), asexual (n = 11; 4.3%), homosexual (n = 8; 3.1%), queer (n = 8; 3.1%), and pansexual (n = 8; 3.1%). Smaller proportions identified as questioning (n = 6; 2.3%), demisexual (n = 2; 1.0%), fluid (n = 1; 0.4%) and autosexual (n = 1; 0.4%). A majority (n = 181; 70.7%) of participants reported being single. Participants reported a median family income of $100,000–124,999 and a median education level of some college.
All study procedures were approved by the University of Kentucky Institutional Review Board. Participants first provided informed consent, and then completed a baseline survey, which contained a battery of self-report questionnaires, including several personality, mood, and well-being/functioning measures. Participants were randomized (1:1) to either the established instructions condition (n = 132; 51.4%), or the weekly instructions condition (n = 125; 48.6%). Participants in the established instructions condition completed each measure according to its established instructions. Those in the weekly instructions condition received the same measures, although the personality measures included a modified set of instructions and items written to pertain to the past week, detailed below. Each week for six more weeks, all participants received an email with a link to complete the same set of personality, mood, and well-being/functioning measures as they completed at baseline. These measures were delivered in the same order at each occasion. Participants received a maximum of five automated reminder emails spaced eight hours apart if they did not complete the weekly survey within the eight-hour window. Following the sixth week, all participants received a debriefing form describing the study and were compensated with course credit prorated based on the number of weekly surveys completed.
The weekly instructions and items of the personality measures were written to modify each measure as minimally as possible while also making it clear participants should respond considering the past week of their experiences. For instance, the last line of the established instructions for the PID-5-BF reads “We’d like you to take your time and read each statement carefully, selecting the response that best describes you” which we modified to “We’d like you to take your time and read each statement carefully, selecting the response that best describes you IN THE PAST WEEK” in the weekly instructions condition. Because we were concerned that participants would not closely attend to these instructions, we further modified each item to specify a past-week time period. For instance, the first PID-5-BF item is “People would describe me as reckless” which we modified to “In the past week, people would describe me as reckless.” We retained the response options for all measures.
We did not modify the instructions of the mood or well-being measures for any participants, because these measures were already written with instructions asking participants to focus on their experiences over the past week. To align our measure of functioning, the Work and Social Functioning Scale (WSAS; Mundt et al., 2002), with these three measures, we did modify its instructions to include the phrase “IN THE PAST WEEK” as well. We included these measures after the personality measures for two reasons. First, we wanted to evaluate the degree to which participants responded differently to commonly administered measures of past-week symptoms, well-being, and functioning after completing measures of either general or past-week personality. Second, we wanted to confirm that participants’ responses to the personality measures in the weekly instructions condition were more similar to their responses on these mood, well-being, and functioning scales than participants’ responses in the established instructions condition were.
In the established instructions condition, 54 participants (40.9%) completed measures at all six weeks with an average completion of 4.77 weeks. In the weekly condition, 35 participants (28.0%) completed measures at all six weeks with an average completion of 4.38 weeks. The average number of weekly responses completed did not significantly differ between conditions, t(255) = 1.34, p = .18, 95% CI [–.18, .94].
Participants first reported on their demographic characteristics. These characteristics included age, gender identity, racial/ethnic identity, estimated family income, education, marital status, and sexual identity.
The Five-Factor Borderline Inventory Short Form (FFBI-SF; DeShong et al., 2016) is a 48-item self-report questionnaire designed to measure maladaptive personality traits related to BPD from the perspective of the Five Factor Model. The FFBI-SF consists of 12 4-item subscales representing facets of neuroticism (i.e., anxious uncertainty, dysregulated anger, despondence, self-disturbance, behavioral dysregulation, affective dysregulation, and fragility), low agreeableness (i.e., manipulativeness, distrustfulness, and oppositionality), low conscientiousness (i.e., rashness), and openness (i.e., dissociative tendencies). Each item is rated on a 5-point Likert-type scale from 1 (disagree strongly) to 5 (agree strongly) and summed to create each subscale score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The Level of Personality Functioning Scale-Brief Form-2.0 (LPFS-BF-2.0; Weekers et al., 2019) is a 12-item self-report questionnaire composed of two 6-item subscales designed to assess difficulties in self and interpersonal functioning as defined by Criterion A of the AMPD (APA, 2022). Each item is rated on a 4-point Likert-type scale from 1 (completely untrue) to 4 (completely true) and summed to create each subscale score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The NEO Five Factor Inventory 3 (NEO-FFI-3; McCrae & Costa, 2010) is a 60-item self-report measure with five 12-item subscales designed to assess each of the Big Five personality neuroticism, extraversion, conscientiousness, agreeableness, and openness. Each item is rated on a 5-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree) and summed to create each subscale score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The Personality Inventory for DSM-5-Brief Form (PID-5-BF; Krueger et al., 2013) is a 25-item self-report measure with five 5-item subscales intended to assess each of the maladaptive personality traits included in the Alternative Model of Personality Disorders (AMPD; APA, 2022): negative affectivity, detachment, antagonism, disinhibition, and psychoticism. Items are rated using a 4-point Likert-type scale from 0 (very false or often false) to 3 (very true or often true) and summed to create each subscale score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The Overall Anxiety Severity and Impairment Scale (OASIS; Norman et al., 2006) is a 5-item self-report questionnaire used to assess anxiety severity over the previous week. Each item is rated on a 5-point Likert-type scale using unique anchors for each item from 0 (none) to 4 (extreme) and summed to create a total score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The Overall Depression Severity and Impairment Scale (ODSIS; Bentley et al., 2014) is a 5-item self-report measure used to assess depression severity over the previous week. Each item is rated on a 5-point Likert-type scale using unique anchors for each item from 0 (none) to 4 (extreme) and summed to create a total score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The Quality of Life Enjoyment and Satisfaction Questionnaire – Short Form (Q-LES-Q-SF; Endicott, 1993) is a 14-item self-report questionnaire intended to measure satisfaction with a range of physical, emotional, social, and functional life domains. Each item is rated from 1 (very poor) to 5 (very good) and summed to create a total score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
The Work and Social Adjustment Scale (WSAS; Marks, 1986) is a 5-item self-report measure of impairment in functioning across five life domains due to mental health difficulties over the previous week. Each item is rated on a 9-point Likert-type scale from 0 (not at all) to 8 (very severely) and summed to create a total score. Means and internal consistencies for each subscale in each randomization condition are presented in the Results.
To explore if differences in the timeframe specified in the instructions and items led to differences in between- or within-person internal consistencies in personality measures (Aim 1), we first calculated between- and within-person McDonald’s ωs for all measures using Geldolf et al.’s (2014) recommendations. Specifically, we fit a multilevel confirmatory factor analysis (CFA) for each measure separately in Mplus (Version 8.11; Muthén & Muthén, 2012–2022), applying full information maximum likelihood (FIML) estimation to model data missingness. We applied a correlated factors model for each measure, defining each factor as an established subscale from each measure and allowing each item to load on its established subscale. We retained the same correlated factors structure at the between- and within-person levels for each measure. We then applied standard equations to calculate ωs at the between- (b) and within-persons (w) levels, ωb/w=(Σλbj/wj)2(Σλbj/wj)2+Σθbj/wj. Between-person ωs indicate the degree to which items represent a unitary construct to different people, whereas within-person ωs indicate the degree to which items represent a unitary construct to any given person over time. Finally, we used the cocron package (Version 1.0–1; Diedenhofen, 2016) in R (Version 4.3.1; R Core Team, 2023) to compare ωs at each level between conditions.^1^ Because the test statistic to compare ωs is approximately distributed as χ^2^ when the null hypothesis is true, with df = 1 and assuming α = .05, we had 80% power to detect χ^2^ ≥ 3.84 with λ = 7.85 (Faul et al., 2009).
We then conducted preliminary analyses to test for any differences in demographic variables between randomization conditions, zero-order correlations among all measures of interest, and one-week rank-order stability of all measures of interest (Aim 2). We tested for differences in demographics between randomization conditions using independent samples t-tests for continuous variables (i.e., age, national subjective social status, community subjective social status), χ^2^ tests of independence for categorical variables (i.e., gender, racial/ethnic identity, sexual identity, marital status), and Mann-Whitney U tests for ordinal variables (i.e., income, education) in SPSS (Version 29.0.2.0; IBM Corp., 2023). We then calculated Pearson correlations among continuous and dichotomous variables and Spearman’s correlations involving ordinal variables in SPSS. We included a dummy-coded randomization condition variable in these correlations as well as the person-specific personality, mood, and well-being/functioning variables described below. To test the one-week rank-order stability of each measure, we calculated Pearson correlations between all six pairs of successive timepoints for each measure within each randomization condition and calculated an average of those week-to-week stability statistics, applying pairwise deletion in the presence of missing data. We compared the average one-week rank-order stability using Preacher’s (2002) implementation of Cohen and Cohen’s (1983, p. 54) formula 2.8.5 to compare two independent correlations. Conducting two-tailed tests with n1 = 132, n2 = 125, and assuming α = .05, we had 80% power to detect between-group differences in continuous variables of d ≥ .35 (Faul et al., 2009). With df = 1 for tests of independence for categorical variables and assuming α = .05, we had 80% power to detect χ^2^ ≥ 3.84 with λ = 7.85 (Faul et al., 2009). Finally, with 257 participants and assuming α = .05 with two-tailed tests, we had 80% power to detect rs and ρs ≥ .17 (Faul et al., 2009).
To test if the latent constructs producing observed scores on measures with established instructions differed from those producing observed scores on measures with weekly instructions (Aim 3), we tested the measurement invariance of each subscale using two-level multi-group CFAs in Mplus, applying full information maximum likelihood (FIML) estimation to model data missingness. We first evaluated scalar equivalence for each subscale by fixing the factor structure, loadings, and means to be the same between the established instructions group and the weekly instructions group at both between- and within-person levels (restricted model; Figure S1a-c). We then freed the within-person loadings and between-person means to evaluate metric equivalence (semi-restricted model) and, finally, freed both the within- and between-person factor loadings to evaluate configural equivalence (unrestricted model). We evaluated model fit using Hu and Bentler’s (1999) standards (i.e., good RMSEA ≤ .06, CFI/TLI ≥ .95, SRMR ≤ .06; acceptable RMSEA ≤ .08, CFI/TLI ≥ .90, SRMR ≤ .08) To compare each model, we conducted χ^2^ difference tests using the Satorra-Bentler scaled χ^2^ to adjust for the continuous non-normal distributions of these models (Satorra & Bentler, 2010). Given the number of tests conducted, we applied the Benjamini-Hochberg procedure to each set of comparisons separately (i.e., restricted to semi-restricted models; restricted to unrestricted models).
To test if differences in the timeframe specified in the instructions and items led to differences in the standard deviations (Aim 4) and means (Aim 5) of personality measures, we conducted a series of multiple regressions. First, we calculated person-specific standard deviations and means for each personality subscale using all available data provided by each participant. To better isolate the impact of the timeframe specified in the instructions, for Aim 4 we regressed these person-specific standard deviations on a dummy-coded randomization condition variable and several covariates, applying listwise deletion in the presence of missing data. We included person-specific means of the respective personality measure as a covariate to account for individual differences in the personality dimension tested. We included the total number of weekly responses completed as a covariate to account for any artificially elevated standard deviations driven by participants who completed fewer but more extreme weekly responses. Finally, we included demographic characteristics (i.e., age, gender, racial/ethnic identity, sexual identity, income, education, national subjective social status, and local subjective social status) as covariates to account for any existing associations with personality dimensions.
We repeated these analyses to test Aim 5 but replaced person-specific standard deviations with person-specific means of each scale in each model. To compare if any between-condition differences in personality measures influenced subsequent mood, well-being, or functioning measures, we repeated the above multiple regression analyses with each of our mood, well-being, and functioning measures. Assuming α = .05, with 257 participants and 13 predictors in each multiple regression model, each evaluated as a two-tailed test, we had 80% power to detect individual predictors explaining R^2^ ≥ .029 (Faul et al., 2009). Given the number of tests conducted in Aims 2 and 3, we applied the Benjamini-Hochberg procedure (Benjamini & Hochberg, 1995) to better control our Type 1 error rate using α = .05 for each aim.
Finally, to explore between-condition differences in within-person effects (Aim 6), we first disaggregated within- from between-person variability for all personality measures following Curran and Bauer’s (2011) recommendations. We calculated the person-mean for all personality measures and subtracted this person-mean from the raw score at each timepoint. We then ran a series of HLMs using proc mixed in SAS (Version 9.4) to regress each mood, well-being, and functioning measures, in separate models, on within-person personality measure scores, a dummy-coded instructions condition variable (0 = established instructions, 1 = weekly instructions), and the product of the within-person personality measure scores and the dummy-coded instructions condition variable. We applied restricted maximum likelihood estimation, modeling random intercepts, an autoregressive lag-1 residual covariance structure, and used the Kenward-Rogers method to calculate degrees of freedom. With N = 257, we had 80% power to detect effects with ts ≥ 2.82 (Murayama et al., 2022). Given the number of tests, we again applied the Benjamini-Hochberg procedure to the family of models for each mood, well-being, and functioning measure, using α = .05 for each measure.
This study was not preregistered. However, we report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. Code for all analyses is available at https://doi.org/10.17605/osf.io/gznav. Data is available upon reasonable request from the first or last author.
Between-persons, items from all measures in the established instructions condition but one (FFBI-SF-A4-Oppositional; ω = .79) exhibited good-to-excellent internal consistencies, ωs: .80-.99 (Table 1). Similarly, items from all measures in the weekly instructions condition but one (NEO-FFI-3-Openness; ω = .79) exhibited good-to-excellent internal consistencies, ωs: .80-.99. The reliabilities of personality measures were generally similar between conditions, with those in the weekly instructions condition exhibiting slightly greater ωs on average, ω¯ = .93, than those in the established instructions condition, ω¯ = .90. Only four scales exhibited significantly different ωs between NEO-FFI-3-Agreeableness, FFBI-SF-A2-Manipulativeness, FFBI-SF-A4-Oppositional, and FFBI-SF-N6a-Affective Dysregulation, ps < .01. In each case, those in the weekly instructions condition demonstrated significantly greater ωs than those in the established instructions condition. Of note, there were no significant between-condition differences or consistent between-condition patterns distinguishing the internal consistencies of items from the mood, well-being, and functioning measures, ps > .05.
Within-persons, only items from one personality scale in the established instructions condition, FFBI-SF-O1-Dissociative Tendencies, exhibited acceptable internal consistency, ω = .72; items from all other scales were in the unacceptable-to-questionable range, ωs: .35-.60. By contrast, items from 12 personality scales in the weekly instructions condition exhibited acceptable-to-good internal consistency, ωs: .73-.83, with the remainder in the unacceptable-to-questionable range, ωs: .42-.69. Over half of the personality scales in the weekly instructions condition demonstrated significantly greater ωs than scales in the established instructions condition, ps < .03, reflected in the greater average ω in the weekly instructions condition, ω¯ = .69, compared to the established instructions condition, ω¯ = .50. As in the between-persons results, there were no significant differences or consistent patterns distinguishing the randomization conditions on any mood, well-being, or functioning measure, ps > .25.
All personality measures demonstrated relatively high average one-week rank-order stability, r¯s: .59-.93 (Table 2). When comparing instruction conditions, 18 of the 24 personality measures (75%) demonstrated greater one-week rank-order stability when using established instructions compared to weekly instructions, zs > 2.30, ps < .03 (Tables 2, S3). By contrast, of the mood, quality of life, and functioning measures, only the WSAS demonstrated a significant between-condition difference, z = 2.31, p = .02.
Across indices, all PID-5-BF subscales except Psychoticism, both LPFS-BF-2.0 subscales, and the FFBI-SF subscales of N5-Behavioral Dysregulation, N6a-Affective Dysregulation, A4-Oppositional, and O1-Dissociative Tendencies demonstrated acceptable-to-good fit when all parameters were set equal across groups (Table 3). All measures except PID-5-BF-Psychoticism, FFBI-SF-N3-Despondence, and FFBI-SF-N4-Self-Disturbance demonstrated at least acceptable fit by RMSEA, although fewer measures fit acceptably using CFI, TLI, and SRMR indices.^2^ However, the vast majority of these subscales (n = 15; 78.9%) demonstrated scalar equivalence compared to the semi-restricted models and a similar percentage (n = 15; 71.4%) demonstrated scalar equivalence compared to the unrestricted models.
There were no significant differences in demographic characteristics between randomization conditions, ps > .15, except community subjective social status. At baseline, participants in the established instructions condition reported higher community subjective social status (M = 5.44, SD = 1.82) than those in the weekly instructions condition (M = 4.94, SD = 1.77), t(254) = 2.19, p = .03, 95% CI [.05, .93], d = .27. Although primarily small-to-medium sized, certain demographic characteristics (i.e., sexual minority identity, lower national and community subjective social status, and psychotropic medication use) were significantly associated with greater mean scores on neuroticism-related measures and lower mean scores on extraversion-, conscientiousness-, and agreeableness-related measures (Table S2). By contrast, there were no consistent associations between demographic characteristics and standard deviations of our measures of interest (Table S1). A similar proportion of participants reported clinically elevated past-week depression at baseline in the established instructions condition (n = 31; 23.8%) as in the weekly instructions condition (n = 36; 29.0%; χ^2^(1) = .88, p = .35). A similar proportion of participants also reported clinically elevated past-week anxiety in the established instructions condition (n = 60; 45.8%) as in the weekly instructions condition (n = 62; 49.6%; χ^2^(1) = .37, p = .54).
After applying the Benjamini-Hochberg procedure, we found significantly greater person-specific standard deviations in the weekly instructions condition than the established instructions condition on all personality scales except PID-5-BF-Antagonism, ds: .08-.94, ps < .04 (Tables 4, S4-S31).^3^ These differences tended to be most pronounced among neuroticism-focused scales (i.e., NEO-FFI-Neuroticism, PID-5-BF-Negative Affectivity, FFBI-SF-N1-Anxious Uncertainty, FFBI-SF-N3-Despondence, FFBI-SF-N5-Behavioral Dysregulation, FFBI-SF-N6a-Affective Dysregulation; ds: .43-.84), extraversion-focused scales (i.e., NEO-FFI-Extraversion, PID-5-BF-Detachment; ds: .83-.93), and conscientiousness-focused scales (i.e., NEO-FFI-Conscientiousness, PID-5-BF-Disinhibition, FFBI-SF-C6-Rashness; ds: .38-.94), as well as general personality functioning, ds: .53-.69. Although generally in the same direction, only Q-LES-Q-SF scores exhibited significantly greater standard deviations in the weekly instructions condition compared to the established instructions condition, d = .29, p = .02.
By contrast, 11 personality scales exhibited significant mean differences between randomization conditions after applying the Benjamini-Hochberg procedures, ds: –.96 - –.21, ps < .02 (Tables 5, S32-S59).^4^ In each of these cases, participants in the established instructions condition tended to exhibit higher mean scores than those in the weekly instructions condition. There were fewer consistent patterns in these between-condition differences with some neuroticism-focused scales (i.e., PID-5-BF-Negative Affectivity, FFBI-SF-N2-Dysregulated Anger, FFBI-SF-N3-Despondence, FFBI-SF-N4-Self-Disturbance, FFBI-SF-N5-Behavioral Dysregulation; ds: –.49 - –.35), some agreeableness-focused scales (i.e., FFBI-SF-A1-Distrustfulness, FFBI-SF-A2-Manipulativeness, FFBI-SF-A4-Oppositional; ds: –.71 - –.21), and individual conscientiousness-focused (FFBI-SF-C6-Rashness; d = –.44), openness-focused (i.e., NEO-FFI-Openness; d = –.96) and personality functioning scales (LPFS-BF-2.0-Other-Dysfunction; d = –.31) demonstrating significant between-condition differences. There were no significant mean-level between-condition differences on any mood, well-being, or functioning scale, ds: <–.01-.10, ps > .20.
Because we used a 0/1 dummy-coded instructions condition variable, the conditional main effect of each personality measure represents the strength of the average within-person association at any given timepoint between the personality measure and mood, well-being, or functioning outcome in the established instructions condition. When this term is significant, it thus indicates that reporting greater or lower personality scores than participants’ personal average using the established instructions was significantly associated with greater or lower mood, well-being, or functioning at the same timepoint. The product term then represents the difference in the strength of the average within-person association at any given timepoint between the established instructions condition and the weekly instructions condition. When this term is significant, it indicates that participants in the weekly instructions condition demonstrated a significantly more positive or more negative association between within-person changes in personality measures and mood, well-being, or functioning measures at a given timepoint.
Of the 48 effects of interest (two effects of interest per personality measure × 24 personality measures) for the ODSIS, 27 remained significant after applying the Benjamini-Hochberg procedure (Tables S60-S83, S156). Two-thirds (n = 18; 66.7%) of these effects were product terms representing stronger relations with within-person changes in ODSIS scores in the weekly instructions condition. Across all models, personality measures with weekly instructions were 3.48 times more strongly associated with within-person changes in ODSIS scores than personality measures with established instructions.
Of the 48 effects of interest for the OASIS, only 11 effects remained significant after applying the Benjamini-Hochberg procedure (Tables S84-S107, S157). However, nine of these effects (81.8%) were product terms representing stronger relations in the expected directions with within-person changes in OASIS scores in the weekly instructions condition. Measures with weekly instructions were 2.96 times more strongly associated with within-person changes in OASIS scores than personality measures with established instructions across all models.
Of the 18 significant effects remaining after applying the Benjamini-Hochberg procedure to the WSAS models, 11 (61.1%) were product terms representing stronger relations in the expected directions with within-person changes in WSAS scores in the weekly instructions condition (Tables S108-S131, S158). Measures with weekly instructions were 1.95 times more strongly associated with within-person changes in WSAS scores than personality measures with established instructions across all models.
Finally, of the 23 significant effects remaining after applying the Benjamini-Hochberg procedure to the Q-LES-Q-SF models, 17 (73.9%) were product terms representing stronger within-person relations in the weekly instructions condition (Tables S132-S155, S159). Measures with weekly instructions were 2.25 times more strongly associated with within-person changes in Q-LES-Q scores than personality measures with established instructions across all models.
In this study, we evaluated the impact of instructing people to reflect on their personality over a specified time period compared to no specific time period on indices of internal consistency, rank-order stability, measurement invariance, variability, and means of several common measures of personality administered once per week for six weeks as well as associations between within-person changes in these measures and measures of symptoms, well-being, and functioning. Although the between-person internal consistencies of the personality measures were good-to-excellent and generally similar between conditions, the within-person internal consistencies were significantly higher in the weekly instructions condition than the established instructions condition, ranging from unacceptable to good, depending on the measure. In the weekly instructions condition, the personality measures tested also tended to exhibit lower rank-order stability, and somewhat lower mean scores but significantly greater variability and stronger associations with measures of symptoms, well-being, and functioning compared to the established instructions condition. By contrast, we found almost no between-condition differences on indices of internal consistency, measurement invariance, variability, or means of measures of symptoms, well-being, and functioning with instructions to consider the past week. Together, these results can guide researchers in planning repeated measures studies of personality and comparing the results of such studies with cross-sectional findings.
Across conditions, items from all but two personality measures demonstrated good-to-excellent between-person internal consistencies, ωs ≥ .80. Although internal consistencies were slightly higher on average in the weekly instructions condition, there were only four significant between-condition differences in the internal consistencies of the personality measures. It should be noted that all the measures included in this study were initially validated as between-person measures. This psychometric property may thus translate well to repeated measures designs. Researchers designing such studies primarily with questions about individual differences may expect the internal consistencies reported in the initial validation studies of personality measures to generalize to repeated measures samples almost regardless of the instructions delivered.
By contrast, items from all measures demonstrated lower within- than between-person internal consistencies, which may be expected given the aggregation of responses that contribute to between-person internal consistencies (Epstein, 1980). However, items from personality measures in the weekly instructions condition tended to demonstrate greater within-person internal consistency than items in the established instructions condition. This discrepancy was especially pronounced for measures of or closely related to neuroticism, extraversion, conscientiousness, and general personality functioning, with somewhat less discrepancy for measures of or related to openness. Asking participants to explicitly focus on a clearly defined time period may allow them to approach items in a more similar way from one timepoint to the next compared to providing no guidance on a time period to consider. Without a clear time period on which to focus, participants in the established instructions condition may have considered their personality over different time periods at different timepoints in the study, leading to less internally consistent responses from one timepoint to the next. We encourage future researchers to test this hypothesis by collecting qualitative descriptions from participants about their thought process in responding to repeatedly assessed personality items. However, researchers designing repeated measures studies primarily with questions about within-person processes may be encouraged to adapt the instructions of their measures to focus participants on a specific time period of interest to facilitate more internally consistent results.
The fact that 70–80% of the subscales assessed demonstrated scalar equivalence further suggests that both versions of these personality measures reflect the same latent constructs, just at different timescales. Thus, researchers may be encouraged to use the version that best suits their study design and psychometric needs.
Despite the greater within-person internal consistency demonstrated in the weekly instructions condition, participants in this condition also demonstrated significantly more variable and less rank-order stable responses than those in the established instructions condition. By more consistently focusing on a relatively brief but recent time period, those in the weekly instructions condition likely responded based on episodic memories of what events occurred and how they responded over the prior week, whereas those in the established instructions condition may have responded more from semantic memory of how they saw themselves in general across different time periods (Robinson & Clore, 2002). Researchers interested in how personality changes over time may be encouraged to use measures with instructions tailored to the time period over which they are measuring personality so participants will be more likely to focus on how their personality was expressed during that specified time frame. Accumulating data suggest that personality traits change across the lifespan (Roberts et al., 2006; Roberts & Mroczek, 2008) and in response to treatment (Roberts et al., 2017). Thus, it is important understand whether existing personality measures, developed with the assumption that traits are static, can adequately capture change across different timeframes of interest.
Regardless of these differences in variability, we found some mean-level differences between instruction conditions when averaged across the study period, with those in the weekly instructions condition reporting somewhat lower mean scores on most, but not all, personality measures. These findings further suggest that those in the weekly instructions condition were relying on episodic memory rather than semantic memory in responding, because briefer time periods tend to preclude the most extreme responses people can exhibit when considering longer time periods (Peasgood et al., 2023; Walentynowicz et al., 2018). Measures tapping into broader constructs or dimensions (e.g., NEO-FFI-3, PID-5-BF), tended to demonstrate more similar means between conditions than the FFBI-SF. Measures assessing broader constructs may be more likely to include a mix of items that are more relevant to lifetime experiences (e.g., “I use people to get what I want” [PID-5-BF-Antagonism item #22]) or weekly experiences (e.g., “I craved attention” [PID-5-BF-Antagonism item #19]) whereas measures assessing more narrow constructs may include items that are all relevant to either lifetime or weekly experiences. If the primary focus is to directly compare results with previous cross-sectional results with established instructions, those measures could be used. The use of measures with weekly instructions may simply need the caveat that the resulting mean scores may differ slightly and somewhat unpredictably from measures with established instructions.
By contrast, we found no consistent differences between conditions among measures of symptoms, quality of life, or functioning when these included past-week instructions. As with the personality measures, items from these measures generally exhibited lower within- than between-person internal consistency. However, neither the between- nor within-person internal consistencies differed between groups. Only the Q-LES-Q demonstrated significantly greater variability in the weekly instructions condition, and there were no significant differences or consistent patterns in mean differences between conditions on any symptom, quality of life, or functioning measure. Thus, self-report measures designed to assess past-week experiences may perform similarly regardless of the instructions of the measures completed before them. Researchers interested in conducting repeated measures studies that include measures with instructions focusing on different time periods should not expect measures with instructions to focus on general time periods to bias responses on subsequent measures with instructions to focus on specific or more recent time periods. Of course, there are many permutations of these ordering effects that could be explored, which remain open questions for future researchers.
Finally, within-person changes in personality measures and measures of symptoms, well-being, and functioning were roughly 2–3 times more strongly associated with personality measures with weekly instructions than those with established instructions. These results suggest that asking participants to consider the same time period when responding to personality measures as well as measures of symptoms, well-being, and functioning enhances the strength of the association between these measures. Thus, researchers interested in how personality and associated constructs change together over time may be encouraged to specify the same time period for both sets of measures. However, researchers interested in the relative stability of general personality ratings compared to more rapid changes in associated constructs may expect to find fewer and typically smaller-sized associations. We encourage future researchers to use these data to plan their studies to align measurement and theory (Kaurin et al., 2023).
These findings should be interpreted in the context of our study’s limitations. We selected personality measures designed to capture normal variation in the Big Five, measures relevant to Criteria A and B of the AMPD, and borderline personality disorder. Although we selected these to cover a range of relevant personality research, it is possible that other measures of interest would perform differently in this design. Similarly, we selected comparison measures of mood, well-being, and functioning with past-week instructions, which may be expected to be more strongly associated with personality measures with past-week instructions than with general, trait-like instructions (Aim 6). We encourage future researchers to replicate our design using comparison measures of trait-like constructs (e.g., avoidance, hope optimism) to evaluate the impact of instructions on these relations. We focused on one weekly instructions compared to general instructions. Researchers conducting intensive longitudinal designs may be interested in comparisons with more momentary measures, whereas researchers conducting yearly panel studies may be more interested in measures with past-year instructions. Although beyond the scope of this design, we encourage such tests to more fully characterize how the time period of instructions impacts performance of these measures.
Because our sample was an unselected undergraduate sample and thus was primarily younger, white, and female, these results may not generalize to more diverse, non-college educated, or clinically severe samples. More diverse samples may provide greater variability in responses on these measures. We collected data across an unspecified six-week time period, so these results pertain most directly to a relatively brief span of time without any associated interventions or further experimental manipulations. We encourage intervention researchers to replicate these results in a treatment context to test their generalizability.
Given the enthusiasm around characterizing and intervening on personality change (Beck & Jackson, 2021; Roberts et al., 2017; Sauer-Zavala et al. (2023b)), it is important to use measures that are most appropriate to capture this change. Together, our results emphasize the importance of aligning the time period specified in the instructions of personality measures with researchers’ time period of interest in repeated measures designs. Researchers interested in weekly changes in personality are encouraged to adapt the instructions and items of well-validated personality measures to focus participants on their personality over the prior week. Doing so is likely to lead to similar between-person internal consistency; greater within-person internal consistency; greater variability; slightly lower, though generally similar mean scores; and larger within-person associations with related measures without substantially impacting other weekly measures of symptoms, well-being, or functioning compared to using personality measures with their established, general instructions.