Authors: Raimundo X. Rodriguez (1.Interdepartmental Neuroscience Program, Yale School of Medicine), Stephanie Noble (2.Dept. of Psychology, Northeastern University; 3.Dept. of Bioengineering, Northeastern University; 4.Center for Cognitive and Brain Health, Northeastern University), Chris C. Camp (1.Interdepartmental Neuroscience Program, Yale School of Medicine), Dustin Scheinost (1.Interdepartmental Neuroscience Program, Yale School of Medicine; 5.Dept. of Radiology and Biomedical Imaging, Yale School of Medicine; 6.Dept. of Biomedical Engineering, Yale School of Engineering and Applied Science; 7.Dept. of Statistics and Data Science, Yale University; 8.Child Study Center, Yale School of Medicine; 9.Wu Tsai Institute, Yale University; 10.Yale Biomedical Imaging Institute, Yale School of Medicine)
Categories: Article, Minor component analysis, MCA, low-dimensional, reliability, discriminability, connectome-based predictive modeling, CPM, fingerprinting, intraclass correlation, ICC, networks
Source: Nature neuroscience
Authors: Raimundo X. Rodriguez, Stephanie Noble, Chris C. Camp, Dustin Scheinost
High-amplitude co-activation patterns are sparsely present during resting-state fMRI yet drive functional connectivity and resemble task activation patterns. However, little research has characterized the remaining majority of the resting-state signal. Here we introduce caricaturing, a method to project resting-state data to a subspace orthogonal to a manifold of co-activation patterns estimated from the task fMRI data. This removes linear combinations of these co-activation patterns from resting-state data to create caricatured connectomes. We used task data from two large-scale neuroimaging datasets to construct a manifold of task co-activation patterns and constructed caricatured connectomes. These connectomes exhibit lower between-individual similarity and higher identifiability and could be used to predict phenotypic measures, representing individual differences in behavior, often to a greater degree than standard connectomes. Our results show there is useful signal beyond the dominating co-activations that drive resting-state functional connectivity, which may better characterize the brain’s intrinsic functional architecture.
Functional connectivity at rest is driven by bursts of short, spatially distributed co-activation events, which make up only a small fraction of the scan^1–5^. These co-activations increase the temporal correlation between functional networks and the nonstationarity of the resting-state signal^1,3^ and mimic task-induced activity patterns^3,5–11^. However, they may be problematic because they account for only a small fraction of the signal and occur sporadically^12^, leading to low reliability^13,14^ and predictive utility^15^.
Relatedly, two theoretical frameworks exist to emphasize individual differences in functional connectivity^13^. The spotlight approach “[blurs] the irrelevant features while retaining and enriching relevant ones”. Task-induced changes in the fMRI signal achieve this goal by consistently modulating the co-activations across individuals^13,16^. Since task paradigms intentionally elicit these strong co-activations, they are more densely sampled during tasks, driving a comparatively stable connectivity pattern. As a result, task-based connectomes exhibit greater within- and between-individual similarity, reliability, and predictive utility than resting-state connectomes. These improvements have been observed across classic task paradigms^13–15,17,18^, as well as modern naturalistic and movie paradigms^19–21^. Methods which transform resting-state connectomes into task-based connectomes similarly improve reliability and prediction^22,23^.
The second framework—the caricature approach—exaggerates “the most prominent features of each individual”^13^. Like a caricature, individuals would become less similar to each other, reducing between-individual similarity and increasing individual differences^13^. While several approaches exist to statistically separate group and individual components of a connectome^24–26^, a more interpretable approach may be to remove the co-activation events^12^. As task-induced co-activation patterns lie on a low-dimensional manifold common across participants^6,27^, an eigendecomposition of task fMRI data can flexibly identify these patterns in an unsupervised manner. Projecting resting-state data onto a subspace orthogonal to this manifold would remove linear combinations of these co-activation patterns from the resting-state data. Since individuals engage similar task-relevant co-activation patterns, removing them would reduce between-individual similarity as a caricature would. If individual differences are also enhanced, removing the task-like co-activations in resting-state fMRI would formalize this thought experiment by using the signals typically discarded by others^1,3–5,27–30^.
In this work, we investigate the signal beyond co-activation events^12^ to implement caricaturing. We compared the within- and between-individual similarity of the projected data (i.e., caricatured connectomes) to standard resting-state connectomes using four datasets. We also investigated the multivariate and univariate reliability of caricatured connectomes. Finally, we tested whether these understudied signals carried information about individual differences in behavior using Connectome-based Predictive Modeling (CPM). Caricatured connectomes behaved like caricatures, exhibiting lower between-individual similarity but higher multivariate reliability. Caricatured connectomes significantly predicted several phenotypic measures. Importantly, these results suggest that a hidden signal—existing beyond the dominating co-activation patterns that drive resting-state functional connectivity—contains information about individual differences in behavior and may be better for characterizing the brain’s intrinsic functional architecture.
To implement caricaturing, we projected the resting-state fMRI time series away from a manifold of task co-activation patterns, estimated from the task-based time series (Figure 1A). First, task-based time series are temporally concatenated across tasks and participants, and the corresponding covariance matrix is constructed. Group-level eigenvectors are then calculated. Next, a projection matrix is created to project an fMRI time point into a subspace orthogonal to the eigenvectors that explain the most variance. These top eigenvectors represent the low dimensional (i.e., dominant) co-activation patterns that recur across tasks^27^. Multiplying this projection matrix by each time point in a resting-state scan creates a new time series without information from these dominant co-activation patterns. Finally, caricatured connectomes are made as usual by correlating time series pairs.
Prior work has removed signals from resting-state data to improve data quality. Generally, these works use principal component analysis (PCA) as a form of data dimensionality reduction and remove low-variance principal components (i.e., the bottom eigenvectors). As the bottom eigenvectors are typically considered noise, these approaches putatively denoise resting-state data, resulting in improved participant identifiability and predictive utility^31–35^.
In contrast to these works, caricaturing is an implementation of Minor Component Analysis (MCA). Whereas PCA is a dimensionality reduction technique that uses the top components, MCA extracts components that account for the smallest variances. MCA is useful in situations where one is interested in the weaker, less significant patterns in data, as it emphasizes small fluctuations or patterns that may be overlooked by PCA. Thus, caricaturing challenges the hypothesis implicit in many works using PCA that the most important signals in resting-state are the first components (i.e., the high-amplitude co-activation patterns). Instead, it discards the dominant co-activation patterns and keeps the remaining signal. In other words, caricaturing investigates the signal previously removed in PCA-based approaches.
We used the Human Connectome Project (HCP)^36^, UCLA Consortium for Neuropsychiatric Phenomics (CNP)^37^, and the Yale Test-Retest (TRT)^38^ datasets for primary analysis. The Yale Transdiagnostic Dataset^39^ was used for external validation. Demographics for these datasets are summarized in Supplementary Figure 1. The HCP and CNP datasets have rich task data and were used to create the manifold of task co-activations and associated projection matrices, independently. Using two datasets to create different manifolds highlights that caricaturing is general to the tasks and datasets used. Resting-state data from the HCP and TRT datasets were projected away from the manifold, creating ‘caricatured’ connectomes. ‘caricaturedHCP’ or ‘caricaturedCNP’ indicates which dataset–the HCP or CNP–was used to generate the manifold. When using only the HCP to create caricatured connectomes, we used a validated subsampling procedure (Supplementary Figures 2–5) to avoid data leakage. In short, for any iteration of a downstream analysis, participants were either used for eigendecomposition or for the analysis, but not both. Supplementary Figure 6 highlights the diminished correlation structure of the caricatured connectomes compared to standard connectomes. Sensitivity analyses investigating how demographic information influences the eigenvector structure are shown in the Supplement (Supplementary Figure 7).
Caricatured connectomes from HCP and TRT were evaluated and compared to ‘Standard’ connectomes (i.e., those created using standard resting-state fMRI data) in multiple downstream analyses (Figure 1B). First, we characterized within- and between-individual similarity in the HCP and TRT datasets. Second, we analyzed multivariate reliability via fingerprinting—identifying individuals from a pool of participants—and discriminability and univariate reliability via intra-class correlation (ICC) in the HCP and TRT datasets. Note that discriminability is distinct from a similar method, differential identifiability^32^, which also has been used in the literature^33,35,40–46^. Multivariate reliability reflects the stability of multidimensional data, such as whole-brain patterns, while univariate reliability reflects the reliability of each measurement individually. For the similarity and reliability analyses, we truncated the HCP time series to 176 frames before constructing the connectomes as previously done^13^ to avoid a potential ceiling effect since longer scan duration improves connectome stability^47–50^ and fingerprinting accuracy^13^. Third, we performed connectome-based predictive modeling (CPM) in the HCP dataset for age, IQ, and sex. The Yale Transdiagnostic Dataset was used for external validation of these models. Age, sex, and IQ were chosen as they have moderate to large effect sizes and are commonly used for benchmarking predictive models. Additional models for less common variables with a wider range of effect sizes are presented in the supplement. These variables include BMI, a proxy score for borderline personality disorder (BPD), and two measures of task performance during the Emotion and Relational tasks in the HCP dataset. Models for continuous variables (e.g., age and IQ) used ridge regression; models for classification (e.g., sex) used support vector machines (SVM). Models were trained and tested using 1000 iterations of 10-fold cross-validation. Significant prediction based on caricatured connectomes suggests that the signal beyond dominant co-activation patterns contains information about individual differences in behavior.
Unless otherwise noted, all statistical tests were either non-parametric permutation or subsampling tests with effect sizes reported as percent increase or decrease. Bonferroni correction was applied when needed. Detailed statistics information is elaborated in Section 4.6.
In the HCP, we replicated previous results^27^ to verify that task-driven activity lies on a low-dimensional manifold. As in caricaturing, group-level eigenvectors were calculated using concatenated task data from every participant in the HCP dataset. We then correlated the top five eigenvector time series with the task block regressors. In line with previous results, the time series for the first eigenvector was strongly correlated with the full task block regressor, created by combining the regressors across all tasks (mean r=0.387; Supplementary Figure 2A). Additionally, the time series for the fifth eigenvector was correlated with the absolute value of the derivative of the full task block regressors (mean r=0.113). The remaining eigenvector time series differentially correlated with the task regressors (Supplementary Figure 2A).
We investigated within- and between-individual similarity for caricatured and Standard connectomes as a caricature should decrease between-individual similarity. As expected in the HCP, caricaturedHCP connectomes showed 53% lower between-individual similarity (p’s<0.008; subsampling test with Bonferroni correction; Figure 2A). However, these connectomes also showed 42% lower within-individual similarity (p’s<0.008; subsampling test with Bonferroni correction; Figure 2A). Likewise, caricaturedCNP connectomes exhibited a 39% decrease in within-individual similarity and a 48% decrease in between-individual similarity (p’s<0.0001; Cohen’s dwithin=−2.36, dbetween=−2.47; paired t-test with Bonferroni correction; Figure 2B). Lastly, in the TRT dataset, caricatured connectomes followed suit (Figure 2C) with significantly decreased within- (26%) and between-individual (41%) similarity (p’s<0.0001; Cohen’s dwithin=−2.88, dbetween=−3.51; paired t-test with Bonferroni correction). Notably, these changes contrast with spotlighting, which increases both within- and between-individual similarity^13^.
We investigated multivariate reliability to determine whether changes in connectome similarity had a wider effect on the data. Reliability is broadly a function of within- and between-individual similarity, where a higher ratio between the two is associated with higher reliability. In the HCP dataset, caricaturedHCP connectomes exhibited significantly better fingerprinting than Standard connectomes, increasing accuracy by 41% (p’s<0.004; subsampling test with Bonferroni correction; Figure 3A). Results were similar when using caricaturedCNP connectomes, with a 42% average increase in accuracy (p’s<0.004; permutation test with Bonferroni correction; Supplementary Table 1). In the TRT dataset, we also assessed the perfect separability rate (PSR)^38^, an extension of fingerprinting for datasets with more than two scan sessions per participant. Caricatured connectomes improved PSR by 281% (p’s<0.004; permutation test with Bonferroni correction; Supplementary Table 2). Using the HCP, caricaturedHCP and caricaturedCNP connectomes increased discriminability on average by 4% compared to their Standard counterparts (p’s<0.004; subsamplingHCP and permutationCNP tests with Bonferroni correction; Figure 3B; Supplementary Table 3). Lastly, in the TRT dataset, discriminability was 2% higher for caricaturedHCP and caricaturedCNP connectomes than Standard connectomes (pHCP=0.048, pCNP=0.26; permutation test with Bonferroni correction; Supplementary Table 3). Discriminability was near-perfect in the TRT dataset, leaving little room for large improvements. Likely, the small number of participants inflates discriminability. While both within- and between-individual similarity decreased with caricatured connectomes, between-individual similarity decreased more than within-individual similarity, increasing multivariate reliability.
Given the changes of within- and between-individual similarity and multivariate reliability, we investigated univariate (i.e., edge-level) reliability to understand how individual edges are changed after projection. Surprisingly, ICC was lower in caricatured connectomes than in Standard ones. In the HCP dataset, caricaturedHCP and caricaturedCNP connectomes had a 25% decrease in ICC compared to Standard connectomes (p’s<0.0001; Wilcoxon’s rHCP=−0.41, rCNP=−0.40; Wilcoxon’s signed rank test with Bonferroni correction; Figure 4A–B). In the TRT dataset (Figure 4C), ICC was 12% lower for the caricaturedHCP connectomes and 7% lower for the caricaturedCNP connectomes (p’s<0.0001; Wilcoxon’s r=−0.16; Wilcoxon’s signed rank test with Bonferroni correction). The variance components from the ICC are discussed in Section SI 1.5.
Finally, we tested whether caricatured connectomes contained information about individual differences in behavior using CPM. Caricatured connectomes significantly predicted (p’s≤0.032, permutation tests) age, IQ, sex, BMI, BPD (in all but one condition), relational task performance (in two conditions), and emotional task performance (in one condition) (Figure 5; Supplementary Figure 10; Supplementary Tables 4–5). Models built on Standard connectomes were similarly significantly predictive (p’s≤0.031, permutation tests). Finally, we assessed external validation (see Section SI 1.7) to account for possible overfitting in the models. Models using caricatured connectomes were significant (Supplementary Figure 16). These results suggest that signals that are typically ignored^1,3–5,27–35,51^–for example, the signals emphasized by caricaturing–contain rich information about individual differences in behavior.
As multivariate reliability provides an upper limit for prediction^52^, we examined whether improved CPM results for caricatured connectomes accompanied the increased multivariate reliability. Results showed that caricaturing significantly improved prediction performance in most cases. For age, IQ, and BMI, CPM models built from the caricaturedHCP connectomes were significantly better than those from Standard connectomes (p’s≤0.0001; Cohen’s dage=3.96, dIQ=1.51, dBMI=5.21; corrected paired t-test with Bonferroni correction), explaining 31% more variance for age, 34% more variance for IQ, and 18% more variance for BMI (Figure 5A–B; Supplementary Figure 10A; Supplementary Figure 11; left panels). Results were similar for caricaturedCNP connectomes (Figure 5A–B; Supplementary Figure 10A; Supplementary Figure 11; right panels), with a 15% average increase in explained variance for age, IQ, and BMI (p’s≤0.0022; Cohen’s dage=3.15, dIQ=1.43, dBMI=4.39; corrected paired t-test with Bonferroni correction). Sex classification was more accurate—significantly in all but one condition—for caricaturedHCP and caricaturedCNP connectomes (significant p’s≤0.0072; significant Cohen’s dHCP=1.87, dCNP=1.52; corrected paired t-test with Bonferroni correction; Figure 5C). For task performance, caricatured connectomes significantly outperformed Standard connectomes for six of the eight conditions (Supplementary Figure 10C–D). BPD was the only measure where prediction performance significantly decreased with caricaturing (Supplementary Figure 10B). Together, these results suggest caricaturing can improve prediction performance for phenotypes with a range of effect sizes.
We also examined the multicollinearity of each model’s features to assess model interpretability. Across all phenotypes, multicollinearity was lower for models built on caricaturedHCP connectomes (Supplementary Figures 12 and 13). The numbers of features, however, were not consistently different in models built on caricaturedHCP connectomes compared to those built on Standard connectomes (Supplementary Figure 14). Even though some models had similar numbers of features, overlap in the features selected was relatively low (Supplementary Figure 15). Given the difference between models from Standard and caricatured connectomes, we explored combining both connectomes into a single model. Overall, these combined models showed numerically improved predictive performance (see Section SI 1.8). Thus, caricatured connectomes increased CPM performance over Standard connectomes, partially by improving a connectome’s feature space and multivariate reliability.
We also tested creating a low-dimensional manifold of co-activation patterns using resting-state or movie data. We computed the overlap between the task-derived and rest-derived manifolds to assess whether the manifolds were the same (Supplementary Figure 18). As expected, these manifolds overlapped, signifying that eigenvectors can be nearly equivalently derived from task or rest. These results support a primary that brain-wide patterns present during specific tasks are not only present during resting-state, but also dominant. Further, we created caricatured HCP connectomes using the top five eigenvectors derived from the CNP resting-state or Yale Transdiagnostic movie data. Next, we replicated the above analyses (see SI Section 1.9 and 1.10). Both within- and between-individual similarity were reduced and fingerprinting accuracy increased in the caricatured connectomes (Supplementary Figure 19; Supplementary Table 6; Supplementary Figure 21; Supplementary Table 7). Lastly, we obtained similar prediction results using the rest and movie-derived eigenvectors for caricaturing (Supplementary Figure 20; Supplementary Figure 22).
In this work, we introduced caricaturing—a method to project resting-state data to a subspace orthogonal to a manifold of co-activation patterns estimated from task fMRI data. This projection removes linear combinations of these co-activation patterns from the resting-state data. It also formalizes the thought experiment from Finn et al.^13^ and “caricatures” resting-state connectomes. These connectomes exhibited decreased between- and within-individual similarity and increased multivariate reliability and predictive utility. As caricaturing identifies correlates of task-relevant variance and projects the signal orthogonally, it is an initial attempt to remove task-like co-activations from rest. Aligned with Iraji et al.^12^, our results suggest that there is valuable signal beyond the dominating co-activations that drive resting-state functional connectivity. Therefore, caricaturing opens the door for future studies using these signals, which may better characterize the brain’s intrinsic functional architecture.
Resting-state functional connectivity can be difficult to interpret^53^. Previous works—including functional gradient mapping—have shown overlap between manifolds, or gradients, generated from task data and those from resting-state data^6,54,55^. If resting-state combines task-like co-activation events^3,5–11^ and intrinsic functional architecture, it should be possible to separate the two, similar to separating group and individual parts of a connectome^24–26^. Since our method projects resting-state data to a subspace orthogonal to a task-relevant manifold, the remaining signal may represent an intrinsic functional signal. Our results suggest that information about individual differences is present in this remaining signal, potentially at a greater degree than that found in the co-activation events. Despite these results, the signal used in caricaturing is typically discarded in favor of studying the co-activation events^1,3–5,27–30^. Further, many works use PCA to remove low-variance components from connectomes^31–35,51^. Together, such approaches treat these low-variance components as noise. By using MCA, caricaturing eschews this common practice in neuroimaging, emphasizing the importance of these generally discarded signals. Our results and others^12^ demonstrate that the resting-state signal contains much more information than previously used.
While the composition of the caricatured signal is unknown, within-individual state differences, spontaneous activity, and neural variability likely comprise it. These smaller components will become relatively enhanced as dominant task-like components are silenced with MCA. Additionally, individual differences appear in caricatured connectomes fundamentally differently than in standard connectomes. They are in different edges (Supplementary Figure 15) and represent information more efficiently across edges (Supplementary Figures 12–13). For example, neural variability (e.g., variability in brain state engagement^56^) has often been considered a noise source, much like the bottom principal components in PCA. However, it is critical for brain function^57^. Caricaturing might prove to be a valuable tool to study spontaneous activity and neural variability.
As theorized^13^, caricaturing decreases between-individual similarity by exaggerating individual-specific qualities. In other words, caricaturing de-emphasizes common traits across individuals. This approach contrasts spotlighting, which uses tasks to elicit greater within-individual and between-individual similarity. General task-related signals likely drive similarity in this case. For example, even if an individual engages in different tasks during two scans, the generality of task-related signals increases their similarity (e.g., default mode deactivation). When the general task-related or group-level signals are removed, within-individual similarity will decrease. However, both approaches–as well as others that specifically seek to remove group-level signals^24–26^–can enhance individual differences as the ratio of within-individual and between-individual information increases. Future work should determine whether it is possible to maximize within-individual similarity while minimizing between-individual similarity.
Paradoxically, both “adding” and “subtracting” task-like co-activation patterns appear to enhance individual differences. We speculate that task-like co-activation patterns as well as the remaining signal in resting-state carry individual differences. However, we posit that resting-state is the wrong combination of the two. The sparseness of task-like co-activations in resting-state^1–5^ limits the information about individual differences that can be extracted. Having the participant perform a task increases the frequency and intensity of the co-activation patterns, allowing for more individual differences to be extracted. By contrast in resting-state, even though the task-like co-activations are sparse, they cover up the remaining signal, which also contains rich individual differences. Removing the co-activations allows one to access this signal for improved detection of individual differences.
Spotlighting and caricaturing are analogous to precoloring and prewhitening—two approaches from time series analysis, including in neuroimaging, to account for autocorrelations^58^. In precoloring, a large correlation is added to the time series to swamp out the unknown, existing autocorrelation. The total autocorrelation can be estimated as only the added component and input into the model to improve statistical inference. In spotlighting, tasks add a structure on top of rest, masking its unconstrained nature and improving downstream analyses (i.e., identification and prediction) in a conceptually similar way. However, precoloring also acts as a low-pass filter, which may remove signals of interest and degrade power^59^, similar to how tasks may obscure useful underlying signals. In contrast to precoloring, prewhitening estimates the true autocorrelation to remove it directly. Once removed, classic statistical inference is valid. Similarly, our caricaturing method estimates group-derived, task-like co-activations and removes them from resting-state data to improve identification and prediction. In most applications, prewhitening is preferred to precoloring. However, prewhitening requires an accurate autocorrelation model, which can be difficult to determine with real data, similar to how it is difficult to estimate true task-like co-activation patterns. As with prewhitening and precoloring, future research will clarify the strengths and weaknesses of spotlighting and caricaturing and how they complement each other.
Caricatured connectomes have greater multivariate but lower univariate reliability. While often going hand in hand, increasing multivariate does not guarantee an increase in univariate reliability^60^. They are distinct. Similarly, improving univariate reliability may not improve predictive utility^38,61,62^. Our results further explain this observation. Prediction methods are inherently multivariate, reflecting a pattern distributed across many features. Improving each feature’s reliability (i.e., univariate) may not make the overall pattern more reliable or, in turn, predictive. Therefore, methods for increasing multivariate reliability instead of univariate reliability are better equipped to improve predictive models.
There are several limitations of our work. Although we project resting-state data away from a manifold of task co-activation patterns, it is unclear how much task-relevant information is removed from the resting-state data. In line with previous research^27^, the eigenvectors temporally track with task designs. Thus, a portion of information in the eigenvectors is certainly attributable to the task. However, some task-relevant signals may not resemble the selected co-activation patterns. Future research is needed to determine how much task-relevant information is present in resting-state and how it can be removed. Second, we only used linear methods to define the manifold. However, many nonlinear methods exist^8,56,63^ which may better identify task co-activation patterns. Further advances through nonlinear techniques may reveal complementary results. Third, we used correlation to measure distance for multivariate reliability analyses. However, other metrics, such as geodesic distance, have been implemented^64,65^ and may yield slightly different results. Fourth, though our main goal was not to increase prediction performance, observed improvements were modest. Direct comparisons to the broader literature are difficult, given the breadth of approaches. However, since caricaturing operates on the data, it can be combined with other advances (like deep neural networks) for further improvements. Fifth, it is important to note whether caricaturing is biased for various demographic factors (e.g., ethnicity). Variance explained in the resting-state data showed weak to moderate correlations with demographic variables (Supplementary Figure 7), suggesting that caricaturing is not overly learning from demographic distribution information. Future work should continue to assess whether any biases may be embedded in caricaturing. Finally, since caricaturing uses MCA, caricatured data may contain a higher proportion of unstructured noise than standard connectomes. Nevertheless, given the demonstrated identifiability and predictive utility, they likely retain strong neural signals.
In conclusion, we introduce a caricaturing method that projects resting-state fMRI data away from a manifold of task co-activation patterns to putatively remove task co-activation patterns. Our work suggests that the signal remaining after projection maintains information about individual differences, often to an enhanced degree compared to the standard resting-state signal. If resting-state combines intrinsic functional architecture and task-like co-activations, the remaining signal may better represent this intrinsic functional architecture and is a topic for further study. Meanwhile, caricaturing can be applied to existing and future resting-state data to evaluate novel sources of individual differences.
Four datasets were used in this the Human Connectome Project (HCP)^36^, the UCLA Consortium for Neuropsychiatric Phenomics (CNP)^37^, the Yale test-retest dataset (TRT)^38^, and the Yale Transdiagnostic Dataset. HCP and CNP were chosen for their wide array of task-based data available in addition to resting-state data. The TRT dataset was selected because participants were scanned four times across different days, yielding ideal data for assessing ICC and discriminability. The Yale Transdiagnostic Dataset was selected for its movie data and for external validation.
For the HCP data, we only used participants with data for each of the seven tasks (EMOTION, GAMBLING, LANGUAGE, MOTOR, RELATIONAL, SOCIAL, WM) and both resting-state scans for both the left-to-right (LR) and right-to-left (RL) phase encodings from the S1200 release. We also removed participants from analysis if the mean motion across all of their scans was greater than 0.1mm, if any scan’s mean motion was greater than 0.15mm, or if they were missing any data. Based on these criteria, 661 participants (males: 316, 345) remained. For the CNP data, we only used participants with data for each of the six tasks (BART, PAMENC, PAMRET, SCAP, STOPSIGNAL, TASKSWITCH). We further excluded participants if their mean motion across these scans was greater than 0.1mm, if the mean motion in any scan was greater than 0.15mm, or if any of their scans were missing data. From these criteria, we remained with 136 participants (males: 78, 58). For the TRT dataset, participants were scanned six times on four days. Each scan was collected at rest for six minutes, but for some runs, scans were shorter and were still included in our analysis. The resulting data comprised 12 participants (males: 6, 6). For the Yale Transdiagnostic Dataset, individuals were scanned under 8 different conditions of which we included scans from baseline resting-state and movie watching. Each scan was six minutes. For the movies task, participants watched three movie clips presented in the same order without a break (Inside Out, The Princess Bride, and Up). Only participants who successfully completed each of the 8 conditions were included. We also excluded participants if the mean motion in any scan was greater than 0.2mm, if any scan was missing data, or if the participant had potential neurological concerns. From these criteria, 237 individuals (males: 105, 132) remained. Additional demographics for the four datasets are summarized in Supplementary Figure 1.
Consistent preprocessing steps were applied to all datasets. For the HCP data, we started with the minimally preprocessed data for the HCP dataset^66^, which includes motion correction and non-linear registration into common space. For the CNP data, skull-stripping was performed with OptiBet^67^. The data were then registered into common space. Motion correction was done with SPM8. The TRT dataset was first skull-stripped in FSL^68^ and then registered into common space. Motion correction was performed in SPM5. The data were also iteratively smoothed to a 2.5mm Gaussian kernel equivalent^69,70^. For the Yale Transdiagnostic data, skull-stripping was performed with OptiBet^67^. The data were then registered into common space. Motion correction was done with SPM8.
For all datasets, further preprocessing was performed in volume space using BioImage Suite^71^. These steps included regressing 24 motion parameters, regressing the mean white matter, gray matter, and CSF time series, removing linear and quadratic trends, and applying a low-pass Gaussian filter (cutoff frequency ~0.12 Hz for HCP, CNP, and Yale Transdiagnostic Dataset and ~0.19Hz for TRT). The preprocessed fMRI data were parcellated into 268 nodes using the Shen 268 atlas. Standard functional connectomes were generated by correlating the blood oxygenation level–dependent time series between all node pairs, with each edge Fisher transformed. Caricatured connectomes were generated similarly after projecting the time series data away from the task manifold. For more detailed accounts of the HCP and CNP see Gao et al., 2021^8^, of the TRT see Noble et al., 2017^38^, and of the Yale Transdiagnostic Dataset see Greene et al, 2022^39^.
In caricaturing, we project resting-state data away from a task manifold. This method is an implementation of Minor Component Analysis (MCA) and has two parts. The first is to define a task manifold from group-level, task fMRI. First, we temporally concatenate all task scans for individuals. Then, we perform eigendecomposition (eig.m in Matlab) on the covariance matrix describing the covariance between all pairs of node-level time series in the concatenated data. Each eigenvector is a common spatial activity pattern across tasks. The second part is to project resting-state data away from this manifold. First, we sort the eigenvectors in descending order by how much variance they explain in the data, as described by their respective eigenvalues. Then, we create a matrix of eigenvectors excluding the top ones (e.g., the first five, as in this work). This matrix is multiplied by its transpose to obtain the projection matrix. Next, we multiply the projection matrix and each time point from a resting-state scan, orthogonalizing them to the task manifold. Caricatured connectomes are created by correlating these orthogonalized time series.
Using fMRI time series data from one participant, the data can be represented as a matrix with dimensions t×n, where t is the number of frames in the scan and n is the number of nodes in the atlas. To estimate the eigenvectors, the time series for each node are z-scored, and the covariance matrix is computed as the covariance between all pairs of node-level time series. The n×n covariance matrix is then input into an eigendecomposition algorithm. The outputs are an n×n matrix V of eigenvectors, which are orthonormal patterns of co-activation in the brain, and an n×1 vector D of eigenvalues, which denote how much variance each eigenvector explains. We then reorder these outputs (maintaining the variable names V and D) such that each successive eigenvector explains less variance in the data.
To extend this framework to multiple scans, first, each time series is z-scored individually, and then they are concatenated along the time dimension. Thus, if using m scans where each scan i has ti frames, the final matrix prior to calculating the covariance matrix will have dimensions (∑i=1mti)×n. The resulting eigenvectors are co-activation patterns that explain variance in the concatenated data.
Using a time series matrix M with dimensions t×n and a sorted eigenvector matrix V with dimensions n×n, we first choose a subset of eigenvectors onto which we will project the time series. Let v be vector of a subset of the integers from 1 to n indicating which eigenvectors will be used for projection. Then, we can create a new matrix V^ where the i^th^ column is equal to the (vi)th column of V where vi is the i^th^ element of v. We then create a projection matrix P=V^×V^T, where T indicates the transpose of a matrix. To project M onto the desired eigenvectors, we simply multiply it by the projection matrix to obtain M^=M×P. The resulting matrix M^ still retains the same dimensions as M but now with only information from co-activation patterns that can be constructed by the desired eigenvectors. Thus, the projected time series data can still be used for downstream connectomics analysis with the same dimensionality.
The first five eigenvectors obtained by concatenating time series across participants and tasks strongly reflected various aspects of the task structure^27^. Based on this result, we implemented our framework by projecting each participant’s z-scored resting-state time series onto the last 263 eigenvectors of the task time series across multiple participants. Thus, we remove information from the top 5 group-level task eigenvectors to remove dominating signals in the task data from the resting-state data.
Time series data were parcellated according to the Shen268 (268 nodes) atlas^72^, whereby the mean time course for each node was computed as the average of all voxel-level time series in that node. Connectomes were then constructed by taking the Fisher transform of the Pearson correlation between all pairs of node-wise time series. As a subsampling procedure was used in some analyses to ensure there was no data leakage between the data used to construct the eigenvectors and the connectomes from which those eigenvectors were projected away, we constructed caricatured resting-state connectomes in the REST1 and REST2 HCP data for each subsample separately.
We evaluated how caricaturing affects downstream connectome metrics. We assessed the connectomes constructed from projected resting-state time series (referred to as caricatured connectomes) via within- and between-individual similarity, fingerprinting, discriminability, intraclass correlation (ICC), and connectome-based predictive modeling (CPM).
To calculate similarity within and between individuals, we extracted and vectorized the upper triangle of the connectome. The within-individual similarities were computed as the correlation between vectorized connectome pairs of the same individual across scans. The between-individual similarities were calculated as the correlation between vectorized connectome pairs between different individuals.
We performed fingerprinting as described in Finn et al., 2015^73^. Given two groups of distinct connectomes that span the same participants, we labeled one group as the ‘Database’ and the other as the ‘Target Set’. For each connectome in the ‘Target Set’, the Pearson correlation between that connectome and each in the ‘Database’ was calculated. The identity of the connectome in the ‘Database’ that corresponded to the highest correlation was assigned as the predicted identity of the current connectome in the ‘Target Set’. After repeating this for all connectomes in the ‘Target Set’, the fingerprinting accuracy for this label of ‘Database’ and ‘Target Set’ was calculated as the number of participants correctly identified divided by the number of participants. Perfect separability analysis–described in Noble et al., 2017^38^–is a simple extension to datasets with more than two scans per participant. From this, we can calculate the perfect separability rate (PSR), the percentage of scans for which all within-individual similarities are higher than any between-individual comparison.
Although fingerprinting serves as an excellent metric for participant identifiability, because it is binary in its methodology (i.e., correct vs. incorrect identification), it potentially leaves out information. Discriminability seeks to overcome this limitation by centering the method around the ratio of between-individual measurement distances that exceed within-individual measurement distances^52,74^. Furthermore, like perfect separability analysis, discriminability allows for any number of measurements per participant. We started by constructing a distance matrix between all pairs of measurements, using the correlation distance, or 1 minus Pearson’s r. Discriminability was calculated as the proportion of instances within-individual measurements were closer in distance than between-individual measurements across all possible combinations. Note that discriminability is distinct from a similar method, differential identifiability^32^, which also has been used in the literature^33,35,40–46^.
Whereas fingerprinting and discriminability are measures of multivariate reliability, that is, how reliable are connectomes as a whole, intraclass correlation (ICC) is a measure of univariate reliability, or how reliable are the edges of a connectome^75^. ICC^76^ is the fraction of variance due to the participant divided by the variance due to error. In the HCP data, there were only two measures per participant in each condition, so the variance components were estimated with a 2-way ANOVA. Symbolically, using subscripts p (participant), r (run), and e (residual) to represent the factors, this can be represented as ICC(x)=σp2(x)σp2(x)+σr2(x)+σpr,e2(x), where x is an edge in the connectome. In the TRT data, since the 24 scans per participant were partitioned by day and run, the variance components were estimated with a 3-way ANOVA. Here, adding d (day) to the factors, we get ICC(x)=σp2(x)σp2(x)+σr2(x)+σd2(x)+σpr2(x)+σpd2(x)+σrd2(x)+σprd,e2(x). We set negative variance components (which were small in magnitude) to zero before computing ICC as in prior work^77^. For more information, see the shared code provided in the links below.
For our CPM analyses, we chose sex, fluid intelligence (IQ: PMAT in HCP and WASI-MR in Yale Transdiagnostic Dataset), and age to predict in the HCP dataset. These phenotypes are common in benchmarking analyses and typically have larger effect sizes. We also chose to predict body-mass index (BMI), a proxy score for borderline personality disorder (BPD), participant accuracy during the relational blocks in the Relation task, and participant accuracy during the shape blocks in the Emotion task. BMI-connectome models have been shown to present with moderate to high effect sizes^78^. The BPD score was previously constructed and validated^79,80^ and used for CPM^81^, where it was moderately predictable. The two task performance scores were previously shown to predict poorly^82^.
In all cases, models were built with 10-fold cross-validation where models were trained on 90% of the families and tested in 10% of the families in each fold. In each fold, feature selection was done to reduce the number of connectome edges used to build the model. Here, in the 90% of families used to build the model, edges were associated with the phenotype by either correlation (if continuous) or t-test (if binary). The resulting p-values for each edge were then observed and edges with a p-value less than 0.05 were used to build the model. For each phenotype, 1000 iterations of this 10-fold cross-validation were performed. For the continuous variables, the models were built using ridge regression. The hyperparameter λ was chosen via nested 10-fold cross-validation using MATLAB’s default parameters for the lasso.m function. As lasso.m implements elastic net regression, we set the α parameter to 0 to use ridge regression. The search space is then a geometric sequence of length 100 where the maximum λ (λmax) is the largest λ that can produce a non-null model and the minimum is equal to λmax*10^−4^. Then, λ is chosen as the largest such that the mean squared error is within 1 standard error of the minimum mean squared error. For sex, the models were built using linear support vector machine (SVM). The hyperparameter λ was not optimized and was set to 0.
This section provides specific details for all tests performed on the results of this research.
For similarity analysis performed on the HCP data using the LR and RL phase encodings as the two scans per participant, tests assessing differences across the medians of within-individual similarity distributions were performed between REST1 and caricaturedHCP REST1, and REST2 and caricaturedHCP REST2. We also performed tests to assess the same differences in medians of between-individual similarity distributions. The test performed is a paired, one-way non-parametric subtraction test performed in both directions whereby one distribution of median similarity is subtracted from the other, and 1 minus the proportion of differences that are greater than 0 is the resulting p-value. We then use Bonferroni correction to adjust the p-values across conditions (i.e., REST1 and REST2) and sub-analysis (i.e., within-individual and between-individual similarity). Thus, to be significant, a test must produce an uncorrected p-value less than 0.05/24. If an uncorrected p-value is returned as 0, we say it is less than 0.001, since there are 1000 observations.
For the similarity analysis involving HCP connectomes projected onto the CNP eigenvectors, we compared the full distributions of within-individual and between-individual similarity between REST1 and caricaturedCNP REST1, and REST2 and caricaturedCNP REST2. A paired t-test was used here, and Bonferroni correction was applied across conditions and sub-analysis (multiplying the p-value by 4 to correct).
For the TRT similarity analysis, we compared the distribution of within-individual similarity in Standard resting-state connectomes to both caricaturedHCP and caricaturedCNP connectomes. The same was done for between-individual similarity. We again used the paired t-test with Bonferroni correction to account for the four tests performed.
For fingerprinting performed between LR and RL phase encodings for each scan condition, tests assessing differences in mean accuracy were performed between REST1 and caricaturedHCP REST1, and REST2 and caricaturedHCP REST2. This was done via the same non-parametric subsampling test described above. Afterwards, Bonferroni correction was applied across these two tests, multiplying uncorrected p-values by 2 to correct. Thus, to be significant, a test must produce an uncorrected p-value less than 0.05/22. If an uncorrected p-value is returned as 0, we say it is less than 0.001, since there are 1000 observations.
For the fingerprinting analysis involving HCP connectomes projected onto the CNP eigenvectors, we compared accuracies between caricaturedCNP connectomes and their standard counterparts. We performed a permutation test with 1000 permutations in which the labels for caricatured versus standard connectomes were shuffled with a probability of 0.5 for each scan to construct two distributions of null fingerprinting accuracies. The measured difference between REST1 and caricaturedCNP REST1 accuracy and REST2 and caricaturedCNP REST2 accuracy were compared to the distribution of differences between the constructed null accuracy distributions. Thus, the resulting uncorrected p-value in each case was 1 minus the proportion of times the empirical difference was greater than the null differences. If an uncorrected p-value was returned as zero, we stated that it was less than 0.001. Since this test was one-tailed and we compared two conditions, the resulting p-values underwent Bonferroni correction by multiplying them by four.
For the TRT perfect separability analysis, we compared the PSR in Standard resting-state connectomes to both caricaturedHCP and caricaturedCNP connectomes. For both comparisons, we used the permutation test described above. Again, the resulting p-values underwent Bonferroni correction by multiplying them by four. Importantly, statistical inference for PSR is not well-behaved, so the p-values are likely inaccurate. However, the discriminability analysis overcomes these limitations.
For the discriminability analysis using LR and RL phase encoded HCP connectomes, each subsample iteration yielded a single discriminability value. These were all pooled together to compare REST1 to caricaturedHCP REST1 and REST2 to caricaturedHCP REST2. The same non-parametric subsample test was used in each case, so to be significant, a test must produce an uncorrected p-value less than 0.05/22. If an uncorrected p-value is returned as 0, we say that it is less than 0.001, since there are 1000 observations.
To compare discriminability between REST1 and caricaturedCNP REST1 and REST2 and caricaturedCNP REST2 in the HCP dataset, we used the same type of permutation test as for fingerprinting accuracy. With 1000 iterations, if caricatured discriminability was always greater, we defined the uncorrected p-value as less than 0.001. Using the Bonferroni method to correct for both tests and the fact that the test was one-tailed, we multiplied each p-value by 4.
Likewise, we used the same permutation test for the analysis where discriminability was computed in the TRT dataset. Comparisons were between caricaturedHCP connectomes and the Standard TRT connectomes and between caricaturedCNP connectomes and the Standard TRT connectomes. Bonferroni correction was applied by multiplying each p-value by 4.
For ICC calculated using the LR and RL phase encodings for each scan condition in the HCP dataset, 1000 subsample iterations were performed, yielding an ICC value for each edge in each iteration. The ICC values for each edge were averaged across iterations, yielding a mean ICC value for each edge in each scan condition. To compare REST1 to caricaturedHCP REST1 and REST2 to caricaturedHCP REST2, we used a Wilcoxon signed rank test and applied Bonferroni correction by multiplying the resulting p-values by 2.
For the ICC analysis in the HCP dataset, where caricatured connectomes used CNP-derived eigenvectors, a single calculation yielded an ICC value for each edge in the connectome. Comparing REST1 to caricaturedCNP REST1 and REST2 to caricaturedCNP REST2, we used a Wilcoxon signed rank test and multiplied the resulting p-values by 2 to correct for multiple comparisons.
For the ICC analysis in the TRT dataset, two comparisons were performed using a Wilcoxon signed rank test. Edge ICC in the caricaturedHCP connectomes and edge ICC in the caricaturedCNP connectomes were compared to edge ICC in the Standard TRT connectomes. Bonferroni correction was applied by multiplying the resulting p-values by 2.
The correlation between actual and predicted phenotype assessed model performance for continuous variables. For the binary phenotype, model accuracy was assessed as the percentage of participants correctly classified. To assess whether a model was significantly predictive, we implemented a permutation test whereby in each of the 1000 iterations of 10-fold CV, the phenotype was randomly permuted before the model was constructed. This was done separately for each condition and connectome type, resulting in four distributions of null prediction accuracies for each phenotype and manifold dataset. Thus, the resulting p-value in each case was 1 minus the proportion of times the median empirical accuracy was greater than the null accuracies. If a p-value was returned as zero, we stated that it was less than 0.001. P-values were not corrected for multiple comparisons as we were not so much interested in whether predictions were significant for a specific phenotype, but rather, that some predictions might be significant at all.
For both projections (i.e., caricaturedHCP and caricaturedCNP), caricatured REST1 was compared to Standard REST1 and caricatured REST2 was compared to Standard REST2, using the 1000 iterations of cross-validation to estimate the true model accuracy. Here, we used a corrected paired t-test similar to Nadeau and Bengio, 2003^83^ and referred to as the “corrected repeated k-fold CV test” in Bouckaert and Frank, 2004^84^. For each random subsample i, we calculated prediction accuracy for the caricatured data and the standard data, say ai and bi. Letting the mean be m=11000∑i=11000(ai−bi) and the estimated variance be σ^2=1999∑i=11000(ai−bi−m)2, we arrive at the t-statistic t=m(11000+19)σ^2 where 19 is an added correction factor to account for the lack of independence in sample pairs. This is input into the t-distribution with 999 degrees of freedom to compute the p-value. As this is a one-sided test to determine whether accuracy is greater for the caricatured connectomes, the p-value is multiplied by 2. Finally, Bonferroni correction is applied within each sub-analysis (i.e., age and IQ) and within each method for constructing eigenvectors, so every p-value is again multiplied by 2 to correct for multiple comparisons. For the binary phenotype, model accuracy was assessed as the percentage of participants correctly classified. The same test was applied to compare REST1 to caricatured REST1 and REST2 to caricatured REST2, and the same statements regarding p-values and correction for multiple comparisons apply.