Authors: Brian Pho, Ryan Andrew Stevenson, Sara Saljoughi, Yalda Mohsenzadeh, Bobby Stojanoski
Categories: Original Research, ADHD, Cognition, Development, FMRI, Movie-watching
Source: Developmental Cognitive Neuroscience
Authors: Brian Pho, Ryan Andrew Stevenson, Sara Saljoughi, Yalda Mohsenzadeh, Bobby Stojanoski
Youth diagnosed with Attention-Deficit/Hyperactivity Disorder (ADHD) often show deficits in various measures of higher-level cognition, such as, executive functioning. Poorer cognitive functioning in children with ADHD has been associated with differences in functional connectivity across the brain. However, little is known about the developmental changes to the brain’s functional properties linked to different cognitive abilities in this cohort. To characterize these changes, we analyzed fMRI data (ADHD = 373, NT = 106) collected while youth between the ages of 6 and 16 watched a short movie-clip. We applied machine learning models to identify patterns of network connectivity in response to movie-watching that differentially predict cognitive abilities in our cohort. Using out-of-sample cross validation, our models successfully predicted IQ, visual spatial, verbal comprehension, and fluid reasoning in children (ages 6 – 11), but not in adolescents with ADHD (ages 12–16). Connections with the default mode, memory retrieval, and dorsal attention were driving prediction during early and middle childhood, but connections with the somatomotor, cingulo-opercular, and frontoparietal networks were more important in middle childhood. This work demonstrated that machine learning approaches can identify distinct functional connectivity profiles associated with cognitive abilities at different developmental stages in children and adolescents with ADHD.
Attention-Deficit/Hyperactivity Disorder (ADHD) is the most common neurodevelopment disorder among children and adolescents, affecting an estimated 4.8 % of all Canadian children up to 19 years of age (Waddell et al., 2002). One reason ADHD is commonly diagnosed in school-aged children is because the symptoms linked to ADHD are most salient in the classroom (Danielson et al., 2018). For instance, ADHD is best characterized by a persistent pattern of inattention (inability to maintain focus), impulsivity (acting on instinct without thinking), and/or hyperactivity (excessive restlessness and movement) that can interfere with not only completing school-based tasks, but extends to daily functioning (DuPaul and Weyandt, 2006, ADHD, 2010, Long-Term Outcomes of ADHD, 2020, Association Between Childhood Specific Learning Difficulties, 2014)
One of the most common aspects of ADHD is a deficit in processing speed (Moura et al., 2019, WISC-IV, 2013, Jacobson et al., 2011, Katz et al., 2011, Rucklidge and Tannock, 2002, Fosco et al., 2020) and executive functioning, which is comprised of three inhibitory control, cognitive flexibility, and working memory (Rucklidge and Tannock, 2002, WISC-IV, 2006, Willcutt et al., 2005). In recent years there has been considerable interest in examining neural mechanisms associated with ADHD symptomology and cognitive abilities. There is evidence that the brain-based associations of ADHD include reduced activity in the prefrontal cortex, basal ganglia, cerebellum, and parieto-temporal regions, all of which have been shown to support multiple cognitive processes such as cognitive control, working memory, and attention (A review of the biological bases of ADHD, 2003, Friedman and Rapoport, 2015, Krain and Castellanos, 2006). More recent work in this area has focused on identifying changes in functional connectivity profiles across this brain in children with ADHD. For instance, recent studies have found children with ADHD appear to have more diffuse pattern of functional connectivity in resting-state networks, such as the default mode, dorsal and ventral attention, and executive control networks, but also extend to control and reward pathways. Connectivity between regions including temporal and frontal gyri, motor areas, insula, anterior cingulate cortex, and precuneus that together form the default mode network and executive control networks appear to be disrupted or weaker, and regions like the prefrontal cortex are less segregated and less mature (Tomasi and Volkow, 2012, Bos et al., 2017, Marcos-Vidal et al., 2018). Disrupted connectivity in regions including temporal and frontal gyri, motor areas, insula, anterior cingulate cortex, and precuneus, which are strongly linked to ADHD symptoms and deficits in different cognitive domains, including attention and memory (Zhou et al., 2019, Yerys et al., 2019, Zepf et al., 2019). However, several studies have also reported stronger connectivity relative to neurotypical child population in attentional, default mode, and executive control network networks, but also somatosensory areas, and the anterior cingulate gyrus (Marcos-Vidal et al., 2018, Lin et al., 2021)
Although there is a large literature examining the cognitive abilities in individuals with ADHD, many of those studies focused on adults or single-age cohorts of children. Consequently, the changing neural mechanisms associated with cognitive abilities in children and adolescents with ADHD remains poorly understood.
Advancements in applying machine learning to large neuroimaging datasets has proven to be a valuable tool to understand the relationship between cognition and the underlying neural mechanisms (Weinstein et al., 2021, Bertolero and Bassett, 2020, Chen et al., 2022, Cui et al., 2022, Marek et al., 2022, Rosenberg et al., 2018, Shen et al., 2017). For example, machine learning (i.e., Ridge regression) has been successfully applied to resting-state functional connectivity networks to predict fluid and crystalized intelligence in healthy young adults (Tian and Zalesky, 2021). A similar approach was also used to predict links between neural activity in the default mode network and three task control networks (frontoparietal, salience, and dorsal attention) and higher-order cognitive functions, such as, general ability, speed/flexibility, and learning/memory in younger participants (Sripada et al., 2020a). Similar machine learning approaches have been used to predict cognitive abilities, such as attention, and symptomology in children with ADHD based on patterns of functional connectivity (Rosenberg et al., 2017). Although much of this work on applying machine learning to link neural activity with cognitive ability has relied on resting-state data, movie-watching fMRI has been shown to improve functional connectivity-based prediction of behavior compared to resting-state (Caldinelli and Cusack, 2022, Gruskin and Patel, 2022, Finn and Bandettini, 2021). The few studies that have used movie-watching data were able to successfully predict cognitive abilities in neurotypical adult (Finn and Bandettini, 2021) and child populations (Cantlon and Li, 2013). The advantage of using movie-watching data is likely due to reduced motion, increased engagement, but perhaps most importantly, movie-watching requires the integration of various cognitive systems to follow the complexities of the plot. Moreover, individuals often have a unique interpretation of the movie, resulting in enhanced individual signals and therefore richer brain dynamics can be captured by predictive models (Meer et al., 2020, Vanderwal et al., 2017, Vanderwal et al., 2018).
In the current study, we combined movie-watching fMRI and machine learning to identify different patterns of functional network connectivity that best predict cognitive ability in a large cohort of children and adolescents with ADHD. We predicted that not only are there specific neural mechanisms associated with different aspects of higher-level cognition in children and adolescents with ADHD, but those mechanisms change developmentally and are unique to different age groups. To explore changes in the neural mechanisms associated with cognitive functioning across time, participants were divided into three age bins and neural activity was modeled (in response to a movie) to predict the same set of cognitive abilities for each age bin. By splitting participants into three age bins, the models would either 1) predict the same set of cognitive abilities for all three age bins, suggesting a similar functional connectivity profile across development or 2) predict a different set of cognitive abilities for each age bin, suggesting the model captured a functional connectivity profile unique to age cohorts. We compared models using out-of-sample cross-validation (model trained on one age bin to predict the same cognitive ability in a different age bin) to determine the degree to which similar neural properties were associated with cognition across age bins. We analyzed shared functional connectivity profiles by calculating a difference score between the models (feature weights) trained on each age bin, revealing connections that changed the most or the least across age bins.
We obtained data from the Healthy Brain Network (HBN) biobank (releases 1–8) as part of the Child Mind Institute (Alexander et al., 2017). The Chesapeake Institutional Review Board approved the study, and details on the HBN biobank can be found http://fcon_1000.projects.nitrc.org/indi/cmi_healthy_brain_network/. The institutional review board at Ontario Tech University approved secondary analysis of the HBN data.
We included a sample of 479 data sets from children and adolescents between the ages 6–16 in the final analysis. The data sets consisted of a T1-weighted, and functional MRI scan, along with phenotypic data. We excluded participants with lower-quality data, based on visual inspection of the T1 images and functional connectivity matrices, along with those who full-scale intelligent quotient scores under 70 (Supplement).
Phenotypic data included age, sex, clinical diagnosis, and six cognitive measures from the Wechsler Intelligence Scale (Table 1). Clinical diagnoses were provided by up to ten licensed clinicians after interviews with the parents and child (Alexander et al., 2017) which we used to group participants into the ADHD (at least one diagnosis of “ADHD”) or NT (no clinical diagnoses) group. In addition to a single ADHD group (n=373), we divided participants with ADHD into three age bins based on previous work examining brain development and cognitive functioning (Shaw et al., 2006): early childhood (Bin ages 6–8, n=114), middle childhood (Bin ages 9–11, n=147), and adolescence (Bin ages 12–16, n=112). Due to the smaller sample size (n=106), we did not divide the Neurotypical (NT) group into discrete age bins.Table 1Participant demographics.Table 1GroupADHDNTMeasureAllBin 1Bin 2Bin 3All****N373114147112106Age10.57 (2.53)6.03–15.987.73 (0.76)6.03–8.9810.34 (0.88)9.04–11.9613.75 (1.13)12.03–15.9810.12 (2.78)6.05–16.50Sex (M/F)274/9977/37118/2979/3362/44WISC FSIQ100.13 (16.31)70.00–147.00104.13 (15.47)73.00–138.0098.56 (15.90)70.00–141.0098.11 (16.94)71.00–147.00108.58 (14.20)76.00–145.00WISC VSI102.32 (17.12)57.00–155.00106.38 (16.64)67.00–147.00100.73 (16.69)64.00–144.00100.28 (17.46)57.00–155.00105.21 (15.01)64.00–141.00WISC VCI105.04 (15.87)70.00–155.00108.53 (16.06)70.00–146.00104.44 (15.34)70.00–155.00102.29 (15.71)70.00–142.00110.58 (13.93)78.00–155.00WISC FRI101.33 (16.19)67.00–144.00104.58 (15.40)67.00–140.0099.14 (15.75)67.00–134.00100.90 (16.98)67.00–144.00107.42 (15.11)69.00–155.00WISC WMI98.54 (15.18)62.00–138.0099.53 (14.83)67.00–138.0097.21 (14.04)65.00–138.0099.29 (16.77)62.00–135.00103.92 (14.29)72.00–135.00WISC PSI93.65 (15.35)53.00–148.0097.33 (15.29)56.00–148.0093.12 (13.94)56.00–123.0090.59 (16.38)53.00–132.00106.55 (15.80)66.00–155.00For each group and measure, the mean, standard deviation (in brackets), and range are provided.
T1-weighted anatomical and functional MR images were acquired while participants watched a ten-minute clip from the movie ‘Despicable Me’ (Alexander et al., 2017). Neuroimaging data were preprocessed and analyzed using the Automatic Analysis (AA) toolbox (Cusack et al., 2015), SPM 8, and in-house MATLAB scripts (see Supplement for additional details).
We generated functional connectivity matrices for each participant using 264 regions-of-interest (ROI) as defined in the Power et al. (2011) atlas (Power et al., 2011). Individual ROIs comprised of spheres of 5 mm in radius with spatial smoothing full-width half maximum of 6 mm and z-score standardization. We correlated activity in each sphere to every other sphere, resulting in a 264 ×264 functional connectivity matrix.
Cognitive ability was measured using the Wechsler Intelligence Scale for Children Fifth Edition (Wechsler Intelligence). The WISC-V measures a child’s intellectual ability based on five primary Visual Spatial Index (VSI), Verbal Comprehension Index (VCI), Fluid Reasoning Index (FRI), Working Memory Index (WMI), and Processing Speed Index (PSI). In addition, the WISC-V also provides a Full-Scale IQ (FSIQ) score, which is derived from the five primary indices, and normalized by age (description of the cognitive measures can be found in the Supplement).
We used two computational models to examine the relationship between functional connectivity and cognitive partial least squares (PLS) (McIntosh and Lobaugh, 2004, Xie and Redcay, 2022) and Ridge regression (Tavor et al., 2016, Dupré la Tour et al., 2022). Ridge and PLS models are ideal for high-dimensional multicollinear data and have built-in anti-overfitting (regularization) parameters (details about optimal components and alpha values in Supplement).
We first fit standard scaler models to rescale features such that they have the properties of a standard normal distribution with a mean of zero and a variance of one. This is essential because regularized linear models assume features are centered around zero and have variance in the same scale to avoid certain features dominating because of differences in variance. To avoid data leakage between the training and testing set, we fit the standard scaler only on the training set, and it was applied to both the training and testing set.
After we trained the models, we analyzed the model’s feature weights using two methods. First, we assessed feature weight reliability between different computational models using the intraclass correlation coefficient (Tian and Zalesky, 2021) (ICC). In the second method, we used feature weights trained on one subset of the dataset and applied them to predict cognition on a different subset. We based prediction accuracy on Pearson correlations representing the degree of similarity between the model’s predicted values of cognitive ability and the true values. We calculated statistical significance by comparing the observed Pearson r score relative to a null distribution of Pearson r scores generated from 500 random permutations of the dataset. We performed this out-of-sample cross-validation—referred to as cross-prediction—only on the ADHD group and evaluated it using permutation statistics (Supplement).
Model feature weights represent the weight (importance) associated with specific aspects of the functional connectivity matrix. We multiplied the feature weights by a participant’s functional connectivity matrix and used this to calculate the predicted score across all cognitive measures. We then compared this score against the participant’s actual cognitive score to update the feature weights. Thus, the feature weights represent a heat map of important functional connections for predicting cognitive ability.
To explore changing neural mechanisms associated with cognitive development, we subtracted feature weights of models trained on early childhood (Bin 1) from models trained on middle childhood (Bin 2), with respect to cognitive ability. We used absolute values to highlight the magnitude of change between Bin 1 and Bin 2 feature weights. The absolute-value-feature-weight-differences matrix represents the network connections that change in importance between Bins 1 and 2. Large differences (top ten most dissimilar feature weight) represent “distinct" functional connections, while small feature-weight differences (top ten most similar) represent “shared” network profiles. Both the distinct and shared network profiles are important when considering cognitive development as connections that change are equally important as connections that do not change between early and middle childhood. We also computed interclass-correlations to examine developmentally meaningful changes in the feature-weights associated with cognitive abilities across development. We used a split-half approach whereby within age-bin ICC was computed by splitting the sample in half. That is, age bin 1 was split in half (across five iterations) and ICC values were computed between the two halves. The same approach was applied to age bins 2 and 3. Between age-bin ICC values were computed between the two halves of one age bin (e.g., age bin 1) with a sized matched sample in the adjacent bin (e.g., two halves in age bin 1 compared to age bin 2).
Using Ridge regression, we predicted the age (r=0.67, p<0.001) and sex of individuals in the ADHD group (n=373) with an accuracy of 74 % (p<.001). Age (r=0.36, p<0.001) and sex (60 % accuracy, p=0.01) were also predicted in the NT group (n=106). We class-balanced the sex prediction to ensure that the model was not constantly predicting the most prevalent sex. Moreover, we found that mean framewise displacement (scanner motion) was not significantly different across the three ADHD age bins and the NT group (for all pairwise comparisons, t < 1.8832, p > 0.05; see Fig. 1 in Supplementary Material).Fig. 1**: Processing stages for the neuroimaging data.** There are three overall stages to the data preprocessing, modeling, and analysis. Preprocessing involved correcting the raw MRI and fMRI data for motion, coregistering the structural and functional images, normalizing to a standard template, generating a functional connectivity matrix, and splitting the participants by age or diagnosis. Next is modelling and it starts with searching for the optimal parameters for the model, then training and validating the model using the functional connectivity matrices, randomly permutating the data, and ends with extracting the model’s feature weights. Lastly, the feature weights were analyzed by calculating the intraclass correlation coefficient, using the weights to cross-predict cognition in a different age bin, and visualizing the feature weights.Fig. 1
Using Ridge regression, we found the model could predict FSIQ (r=0.38, p=.002), VSI (r=0.31, p=.002), VCI (r=0.39, p=.002), FRI (r=0.30, p=.002), and WMI (r=0.21, p=.004), but failed to predict PSI (r=0.05, p=.26) in the group of participants diagnosed with ADHD (n=373). Conversely, we could not predict FSIQ (r=0.04, p=.42), VSI (r=0.16, p=.11), VCI (r=0.20, p=.05), FRI (r=-0.07, p=.73), WMI (r=0.12, p=.21), and PSI (r=-0.06, p=.70) in the NT group (n=106). These p-values were corrected for multiple comparisons using the max-statistic method (Nichols and Hayasaka, 2003). To determine whether these results were driven by model choice (Table 2), the analysis was replicated using a partial least squares (PLS) model. We found no difference in performance between the two models, except that VCI (r=0.23, p=.04) could be predicted in the NT group using the PLS model. The similarity between the two models was further supported by the ICC analysis which showed the weights produced by the Ridge and PLS model were strongly correlated (> 0.90; Supplement for additional details). Based on these results, we excluded the NT group, and used Ridge Regression for all subsequent analyses.Table 2Scores for predicting six cognitive abilities in ADHD and NT using partial least squares and Ridge regression.Table 2ADHD (n=373)NT (n=106)PLSRidgePLSRidgeWISC Primary IndexPearson rP-valuePearson rP-valuePearson rP-valuePearson rP-valueIntelligence Quotient (FSIQ)0.37.0020.38.0020.04.3880.04.417Visual Spatial (VSI)0.28.0020.31.0020.14.1290.16.107Verbal Comprehension (VCI)0.37.0020.39.0020.23*.0350.20.052Fluid Reasoning (FRI)0.30.0020.30.002*−0.07.737−0.07.734Working Memory (WMI)0.17.0110.21.0040.08.2590.12.213Processing Speed (PSI)0.06.2270.05.257−0.10.791−0.06.698The Pearson r correlations test score represents the linear correlation between the model’s predicted values of the cognitive ability and the true values. The p-value was calculated by comparing the observed Pearson r score to a null distribution of Pearson r scores generated from 500 random permutations of the dataset. Both the PLS and Ridge models predicted FSIQ, VSI, VCI, FRI, and WMI in the ADHD group at significance (p<.011) but failed to predict PSI (p=.23). For the NT group, only VCI was predicted at significance (p=.04) using PLS. PLS and Ridge achieved similar Pearson r correlation scores on both the ADHD and NT groups.*values indicate statistically significant at p<.05 max-statistic corrected.
Across all cognitive measures, models consistently assigned the largest positive weights to connections within three memory retrieval, dorsal attention network, and sensory/somatomotor (mouth), while inter-network connections with the largest positive weights were between memory retrieval and dorsal attention, memory retrieval and frontoparietal and between memory retrieval and cerebellar networks. The largest negative weights were commonly assigned to connections between frontoparietal and visual networks, between dorsal and ventral attention networks, and to networks connected with subcortical areas.
To examine developmental changes in the relationship between neural connectivity profiles and cognitive ability, we divided the ADHD group into three age bins (Table 3). The model successfully predicted FSIQ (r = 0.27, p = 0.02), VSI (r = 0.24, p =0.02), and VCI (r = 0.22, p = 0.03), but not FRI and WMI (p > 0.05) for Bin 1 (ages 6–8); and FSIQ (r = 0.35, p = 0.002), VSI (r = 0.21, p=.02), VCI (r = 0.35, p = 0.002), FRI (r = 0.31, p = 0.004), and WMI (r = 0.29, p = 0.004) for Bin 2 (ages 9–11). The model did not predict any WISC-V measure (p > 0.17) for individuals in Bin 3 (ages 12–16). We found similar results using a smaller sample of Bin 2 (n=113) that matched the sample sizes of Bins 1 and 3; the model could predict FSIQ (r = 0.37, p = 0.002), VSI (r = 0.27, p = 0.01), VCI (r = 0.37, p = 0.002), FRI (r = 0.30, p = 0.006), WMI (r = 0.35, p = 0.002), but not PSI (r = 0.04, p = 0.38). The feature weights are shown in Fig. 2.Table 3Scores for predicting six cognitive abilities in ADHD across three age bins using Ridge.Table 3Age Bins****Bin 1 (n=114)****Bin 2 (n=147)****Bin 3 (n=112)****WISC Primary IndexPearson rP-valuePearson rP-valuePearson rP-valueIntelligence Quotient (FSIQ)0.27.0190.35.0020.11.177Visual Spatial (VSI)0.24.0170.21.0210.09.229Verbal Comprehension (VCI)0.22.0270.35.0020.04.403Fluid Reasoning (FRI)0.05.3470.31.004*−0.01.518Working Memory (WMI)0.05.3570.29.0040.10.183Processing Speed (PSI)−0.09.7920.06.2630.09.257Bin 1 represents early childhood (ages 6–8), Bin 2 represents middle childhood (ages 9–11), and Bin 3 represents adolescence (ages 12–16). The Ridge model successfully predicted FSIQ, VSI, and VCI in Bin 1 (p<.03); FSIQ, VSI, VCI, FRI, and WMI in Bin 2 (p<.02); and no cognitive ability in Bin 3 (p>.17).Values indicate statistically significant at p<.05 max-statistic corrected.Fig. 2: Feature weights used to predict five cognitive abilities in the entire ADHD group, Bin 1, and Bin 2.** Each row represents one of five WISC FSIQ, VSI, VCI, FRI, and WMI. Each column represents one of three ADHD All (ages 6–16), Bin 1 (ages 6–8), and Bin 2 (ages 9–11). Each scale applies to the feature weight matrices in the row. A feature weight matrix represents the average feature weight for all connections between two networks shown for all networks. Darker cells in the feature weight matrix represent more extreme values, while lighter cells represent values closer to zero. Red cells represent positive values (increases in value for that network connection increased the predicted cognitive score), while blue cells represent negative values (increases in value for that network connection decreased the predicted cognitive score). Diagonal cells represent intranetwork connections, while off-diagonal cells represent internetwork connections. The networks are visual (VIS), frontoparietal task control (FPN), default mode (DMN), sensory/somatomotor (hand; SMH), sensory/somatomotor (mouth; SMM), cingulo-opercular task control (CON), auditory (AUD), salience (SAL), memory retrieval (MEM), ventral attention (VAN), cerebellar (CER), subcortical (SUB), and dorsal attention (DAN). Row four represents a summary of connections weights across the networks based by taking the average feature-weights for each group (ADHD: All, Bin 1 and Bin 2) across the different cognitive measures.Fig. 2
The feature weights for FSIQ, VSI, and VCI had positive weights for network connections between memory retrieval and dorsal attention, cingulo-opercular and sensory/somatomotor (mouth) networks, and within memory retrieval, sensory/somatomotor (mouth) networks. Negative weights were learned for connections between dorsal and ventral attention, memory retrieval and sensory/somatomotor (mouth), and between cerebellar and sensory/somatomotor (mouth) networks. For the Bin 2 feature weights, we found a general pattern of less-extreme feature weights (fewer darker-colored cells) across all cognitive measures (Fig. 2, right column) relative to the entire ADHD group and Bin 1. This suggests that the model is not relying on specific network connections, but instead is using a distributed approach to predict cognitive ability. However, the model identified strong negative weights for connections within the sensory/somatomotor (mouth) network associated with predicting VCI scores, implying this network is deemphasized for predicting VCI performance. Interestingly, this connection was assigned a large positive in Bin 1, which shows that the models switched from a positive to a negative weight from Bin 1 to Bin 2 when predicting VCI (see Supplement for additional details).
Using cross-prediction (out-of-sample cross-validation), we found that models trained on Bin 1 and tested on Bin 2 successfully predicted FSIQ (r=0.33, p=.002), VSI (r=0.36, p=.002), VCI (r = 0.32, p = 0.002), and FRI (r = 0.15, p = 0.02) (Fig. 3). We also found the reverse; a model trained on Bin 2 and tested on Bin 1 successfully predicted FSIQ (r = 0.36, p = 0.002), VSI (r = 0.40, p = 0.002), VCI (r = 0.30, p = 0.002), and FRI (r = 0.20, p = 0.01). However, models failed to cross-predict WMI (r = 0.03, p = 0.35) and PSI (r = 0.07, p = 0.18) when trained on Bin 1 and tested on Bin 2, and when trained on Bin 2 and tested on Bin 1; WMI (r = 0.03, p = 0.37) and PSI (r = 0.03, p = 0.40). These results suggest connectivity patterns associated with FSIQ, VSI, VCI, and FRI, but not WMI and PSI, generalize from early to middle childhood.Fig. 3**: Scores for cross-predicting six cognitive ability between Bin 1 and Bin 2.** For each matrix, rows represent the age bin (Bin 1 or Bin 2) the model was trained on, while columns represent the age bin (Bin 1 or Bin 2) the model was tested on. The top-left to bottom-right diagonal represents training and testing the model within the same age bin (same scores as in Table 3), while the bottom-left to top-right diagonal represents the training the model on Bin 2 and testing on Bin 1 and training the model on Bin 1 and testing on Bin 2 respectively. Values within each cell are the Pearson r correlation test score and represent the linear correlation between the model’s predicted values of the cognitive ability and the true values. Purple cells indicate statistically significant at p<.05 after being corrected for multiple comparisons using the max-statistic method, while grey cells indicate not statistically significant.Fig. 3
To identify the most similar and dissimilar feature weights that were trained on Bin 1 and Bin 2, we subtracted (using absolute values) the Bin 2 feature weights from the Bin 1 feature weights for each cognitive measure (e.g., for FSIQ, VSI, VCI, FRI, and WMI). We found the feature weight profiles with the (top ten) most similar networks between early childhood (Bin 1) and middle childhood (Bin 2) across all cognitive measures comprised of four intra-network the frontoparietal, default mode, subcortical, and dorsal attention networks. The feature weights associated with inter-network connections that were most similar between the two age groups primarily included the frontoparietal, default mode, subcortical, and salience structures, although other networks were also found (but to a lesser degree) to be shared between age groups.
Conversely, relatively more intra-network connections were dissimilar between the two age groups across the cognitive measures, such as the sensory/somatomotor (mouth), cingulo-opercular, and memory retrieval networks, but also included cerebellar and ventral attention networks. Most dissimilar (top ten) inter-network connections included the memory retrieval, dorsal attention, sensory/somatomotor networks (mouth and hand) networks. Moreover, connections between cingulo-opercular network and other parts of the brain were more often shared than not between the age groups across the different cognitive measures. Note, the model was not able to predict FRI and WMI in Bin 1 but was able to predict FRI and WMI in Bin 2, which does not reflect direct comparisons of specific cognitive abilities between the age groups (Fig. 4).Fig. 4**: Difference in feature weights between Bin 1 and Bin 2 for five cognitive abilities.** Each row represents one of five WISC FSIQ, VSI, VCI, FRI, and WMI. The left column (grey) represents all feature weight differences between Bin 1 and 2, the center column (pink) represents network connections with the most dissimilar values for Bin 1 and 2 (“distinct” networks), and the right column (green) represents network connections with the most similar values between Bin 1 and 2 (“shared” networks). The distinct network profiles were obtained by thresholding all feature weight differences between Bins 1 and 2 by the ten largest differences. The shared network profiles were obtained by thresholding all feature weight differences between Bins 1 and 2 by the ten smallest differences. For the left and center columns, darker cells represent a larger difference between the feature weights assigned to Bin 1 and 2 when predicting cognition, while lighter cells represent a smaller difference. Diagonal cells represent intra-network connections, while off-diagonal cells represent internetwork connections.Fig. 4
To validate the developmentally distinct predictive accuracy results, we computed the same analysis using a different age binning structure. That is, we increased the age resolution (sample matched), using a sliding window approach, (6–8 (N=69), 7–9 (N=75), 8–10 (N=75), 9–11 (N=75), 10–12 (N=75), 11–13 (N=75) and 12–15 (N=75) and computed prediction accuracy for four cognitive abilities (full scale IQ, verbal comprehension, working memory, and processing speed). We found that in the two youngest (6–8 and 7–9) and the two oldest age bins (11–13, and 12–15) the model could not significantly predict cognition (r<0.21, p> 0.05), but significantly predict cognition all 3 cognitive abilities in the next three age bins (8–10, 9–11, and 10–12; r > 0.26, p< 0.015) other than working memory in 8–10 year olds (r = 0.11, p>0.05). Confirming our results, at no age bin could the model predict processing speed (Fig. 5)Fig. 5**: Prediction accuracy across age bins using a sliding window approach.** Y-axis represent prediction accuracy (correlation between predicted score across four cognitive tests) and actual individual performance on each test. X-axis represent smaller age bins (increased resolution) organized in a sliding window 6–8 (N=69), 7–9 (N=75), 8–10 (N=75), 9–11 (N=75),10–12 (N=75), 11–13 (N=75) and 12–15 (N=75). Prediction accuracy across all age bins for four cognitive 1) Full-scale IQ (black), Verbal comprehension (dark gray), Working memory (light gray) and Processing Speed (orange).Fig. 5
We also examined whether the assigned feature-weights for age-dependent models (based on the three age bins) represent distinct developmental differences. Using a split-half approach, we computed within age-bin ICCs by randomly splitting the sample in half (this was repeated for each age bin). We also computed ICC values for the two halves of each age bin with the adjacent age bin (matched for sample size). That is, we computed ICC values for the two halves for age bin 1 with a matched sample in age bin 2 (means plotted in Fig. 6), this was repeated for age bins 2 and 3. We found that ICC values for split-half within group comparisons were larger (other than bin 1 to bin 2 for VCI) than between group comparisons for across different cognitive measure (Fig. 6). Bin 3 (ages 12–16) results were not plotted because ICC values for within and between comparisons were not significant; all other ICC values were significant (p < 0.05).Fig. 6**: Interclass correlations across age bins.** Y-axis represents interclass correlations (ICC) values (p<.05) for within and between age-bin comparisons for four cognitive measures; full scale IQ (black), verbal comprehension (dark grey), working memory (light grey) and fluid reasoning (steel grey). Each within age-bin values represent the mean of five random iterations, and between age-bin values represent the mean of the two halves, across five iterations.Fig. 6
We demonstrated that applying machine learning to movie-watching fMRI data is a viable tool for predicting demographic and higher-level cognitive abilities in children and adolescents diagnosed with ADHD. In a large cohort of early childhood, middle childhood, and adolescent participants, we built models that successfully predicted age and sex, and we identified shared and distinct neural mechanisms associated with different aspects of higher-level cognition across development in ADHD and NT groups.
Establishing models that can predict age and sex is important because it demonstrates that a dimensional data-driven approach (i.e., machine learning) can be used to extract information from the neural connectivity profile to predict aspects of development. The distinct feature-weight profile used by the model likely reflects that male and female (sex was limited to these two categories) children and adolescents have distinct functional patterns of brain activity and are relying on different neural mechanisms to process the movie. This result replicates and expands on previous work that generated models to predict age and sex (Tian and Zalesky, 2021, Dosenbach et al., 2010, Fair et al., 2013, Rudolph et al., 2017) providing further evidence that biological properties, such as age and sex, could be reliably localized to specific important connections in the brain. Does the same apply to cognition?
Emerging from our results was predicting higher-level cognitive abilities followed an inverted-U pattern. That is, that model prediction was highest in the second age bin (ages 9–11), slightly weaker in the youngest age bin (6−8) and weakest in the oldest age bin (12−16). This pattern of results was consistent when we increased the resolution of the age-bins and replicated our analysis using a sliding window approach; prediction accuracy across different measures of cognition was highest for individuals in middle to late childhood (between the ages of 8 and 12). This may suggest that the link between intra brain functional connectivity in response to movie watching and cognitive ability is strongest and most consistent during middle childhood, weaker and more variable during early childhood and not detectable during adolescence. Although successfully predicting cognitive abilities based on neuroimaging data is consistent with previous studies (in neurotypical children (Sripada et al., 2020b)), we were surprised that the same cognitive systems could not be predicted in adolescence with ADHD. One reason for this pattern of results is that the link between functional connectivity and cognitive ability in adolescence may be too weakly represented in the data. This could reflect more variable cognitive development in this group that was not equally elicited by the movie. Another interesting possibility is that the nature of the relationship between cognition and connectivity profiles do not follow a linear trajectory and linear models are insufficient to capture the link.
Although we could generate models that predicted different cognitive abilities in early and middle childhood, not all cognitive systems were reliably predicted between the cohorts. Beyond that, the models that best predicted cognitive abilities were different for participants in early versus middle childhood. For example, the model could predict fluid reasoning and working memory in middle childhood but not in early childhood. Although previous work has suggested that verbal and visuospatial working memory remain relatively distinct in children ages 4–11 (Alloway et al., 2006), our results suggest this may not be the case. One potential reason why we could predict working memory in middle, but not early childhood, is because the link between these cognitive abilities and the underlying neural mechanisms are either weaker or follow distinct developmental trajectories during this period in children with ADHD (Moura et al., 2019, Mayes and Calhoun, 2006). Indeed, maturing working memory and fluid reasoning are associated with the development of the frontoparietal network (Otero, 2017, Ullman et al., 2014), and atypical development of this network is associated with difficulties in fluid reasoning and working memory in children between the ages of 6 and 12 with ADHD (Van Breukelen, 2006, Sun et al., 2020, Silk et al., 2008).
One similarity across the three developmental stages was that processing speed could not be predicted well in children and adolescents with ADHD. This is likely due to children with ADHD showing the most pronounced deficits in processing speed (Moura et al., 2019, Mayes and Calhoun, 2006). Delayed or more variable development of this cognitive ability may suggest that there is a weakened relationship between neural activity associated processing speed ability, or the representation of processing speed in brain activity may have greater variability, resulting in poorer prediction scores. Similarly, previous work found the lowest prediction scores on measures of speed/flexibility out of three higher-order cognitive functions (General Ability, Speed/Flexibility, and Learning/Memory) using resting-state fMRI data in 9- to 10-year-old children (Sripada et al., 2020b). These results suggest that neither movie-watching or resting-state fMRI is ideal for capturing the neural mechanisms related to processing speed in neurotypical children (Diamond, 2013) or children with ADHD.
We hypothesized models identifying unique network patterns associated with cognitive ability in one age cohort would not generalize to other cohorts. However, this was not the case. Instead, connectivity patterns associated with IQ, visual spatial, verbal comprehension, and fluid reasoning generalize from early to middle to childhood. The shared networks for predicting cognition between early and middle childhood were divided into two intra-network (within) connections and inter-network (between) connections. Shared intra-network connections were predominately made up of the frontoparietal, default mode, subcortical, and dorsal attention, but also included sensory (e.g., visual and auditory) networks. The shared inter-network connections were comprised primarily of the frontoparietal, default mode, memory retrieval, dorsal attention, and salience networks. This is not to say these network connections remain stable during this period; some developmental changes may not be tied to cognitive ability. What the shared network connections do imply is that the models did not change their importance for these network connections when predicting cognition for early and middle childhood. Why would the models highlight these networks? One possibility is that many of the shared networks, which have been linked to higher-level cognitive processing—such as the frontoparietal, memory retrieval, dorsal attention, and salience networks—bridge cognitive maturity and the degree to which they are recruited during movie watching is similar between the two age groups. That is, young children with greater scores on cognitive abilities are recruiting (or not recruiting) these networks during movie watching to the same degree as older children, while the same relationship is true for early and middle childhood participants with lower scores on cognitive abilities.
Although we found many important shared network connections between the two age groups, relatively low explained variance suggests that the shared networks do not capture all, or even most, of the developmental neural mechanisms supporting higher-level cognition in early and middle childhood. The distinct networks for predicting cognition between early and middle childhood were found primarily within the sensory/somatomotor (mouth), cingulo-opercular, and memory retrieval networks, but also included subcortical, cerebellar, and ventral attention networks, and between the sensory/somatomotor networks (mouth and hand), dorsal attention, and memory retrieval networks. In line with our results, the sensory/somatomotor (hand) and memory retrieval networks connections were important when predicting general cognitive ability in a cohort of middle childhood participants (Sripada et al., 2020b). This suggests the network connectivity profile of the sensory/somatomotor (hand), and memory retrieval networks are strongly linked to cognitive development from early to middle childhood. Contrary to our hypothesized outcomes, the connectivity profile in frontoparietal network was not distinctly associated with cognition in early versus middle childhood. Although the frontoparietal network is strongly linked to the development of executive function and intelligence (Baum et al., 2017, Deary et al., 2010), it’s possible that Despicable Me did not trigger the cognitive systems mediated by the frontoparietal network in the youngest two cohorts in our study. Another possibility is that movie-watching in general may not be a context that is sensitive enough to extract neural features associated with executive functioning in young children. Related to both points, previous studies found that the frontoparietal network is not a flexible hub during movie watching (Caldinelli and Cusack, 2022) to the same degree it has been reported to be when participants complete a set of demanding tasks (Cocuzza et al., 2020, Cole et al., 2013). Therefore, the bridge between frontoparietal activity associated with executive functioning during movie-watching and the WISC scales may not be sufficiently strong to find patterns of activity associated with executive functioning development.
Importantly, we were able to replicate all findings using a different partial least squares. In fact, we found a very high correspondence between the feature weights generated by Ridge and partial least squares as measured by the high intraclass correlations. This suggests that the models’ ability to predict cognition is likely not driven by model choice as both the model output (its correlation score) and the model internals (its feature weights) are extremely similar between Ridge and partial least squares. In line with our results, previous studies also reported finding little difference between a Lasso model and Ridge’s correlation score when predicting fluid and crystalized intelligence (Tian and Zalesky, 2021). Thus, perhaps in the space of regularized linear models, the choice of model does not lead to significant performance differences.
One limitation of the current study is that we were unable to predict cognition in age-matched neurotypical children, despite previous studies demonstrating that cognition can be predicted in this populations (Tian and Zalesky, 2021, Finn and Bandettini, 2021, Sripada et al., 2020b). One potential reason is the NT group was smaller than the ADHD group. However, this likely does not account for our results because we were able to predict some cognitive measures in the ADHD group with a comparable sample. Another factor might be data quality; perhaps the noisier data for the NT group was leading to poor predictive performance. This is also an unlikely to account for our results because we could predict age and sex in the NT group. However, the explained variance and accuracy was lower in the NT group compared to the ADHD group, and lower than estimates from other studies (approximately 42 % explained variance but using a different task (Rudolph et al., 2017)). To determine whether our findings are specific to ADHD or generalize to other groups of children, future studies examining distinct neural mechanisms associated with cognitive development in neurotypical populations should replicate our findings using larger samples. It would also be valuable for future work to examine whether neural mechanisms associated with cognitive development are modulated by symptom severity, and in doing so identify differential milestones in neurocognitive development in children with ADHD relative to their neurotypical peers. Furthermore, different models would be valuable, such as those incorporating non-linear relationships between connectivity profiles and cognition, and lesion-modeling (Hebling Vieira et al., 2021) that examine changes in the direction of the relationship between neural mechanisms and cognition across development.
Different higher-order cognitive abilities in a large group of children and adolescents diagnosed with ADHD could be predicted using functional neural activity during movie watching. Prediction scores do not remain constant across development but instead follows an inverted-U developmental trajectory from early childhood to adolescence, and that certain neural mechanisms linked to higher-level cognition were shared we also found several distinct sets of neural mechanisms for predicting cognition between early and middle childhood.
Bobby Stojanoski: Writing – review & editing, Writing – original draft, Visualization, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Yalda Mohsenzadeh: Writing – review & editing, Visualization, Supervision, Methodology, Formal analysis, Conceptualization. Sara Saljoughi: Writing – review & editing, Visualization, Formal analysis. Ryan Stevenson: Writing – review & editing, Formal analysis, Conceptualization. Brian Pho: Writing – review & editing, Writing – original draft, Visualization, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization.
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests Bobby Stojanoski reports financial support was provided by Natural Sciences and Engineering Research Council of Canada. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.