Authors: Tau Ming Liew, Jessica Yi Hui Foo, Howard Yang, Sze Yan Tay, Way Inn Koay, King Fan Yip, Simon Kang Seng Ting, Kaavya Narasimhalu, Weishan Li, Congyuan Tan, Danlin Luo, Rebecca Chong, Rachel Shong, Christopher Sia, Gerald Choon-Huat Koh, Julian Thumboo
Categories: Article, Geriatrics, Alzheimer's disease
Source: Nature Communications
Authors: Tau Ming Liew, Jessica Yi Hui Foo, Howard Yang, Sze Yan Tay, Way Inn Koay, King Fan Yip, Simon Kang Seng Ting, Kaavya Narasimhalu, Weishan Li, Congyuan Tan, Danlin Luo, Rebecca Chong, Rachel Shong, Christopher Sia, Gerald Choon-Huat Koh, Julian Thumboo
Undiagnosed cognitive impairment is a pervasive global issue, often due to subtle nature of early symptoms, necessitating the use of brief cognitive tests for early detection. However, most brief tests are not scalable (requiring trained professionals), and are not designed for lower literacy groups (e.g. in underserved communities). Here, we developed PENSIEVE-AI^TM^, a drawing-based digital test that is less dependent on literacy, and can be self-administered in <5 min. In a prospective study involving 1758 community-dwelling individuals aged 65 and older from Singapore (education range = 0–23 years), our deep-learning model showed excellent performance in detecting clinically-adjudicated mild cognitive impairment and dementia (AUC = 93%), comparable to traditional neuropsychological assessments (AUC = 94%, Pcomparison = 1.000). Results were consistent even across education subgroups. Being less dependent on literacy, PENSIEVE-AI holds promise for broader deployment in literacy-diverse populations similar to Singapore (e.g. some Asian and lower- and middle-income countries), potentially improving early detection and intervention of cognitive impairment.
Undiagnosed cognitive impairment (CI) is a global challenge^1^, with 60–90% of individuals with CI never receiving a formal diagnosis^2,3^. Individuals with undiagnosed CI miss out on timely clinical care^4^ (e.g. cognitive enhancers, behavioral management, and caregiver support)^5–8^, which can affect their well-being^9,10^ and increase their risk of premature nursing home placement^11–13^. They may also not receive adequate support to manage and coordinate the care of their chronic diseases^14,15^, resulting in suboptimal disease management, inappropriate healthcare utilization, and higher healthcare costs^16,17^. Recently, the importance of early diagnosis has been further underscored by growing literature on early interventions for CI^18,19^, such as risk factor modification^20^ and anti-amyloid monoclonal antibodies^21,22^.
Early symptoms of CI are often subtle. Without objective cognitive tests, these symptoms are easily mistaken for normal ageing^1,5,23,24^. To address this inherent challenge, various international bodies^23–25^ have advocated the use of brief cognitive tests to facilitate case-finding among high-risk individuals in the community^25^. Although many brief cognitive tests exist in the literature (e.g. Montreal Cognitive Assessment^26^, Mini-Mental State Examination^27^, Mini-Cog^28^, Memory Impairment Screen^29^, Brief Cognitive Assessment Tool^30^), most are labor-intensive and require trained professionals^1,23,24,31^, which limit their scalability in community settings. Equally important, most tests were developed in populations with high literacy (e.g. White populations)^32^, and are predicated on the assumption that respondents are able to read and write in a language^33^. This may limit the usefulness of cognitive tests in underserved communities with lower literacy (e.g. in some non-White communities, and in lower- and middle-income countries [LMICs]), which often have the largest number of individuals with undiagnosed CI^32,34^. This has also led to call by the 2024 Lancet Commission on dementia care^32^ to address the unmet need for brief cognitive tests that are suited for individuals with lower literacy.
Digital cognitive tests hold promise as scalable tools for detecting CI in community settings, by leveraging artificial intelligence (AI) to automate the administration and scoring of brief cognitive assessments^35^, thereby reducing dependence on trained professionals in case-findings efforts. However, despite their potential, digital cognitive tests is still a relatively nascent field^35^. Few digital tests have undergone rigorous validation for the detection of CI in community settings^36^, especially in populations with lower literacy^37^. To address the unmet need for scalable case-finding tools that are suited for lower literacy groups, we have purpose-built an AI-based digital cognitive test (denoted as PENSIEVE-AI^TM^) which has the following Designed to be self-administered (using touch-screen tablets and pre-recorded audio instructions), thus reducing dependence on trained professionals.Takes <5 min to complete (comprising only four drawing tasks), making it well-suited as a brief case-finding tool in community-settings.Relies on drawing tasks alone, thus reducing dependence on respondents’ ability to read or write in a language^33^, and potentially allowing broader implementation in communities with varying literacy (such as in Singapore and other Asian populations). Arguably, drawing tasks can still be affected by literacy level^38,39^; but they are among the earliest skills that individuals develop before learning to read or write in a language, with the ability to draw shown to pre-date language development even in human civilizations^33^.
Using a large, community-representative sample from Singapore, this study aimed Train an image-based deep-learning model to detect mild cognitive impairment and dementia (MCI/dementia) using the four drawing tasks in PENSIEVE-AI.Examine the effects of key demographic features (e.g. education, test language) in improving model performance, given prior literature on the potential influence of these features on drawing tasks^38,39^.Compare the performance of the deep-learning model to several commonly used assessment tools in detecting MCI/dementia, across participants with lower and higher literacy.
Of note, as a city-state in South-East Asia, Singapore offers a unique testbed to develop the new digital tool. Its 6-million-strong population serves as a microcosm of Asia, representing an amalgamation of Asian culture and comprising multiple Asian ethnicities, including Chinese, Malay, Indian, and other ethnic groups^40^. This diversity provides a robust testing ground for assessing the new tool’s performance across varied cultural and linguistic backgrounds. Additionally, the current cohort of older individuals in Singapore witnessed the country’s transformation from a traditional, lower-income, Asian society to a more westernized, higher-income country^41^. Consequently, this cohort of older Singaporeans encompasses a wide range of educational backgrounds, from minimal formal education to tertiary education. By validating the digital tool in such a heterogeneous population, we sought to demonstrate its potential for broader implementation in similar multiethnic and literacy-diverse settings beyond Singapore, such as in populations across East and South Asia, and potentially in some LMICs.
A total of 1758 participants were included (Table 1), with 239 (13.6%) having clinically-adjudicated MCI/dementia. Given the nature of community recruitment, most cases were in early stages of CI (CDR global ≤1). Participants had a median age of 72 years and a median education of 10 years. Most participants could self-administer PENSIEVE-AI in <5 min (i.e. 69.1% self-administered; and 77.0% completed in under five minutes), with a median completion time of 3.7 min. However, participants with MCI/dementia were more likely to need some supervision to navigate the digital interface, and took longer to complete PENSIEVE-AI (4.6–6.7 min).Table 1Characteristics of the study participants (n = 1758)VariableOverall sample(n = 1758)Normal cognition(n = 1519)MCI(n = 195)Dementia(n = 44)P value^a^Age, median (IQR) [range]72 (68, 76)[65, 101]71 (68, 75)[65, 93]74 (69, 80)[65, 91]80 (76, 82)[66, 101]2.22 × 10^-19^Years of education, median (IQR) [range]10 (9, 12)[0, 23]10 (10, 13)[0, 23]10 (6, 12)[0, 21.5]10 (2, 10)[0, 17]5.73 × 10^-11^Male sex, n (%)640 (36.4)532 (35.0)95 (48.7)13 (29.5)5.77 × 10^-4^Ethnicity, n (%)0.346 Chinese1644 (93.5)1427 (93.9)178 (91.3)39 (88.6) Malay/Indian90 (5.1)73 (4.8)13 (6.7)4 (9.1) Eurasian/Others24 (1.4)19 (1.3)4 (2.1)1 (2.3)MoCA total score, median (IQR)26 (24, 28)27 (25, 28)21 (17, 24)14 (10, 18)2.87 × 10^-85^NTB Global Z-score, median (IQR)-0.2 (-0.6, 0.1)-0.1 (-0.4, 0.2)-1.0 (-1.3, -0.7)-1.6 (-2.0, -1.2)3.70 x 10^-106^Global CDR, n (%)3.25 × 10^-115^ 01488 (84.6)1475 (97.1)13 (6.7)0 (0.0) 0.5241 (13.7)44 (2.9)182 (93.3)15 (34.1) 121 (1.2)0 (0.0)0 (0.0)21 (47.7) 27 (0.4)0 (0.0)0 (0.0)7 (15.9) 31 (0.1)0 (0.0)0 (0.0)1 (2.3)iAD8, median (IQR)0 (0, 1)0 (0, 1)2 (0, 4)6 (4, 8)2.03 × 10^-42^Test language selected in PENSIEVE-AI, n (%)1.56 × 10^-5^ English1119 (63.7)999 (65.8)100 (51.3)20 (45.5) Mandarin Chinese639 (36.3)520 (34.2)95 (48.7)24 (54.5)Mode of administration for PENSIEVE-AI, n (%)3.28 × 10^-24^ Self-administered (i.e. no supervision needed)1214 (69.1)1112 (73.2)95 (48.7)7 (15.9) Minimal supervision by research coordinator544 (30.9)407 (26.8)100 (51.3)37 (84.1)Completion of PENSIEVE-AI in <5 min, n (%)1354 (77.0)1234 (81.2)109 (55.9)11 (25.0)2.53 × 10^-29^Time to complete PENSIEVE-AI (minutes), median (IQR)3.7 (3.1, 4.8)3.6 (3.0, 4.5)4.6 (3.5, 6.1)6.7 (5.1, 8.8)9.78 × 10^-26^MCI mild cognitive impairment, IQR interquartile range, MoCA Montreal Cognitive Assessment, NTB neuropsychological test battery, CDR Clinical Dementia Rating, iAD8 the Eight-item Informant Interview to Differentiate Aging and Dementia (informant version).^a^Test of difference across Chi-square test for categorical variables, and Kruskal-Wallis test for continuous variables.
Study samples were split into approximately 40% for Training sample and 20% for Validation sample (rounded to whole numbers), with the remaining set aside as Test sample. The sample split was done using the random approach, stratified by the clinical diagnosis (i.e. normal cognition, MCI and Dementia) to ensure balanced representation of clinical diagnosis across the split samples. Following the random split, the participant characteristics were largely comparable across the three split samples, as seen in Table 2. Training sample was used to train deep-learning models to distinguish MCI/dementia from normal cognition, and Validation sample was used to fine-tune model hyperparameters. Meanwhile, Test sample (i.e. single hold-out test set) evaluated actual performance of trained models in distinguishing MCI/dementia from normal cognition, and was used to select the best-performing model and the optimal cutoffs.Table 2Comparison of participant characteristics across training, validation and test samplesVariableTraining sample(n = 700)^a^Validation sample(n = 400)^a^Test sample(n = 658)^a^P value^b^Age, median (IQR)72 (69, 76)71 (68, 76)71 (68, 75)0.174Years of education, median (IQR)10 (8, 12)10 (8, 12)10 (10, 13)0.012Male sex, n (%)257 (36.7)147 (36.8)236 (35.9)0.936Ethnicity, n (%)0.470 Chinese657 (93.9)374 (93.5)613 (93.2) Malay/Indian32 (4.6)21 (5.2)37 (5.6) Eurasian/Others11 (1.6)5 (1.2)8 (1.2)MoCA total score, median (IQR)26 (24, 28)26 (24, 28)26 (24, 28)0.576NTB Global Z-score, median (IQR)-0.2 (-0.6, 0.1)-0.2 (-0.6, 0.1)-0.2 (-0.6, 0.1)0.801Global CDR*, n* (%)0.899 0589 (84.1)337 (84.2)562 (85.4) 0.598 (14.0)55 (13.8)88 (13.4) 19 (1.3)7 (1.8)5 (0.8) 23 (0.4)1 (0.2)3 (0.5) 31 (0.1)0 (0.0)0 (0.0)Diagnosis, n (%)0.932 Normal cognition599 (85.6)346 (86.5)574 (87.2) Mild cognitive impairment83 (11.9)44 (11.0)68 (10.3) Dementia18 (2.6)10 (2.5)16 (2.4)IQR interquartile range, MoCA Montreal Cognitive Assessment, NTB neuropsychological test battery; CDR, Clinical Dementia Rating.^a^The study samples were randomly split into approximately 40% for Training sample and 20% for Validation sample (rounded to whole numbers), with the remaining set aside as Test sample.^b^Test of difference across split Chi-square test for categorical variables, and Kruskal-Wallis test for continuous variables.
Table 3 presents the results of trained models in Test sample (n = 658). VGG-16 performed better than SwinTransformer among image-based models (Table 3A); CLIP performed better than CNN-GRU among alternative models (Table 3B). Drawing activities (e.g. replaying audio instructions, repeated drawing attempts, long pauses between drawing strokes) further improved performance of image-based models, with VGG-16 + Drawing activities achieving the best-performing model (area under receiver-operating-characteristic curve, AUC = 93.2%; area under precision-recall curve, PR-AUC = 70.8%). Using this best model, we further examined effects of basic demographics (i.e. age, sex, education, and test language) (Table 3C); of which, only education improved model performance further (i.e. similar AUC of 93.1%, with further improvement of PR-AUC to 74.1%), and hence VGG-16 + Drawing activities + Education was selected as the final model (bold-faced in Table 3). Based on this final selected model, we conducted ablation studies to understand relative contributions of the four drawing tasks in detecting MCI/dementia (Table 3D) – Complex figure recall alone had the greatest utility in detecting MCI/dementia (AUC = 89.8%); adding Complex figure copy improved AUC to 91.8%, and further addition of Clock drawing improved AUC to 92.1%.Table 3Comparison of the performance of trained models for distinguishing MCI/dementia from normal cognition in the Test sample (n = 658)Models^a^AUC, %(95% CI)^b^PR-AUC, %(95% CI)^b^(A) Image-based models VGG-1688.2 (84.5–91.9)59.2 (49.1–69.3)** VGG-16 + Drawing activities**^c^**93.2 (91.0–95.5)**70.8 (62.7–78.9) SwinTransformer83.7 **(**78.9–88.5)52.1 **(**42.6–61.7) SwinTransformer + Drawing activity85.2 **(**80.4–90.1)62.0 **(**52.5–71.6)(B) Alternative models (i.e. sequential model and zero-shot Vision Language Model) CNN-GRU82.8 **(**77.8–87.8)50.8 **(**40.7–60.9) CNN-GRU + Drawing activities84.9 **(**80.2–89.5)56.5 **(**46.8–66.2) CLIP86.6 **(**82.4–90.7)57.7 **(**48.1–67.3) CLIP + Drawing activities88.4 **(**84.5–92.4)66.1 **(**57.9–74.3)(C) Best model^c^ + Basic demographics VGG-16 + Drawing activities + Age92.4 **(**89.7–95.1)70.8 **(**62.7–78.9) VGG-16 + Drawing activities + Sex91.6 **(**88.6–94.6)69.5 (61.5–77.5) VGG-16 + Drawing activities + Education^d^**93.1 (90.6–95.6)**74.1 (66.4–81.8) VGG-16 + Drawing activities + Test language92.8 (90.3–95.3)69.5 (61.1–77.9) VGG-16 + Drawing activities + Age + Sex + Education91.6 (88.7–94.4)69.4 (61.0–77.7) VGG-16 + Drawing activities + Age + Sex + Education + Test language91.8 (88.9–94.8)69.9 (61.7–78.1) Drawing activities + Age + Sex + Education + Test language^e^79.1 (74.0–84.2)46.6 (37.1–56.0) Age + Sex + Education + Test language^e^74.5 (68.6–80.4)37.3 (28.1–46.5)(D) Subset of drawing tasks based on the final selected model^d^ Complex figure copy78.5 (73.7–83.2)36.0 (27.6–44.4) Simple figure copy65.8 (59.5–72.1)23.5 (16.7–30.3) Clock drawing76.2 (70.2–82.2)42.4 (32.6–52.2) Complex figure recall89.8 (86.1–93.5)63.5 (53.6–73.3) Complex figure copy + Simple figure copy78.0 (73.0–83.0)33.0 (25.4–40.6) Complex figure copy + Clock drawing77.0 (71.2–82.8)41.2 (31.7–50.7) Complex figure copy + Complex figure recall91.8 (88.8–94.8)69.6 (61.4–77.9) Simple figure copy + Clock drawing75.3 (69.9–80.7)39.2 (29.9–48.5) Complex figure copy + Simple figure copy + Complex figure recall90.8 (87.3–94.2)68.0 (59.4–76.5) Complex figure copy + Clock drawing + Complex figure recall92.1 (89.1–95.0)73.3 (65.4–81.1)AUC area under the receiver operating characteristics curve, PR-AUC area under the precision-recall curve, CNN-GRU Convolutional neural network–Gated Recurrent Unit, CLIP Contrastive Language-Image Pretraining^a^Further description on the models and model architecture is available in Supplementary Method 5.^b^The 95% CI of AUC were computed using a non-parametric approach proposed by DeLong et al.^71^. The 95% CI of PR-AUC were computed using 1000 bootstrap resampling.^c^This bolded row indicate the best-performing model (VGG-16 + Drawing activities) among the initially trained models in (A) and (B).^d^This bolded row indicate the final selected model (VGG16 + Drawing activities + Education). This model was then used in the ablation studies in (D) to understand the relative contributions of the four drawing tasks in detecting MCI/dementia.^e^These baseline models were provided mainly for comparison purposes, that is, by omitting VGG-16 and Drawing activities to examine the impact on the overall performance. Of note, the baseline models provided further evidence on the contributions of VGG-16 and Drawing activities to the overall performance; with AUC dropping to 79.1% when VGG-16 was omitted, and with AUC further dropping to 74.5% when Drawing activities were omitted.
Table 4 compares the performance of PENSIEVE-AI and other commonly used assessment tools in Test sample (n = 658). PENSIEVE-AI had comparable performance to NTB (Neuropsychological Test Battery) and MoCA (Montreal Cognitive Assessment) for detecting MCI/dementia (AUC = 93.1–95.3%), even across the lower education subgroup (AUC 90.0–95.0%) and the higher education subgroup (AUC = 95.0–98.2%). In contrast, iAD8 (the Eight-item Informant Interview to Differentiate Aging and Dementia) had significantly lower AUC for MCI/dementia, particularly among participants ≤10 years of education (AUC = 73.2%, p < 0.001 when compared to PENSIEVE-AI). For the detection of dementia, all four tools (i.e. PENSIEVE-AI, NTB, MoCA, iAD8) have comparable AUCs of >90%. AUC results remained largely similar in the two sensitivity analyses (Table 4), when prevalence of MCI/dementia was increased to reflect average prevalence in most communities (i.e. 20%^42–47^ and 35%^43,44,46,47^ respectively). Additionally, several post-hoc analyses were conducted to examine potential AI biases in PENSIEVE-AI’s performance across various demographic subgroups. As seen in Supplementary Table 1, PENSIEVE-AI maintained similar AUC in detecting MCI/dementia even across the subgroups of age, sex, ethnicity, test language, and mode of administration.Table 4Performance of PENSIEVE-AI for detecting cognitive impairment in the Test sample (n = 658), and a comparison with the performance of other commonly used assessment toolsAssessment ToolsAll education subgroups≤10 years of education^a^>10 years of education^a^AUC, %(95% CI)P value^b^AUC, %(95% CI)P value^b^AUC, %(95% CI)P value^b^Detection of MCI/dementia PENSIEVE–AI93.1 (90.6–95.6)Ref90.0 (86.3–93.8)Ref98.2 (96.6–99.8)Ref NTB^c^94.0 (91.6–96.4)1.00093.2 (90.2–96.2)0.45696.2 (91.4–100)1.000 MoCA^c^95.3 (93.2–97.4)0.52095.0 (92.4–97.6)0.05795.0 (90.9–99.1)0.372 iAD875.2 (69.0–81.3)1.48 × 10^-7^73.2 (66.3–80.1)4.96 × 10^-5^83.5 (70.0–97.0)0.094Detection of dementia PENSIEVE–AI96.6 (94.6–98.5)Ref95.5 (93.1–97.9)Ref96.5 (89.4–100)Ref NTB^c^98.5 (97.5–99.6)0.14797.9 (96.3–99.4)0.13499.7 (99.0–100)1.000 MoCA^c^95.5 (90.1–100)1.00093.8 (85.1–100)1.00098.9 (97.7–100)1.000 iAD892.0 (83.2–100)0.88490.8 (80.8–100)1.00098.9 (97.2–100)1.000Sensitivity analysis 1 (Prevalence of MCI/dementia=20%)^d^Detection of MCI/dementia PENSIEVE–AI93.4 (90.4–96.3)Ref90.8 (86.4–95.2)Ref97.9 (95.8–100)Ref NTB^c^95.1 (92.5–97.8)0.86793.9 (90.3–97.5)0.58098.2 (95.4–100)1.000 MoCA^c^95.9 (93.6–98.2)0.50195.4 (92.2–98.5)0.19896.7 (92.8–100)1.000 iAD875.5 (68.0–82.9)1.78 × 10^-5^73.9 (65.3–82.4)0.00185.6 (72.2–99.0)0.197Detection of dementia PENSIEVE–AI95.5 (92.9–98.1)Ref94.3 (90.7–97.8)Ref94.6 (83.8–100)Ref NTB^c^98.0 (96.5–99.5)0.15897.8 (95.8–99.8)0.11499.3 (97.9–100)1.000 MoCA^c^94.8 (89.1–100)1.00093.2 (84.4–100)1.00097.7 (95.2–100)1.000 iAD891.5 (82.7–100)1.00091.0 (81.2–100)1.00098.4 (96.0–100)1.000Sensitivity analysis 2 (Prevalence of MCI/dementia=35%)^e^Detection of MCI/dementia PENSIEVE–AI93.1 (89.5–96.7)Ref90.1 (84.7–95.6)Ref99.1 (97.3–100)Ref NTB^c^93.9 (90.1–97.6)1.00093.9 (89.6–98.3)0.80794.2 (82.6–100)1.000 MoCA^c^95.9 (93.1–98.6)0.63095.2 (91.5–98.8)0.28996.4 (91.0–100)1.000 iAD876.0 (68.0–83.9)3.44 × 10^−4^74.2 (65.0–83.4)0.01285.3 (65.6–100)0.528Detection of dementia PENSIEVE–AI92.3 (87.8–96.8)Ref89.9 (83.8–95.9)Ref93.4 (79.7–100)Ref NTB^c^96.5 (93.8–99.2)0.18094.9 (90.9–98.9)0.284100 (100–100)1.000 MoCA^c^91.8 (84.7–98.9)1.00089.8 (79.5–100)1.00099.5 (98.2–100)1.000 iAD890.6 (81.3–100)1.00088.0 (77.1–99.0)1.00099.5 (98.2–100)1.000AUC area under the receiver operating characteristics curve, 95% CI 95% confidence interval, MCI mild cognitive impairment, Ref reference, NTB neuropsychological test battery, MoCA Montreal Cognitive Assessment, iAD8 the Eight-item Informant Interview to Differentiate Aging and Dementia (informant version).^a^Education subgroups were stratified based on median split.^b^P values were based on comparison of AUC between the PENSIEVE-AI and the respective assessment tools, using a non-parametric approach proposed by DeLong et al.^71^ P < 0.05 indicates significant difference in AUC between the PENSIEVE-AI and the respective assessment tools. P values were Bonferroni-adjusted to minimize the risk of Type 1 error in the context of multiple testing.^c^The AUCs of NTB and MoCA were likely overestimated in this study, as their data informed the diagnostic process. Therefore, readers should exercise caution when comparing these results to those of PENSIEVE-AI, viewing them as general indicators rather than reflections of actual, real-world performance.^d^Prevalence of MCI/dementia was readjusted to 20% in the Test sample, based on prior meta-analytic findings that community prevalence was ~15% for MCI and ~5% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 256 participants with normal cognition (80%), 48 participants with MCI (15%), and 16 participants with dementia (5%).^e^Prevalence of MCI/dementia was readjusted to 35% in the Test sample, based on prior meta-lytic findings that community prevalence could be as high as ~25% for MCI and ~10% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 104 participants with normal cognition (65%), 40 participants with MCI (25%), and 16 participants with dementia (10%).
Test statistics of PENSIEVE-AI are plotted in Fig. 1a. Adopting two-cutoff approach, the lower cutoff (probability≥13%) had 85.7% sensitivity and 97.5% negative predictive value, and was used to rule out MCI/dementia (for individuals with probability scores below the cutoff); while the upper cutoff (probability≥45%) had 98.8% specificity and 85.1% positive predictive value, and identified those who were likely to have MCI/dementia (i.e. to rule in MCI/dementia). These two cutoffs provide an intermediate range between them (greyed area in Fig. 1a), identifying those who may be at higher risk and potentially require further monitoring or assessment. The optimal cutoffs varied slightly with changing prevalence of MCI/dementia, as seen in Figs. 1b and 1c.Fig. 1Plot of sensitivity, specificity, NPV and PPV based on probability scores of PENSIEVE-AI in the Test sample (n = 658).a Main results based on all the Test sample (n = 658). Adopting two-cutoff approach, the lower cutoff identifies sensitivity and negative predictive value (NPV) (red lines) which are >85% each, and is used to rule out mild cognitive impairment and dementia (MCI/dementia) when probability scores fall below this threshold. The upper cutoff identifies specificity and positive predictive value (PPV) (blue lines) which are >85% each, and is used to rule in MCI/dementia when probability scores exceed this threshold. The greyed area (demarcated by the lower and upper cutoffs) represents the intermediate range, identifying those who may be at higher risk and may require further monitoring or assessment. b Results based on Sensitivity analysis 1, whereby the prevalence of MCI/dementia was readjusted to 20% in the Test sample, based on prior meta-analytic findings that community prevalence was ~15% for MCI and ~5% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 256 participants with normal cognition (80%), 48 participants with MCI (15%), and 16 participants with dementia (5%). c Results based on Sensitivity analysis 2, whereby the prevalence of MCI/dementia was readjusted to 35% in the Test sample, based on prior meta-analytic findings that community prevalence could be as high as ~25% for MCI and ~10% for dementia. In the Test sample a subset of participants with MCI and dementia were randomly selected to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 104 participants with normal cognition (65%), 40 participants with MCI (25%), and 16 participants with dementia (10%). Source data are provided as a Source Data file.
Effectively, the two cutoffs identified 3 risk categories for cognitive (1) Less likely to have cognitive impairment; (2) Higher risk of cognitive impairment; and (3) Likely to have cognitive impairment. These 3 categories, along with their cross-tabulation with the final diagnoses, are presented in Table 5. In the first category (i.e. Less likely to have cognitive impairment), 92–98% of individuals had normal cognition. In the second category (i.e. Higher risk of cognitive impairment), 20–40% of individuals were diagnosed with MCI. In the third category (i.e. Likely to have cognitive impairment), 85–88% of the individuals had MCI/dementia, with a large proportion having dementia (26–36%). Distinctions between these 3 risk categories are also visible in Fig. 2. The first category (white region with probability scores below the lower cutoff) identified those with normal cognition; the third category (dark grey region with probability scores above the upper cutoff) identified almost all individuals with dementia; while the second category (light grey region between the lower and upper cutoffs) mostly captured those with MCI. Detailed results on test statistics are available in Supplementary Tables 2–4.Table 5Cross-tabulation between the output from PENSIEVE-AI and the final diagnosis in Test sample (n = 658)Output from PENSIEVE-AI ^a^Final diagnosisNormal cognitionMCIDementiaLess likely to have CI, n (%)^b^461 (97.5)12 (2.5)0 (0.0)Higher risk of CI, n (%)^b^106 (76.8)28 (20.3)4 (2.9)Likely to have CI, n (%)^b^7 (14.9)28 (59.6)12 (25.5)Sensitivity analysis 1 (Prevalence of MCI/dementia = 20%)^c^Less likely to have CI, n (%)^d^209 (95.9)9 (4.1)0 (0.0)Higher risk of CI, n (%)^d^42 (70.0)17 (28.3)1 (1.7)Likely to have CI, n (%)^d^5 (11.9)22 (52.4)15 (35.7)Sensitivity analysis 2 (Prevalence of MCI/dementia = 35%)^e^Less likely to have CI, n (%)^f^82 (92.1)7 (7.9)0 (0.0)Higher risk of CI, n (%)^f^16 (55.2)12 (41.4)1 (3.4)Likely to have CI, n (%)^f^6 (14.3)21 (50.0)15 (35.7)CI cognitive impairment, MCI mild cognitive impairment.^a^Two-cutoff approach was adopted for PENSIEVE-AI. The lower cutoff has high sensitivity and negative predictive value (>85% respectively), and is used to rule out MCI/dementia (for individuals with probability scores below the cutoff). The upper cutoff has high specificity and positive predictive value (>85% respectively), and identifies those who are likely to have MCI/dementia. These two cutoffs provide an intermediate range between them, identifying those who may be at higher risk and require further monitoring or assessment.^b^Probability cutoff for the main results (i.e. prevalence of MCI/dementia=13.6%): <13% (Less likely to have CI), 13–44% (Higher risk of CI), ≥45% (Likely to have CI).^c^Prevalence of MCI/dementia was readjusted to 20% in the Test sample, based on prior meta-analytic findings that community prevalence was ~15% for MCI and ~5% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 256 participants with normal cognition (80%), 48 participants with MCI (15%), and 16 participants with dementia (5%).^d^Probability cutoff for the first sensitivity analysis (i.e. prevalence of MCI/dementia=20%): <13% (Less likely to have CI), 13–34% (Higher risk of CI), ≥35% (Likely to have CI).^e^Prevalence of MCI/dementia was readjusted to 35% in the Test sample, based on prior meta-analytic findings that community prevalence could be as high as ~25% for MCI and ~10% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 104 participants with normal cognition (65%), 40 participants with MCI (25%), and 16 participants with dementia (10%).^f^Probability cutoff for the second sensitivity analysis (i.e. prevalence of MCI/dementia=35%): <11% (Less likely to have CI), 11–27% (Higher risk of CI), ≥28% (Likely to have CI).Fig. 2Box plots showing the distribution of PENSIEVE-AI’s probability scores in the Test sample (n = 658).a Main results based on all the Test sample (n = 658). The box plot’s center line, box limits, and whiskers denote the median, lower and upper quartiles, and 1.5× interquartile range, respectively. The red dots represent the individual datapoints. The two horizontal dashed lines represent the two optimal cutoffs for PENSIEVE-AI. The lower cutoff has high sensitivity and negative predictive value (>85% each), and is used to rule out mild cognitive impairment and dementia (MCI/dementia) when probability scores fall below this threshold (as shown by the white region). The upper cutoff has high specificity and positive predictive value (>85% each), and identifies individuals likely to have MCI/dementia (when probability scores exceed this threshold, as shown by the dark grey region). The light grey region (demarcated by the lower and upper cutoffs) represents the intermediate range, identifying individuals who may be at higher risk and may require further monitoring or assessment. b Results based on Sensitivity analysis 1, whereby the prevalence of MCI/dementia was readjusted to 20% in the Test sample, based on prior meta-analytic findings that community prevalence was ~15% for MCI and ~5% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected in the Test sample to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 256 participants with normal cognition (80%), 48 participants with MCI (15%), and 16 participants with dementia (5%). c Results based on Sensitivity analysis 2, whereby the prevalence of MCI/dementia was readjusted to 35% in the Test sample, based on prior meta-analytic findings that community prevalence could be as high as ~25% for MCI and ~10% for dementia. In the Test sample, a subset of participants with MCI and dementia were randomly selected in the Test sample to readjust the prevalence in the dataset (see Methods section for further details). The resulting dataset comprised 104 participants with normal cognition (65%), 40 participants with MCI (25%), and 16 participants with dementia (10%). Source data are provided as a Source Data file.
Brief cognitive tests are crucial for detecting subtle, early symptoms of CI. However, most require trained professionals, limiting their scalability; many were developed from high literacy populations, limiting their usefulness in lower literacy subgroups. In this study, we developed a purpose-built AI tool for early detection of CI, based on 4 drawing tasks that can be self-administered by most participants in <5 min and do not rely on the ability to read or write in a language. The new PENSIEVE-AI was trained and validated using clinically-adjudicated diagnoses in a large, prospectively-recruited community sample. Among trained deep-learning models, VGG-16 demonstrated the highest performance; adding Drawing activities (e.g. pauses between drawing strokes) significantly improved performance, adding Education marginally improved performance, and adding Test language did not improve performance. The best-performing model (VGG-16 + Drawing activities + Education) demonstrated excellent performance in detecting MCI/dementia, comparable to detailed neuropsychological testing and MoCA. Results remained consistent across education subgroups, and when prevalence of MCI/dementia was readjusted to reflect average prevalence in most communities (i.e. ~15–25% for MCI^42–44^ and ~5–10% for dementia)^45–47^.
Our findings highlight a key strength of PENSIEVE-AI in particular, and digital cognitive tests in general. Despite PENSIEVE-AI’s brevity (comprising only 4 test items and taking <5 min to complete), it achieves comparable AUC to detailed neuropsychological testing (which often requires at least 1–2 h to complete). This success is possibly attributed to the capture of additional data on test processes (i.e. drawing activities)^35^, which provides valuable information to offset the reduced number of test items. As seen in our findings, this test process data was highly informative in guiding diagnosis. Plausibly, nuanced behaviors during cognitive testing better reflect subtle cognitive changes than final test scores, especially at early stages of CI. In a way, this process data mimics conventional practice of qualitative observations during detailed neuropsychological testing, providing complementary information to final test scores. Such process data would otherwise not be feasibly captured in pen-and-paper versions of brief cognitive tests, due to the labor intensity of recording such qualitative observations in routine clinical practice.
Although many digital cognitive tests have been developed in the literature^37^, most are pilot studies with smaller samples and primarily correlated with another neuropsychological test (i.e. without evaluation over actual clinical diagnoses)^37^. The Brain Health Assessment (BHA) is among the few that showed promising results^36^. BHA shares some similarities with PENSIEVE-AI – both were trained on gold standard clinical diagnosis in large community samples; BHA also involves 4 tasks capturing various cognitive domains and reported similar AUC (up to 91.9%) for detecting MCI/dementia^36^. However, BHA differs in design from PENSIEVE-AI, requiring trained professionals to administer the 10 min test, whereas PENSIEVE-AI is designed to be primarily self-administered in <5 min. BHA was developed from White populations with high literacy (average education of 16–17 years in the development sample^36^; with recent pilot validations in non-White populations)^36,48,49^, in contrast to PENSIEVE-AI’s development within a multiethnic Asian population with lower literacy (average education of 10 years).
In extant literature, few digital cognitive tests rely solely on drawing tasks, with clock drawing being most widely adopted^50^. Consistent with the literature, our findings indicate that clock drawing alone is insufficient to detect early CI^51,52^, and must be combined with at least one other test that evaluates another cognitive domain^52–54^. Additionally, our findings further show that memory tasks are crucial for detecting early CI, possibly to capture early memory decline related to the most common aetiology (i.e. Alzheimer’s disease). The final model – incorporating a memory task and 3 other drawing tasks – achieved an AUC of >93%, which is among the highest reported to date for drawing-based digital cognitive tests. Consistent with the literature, our findings also suggest some influence of educational attainment on drawing-based tasks^38,39^, whereby the inclusion of education as a covariate further improved model performance (Table 3C). At the same time, the findings also affirm our initial hypothesis that drawing tasks may possibly be less affected by literacy – the education covariate only marginally improved model performance; and after including the education covariate, the final model demonstrated comparable performance to detailed neuropsychological testing even among individuals with lower literacy (Table 4).
PENSIEVE-AI offers a scalable solution for case-finding of CI in the community. To address the global challenge of undiagnosed CI^1–3^, the International Association of Gerontology and Geriatrics has advocated for annual evaluation of cognitive function among older age-groups (e.g. all individuals ≥70 years)^25^. Yet few viable options are available to date for large-scale deployment. Unlike most brief cognitive tests, PENSIEVE-AI does not require trained professionals to administer, making it well-suited as a scalable tool for case-finding of CI in large populations. Given the brevity of PENSIEVE-AI, it can be easily embedded within routinely-conducted comprehensive geriatric assessments in the community, or used as a follow-up assessment tool in conjunction with subjective questionnaires^55,56^ (i.e. to provide more conclusive evidence of CI among individuals screened positive through subjective questionnaires)^57^. Considering the finding that PENSIEVE-AI can be self-administered by a majority of participants, it may also be deployed as standalone kiosks in community settings with high volumes of higher-risk older persons (e.g. primary care clinics), to allow individuals with cognitive concerns to complete brief cognitive evaluations. This approach can be especially cost-saving, as it does not require professional staff to man the community kiosks and at most only needs lay volunteers to be on standby to supervise those with difficulty navigating the digital interface.
At the population level, PENSIEVE-AI can serve as an efficient risk-stratification tool. As shown in Fig. 1, cutoffs for PENSIEVE-AI can be adjusted depending on prevalence of MCI/dementia in different populations, to identify individuals with varying risks of CI. Low-risk individuals ( <10% probability of MCI/dementia) may possibly be reassured and advised to repeat the test after a longer time horizon (e.g. 3–5 years). Intermediate-risk individuals ( 25–40% probability of MCI/dementia) can be advised to consult a physician if concerned about cognition, or to repeat the test in 1 year for closer monitoring. High-risk individuals ( >85% probability of MCI/dementia) will benefit from direct referral to memory clinics for further assessment and management. Notably, the high-risk group largely captures most of the individuals with dementia (Fig. 2); thus, this category can also be the primary focus in communities more interested in detecting dementia than MCI. The risk-stratification approach described here is also summarized in Table 6.Table 6Potential clinical implications based on output from PENSIEVE-AIOutput from PENSIEVE-AIRisk communicationPotential implicationsLess likely to have CI<10% chance to have CI• Repeat the test after a longer time interval (e.g. in 3–5 years)Higher risk of CI25–40% chance to have CI• Consult a physician if concerned about cognition• Repeat the test in 1 yearLikely to have CI^a^>85% chance to have CI• Referral to memory clinic for further assessment and managementCI cognitive impairment.^a^This category captures most of the individuals with dementia, and can be the main focus in communities that are primarily interested in detecting dementia.
Moving forward, there are several avenues to further expand the applicability and utility of PENSIEVE-AI. An immediate direction is to translate the tool into other local languages and dialects in Singapore (e.g. Malay, Tamil, Cantonese, Hokkien, and Teochew), to enhance accessibility and inclusivity within Singapore’s multiethnic population. This effort is readily attainable, as PENSIEVE-AI is largely language-neutral (i.e. the test input is based on drawing data alone) and requires only the translation of test instructions. This approach is also supported by findings from the current study, which show that test language had minimal impact on PENSIEVE-AI’s overall performance (as seen in Table 3C and Supplementary Table 1). On a related note, PENSIEVE-AI may also hold potential for broader implementation in other literacy-diverse populations similar to Singapore (e.g. those across East and South Asia and some LMICs), given current findings that it is less affected by literacy (as shown in Table 3C and Table 4). However, this potential will need to be verified through future validation in populations beyond Singapore, considering prior literature on the potential cultural impact on drawing tasks similar to those used in PENSIEVE-AI^38,58^. Lastly, although current findings demonstrated the usefulness of PENSIEVE-AI for detecting the presence of MCI/dementia cross-sectionally, efforts are also ongoing to evaluate its utility in generating a global cognitive score, alongside further longitudinal evaluations of the psychometrics of this score with respect to test-retest reliability and validity in tracking cognitive decline over time.
Several limitations are notable. First, PENSIEVE-AI is less useful for individuals with severe visual impairment or hand movement difficulties, as it requires the ability to see on-screen figures and draw with a stylus. Second, while being language-neutral is a strength of PENSIEVE-AI, it can also pose a limitation. The drawing-based tasks may plausibly capture less information on the language domain, potentially reducing PENSIEVE-AI’s sensitivity in detecting language dysfunction. This limitation is particularly relevant in young-onset CI, where language dysfunction can be more prevalent as the initial presentation (due to higher proportions of non-Alzheimer’s diseases in young-onset CI, e.g. frontotemporal lobar degeneration). Third, while digital cognitive tests have inherent strengths and appeal^35^, they also present new barriers, particularly for individuals with lower literacy and in LMICs. For example, individuals with lower literacy may be unfamiliar with using technology, and some LMICs may have limited access to touch-screen tablets and technology infrastructure. To mitigate these limitations, we conducted extensive user design iterations in this study to tailor PENSIEVE-AI to the needs of older individuals with less digital literacy (Supplementary Method 1). We also ensured that PENSIEVE-AI is compatible with generic, low-specification touch-screen tablets, and requires only intermittent internet connection to generate results from cloud-hosted deep-learning models. Additionally, we designed PENSIEVE-AI as an assessor- or center-based tool (i.e. not installed on older individuals’ personal devices), so that only a limited number of tablets are needed for large-scale assessments in the community. Fourth, residual AI biases may still exist despite our best efforts to minimize the biases (e.g. through extensive recruitment efforts to obtain community-representative samples, ensuring the project team comprised a diversity of age, sex, ethnicity and professional discipline [i.e. geriatric psychiatrist, geriatrician, neurologist, psychologist], and conducting post-hoc analyses to ensure no systematic biases across demographic subgroups). As an example of residual AI biases, individuals who chose to participate in this study may differ from those who opted not to, potentially reducing the community-representativeness of recruited samples. We mitigated this limitation by employing diverse sources of community recruitment, as well as by emphasizing ‘Detect dementia early’ in our recruitment publicity (rather than a conventional invitation to participate in research, which tends to attract a distinct group of individuals) (Supplementary Method 2). Fifth, while clinicians who determined the diagnoses in this study were blinded to the drawing data from PENSIEVE-AI, they were not blinded to participants’ demographic information (e.g. age, sex and education) because such details are often essential for accurate diagnosis (e.g. information on previous levels of cognitive abilities is critical when making clinical judgment on the presence of “significant cognitive decline”)^59^. While access to this demographic information might introduce potential bias in the results of PENSIEVE-AI, the risk is arguably low. As demonstrated in Table 3C, baseline models (using demographic information alone) contributed minimally to PENSIEVE-AI’s overall performance, in contrast to the substantial contributions of the drawing tasks. Sixth, PENSIEVE-AI is not intended to replace comprehensive clinical and neuropsychological assessments, as it does not provide a definitive diagnosis nor granular information on specific cognitive deficits^18,60,61^.
Using a large community sample, we developed an AI-based, drawing-based cognitive test that can be self-administered in <5 min by most participants. Despite its brevity and ease of use, PENSIEVE-AI demonstrated excellent performance in detecting MCI/dementia, comparable to detailed neuropsychological testing. It can be a valuable tool in situations where detailed neuropsychological testing is not feasible, such as being embedded within community assessments or deployed as community kiosks to identify individuals requiring further intervention. As PENSIEVE-AI is less affected by language or literacy, it holds the potential for broader implementation in other literacy-diverse settings similar to Singapore, such as in populations across East and South Asia and some LMICs.
This study complies with all relevant ethical regulations. The research protocol was reviewed and approved by SingHealth Centralized IRB (reference: 2021/2590). Informed consent was obtained from all participants, or their legally authorized next-of-kin (for participants without mental capacity to consent)^62^. Participants who completed the research assessments received Singapore Dollar $80 as compensation for their time, inconvenience and transportation costs.
This was a nationally-funded study in Singapore to develop an AI tool for early detection of CI (Project PENSIEVE). From March 2022 to August 2024, we prospectively recruited community-dwelling older persons based on the following (1) At higher-risk of CI (i.e. aged ≥65 years^25^, and having at least one of the three chronic diabetes mellitus, hypertension, or hyperlipidemia); (2) Able to follow simple instructions in English or Mandarin Chinese; (3) Did not having severe visual impairment that could affect the ability to complete drawing tasks (note: to ensure generalizability, participants were included as long as they could see pictures on a piece of paper held before them); and (4) Had an informant who knew the participant well (e.g. family member or friend). Recruitment sources included 14 community roadshows by study team, clients of community partners, home visits by community volunteers, media publicity (radio, online articles, and posters), and word-of-mouth referrals from participants who had completed research assessments. To ensure that the recruited samples are representative of the community, the study’s publicity materials placed much emphasis on the key message of ‘Detect dementia early’ (along with direct referrals to memory clinics in the event of significant findings), rather than the conventional invitation to participate in research (which may inadvertently attract a distinct group of individuals). Samples of these publicity materials (e.g. study banner, poster, brochure) are presented in Supplementary Method 2.
The recruited participants received comprehensive assessments, which included semi-structured interviews with participants and their informants, detailed neuropsychological testing, and observational notes of participants’ behavior during assessments. Details on the comprehensive assessments are available in Supplementary Methods 3, 4. Diagnoses of MCI and dementia were made via consensus conference (by 3 dementia specialists). Dementia was diagnosed using the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders–Fifth Edition) criteria^59^. MCI was diagnosed using the modified Petersen criteria^63^. Normal cognition was diagnosed when participants were found not to have dementia or MCI.
The new digital cognitive test (henceforth denoted as PENSIEVE-AI^TM^) comprises 4 drawing tasks, (1) complex figure copy; (2) simple figure copy; (3) clock drawing; and (4) complex figure recall (i.e. recall of complex figure from the first task). Respondents were provided a 12.4-inch touch-screen tablet and a stylus, and asked to follow on-screen voice instructions to complete the 4 drawing tasks on the tablet (of note, the same drawing prompts were used for every assessment, to ensure consistency in administration across different assessments). Throughout the 4 tasks, drawing activities (e.g. drawing motions, replaying audio instructions, repeated drawing attempts) were also captured within the tablet and included as input data for model training. The 4 tasks were designed to cover the cognitive domains of Visuospatial abilities (tasks 1 and 2)^64^, Attention and Executive function (task 3)^64^, Memory (task 4)^64^, and Language (ability to follow through audio instructions). Details on user design of PENSIEVE-AI are available in Supplementary Method 1. Of note, PENSIEVE-AI was completed by participants before the start of comprehensive assessments, and the dementia specialists who were determining the diagnosis in consensus conference were blinded to the drawings and drawing activities from PENSIEVE-AI (but not blinded to participants’ demographic information such as age, sex and education).
Three alternative assessment tools were included in the analyses as comparators to PENSIEVE-AI. These tools represent three common types of assessments in cognitive an informant questionnaire (iAD8; the Eight-item Informant Interview to Differentiate Aging and Dementia)^55^, a brief cognitive test (MoCA; Montreal Cognitive Assessment)^26^ and detailed neuropsychological testing (NTB; Neuropsychological Test Battery)^65^. They are briefly described in the next paragraph, with further details available in Supplementary Method 4. It is important to note that the dementia specialists in this study were blinded to iAD8 results, but not blinded to those of MoCA or NTB. Given that the data from MoCA and NTB were used to inform the diagnostic process, the performance of MoCA and NTB were likely overestimated in this study (i.e. actual performance of MoCA and NTB would be lower than reported). Accordingly, readers should exercise caution when comparing these results to those of PENSIEVE-AI, interpreting them as general indicators rather than reflections of actual, real-world performance.
iAD8^55^ is a brief questionnaire that requires informants to rate changes in participants’ cognition and function in the past few years (through yes/no responses). Its 8 items can be completed in ~3–5 min, with higher scores indicating greater cognitive problems. MoCA^26^ comprises 12 items that test participants in various cognitive domains. It can be completed in ~15–20 min, with higher scores reflecting better cognitive function. The NTB^65^ takes ~60 min to complete, and includes seven neuropsychological tests measuring the key cognitive domains of Visuospatial abilities (Benson Complex Figure Copy), Working memory (Craft Story 21 Immediate Recall), Delayed memory (Craft Story 21 Delayed Recall and Benson Complex Figure Recall), Language (Verbal Fluency–Animal), Attention/Processing speed (Trail Making Test–Part A), and Executive function (Trail Making Test–Part B).
In Training and Validation samples, we experimented with image-based models (i.e. VGG-16^66^ and Swin Transformer)^67^, sequential models (i.e. CNN-GRU)^68^, and zero-shot vison-language models (i.e. CLIP)^69^. While the drawings and the drawing activities were the main input data for model training, we also explored the inclusion of basic demographic features (e.g. age, sex, educational attainment, and test language) to assess their potential effects in improving model performance. Models were trained using focal loss^70^ due to the unbalanced dataset. Focal loss is a technique that helps the model to pay more attention to harder-to-classify examples that it often misclassifies, rather than those it already classifies correctly. It achieves this by dynamically adjusting the contribution of each example to the overall training process. Based on the predicted probability of each example, it reduces the loss contribution from well-classified examples, thereby allowing the model to focus more on harder, misclassified examples. This method is particularly useful in situations where there are far more examples of one class (i.e. normal cognition) compared to another (i.e. MCI/dementia), which can otherwise overwhelm the model. By focusing on the harder, less frequent examples, the model thus improves its ability to identify those rarer examples of MCI/dementia. Further details on model training are presented in Supplementary Method 5.
In Test sample, predicted probabilities of the trained models were compared using area under receiver-operating-characteristic curve (AUC) – supplemented by area under precision-recall curve (PR-AUC) – to select the best-performing model for PENSIEVE-AI in distinguishing MCI/dementia from normal cognition. Thereafter, AUC of PENSIEVE-AI was compared to the AUCs of 3 other commonly-used assessment tools (i.e. iAD8, MoCA, NTB) using a non-parametric approach proposed by DeLong et al.^71–74^, with analyses stratified by education subgroups (i.e. ≤10 years of education and >10 years of education, based on median-split). A two-cutoff approach^75–79^ was adopted for PENSIEVE-AI. First cutoff has high sensitivity and negative predictive value (>85% respectively), and is used to rule out MCI/dementia (i.e. when probability scores fall below first cutoff). Second cutoff has high specificity and positive predictive value (>85% respectively), and identifies those who are likely to have MCI/dementia. This two-cutoff approach has been recommended in recent literature^79^, as it enhances test performance^75–78^, reduces effects of prevalence on test performance^76^, and prioritizes healthcare resources for those more likely to benefit^75^.
As secondary analysis, the performance of PENSIEVE-AI was evaluated for distinguishing dementia from non-dementia. Additionally, two sensitivity analyses were conducted in Test sample to evaluate robustness of results when prevalence of MCI/dementia was readjusted to reflect average prevalence in most Prevalence of MCI/dementia was artificially readjusted to 20%, based on prior meta-analytic findings that community prevalence was ~15% for MCI^42–44^ and ~5% for dementia^45–47^. Readjustment of prevalence was done by randomly selecting only a subset of participants with MCI and normal cognition – for each participant with dementia, 3 participants with MCI and 16 participants with normal cognition were randomly selected (i.e. so that the final dataset corresponded to 5% prevalence for dementia and 15% prevalence for MCI).Prevalence of MCI/dementia was artificially readjusted to 35%, based on prior meta-analytic findings that community prevalence could be as high as ~25% for MCI^43,44^ and ~10% for dementia^46,47^. Readjustment of prevalence was done by randomly selecting only a subset of participants with MCI and normal cognition – for each participant with dementia, 2.5 participants with MCI and 6.5 participants with normal cognition were randomly selected (i.e. so that the final dataset corresponded to 10% prevalence for dementia and 25% prevalence for MCI).
Statistical analyses were conducted in Stata (version 18). No statistical method was used to predetermine sample size. During the initial study planning, we estimated that at least 1000 samples would be required (with each sample providing 4 drawing data), guided by a well-known classification challenge in recent literature (Tiny ImageNet)^80^. No data were excluded from the analyses.
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Supplementary Information Reporting Summary Transparent Peer Review file
Source Data