Authors: Juan Antonio Becerra-García, Sara Barbeito, Eleni Petkari, Ana Calvo, Teresa Sánchez-Gutiérrez
Categories: Research, APSS, Factorial structure, Paranoid ideation, Psychometric properties, Psychotic-like experiences, Psychoticism
Source: BMC Psychology
Psychotic-like experiences are common in the general population and an important indicator of psychological vulnerability. One of the used instruments for their assessment is the Adolescent Psychotic-Like Symptom Screener (APSS). However, there are few studies on the APSS psychometric characteristics in different cultural contexts. The aim of this study is to adapt the instrument in the Spanish context and analyze its psychometric properties in an adult population.
Upon the translation and adaptation of the APSS to Spanish, a sample of 287 participants (19–60 years) completed the questionnaire online together with the Symptom Assessment-45 Questionnaire (SA-45) and an ad-hoc survey on clinical and sociodemographic variables. The sample was then randomly split into two halves, the first being the calibration (n = 144) and the second the validation sample (n = 143). An exploratory factor analysis was performed with the former, and a confirmatory factor analysis was performed with the latter, together with internal consistency and convergent validity analyses.
The APSS Spanish version presents a unifactorial structure comprising 6 out of the 7 original items with adequate fit and good internal consistency. This single factor structure is invariant across age, sex and history of self-reported psychopathology. The instrument also shows significant positive correlations with the SA-45 psychoticism and paranoid ideation subscales.
To the best of our knowledge, this is the first study that has conducted a psychometric analysis of the APSS in adult population. The Spanish version of the APSS shows adequate reliability, construct and convergent validity in adults, therefore provides a handy tool to be used for the screening of psychotic-like experiences in the Spanish general population.
The online version contains supplementary material available at 10.1186/s40359-024-02172-z.
Keywords: APSS, Psychotic-like experiences, Factorial structure, Psychometric properties, Psychoticism, Paranoid ideation
The dimensional approach to psychosis suggests that psychotic manifestations are experienced on a continuum that ranges from severe and frequent hallucinatory and/or delusional symptoms to subclinical psychotic experiences (such as occasional delusional or hallucinatory ideation) in the general population [1–4].
Research on subclinical psychotic experiences has mostly focused on the analysis of attenuated positive psychotic symptoms or psychotic-like experiences (PLEs, such as magical thinking, transient subclinical delusional or hallucinatory ideation); attenuated negative symptoms (such as lack of motivation or social withdrawal) are examined less frequently or treated as a broader group of manifestations (as, for example, “psychotic experience and negative symptom traits - PENS”) [3, 5, 6].
The PLEs are mostly transient (80%) and are commonly present in the general population (5–10%), making their appearance at any time from childhood to adulthood. Though their occasional appearance is not considered a risk factor for mental illness [2, 3, 5], the detection of these PLEs is relevant for the clinical practice and prevention. In this regard, the presence of recurrent, persistent, or coexisting PLEs with other psychopathological symptoms has been related to an increase in psychological distress [1–4], related to impaired quality of life in different age groups [5] and to worse social functioning in adulthood [6].
When it comes to the assessment of the PLEs in the general population, it is common practice to use brief self-report questionnaires [7, 8], which have shown high precision, adequate agreement with structured interviews administered to assess psychotic symptoms, and predictive validity regarding the psychosis onset [8–10].
One such instrument is the Adolescent Psychotic-Like Symptom Screener (APSS) [7], which is specifically aimed at screening the existence of attenuated positive psychotic symptoms or PLEs. Although the APSS is aimed at the adolescent population (i.e. between 11 and 16 years) [7, 11, 12], its brevity and ease of use has instigated its use in different countries with adult general population (over 18 years of age) and clinical samples of adult patients [13, 14]. Albeit its wide use, the psychometric characteristics of the APSS have been worldwide underexplored [15].
There are only three studies have addressed the psychometric properties of this measure. Specifically, the APSS total score was found to have adequate predictive capacity (with a sensitivity of 70% and a specificity of 82.6%) for identifying adolescents between 11 and 13 years of age with interview-validated psychotic experiences [7]. Similarly, an APSS-brief version (APSS-3, made up of items 3, 4 and 6) showed good discriminant validity (with sensitivity of 80% and specificity of 76%) for identifying the presence of psychotic disorders in an adult general population [13]. The APSS factorial structure has been examined only in one recent study with a sample of adolescents in China [16]. The exploratory factor analysis findings proposed a 7-items two-factor model (factor 1 -thought: items 1–2, and factor 2-volition: items 3–7), whereas the confirmatory factor analysis findings showed that the 5-items unidimensional model (version APSS-5, after deletion of items 1 and 2 due to their low psychometric properties) the APSS had adequate reliability.
As observed, to date there are few studies on the APSS psychometric characteristics in different cultural contexts, coming from Ireland, Brazil, and China [7, 13, 16]. Given there are no data in the Spanish research and clinical settings on the psychometric characteristics of this instrument, it is important to examine whether the Spanish version of the APSS behaves well when it comes to the assessment of the subclinical symptoms with general population adult samples.
The sample consisted of 287 adults approached using snowball sampling. From the total sample, 218 participants (76%) were women and 69 (24%) men, aged between 19 and 60 years (Mean = 36.99; SD = 9.07), 251 participants (87.5%) had university studies, 163 (56.8%) reported being married or having a partner and 238 (82.9%) indicated that they were employed. A total of 216 participants (75.3%) had no personal self-reported history of psychopathology.
The inclusion criteria were (a) being above 18 years of age, (b) living in Spain and having access to the internet and (c) being fluent in Spanish. Exclusion criteria were (a) presenting missing data on any survey item and (b) self-reported history of psychotic psychopathology (reporting having received a diagnosis of any psychotic condition, such as schizophrenia, schizophreniform or schizoaffective disorders). Out of the initial sample (n = 289) two participants were excluded.
The sample size calculations for the total sample and the subgroups followed common procedures used in scale validation studies. The power analysis conducted using G*Power software [17] estimated a required total sample size of n = 210 to achieve 95% power (1-β) with an effect size d = 0.50 and α = 0.05. The subject-per-item ratio was used to estimate the minimum sample size for each subgroup (i.e. calibration, validation samples) [18]. That is, considering a minimum ratio of 20 participants for each instrument item [18], the estimated sample size for each subgroup was n = 140. The total sample (n = 287) also met the criterion of having 20 participants per each instrument item.
An ad hoc questionnaire was created to collect information on sociodemographic data (age, sex, educational level, civil status, work activity, and self-reported history of psychopathology).
The Adolescent Psychotic-Like Symptom Screener (APSS) [7] is a self-report instrument assessing different attenuated positive psychotic symptoms, specifically hallucinatory (visual and auditory) and delusional (control, persecution, and grandiosity) PLEs. It consists of seven items with a 3-point Likert-scale response format (“Yes, definitely” = 1, “Maybe” = 0.5, “No, never” = 0). Items 1 to 4 derived from the Diagnostic Interview Schedule for Children, and items 5 to 7 were designed by the instrument authors, aiming at the assessment of a wider range of PLEs with this screening instrument [7]. The total score ranges from 0 to 7, with higher scores indicating greater presence of psychotic-like experiences. The details of creating the APSS Spanish version are provided below (Procedure section). The Spanish version of the instrument can be seen at Table 2.
The Spanish version of the Symptom Assessment-45 Questionnaire (SA-45) [19] provides a self-reported measure of nine psychopathological dimensions [19]. Each subscale consists of five items with a Likert-type response format (ranging from 0 = “Not at all” to 4 = “Very much or extremely”). In this study, only the psychoticism (which examines auditory hallucinatory symptoms and thought control) and the paranoid ideation subscales were administered. The total score in these subscales ranges from 0 to 20 points with higher scores indicating higher levels of psychoticism and paranoid ideation. In the Spanish validation study of the SA-45 the Cronbach’s α values for the psychoticism and paranoid ideation subscales were 0.63 and 0.71, respectively [19]. For this study the internal consistency for the psychoticism subscale was α = 0.60, whereas for the paranoid ideation was α = 0.75.
The study was approved by the Ethics Committee of the Universidad Internacional de La Rioja (RE: PI002/2019) and followed the guidelines and ethical principles of the Declaration of Helsinki. The instrument adaptation followed the International Test Commission Rules [20]. First, the APSS was translated from English to Spanish by a bilingual translator following a standardized translation process [21]. Second, a back translation was performed by a bilingual researcher specialized in psychosis to confirm its similarity to the original English version. Discrepancies were discussed and resolved with the team. The final Spanish version of the APSS was launched through an online survey created at Google Forms. The survey also included the ad hoc questionnaire, and the SA-45 psychoticism and paranoid ideation subscales and it was disseminated through different web platforms from April to September 2019. Before the start of the online survey, the participants had to read the information about the study and explicitly give their informed consent if they agreed to participate. Electronic informed consent was necessary in order to access the survey.
Before proceeding with the analyses, data was checked for missing values (exclusion criterion). The total sample was randomly divided into two subgroups (n1: calibration sample and n2: validation sample). Descriptive statistics were used to calculate the sociodemographic and psychological characteristics of both subgroups and χ^2^ and Student’s t tests were used to check for potential differences between them.
To validate the instrument, first an exploratory factor analysis (EFA) was performed with the calibration sample (n1 = 144) following the Kaiser principle and using the principal component method with varimax rotation, after performing fit checks with the Kaiser-Meyer-Olkin index and the Bartlett sphericity test. Components were chosen if having Eigenvalues > 1.
The Cronbach’s α and McDonald’s ω coefficients were calculated to determine the instrument’s reliability. Values greater than 0.7 in these coefficients were regarded as indicators of acceptable internal consistency [22]. The item-test association was assessed to determine the discriminative items power (considering a minimum acceptable score ≥ 0.30) [23].
Second, a series of multigroup confirmatory factor analyses (CFA) was performed with the validation sample (n2 = 143), to determine the goodness-of-fit of the factorial structure identified in the EFA, compare it with a model including all the items and with previous solutions proposed in the literature. For this, to account for type I and II errors following the recommendations of Hu and Bentler [24] we used the combination of the following indices as indicators of adequate model the root mean square error of approximation (RMSEA) with values < 0.08 indicating a good fit; and the comparative fit index (CFI), with > 0.95 showing adequate fit and > 0.90 showing acceptable fit [24, 25]. As the multivariate normality was violated as suggested by the Mardia test (Mardia = 125.93; p < 0.001), the diagonally weighted least squares (DWLS) was selected as the CFA estimation method. To check whether the single factor model stands across several groups, measurement invariance analyses were performed for sex, age and self-reported psychopathology. The measurement invariance calculation includes four related steps [26, 27]: (1) configural invariance (factor equivalence across groups), (2) metric invariance (equal factor loading between groups), (3) scalar invariance (equivalent item intercepts in the groups) and (4) strict invariance (similar error variance across groups). Each step was tested only when the requirement fit (CFI > 0.90 and RMSEA < 0.08) of the previous step were met.
Finally, the convergent validity was tested using the Pearson product-moment correlation calculated between the total APSS score for the whole sample (n = 287) and the SA-45 psychoticism and paranoid ideation subscales. The SPSS 25.0 program was used to perform the initial checks and descriptive analyses, the sample split, the Pearson correlation and χ^2^ and t tests. The JAMOVI 2.3 software was used for the EFA analysis and for the calculation of the reliability coefficients. The JASP 0.18.1 program was applied for the multigroup CFA analysis.
Table 1 summarizes the sociodemographic, clinical, and psychological variables for the calibration and calibration sample. There were no significant differences between the subsamples for any of the sociodemographic characteristics, the PLEs, psychoticism and self-reported paranoid ideation (see Table 1).
The APSS showed adequate internal consistency (α = 0.76 and ω = 0.77). As can be seen in Table 2, item 2 did not meet the minimum required discrimination criteria (r item-total ≥ 0.30). The reliability coefficients values were increased upon its deletion, thus this item was removed and it was not included in the performed factor analyses.
The Kaiser-Meyer-Olkin index (KMO = 0.86) and Bartlett sphericity test (χ^2^ = 351; df = 15; p < 0.001) in the calibration sample indicated the suitability of the data for factor analysis. The performed EFA in n1 identified the presence of a single factor (which encompassed 6 items) with an eigenvalue greater than 1 (eigenvalue = 3.50, see Fig. 1) that explained 58.4% of the total variance.
Fig. 1 Sedimentation figure, which represents the values of the factor analysis for the adolescent psychotic-like symptom screener (APSS-6) in the calibration sample
The CFA conducted in n2 confirmed that the unifactorial internal structure model is suitable for the APSS-6 (see Table 3). Considering the combination of CFI and RMSEA values, the unidimensional models of 6 items (APSS-6) and 7 items (APSS) showed a superior fit compared to the models previously proposed in the literature. However, the reliability for the 6-item factorial model seems to be better (see Table 3). All the items included in EFA and CFA showed factor loadings greater than 0.30 in a single factor (see Table 4).
The results of the measurement invariance of the APSS-6 one-factor model across different groups are presented in Table 5. The analyses showed an adequate fit CFI and RMSEA indices for configural invariance across sex, age and history of psychopathology. The fit indices indicated that there was metric non-invariance for the different groups (Table 5), therefore, further invariance analyses were not performed.
Lastly, the APSS-6 total score showed a significant positive correlation with the SA-45 psychoticism and paranoid ideation subscales (see Fig. 2).
Fig. 2 Scatter plots and r values for the relationship between the adolescent psychotic-like symptom screener (APSS-6) total score and symptom assessment-45 questionnaire (SA-45) psychoticism and paranoid ideation subscales scores. Note: fitted values (black line); 95% confidence interval (grey area)
This study aimed to assess the psychometric properties of the APSS among adults in Spain. The results showed an adequate overall reliability of the APSS, with similar levels of internal consistency to those reported by the only study that performed a similar examination with Chinese population [16]. We propose an APSS Spanish version with six items, upon the deletion of item 2, as this item shows low item-total correlation (r < 0.30) and the reliability coefficients values increased when it was removed. Importantly, the CFA findings indicated a better fit for the 6-item model compared to the models previously identified in the literature [16] and higher reliability than the 7-item model (though it also showed adequate fit). This finding aligns with the psychometric performance of the APSS items in the adolescent population, where item 2 presented the worst predictive validity of all the questionnaire items [7] and a lower inter-item correlation score [16]. In all, this could point to cross-cultural stability in the psychometric performance of the APSS, also across different age groups.
The internal structure analysis showed that the APSS-6 is a unifactorial scale that assesses a global dimension of subclinical psychotic experiences. This single dimension solution explained a percentage of variance higher than 50%, as recommended in the psychometric literature [28, 29]. Also, the APSS-6 showed high fit indices, suggesting a stable factorial solution of the instrument. Additionally, the single factor structure of the APSS-6 is invariant across age, sex and history of self-reported psychopathology confirming that the APSS-6 items load on a single factor in these groups. However, when used for comparing subclinical psychotic symptoms across males-females, different age range participants or people with or without history of non-psychotic psychopathology the findings should be interpreted with cautiousness, as further measurement invariance was not confirmed in our sample [26, 27].
Together with the solid factorial structure, the APPS-6 shows convergent validity as it correlates significantly both with the psychoticism and paranoid ideation subscales of the SA-45 [19]. The paranoid ideation and psychoticism subscales present a low and moderate positive correlation with the APSS-6 respectively. The magnitude of these correlations could be due to the fact that the SA-45 subscales [19], assess only specific PLEs, thus they do not match with the wider range of attenuated psychotic symptoms included in the APSS-6 [7, 15].
The convergent validity findings indicate that the instrument can indeed be used when the aim is to assess a general dimension of subjective psychotic-like experiences in the general population. This dimension consists of manifestations (1) hallucinatory experiences, (2) thought control beliefs, and (3) self-reported paranoid ideation. This PLEs conceptualisation adds to previous suggestions made based on similar psychometric measures, such as the Launay-Slade Hallucinations Scale, Magical Ideation Scale or Delusion Inventory, among others [30, 31] offering a shorter but equally accurate tool to be considered in future studies when the aim is to provide a unified evaluation of the different PLEs [3].
To the best of our knowledge, this is the first study that has conducted a psychometric analysis of the APSS in adult population, adding on the literature that supports its good psychometric properties with adolescents [7, 16]. Also, the use of rigorous statistical procedures has allowed the examination of several aspects not previously examined, such as internal consistency, convergent validity, construct validity and measurement invariance of the instrument in adult population [13–15]. Importantly, the findings support the use of a web-based version of the APPS suggesting that an assessment of self-reported subclinical psychotic experiences online is feasible and valid, strengthening previous reports [11, 14]. Finally, this is the first study to use well-defined validated instruments for checking the convergent validity of the APSS, establishing the instrument as a handy alternative for a quick and valid evaluation of psychotic-like experiences.
Despite its strengths, the study does not come without limitations. First, administering the APSS in an online format and using non-probabilistic incidental sampling could determine a possible selection bias. Second, the majority of the sample were female, people with high education and employed. Mixed results have been found in the PLEs rates in men and women. That is some studies report higher prevalence of PLEs in men [4, 32] while other studies indicate a higher lifetime prevalence of PLEs in women [3, 33]. Importantly, previous studies have also reported a relationship between low socio-economic and educational level and the manifestation of PLEs in the general population [34, 35]. For these reasons the sample composition may have influenced the measurement invariance values and somehow limit the generalizability. It would thus be relevant to complement the psychometric analysis of the APSS-6 using a sample with similar proportions of participants by sex, educational level, and employment status. Third, the APSS is a self-report screening instrument, therefore, consider potential drawbacks inherent to the evaluation of psychotic-like experiences should be considered such as social desirability, predictive validity or possible response biases and false positives in the report of PLEs [15, 30, 31].
Given the advantages of the APSS use in clinical routine and research, a wider evaluation of its psychometric properties based on representative samples with greater variability and larger size, should be considered. Also, given the role of the subclinical psychotic experiences in the psychosis continuum [1–4] it would be crucial to examine whether the instrument shows good discriminatory validity for distinguishing between people in high risk for developing psychosis, reinforcing its use as a screening instrument in the clinical practice. One may conclude that the APSS-6 Spanish version has adequate psychometric properties that together with its brevity and ease of use provide an ideal candidate for its usage in the everyday practice when targeting the exploration of subclinical psychotic experiences with adult population.
Below is the link to the electronic supplementary material.
Not applicable.
JABG wrote the first draft of the manuscript, collected data, conducted the statistical analyses, and submitted the final version. JABG, SB, EP, AC, and TSG contributed to the interpretation of the results, participated in the drafting of the manuscript, and approved the final version.
This study was partially funded by Universidad Internacional de La Rioja (UNIR, http://www.unir.net) under the Research Projects Strategy RETOS 2018–2020 (grant RT-2018-01), by the Spanish Ministry of Economy, Industry and Competitiveness (MINECO, grant PSI2017-82542-R), and by the Grants for the Translation of Scientific Articles and Publication Fees in Open Access Journals 2023/2024 of the Universidad Internacional de La Rioja (UNIR).
The datasets generated and analyzed during the study are available upon request from the corresponding author.
The study was approved by the International University of La Rioja Research Ethics Committee (RE: PI002/2019) and followed the principles of the Declaration of Helsinki. All participants provided informed consent previously to participate in this study.
Not applicable.
The authors declare no competing interests.
The datasets generated and analyzed during the study are available upon request from the corresponding author.