Authors: Marina Krnic Martinic, Dawid Pieper, Angelina Glatt, Livia Puljak
Categories: Research Article, Definition, Research methodology, Systematic review
Source: BMC Medical Research Methodology
Authors: Marina Krnic Martinic, Dawid Pieper, Angelina Glatt, Livia Puljak
A standard or consensus definition of a systematic review does not exist. Therefore, if there is no definition about a systematic review in secondary studies that analyse them or the definition is too broad, inappropriate studies might be included in such evidence synthesis. The aim of this study was to analyse the definition of a systematic review (SR) in health care literature, elements of the definitions that are used and to propose a starting point for an explicit and non-ambiguous SR definition.
We included overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks. We extracted the definitions of SRs, as well as the inclusion and exclusion criteria that could indicate which definition of a SR the authors used. We extracted individual elements of SR definitions, categorised and quantified them.
Among the 535 analysed sources of information, 188 (35%) provided a definition of a SR. The most commonly used reference points for the definitions of SRs were Cochrane and the PRISMA statement. We found 188 different elements of SR definitions and divided them into 14 categories. The highest number of SR definition elements was found in categories related to searching (N = 51), analysis/synthesis (N = 23), overall methods (N = 22), quality/bias/appraisal/validity (N = 22) and aim/question (N = 13). The same five categories were also the most commonly used combination of categories in the SR definitions.
Currently used definitions of SRs are vague and ambiguous, often using terms such as clear, explicit and systematic, without further elaboration. In this manuscript we propose a more specific definition of a systematic review, with the ultimate aim of motivating the research community to establish a clear and unambiguous definition of this type of research.
In 1990, the term evidence-based medicine (EBM) was coined [1]. It was hailed as a new approach for teaching and practising clinical medicine [2], incorporating “the best available external clinical evidence from a systematic search” [3]. When it comes to the best available evidence about treatment, randomised controlled trials (RCTs) and a systematic review (SR)/meta-analysis are considered the “gold standard” [1].
The EBM movement has been widely adopted, and evidence syntheses are regularly used to support clinical guidelines and recommendations for practice. However, it has been suggested that the EBM might be a movement in crisis [4], as there is “too much evidence” [4]. A study published in 2016 indicated that more than 8000 systematic reviews were being indexed annually in MEDLINE, corresponding to a three-fold increase over the last decade [5]. A search conducted in October 2019 showed that more than 15,000 studies published in 2018 were marked with a systematic review tag in PubMed.
Furthermore, some SRs might actually be misleading, redundant and conflicted [6]. A recent overview of systematic reviews found 12 systematic reviews and two major guidelines about thrombolytic therapy for pulmonary embolism published within less than 2 years. The results of those evidence syntheses were discordant, and the benefit-to-risk ratio was elusive [7]. Inclusion and exclusion criteria played a part in the origin of the discordant results [7].
Just as different inclusion and exclusion criteria might be a problem when conducting a systematic review, the same can happen in overviews of systematic reviews (OSRs) or other types of studies analysing systematic reviews, where results will depend upon inclusion criteria. The problem here is that a standard or consensus definition of a systematic review does not exist.
For example, in a study that reported about the increasing popularity of SRs, Page et al. [5] used the PRISMA-P explanation of a SR [8]. Using a definition when searching for SRs is important because there are studies that may call themselves SRs but are not SRs; we can only speculate that authors use a descriptor SR to label their studies because they are not aware of what a systematic review is, or because systematic reviews are considered to be a higher standard of review.
Therefore, if there is no definition about a systematic review in secondary studies that analyse them or the definition is too broad, inappropriate studies might be included in such evidence synthesis. The aim of this study was to explore and analyse the definition of a systematic review (SR) in health care literature, the elements of definitions that are used and propose a starting point for a new, explicit SR definition.
This was a methodological study, for which we developed a protocol a priori. The study protocol is available from the corresponding author on request.
We aimed to collect definitions of systematic reviews in the health care literature. As numerous collections of SRs have already been published in the past, we relied on existing resources. We used three different i) OSRs about healthcare interventions, ii) studies that have analysed the methodological quality of systematic reviews and iii) relevant textbooks/internet sources that define systematic reviews.
We included OSRs and methodological studies identified previously by Pieper et al. [9, 10]. We used a validated filter for retrieval [11]. Additionally, we searched for EBM-related and epidemiology handbooks published in English or German. There was no systematic search for the handbooks. We compiled a list of relevant handbooks known to us, using the same methodological approach as described by other authors in similar projects [12].
Furthermore, we searched Google Scholar between the 24th January, 2018 and the 7th February, 2018 using the following search “definition of a systematic review”, “definition of systematic review”, “definition of the systematic review”, “defined a systematic review”, “defined the systematic review”, “systematic review was defined”. Those phrases were used to search any part of the manuscript – without any restrictions. We analysed the first 50 search results for each search phrase if there were more than 50 search phrases retrieved for a phrase. We excluded duplicate manuscripts found by searching multiple sources before starting the analysis.
We piloted a data extraction form in Microsoft Excel on a sample of ten manuscripts. Two authors piloted the data extraction form (LP, MKM). Furthermore, based on the advice of a third author (DP), the form was further refined. Thus, in an iterative process between the authors, the form was modified where necessary to avoid any misunderstandings or later disagreements.
We extracted the following i) whether the analysed literature sources reported a definition of a systematic review and ii) inclusion and exclusion criteria defining systematic reviews. We extracted the relevant exclusion criteria when they had explicit statements about studies that were not included because certain aspects of them were not considered to be characteristics of a systematic review.
When we found a definition or inclusion/exclusion criteria defining systematic reviews, the text was extracted verbatim. Subsequently, from those definitions and the inclusion/exclusion criteria, we extracted elements of a systematic review definition. The definition elements were defined as distinct methodological components and their attributes. Elements described with similar adjectives were not combined; instead, we presented all unique elements separately in order to present a wide variety of adjectives and attributes used in the definitions of SRs. We did not use an a priori defined list of those elements; instead, we presented elements that we found in the analysed sources of information and we kept expanding the list of elements as we found new variations of the elements of the SR definition.
One of the elements of a definition we used was the presence of a meta-analysis (MA), but only if the authors explicitly indicated that the MA was considered as defining characteristic of a SR. For example, in a study published in 2013, Aziz wrote explicitly that SRs without MA were not included because “these were not considered SRs” [13].
Extracted individual elements of the SR definition were then categorised into groups. For example, if a SR definition “systematic search”, “reproducible search” or “keywords searched”, those elements were sorted into a category called “Search”. The process of forming categorisations was iterative between the authors until we reached a consensus about the categories that will be used.
We extracted reference(s) for a definition of a systematic review or inclusion criteria referring to the systematic review, if available. We recorded the 2017 Journal Citation Reports (JCR) Journal Impact Factor (JIF) of a journal from the Web of Science. We hypothesised that manuscripts published in journals with a higher JIF would have a higher prevalence of SR definitions, due to the higher reporting standards.
For all data, one author (MKM) extracted data and the second author (LP) verified the extractions. Furthermore, one author (LP) categorised the definition elements and the second author verified the categorisations (MKM). Any discrepancies in opinion were resolved via discussion.
For the analysis of definitions from textbooks and Internet sources, we extracted the definitions verbatim and indicated the field from which the definition came from, such as medicine, psychology and social sciences. During the analyses of the textbooks, if the definition in the text was supplemented with a table, we treated this as one source of information and the extracted elements of the SR definition from both text and table. One author extracted data and the second author verified the extractions from the textbooks and Internet sources.
Descriptive statistics, including frequencies and percentages, were used to describe the categories of elements of a systematic review definition/inclusion criteria. We also analysed the frequency of each category by counting the categories of elements that were used in each source. If at least one element was used in a certain category, we considered that this category of elements was present in the information source. We expressed the JIF as the mean and standard deviation (M ± SD), we used a t-test to analyse the difference in the JIR between the information sources with and without a SR definition. For the analyses, we used the MedCalc statistical software, v 15.2.1 (©MedCalc Software bvba, Ostend, Belgium). The statistical significance was set at P < 0.05.
After searching for OSRs and methodological studies, from the 347 identified full texts, we included 308 studies. We excluded 39 studies because 31 were duplicates and an additional eight manuscripts were excluded because they were written in Chinese or did not fit our inclusion criteria (commentaries, traditional narrative reviews, reviews of an unspecified type of reviews or analysed rapid reviews).
By searching Google Scholar we found 531 hits. Based on the limits we set, analysing 50 hits per search phrase, we analysed a total of 238 bibliographic records from Google Scholar. After removing the duplicates that we already had in the first cohort of the included studies, we included the remaining 200 manuscripts from this cohort of studies. Additionally, we analysed 27 textbooks. In total, we analysed 535 sources of 508 manuscripts from peer-reviewed journals and 27 from textbooks.
Among the 535 analysed sources of information, 188 (35%) defined what they consider to be a systematic review, 62 (18%) had an inclusion criteria in the methods that allowed us to extract information about what the authors considered to be a systematic review and 59 (18%) had exclusion criteria that we used as well for determining the authors’ definition of a SR. Some sources of information had both a definition of a SR and/or inclusion/exclusion criteria; in total there were 226 sources of information from which we could extract information related to the authors’ definition of a SR.
Among the 508 manuscripts, we found a JIF for 401 manuscripts, of which 113 had a SR definition, and 288 did not. Journals that did not provide SR a definition had a higher JIF (4.4 ± 5.1) than those with a definition (3.7 ± 4.5), but this difference was not significant (P = 0.099).
Many of the analysed sources explicitly mentioned relevant organisations, checklists and databases for defining what they considered to be a SR, some of the analysed sources of information only provided literature references to support their definitions or inclusion/exclusion criteria.
Explicit mentions of the names of the organisations, checklists, databases associated with a definition of SRs or criteria for the inclusion of SRs were found in 43 out of 535 (8%) analysed sources of information. Those were Cochrane (N = 24), the PRISMA statement (N = 13), criteria of Database of Reviews of Effect (DARE) (N = 5), National Institute for Health and Care Excellence (NICE) (N = 3), NHS Centre for Reviews and Dissemination (N = 3), Campbell collaboration (N = 2) National Health and Medical Research Council (N = 1), QUOROM (QUality Of Reporting of Meta-analyses) recommendations (N = 1), Guidelines from Agency for Healthcare Research and Quality (AHRQ) (N = 1), Institute of Medicine (IOM) (N = 1) and author Andy Oxman (N = 1), referred to as the “Oxman criteria“. Cochrane was mentioned most commonly, either as a reference to a whole organisation, the Cochrane Handbook for Systematic Reviews of Interventions or a specific Cochrane the Dutch Cochrane Centre in one source of information. Details about the definitions and references provided in those 43 studies are shown in Additional file 1: Table S1. The most commonly used supporting references in those studies were the manuscripts by Moher et al. and Liberati et al. describing the PRISMA statement, the PRISMA-P checklist, and the Cochrane Handbook (Additional file 1: Table S1).
The most commonly used literature references that were used to support the statements provided in the definitions of SRs or the inclusion/exclusion criteria were also manuscripts describing the PRISMA statement and Cochrane Handbook (Additional file 1: Table S2).
After analysing all the definitions of SRs and the inclusion/exclusion criteria for SRs, we extracted 188 individual elements of a SR definition; we categorised them into the following 14 self-identification, indexing, aim/question, overall methods, search, identification of studies, selection of studies, study eligibility, data extraction, quality/bias/appraisal/validity, analysis/synthesis, describing included studies, reporting and unclear (Table 1). Table 1Categories and elements of systematic review definition found in health care literature; percentage calculated from 226 sources of information that had a SR definition, or inclusion/exclusion criteria that could be used for extracting individual elements of SR definitionCategoryElement of definitionN (%)Self-identified as a systematic reviewManuscript that identifies itself as a systematic review in title, abstract or in methods30 (13)IndexingIndexed as SR1 (0.4)Aim/research questionSpecific research question66 (29)Clearly stated set of objectives12 (5.3)Clearly formulated research question11 (4.8)Focused research question3 (1.3)Reported research question2 (0.9)Clinical question including participants, interventions, controls, outcomes and study design (PICOS)2 (0.9)Explicit clinical question1 (0.4)Clearly stated topic of review1 (0.4)Explicitly reported pre-defined objectives1 (0.4)Stated goal implied a critical and comprehensive intent1 (0.4)Clear statement of the topic1 (0.4)Defined clinical topic1 (0.4)Explicit statement of questions being addressed1 (0.4)Overall methodsSystematic methods22 (9.7)Explicit methods21 (9.2)Systematic method to minimize risk of bias9 (4)Systematic approach, in an attempt to minimize biases and random errors, documented in the Materials and Methods section8 (3.5)Explicit method to minimize risk of bias7 (3.1)Reproducible methods5 (2.2)Using a systematic approach5 (2.2)Methods described in explicit detail4 (1.8)Well-defined methods2 (0.9)Overall methods defined study as systematic review1 (0.4)Overall Conduct defined study as a systematic review1 (0.4)Systematic review methodology on closer inspection of the methods section1 (0.4)Specific methods1 (0.4)Repeatable methods1 (0.4)Rigorous methods1 (0.4)Different components of the review process documented in the ‘methods section’1 (0.4)Using methods to provide more reliable findings1 (0.4)Using methods from which conclusions can be drawn1 (0.4)Using methods based on which decisions can be made1 (0.4)Exhaustive review of the literature1 (0.4)Systematic approach1 (0.4)SearchSystematic search29 (13)Reported search strategy13 (5.8)Comprehensive search strategy12 (5.3)Searched at least two databases/sources10 (4.4)Exact search criteria reported9 (4.0)Searched at least one database9 (3.9)Reported search methods7 (3.1)Attempt to collate all empirical evidence7 (3.1)Reported all information sources6 (2.6)Transparent search strategy6 (2.6)Detailed and comprehensive search strategy (as identified naming of databases and years of searching and example or actual terms)4 (1.8)Detailed and specific search strategy with key-words that enabled reproduction of the literature search4 (1.8)Names of databases reported4 (1.8)Explicit search criteria that are available to review3 (1.3)Description of data sources and search dates2 (0.4)Keywords searched2 (0.9)Detailed search of the literature for relevant studies2 (0.9)Explicit description of search strategy2 (0.9)Adequate searching methods2 (0.9)Replicable search method2 (0.9)Reported search sources1 (0.4)Description of sources1 (0.4)Reported details of databases searched1 (0.4)Reported dates of search1 (0.4)Included relevant search strategy1 (0.4)Adequate search strategy1 (0.4)Appropriate search strategy1 (0.4)Detailed search strategy1 (0.4)Non-selective search strategy1 (0.4)Explicit search strategy1 (0.4)Prescriptive search strategy1 (0.4)Reproducible search strategy1 (0.4)Rigorous search process1 (0.4)Explicitly reported search strategy details1 (0.4)Thorough search of evidence1 (0.4)Comprehensive search of evidence1 (0.4)Reported search processes1 (0.4)Extensive use of search string combinations1 (0.4)Description of evidence retrieval methods1 (0.4)Explicit and organized approach to searching1 (0.4)Attempt to search all empirical evidence1 (0.4)Adequately attempt to retrieve all relevant data1 (0.4)Review trying to collect all available evidence1 (0.4)Structured search of bibliographic and other databases1 (0.4)Searched at least Medline1 (0.4)Searched at least two databases (of which one is Medline)1 (0.4)Identification of studiesExplicit methods to identify relevant research14 (6.2)Systematic methods of identification of studies10 (4.4)Attempt to identify all empirical evidence6 (2.6)Reported methods for identification of studies2 (0.9)Transparent procedure to find relevant research2 (0.9)Formal process of identifying literature1 (0.4)Selection of studiesExplicit methods to select relevant research14 (6.2)Systematic methods of selection of studies13 (5.8)Reported methods for selection of studies6 (2.6)Transparent selection of studies2 (0.9)Reproducible selection of studies4 (1.8)Reproducible approach for selecting the studies1 (0.4)Clear description of selection criteria1 (0.4)Clear study selection criteria1 (0.4)Relevant study selection criteria1 (0.4)Detailed description of the studies’ selection process (number of articles included and excluded in each step)1 (0.4)Study eligibilityReported inclusion and exclusion criteria31 (14)Pre-defined/pre-specified eligibility criteria20 (8.8)Outcome defined using a validated tool or diagnostic criteria13 (5.8)Only Cochrane systematic reviews12 (5.3)Reported inclusion criteria6 (2.6)Explicitly reported inclusion and exclusion criteria6 (2.6)Articles that meet PRISMA definition of a systematic review5 (2.2)Definitions of the population(s), intervention(s), comparator(s) and outcome(s) of interest2 (0.9)Inclusion/exclusion criteria that are relevant in terms of the PICO framework3 (1.3)Reviews published in Database of Reviews of Effects (DARE)2 (0.9)Reviews were judged to be systematic if they synthesized peer reviewed articles1 (0.4)Studies meeting minimum methodological standards1 (0.4)Reference to study designs1 (0.4)Data extractionSystematic data collection12 (5.3)Systematic methods to extract data4 (1.8)Explicit methods to collect data3 (1.3)Data extraction by 2 independent reviewers2 (0.9)Reported data abstraction from trials2 (0.9)Independent data extraction1 (0.4)Explicit approach to extracting1 (0.4)Organized approach to extracting1 (0.4)Explicit methods to extract data1 (0.4)Performed data extraction1 (0.4)Extracting the information from the studies following a priori protocol1 (0.4)Quality, bias, appraisal, validityQuality assessment of evidence27 (12)Critical appraisal of the studies25 (11)Risk of bias assessment19 (8.4)Systematic methods to critically appraise relevant research13 (5.8)Explicit methods to critically appraise relevant research13 (5.8)Reported validity assessment11 (4.9)Attempt to appraise all empirical evidence6 (2.6)Full assessment of methodological quality of included studies5 (2.2)Consideration of internal and external validity of the research3 (1.3)Provided sufficient details about individual included studies to enable assessment of quality by a reader2 (0.9)Reported at least one or more aspects of validity assessment of original studies2 (0.9)Transparent procedures to evaluate relevant research2 (0.9)Full report of methodological quality of included studies1 (0.4)Transparent process to minimize risk of bias1 (0.4)Explicit approach to critically evaluating studies1 (0.4)Organized approach to critically evaluating empirical literature1 (0.4)Systematic approach for assessing the studies1 (0.4)Reproducible approach for assessing the studies1 (0.4)Assessed methodological features of the included studies1 (0.4)Adequate methods to appraise included studies1 (0.4)Transparent methodological criteria are used to exclude papers that do not meet an explicit methodological benchmark1 (0.4)Evaluate the retrieved studies using prospectively defined methodological criteria1 (0.4)Analysis, synthesisSynthesis of results34 (15)Presence of meta-analysis19 (10)Systematic methods of analysis of studies18 (8.0)Explicit methods to analyze data17 (7.5)Systematic synthesis of findings10 (4.4)Quantitative synthesis9 (4.0)Synthesis of the included evidence, whether narrative or quantitative7 (3.1)Attempt to synthesize all empirical evidence6 (2.6)Systematic analysis of results2 (0.9)Unbiased synthesis of study findings2 (0.9)Transparent procedures to synthesize the results of relevant research2 (0.9)Analyze results appropriately1 (0.4)Systematic analysis1 (0.4)Plausible analysis of data1 (0.4)Plausible synthesis of data1 (0.4)Summary of results1 (0.4)Systematic analysis1 (0.4)Meta-analysis or best evidence synthesis1 (0.4)Formal analysis contained in the methods1 (0.4)Makes judgement about research question1 (0.4)Relying on statistical significance to make judgments about what works1 (0.4)Transparent process of interpretation of the findings of the studies included in the review1 (0.4)Rigorous conclusions about outcomes1 (0.4)Describing included studiesSystematic presentation of characteristics of included studies4 (1.8)Systematic synthesis of characteristics of included studies4 (1.8)Clearly identified all included studies2 (0.9)Reported trial characteristics1 (0.4)Systematic presentation of main information1 (0.4)Described main characteristics of included studies1 (0.4)Adequate methods to describe included studies1 (0.4)Description of the number and nature of included studies1 (0.4)Description of the types of primary studies included1 (0.4)Accounted for identified studies1 (0.4)ReportingUsed PRISMA or predecessor guidelines for reporting3 (2)Presented results appropriately1 (0.4)Systematic presentation of findings1 (0.4)Flow chart present1 (0.4)Reported level of evidence for their recommendations1 (0.4)Reported sufficient information to allow a level of evidence grading1 (0.4)Published in a journal conforming to PRISMA standards1 (0.4)A review that has methods and results section1 (0.4)Unclear“It was apparent in the text that a systematic review had been undertaken”4 (1.8)“Reviews were included if they were systematic”1 (0.4)
Elements were sorted according to those categories (Table 1). The highest number of SR definition elements was found in categories related to searching (N = 51), analysis/synthesis (N = 23), overall methods (N = 22), quality/bias/appraisal/validity (N = 22) and aim/question (N = 13) (Table 1).
Among the 226 sources of information that had a SR definition or inclusion/exclusion criteria that could be used for extracting individual elements of a SR definition, 59 used only one category, 62 used two categories, while 105 used from three to ten categories of the SR definition elements. When we looked at the combinations that were used, none of the combinations of various categories was used more than ten times. The most commonly used combination of SR definition categories was used in nine of the manuscripts/books, and it used the following five i) aim/research question, ii) search, iii) study eligibility, iv) quality, bias, appraisal, validity and v) analysis/synthesis. However, those nine manuscripts had different wording of the SR definition, as shown in Additional file 2: Table S3; they did not use one consistent definition.
The same five categories were the most commonly used SR definition categories in our sample of information sources, with the following i) search (N = 122), ii) aim/research question (N = 93), iii) analysis/synthesis (N = 90), iv) study eligibility (N = 89) and v) quality, bias, appraisal, validity (N = 81).
We found that authors of manuscripts and textbooks use various definitions of systematic reviews; in 535 sources of information, we found 188 different elements of a SR definition. The most commonly used categories of SR definition elements were related to searching, analysis/synthesis, overall methods, quality/bias/appraisal/validity and aim/question. The most commonly used reference resources were the Cochrane and PRISMA statement [14, 15].
However, as our study showed, there is no uniformly used definition of a SR. We analysed various sources of information, including overviews of SRs and methodological studies about SRs because those studies included SRs and we expected that therefore they should provide a definition of a SR. Our expectations were not met; as we found that one-third of those information sources used an explicit definition of a SR. In another one-third of the information sources, we found either inclusion or exclusion criteria, from which we could deduce what they consider to be, or not to be, a SR.
Also, we found that journals that did not provide SR definition had a higher JIF than those with a definition, but this difference was not significant. This finding was not in line with our hypothesis, and it shows that in this respect journals with higher JIF did not have higher expectations from authors in terms of transparent reporting about what was considered to be a SR.
When extracting the elements of SR definitions, we tried to be as detailed as possible, to capture various terminology used in those definitions. We found many variations of similar concepts, but also many vague terms. Such vague terms were frequently reflected in the usage of the word systematic, such “systematic methods”, “systematic approach”, “systematic search”, “systematic synthesis”, “systematic analysis” and “systematic presentation”, without actually explaining what systematic means. We also found two expressions that were completely unclear about what the authors consider to be a SR, including “Reviews were included if they were systematic” and “It was apparent in the text that a systematic review had been undertaken”.
There were ten elements of a SR definition that used the type and number of sources that were searched in a SR, as an element of the SR definition. It has been suggested previously that a minimum number and types of sources should define SRs because searching only one database may not be universally considered a systematic search [16].
It could be argued that our categorisation was too detailed, as some of our categories of SR definition elements sound similar, for example, categories search, selection of studies, identification of studies and study eligibility. We left those categories as they were on purpose, because it may not be perfectly obvious what the difference between them is; for example, the term selection of studies in the Cochrane reviews is reserved for the description of the screening of abstracts and full texts, but it is unclear whether all authors use this term in the same context. Furthermore, it is unclear whether the identification of studies refers to searching, screening or eligibility, i.e. the inclusion/exclusion criteria. Because of this ambiguity, we chose to present more detailed categories.
The most commonly used individual five categories of the SR elements were also used as the most common combination of elements in the analysed sources of information, but only nine manuscripts used this combination of the five elements. Those five categories of elements are also included in the definition of SRs from the Cochrane Handbook [14].
In section 1.2.2 of the Cochrane Handbook, titled What is a systematic review?, the following definition can be found [quote]: “A systematic review attempts to collate all empirical evidence that fits the pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimising bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993). The key characteristics of a systematic review a clearly stated set of objectives with pre-defined eligibility criteria for the studies; an explicit, reproducible methodology; a systematic search that attempts to identify all the studies that would meet the eligibility criteria; an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias; and a systematic presentation, and synthesis, of the characteristics and findings of the included studies” [14].
Also, Cochrane was the most commonly mentioned organisation in the definitions of SRs; 13% of the manuscripts/textbooks mentioned Cochrane as a source of the SR definition. Therefore, one could argue that the Cochrane’s definition could be used as a formal definition of what a SR is. However, the Cochrane’s definition is also vague, as it is unclear what it means “explicit, systematic methods” or “explicit, reproducible methodology”. Someone can explicitly describe the methodology that is not adequate. This inadequate methodology may also be reproducible, but that does not mean that it is good. Furthermore, the Cochrane definition of a SR repeatedly uses the adjective “systematic”, without explaining what the meaning of systematic is.
Two references used in Cochrane’s definition of a SR are those of Antman et al. [17] and Oxman et al. [18]. We also analysed which references were used to support the definitions of SRs in the manuscripts and textbooks; we found that the authors most commonly referred to the PRISMA statement [15] and Cochrane Handbook. However, the definition of SRs from the PRISMA statement manuscripts also uses vague terms such as clearly, systematic and explicit, without going into details of what they entail [15].
The research community would benefit from having a very specific definition of a SR. The five most commonly used SR definition elements that we identified could be used to create a more elaborate and unambiguous definition of a SR. We believe that the international research community should create an unambiguous SR definition; we hope that this study will be a starting point in that direction. As a first step, we suggest starting with the following
A systematic review is a review that reports or includes the i)research questionii)sources that were searched, with a reproducible search strategy (naming of databases, naming of search platforms/engines, search date and complete search strategy)iii)inclusion and exclusion criteriaiv)selection (screening) methodsv)critically appraises and reports the quality/risk of bias of the included studiesvi)information about data analysis and synthesis that allows the reproducibility of the results
Some of those elements are mentioned in the SR definition from the Cochrane Handbook [14], as shown above, but the Cochrane’s definition still leaves a lot of ambiguity in several aspects. Those elements should be more specific in future. For example, which details should the clinical question report, how many databases/sources should be searched to be considered systematic, whether key methodological aspects (screening of titles and abstracts, screening of full texts, data extraction and risk of bias assessment) should be done by two authors independently or done by one author and verified by another. The naming of the databases is important for ensuring transparency and reproducibility, which should be features of a systematic approach. Those and other considerations should be taken into account in further efforts to clarify what exactly makes a SR.
Information presented in this manuscript could help inform a consensus meeting or a similar gathering where interested SR researchers could contribute to standardising a SR definition. A similar approach was recently suggested for the definition of a predatory journal. Cobey et al. have conducted a scoping review in which they summarised the literature on predatory journals, described its epidemiological characteristics and extracted empirical descriptions of the potential characteristics of predatory journals. In their conclusions, they informed readers that the results will be shared with attendees that will attend a stakeholder meeting seeking to develop a standardised definition for what constitutes a predatory journal [19].
One limitation of our study could be the use of information sources published within a certain period of time. However, this type of work, which relies on analysis of published literature, usually suffers from a time lag. Each new update of the search results in new literature sources to analyse, and time lag appears again by the time analysis is completed.
Furthermore, in our approach, we analysed both expressions that appeared to be definitions of SRs and the characteristics of SRs eligible for inclusion. It may be considered that the inclusion criteria for SRs are not eligible elements to define what a SR is. However, we considered that the eligibility and inclusion criteria which describe SRs would be useful in our analysis, as we seldom found explicit statements about the definition of a SR. We consider that the range of descriptors we found indicates a very rich vocabulary used by authors who are defining or searching for SRs and that our approach is an adequate starting point towards building a future consensus definition of a systematic review. Likewise, it could be argued that we are mixing a definition of a SR with measures of the quality of a SR. However, in the absence of an existing definition, we believe that we should assess all the descriptors used for SRs and report them explicitly and transparently, then readers can see for themselves that some of those may overlap with quality descriptors. Also, for searching Google Scholar we used a limited number of phrases. Google used to include details for searching the advanced interface, which is no longer available but this search information could be available from other sites (mostly libraries) which we did not utilize.
Our analysis is also limited by the fact that we only focused on the definitions, while we acknowledge that some relevant information might also be found in the explanatory text to the definition, if available.
The majority of manuscripts that include SRs actually do not provide a definition of what they consider to be a SR. The most commonly used reference sources of a SR definition use vague and ambiguous terms. We propose a new definition of a systematic review, which is open for further commenting and elaboration, with the aim of motivating the research community to create a more specific definition of this type of research.
Additional file Table S1. Definitions of systematic reviews which explicitly quoted specific organizations/checklists/criteria. A table that contains definitions of systematic reviews, extracted from analyzed data sources, in which the authors hav explicitly quoted specific organizations, or checklists or criteria. Table S2. All references that were used to support definition of systematic review, or inclusion/exclusion criteria that could be used as a proxy for a definition. A list of references that autohrs of analyzed information sources have used in their manuscript to support either a definition of systematic review, or inclusion criteria, or exclusion criteria that could be used as a proxy indicator of a systematic review definition Additional file Table S3. Definitions of a systematic review from nine manuscripts with the most commonly used combination of categories. A file containing verbatim extracted definitions that were used in nine manuscripts that had the most commonly used combination of categories of systematic review definitions