Rapid Arterial Occlusion Evaluation (RACE) Tool in Detecting Large Cerebral Vessel Occlusions; a Systematic Review and Meta-Analysis

Authors: Iman Chehregani Rad, Amir Azimi

Abstract

Introduction:

Large vessel occlusion (LVO) strokes are linked to higher mortality rates and a greater risk of long-term disability. This study aimed to evaluate the diagnostic performance of the Rapid Arterial Occlusion Evaluation (RACE) tool in detecting LVO through a systematic review and meta-analysis.

Methods:

A comprehensive search was conducted across online databases including PubMed, Embase, Scopus, and Web of Science, up to June 25th, 2023. Additionally, a manual search on Google and Google Scholar was performed to identify studies that assessed the diagnostic accuracy of the RACE scale in detecting LVO among patients with stroke symptoms.

Results:

Data extracted from 43 studies were analyzed. The optimal cut-off points were determined to be 3 and 4, with a sensitivity of 0.86 (95% confidence interval (CI): 0.78, 0.91) and specificity of 0.57 (95% CI: 0.49, 0.67) for cut-off ≥3, and a sensitivity of 0.78 (95% CI: 0.70, 0.84) and specificity of 0.68 (95% CI: 0.59, 0.75) for cut-off ≥4. Subgroup meta-regression analysis revealed significant variations in sensitivity and specificity. RACE scale’s sensitivity was significantly higher in LVO detection in suspected stroke cases, in pre-hospital settings, prospective design studies, and when considering both anterior and posterior occlusions for LVO definition. RACE scale’s specificity was significantly higher when evaluating confirmed stroke cases, in-hospital settings, and considering only anterior occlusions for LVO definition and retrospective design studies. Notably, RACE exhibited higher sensitivity and specificity when utilized by neurologists and physicians compared to other emergency staff. Despite these variations, our study found comparable diagnostic accuracy across different conditions.

Conclusion:

A high level of evidence indicates that the RACE scale lacks promising diagnostic value for detection of LVOs. A sensitivity range of 0.69 to 0.86 is insufficient for a screening tool intended to aid in the diagnosis of strokes, considering the substantial morbidity and mortality associated with this condition.

1. Introduction:

Large vessel occlusion (LVO) is a critical condition characterized by the blockage of major cerebral arteries, accounting for 24-46% of acute ischemic strokes (AIS). Compared to non-LVO ischemic strokes, LVO strokes are linked to higher mortality rates and a greater risk of long-term disability (1, 2). Notably, a meta-analysis revealed that LVO patients had a 64% incidence of unfavorable outcomes and a mortality rate of 26.2%, in contrast to the 24% poor outcomes and 1.3% mortality rate observed in non-LVO cases (2).

Endovascular treatment (EVT) has emerged as the gold standard in management of LVO (3). Guidelines strongly recommend initiating EVT within 6 hours from the patient's last known well time (4). Studies suggest that even within a window of 6 to 16 hours since the last known well time, LVO patients can benefit from EVT (5). Some studies have even extended this treatment window to 6 to 24 hours (3). However, not all healthcare facilities have the capability to perform EVT. Thus, it becomes imperative to rapidly identify LVO cases and bypass primary emergency centers, directing patients to Comprehensive Stroke Centers (CSCs) to ensure they don't miss the optimal treatment window (6).

Diagnosing LVO typically relies on advanced imaging techniques such as magnetic resonance imaging (MRI) and computed tomography (CT) scans (1). Nonetheless, specialized scales have been designed for rapid and accurate LVO diagnosis, including, FAST-ED (Face-Arm-Speech-Time-Eye-Disturbance), VAN (Vision-Aphasia-Neglect), PASS (Prehospital Acute Stroke Severity), LEGS (Lever Arm Test-Gaze-Eyes-Visual Field-Speech), CPSSS (Cincinnati Prehospital Stroke Severity Scale), LAMS (Los Angeles Motor Scale), RACE (Rapid Arterial Occlusion Evaluation), and NIHSS (National Institutes of Health Stroke Scale) (7-14). Among these, the RACE scale stands out for its simplicity and rapid assessment. The RACE scale comprises five essential assessment of facial palsy, arm motor function, leg motor function, head and gaze deviation, as well as aphasia and agnosia, with scores ranging from 0 to 9 (7).

While a previous study has systematically evaluated the RACE scale's performance (15), its clinical utility, applicability in diverse settings, with varying assessors, and under various study designs remain underexplored. Furthermore, the previous systematic review is outdated and it is essential to incorporate the findings of the many newly published studies. Therefore, the objective of this systematic review and meta-analysis was to assess the diagnostic accuracy of the Rapid Arterial Occlusion Evaluation (RACE) scale using different cut-off values and to ascertain its effectiveness in detecting LVO across diverse clinical conditions.

2. Method

2.1 Study design and search strategy

This systematic review and meta-analysis aimed to evaluate the effectiveness of the RACE tool in detecting LVO among suspected or confirmed stroke patients. The study was guided by the PICO (patient, intervention, comparison, outcome) framework, where P referred to suspected or confirmed stroke patients, I to the RACE scoring tool, C to radiographic imaging techniques such as CT angiography (CTA), Magnetic Resonance Imaging (MRI), Transcranial Doppler ultrasound (TCD) or Digital Subtraction Angiography (DSA), and O to patients with final diagnosis of LVO. A comprehensive search strategy was employed, covering databases including Medline (via PubMed), Scopus, Embase, and Web of Science, up until June 25th, 2023. Relevant keywords related to "RACE" and "Stroke" were selected using MeSH and Emtree terms, guided by consultations with field experts and reviews of pertinent literature. Additionally, manual searches were performed on the Google and Google Scholar search engines. The complete search strategy is provided in Supplementary Material 1.

2.2 Selection criteria

Inclusion criteria encompassed articles investigating the utility of the RACE tool in detecting LVO among suspected or confirmed stroke patients, irrespective of LVO definition and setting. Exclusion criteria comprised reviews, studies lacking non-LVO control groups, studies without reported cut-off points, studies with no required data, and duplicate publications.

2.3 Data collection

After eliminating duplicates, two reviewers independently assessed the retrieved records. The screening process was conducted in two an initial evaluation of titles and abstracts, followed by a comprehensive examination of full-text articles to determine their eligibility based on selection criteria. Data from selected articles were synthesized and extracted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (16). The extracted information included study characteristics such as the first author's surname, publication year, study design, age and sex distribution of patients, sample size, the number of patients with LVO, the LVO definition used in the study, and the reference standard test employed. Additionally, diagnostic performance metrics of the RACE tool, including true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), were extracted. These metrics were subsequently used to calculate sensitivity, specificity, and other relevant variables.

2.4 Quality assessment and certainty of evidence

The Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool (17) was used to evaluate the quality and risk of bias in the included studies. QUADAS-2 assesses bias and applicability across domains related to patient selection, index test, reference standard, and flow and timing, and each domain’s risk is rated as high, unclear, or low. The certainty of evidence was determined utilizing the Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) approach (18).

2.5 Statistical analysis

Statistical analyses were conducted using STATA v17.0 statistical software, with calculations performed using the 'midas' package. We used the collected TP, TN, FP, and FN values to calculate diagnostic accuracy metrics for the RACE tool, including sensitivity, specificity, the area under the receiver operating characteristic curve (AUC), diagnostic odds ratio (DOR), positive likelihood ratio (PLR), and negative likelihood ratio (NLR). Since the diagnostic value of RACE was reported at various cut-off points, we stratified the results based on reported cut-off points. In articles with multiple reports for various LVO definitions (anterior LVO and both anterior and posterior LVO) and population (suspected or confirmed stroke), the report with the broadest definition was included in the main analysis. Heterogeneity among the studies was assessed using the I2 test, and we applied a bivariate random-effects model for our analysis. Additionally, subgroup analysis was performed on studies based on criteria such as the study population (suspected or confirmed stroke), study design, the setting in which they were conducted, assessor specialties, and the definition of LVO. Publication bias was investigated using Deeks’ funnel plot asymmetry test.

3. Results

3.1. Study characteristics

A total of 345 articles were initially collected; 332 were identified through a systematic search and 13 through manual search. After duplicate removal and review of titles and abstracts, 60 articles underwent a comprehensive full-detail assessment. Ultimately, 43 articles were included in this systematic review and meta-analysis (7, 8, 11, 13, 19-57) (Figure 1). Among the selected articles, 20 were conducted in the USA, 7 in Spain, 4 in China, and 3 in the Netherlands. The rest of the articles were conducted Switzerland, Finland, Brazil, France, Hungary, Australia, and Denmark. One of the included articles involved multinational study populations. Twenty-one studies had a prospective design, and 21 were retrospective studies. One study had separate retrospective and prospective reports.

In total, the collective sample size encompassed 37,811 patients. The average age of participants ranged from 62.04 to 73.5 years. In 13 articles, sex-specific patient numbers were not reported. However, among the remaining articles, out of a total of 31508 patients, 16016 (50.83%) were male. Regarding the RACE cut-off points, seven articles used a cut-off point of ≥1, eight used ≥2, eleven used ≥3, twelve used ≥4, thirty-seven used ≥5, ten used ≥6, nine used ≥7, eight used ≥8, and seven used =9. RACE evaluations were conducted in a prehospital setting in 24 studies, while they were conducted in an in-hospital setting in 17 studies. One study did not specify the setting, and in another study, data were collected from both prehospital and in-hospital medical records. RACE assessments were carried out by physicians and neurologists in 12 articles and by other Emergency Medical Services (EMS) and Emergency Department (ED) staff in 25 studies. Four studies did not mention the assessors, and in two studies, it was noted that the assessors were members of the stroke team certified in National Institutes of Health Stroke Scale (NIHSS) or were certified research personnel, although their specific specialty remained unclear. In 20 studies both anterior and posterior circulations were evaluated for LVO, whereas the definition of LVO was restricted to anterior circulation occlusions in 7 studies. Two studies, evaluated their study population based on each of the above definitions of LVO, separately. Additionally, in 14 studies, the exact definition of LVO was not provided. Table 1 provides the characteristics of the included studies.

3.2. Meta-analysis

3.2.1. Diagnostic value of RACE in detection of LVO

In the meta-analysis of RACE for LVO detection, performance varied across different cut-off points. Duo to scarcity of data, conducting an analysis at ≥1 cut-off was not possible. For ≥2 cut-off, RACE had an AUC of 0.71, sensitivity of 0.94, and specificity of 0.33 with a DOR of 7. The ≥3 cut-off yielded an AUC of 0.79, sensitivity of 0.86, specificity of 0.57, and DOR of 8. At ≥4 cut-off, the AUC was 0.79, sensitivity and specificity were 0.78 and 0.68, respectively, and DOR was 7. Raising the threshold to ≥5 resulted in an AUC of 0.80, sensitivity of 0.69, specificity of 0.80, and DOR of 9. The ≥6 cut-off had an AUC of 0.77, sensitivity of 0.6, specificity of 0.80, and DOR of 6. Using ≥7 cut-off, the AUC was 0.70, sensitivity was 0.43, specificity was 0.89, and DOR was 6. At ≥8 cut-off, sensitivity was 0.26, specificity was 0.95, and DOR was 6. Finally, for ≥9 cut-off, sensitivity was 0.12, specificity was 0.99, and DOR was 10 (Table 2). Due to the low sensitivity observed at cut-offs ≥8 and ≥9, the AUC values for these specific cut-off points are not reported in the results and the table (Supplementary Figure 1-9).

After evaluating the AUC, sensitivity, and specificity, the optimal cut-off points for the RACE scale in detecting LVO were determined to be ≥3 and ≥4. However, since the majority of articles used a cut-off of ≥5 as the optimal, further analyses were conducted using this cut-off.

3.2.2. Subgroup analysis

At the cut-off point of ≥5, there was significant heterogeneity among the studies (I2 = 100.0%). As a result, we performed a meta-regression analysis to explore the potential factors responsible for this observed heterogeneity across the studies. Our analysis revealed that the most likely contributors to this heterogeneity included variations in study settings, the specialties of the assessors using the RACE scale, and the definition of LVO (p<0.001).

Subgroup analysis unveiled significant sensitivity and specificity variations based on settings, assessor specialty, population, LVO definition, and study designs. Higher sensitivity was observed when RACE investigated suspected stroke patients (p < 0.001), in pre-hospital settings (p < 0.001), when considering both anterior and posterior circulation occlusions in LVO definition (p < 0.01), and in prospective study designs (p < 0.01). Conversely, higher specificity was found when RACE investigated confirmed stroke patients (p < 0.01), in in-hospital settings (p < 0.01), only anterior circulation occlusions was considered in LVO definition (p < 0.01), and in retrospective study designs (p < 0.001). Furthermore, according to the analysis, both sensitivity (p < 0.001) and specificity (p < 0.001) were higher when the assessor was a neurologist or physician compared to other ED and EMS staff. Despite these variations, our study found comparable diagnostic accuracy across different conditions (Table 3).

3.3. Publication bias

The Deeks’ funnel asymmetry test demonstrated no evidence of publication bias or asymmetry in cut-off points ≥2 (p=0.6), ≥3 (p=0.31), ≥4 (p=0.74), ≥5 (p=0.2), ≥6 (p=0.08), ≥7 (p=0.81), ≥8 (p=0.96), ≥9 (p=0.89) (Supplementary Figure 10).

3.4. Risk of bias assessment

The QUADAS-2 guidelines were used to assess the quality of the included articles. In patient selection, 4 studies had high risk of bias due to inappropriate exclusions. In the Index test domain, 10 articles had unclear risk due to no mention of assessor blinding. In the reference standard section, 16 studies had unclear 6 due to no mention of the utilized reference standard, 10 due to no mention of reference standard assessor blinding. In flow and timing, 11 articles had unclear risk due to uncertain patient inclusion in the final analysis (Table 4).

3.5. Certainty of evidence

The certainty of evidence in the included articles was evaluated in accordance with the GRADE guidelines. These articles were structured as observational studies, and in line with the GRADE guidelines, the initial level of evidence was set as low. The included studies exhibited significant risk of bias and substantial heterogeneity. Additionally, the presence of a very large magnitude effect size, plausible confounders and dose-response gradient effect could potentially elevate the level of evidence by four two for the very large magnitude effect size, one for plausibility, and one for the dose-response gradient effect. The presence of a plausible co-founder is due to the enhanced diagnostic value achieved through the consideration of assessor-based grouping. Additionally, the existence of a dose-response gradient effect is a result of evaluating the performance of this scale at various cut-off points. Consequently, the certainty of evidence for the predictive value of the RACE scale in LVO detection was classified as high (Table 5).

4. Discussion:

The current review assessed the RACE scale’s diagnostic efficacy in detecting LVOs. Our result demonstrated that the optimal cut-off for the RACE scale is set at ≥3 and ≥4, with a sensitivity of 0.86 (95% CI: 0.78, 0.91) and specificity of 0.57 (95% CI: 0.49, 0.67) for cut-off 3, and a sensitivity of 0.78 (95% CI: 0.70, 0.84) and specificity of 0.68 (95% CI: 0.59, 0.75) for cut-off 4.

In 2014, de la Ossa and colleagues introduced the RACE scale for the first time, building upon the components of the NIHSS. The primary goal of developing this tool was to provide paramedics with an easy to use and accurate method of identifying patients with LVOs and facilitating the rapid transfer to CSCs. The RACE scale assigns scores based on assessments of facial palsy (scored from 0 to 2), arm motor function (scored from 0 to 2), leg motor function (scored from 0 to 2), gaze (scored from 0 to 1), and aphasia or agnosia (scored from 0 to 2). In their study, the authors sought to validate the RACE scale in a prehospital setting, where it was administered by trained medical emergency technicians as part of a prospective study. The results revealed an AUC of 0.82, a sensitivity of 0.85, and a specificity of 0.68 (7). Subsequently, numerous studies have been conducted with varying study designs, in different clinical settings, and involving various clinical specialties to validate its utility in diverse situations.

A review conducted by Smith et al. assessed the performance of various scales in identifying LVO in both suspected and confirmed stroke patients. They found that, based on the analysis of three studies, the RACE score of ≥5 demonstrated a sensitivity and specificity of 0.69 and 0.81, respectively, for suspected stroke patients. Additionally, in confirmed stroke patients, the review revealed a sensitivity of 0.67 and a specificity of 0.85 based on an analysis of two articles. It's important to note that this review had limitations due to the relatively small number of included articles. In contrast, our review provides more robust results as it incorporates a larger number of articles for analysis (15).

In de la Ossa's study, the RACE scale was originally developed and validated in a prehospital setting, carried out by EMS technicians for individuals with suspected stroke (7). However, our subgroup analysis in this study revealed variations in sensitivity and specificity across different conditions. Nevertheless, the RACE scale demonstrated relatively comparable diagnostic accuracy across these variations. The results of our study indicated that when administered by neurologists and ED physicians, the sensitivity and specificity of the RACE scale is slightly superior compared to its use by other ED or EMS staff. This difference could be attributed to the higher level of knowledge and expertise possessed by neurologists in assessing stroke patients, as compared to other healthcare providers in a clinical setting. This could underscore the importance of training the ED and EMS staff for using RACE scale. Our study also revealed that RACE exhibits higher specificity when conducted in confirmed AIS patients. This supports previous findings that the moderate specificity of the RACE scale may result from including patients with hemorrhagic stroke, who typically exhibit severe symptoms and high RACE scores (7).

The de la Ossa study identified a cut-off of ≥5 as the optimal for identifying LVO patients using the RACE scale (7). As a result, many subsequent studies have focused on evaluating the diagnostic performance of RACE specifically at this cut-off. However, our study prioritized not missing LVO patients, considering the severe consequences in terms of mortality and morbidity if the therapeutic window is missed. To address this, we recommend utilizing cut-offs of ≥3 or ≥4, which showed higher sensitivity rates at the expense of lower specificity rates. This approach aims to ensure that LVO patients are directed to CSCs where they can receive appropriate care. It is crucial to note that some patients classified as false positives are actually hemorrhagic stroke patients, who also benefit from referral to CSCs. Therefore, using cut-offs of ≥3 and ≥4 has the potential to improve outcomes for suspected patients.

The ideal prediction tool for LVOs should incorporate several key features to ensure its effectiveness in a prehospital setting. First and foremost, the tool should prioritize simplicity and clarity to reduce the risk of misinterpretation when responding to different commands. It is crucial to employ a binary scoring system rather than relying on ordinal gradations. Speed is paramount to expedite decision-making and minimize time delays. Furthermore, the tool's design should ensure reproducibility among different healthcare personnel, which can be achieved by excluding items that might pose scoring challenges. In this regard, nuanced findings such as neglect should be avoided to ensure easy usability. RACE scale is a short and fast screening tool containing items that examine gaze deviation and palsy and motor symptoms that are the most predictive subitems of LVO and it was validated in a prospective prehospital validation cohort (1, 58-60). The RACE test has certain limitations, notably it’s a non-binary scoring system with three possible responses for most items, complicating the distinction between mild, moderate, and severe deficits and requiring nuanced interpretation. Additionally, the inclusion of assessments for aphasia and agnosia/neglect in the RACE test has been shown to demand the expertise of trained and experienced staff for accurate evaluation, potentially limiting its practicality due to the need for specialized skills and the relatively modest gains in diagnostic accuracy.

The de la Ossa study underscores the strong correlation between RACE and NIHSS, but it's crucial to note that RACE's distinct scoring and focus may not perform as effectively as NIHSS in LVO detection. In this regard, it’s important to note that RACE places greater emphasis on motor symptoms, allowing for an additional point assessment for facial weakness and up to two extra points for leg weakness. However, motor symptoms, while associated with higher NIHSS scores, may not effectively differentiate between non-LVOs and LVOs, as they can also manifest in subcortical or lacunar strokes. Conversely, the RACE scale assigns only one point for gaze deviation, a characteristic sign of cortical or brain stem dysfunction and a potent LVO stroke discriminator. Furthermore, while NIHSS evaluates both fluency and comprehension, RACE focuses solely on speech with commands, potentially missing expressive aphasia—a strong LVO stroke indicator. Lastly, the RACE scale limits aphasia evaluation to individuals with right weakness and neglect to those with left-sided weakness, overlooking concurrent neglect and aphasia possibilities and left-handed patients with right hemisphere dominance (13).

While our findings provide valuable insights, this study has some limitations. In our study, performing a meta-analysis for a cut-off of ≥1 was not feasible. Furthermore, the role of staff training was inadequately addressed in the majority of the included studies and limited us in evaluating the role of training in diagnostic accuracy of RACE. Additionally, a notable number of studies exhibited a serious risk of bias, raising concerns about the overall quality of the evidence.

5. Conclusion:

A high level of evidence indicates that the RACE scale lacks promising diagnostic value for detection of LVOs. A sensitivity range of 0.69-0.86 is insufficient for a screening tool intended to aid in the diagnosis of strokes, considering the substantial morbidity and mortality associated with this condition.

6. Declarations:

6.1 Acknowledgment

The completion of this article would not have been possible without the valuable contributions and expertise of Dr. Mahmoud Yousefifard and Dr. Koohyar Ahmadzadeh. We appreciate their collaboration and guidance throughout the research process.

6.2 Author contributions

Both authors had similar contributions in all steps of the study. The two authors have read and approved the final version of manuscript.

6.3 Conflicts of interest

None

6.4 Funding

None

6.5 Using artificial intelligent chatbots

None

Supplementary materials