Authors: Mohammad Hossein Abbasi, Kristy Yuan, Scott E. Kasner, Ellen McPartland, Karrima C. Owens, Kelly L. Sloane
Categories: Original Research, Cerebrovascular Disease/Stroke, Digital Health, Quality and Outcomes, disability, modified Rankin Scale, stroke, telehealth
Source: Journal of the American Heart Association: Cardiovascular and Cerebrovascular Disease
The modified Rankin Scale (mRS) is commonly used to measure disability after stroke, traditionally assessed through telephone or in‐person evaluation. Here, we investigated the validity of mRS assessment through an automated text messaging system based on the simplified mRS questionnaire as an alternative method to traditional methods of assessment.
A total of 250 patients admitted to 3 hospitals within the University of Pennsylvania Health System with ischemic or hemorrhagic stroke were enrolled. Participants received automated text messages sent 48 hours before their outpatient appointment at about 90 days after stroke. The mRS scores were assigned on the basis of participant responses to 2 to 4 text questions eliciting yes/no responses. The mRS was then evaluated in person or by telephone interview for comparison. Responses were compared with κ. A total of 142 patients (57%) completed the study. The spontaneous response rate to text messages was 46.5% and up to 72% with an additional direct in‐person or phone call reminder. Agreement was substantial (quadratic‐weighted κ=0.87 [95% CI, 0.83–0.89]) between responses derived from the automated text messaging and traditional interviews. Agreement for distinguishing functional independence (mRS 0–1) from dependence (mRS 2–5) was substantial (unweighted κ=0.79 [95% CI, 0.69–0.90]).
An automated text messaging system is a feasible method for remotely obtaining the mRS after stroke and a potential alternative to traditional in‐person or telephone assessment. Further studies are needed to evaluate the generalizability of text message–based approaches to stroke outcome measurement.
Keywords: disability, modified Rankin Scale, stroke, telehealth
Subject Categories: Digital Health, Quality and Outcomes, Cerebrovascular Disease/Stroke
Stroke affects close to 800 000 individuals in the United States annually and is the leading cause of adult disability in the United States. ^1^ The modified Rankin Scale (mRS) is a widely used outcome measure in clinical and research practices for assessing poststroke disability. ^2^ It is traditionally assessed through an in‐person evaluation or structured telephone questionnaire, ^3^ but both can be operationally difficult and time intensive, and appointments or calls may be missed. Text messaging may represent a user‐friendly, accessible method for obtaining patient‐reported outcomes. Prior text message interventions for patient follow‐up have been implemented for a variety of indications related to clinical follow‐up and assessment of patient satisfaction, including surveillance of care for neurosurgical conditions, ^4^ , ^5^ psychiatric disturbance, ^6^ , ^7^ , ^8^ smoking and drinking cessation, ^9^ , ^10^ diabetes management, ^11^ , ^12^ and postdischarge primary care follow‐up. ^13^ Text message–based approaches have grown in popularity over recent years, accelerated by the coronavirus pandemic and the essential need for strategies of remote patient monitoring. ^14^ , ^15^ , ^16^
Previously, the simplified mRS questionnaire (smRSq) was validated for assessment of poststroke disability status, with good reproducibility noted among raters certified in evaluation of mRS
^17^
,
^18^
,
^19^
and those uncertified.
^20^
Methods of investigation ranged from surveys sent to participants through the mail to clinical vignettes given to raters, and the agreement (quadratic‐weighted κ [κw]) between smRSq and traditional interrogative methods ranged from 0.64 to 0.90. In recent years, technology‐based solutions have been explored for remote assessment of outcome measures, particularly in the patient population with stroke, where in‐person appointments may present a challenge in terms of mobility and accessibility. Two recent studies have investigated the use of remote methods of assessing mRS after stroke. The first study involved an electronic, web‐based application designed to guide raters to an mRS score based on the smRSq algorithm.
^20^
Raters with varied clinical backgrounds were given clinical vignettes and asked to rate the mRS for each case described using the application. The study found good validity compared with reference mRS scores when raters used this intervention. Another recent study investigated text message–based prompts for obtaining the mRS for patients after discharge from stroke hospitalization.
^21^
This study demonstrated validity for assessing the mRS score by an automated text messaging platform compared with traditional methods, though the response rate was low (22.8%) at 90 days.
In this study, we tested the validity of an automated text message–based program for obtaining the mRS score at 90 days after stroke. We hypothesized that this automated text message program, based on smRSq, would have good validity to remotely assess the mRS score at 90 days after stroke compared with in‐person or telephone interviews.
The data that support the findings of this study are available from the corresponding author upon reasonable request.
We designed a 90‐day postdischarge intervention using automated texting for assessment of the mRS score in collaboration with the platform WayToHealth (WTH). WTH is a web‐based platform that supports technology‐based interventions to support behavior change interventions. WTH is integrated with the electronic medical record supported by the Penn Center for Health Care Innovation and Center for Health Incentives and Behavioral Economics. With WTH, we customized an automated text message algorithm to message patients at specified intervals after discharge. We enrolled patients at 3 hospitals of Penn Medicine in Philadelphia, Pennsylvania, between August 2022 and February Hospital of the University of Pennsylvania, Pennsylvania Hospital, and Penn Presbyterian Medical Center. Patients were eligible if they were hospitalized with an ischemic or hemorrhagic stroke, they (or a surrogate) could receive and respond to text messages in English. Participants were excluded if life expectancy was <90 days. This project was reviewed and determined to qualify as quality improvement by the University of Pennsylvania Institutional Review Board. Patients or surrogates were approached before discharge to confirm eligibility and provide verbal assent, and then opted in to receive text messaging from WTH. Participants were able to decline participation in the program by responding “stop” at any time.
Participants received a text message 48 hours before their scheduled appointment, ideally at 90 days but allowing 30 to 180 days after stroke. Based on the line of questioning described in the smRSq questionnaire, WTH used branching logic to ask 2 to 4 yes/no questions and determine the mRS score. ^17^ The initial question asked, “Can you live alone without the help of another person? This means being able to bathe, use the toilet, shop, prepare meals, and manage finances.” Subsequent questions were asked depending on the yes or no response from the participant. If the participant did not respond within 24 hours of the initial message, a second automated text message with the same content was sent. At the in‐person visit, mRS was obtained through an interview by an individual rater (M.H.A.), who was trained in obtaining the mRS score and was blinded to text message responses. The interview consisted of questions regarding the patient's ability to complete activities of daily living, mobility, and dependence. ^22^ The interview started with the question, “Could you live alone without help from others?” and then included subsequent questions like “Can you do everything you were able to do before your stroke?” or “What activities like housekeeping, work, paying bills, and shopping are you no longer able to do since your stroke?” to characterize the degree of disability of the participant on the basis of mRS training and interview questions developed by Wilson et al. ^22^ If the participant was unable to attend an in‐person visit, a telephone assessment was performed using the same scripted interview and rater. In situations in which the surrogate responded to messages on behalf of the patient, the same surrogate was interviewed for the in‐person/telephone‐based assessment. The questions used in the structured in‐person/telephone interview are similar but not identical to those used in the WTH text message–based approach.
Demographic and clinical characteristics were collected at baseline (Table). Participants also completed a survey of frequency of use and comfort with technology and living situation at 90‐day follow‐up.
We assessed agreement between mRS scores derived from text messages and those from traditional methods (in‐person or telephone) using Cohen's κ, κw for ordinal scores, and unweighted for scores dichotomized as functionally independent (mRS 0–2) or dependent (mRS 3–5). The differences between text‐based and traditional mRS scores were normally distributed and reported as mean (±SE). Linear regression was used to estimate the relationship between methods without an assumption of forcing the intercept through the origin. Sensitivity and specificity of the dichotomized data were also calculated. Participant characteristics were compared between enrollees who completed the study and those who did not. A P value <0.05 was considered statistically significant. STATA version 5.0 (StataCorp LLC, College Station, TX) was used for analyses.
Of 250 patients enrolled, 24 withdrew before receiving the study text message. Complete mRS assessments were received from 105 (46.5%) solely on the basis of the automated process. An additional 58 (25.5%) responded to the text message mRS assessment after a brief reminder from a member of the study team. In total, 142 (72%) of participants completed the text message response. Baseline characteristics of those who completed the study and those who did not are described in the Table. Those who responded via the fully automated process were younger, more likely to be White individuals, had milder strokes, and had fewer chronic comorbidities than nonresponders (Table S1). Of these 163 whose mRS was assessed via text messaging, 142 completed the traditional mRS interview and constituted the final study population (Figure 1). At study completion, 57% of participants reported electronic technology use without assistance or perceived difficulties, but 17% perceived difficulties with the technology, 9% required assistance of another person, and 13% reported no technology use. The majority (85%) were living at home.
Figure 1 Study flow diagram.
The final mRS measured through traditional methods (in‐person or telephone) and text message was a median of 2 (1–3). There was excellent agreement between text‐based and standard interviews (κw=0.87 [95% CI, 0.83–0.89]; N=142) when evaluating ordinal mRS scores (Figure 2). The mean difference between methods was 0.18 (±0.07) points. Agreement with text message–based mRS did not differ depending on method of assessment (in‐person [κw=0.86, n=69] or phone [κw=0.87, n=73]), nor by person responding (patient [κw=0.86, n=90] or surrogate [κw=0.78, n=52]).
Figure 2 Comparison of modified Rankin Scale (mRS) by method of assessment (text message or traditional interview) and by person providing response (patient or surrogate).
With mRS dichotomized (mRS 0–1 versus 2–5), there was substantial agreement between text‐based and standard interviews (unweighted κ=0.79, 95%CI:0.69–0.90) with similar unweighted κ=0.75 for dichotomy of mRS 0 to 2 versus 3 to 5 and k=0.63 for dichotomy of mRS 0 to 3 versus 4 to 5. Discordant scores were assigned to 14 participants (Figure 2). Taking the standard interview as the reference criterion, the sensitivity of text‐based assessment was 90% and specificity was 90%. Similar results were obtained for in‐person and phone assessments. There was greater agreement for dichotomized mRS when the patient responded compared with surrogates, though confidence limits were overlapping.
Text message communication is an emerging approach to patient engagement to support quality improvement and research. Text messaging allows for automation and avoids common pitfalls of telephone or in‐person assessments such as unanswered calls, investment of time and effort among staff, and transportation and cost issues preventing patients from coming to the clinic. As the mRS is commonly used to measure outcome after stroke in practice and in clinical trials, efforts to streamline and automate its collection should improve efficiency and access. Our study was launched as part of a quality improvement initiative to enhance assessment of outcome after recent stroke. We found that a text message system of obtaining mRS at 90 days after stroke was feasible, yielding a 42% spontaneous response rate. An additional 23% were willing to complete the process with an additional nudge. Text‐based mRS had excellent agreement with standard interview‐based mRS scores. Patient‐reported mRS may have had slightly better reliability between modalities (κw=0.86, n=90) than surrogate‐reported scores (κw=0.78, n=52). Further development may improve implementation and integration into clinical and research workflows. Automated text messages from WTH can also be dispensed in multiple languages, and outcome data obtained from WTH can be incorporated into the electronic medical record.
The smRSq, based on which we designed the current automated text messaging system, has also shown high agreement with reference scores (κw=0.90) when electronically used to assess the mRS scores.
^20^
The agreement (κw) between smRSq and traditional interrogative methods ranged from 0.64 to 0.90. One study evaluated the validity of smRSq that was self‐administered through a survey sent to subjects through the mail and had a κw of 0.64.
^18^
Other studies involved physician determination of mRS on the basis of direct questioning of subjects or on clinical vignettes, and κw was higher, from 0.79
^19^
to 0.86
^17^
to 0.90.
^20^
Two recent studies have evaluated mRS through electronic or text communications and have found good reproducibility and validity. The first study involved a web‐based application of the smRSq, which raters could use as a guide for a final mRS score. Sixteen raters from diverse clinical backgrounds participated, evaluating 24 different clinical vignettes. The study found strong agreement between the scores assigned by the individual raters and reference scores (κw=0.90).
^20^
This intervention further validated the use of the smRSq, but it relied on the individual rater to assign an mRS score rather than an automated process.
Another recent study assessed an automated text message system similar to our program.
^21^
They enrolled 350 participants to receive 3 sets of text messages at 30‐day intervals after discharge. The intervention showed strong agreement between text‐based and in‐person assessments (κw=0.80) for both ordinal and dichotomized mRS scores. Although they had a large initial sample of 350 participants, they had a low response rate at 90 days of 22.8%. In addition to the low response rate, the study's other limitations included the interval of 4 weeks between text message response and in‐person assessments and the lack of inclusion of surrogate participation. The poststroke period reflects a dynamic time in recovery, and there may be differences in function and level of disability that change rapidly between the first and second assessment. In addition, many individuals with recent stroke may not be able to receive and respond to text messages, depending on their impairments and level of disability, so inclusion of surrogates will be an important aspect of validating an outcome measure.
In contrast to other published studies, our study involved a narrow interval between types of assessments (in‐person/telephone versus text message based) with a higher response rate. We also sought to include surrogates as responders, as this reflects the real‐world approach to mRS assessment, where participants may not be able to self‐report due to severity of disability. Our reasonable spontaneous response rate, which was higher than prior studies of text message–based mRS assessments, was also augmented by additional responses that occurred after a nudge from a member of the study team, rather than an automated text message, suggesting that some patients and caregivers may benefit from outreach that is not solely technology based. The platform used here, WayToHealth, can integrate with electronic medical records, allowing for the mRS to be auto‐populated into the chart for review by the participant's clinician.
Our study had several limitations. The participant sample was modestly sized and was drawn from 3 hospitals, all in the city of Philadelphia, so generalizability to other populations may be limited. Moreover, our findings indicated that patients who were younger, White individuals, and with fewer comorbidities were more likely to respond, so further investigation is necessary to determine the generalizability to other patient populations, especially those from underrepresented groups and from caregivers, and to determine the extent to which stroke severity is associated with response status. In addition, though our response rate was higher than other studies using a technology‐based mRS questionnaire, there was a substantial number of nonresponders. Several patient‐level factors may have contributed to this finding, including location of the patient at 90 days (home or facility), comfort with technology and text messaging, and access to a mobile phone. We included only participants who could send or receive text messages themselves or by a caregiver; based on observational studies, the rate of Internet and smartphone access is high among stroke survivors and their caregivers (up to 86%–100%), but there may be some patients missed who are unable to access a text message–capable device. ^23^ Additionally, rating of mRS for non–technology‐based assessments was performed by a single certified rater. The use of a single rater eliminated concern for interrater reliability, but replication with multiple raters may be needed to support widespread adoption. Finally, the method of analysis, weighted κ, weights disagreement on the basis of the difference between the 2 scores, so it is a robust measure for ordinal scales but also may amplify the apparent agreement if only single‐category differences are present.
Our study expands upon the evolving role of digital health applications in the care of patients with cerebrovascular disease. Text message–based outcome measures in this study yielded reasonable response rates from patients and demonstrated substantial agreement with the traditional methods for obtaining these measures. Text message–based programs like WTH are a promising direction for postdischarge outcome assessments for stroke and other cardiovascular or cerebrovascular conditions. As a result of these promising findings, our institution is continuing to collect the mRS through this automated technology‐based system. Future studies will be needed to determine which patient subgroups are less likely to respond to text messaging and to determine if other outcomes measures may be validly collected through this platform.
None.
Dr Kasner has received grant funding (to institution) from Genentech, Diamedica, Bristol‐Myers Squibb, Bayer, Stryker, and Medtronic; consulting fees from Shionogi, Abbvie, Artivion, and NeuExcell; and royalties from UpToDate and Elsevier. The remaining authors have no disclosures to report.