Authors: Katie Esser (aDepartment of Speech-Language Pathology & Audiology, Towson University, MD), Erin M. Picou (bDepartment of Hearing & Speech Sciences, Vanderbilt University Medical Center, Nashville, TN), Benjamin W. Y. Hornsby (bDepartment of Hearing & Speech Sciences, Vanderbilt University Medical Center, Nashville, TN)
Categories: Research Articles
Source: American Journal of Audiology
Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue
Authors: Katie Esser, Erin M. Picou, Benjamin W. Y. Hornsby
Abstract
Purpose:
Understanding speech in challenging environments can be fatiguing for individuals with and without hearing loss. Empirical research examining interventions to reduce such fatigue, however, is limited. Our study investigated the effects of imperfect captions, such as those created using automated speech recognition systems, on listening-related fatigue during a challenging speech task.
Method:
Twenty-two adults (aged 18–63 years) with essentially normal hearing completed a sustained dual task designed to induce listening-related fatigue. The primary task was audiovisual sentence recognition in quiet, with speech levels individually adjusted for ~70% correct performance (without captions). The secondary task was response time (RT) to topic words presented during the speech task. Participants completed the dual task with or without captions. Subjective fatigue ratings were obtained before, during, and after the dual task. Vigilant attention was measured via the secondary task and before and after the dual task using visual stimuli (RTs to a visual marker presented at random intervals).
Results:
Subjective fatigue ratings increased significantly over the course of the dual task. However, this increase was larger for the group who had captions, even though they had better sentence recognition overall. Evidence of behavioral fatigue (slowed RTs over time) was also present but only for those in the Caption group. Anecdotal reports from study participants suggest that the increased fatigue for the Caption group was related to the cognitive challenge of combining information from the time-locked audiovisual cues and the time-delayed, and imperfect, text captions.
Conclusions:
Even though they were time-delayed and contained inaccuracies, captions improved speech recognition. However, this benefit was accompanied by greater increases in subjective and behavioral fatigue. Therefore, in some conditions imperfect captions can have negative consequences for listening-related fatigue. Further research is needed to determine whether this pattern holds in different circumstances, such as with audio-only stimuli.
Supplemental Material:
https://doi.org/10.23641/asha.29646482
Understanding speech in challenging listening situations, like when speech is low-level or when the signal-to-noise ratio is unfavorable, can require considerable mental (or “listening”) effort (Alhanbali et al., 2017; Kramer et al., 2006). Maintaining such high levels of effort over a sustained period of time increases the risk for developing “listening-related fatigue” (Davis et al., 2021). When recurrent and severe, such fatigue can be especially problematic for those with hearing loss, negatively impacting their emotional state (Davis et al., 2021), need for recovery (Nachtegaal et al., 2009), and work performance (Kramer et al., 2006). Interventions such as hearing aids (Holman, Drummond, & Naylor, 2021; Hornsby, 2013) and cochlear implants (Hornsby et al., 2024) have been shown to decrease listening-related fatigue in certain conditions. However, some listeners with hearing loss continue to report experiencing listening-related fatigue even with use of hearing devices (Davis et al., 2021; Hornsby et al., 2024; Hughes et al., 2018). As such, there is a need for alternative and supplemental interventions to reduce listening-related fatigue.
Captions, or text transcriptions of audio, are widely available and are often required by law for many forms of recorded media and live entertainment (Closed captioning of televised video programming, 47 C.F.R. 79.1, 2024). Captions provide a supplement to audio and audiovisual information that may otherwise be difficult to access when there is poor audibility due to the nature of the listening situation or due to hearing loss. Captions can be transcribed and edited by humans and time-locked to the visual material for recorded media (e.g., open and closed captions), or they may be generated by automated speech recognition systems for live or recorded media (National Deaf Center on Postsecondary Outcomes, 2022). Captions for recorded media (“subtitles”) can, in theory, convey the auditory information with 100% accuracy but automated captions can vary wildly in their transcription accuracy (Kuhn et al., 2023) and timing relative to the visual material (Jiline et al., 2020). Live captioning by humans (e.g., Communication Access Realtime Translation; CART) is often thought of as the “gold standard” for improving accessibility of live events but this approach can contain errors, similar to automated captions (e.g., Romero-Fresco & Fresno, 2023). Although automated captions in particular have timing and accuracy limitations, they are relatively easy to access at little to no cost to the consumer (e.g., on Zoom, YouTube, social media). Therefore, automated captions hold promise as a cost-effective solution to increasing access to spoken language. The captions used in the current study are meant to approximate these types of automated captions and are described in more detail below.
Benefits and Limitations of (Imperfect) Captions
A recent systematic review by Zhong et al. (2022) summarized the evidence supporting benefits of captions for adults with and without normal hearing. The review revealed that captions can provide substantial speech-understanding benefits in a wide range of listening conditions, but especially in challenging conditions (e.g., listening in competing background noise). The characteristics of the captions themselves also impact their benefits, with higher transcription accuracy (i.e., > 70%) and minimal text delays (i.e., < 1 s) leading to larger benefits.
On the other hand, “imperfect captions” (i.e., those with lower accuracy and timing delays relative to the audio signal) can limit the benefits of captions. For example, Zekveld et al. (2009) found that the introduction of captions to an audio signal decreased subjective ratings of performance and increased subjective ratings of effort when captions were imperfect (< 70%) and delayed (1.1 s) relative to the audio signal. This suggests that imperfect captions have the potential to increase subjective effort, especially if they are delayed and inaccurate, at least for audio-only speech signals. However, none of the studies included in the Zhong et al. (2022) review supplemented audio and captions paired with reliable facial cues. This is an important limitation because captions are commonly used when facial cues are available (including recorded media and live settings).
More recently, Zhong et al. (2023) investigated the effect of “imperfect” captions (i.e., captions with deletion errors and timing delays relative to the auditory signal) on audiovisual sentence recognition and listening effort (subjective only) in young listeners with normal hearing. Several conditions were tested in which audibility and access to facial cues varied. When noise was present, combined access to captions and facial cues provided significant benefits for sentence recognition performance and subjective listening effort. Conversely, when tested in quiet at a conversational speech level (60 dB SPL), access to captions and facial cues did not significantly improve sentence recognition or subjective ratings of listening effort. However, participants did report significant perceived (subjective) speech-understanding benefit from captions in all test conditions, even in quiet. In other words, even though the study captions contained deletion errors (i.e., captions were ~87% accurate) and were delayed 500 ms in time relative to the auditory stimuli, they provided substantial objective and subjective speech-understanding benefit. These results suggest that such imperfect captions may, at a minimum, improve perceived speech understanding and, potentially, reduce subjective listening effort in certain conditions (e.g., when audibility is poor), even with audiovisual stimuli.
However, while many studies have shown the benefits (and limitations) of captions in the context of speech understanding, less is known about how captions, particularly imperfect captions, impact exerted listening effort or listening-related fatigue. Listening effort, or the deliberate use of mental resources (e.g., attention, working memory) during a listening task, can increase when the listening situation is demanding (e.g., poor signal-to-noise ratio) and due to certain characteristics of the listener (e.g., hearing loss; Peele, 2018; Pichora-Fuller et al., 2016). When discussing the construct of listening effort (or any type of mental or physical effort) it is important to distinguish between the effort required to complete a task and the effort an individual actually applies toward a task. Francis and Love (2020) refer to the effort (mental resources) required to complete a task as demanded effort. The magnitude of this effort is inferred from the task properties. For example, higher listening effort may be demanded in challenging acoustic settings (e.g., in noise or reverberation) or in more cognitively challenging listening tasks (e.g., dual tasks, nonnative speech materials). However, regardless of the demanded effort, the amount of effort applied toward a task by a given individual can vary (e.g., the listener may apply a lot or a little effort toward the task). The listening effort applied during a task can be measured using subjective, physiological, or behavioral methods (see Francis & Love, 2020, for a review). Effort measured using these techniques has been referred to as assessed (subjective) and exerted (physiological or behavioral) effort (Francis & Love, 2020). Previous work with captions has examined subjectively assessed listening effort (see review by Zhong et al., 2022); in this article, we focus on behavioral techniques that measure exerted effort over time.
Subjective measures use surveys or questionnaires to evaluate an individual's self-perceived, assessed effort. Behavioral measures of listening effort can be used to infer the degree to which mental resources (e.g., attention, working memory) are being exerted during the listening task. A variety of tasks (e.g., memory recall, simple response time [RT], and dual-task measures) have been used for this purpose. For example, verbal and motor RTs have been used as a proxy of effort on listening tasks. The assumption being that as more effort by means of the use of cognitive resources is exerted during the task (to decode, recognize, and repeat the auditory signal), verbal and motor RTs will slow (Gatehouse & Gordon, 1990; Hornsby, 2013; McGarrigle et al., 2019; Picou & Ricketts, 2014). The dual-task paradigm is an example of an approach that often utilizes RT measures as an index of behavioral listening effort. Again, with longer RTs suggesting increased allocation of resources (effort) to the primary task (listening), away from the secondary RT task (see Gagne et al., 2017, for a review). A challenge with using simple, or dual-task RTs as a proxy for cognitive resource allocation is that response speed during a task can be affected by more than exerted effort (e.g., motor RTs, practice effects, listener motivation, age, etc.; see Gagne et al., 2017; Kuchinsky et al., 2024). The impact of such confounding variables complicates the interpretation of differences in effort applied by individuals or groups when they are tested in different listening conditions (i.e., a between-groups design). This distinction is relevant to the current study design, as described below.
Listening-Related Fatigue
By improving speech understanding and reducing listening effort, captions might also be an effective tool for reducing listening-related fatigue. The construct of fatigue, like effort, is complex and can be defined and quantified using subjective, physiological, and behavioral measures (see Hornsby et al., 2016, for a review). Our focus in this article is on subjective and behavioral assessment of fatigue. Specifically, subjective feelings of fatigue that may develop in response to sustained effort can, in some cases, be accompanied by behavioral changes, specifically decrements in mental or physical performance over time. This approach stems from our everyday experiences and longstanding research highlighting the difficulties in maintaining optimal performance in a fatigued state (Hockey, 2013). For example, fatigue-related decrements in physical performance are familiar to athletes. However, similar fatigue-related performance changes can, in some cases, also be observed on cognitive tasks, such as those used to assess listening effort (Hockey, 2013). Thus, systematic changes in exerted effort over time (e.g., a slowing of RTs over the course of a listening task) have been used as a behavioral marker of listening-related fatigue (Hornsby, 2013; Moore et al., 2017).
While many factors can affect the rate and magnitude of fatigue development, as noted above, fatigue is commonly associated with the sustained application of high levels of mental effort (Hockey, 2013). Thus, interventions that reduce the applied effort during a listening task may also be effective for reducing any resultant listening-related fatigue. For example, Alhanbali et al. (2017) and Hornsby et al. (2024) found strong, positive correlations between subjective ratings of effort and fatigue in everyday settings. Individuals who reported a need for high levels of effort when listening in everyday settings were more likely to report experiencing higher fatigue in their daily lives.
In addition, the relationship between applied effort and fatigue has also been demonstrated in laboratory settings (Earle et al., 2015; Hornsby, 2013). Hornsby (2013) examined the effects of hearing aid use on listening effort and fatigue in adults with hearing loss. Subjective and behavioral (auditory recall and RTs to a visual marker) measures of effort and fatigue were obtained over the course of a ~50-min, sustained and cognitively challenging, speech-in-noise task. Subjective ratings of fatigue increased significantly over time regardless of whether participants were tested with or without hearing aids. However, hearing aid use did reduce listening effort, measured behaviorally via auditory recall and visual RT measures, during the task. In addition, measures of behavioral listening effort were stable throughout the duration of the task when participants used hearing aids but effort (measured behaviorally) increased over time (i.e., a performance decrement over time) when participants did not use hearing aids, suggesting that hearing aids may facilitate reductions in behavioral listening-related fatigue. Interestingly, subjective and behavioral measures had weak or no correlation, suggesting that these two types of measures may assess different underlying aspects of fatigue. A similar lack of correlation has been found in other studies of listening effort and fatigue (e.g., Alhanbali et al., 2019; Bryant et al., 2004), underscoring the importance of including both subjective and behavioral measures in listening-related fatigue investigations.
Prior research has shown that listening-related fatigue can be a significant problem for some individuals with hearing loss. Despite its negative effects, empirical work examining interventions to reduce listening-related fatigue is limited. The current study used a similar methodological approach to Hornsby (2013) to expand on the work of Zhong et al. (2023) by focusing on the impact of captions on listening-related fatigue (subjectively and behaviorally) with audiovisual stimuli (visible talker with an audio signal). To answer our research questions, we tasked participants with completing a sustained (~50 min), complex, dual task using auditory–visual speech materials. The task was designed to induce listening-related fatigue and was completed either with or without access to imperfect text captions. As in the study by Zhong et al. (2023), imperfect captions were used because they represent an accessible and inexpensive method of captioning. Fatigue effects that developed while completing the task (with or without captions) were quantified with subjective and behavioral measures. Specifically, increased (higher or worse) subjective ratings and decrements in behavioral performance (speech recognition, topic-word RTs, and visual RTs) over time were assumed to be reflective of increased listening-related fatigue. Given previous findings showing captions can reduce subjective listening effort (Zhong et al., 2023), and the well-known relationship between effort and fatigue (e.g., Hockey, 2013), we hypothesized that, despite their imperfections, captions would reduce subjectively and behaviorally measured fatigue.
Method
Participants
Twenty-four adults participated; however, the data of two participants were excluded due to technical issues (n = 1) or inadvertently exceeding an age cutoff criterion (n = 1). The remaining 22 participants (18 women; M = 36 years, 18–63) had normal to near-normal hearing in at least one ear, as measured via sound-field testing in quiet (mean binaural three-frequency pure-tone average [PTA; 500, 1000, and 2000 Hz] = 11.6 dB HL; 2–26 dB HL). To rule out confounds related to hearing acuity, our two groups were matched for sound-field PTA (see Table 1). All participants were native English speakers who denied a history of speech, language, reading, and/or cognitive difficulties; an unmanaged sleep disorder; diagnosis of chronic fatigue syndrome or clinical depression; or regular use of medication that can impact fatigue or energy levels.
To investigate the effects of captioning, participants were randomly assigned to one of two groups. One group had captions available during testing (“Caption”) and the other group did not (“No Caption”). We chose to use a between-groups, rather than repeated-measures, design for two primary reasons. First, we wanted to minimize confounds due to learning effects that could occur if speech test materials were replicated across test sessions—this would be required in a repeated measures design given the duration of our test session and the limited number of sentence materials available for use during the dual task (see below). Second, there was concern that after completing the fatiguing task once, participants might differentially modify their effort strategy on the second day resulting in more individual variability (e.g., some might reduce their applied effort to minimize fatigue effects, others might feel more familiar with the task and apply more consistent effort, and others could change their listening strategies in other ways). Using a between-groups design allowed participants to complete all study procedures in a single test session. One consequence of a between-groups design, however, is the reduction in statistical power compared to a repeated-measures design. In addition, as noted earlier, potential between-group differences in motivation, cognitive ability, motor RTs, and so forth could impact behavioral estimates of exerted effort, potentially confounding comparisons between groups (i.e., caption vs. no captions conditions). Such differences could also, potentially, affect between-group differences in the magnitude of subjective fatigue ratings. While potentially important, we believe the impact of any such between-group differences will be minimal given our focus on changes (decrements) in subjective and behavioral (RT) measures of listening-related fatigue, rather than on the absolute values of subjective ratings or RTs at any one point in time.
Demographic information for the two groups is displayed in Table 1. An analysis of variance (ANOVA) revealed the groups did not differ significantly with regards to PTA, age, or biological sex (p > .05). In addition to the study tasks, participants completed a questionnaire regarding their use of recorded media (e.g., movies, TV, YouTube, TikTok) and captions. Participants answered the following yes-or-no “Do you like to use closed captions or subtitles when you watch shows or movies?” Responses to this question are summarized in Table 1. Additional questions included in the questionnaire asked how often participants use captions and why they do or do not like to use captions. This questionnaire was administered to facilitate exploratory analysis of the effect of preference for caption use on our outcome measures. The full questionnaire is available in the Appendix.
Participants also completed the 40-item version of the Vanderbilt Fatigue Scale for Adults (VFS-A-40; Hornsby et al., 2021). The VFS-A-40 measures subjective listening-related fatigue providing a total score and four subscale fatigue (a) emotional, (b) social, (c) cognitive, and (d) physical. Total and subscale scores were calculated for each participant. The possible total scores range from 0 to 160, with higher scores indicating more listening-related fatigue. Mean total scores were not significantly different between the No Caption and Caption groups (see Table 1). A summary of the VFS-A-40 total and subscale scores by group is available in Supplemental Material S1. Prior to the start of testing, all participants provided informed consent under Vanderbilt University Medical Center Institutional Review Board Protocol #231065. Participants were paid for their time.
Dual-Task Paradigm
Primary Task
Participants completed a sustained, approximately 50-min, dual task. The primary task was audiovisual sentence recognition (Audiovisual Connected Speech Test [AV CST]; Cox et al., 1989). Participants were instructed to listen to the sentences and to repeat back what they heard. The AV CST consists of 24 pairs of passages from a children's encyclopedia that are spoken by a female talker; each individual passage contains nine or 10 sentences about a single topic (e.g., windows). Thus, each passage pair consists of approximately 20 sentences (nine to 10 sentences for each passage topic) and contains 50 “key words,” which are used for scoring (25 key words per passage). The video recording includes the head and upper shoulders of the talker speaking the CST sentences. During the dual task, the AV CST sentences were presented in quiet, via sound-field loudspeaker, at a low sensation level. Speech presentation levels were individualized (details provided in the Procedure section below, see Training Session subsection) to achieve approximately 70% correct recognition on the AV CST key words when captions were not available. The presentation order of AV CST passage pairs was fixed for all participants. This approach ensured that any AV CST passage pair order effects would be consistent across participants.
Captions. The AV CST materials were presented either with or without captions. The captions used were the same as those described by Zhong et al. (2023). These captions have a 500-ms delay relative to the onset of the audio track. In addition, each sentence was missing one or two of the scored key words. These characteristics were chosen to approximate the onset delays and textual errors observed in some captions resulting from automatic transcription (Gupta et al., 2018; Jiline et al., 2020; Slaney et al., 2020). Given the number of key words that were intentionally missing from the captions, only 86%–88% of spoken words in each passage pair, and only 60% of scoring key words, were accurately presented via captions. Therefore, the maximum score that could be obtained by reading the captions alone (i.e., with no audio or video information) would be 60% correct. Topic words (see definition under Secondary task subsection) were included in the captions the majority of the time. When a topic word was a scored key word, there was a chance that it was deleted from the text captions. All words, including topic words, scored key words, and all other words in the sentences, remained intact in the audio portion of the stimulus (i.e., the only auditory manipulation was the presentation level, no words were ever omitted from the audio).
Secondary task. For the secondary task, participants were instructed to press a button as fast as possible whenever the target “topic word,” or its derivative, occurred in a CST sentence. Before each new passage started, the target topic word for that passage was shown on the video monitor. For example, during the passage about “windows,” participants were instructed to press a button every time the talker said the word “window” or “windows.” Participants were instructed to prioritize the primary, speech recognition, task over the secondary task. Those in the Caption group were told to press the button as quickly as possible when they heard the topic word, not if or when they read the topic word using the captions. Fast and accurate responses require participants to remain vigilant for topic words throughout the test session. RTs in milliseconds and counts of topic word misses and false alarms were calculated. See Supplemental Material S1 for details as to how RTs, misses and false alarms were calculated and for a list of the target topic words for each passage. RTs were interpreted as a behavioral measure of listening effort; longer times would indicate more exerted listening effort, and greater increases in RTs over time would indicate greater listening-related fatigue.
Listening-Related Fatigue Measures
The impact of text captioning on participant fatigue was assessed via subjective ratings and changes in objective (behavioral) measures completed over the course of the dual task. While participants completed the dual task continuously, for analysis purposes, we divided the dual task into three blocks, with each block containing eight passage pairs and 400 scoring key words.
Subjective Fatigue Ratings
Right Now Fatigue Scale
Subjective ratings of participants' current fatigue levels were obtained using a single query, “How physically/mentally fatigued do you feel right now?” (Hornsby, 2013). Participants responded by selecting a number on an 11-point visual analogue scale. The scale contained verbal descriptors at the endpoints and ranged, in 10-point increments, from 0 (Not at all fatigued) to 100 (Extremely fatigued). Participants could select any number between 0 and 100. These subjective ratings were collected four once before, twice during (after every eight passage pairs—approximately 160 sentences and 400 scoring key words), and once immediately after the dual task. Higher Right Now Fatigue Scale (RNFS) ratings were interpreted as higher subjective fatigue. During the dual task, the RNFS instructions and a visual scale were presented on the computer screen in black text on a white background. Participants verbally stated the number (from 0 to 100) that corresponded to their current fatigue level. This information was manually recorded by the experimenter.
Behavioral Fatigue Measures
We assessed behavioral fatigue resulting from completion of the listening task by monitoring changes in performance over time on three behavioral
- Changes in AV CST speech recognition Each block of the dual task had the same number of scored key words (400). Decreases in AV CST scores over time were interpreted as evidence of fatigue development.
- Changes in AV CST topic-word RTs: Each block of the dual task contained a different number of topic words that were used to obtain RT measures, with 98, 88, and 89 topic words occurring in Blocks 1, 2, and 3, respectively. Increasing (slower) RTs over time were considered suggestive of increased fatigue.
- Psychomotor Vigilance Task (PVT; Dinges & Powell, 1985): Participants completed the PVT twice, once immediately before starting the dual task and then a second time immediately after the dual task (see Figure 1 for test procedure in chronological order). The PVT is a simple visual reaction time task that measures vigilant (sustained) attention. The PVT has been widely used in the broader area of fatigue research and is sensitive to factors known to affect mental fatigue (e.g., Lieberman, 2007; Lim et al., 2010). This task requires participants to remain vigilant for a millisecond counter to appear (visually) on a computer screen and press a button as fast as possible whenever it appears. After pressing the button, the participant's RT (in milliseconds) is briefly displayed on the screen. The counter then disappears prompting the participant to remain vigilant for the next trial. The PVT consists of 100 trials where the visual counter appears. The time between trials randomly varies with intertrial intervals ranging approximately from 1 to 10 s. The total test duration is about 10 min. Slower RTs after completing the dual task were considered indicative of increased fatigue.
Additional Measures
Quiz
Following the conclusion of the dual task, participants completed a bespoke content-based, multiple-choice quiz based on information extracted from 12 of the 24 AV CST passage pairs used in the dual task (questions came from every other passage presented during the 50 min of testing). These 12 “quiz” passage pairs were equally distributed throughout the dual task. There were six questions derived from each of the 12 “quiz” passage pairs, for a total of 72 questions. Each question had four multiple-choice options with one correct answer. The quiz was designed such that general knowledge about a topic would be insufficient to perform well on the quiz, but listening to CST passages first would make the quiz easy to complete. For example, one question from the “Window” passage reads, “Windows often display pictures from what?” with the following options for (a) The Bible, (b) Scriptures, (c) Fables, and (d) Mythology. The sentence from which this question was derived states, “These windows displayed pictures from the Bible.” Although any of the options could be correct based on general knowledge, the correct answer for this question is “a,” because that is what participants heard during the AV CST. A copy of the quiz is provided in Supplemental Material S2.
The primary purpose for including the quiz was to encourage participants to stay actively engaged and apply sustained effort throughout the listening task. To this end, we included a minor deception to encourage effortful participation in the study. Specifically, participants were told that those whose “performance” was in the top 20% of all study participants would be included in an additional drawing for a $250 gift card. In fact, all participants were entered into this drawing regardless of their performance. After the study, participants were told of the deception and given the opportunity to withdraw their data. No participants withdrew their data from the study.
Test Environment
All study tasks were completed in a sound-attenuating booth. Auditory stimuli were presented in the free field and delivered via a single Bowes and Wilkins 685 S2 loudspeaker, mounted on a speaker stand with a height of 43 in.; visual stimuli were presented via custom software using Neurobehavioral System's Presentation program (Version 23.0, Build 10.27.1) and delivered to a 24-in. Dell monitor with a resolution of 1080 × 1920. Participants were seated directly in front (0° azimuth), approximately 50 in. (~1.3 m), from the Bowes and Wilkins loudspeaker. Key word RTs were recorded using a Targus number pad.
Procedure
Consent, Surveys, and Pure-Tone Threshold Measures
After providing written and oral consent, participants completed two questionnaires, the VFS-A-40 and the media use questionnaire. Pure-tone, sound-field thresholds at all octave and interoctave frequencies from 125 to 8000 Hz were also obtained for each participant using the same loudspeaker configuration as for speech testing.
Training session. Next, participants completed a training session to orient them to the main study tasks (details regarding these tasks are discussed in the main study tasks section). For the practice dual task, an initial presentation level for the first three passages was chosen individually (based on pilot work and Speech Intelligibility Index; SII [American National Standards Institute S3.5, 1997] predictions) to achieve a score of approximately 70% correct. Next, three additional passages were presented at either the same level (if the initial score approximated 70% correct) or at a lower or higher level if needed to better approximate a score of 70% correct. The new level was determined using SII calculations incorporating a proficiency factor based on the deviation of the initial measured score from predictions. Finally, two additional passages were presented either at the same level to confirm a score of ~70% correct or adjusted again (higher or lower) based on the prior scores and SII calculations incorporating an average proficiency factor derived from the scores on the initial six test passages. The presentation level of these final two practice passages was used during the main AV CST dual task for each participant (Caption and No Caption groups).
Participants assigned to have captions available during testing (i.e., Caption group) also repeated the first two practice passages (at the final presentation level described above) with captions visible. This was done to familiarize participants with the style of captions prior to the start of the main study procedures. Participants in the Caption group were explicitly instructed to actively listen to the audio and read the captions, and to not solely read the captions. Presentation levels were similar between the groups, with a mean level of 22 dBA (SD = 6 dB) for the No Caption group and 24 dBA (SD = 11 dB) for the Caption group. We chose to utilize test conditions that required such low presentation levels, in part, to limit ceiling effects. In addition, this study provides control data for future studies that will examine the utility of captions for individuals with hearing loss while also listening in quiet settings (e.g., via video teleconferencing in a classroom or office setting where noise levels are typically very low), with and without the use of hearing devices.
Main study session. After training, participants began the main study tasks. Before the dual-task paradigm, participants completed a baseline PVT (100 trials; ~10 min). They then completed the dual-task paradigm, which lasted ~50 min. The dual-task procedures were continuous (i.e., no breaks were taken once the procedure started). Before the start of the dual task, and after every eight passage pairs, participants provided subjective ratings of their fatigue using the RNFS. This resulted in a total of four RNFS ratings throughout the task. After the dual task was completed and the final RNFS rating was obtained, participants completed a second (posttask) PVT (also 100 trials; ~10 min). After this, participants completed the content-based quiz described above (see Figure 1 for depiction of study procedures in chronological order).
Data Analysis
All analyses were conducted in R (Version 4.3.0; R Core Team, 2023). Study outcomes included (a) measures of vigilant attention (PVT RTs) obtained before and after completing the dual task, (b) measures obtained during the dual task, including sentence recognition scores, RTs to topic words, and subjective fatigue ratings, and (c) quiz scores obtained after completing the dual task. Prior to analysis, RTs to the PVT and to topic words that were 3 SDs above or below the mean were removed.
Outcome measures were analyzed by building linear mixed-effects models (lme4; Bates et al., 2015) for each outcome; models were analyzed using the ANOVA function in base R. All models included group (Caption and No Captions groups) and block, in addition to their interaction, as fixed factors, and participant as a random intercept. Dependent variables were task scores (e.g., RTs, mean speech recognition scores, subjective fatigue ratings). The emmeans function (Lenth, 2023) was used to explore main effects and interactions with corrections for familywise error rate (Benjamini & Hochberg, 1995). Prior to analysis, the mean sentence recognition scores for each block were calculated for each participant. Trial-level data were used for the PVT RT and key word RT models.
Results
Sentence Recognition
Analysis of sentence recognition scores revealed a significant main effect of group, F(1, 20) = 5.30, p = .032, partial η^2^ = .21. The Caption group outperformed the No Caption group throughout the task, with average scores across blocks of 77% and 63% for participants in the Caption and No Caption groups, respectively. There was no significant main effect of block. The Group × Block interaction, however, was significant, F(2, 481) = 3.63, p = .027, partial η^2^ = .01. Further exploration of the interaction revealed that sentence recognition performance was significantly better among the Caption group at both Block 1 and Block 2 (estimated marginal means = 15% and 13% better; p = .014, .037; see Figure 2), but not at Block 3 (estimated marginal mean = 11% better; p = .07; see Figure 2). This small but significant effect results from a small decrease in scores over time for the Caption group (consistent with a fatigue-related performance decrement and a small increase in scores over time for the No Caption group (consistent with learning effects).
Subjective Fatigue Ratings
Analyses of subjective fatigue ratings obtained over the course of the dual task revealed a significant main effect of block, F(3, 60) = 33.92, p < .0001, partial η^2^ = .63, and a significant Group × Block interaction, F(3, 60) = 2.89, p < .01, partial η^2^ = .13. Specifically, fatigue ratings increased over time, but more so for the Caption group (subjective rating estimated marginal mean increased by 36.36 points from Block 0 to Block 3; p < .0001; see Figure 3) than the No Caption group (subjective rating estimated marginal mean increased by 20 points from Block 0 to Block 3; p = .0001; see Figure 3). There was no significant difference in fatigue ratings between the Caption and No Caption group at Block 0, meaning that participants in both groups started the task at similarly low levels of fatigue. The main effect of group was not significant.
Topic-Word RTs
There were no significant main effects of group or block on topic-word RTs. The two-way interaction was also not significant. However, we observed a notable increase in topic-word RTs from the second to third block for the Caption group. Given our a priori hypothesis that changes in RTs to topic words might provide a proxy measure of fatigue, we conducted follow-up testing using pairwise comparisons to further explore this increase in RTs in the Caption and No Caption groups separately. The increase in RTs from the second to third block for the Caption group was significant (estimated marginal mean = 55 ms; p = .047; see Figure 4), suggesting that participants in the Caption group experienced an increase in fatigue (i.e., an increase over time in exerted listening effort) from the second to third block. No statistically significant increase in RTs was observed for the No Caption group (estimated marginal mean = −1 ms; p = .99).
PVT RTs
Analyses revealed a significant main effect of time (pre–dual task vs. post–dual task) on PVT RTs, F(1, 4423.8) = 28.94, p < .001*,* partial η^2^ < .001 (see Figure 5), suggesting the demanding dual task was fatiguing for both groups. However, the main effect of group and the Group × Time interaction were not statistically significant, F(1, 20) = 0.01, p = .921, partial η^2^ < .001. While the Group × Time interaction was not statistically significant, RTs did increase slightly more for the Caption group (estimated marginal mean = 8 ms increase; p = .0001) compared to the No Caption group (estimated marginal mean = 7 ms increase; p = .0003), consistent with the pattern of results observed in the subjective and topic-word RT data.
Quiz
Analysis of participants' posttask quiz scores revealed a significant main effect of block, F(2, 1558.83) = 21.02, p < .001, partial η^2^ = .03. Answers to questions from information in the later passages (third/final block) were more likely to be correct compared to questions based on information from the first and second blocks. This was the case for both the Caption group (estimated marginal mean = 12 percentage point difference, p = .01; see Figure 6) and the No Caption group (estimated marginal mean = 24 percentage point difference, p < .0001; see Figure 6). Quiz scores were also significantly higher for the Caption group (M = 66%) than the No Caption group (M = 56%), but only for questions based on information presented toward the beginning of the task, t(526) = −2.24, p = .026.
Discussion
The purpose of this study was to evaluate the effects of imperfect captions on listening-related fatigue. Given previous findings showing captions, under certain conditions, could reduce subjective listening effort (Zhong et al., 2023) and the well-known relationship between effort and fatigue (e.g., Hockey, 2013), we hypothesized that, despite their imperfections, captions would reduce subjective and behaviorally measured fatigue. In contrast to this hypothesis, our results suggest that imperfect captions paired with audiovisual stimuli increase subjective listening-related fatigue (see Figure 3), even though access to these captions improves speech recognition (see Figure 2) and memory for material presented early during the testing block (see Figure 6). Furthermore, we found that imperfect captions provided no benefit to vigilant attention (PVT RTs), behaviorally measured listening-related fatigue (topic-word RTs), or memory for material toward the end of the task (posttask quiz). In other words, despite providing some benefits in terms of speech recognition, imperfect captions paired with audiovisual speech showed indications of negatively affecting participants in terms of listening-related fatigue.
The finding of increased fatigue when captions were available is especially surprising given the improvement in speech recognition for the Caption (77%) versus the No Caption (63%) group. With the improvement in understanding, we expected to see reduced listening effort for the Caption group. However, our results of increased fatigue are more suggestive of greater listening effort for our Caption group as they completed the ~50-min study task. Although the difference was not statistically significant, our finding that average topic-word RTs across blocks were greater (slower) for the Caption group compared to the No Caption group (see Figure 4) is consistent with the idea that the Caption group listening condition was more effortful. The lack of statistical significance in this analysis may be due, in part, to the reduced statistical power of our between-groups design.
The idea that imperfect captions, under some conditions, can increase listening-related fatigue is consistent with some prior studies of listening effort. Zekveld et al. (2009) examined imperfect captions paired with an audio only signal (i.e., no visual cues) and found that captions with lower accuracy rates (60%–70% accuracy vs. 80% accuracy) and increased timing delays (average text delay of 1,100 ms vs. no delay) were associated with increased subjective effort. In the current study, our behavioral measure of listening effort (topic-word RTs) was not significantly different between the Caption and No Captions groups. However, topic-word RTs did increase at the end of testing (suggestive of fatigue) but only for the Caption group, highlighting that the captions in our study did influence behavioral listening-related fatigue (and possibly behavioral listening effort).
The negative effects of captions in our study may have been exacerbated by our test condition. Recall that, consistent with some currently available real-time, automated captioning programs, our text captions were delayed by 500 ms relative to the audiovisual signal, and caption accuracy was imperfect. While 86%–88% of all speech was captioned accurately, only 60% of the scoring key words were captioned (40% of key words were deleted from the captions). Anecdotal reports from our participants confirmed that integrating the time-delayed (and incomplete) text information with the (low level) audio and visual (facial) information was mentally challenging. We received comments from our participants such as, “The captions were very distracting. It was hard to balance reading the text and straining to hear what the woman was saying,” and “The captions missing words threw me off, and the delay made it difficult to follow.” As such, integrating the three sources of information may have demanded a great deal of cognitive resources (e.g., Alsius et al., 2005). If so, such high cognitive demands would require a large allocation of effort toward the task and thus increase the likelihood of fatigue development, consistent with Hockey's (2013) model of fatigue. Additional research is needed to determine whether using imperfect captions in common conditions where audiovisual integration is not prioritized (e.g., captions paired with audio-only stimuli as in audio streaming, phone conferencing, or classroom lectures with slide presentations) might result in reduced cognitive demands. If so, such captions could provide benefits not only in terms of speech recognition but also reduced listening effort and fatigue.
Another factor to consider is that our participants had normal hearing. It is possible that individuals with hearing loss may be less, or more, affected by captions in our test condition. For example, adults with hearing loss may have stronger audiovisual integration than those with normal hearing. A comparative investigation of audiovisual integration by means of the McGurk illusion (i.e., incongruent auditory and visual input) revealed that individuals with hearing loss were more likely to provide visually biased responses, while participants with normal hearing provided responses that matched the auditory input (Roseman & Thiel, 2018). Individuals with hearing loss also demonstrate greater benefit from the addition of visual cues to auditory input. Moradi et al. (2016) found that audiovisual speech recognition for older adults with hearing loss was slightly poorer than older adults with normal hearing. However, older adults with hearing loss needed less time to correctly identify words in sentences with low semantic predictability during audiovisual presentation than older adults with normal hearing. This suggests that, particularly in more cognitively challenging conditions (i.e., low semantic predictability), older adults with hearing loss experience greater benefit from the addition of visual cues to auditory input. Although neither of these studies included the use of captions (imperfect or otherwise), they highlight the potential impact of hearing loss on the ability to integrate and utilize audiovisual information. Future work is needed to better understand how the benefits and limitations of captions, particularly ones that are imperfect, are impacted by hearing loss.
Limitations and Future Directions
In addition to the investigation of imperfect captions for listeners with hearing loss, future work is warranted to explore some of the limitations in the current work. For example, the test conditions used in our study (i.e., in quiet with audiovisual speech presented at a low sensation level) were not reflective of all conditions where imperfect captions may be used, such as in a classroom or conference session. These conditions, with the lack of a competing auditory signal (e.g., background talkers), may have limited the benefits provided by captions. In a study comparing the effects of background noise on caption benefit, young listeners with normal hearing demonstrated similar word recall, recognition, and recognition confidence in quiet, regardless of whether captions were presented or not. However, in the presence of background noise, these listeners demonstrated a greater benefit to word recall, recognition, and recognition confidence with captions (Payne et al., 2022), suggesting benefits of captions might be different in noise than at low sensation levels in quiet.
Additionally, although we classify the captions used in our study as “imperfect,” automated captions contain other distinct types of errors that were not reflected in the study captions. Namely, there were no substitution errors (e.g., “cap” instead of “cat”). Crandell et al. (2022) specifically examined the effects of different types of captioning errors on recall among older adults, finding that substitution and deletion errors had the largest negative impact on recall (as compared to insertion errors). Furthermore, recall was poorer with increasing levels of hearing loss. The authors posit that the participants may have relied more on the captions to compensate for poor audibility (due to both the stimulus presentation level and their hearing loss). While our participant pool was younger (average age = 36 years), anecdotal reports from our participants corroborate what Crandell et al. note about audiovisual integration. Specifically, several of our participants expressed that they had difficulty integrating the time-delayed, imperfect captions with the time-locked audio and visual cues. If this difficulty led to differences between groups in their sustained effort over time during the dual task, it may have played a role in the higher fatigue ratings observed in the Caption group. Further investigation is needed to determine the extent to which captions with better time-locking to the audiovisual stimuli and greater transcription accuracy may reduce listening-related fatigue. Such findings could provide evidence to support that the quality of captions matters as much, or even more so than access to captions alone.
Additionally, the use of imperfect captions in our study is not a limitation, per se, as they are more reflective of the transcription accuracy that automated speech-recognition systems provide. However, the nature of imperfect captions may be such that, regardless of hearing status, individuals derive minimal benefit from their supplement to auditory–visual information. Automated captions, which are characterized by imperfect transcriptions, are routinely used in classrooms or in conference sessions. Our data and findings from previous studies (e.g., Crandell et al., 2022) suggest that in certain settings where audibility is poor, captions may increase speech recognition at the cost of increased effort and fatigue. While it is certainly valuable to improve speech understanding, it is crucial to consider the impact of imperfect captions on effort and fatigue, which heavily contribute to social–emotional well-being and social engagement (Holman, Hornsby, et al., 2021).
The issue of imperfect captions is also an issue of accessibility. Nonautomated, human transcribers trained in CART for live programming can transcribe technical words and terminology with at least 96% accuracy (National Court Reporters Association, 2023), which is crucial for understanding in classrooms and conferences. In our quiz data, we observed a significantly greater benefit of having access to imperfect captions only for material that was presented toward the beginning of the task. Additional work is needed to determine whether improving caption accuracy leads to a greater improvement in learning throughout a single session. Should it be the case that more accurate captions do improve learning, it would be that much more important to provide CART or other forms of accurate captioning to students and professionals.
Lastly, the ability of our test measures to detect fatigue may be limited. Word-level RTs (i.e., isolated words that are not embedded in sentences) have been used to index behavioral listening effort (Huang et al., 2023; Picou & Ricketts, 2014). The use of RTs to topic words embedded within sentences is novel and thus might not be optimal for assessing listening effort and listening-related fatigue effects. More work is needed to develop measures that are sensitive to fatigue during sentence recognition tasks, not just word recognition tasks. Further investigations into listening-related fatigue and imperfect captions can inform how listeners with normal hearing (and with hearing loss) can leverage audiovisual information to both improve speech understanding and reduce fatigue.
Conclusions
The purpose of this study was to evaluate the effects of imperfect captions on listening-related fatigue using behavioral and subjective measures of fatigue during a sustained listening task. The results of this study demonstrate that access to imperfect captions led to significantly greater sentence recognition performance. However, although subjective fatigue significantly increased over time for those with captions and without captions, increases in subjective fatigue were greater for the group with captions. In addition, behavioral fatigue increased significantly toward the end of the task for the group with captions, evidenced by slower topic-word RTs. Visual RTs showed an increase in fatigue in both groups. Future work is needed to determine the effect of imperfect captions on listening-related fatigue compared to more accurate captions. These investigations should also include listeners with hearing loss, given they are more likely than those without hearing loss to experience severe levels of listening-related fatigue (Hornsby et al., 2023, 2024). This is particularly important as it relates to supplementing traditional interventions offered by clinicians (e.g., amplification and cochlear implants) and improving accessibility in public spaces and events. As discussed, captions can be “imperfect” in many ways (i.e., containing deletion, substitution, and insertion errors; time-delayed relative to the audiovisual stimulus). With the increased reliance on automated speech recognition systems and artificial intelligence, “imperfect” captions will become even more ubiquitous. As such, there is a need to identify the types of imperfections that influence speech understanding and listening-related fatigue and how they interact with environmental and individual factors such as noise and hearing loss. These discoveries can in turn guide the implementation of “imperfect” captions, improving accessibility in an effective and economical manner.
Data Availability Statement
De-identified data will be made available to researchers upon reasonable request to the corresponding author.