Authors: Michelle E. Stepan (1Department of Psychiatry, University of Pittsburgh), Erik M. Altmann (2Department of Psychology, Michigan State University), Kimberly M. Fenn (2Department of Psychology, Michigan State University)
Categories: Article, sleep deprivation, caffeine, placekeeping, vigilant attention, procedural error
Source: Journal of experimental psychology. Learning, memory, and cognition
Doi: 10.1037/xlm0001023
Authors: Michelle E. Stepan, Erik M. Altmann, Kimberly M. Fenn
Sleep deprivation impairs a wide range of cognitive processes, but the precise mechanism underlying these deficits is unclear. One prominent proposal is that sleep deprivation impairs vigilant attention, and that impairments in vigilant attention cause impairments in cognitive tasks that require attention. Here, we test this theory by studying the effects of caffeine on visual vigilant attention and on placekeeping, a cognitive control process that plays a role in procedural performance, problem solving, and other higher-order tasks. In the evening, participants (N = 276) completed a placekeeping task (UNRAVEL) and a vigilant attention task (the Psychomotor Vigilance Task, or PVT) and were then randomly assigned to either stay awake overnight in the laboratory or sleep at home. In the morning, participants who slept returned to the lab, and all participants consumed a capsule that contained either 200 mg of caffeine or placebo. After an absorption period, all participants completed UNRAVEL and PVT again. Sleep deprivation impaired performance on both tasks, replicating previous work. Caffeine counteracted this impairment in vigilant attention but did not significantly affect placekeeping for most participants, though it did reduce the number of sleep-deprived participants who failed to maintain criterion accuracy. These results suggest that sleep deprivation impairs placekeeping directly through a causal pathway that does not include visual vigilant attention, a finding that has implications for intervention research and suggests that caffeine has limited potential to reduce procedural error rates in occupational settings.
Sleep deprivation causes deficits in a wide range of activities, but there is disagreement about the underlying cognitive mechanisms. A prominent view, the vigilance hypothesis (Lim & Dinges, 2010), holds that deficits in vigilant attention explain the majority of cognitive performance deficits due to sleep loss. The strong interpretation of this view is that sleep deprivation impairs vigilant attention directly and impairs other cognitive processes indirectly because they require attention (Balkin, Rupp, Picchioni, & Wesensten, 2008; Doran, Van Dongen, & Dinges, 2001). This view is parsimonious, in that it explains the deficits caused by sleep deprivation in terms of one process shared by many tasks. Supporting this view, sleep deprivation produces large and widely-replicated performance deficits in tasks that primarily measure vigilant attention (Lim & Dinges, 2008, 2010) but has weaker effects on tasks that measure other or more heterogeneous cognitive processes (Lim & Dinges, 2010; Lo et al., 2012). For some tasks, including a psychomotor control task (Bolkhovsky, Ritter, Chon, & Qin, 2018), 24 hr of sleep deprivation had no effect on performance. In a study using a modified Sternberg task, different measures showed different effects; sleep deprivation impaired measures of probe encoding and response selection but not measures of working memory scanning, suggesting that sleep deprivation affected non-executive processes but spared executive processes (Tucker et al., 2010). Thus, sleep deprivation consistently impairs performance on vigilant attention tasks but has mixed effects for tasks that measure other aspects of cognition.
One implication of the vigilance hypothesis is that attention should fully statistically mediate individual differences in how sleep deprivation affects performance of tasks that require attention. We tested this implication in a previous study, finding that sleep deprivation directly accounted for a significant amount of between-participants variability in performance of a criterion task after we statistically removed the mediating effect of sleep deprivation on attention (Stepan, Altmann, & Fenn, 2020). This result is not consistent with the strong interpretation of the vigilance hypothesis, which implies that all causal paths linking sleep deprivation to task performance run through vigilant attention. Instead, this result indicates that sleep deprivation can directly affect cognitive processes through causal paths that do not include attention. This result is most consistent with the neuropsychological hypothesis (Lim & Dinges, 2010), the view that sleep deprivation effects are akin to specific neuropsychological deficits in a range of cognitive processes and cannot be solely explained by general attention deficits (Chuah, Venkatraman, Dinges, & Chee, 2006; Harrison & Horne, 2000; Mu, et al., 2005).
Another implication of the vigilance hypothesis concerns effects of interventions. Specifically, interventions that mitigate attention deficits should also mitigate a wide range of other performance deficits, if causal paths linking sleep deprivation to task performance run through vigilant attention. We test this implication here, using an experimental approach. The intervention we use is caffeine, which is known to improve vigilant attention under conditions of sleep deprivation (Beaumont et al., 2001; McLellan et al., 2005; Wesensten et al., 2002). Effects of caffeine on vigilant attention have been widely demonstrated using the Psychomotor Vigilance Task (PVT), a simple reaction time task that taps into sustained attention and vigilance processes (Doran et al., 2001). Sleep-deprived participants who receive caffeine exhibit fewer attentional lapses on the PVT compared with sleep-deprived participants who receive placebo (Kamimori, Johnson, Thorne, & Belenky, 2005; Killgore et al., 2008; McLellan et al., 2005). Often, sleep-deprived performance with caffeine is similar to baseline levels, suggesting that caffeine may eliminate vigilant attention impairments due to sleep deprivation. This interpretation should be treated with caution, however, because rested control groups are infrequently used, making it difficult to know how performance of sleep-deprived individuals who have consumed caffeine compares to performance of rested individuals at the same timepoint (an issue we address in this study). The timepoint is important because performance on the PVT shows strong circadian variation (Kline et al., 2010).
As a second cognitive process for testing the vigilance hypothesis, we focused on placekeeping. Placekeeping is a cognitive control process required when a set of steps or subtasks must be performed in a specified order, without repetitions or omissions, despite interruptions that make it difficult to remember one’s place in the correct sequence (Altmann, Trafton, & Hambrick, 2014). Placekeeping is impaired by sleep deprivation (Stepan, Fenn, & Altmann, 2019) and makes an attractive target for potential interventions because it supports performance of a wide array of cognitive tasks. Most obviously, placekeeping supports performance of procedures, which consist of sequences of steps to be performed in a specified order. Other examples are multitasking environments that require interleaving of procedures (Burgoyne, Hambrick, & Altmann, in press), and event counting (Carlson & Cassenti, 2004), where the steps are events to be counted and the sequence is the positive integers. Placekeeping also plays a role in problem solving, allowing the solver to search a problem space efficiently, without repeatedly exploring failed paths, and effectively, without omitting solution paths (Hambrick, Burgoyne, & Altmann, 2020). Problem solving is, in turn, a basis of fluid intelligence (Gf), and placekeeping ability is strongly related to Gf (Hambrick & Altmann, 2015), even more so than working memory capacity (Burgoyne, Hambrick, & Altmann, 2019). This broad scope makes placekeeping a suitable criterion process for testing whether sleep deprivation has direct effects on processes other than attention.
There are several reasons to think that placekeeping requires attention and thus should, according to the vigilance hypothesis, show benefits of caffeine under conditions of sleep deprivation. Attention is necessary for encoding and retrieval processes associated with episodic memory, which stores memories of past performance that allow the system to determine which steps of the task remain to be performed (Altmann & Trafton, 2015). Attention is also necessary for maintenance processes in episodic memory, which support placekeeping when there are temporal gaps (interruptions) between when a step is performed and when a memory of that performance is required (Altmann, Trafton, & Hambrick, 2017; Stepan et al., 2019). Moreover, the stimulus materials in our placekeeping task generate response conflict in the manner of an incongruent Stroop or flanker stimulus, such that attention is required to override priming of incorrect steps (as we discuss further in the Materials section). Finally, the mapping from stimuli to responses in our placekeeping task is variable rather than consistent, which should block automatization as a mechanism that reduces need for attention (Shiffrin & Schneider, 1977).
No studies to date have directly measured effects of caffeine on placekeeping under conditions of sleep deprivation. This gap is itself important to address, given the role of placekeeping in procedural performance. Major accidents have been attributed to procedural errors occurring under conditions of sleep deprivation (e.g., Navy Office of Information, 2017), so it would be useful to know if caffeine mitigates effects of sleep deprivation on procedural performance. Thus, the theoretical question we address concerning the vigilance hypothesis has implications for predicting which interventions will be beneficial for which cognitive activities.
There is mixed evidence on whether caffeine affects performance of higher-order cognitive tasks in which vigilant attention or placekeeping may play a role. Some studies have found that caffeine improved sleep-deprived performance on certain tasks assessing problem solving (Killgore et al., 2009) or reasoning and strategy development (Wesensten, Killgore, & Balkin, 2005). However, these tasks were administered only once to sleep-deprived individuals, with no rested baseline measure and no rested control group, so whether caffeine completely restored performance to rested levels, or whether caffeine affected rested performance, is unknown. Caffeine did not improve performance on a number of other tasks assessing problem solving (Killgore et al., 2009; Wesensten, et al., 2005), working memory (Wesensten et al., 2002), verbal fluency (Wesensten et al., 2005), emotion-guided decision making (Killgore, Grugle, & Balkin, 2012; Killgore, Lipizzi, Kamimori, & Balkin, 2007), perseveration (Gottselig, et al., 2006), inhibitory control (Wesensten et al., 2005), or humor appreciation (Killgore, McBride, Killgore, & Balkin, 2006), and had a limited effect on rifle shooting, improving speed-based elements of the task but not precision (Tikuisis, Keefe, McLellan, & Kamimori, 2004). Caffeine was also associated with poorer inhibitory control in individuals with poor sleep quality (Anderson, Hagerdorn, Gunstad, & Spitznagel, 2018). Interpretation of these results is often complicated by small sample sizes, a lack of baseline measures when the task can only be administered once due to learning effects, and a lack of rested control groups. Nonetheless, these results suggest that whether caffeine improves performance of a given task is not guaranteed.
Based on previous research, we expect that sleep deprivation will impair visual vigilant attention and placekeeping, and that caffeine will counteract this impairment for visual vigilant attention. Of interest is whether caffeine counteracts the impairment for placekeeping as well. To address this question, we conducted an experiment in which we manipulated three condition (sleep^1^, sleep-deprived), pill (placebo, caffeine), and task (vigilant attention, placekeeping). Our primary interest was in whether the pill and task factors would interact. A null interaction would be consistent with the vigilance hypothesis, because it could mean that effects of caffeine on vigilant attention were reflected in placekeeping. In contrast, a significant interaction would be more consistent with the neuropsychological hypothesis. If caffeine affected performance of the two tasks differently, with no effect on placekeeping, this would indicate that impairments in placekeeping caused by sleep deprivation cannot be counteracted by improved vigilant attention.
We measured vigilant attention with the PVT and placekeeping with the UNRAVEL task (Altmann et al., 2014), which we describe below. Following best practices, we used a large sample (N = 276), a control group that slept, and baseline measures administered under double-blind conditions. For a reference point concerning sample size, a meta-analysis of 70 studies of acute sleep deprivation (< 48 hours) found an average sample size of N = 21.3 (Lim & Dinges, 2010). The sleep control group allows us to measure control performance at the same timepoint as treatment performance, to avoid confounds with time-of-day and practice or other sequential effects. The baseline measures allow us to reduce error variance by controlling for stable individual differences in task performance.
We administered the baseline measures in the evening, when neither participants nor research assistants were aware of group assignments. Participants were then randomly assigned to go home and sleep or stay awake overnight in the laboratory. We assessed performance again the following morning. For the morning session, sleep and sleep-deprived participants were randomly assigned to receive placebo or caffeine, forming four sleep placebo, sleep caffeine, sleep-deprived placebo, and sleep-deprived caffeine. Every participant performed both tasks in both sessions.
Participants were undergraduate students at Michigan State University. Participants used caffeine in moderation (up to 400 mg daily), had no heart conditions, had never been diagnosed with memory or sleep disorders, were not color blind, did not have a strong time-of-day preference [scores of 42–58 on the Morningness-Eveningness Questionnaire (Horne & Östberg, 1975)], did not have major sleep disturbances [scores of 0–10 on the sleep disturbance section of the Pittsburgh Sleep Quality Index (Buysse et al., 1989)], and were native English speakers. To meet the native-English criterion, participants had to have learned English as their first language, be fluent in English, and not be an international student. Participants slept a minimum of 6 hours the night before the study began and woke up on the first day of the study by 00. Participants refrained from napping on the first day of the study and had not consumed any caffeine, alcohol, or drugs for 24 hours prior to the study. All criteria were assessed using self-report.
Participants recorded their sleeping habits in a sleep diary for five nights leading up to the study. Data from the sleep diaries are reported in Table 1. Participants reported sleeping between 7.5 and 8 hrs on average each night, including the night before the first day of the study. There were no differences in sleep time between sleep and sleep-deprived participants.
Of an initial sample of 305 participants, 2 were excluded due to technical problems, 7 due to experimenter error, 5 for attrition, 5 for missing PVT data, and 10 for failing to meet an accuracy criterion for the UNRAVEL task in the evening session, as described in the Materials section. Data from the remaining 276 participants [demographic information for n = 18–26 years old (M = 19.09, SD = 1.28), 182 females] were submitted to analysis. As described in the Procedure section, these participants were randomly divided into four sleep placebo (n = 61), sleep caffeine (n = 68), sleep-deprived placebo (n = 77), and sleep-deprived caffeine (n = 70). Data from the placebo groups (n = 138) were reported by Stepan et al. (2020); there we analyzed the between-participant variability, which here we treat as error variance. Our stopping rule for data collection was that we collected data through two full semesters. Informed consent was obtained from all participants, who received course credit as compensation. The study was approved by Michigan State University’s Institutional Review Board.
We used the PVT (Wilkinson & Houghton, 1982; Dinges & Powell, 1985) to measure visual vigilant attention. We used the version available with the Inquisit software platform (millisecond.com, Seattle, WA).
In the PVT, participants monitor a blank computer screen for the appearance of a large red circle in the center of the screen and were instructed to make a mouse click as quickly as possible. Each appearance of the red circle is designated a trial. When the participant clicked the mouse, the red circle offset and the participant received feedback on their reaction time. The feedback message remained on the screen for 0.5 s. The circle appeared at random intervals between 1 and 10 s. The task lasted 10 min. The dependent variable of interest was the rate of lapses, a lapse being a reaction time greater than 500 ms.
We used the UNRAVEL task (Altmann et al., 2014) to measure placekeeping. The task is implemented as custom software developed in Python.
‘UNRAVEL’ is an acronym identifying seven procedural steps (one step per letter of the acronym) and the correct order in which to perform them (the order of the letters in the acronym). On each trial, the participant performs one step, advancing to the next step on the next trial. When the participant reaches ‘L,’ he or she directly starts over with ‘U,’ generating continuous performance. Performing a step involves applying a two-alternative forced-choice decision rule unique to that step to a randomly generated stimulus. An UNRAVEL session contained four test blocks, each with an average of 66 trials, and took about 35 minutes to complete.
Figure 1 shows two examples of randomly generated UNRAVEL stimuli as well as the choice rules for the different steps. Stimuli contain no information about which step is to be performed; therefore, participants have to remember which step is currently correct. Furthermore, stimuli contain features that are often incongruent with the correct step or response. For example, the stimulus may contain the letter ‘U,’ which primes the ‘U’ step even if a different step is currently correct. We assume that incongruencies like this generate response conflict that has to be managed with attention in order to generate correct responses.
Performance was randomly interrupted with a transcription typing task. For each interruption, participants had to correctly transcribe two strings of letters, each consisting of the 14 possible UNRAVEL responses presented in random order. This task takes about 20 seconds. After an interruption, participants are expected to resume where they left off in the UNRAVEL sequence prior to the interruption. There were ten interruptions per test block.
In the evening, participants were given computer-based instructions on how to perform the UNRAVEL task, followed by a practice phase to familiarize them with the task and assess their understanding. If a participant had 15 or more incorrect trials (correct trials are defined below) during the practice phase, they completed the practice phase a second time. If a participant had fewer than 15 incorrect trials, they progressed to the test blocks. In the morning, participants completed a brief practice phase to remind them of the rules and procedure before completing the test blocks.
The dependent variable of interest was the rate of placekeeping errors. A placekeeping error occurs when a participant performs a step that is incorrect relative to the step performed on the previous trial. For example, if a participant performed ‘N,’ ‘R,’ ‘V,’ and ‘E’ on four successive trials, the ‘V’ trial would be scored as a placekeeping error because the correct step after ‘R’ is ‘A,’ but the ‘E’ trial would be scored as correct because it followed ‘V.’ We distinguish between two types of placekeeping errors, post-interruption errors and non-interruption errors. A post-interruption error occurs on a trial that immediately follows an interruption, whereas a non-interruption error occurs on a trial that immediately follows another trial. Post-interruption errors occur at a greater rate and reflect a greater burden on memory processes (Stepan et al., 2019), so we analyze the two error types separately.
We also recorded decision-rule errors, which occur when a participant selects the correct step in the sequence but makes the incorrect response for the two-alternative forced-choice decision. Decision-rule errors are reported in the supplemental online material (SOM).
Participants received feedback on their accuracy at the end of each test block. A trial was scored as correct if there was neither a placekeeping error nor a decision-rule error. If the proportion of correct trials in a test block was below 70%, the feedback message at the end of the block asked the participant to be more accurate. A session was coded as a failure if the participant’s accuracy was below 70% on two or more blocks, on grounds that the participant was unable or unwilling to respond to the instruction to be more accurate. We excluded participants who failed the evening session (n = 10) from our sample because we could not be sure that they understood the task.
Participants were recruited for a study on sleep deprivation and caffeine. All participants arrived at the laboratory at 00 having been advised that they would either stay awake all night or go home and sleep. Participants were run in groups of up to 11 (M = 7 participants), across two testing rooms. When participants arrived for the evening session, they completed sleepiness and mood assessments (reported in the SOM) and then UNRAVEL, PVT, and other cognitive assessments associated with another study. These assessments and tasks lasted approximately 2 hrs. Participants were then randomly assigned to either a sleep or a sleep-deprived cohort, subject to the constraint that the sleep-deprived cohort contain five or six participants. Participants and research assistants were blind to condition up to this point. Participants in the sleep cohort received a Charge 2 activity monitor (Fitbit Inc., San Francisco, CA) to track their sleep at home that night. Actigraphy data are reported in Table 2.
Participants in the sleep-deprived group stayed awake in the laboratory overnight, monitored by two trained research assistants, who were not the individuals who administered the evening or morning sessions. At three timepoints (00:30, 30, and 30), participants were given a capsule. The first two capsules (00:30 and 30) contained placebo and the last capsule (08:30) contained either placebo or 200 mg of caffeine. The caffeine administration schedule is shown in Table 3.^2^ Capsules were distributed in a double-blind fashion, with participants and research assistants blind to condition.
Sleep-deprived participants were allowed to read, do homework, watch TV or movies, play board or card games, or engage in other quiet activities, but were not permitted activities that would activate the autonomic nervous system. Participants were allowed to consume any food or beverage that did not contain caffeine or alcohol. Every two hours (01:00, 00, 00, and 00) participants were taken into a different testing room and seated at a different computer than during the evening and morning sessions and completed sleepiness and mood assessments. Participants were sleep-deprived for approximately 24 hours before starting the morning tasks.
Sleep participants returned the next morning at 30 and were randomly assigned to receive a capsule that contained either placebo (n = 61) or 200 mg of caffeine (n = 68). Capsules were again distributed in double-blind fashion. At 00, after an absorption period, all participants began the morning tasks, which included sleepiness and mood assessments, UNRAVEL, PVT, and other cognitive tasks associated with another study. During testing, participants were seated at individual computers with dividers between them so that members of different groups were not interacting with each other. The morning session lasted approximately 1.5 hr, after which sleep-deprived participants were given a ride home to avoid having them drive in a sleep-deprived state.
Our basic design was a 2 (condition: sleep, sleep-deprived) x 2 (pill: placebo, caffeine) x 2 (task: PVT, UNRAVEL) analysis of covariance (ANCOVA), with condition and pill as between-participants factors and task as a within-participants factor. The dependent variable was the event rate. An event was either a lapse in the PVT or a placekeeping error in the UNRAVEL task, and we formed event rates by dividing the number of events by the number of trials on which that event could occur. The covariates were the baseline event rates collected during the evening session, which control for stable individual differences in task performance. To stabilize the variance, which scales with the mean in rate data, we transformed all event rates using the arcsine-root transformation, which is appropriate for proportion data containing true zeroes (and which, unlike the Box-Cox transformation, does not require estimation of a free parameter). In the SOM we report raw morning and evening event rates and raw lapse counts for the PVT.
We conducted three analyses using this design. In Analysis 1 we considered our whole sample (N = 276). In Analyses 2 and 3 we partitioned the sample by separating the worst performers in the morning session from the rest. The purpose of this partitioning was to guard against misinterpreting effects from Analysis 1 that might be driven by participants who are not performing the task as instructed, whether due to unwillingness or inability to perform after sleep deprivation. In Analysis 2, we partitioned the sample based on UNRAVEL performance, according to the a priori criterion for session failure we described in the Materials section. In Analysis 3 we partitioned the sample based on PVT performance, separating the same number of the worst performers as we did in Analysis 2.
As we noted in the Materials section, post-interruption and non-interruption errors in the UNRAVEL task occur at different rates and measure different levels of memory load, so we examine them separately within each analysis. Thus, we first compare the PVT lapse rate with the UNRAVEL post-interruption error rate, then compare the PVT lapse rate with the UNRAVEL non-interruption error rate. These two comparisons are not orthogonal, because the PVT data are the same and because post-interruption and non-interruption error rates are correlated. Nonetheless, this approach allows us to analyze all data without obscuring any differences between trial types and to obtain a degree of internal replication for the UNRAVEL results.
In this analysis we applied our design to the full sample. Averaged across conditions, the morning PVT lapse rate was M = .105 (SD = .108), the UNRAVEL post-interruption error rate was M = .209 (SD = .204), and the UNRAVEL non-interruption error rate was M = .045 (SD = .112).
Estimated mean event rates by condition are reported in Table 4. The overall pattern is that sleep deprivation impaired performance on both tasks and caffeine improved performance on both tasks. For the comparison of PVT lapses with UNRAVEL post-interruption errors, the main effect of condition was significant, F(1, 270) = 57.9, p < .001, ηp^2^ = .177, indicating that sleep deprivation impaired performance. The main effect of pill was significant, F(1, 270) = 12.6, p < .001, ηp^2^ = .045, indicating that caffeine improved performance. There was no main effect of task, F < 1, indicating that lapses in the PVT occurred at about the same rate as post-interruption errors in UNRAVEL. The condition x pill interaction was not significant, F(1, 270) = 2.4, p = .121, ηp^2^ = .009, suggesting that the improvement caused by caffeine did not differ for the sleep-deprived and sleep groups. The condition x task interaction was not significant, F(1, 270) = 1.1, p = .303, ηp^2^ = .004, indicating that the impairment caused by sleep deprivation did not differ by task. The pill x task interaction was not significant, F(1, 270) = 2.2, p = .138, ηp^2^ = .008, suggesting that the improvement caused by caffeine did not differ by task. The condition x pill x task interaction was not significant, F < 1.
For the comparison of PVT lapses with UNRAVEL non-interruption errors, the main effect of condition was significant, F(1, 270) = 55.3, p < .001, ηp^2^ = .170, indicating that sleep deprivation impaired performance. The main effect of pill was significant, F(1, 270) = 15.8, p < .001, ηp^2^ = .055, indicating that caffeine improved performance. Here, there was a main effect of task, F(1, 270) = 21.0, p < .001, ηp^2^ = .072, indicating that lapses in the PVT were more frequent than non-interruption errors in UNRAVEL. The condition x pill interaction was marginally significant, F(1, 270) = 2.8, p = .096, ηp^2^ = .010, suggesting that the improvement caused by caffeine may have differed somewhat for the sleep-deprived and sleep groups. The condition x task interaction was not significant, F(1, 270) = 1.5, p = .228, ηp^2^ = .005, suggesting that the impairment caused by sleep deprivation did not differ by task. Here, the pill x task interaction was significant, F(1, 270) = 4.6, p = .033, ηp^2^ = .017, indicating that the improvement caused by caffeine differed by task. The condition x pill x task interaction was not significant, F < 1.
To probe the pill x task interaction, we performed separate condition x pill ANCOVAs for each task. For the PVT, there were main effects of condition, F(1, 271) = 50.1, p < .001, ηp^2^ = .156, and pill, F(1, 271) = 22.9, p < .001, ηp^2^ = .078. These effects are consistent with previous research indicating that sleep deprivation impairs vigilant attention and that caffeine mitigates the impairment. The condition x pill interaction was not significant, F(1, 271) = 2.0, p = .163, ηp^2^ = .007, suggesting that caffeine effects did not differ for the sleep-deprived and sleep groups. For non-interruption errors in UNRAVEL, the main effect of condition was significant, F(1, 271) = 21.2, p < .001, ηp^2^ = .072, and the main effect of pill was marginally significant, F(1, 271) = 2.8, p = .098, ηp^2^ = .010. The condition x pill interaction was not significant, F(1, 271) = 1.4, p = .230, ηp^2^ = .005.
These results indicate that caffeine affected performance on both tasks, although the effect on one of the UNRAVEL measures was marginal. In the next analysis we ask whether these results are robust to the influence of participants who are not performing the UNRAVEL task as instructed.
In this analysis we partition our data according to whether participants passed or failed the morning session of the UNRAVEL task. As we noted in the Materials section, we code an UNRAVEL session as a failure if the participant’s accuracy is below 70% on two or more test blocks. When accuracy is below 70% on a block, the participant is asked to be more accurate at the end of the block, so accuracy below 70% on two or more blocks is evidence that the participant was not performing the task as instructed. Partitioning the data according to this criterion allows us to test whether the effect of caffeine on task performance is robust to the influence of this particular source of error variance. We have used this same partitioning criterion in all our previous experimental work with the UNRAVEL task (Altmann & Trafton, 2015; Altmann et al., 2014, 2017; Altmann & Hambrick, 2017; Stepan et al., 2019).
Of 276 total participants, 22 (8.0%) failed the morning UNRAVEL session. Averaged across conditions, the PVT lapse rate for these 22 participants was M = .232 (SD = .129), the UNRAVEL post-interruption error rate was M = .648 (SD = .157), and the UNRAVEL non-interruption error rate was M = .320 (SD = .251).
Given the small sample size of participants who failed the morning session of UNRAVEL, we do not perform inferential statistics on event rates. Instead, we examine group effects on the rate of session failures. The 22 participants who failed included more sleep-deprived participants (22, n = 147) than sleep participants (0, n = 129), χ^2^(1, n = 276) = 21.0, p < .001. There were more sleep-deprived placebo participants (16, n = 77) than sleep-deprived caffeine participants (6, n = 70), χ^2^(1, n = 147) = 4.3, p = .038, and more sleep-deprived caffeine participants than sleep placebo participants (0, n = 61), p = .030 and sleep caffeine participants (0, n = 68), p = .028 by Fisher’s Exact Test.
These results indicate that a night of sleep deprivation made some participants unable or unwilling to meet the modest accuracy criterion that they had met the night before (70% or greater accuracy for at least three of four test blocks). Caffeine reduced the number of such participants, but not to the level of the sleep group.
Here we examine event rates for the large majority of participants (254 of 276, or 92.0%) who passed the morning UNRAVEL session. The cell counts for this analysis were roughly balanced (sleep placebo n = 61, sleep caffeine n = 68, sleep-deprived placebo n = 61, and sleep-deprived caffeine n = 64). Averaged across conditions, the PVT lapse rate was M = .094 (SD = .099), the UNRAVEL post-interruption error rate was M = .171 (SD = .158), and the UNRAVEL non-interruption error rate was M = .021 (SD = .037).
Estimated mean event rates by condition are reported in Table 5 and plotted in Figure 2. The overall pattern is that sleep deprivation impaired performance on both tasks, but caffeine improved performance only for the PVT. For the analysis comparing PVT lapses with UNRAVEL post-interruption errors (left versus middle panels of Figure 2), the main effect of condition was significant, F(1, 248) = 30.7, p < .001, ηp^2^ = .110, indicating that sleep deprivation impaired performance. The main effect of pill was significant, F(1, 248) = 7.3, p = .007, ηp^2^ = .029, indicating that caffeine improved performance. There was no main effect of task, F < 1, indicating that lapses in the PVT occurred at about the same rate as post-interruption errors in UNRAVEL. The condition x pill interaction was not significant, F < 1, indicating that the improvement caused by caffeine did not differ for the sleep-deprived and sleep groups. The condition x task interaction was not significant, F(1, 248) = 1.2, p = .280, ηp^2^ = .005, indicating that the impairment caused by sleep deprivation did not differ by task. Of central interest is that the pill x task interaction was significant, F(1, 248) = 5.6, p = .019, ηp^2^ = .022, indicating that caffeine affected performance of the two tasks differently. The condition x pill x task interaction was not significant, F < 1, indicating that this differential effect of caffeine on the two tasks was not further moderated by sleep deprivation.
To probe the pill x task interaction, we performed separate condition x pill ANCOVAs for each task. For the PVT, there were main effects of condition, F(1, 249) = 34.8, p < .001, ηp^2^ = .123, and pill, F(1, 249) = 17.3, p < .001, ηp^2^ = .065. The condition x pill interaction was not significant, F < 1, indicating that caffeine effects did not differ for the sleep-deprived and sleep groups. For post-interruption errors in UNRAVEL, the main effect of condition was significant, F(1, 249) = 9.9, p = .002, ηp^2^ = .038, but the main effect of pill was not, F < 1; thus, sleep deprivation impaired placekeeping and caffeine did not mitigate this impairment. The condition x pill interaction was not significant, F < 1.
For the analysis comparing PVT lapses with UNRAVEL non-interruption errors (left versus right panels of Figure 2), the results were similar. The main effect of condition was significant, F(1, 248) = 31.0, p < .001, ηp^2^ = .111, indicating that sleep deprivation impaired performance. The main effect of pill was significant, F(1, 248) = 11.7, p = .001, ηp^2^ = .045, indicating that caffeine improved performance. There was a main effect of task, F(1, 248) = 20.6, p < .001, ηp^2^ = .077, indicating that lapses in the PVT were more frequent than non-interruption errors in UNRAVEL. The condition x pill interaction was not significant, F < 1, indicating that the improvement caused by caffeine did not differ for the sleep-deprived and sleep groups. The condition x task interaction was significant, F(1, 248) = 17.4, p < .001, ηp^2^ = .066, indicating that the impairment caused by sleep deprivation differed by task. Here again, the pill x task interaction was significant, F(1, 248) = 12.7, p < .001, ηp^2^ = .049, indicating that caffeine affected performance of the two tasks differently. The condition x pill x task interaction was not significant, F < 1, indicating that this differential effect of caffeine on the two tasks was not further moderated by sleep deprivation.
To probe the pill x task interaction—and to test whether sleep deprivation affected non-interruption errors, given the condition x task interaction in the omnibus analysis—we performed a separate condition x pill ANCOVA for non-interruption errors in UNRAVEL. A corresponding analysis for the PVT is reported earlier in this section. The main effect of condition was significant, F(1, 249) = 3.9, p = .049, ηp^2^ = .015, but the main effect of pill was not, F < 1; thus, as for post-interruption errors, sleep deprivation impaired placekeeping and caffeine did not mitigate the impairment. The condition x pill interaction was not significant, F < 1.
Finally, to assess how fully caffeine counteracted sleep deprivation effects on PVT performance, we compared morning performance of sleep-deprived caffeine participants with morning performance of sleep placebo participants, with evening performance as a covariate. The difference was not significant, F(1, 122) = 1.5, p = .226, ηp^2^ = .012, indicating that caffeine mitigated the effect of sleep deprivation to the point that the impairment was not detectable in comparison with a control group that had slept overnight and was tested at the same diurnal time.
These results indicate that the effect of caffeine on UNRAVEL performance in Analysis 1 was driven by the small minority of participants who failed the morning UNRAVEL session, and that excluding these participants left the effect of caffeine on PVT performance intact. In the next analysis, we ask whether the worst PVT performers similarly drive the effect of caffeine on PVT performance.
The PVT does not include an a priori diagnostic criterion analogous to the one in the UNRAVEL task for assessing failed performance on a session. Nonetheless, we sought to conduct an analysis comparable to the one in the previous section to address the possibility that the effect of caffeine on PVT performance was driven by the worst PVT performers, the same way that the effect of caffeine on UNRAVEL performance was driven by the worst UNRAVEL performers.
We identified the 22 participants with the highest PVT lapse rates and treated those as we did the 22 participants who failed the morning UNRAVEL session. There were 8 participants in common across the two groups of 22.
For the 22 worst morning PVT performers, the PVT lapse rate, averaged across conditions, was M = .380 (SD = .086), the UNRAVEL post-interruption error rate was M = .408 (SD = .253), and the UNRAVEL non-interruption error rate was M = .178 (SD = .243).
The 22 worst performers included more sleep-deprived participants (18, n = 147) than sleep participants (4, n = 129), χ^2^(1, n = 276) = 7.8, p = .005, and more placebo participants (18, n = 138) than caffeine participants (4, n = 138), χ^2^(1, n = 276) = 9.7, p = .002. There were more sleep-deprived placebo participants (14, n = 77) than sleep-deprived caffeine participants (4, n = 70), χ^2^(1, n = 147) = 5.3, p = .021. The number of sleep-deprived caffeine participants did not significantly differ from the number of sleep placebo participants (4, n = 61), p = .562 or the number of sleep caffeine participants (0, n = 68), p = .120 by Fisher’s Exact Test.
These results indicate that the worst PVT performers were primarily sleep-deprived placebo participants. Caffeine reduced the number of sleep-deprived worst performers to levels comparable to the number of sleep participants.
In this analysis we applied our design to the 254 participants that remain after excluding the 22 worst PVT performers. The cell counts for this analysis were again roughly balanced (sleep placebo n = 57, sleep caffeine n = 68, sleep-deprived placebo n = 63, and sleep-deprived caffeine n = 66). Averaged across conditions, the PVT lapse rate for this group was M = .081 (SD = .070), the UNRAVEL post-interruption error rate was M = .192 (SD = .190), and the UNRAVEL non-interruption error rate was M = .034 (SD = .085).
Estimated mean event rates by condition are reported in Table 6. The overall pattern is that sleep deprivation impaired performance on both tasks, and caffeine improved performance on the PVT but had mixed effects on UNRAVEL. For the comparison of PVT lapses with UNRAVEL post-interruption errors, the main effect of condition was significant, F(1, 248) = 48.4, p < .001, ηp^2^ = .163, indicating that sleep deprivation impaired performance. The main effect of pill was significant, F(1, 248) = 4.3, p = .039, ηp^2^ = .017, indicating that caffeine improved performance. There was no main effect of task, F < 1, indicating that lapses in the PVT occurred at about the same rate as post-interruption errors in UNRAVEL. The condition x pill interaction was not significant, F(1, 248) = 1.5, p = .226, ηp^2^ = .006, suggesting that the improvement caused by caffeine did not differ for the sleep-deprived and sleep groups. The condition x task interaction was not significant, F(1, 248) = 1.2, p = .276, ηp^2^ = .005, indicating that the impairment caused by sleep deprivation did not differ by task. The pill x task interaction was not significant, F(1, 248) = 1.3, p = .249, ηp^2^ = .005, suggesting that the improvement caused by caffeine did not differ by task. The condition x pill x task interaction was not significant, F < 1.
For the comparison of PVT lapses with UNRAVEL non-interruption errors, the main effect of condition was significant, F(1, 248) = 49.2, p < .001, ηp^2^ = .165, indicating that sleep deprivation impaired performance. The main effect of pill was significant, F(1, 248) = 4.6, p = .033, ηp^2^ = .018, indicating that caffeine improved performance. There was a main effect of task, F(1, 248) = 22.2, p < .001, ηp^2^ = .082, indicating that lapses in the PVT were more frequent than non-interruption errors in UNRAVEL. The condition x pill interaction was not significant, F(1, 248) = 1.2, p = .270, ηp^2^ = .005, suggesting that the improvement caused by caffeine did not differ for the sleep-deprived and sleep groups. The condition x task interaction was not significant, F(1, 248) = 2.6, p = .108, ηp^2^ = .010, suggesting that the impairment caused by sleep deprivation did not differ by task. The pill x task interaction was significant, F(1, 248) = 4.9, p = .028, ηp^2^ = .019, indicating that the improvement caused by caffeine differed by task. The condition x pill x task interaction was not significant, F < 1.
To probe the pill x task interaction, we performed separate condition x pill ANCOVAs for each task. For the PVT, there were main effects of condition, F(1, 249) = 50.1, p < .001, ηp^2^ = .170, and pill, F(1, 249) = 11.7, p = .001, ηp^2^ = .045. The condition x pill interaction was not significant, F(1, 249) = 2.6, p = .108, ηp^2^ = .010, suggesting that caffeine effects did not differ for the sleep-deprived and sleep groups. For non-interruption errors in UNRAVEL, the main effect of condition was significant, F(1, 249) = 14.0, p < .001, ηp^2^ = .053, but the main effect of pill was not, F < 1. The condition x pill interaction was not significant, F < 1.
These results indicate that caffeine still affected PVT performance after excluding the worst PVT performers—in contrast with Analysis 2, where caffeine no longer affected UNRAVEL performance after excluding the worst UNRAVEL performers. Thus, the effect of caffeine was more robust for PVT performance than for UNRAVEL performance.
We tested the proposal that sleep deprivation impairs vigilant attention and impairs performance of other tasks because they require vigilant attention (Balkin et al., 2008; Doran et al., 2001; Lim & Dinges, 2010). Our approach was to ask whether an intervention known to mitigate effects of sleep deprivation on vigilant attention, caffeine, also does so for placekeeping, a cognitive control process that supports linear thinking in procedural and other higher-order tasks. We measured visual vigilant attention using the PVT and placekeeping using the UNRAVEL task.
We found that sleep deprivation impaired performance on both tasks but caffeine affected performance selectively. Specifically, caffeine improved performance on both tasks in the full sample (Analysis 1), but its effect on UNRAVEL was driven by a small number of poor performers, namely the 8% who were not following task instructions. When we excluded these participants, caffeine did not affect performance on the UNRAVEL task (Analysis 2). When we excluded the same number of the poorest PVT performers, caffeine still affected performance on the PVT (Analysis 3). We conclude that caffeine selectively improved visual vigilant attention but not, for the large majority of participants, the cognitive processes required for placekeeping.
These results converge with those from a previous study in which we asked whether individual differences in vigilant attention fully statistically mediate individual differences in the effects of sleep deprivation on placekeeping (Stepan et al., 2020). In that study, we analyzed the between-participant variability in the placebo group of the present study, examining its systematic components rather than treating it as error variance as we do here. We found that deficits in PVT performance explained some, but not all, variability in UNRAVEL performance. That finding was not consistent with a strong interpretation of the vigilance hypothesis, in which the causal path from sleep deprivation to performance deficits runs strictly through vigilant attention. That finding was consistent with the neuropsychological hypothesis, suggesting that sleep deprivation causes specific deficits to an array of cognitive processes, and those deficits cannot be solely explained by attention failures.
The finding of Stepan et al. (2020) was also consistent with a weaker interpretation of the vigilance hypothesis in which vigilant attention explains most, but not necessarily all, variability in sleep deprivation effects. However, the present results are not consistent with even this weaker interpretation, at least for those participants who passed the morning UNRAVEL session. For those participants, even though their performance was impaired by sleep deprivation, this impairment was not detectably counteracted by improved vigilant attention.
Caffeine did significantly reduce the proportion of sleep-deprived participants who failed the morning UNRAVEL session (though not to the level of the sleep group). Improved vigilance may have played a role in this effect, although the mechanism of action is unclear, given that improved vigilance did not affect UNRAVEL performance for the majority of participants. Sleep deprivation blunts responses to feedback (Whitney, Hinson, Jackson, & Van Dongen, 2015), so one possibility is that the participants who were most strongly affected by sleep deprivation did not register the feedback asking them to be more accurate when they failed a block. Improved vigilance due to caffeine may have mitigated this effect, leading a few additional participants to pay attention to feedback and adjust their performance accordingly.
A related question is whether caffeine affected the ability to perform at criterion accuracy, perhaps through improved feedback processing, or the willingness to do so. Consistent with either possibility, moderate doses of caffeine (under 200 mg) can increase motivation as well as cognitive factors such as concentration (Garrett & Griffiths, 1997). The mechanisms behind this selective effect of caffeine for those most affected by sleep deprivation will be important to study in future work. In the interim, this particular selectivity may help explain some of the mixed effects of caffeine we surveyed in the introduction, if a small proportion of the most-affected participants drove an effect in some studies but not others. Moreover, this possibility is more likely with small samples than with large ones.
Selective effects of caffeine for some tasks but not others are consistent with studies that have simulated effects of sleep deprivation on performance using cognitive architectures. Simulations developed within a cognitive architecture share common mechanisms that allow for precise comparison of cognitive processing across tasks. Simulations of sleep deprivation effects suggest that the mechanisms responsible for impairments on the PVT (Gunzelmann, Gross, Gluck, & Dinges, 2009) are distinct from those responsible for impairments of declarative memory (Gunzelmann et al., 2007). Declarative memory does not play an obvious role in vigilant attention but plays a central role in placekeeping, in that it stores episodic items representing past performance and semantic items representing the step sequence (Altmann & Trafton, 2015; Altmann et al., 2017). Thus, this simulation work suggests that sleep deprivation may have acted on PVT and UNRAVEL performance through different mechanisms, not all of which are necessarily affected by caffeine.
One candidate target of caffeine effects in the PVT model of Gunzelmann and colleagues (2009) is the process that causes the system to transition temporarily into the sleep state when there is little or nothing to do. In the model, sleep deprivation reduces the value of cognitive operations. If no operation is valuable enough to perform on a given system cycle, then no operation is performed, and the effective value of cognitive operations is further reduced on the next cycle. Thus, sleep deprivation increases the probability of a positive feedback loop that, once triggered, drives the system toward a sleep state.
In terms of this model, caffeine may counteract the effect of sleep deprivation by effectively increasing the value that the system assigns to cognitive operations. Performance on the PVT should be particularly sensitive to any factor that changes the value of cognitive operations, up or down, because there are few relevant operations to begin with, and none are relevant while the system is waiting for something to happen. Performance on tasks in which there is more to do should be less sensitive to such factors, as other researchers have also suggested (Pilcher, Band, Odle-Dusseau, & Muth, 2007). This view of selective caffeine effects seems broadly consistent with their underlying neurobiology. Sleep deprivation causes buildup of extracellular adenosine, which is a source of sleep pressure (Basheer, Strecker, Thakkar, & McCarley, 2004; Huang, Urade, & Hayaishi, 2011; Porkka-Heiskanen, Strecker, & McCarley, 2000; Scammell, 2001; Strecker et al., 2000). Caffeine is an adenosine antagonist (Fredholm et al., 1999; Huang et al., 2011), reducing sleep pressure. Any factor that modulates sleep pressure should be more likely to affect performance if an operations “vacuum” occurs regularly in the task environment, as it does in vigilance tasks.
In terms of effects of caffeine on visual vigilant attention, our results extend prior work in two ways. First, we found that caffeine mitigated effects of sleep deprivation to the point where sleep-deprived performance did not significantly differ from sleep non-caffeinated performance (see Analysis 2). Thus, for at least some tasks, caffeine can be an effective intervention, restoring performance to a level that can be considered normal. Second, we found that caffeine improved performance for individuals who had slept, as well as sleep-deprived individuals. This finding may mean that caffeine enhances visual vigilant attention generally, independent of effects of sleep deprivation. However, our sleep group was arguably mildly sleep restricted, having obtained an average of only 5 hrs 42 min of sleep before the morning session (Table 2). Therefore, caffeine may simply have mitigated effects of mild sleep restriction in the sleep group. The latter explanation is consistent with the adenosine-antagonist action of caffeine, given that adenosine buildup is associated with sleep loss.
In terms of practical implications, the vigilance hypothesis implies that interventions that improve vigilant attention can be administered in many different settings with confidence that they would improve performance on a wide range of measures. In contrast, our results suggest that whether a given intervention improves performance on a specific task is an empirical question, not necessarily informed by whether the intervention improves vigilant attention. An implication for intervention research is that successful interventions may need to be tailored to specific cognitive or neurobiological mechanisms affected by sleep deprivation. For occupational settings, our results suggest that where sleep deprivation and tasks requiring placekeeping (such as procedures) are prevalent and placekeeping errors are costly, caffeine is unlikely to provide much benefit beyond a small proportion of individuals for whom sleep deprivation effects are especially strong.
Despite the overall strengths of this work, there are several limitations. As we noted above, our sleep control group was arguably mildly sleep restricted on the morning of the study. Although this is a limitation, it may also suggest that our effect sizes underestimate the impairments caused by sleep deprivation and does not necessarily compromise external validity given that sleep restriction is a common condition societally. Participants slept at home rather than in the laboratory, so we could not control sleep environment or the timing of lights-out; this was a trade-off we made in order to collect a larger sample. The timing parameters for the version of the PVT we used (Wilkinson & Houghton, 1982) are slightly different from published standards (e.g. Basner & Dinges, 2011), complicating comparisons with existing studies. Finally, we did not control or objectively assess compliance with the instruction to refrain from drug and alcohol use for 24 hr prior to the start of the study and for the duration of the study. Although there is no reason to think that failure to comply (and associated false reporting) differed across groups, we cannot rule out the possibility of interactions between non-compliant drug or alcohol use on one hand and sleep deprivation or caffeine administration on the other.
In summary, we found that caffeine mitigated negative effects of sleep deprivation on visual vigilant attention but had no effect on placekeeping, beyond a small subsample performing at the margins of criterion accuracy. Our results are evidence that at least some effects of sleep deprivation are not caused by attention deficits and thus may not be counteracted by interventions that improve attention. In future work, it will be important to ask what other processes might be affected directly by sleep deprivation to develop a more complete understanding of the direct and indirect effects of sleep deprivation on cognitive processes.