Authors: Kaiyue Wang, Zhuyuan Jiang, Suqi Huang, Jiehui Qian
Categories: Article, binocular disparity, depth perception, feature binding, working memory
Source: Journal of Vision
Doi: 10.1167/jov.21.7.8
For decades, working memory (WM) has been a heated research topic in the field of cognitive psychology. However, most studies on WM presented visual stimuli on a two-dimensional plane, rarely involving depth perception. Several previous studies have investigated how depth information is stored in WM, and found that WM for depth is even more limited in capacity and the memory performance is poor compared to visual WM. In the present study, we used a change detection task to investigate whether dissociating memory items by different visual features, thereby to increase their perceptual separateness, can improve WM performance for depth. Memory items presented at various depth planes were bound with different colors (Experiments 1 and 3) or sizes (Experiment 2). The memory performance for depth locations of visual stimuli with homogeneous and heterogeneous appearances were tested and compared. The results showed a consistent pattern that although separating items with various feature values did not affect the overall memory performance, the manipulation significantly improved memory performance for the middle depth locations but impaired the performance for the boundary locations when observers fixated at the center of the whole depth volume. The memory benefits of feature separation can be attributed to enhanced individuation of memory items, therefore facilitating a more balanced allocation of attention and memory resources.
Keywords: working memory, depth perception, binocular disparity, feature binding
Working memory (WM) is deemed as a limited-capacity memory system that can temporarily process and maintain information. It is suggested to serve as a link among sensory perception, long-term memory, and motor action, and therefore plays an important role in supporting other higher cognitive functions (Baddeley, 2003; Baddeley, 2012). Although WM has long been a heated research topic in the field of cognitive psychology, most studies focused on WM that is responsible for holding visual or verbal information in a 2D context, and fewer investigated WM involving depth information (Qian & Zhang, 2019; Qian, Li, Zhang, & Lei, 2020; Reeves & Lei, 2017; Zhang, Gao, & Qian, 2020).
Studies on visual WM (vWM) have shown that its storage capacity is severely limited and only up to four items can be stored (Awh, Barton, & Vogel, 2007; Cowan, 2001; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Zhang & Luck, 2008). These studies often used a change detection paradigm – memory stimuli, such as colors, shapes, or letters, were briefly presented and observers needed to judge whether the test stimulus were different from the memorized one after various retention intervals. Neuro- and electro-physiological studies provide supportive evidence for the limited capacity of vWM (Anderson, Vogel, & Awh, 2011; Vogel & Machizawa, 2004; Xu & Chun, 2006). Research also suggests that cognitive factors, such as attention (Griffin & Nobre, 2003; Murray, Nobre, Clark, Cravo, & Stokes, 2013; Qian, Zhang, Lei, Han, & Li, 2020), familiarity (Olson & Poom, 2005), spatial configuration (Li, Qian, & Liang, 2018; Jiang, Makovski, & Shim, 2008; Jiang, Olson, & Chun, 2000), and long-term memory (Brady, Störmerb, & Alvarez, 2016), may help to overcome the capacity limitations of vWM. In addition, several studies have shown that when memory items were separated in different depth surfaces or planes, vWM performance was improved (Chunharas, Rademaker, Sprague, Brady, & Serences, 2019; Xu & Nakayama, 2007), especially for those objects closer to the observer (Qian, Li, Wang, Liu, & Lei, 2017; Qian, Zhang, Wang, Li, & Lei, 2018).
Recently, emerging studies investigated the nature of memory representations for depth information (e.g. Qian et al., 2020; Reeves & Lei, 2017), and found that the accuracy for holding one depth position in a change detection task was consistently below 80% and even further decreased with memory load (Qian & Zhang, 2019). Compared to vWM, whose capacity is suggested to be about four for colors (e.g. Cowan, 2001) and lower for more complex stimuli (e.g. Alvarez & Cavanagh, 2004; note that some suggested vWM capacity can be down to one in some cases, see Olsson & Poom, 2005), working memory for depth (WMd) seems to have much more limited capacity, as the memory performance for depth remains low even when changes in depth is very prominent to detect (Qian et al., 2020). Despite the overall low change detection accuracy, previous study has shown that memory performance for the boundary depth locations is better than that for the fixated center depth locations (Qian & Zhang, 2019), which might be attributed to the differences in visual processing of items at these two locations. One possibility is that because the boundary depth locations can be deemed as “anchor” and serve to better localize the other depth locations, they are more likely to be prioritized for attention and processing. Indeed, research shows that the identification of an object's depth location could be promoted when a farther object was displayed (Sousa, Brenner, & Smeets, 2011) and perceptual and memory performance could be enhanced when target item was presented at the nearest depth plane (Plewan & Rinkenauer, 2020; Qian et al., 2017; Qian et al., 2018), which all indicate the special role of boundary depth locations. Moreover, because we are apt to attend to objects in depth serially rather than to simultaneously spread attention across multiple depths (He & Nakayama, 1995), the prioritization of boundary processing results in the “boundary advantage.” Another possibility is that attention simultaneously distributed across depth planes, and each plane interferes with its neighbors on both sides while the boundary items have only one neighbor and thus are interfered with less, resulting in better performance for them. Regardless of the serial or parallel mechanism of attentional processing, the boundary advantage seems to be robust in WMd.
On the other hand, the surprisingly low accuracy found for WMd with low memory load may be due to one (or some) of the following inaccurate perception, poor transfer to memory, poor maintenance in memory, and failures in retrieval from memory. It is unclear which specific process can be accounted for the low memory performance for depth, although some researchers suggested that poor maintenance is a probable account (e.g. Qian et al., 2020). On the other hand, because past research has shown that our perception for depth is fairly accurate (e.g. the stereoacuity threshold can be less than 10 arcsec; McKee & Taylor, 2010), the low memory performance is unlikely to be due to limited perceptual precision for one depth location. However, for multiple depth locations, it may take longer for perceiving and transferring to memory with certain precision. In other words, the poor performance for WMd may be attributed to the insufficient encoding of multiple depth locations with limited time. Indeed, researchers suggested that when multiple depth planes are presented, attention is likely to be spread over a larger area in depth (Finlayson & Grove, 2015). Spreading attention laterally degrades processing efficiency for each location within the attended region (White & Carrasco, 2011; Eriksen & James, 1986), and this may also apply when attention is spread out in depth.
If presenting items in separate depth planes facilitates the memory performance for their visual features (Chunharas, Rademaker, Sprague, Brady, & Serences, 2019; Qian et al., 2017; Qian et al., 2018), one may suspect if the reverse is true – whether separating items by different visual features could improve the memory performance for their depth locations. The memory benefits of depth separation may arise from a process of reducing neural competition by utilizing cortically separable resources (Chunharas et al., 2019), and thus facilitating the encoding of visual features (Nakayama & Silverman, 1986). Conversely, it is possible that presenting items that each has distinct feature values (past research on WMd used memory items with homogeneous appearance; Qian et al., 2020; Qian & Zhang, 2019; Zhang, Gao, & Qian, 2020) may enhance their perceptual separateness, and the memory performance for their depth locations may be further improved. Indeed, in real life we can almost always memorize the spatial locations of multiple objects that have various colors, shapes, and sizes.
We hypothesize that WMd performance can be improved by assigning distinct feature value (e.g. a different color) to each memory item. The underlying mechanism of improvement may be attributed to the following possibilities. First, it can be more cognitively demanding to encode homogenous visual stimuli and more difficult to retain homogeneous memory representations. Enhancing perceptual separateness may decrease the between-item interferences, and therefore facilitates the overall encoding or maintenance process of depth representations. Second, because the previously observed boundary advantage indicates that boundary locations are primarily attended and processed and the less distinct intermediate depth locations are likely to be obscured, enhancing perceptual separateness may help observers to better individualize the items at the intermediate depth locations. As a result, attention may be re-allocated from boundaries to the center, and memory performance can be improved for the center locations and weakened for the boundaries. In other words, an overall improvement in memory performance indicates that enhancing perceptual separateness could reduce the between-item interference during encoding or retention, whereas an improvement only for the center depth locations indicates a more balanced allocation of attention and memory resources due to better individuation.
In this study, we used a change detection task to investigate the effect of increasing perceptual separateness on WMd. The participants were asked to detect any change in depth location between briefly presented memory items and a test item after a period of retention. Each memory item was bound with a distinct color (Experiments 1 and 3) or size (Experiment 2). By comparing with the control experiment where memory items were homogeneous in feature, we investigated whether there were benefits for memorizing depth locations by individualizing items with various visual feature values. In addition, location of fixation was also manipulated to test the robustness of outstanding performance for boundary locations (Experiment 3). Because previous study suggests that allocation of attention in 3D space can be modulated by experimental settings (Plewan & Rinkenauer, 2020), the result patterns corresponded to different fixation locations may indicate the specific attentional process involved and thus to clarify its underlying mechanism.
In Experiment 1, we investigated the effect of increasing perceptual separateness on WMd by binding depth with color. The memory performance for depth locations of visual stimuli with homogeneous and heterogeneous appearances were tested and compared. In the experimental group, the memory items were squares with highly distinctive colors; in the control group, the memory squares had a homogeneous color of blue. We expected to observe improvements for the experimental group.
Fifty-nine participants from Sun Yat-Sen University (SYSU) with normal vision or corrected-to-normal vision were recruited for payments. All participants were required to pass a screening test before the formal experiment to ensure that they could accurately perceive the disparity-defined depth (see Qian & Zhang, 2019 for details of the screening test). A total of 30 participants (8 male subjects; mean age = 22.5 years) passed the screening test and took part in the formal experiment. This study (including Experiments 1, 2, and 3) has been approved by the Institutional Review Board (IRB) of SYSU Department of Psychology. Written informed consent approved by the IRB was obtained from each participant prior to all of the experiments.
The participants viewed the stimuli presented on the gray background (102 cd/m^2^) through a Wheatstone stereoscope on a pair of 21 inch ViewSonic monitors. The display resolution was set to 1920 × 1080 pixels and the refresh rate was 60Hz. The viewing distance was 75 cm.
The memory array was composed of a set of squares that were arranged in a circular configuration with a radius of 3.5 degrees from the center of the screen (see Figure 1). There were either two (set size 2) or four memory items (set size 4) presented. Any two nearest neighboring items were separated by 3.5 degrees. Each square occupied a depth plane, which was randomly selected from seven depth planes without replacement on every trial. The seven depth planes (target depth) were separated by relative disparities, ranging from −0.51 degrees to +0.51 degrees with a step of 0.17 degrees. These disparities were selected to ensure the reliable fusion of left- and right-eye images, so that the items appeared to be clearly separated in depth (Blakemore, 1970). Each square subtended approximately 0.80°× 0.80 degrees of visual angle. In the multicolor condition, the color of each memory item was randomly selected from six different colors on every red, green, yellow, blue, cyan, and magenta. In the control (single color) condition, the memory items were all blue (25.8 cd/m^2^).
Figure 1. Stimuli and procedure in the experiment. Top: task sequence. Bottom: the side view of the memory display in the multicolor condition. The stimuli were arranged in a circular configuration, and different types of lines indicated that they were presented at different depths. No line or circle was presented in the actual experiment.
The participants were seated in a dark room to complete the experiment. They received a short period of training (2–5 minutes) to familiarize themselves with the stimuli and tasks. At the beginning of the experiment, they needed to first confirm that the left- and right-eye images (red cross with a size of 0.65 degrees × 0.65 degrees) could be fused smoothly to form depth perception in an image fusion stage, by pressing a key on the keyboard. Each trial began with a black fixation cross presented for 400 ms at the center of the screen. Then, the memory array was presented for 800 ms, following a blank screen for 900 ms. In the test stage, a test item was presented until response. There was a blank screen interval of 1000 ms between each trial. The task sequence is shown in Figure 1.
The participants were asked to judge whether the depth position of the test item was different from the memory item at the same 2D location, by pressing a key to indicate that they were different and pressing another key to indicate the same. In half of the trials, the depth positions of two were different. The test item would randomly appear in a new depth plane that had not been previously occupied by any memory item on that trial. There were an equal number of trials for each set size and for each depth plane that was selected to be tested (target depth). Half of the participants ran the multicolor condition, and the other half ran the control condition. Each participant needed to complete 560 trials, including 40 trials for each set size and target depth condition. All trials were randomized in order and were inter-mixed within a block.
The performance of change detection task was assessed by detection sensitivity (d’) and the response criterion (β) based on hit and false alarm rates (see Supplementary Materials). A 2 × 7 × 2 (set size × target depth × group) mixed-design ANOVA was used to analyze d’ and β separately. The same analysis was also performed on change detection accuracy (percent of correct), these results are reported in Supplementary Materials as they were consistent with the results of d’. Greenhouse-Geisser correction was used if the spherical assumption was violated. Bonferroni adjustment was used for pairwise comparisons to correct for type I errors.
d’. The results are shown in Figure 2. ANOVA showed that the main effect of set size was significant, F(1,28) = 48.52, p < 0.001, ηp^2^ = 0.63, and d’ decreased as the set size increased. The main effect of target depth was significant, F(6,168) = 9.03, p < 0.001, ηp^2^ = 0.24, and there was a significant quadratic trend that d’ first decreased and then increased as the target depth changed from the nearest (crossed disparity) to the farthest (uncrossed disparity), F(1,28) = 16.65, p < 0.001, ηp^2^ = 0.37. The main effect of group was not significant, F(1,28) = 1.36, p = 0.253, ηp^2^ = 0.05. The interaction between target depth and group was significant, F(6,168) = 2.32, p = 0.036, ηp^2^ = 0.08. The interaction between set size and target depth was not significant, F(6,168) = 1.81, p = 0.139, ηp^2^ = 0.06. The interaction between set size and group was not significant, F(1,28) = 0.36, p = 0.556, ηp^2^ = 0.01. The three-way interaction was not significant, F(6,168) =2.018, p = 0.067, ηp^2^ = 0.07.
Figure 2. Results of d’ in Experiment 1. (A) Comparison of d’ between the multicolor group and the control group as a function of set size; data averaged across target depth. (B) Comparison of d’ between the two groups as a function of target depth; data averaged across set size. (C) For a set size of 2, d’ as a function of target depth. (D) For a set size of 4, d’ as a function of target depth. The error bar indicates one standard error here and in other figures.
The simple effect analysis was carried out to test the significant interaction between target depth and group, which showed that there was no significant difference in d’ between the multicolor and control groups for any target −0.51 degrees, t(28) = −0.49, p = 0.627, Cohen's d = 0.19; −0.34 degrees, t(28) = 0.07, p = 0.943, Cohen's d = 0.03; −0.17 degrees, t(28) = 0.58, p = 0.569, Cohen's d = 0.22; 0 degrees, t(28) = 2.05, p = 0.082, Cohen's d = 0.77; +0.17 degrees, t(28) = 1.91, p = 0.073, Cohen's d = 0.72; +0.34 degrees, t(28) = 1.34, p = 0.191, Cohen's d = 0.51; +0.51 degrees, and t(28) = 1.50, p = 0.144, Cohen's d = 0.57. However, it seems contradictory that the interaction between target depth and group was significant yet none of the simple effects was significant. Because the difference was marginally significant for target depth of 0 degrees and +0.17 degrees, we think that there might be a weak effect for 0 degrees and +0.17 degrees, and it was possible that significant difference only occurred for the joint effect of these two target depths. To provide a better understanding of the seemingly contradictory statistical results, here we performed a follow-up analysis that combined the data for 0 degrees and +0.17 degrees. The results showed that the mean d’ was significantly larger for the multi-color group than for the control group, t(28) = 2.07, p = 0.048, Cohen's d = 0.78.
β. The results are shown in Figure 3. The results showed that the main effect of set size was significant, F(1,28) = 6.15, p = 0.019, ηp^2^ = 0.18. β decreased with set size, showing that the participants were less conservative with larger memory load. None of the other effects was target depth, F(6,168) = 1.03, p = 0.385, ηp^2^ = 0.04; group, F(1,28) = 0.48, p = 0.496, ηp^2^ = 0.02; set size × target depth, F(6,168) = 1.25, p = 0.284, ηp^2^ = 0.04; set size × group, F(1,28) = 0.02, p = 0.884, ηp^2^ = 0.001; target depth × group, F(6,168) = 0.99, p = 0.435, ηp^2^ = 0.03; set size × target depth × group, F(6,168) = 2.89, p = 0.059, ηp^2^ = 0.07.
Figure 3. Results of β in Experiment 1. (A) Comparison of β between the multicolor group and the control group as a function of set size; data averaged across target depth. (B) Comparison of β between the two groups as a function of target depth; data averaged across set size.
The results for the control group were consistent with our previous findings (Qian & Zhang, 2019), which showed that detection sensitivity was lowest around the middle zero-disparity (fixation) plane and gradually increased as depth planes became nearer or farther. By comparing the task performance between the multicolor and control groups, we found that for most of the depth planes, coloring the memory items with distinctive feature values does not make a significant difference in the memory performance. This indicated that color separation did not result in an overall enhancement of the encoding process by reducing competition of resource, or facilitating maintenance by preventing rapid decay. Only for the depth planes of 0 degrees and +0.17 degrees, memory performance was significantly improved (however, note for a slight drop in performance at −0.51 degrees at set size 4 for the multi-color group), suggesting that color separation may facilitate individuation of the memory items, which further helps to re-allocate attention across depth planes in a more balanced way.
In addition, the mean β was greater than one under all conditions, indicating that the participants tended to make a conservative response of “no change,” but this bias became smaller as memory load increased. These results were consistent with Qian and Zhang (2019), and perceptual separateness did not affect the pattern of response criteria for detecting depth changes. To simplify the Results section, the results of β in Experiments 2 and 3 were reported in Supplementary Materials.
Previous studies have demonstrated that there is asymmetric collateral binding between visual features (such as color, shape, letter, etc.) and 2D spatial locations – when the task involves judging visual features, observers seem to automatically encode the 2D spatial information to bind with the corresponding visual features, even though the former is task-irrelevant; when the task is to respond to the 2D spatial locations, observers only encode and store the spatial location information and tend to ignore the visual feature information that is task-irrelevant (Elsley & Parmentier, 2015; Guérard, Morey, Lagacé, & Tremblay, 2013; Jiang et al., 2000; Kondo & Saiki, 2012; Logie, Brockmole, & Jaswal, 2011). Because depth can be also deemed as a type of spatial location, the lack of significant effect of color separation on the overall memory performance might be due to the binding between depth and color being weak in a memory task for depth, or due to the colors being ignored when depth is to be reported.
To rule out this possibility, we investigated the effect of increasing perceptual separateness on WMd by binding depth with size in Experiment 2. The memory performance for depth locations of items with various sizes and items with a homogeneous size was compared. It is known that relative size is a powerful monocular depth cue. Studies show that size and depth are interdependent, manipulating either one would significantly affect the perception of the other (Qian & Petrov, 2013; Qian, Liu, & Lei, 2016; Zhang, Qian, Liang, & Huang, 2018) and the binding between depth and size is considered to be automatic (Markov, Tiurina, & Utochkin, 2019). Therefore, one may suspect that binding depth with size would produce a memory benefit greater than binding depth with color.
Here, various sizes were randomly bound with depth locations, therefore it is impossible for observers to perceive size as a reliable depth cue nor introducing cue conflicts with disparity. We tested size separation in this experiment and further compared it with the effect of color separation in Experiment 1 to explore whether visual features that differed in their binding efficiency with depth could produce different separation effect on WMd.
Another 16 participants with normal or corrected-to-normal vision from SYSU were recruited. A total of 15 participants (4 male subjects; mean age = 23.1 years) successfully passed the screening test and took part in the formal experiment. Written informed consent approved by the IRB was obtained from each participant prior to the experiment.
The stimuli and experimental procedures were similar to Experiment 1, except that each memory item was associated with a unique size instead of color (see Figure 4^4^). Each participant ran an experimental condition of multi-size. In this condition, the size of blue squares was randomly selected from six different 0.60 degrees × 0.60 degrees, 0.75 degrees × 0.75 degrees, 0.90 degrees × 0.90 degrees, 1.05 degrees × 1.05 degrees, 1.30 degrees × 1.30 degrees, and 1.45 degrees × 1.45 degrees. For the control (single-size) group, we used the data collected from the control group in Experiment 1, in which the size of all blue squares was identical. In addition, we only used a set size of four, because the effect of set size was not of our primary interests and it did not interact with other effects in Experiment 1. The participants were asked to judge whether the depth position of the test item was different from the memory item at the same 2D location, by pressing one of the two keys on the keyboard. Each participant completed a total of 280 trials, including 40 trials for each target depth condition.
Figure 4. Stimuli and procedure of Experiment 2. Each memory item was associated with a unique size. Different types of lines indicated that they were presented at different depths. No line or circle was presented in the actual experiment.
A 7 × 2 (target depth × group) mixed-design ANOVA was used to analyze detection sensitivity, d’ (see Supplementary Materials for hit and false alarm rates, the results of change detection accuracy, and response bias, β). To further compare the effect of color separation and size separation, we ran an additional 7 × 2 (multicolor versus multisize group) mixed-design ANOVA. Only data from set size of four in Experiment 1 was used for this analysis. Greenhouse-Geisser correction was used if the spherical assumption was violated. Bonferroni adjustment was used for pairwise comparisons to correct for type I errors.
d’. The results are shown in Figure 5. ANOVA showed that the main effect of target depth was significant, F(6,168) = 3.69, p = 0.008, ηp^2^ = 0.12, and there was a significant quadratic trend that d’ first decreased and then increased with target depth. The main effect of group was not significant, F(1,28) = 0.41, p = 0.527, ηp^2^ = 0.01. The interaction between target depth and group was significant, F(6,168) = 5.21, p < 0.001, ηp^2^ = 0.16.
Figure 5. The results of d’ in Experiment 2 (A) and a comparison between the multicolor group in Experiment 1 and the multisize group in Experiment 2 (B).
The simple effect analysis for testing the significant interaction between group and target depth showed that the task performance of the two groups were not significantly different (p values > 0.10), except for target depth 0 degrees, t(28) = 2.06, p = 0.049, *Cohen’*s d = 0.79, and +0.17 degrees, t(28) = 2.63, p = 0.014, *Cohen’*s d = 0.99.
Comparison between color separation and size separation. ANOVA showed that the main effect of target depth was significant, F(6,168) = 5.32, p = 0.001, ηp^2^ = 0.16. The main effect of group was not significant, F(1,28) = 0.05, p = 0.830, ηp^2^ < 0.01. The interaction between target depth and group was not significant, F(6,168) = 0.83, p = 0.546, ηp^2^ = 0.03.
Consistent with the result of Experiment 1, we found that separating memory items with various sizes did not significantly improve the overall performance and the memory performance was only improved at depth planes of 0 degrees and +0.17 degrees (also note for a slight but non-significant drop in performance at −0.51 degrees for the multisize group, consistent with Experiment 1). Although perceived size significantly modulates depth perception (Landy, Maloney, Johnston, & Young, 1995), and the binding between depth and size is considered to be automatic (Markov, Tiurina, & Utochkin, 2019) and stronger than the binding between depth and color, there was no significant difference in memory performance between Experiments 1 and 2, suggesting that increasing perceptual separateness either by color or size produced similar effects on WMd. This was probably due to the various sizes that were randomly bound with depth locations in our study, therefore size here was merely perceived as an ordinary feature dimension, rather than a powerful cue that associated with depth processing. In other words, separating items with various visual features produces consistent effects on memory performance for depth, regardless of their different binding strength with depth.
Although the improvements at depth planes of 0 degrees and +0.17 degrees were consistently observed in Experiments 1 and 2, there are three possible confounds needed to be clarified. First, because the fixation was always presented at the depth plane of 0 degrees, which coincided with the middle of the whole depth volume in our experimental settings, it is unclear whether the improvements at 0 degrees and +0.17 degrees was due to the fact that they were around the middle depth or around the fixation. As previous studies involved multiple depths mostly set the fixation location to the center of whole depth volume (e.g. Finlayson & Grove, 2015; Plewan & Rinkenauer, 2017; Theeuwes & Pratt, 2003), it is unclear how fixation location affects the allocation of attention in depth. Second, because Experiments 1 and 2 used a between-subject design and both used the same control group for comparison, it is possible that the improvements at 0 degrees and +0.17 degrees was due to some random errors like unfortunate sampling for the control group. Third, since verbal memory was not controlled in Experiments 1 and 2, participants might be able to use a strategy to verbally rehearse the depth position of an item by associating the location with a specific feature value (e.g. “green square in middle”). This might have enhanced the performance for the heterogeneous display.
To rule out these possibilities, in Experiment 3, we manipulated the plane of fixation (near versus middle versus far) to test whether it affects the memory performance for WMd and used a verbal suppression task to prevent rehearsing during retention. A within-subject design was employed to better control for individual difference. We aimed to replicate Experiment 1 in the middle fixation condition, and the results in the near (far) fixation condition would clarify whether the plane of fixation affected the improvements – the possibility can be ruled out if there was no improvement for the heterogeneous display at the near (far) depth planes.
Fifteen out of 20 (3 male subjects; mean age = 22.9 years) participants successfully passed the screening test and took part in the formal experiment. Written informed consent approved by the IRB was obtained from each participant prior to the experiment.
The stimuli and experimental procedures were similar to Experiment 1 (set size 4), except for the following changes. First, three random meaningless letters (e.g. ACG) were presented at the beginning of each trial and the participants were instructed to repeat aloud throughout the trial. Second, the fixation was presented in the near (−0.51 degrees), the middle (0 degrees), and the far (+0.51 degrees) depth planes in separate experimental blocks. Participants needed to complete six blocks (3 fixation planes × 2 single color and multicolor) of 840 trials in total, with 20 trials for each target depth in each block. The order of the six blocks was counterbalanced across participants.
A 3 × 7 × 2 (fixation plane × target depth × display) repeated measures ANOVA was used to analyze detection sensitivity, d’ (see Supplementary Materials for results of change detection accuracy and response bias, β). Greenhouse-Geisser correction was used if the spherical assumption was violated. Bonferroni adjustment was used for pairwise comparisons to correct for type I errors.
d’. The results of ANOVA showed that the main effect of fixation plane was significant, F (2, 28) = 13.69, p < 0.001, ηp^2^ = 0.49. d’ for the middle fixation was larger than that for both the near (Mmiddle~~-near = 0.43, SE = 0.09, p = 0.001) and the far fixation (Mmiddle~~-far = 0.36, SE = 0.08, p = 0.002); there was no significant difference in d’ between the near and far fixations, Mnear~~-far = −0.06, SE = 0.09, p = 1.00. The main effect of target depth was significant, F (6, 168) = 8.13, p < 0.001, ηp^2^ = 0.37, and there was a significant quadratic trend that d’ decreased and then increased with target depth, F (1, 14) = 41.71, p < 0.001, ηp^2^ = 0.75. The main effect of display was not significant, F (1, 14) = 0.20, p = 0.665, ηp^2^ = 0.01. The interaction between fixation plane and target depth was significant, F (12, 168) = 15.24, p < 0.001, ηp^2^ = 0.52. The simple effect analysis showed for the middle fixation, the quadratic trend that d’ first decreased and then increased with target depth was significant, p = 0.001; for the near fixation, there was a significant linear trend that d’ decreased from near to far, p = 0.002; and for the far fixation, there was a significant linear trend that d’ decreased from far to near, p < 0.001. The interaction between fixation plane and display was not significant, F (2, 28) = 1.01, p = 0.377, ηp^2^ = 0.07. The interaction between target depth and display was not significant, F (6, 84) = 1.59, p = 0.202, ηp^2^ = 0.10.
Importantly, the three-way interaction was significant, F (12, 168) = 3.53, p < .0001, ηp^2^ = 0.20. To further interpret the three-way interaction, we performed a 7 (target depth) × 2 (display) repeated measures ANOVA separately for each fixation condition. Results showed that the interaction between target depth and display was significant only for the middle fixation, F (6, 84) = 8.47, p < 0.001, ηp^2^ = 0.38, but for neither the near fixation, F (6, 84) = 0.69, p = 0.661, ηp^2^ = 0.05, nor the far fixation, F (6, 84) = 1.21, p = 0.311, ηp^2^ = 0.08. The simple effect analysis for the middle fixation showed that memory performance for the two displays was significantly different at the target depth of 0 degrees (Mmulti~~-control = 0.63, SE = 0.16, t (14) = 3.91, p = 0.002), +0.17 degrees (Mmulti~~-control = 0.62, SE = 0.12, t (14) = 5.41, p < 0.001), and +0.51 degrees (Mmulti~~-control = −0.46, SE = 0.18, t (14) = −2.65, p = 0.019). The results are shown in Figure 6.
Figure 6. The results of d’ in Experiment 3. (A) Comparison of d’ between the two displays as a function of target depth for the middle fixation. (B) Comparison of d’ between the two displays as a function of target depth for the near and the far fixations.
For the middle fixation condition, Experiment 3 again showed significant improvements for the multicolor display at the depth planes of 0 degrees and +0.17 degrees, which is in accordance with the findings in Experiments 1 and 2. These suggested that verbal memory did not contribute to the observed memory benefits for the heterogeneous-feature condition. In addition, we found that the performance at the depth plane of +0.51 degrees was significantly higher in the control condition than in the multi-color condition. Indeed, Figure 6 showed that the U-shaped performance pattern for the single-color display seemed to flatten out for the multicolor display. The significant performance decrease at the depth plane of +0.51 degrees, combined with the observed performance decrease (though not significant) at the boundary location of −0.51 degrees in Experiment 1 (Figure 2D) and 2 (Figure 5A), will be further discussed in the General Discussion.
For the near and far fixation conditions, there was no improvement for the heterogeneous display at the near and far depth planes, indicating that the improvements at 0 degrees and +0.17 degrees cannot be attributed to the effect of fixation plane. This is not due to a ceiling effect, because the mean accuracy for the near fixation was 0.81 (SE = 0.03) and for the far fixation was 0.82 (SE = 0.03), which was far from the perfect performance. However, there was also no improvement around the intermediate depths or around the depth planes with lowest performance for the heterogeneous display in these two conditions – the performance decreased linearly with the distance between the target depth and the fixation plane both for the homogeneous and heterogeneous displays. The possible mechanism underlying the linear trend of memory performance will be discussed in the General Discussion.
In the present study, we used a change detection paradigm to investigate whether increasing the perceptual separateness improves the memory performance for depth. The results showed a consistent pattern that although the overall performance was not affected by color separation (Experiments 1 and 3) and size separation (Experiment 2), there were reliable and significant memory benefits for intermediate depths and possible memory loss at boundary locations with middle fixation. However, memory performance for depth was not affected in the near and far fixation conditions.
In a homogeneous display, we found a linear decrease in memory performance from the fixated edge to farther depth planes (near or far fixation condition), and a U-shaped performance function with higher d’ at the edges and lower d’ at the fixated middle plane (middle fixation condition). The observed differences between the fixation conditions cannot be due to different sensory sensitivity as all the items are equally detectable at every depth plane, given the small range of disparities used in this study. Indeed, for the control condition in Experiment 3, items at the middle (0 degrees) plane were nearly equal in the d’ for the three fixation conditions, implying that the plane of fixation has little or null sensory effect, and the difference in results mainly reflect the effect of attentional allocation as processing and encoding memory items rely on attentional engagement (Kane & Engle, 2000; Miller, Gross, & Unsworth, 2019). We think that attention may play an important role in tasks involving depth perception – it is primarily deployed to the fixated edge or split between the edges when the middle location is fixated. This is because that perception of boundaries helps to enhance the precision of perceiving the intermediate depths (Foley, 1985; Gogel, 1972; Sousa et al., 2011), therefore they are more likely to be prioritized in encoding and processing than the less distinct intermediate depths. Intuitively, memory items with homogeneous appearance may be perceived as one big chunk, and detecting changes at the edges of the chunk can be easier. This could explain the linear decrease in performance in the near and far fixation conditions, that is, attention was first distributed around the fixated edge, and then be directed to other locations following a natural order in proximity. For the middle fixation condition, attention may split between the edges but not to the middle, resulting in worse performance at the intermediate depths.
However, in a heterogeneous display, boundary depth locations may no longer be a dominating factor in the allocation of attention. When fixation is on the middle plane, we consistently found across the three experiments that the memory advantage for the boundary locations diminished and the performance at the intermediate depths improved for the heterogeneous display. Although in Experiments 1 and 2, the decrease in performance at the boundary location (−0.51 degrees) did not reach statistical significance, this is possibly due to large error variances and lower statistical power in a between-subject experimental design. Because the worse memory performance for intermediate depths in a homogeneous display is possibly due to the way that they were more likely to be obscured in a set of apparently uniform memory items, we think that assigning items with distinct feature values may increase the perceptual separateness between items and help the intermediate depth locations to be better individualized. As a result, attention can be re-allocated to the middle and memory resources can be balanced across depth planes. This improves the poor performance for the intermediate depths observed in a homogeneous display, yet at the cost of the boundaries. In other words, increasing perceptual separateness affects memory performance for depth by overcoming the limits of attention allocation.
When fixation is on a boundary plane, the performance for the heterogeneous display was consistent with that for the homogeneous display, which kept to deteriorate as the tested item became away from fixation. This indicates that attention is not re-allocated in this case. We think that two possible explanations may account for the lack of effect. One possibility is that boundary depth locations were already highly distinct, and focusing attention on the fixated edge plane is so obviously beneficial that observers never attempt to re-allocate attention. In other words, the benefits of increasing perceptual separateness only work for depth locations that have lowest memory performance due to a lack of distinctiveness, therefore no improvements were observed for the boundary locations opposite to the fixated edge. Another possibility is that when fixation is on one edge, attention may not be easily diverted to the opposite edge due to the constraints on size of the attentional span in depth. Although studies involving perceptual tasks often indicate the size of attentional focus on 2D frontoparallel plane is apparently larger than one degree of visual angle (e.g. Eriksen & James, 1986), it is unclear how large is the attentional span in stereoscopic depth. Because binocular disparity has a narrow fusion range of slightly above 1 degrees in fovea, we suspect that the size of attentional span is smaller than this range. In our study, although attention could reach to the two boundary locations about half a degree away when fixating on the middle plane, it might not be able to cross the whole depth volume when fixating on one edge. Further exploration in needed to clarify the exact mechanism underlying the lack of effect.
One may suspect that increasing perceptual separateness enhances the visual saliency of items at the intermediate depth locations, and therefore facilitates the encoding for these locations. Although it is hard to tell whether the colors used in our study differed in terms of visual saliency, one might assume that an item with an extreme (largest or smallest) size is more visually salient. In order to test this possibility, we have calculated the mean accuracies of items with the largest size, the smallest size, and the other sizes on each trial in Experiment 2, and conducted a 1-way repeated measures ANOVA to compare whether there were significant differences. The results showed that the main effect of size was not significant, F (2, 28) = 0.21, p = 0.813, ηp^2^ = 0.02, indicating that visual saliency does not affect the performance.
We used set sizes of two and four in Experiment 1. Although there was no significant three-way interaction effect (p = 0.067), we noticed that the mean color separation effect on intermediate depths (0 degrees and + 0.17 degrees) was greater for a set size of four (Mmulti~~-control = 0.64) than for a set size of two (Mmulti~~-control = 0.34). This is consistent with previous studies that showed similar findings of more significant improvements on memory performance for colors separated by depth for a larger set size (Chunharas et al., 2019; Qian et al., 2017). We think that the smaller effect for set size two was due to that the memory task was relatively easier. For a set size of two, only two depth positions were presented and both were equally likely to be prioritized in processing for their “boundary” locations. For a set size of four, depth positions at the middle were more likely to be deprived of memory resources. In other words, the more memory items presented, the fewer memory resources are available for the less distinct intermediate depths. Therefore, when the memory load is high, increasing perceptual separation helps to balance the resources among memory items, and the memory performance for the less distinct locations can be improved. To test this hypothesis, we performed additional analyses to compare d’ separately for the two set sizes to examine the color separation effect. We found that for set size two, the interaction between depth plane and group was not significant (p = 0.435); for set size four, the interaction between depth plane and group was significant (p = 0.003), and simple effect analysis showed that the memory performance was significantly different only at the depth plane of 0 degrees and +0.17 degrees between the multicolor group and the control group, p values < 0.05. These results suggest that increasing perceptual separation produces greater benefits for a larger set size, which is in line with the prediction from the hypothesis that the memory benefit originated from more balanced resource allocation.
There was no overall enhancement of the memory performance for the whole range of depth planes, suggesting that perceptual separation by color or size does not facilitate the overall encoding or retention process. The highest change detection accuracy across the three experiments had an average about 70% for a set size of four, therefore the lack of overall improvement is not due to the ceiling effect. We think that the lack of improvement may be attributed to the weak binding between an object's feature and its depth location. The phenomenon of asymmetric collateral binding between visual features and 2D spatial locations suggests that 2D spatial information is automatically encoded in a task of judging visual features, whereas visual feature information seems to be ignored in a task of judging 2D locations (Elsley & Parmentier, 2015; Guérard, Morey, Lagacé, & Tremblay, 2013; Jiang et al., 2000; Kondo & Saiki, 2012; Logie, Brockmole, & Jaswal, 2011). The asymmetry could also be present for binding between depth and visual features. Past research has shown that memory performance for colors is affected when colors are bound with depth (Chunharas et al., 2019; Qian et al., 2017; Qian et al., 2018; Xu & Nakayama; 2007), however, our study shows that binding depth with colors or sizes does not significantly affect the overall memory performance for depth. We think that binding between an object's feature and its depth location is not automatic and can be quite weak, if existed. Therefore, the weak binding cannot reduce interference between items to enhance the overall encoding or retention process, and thus no overall improvement was observed.
Although our findings consistently showed that increasing perceptual separateness promoted the performance at the depth plane of 0 degrees and +0.17 degrees with the middle fixation, it is unclear why the memory benefit is absent at the depth plane of −0.17 degrees. One possibility is that the allocation of attention in depth is asymmetrically biased toward positive disparity. To our knowledge, there is no previous study that has investigated the natural distribution of attention in depth. However, several studies have demonstrated a systematic overestimation bias that memorized depth is likely to be recalled as farther (e.g. Campagnoli et al., 2017; Zhang et al., 2020). Zhang et al. (2020) suggested that this might be partly due to a tendency toward a “default” position of dark vergence (the position where the eyes converge at in absence of visual input, about 1 m). Similarly, this default position, which is farther than the point of fixation in our study, may facilitate the performance in the depth plane with a positive disparity (farther).
Our study has shown several interesting findings. First, the memory benefits for the boundary depth locations in a homogeneous display suggest that attention plays a crucial role in WMd. Attention is primarily oriented to the fixated boundary location (the nearest or farthest depth plane) or split between the boundary locations when the middle is fixated. Second, our study is the first to report that the memory performance for depth decreases as the tested item is presented away from the attended plane(s). Such a linear trend is very strong and reliable, indicating a default pattern of attention allocation in depth. Third, when fixating at the middle of the whole depth volume, increasing the perceptual separateness by binding distinctive feature values to memory items improved memory performance for less distinct depth locations. The benefits of feature separation may be attributed to enhanced individuation of memory items, therefore facilitating a more balanced allocation of attention and memory resources. To summarize, our findings shed light on the role of attention in temporally holding depth information, which provides unique contributions to the current literature on WM.
Supported by the Guangdong Basic and Applied Basic Research Foundation (2021A1515010840), the National Natural Science Foundation of China (31500919) and the Fundamental Research Funds for the Central Universities (20wkzd12). The authors have no competing financial interests that might be perceived to influence the results and/or discussion reported in this paper.
Commercial none.
Corresponding Jiehui Qian.
Email: qianjh3@mail.sysu.edu.cn.
Address: Department of Psychology, Sun Yat-Sen University, Guangzhou 510006, China.