Expectation-driven sensory adaptations support enhanced acuity during

categorical perception

Authors: Tim Sainburg, Trevor S McPherson, Ezequiel M Arneodo, Srihita Rudraraju, Michael Turvey, Bradley H Theilman, Pablo Tostado Marcos, Marvin Thielk, Timothy Q Gentner

Abstract

Expectations can influence perception in seemingly contradictory ways, either by directing attention to expected stimuli and enhancing perceptual acuity, or by stabilizing perception and diminishing acuity within expected stimulus categories. The neural mechanisms supporting these dual roles of expectation are not well-understood. Here, we trained European starlings to classify ambiguous song syllables in both expected and unexpected acoustic contexts. We show that birds employ probabilistic, Bayesian, integration to classify syllables, leveraging their expectations to stabilize their perceptual behavior. However, auditory sensory neural populations do not reflect this integration. Instead, expectation enhances the acuity of auditory sensory neurons in high-probability regions of the stimulus space. This modulation diverges from patterns typically observed in motor areas, where Bayesian integration of sensory inputs and expectations predominates. Our results suggest that peripheral sensory systems use expectation to improve sensory representations and maintain high-fidelity representations of the world, allowing downstream circuits to flexibly integrate this information with expectations to drive behavior.

Introduction

Categorical perception groups smoothly varying signals into discrete classes, affording generalization at the expense of acuity. In many settings, categorical perception is driven by expectation. For example, in speech, as contexts change, perception is biased towards likely sounds, words, and phrases [1, 2, 3], reflecting a shift in prior expectations. This warping of perception towards expected categories is called the perceptual magnet effect [4, 5] and can be formally described as a process of Bayesian inference over acoustic distributions [6, 7, 3, 8, 9]. Under this framework, an optimal perceiver resolves sensory ambiguity by integrating noisy or imperfect sensory information with information about their prior expectations in a particular context. In contrast, prior expectations also play a pivotal role in enhancing sensory acuity by refocusing attention toward expected regions of stimulus space [10, 11, 12]; this ability of priors to bias our sensory systems enables more accurate discrimination of closely-related signals [13, 14, 15]. In the domain of language, this phenomenon is exemplified by listeners ability to fine-tune speech recognition based on the characteristics of the speakers voice [16]. It remains unknown how these dual contrasting functions of prior expectations - generalization underlying categorization and sharpened sensory acuity - are implemented neurally. Specifically, the extent to which early sensory representations are influenced by prior expectation remains unclear. This ambiguity is highlighted in speech sound perception, which can manifest as either categorical or continuous depending upon the specific task at hand [17, 18]. The task-dependent nature of perception fuels ongoing debates about the inherent nature of categorical perception; whether it is an intrinsic aspect of the sensory systems responsible for perception, or a product of downstream decision-making processes [19, 20, 21, 22]?

One possibility is that sensory codes are not impacted by expectation. Most empirical evidence for probabilistic integration derives from work on regions of the brain associated with motor and decision making [23, 24, 25, 26, 27, 28], and suggests that sensory populations encode a likelihood distribution [29] integrated with expectation by downstream circuits. Alternatively, expectation and context could be integrated in sensory systems themselves, which then are used as a substrate for decision making; such modulation could occur via feedback loops with higher-order systems (e.g. decision-making and multi-sensory integration; [30]) or through local network dynamics [21, 31, 32, 33]. Both theoretical work [31, 21, 19, 34] and recent brain wide analyses [22, 35] raise the possibility that even early sensory brain regions engage in probabilistic integration of information about sensation and expectation. However, it is also possible that expectation impacts sensory codes, but does so in a manner that is inconsistent with classical Bayesian integration. Instead, sensory modulation may be related to the active role that prior expectations play in shaping our interaction with the environment. For example, it is known that prior expectations drive attention and active sensing [36, 37], which modulates sensory encoding [15, 11, 38, 39] and improves sensory acuity [13, 14, 12, 40, 41]; these findings suggest that expectation may flexibly focus neural resources on regions of sensory space where signals are expected, thus improving perceptual acuity rather than pulling percepts toward expected categorical representations.

These three hypotheses can be restated in the language of Bayes rule. Noisy measurements of physical signals received by sensory systems represent the likelihood of a particular stimulus. The likelihood is then integrated with prior expectations to form a posterior probability on which decisions are made (Figure 1A). In the feed-forward probabilistic integration account, sensory systems represent only the likelihood, and prior probabilities and the posterior distribution are not reflected in the sensory brain (Figure 1B). In contrast, the sensory probabilistic integration account predicts that sensory populations will reflect the Bayesian integration of prior and likelihood (Figure 1C). Finally, in the attention and active sampling account, expectations sharpen the stimulus likelihood without directly integrating the prior probability distribution (Figure 1D).

Here we analyze the activity patterns of sensory neuron populations in an auditory perceptual decision-making task to discern which computational model (feed-forward probabilistic integration, sensory probabilistic integration, or attention and active sampling) most accurately explains neural processing in sensory systems. Songbirds provide an important opportunity to study mechanisms of categorical vocal perception in neurobiological detail as they perceive some elements of song categorically [42, 2] and those elements are biased by expectation [43]. We developed methods to explicitly impose probabilistic predictive information in a sequence of birdsong syllables and trained European starlings, a songbird with complex vocal repertoires, to classify smoothly varying syllables while controlling sequential predictive song structure. To begin, we show that categorical perception of vocal elements is well-explained by Bayesian probabilistic integration, and that sensory neural responses capture the perceptual uncertainty (i.e. the likelihood) of a Bayesian model representing the birds’ behavior. We observe that sensory neural responses are directly modulated by the predictive structure of vocal sequences, thereby ruling out the feed-forward probabilistic integration hypothesis. These response biases do not align, however, with a Bayesian integration of prior and likelihood (i.e. the posterior) as predicted by the sensory probabilistic integration hypothesis. Instead, we find that the bias is consistent with dynamic changes in sensory acuity (i.e. the likelihood of the Bayesian model) in predicted regions of acoustic space. As a consequence, an unbiased representation of the sensory signal is left available for flexible use in behavior.

Results

Categorical decision-making is modulated by expectation

Sensory neuroscience and psychophysics have long, productive histories founded on the idea of relating parametric change in a stimulus to quantifiable changes in both neural activity [44] and behavior [45]. Implicit in this approach is the strong assumption that sensory inputs can be discretized into stimulus events parametrically varying along one or two continuous dimensions. This approach is ideally suited to investigate how simple, non-natural, and easily controllable signals are perceived behaviorally or encoded neurally, but neglects the natural history of sensory systems, which are adapted to complex ethologically relevant signals like birdsong [46, 47]. Attempts to apply the same kind of parametric stimulus control to natural stimuli are rare because natural signals tend to vary along multiple dimensions simultaneously [48, 49].

To address this challenge, we developed a behavioral paradigm to control context-dependent categorical perception in a natural stimulus space by synthesizing smoothly varying starling song syllables using a neural network. We captured the complex spectro-temporal statistics of song acoustics using a deep convolutional variational autoencoder (Figure 2A) [50] trained on a large library of conspecific song. From the latent space of this network, we synthesized acoustic continua (N=9), each comprising 128 synthetic syllables (morphs) that smoothly vary between two acoustically distinct syllable endpoints. We trained starlings using a two-alternative choice (2AC) category learning task to classify the naturalistic syllable morphs that lie along these continua (Figure 2B). We divided each continuum at the midpoint and reinforced one half with food for pecks into the left response port and the other half for pecks into the right response port (Figure 2C bottom). We trained birds (n=20) on the syllable classification task to obtain psychometric functions for each syllable continuum (Figure 2D) and then introduced cue syllables preceding the target (to-be-classified) syllable. Each cue syllable provided predictive information about the likely response category of the target syllable (Figure 2E). All subjects learned the task to at least 75% accuracy (Supplementary Table 1) performing a total of 4.8 million behavioral trials.

We fit a psychometric model (Figure 3A) to each subject’s classification behavior for each syllable continuum (Figure 3B), then used the parameters of the fit psychometric model to understand how the cue affected behavior. Under the Bayesian integration hypothesis, (Figure 2F-H) categorical perceptual decision-making (i.e., syllable classification) is modulated by integrating the likelihood imposed by the stimulus (i.e., the target syllable) with the prior imposed by its sequential context (i.e., the preceding cue syllable). As a result, the decision boundary (Figure 3A, inflection point) shifts in the direction predicted by the cue (Figure 3C top). We also examined whether information from the cue and target syllables are treated independently, in a non-Bayesian way, as evidenced by an overall shift in the probability of a left or right response, but not a shift in the decision boundary (Figure 3C bottom).

Across each syllable continuum and for each bird we observed robust shifts in the decision boundary (Figure 3B, D-F; Linear Mixed Effects; psychometric inflection ~ cue probability + (1∣subject) + (cue probability∣subject); cue β=-11.12, SE = 1.158, z= −9.6, p <0.001), consistent with Bayesian integration underlying context-dependent categorical perception. To examine this shift more closely, we fit the Bayesian model (in particular the likelihood) to each birds behavioral data on uncued trials (for each continuum) and predicted the inflection point shift given each cue probability. The red dashed line in Figure 3F depicts a linear regression showing the close correspondence between the observed shift in inflection point and that predicted by the Bayesian model. In addition, we observed an overall shift in decision probability (Figure 3D), suggesting that on a subset of trials subjects responded independently to the cue or the target syllable alone, which aligns with previous findings that animals alternate between decision-making strategies from trial-to-trial [51].

We observed substantial variation in the slope of the psychometric functions fit to each birds behavior. Some individuals showed a much sharper categorical boundary than others (e.g. B1432 vs B1110 in Figure 3B) and the mean slope (averaged across individuals) also varied between syllable continua (Figure 3G). The slope of the psychometric curve reflects uncertainty in the Bayesian model. Under greater uncertainty about the target syllable, the Bayesian model predicts that integration with the cue stimulus will result in a greater shift in categorical perception (i.e. the inflection point; Figure 3H [52]). Consistent with this, we observed a smaller inflection point shift in the direction of the cue as the slope of the psychometric curve steepens (Fig. 3I; Linear Mixed Effects; inflection shift ~ psychometric slope + (1∣subject) + (psychometric slope∣subject); cue probability: β=−0.401, SE = 0.099, z= −4.03, p <0.001), which again matches the quantitative prediction of the model (Figure 3I, red dashed line).

Both the likelihood of a given stimulus and its prior probability were reflected in the response times of birds. Response times were longer on incorrect trials than correct trials (Extended Data Figure 1A; Linear Mixed Effects; response time ~ correct response + (1∣subject); correct response: β=-0.298, SE <0.001, z= −339.448, p <.001), suggesting challenging decisions take longer to make. When looking only at trials where the bird was correct and controlling for side bias (see Methods), we find that response times decreased proportional to the prior probability imposed by the cue (Extended Data Figure 1B) and that response times near the center of the morph increased following the birds psychometric slopes for each morph (i.e. the likelihood; Extended Data Figure 1C,D).

Sensory neural responses reflect behavioral likelihood

These behavioral results indicate that in our task birds are probabilistically integrating expectations with sensory experiences to categorize song syllables. To investigate whether sensory forebrain neural populations reflect Bayesian integration, we recorded extracellular neural spiking activity using 1-2 (unilaterally or bilaterally) implanted 32-64 channel 1-8 shank silicon electrode arrays from freely behaving subjects (N=10) while they completed trials on the syllable categorization task and passively listened to the same stimuli (during both sleeping and waking states). We targeted electrode arrays broadly across the auditory forebrain, including the primary auditory region Field L (Extended Data Figure 2) and secondary regions CM (Caudal Mesopallium), NCM (Caudomedial Nidopallium), and medial NCL (Caudolateral Nidopallium). We recorded from a total of 13,854 putative single-neurons (See “Spikesorting”).

We analyzed spike train data as spike vectors over the different syllable continua by convolving the time histogram (bin width=10ms) of the stimulus-aligned spike train for each trial with a Gaussian kernel (σ=25ms; Figure 4A-E). Figure 4E and F show sample spike trains and trial-averaged spike vectors for a sample unit for each syllable continuum. From the trial averaged spike vectors, we computed a cosine similarity matrix between spike vectors for each syllable on each continuum (Figure 4I) from which we then computed a neurometric function (Methods, Figure 4J). We also used the cosine similarity matrix to compute a metric for each units task relevance (Figure 4K-L; see “Assessing task relevance for units”) reflecting the similarity of the unit’s response within versus between syllable categories. Importantly, this analysis is not meant to suggest these neurons reflect learned categories, only that they show response variance over the task-relevant stimulus space. Of the 13,854 units recorded, 7,994 had task relevant responses to the syllable continua (See “Subsetting task-relelvant units”). On average, the spike vector responses for these task-relevant units changed smoothly across the syllable continua, but the degree of this smoothness varied (Figure 4M-N).

To assess whether neural responses reflected behavior, we compared the slope of the neurometric function to the slope of the psychometric function for each bird and syllable continuum. The neurometric slopes were well predicted by the psychometric function (Figure 4, P; Linear Mixed Effects; Log Neurometric Slope ~ Log Psychometric Slope + (1∣unit) + (Log Psychometric Slope∣unit); Log Psychometric Slope: β=0.172, SE <0.005, z= −37.006, p <.001). Because in the Bayesian decision-making model the slope of the psychometric function is modulated by the likelihood, i.e. stimulus uncertainty, it follows that these neural responses carry information about the stimulus uncertainty. Additionally, we assessed whether individual bird-level variation in psychometric slopes was reflected in the unit neurometrics, and found that it was not (see Methods), indicating a possible downstream role for estimates of stimulus uncertainty in decision making.

Expectation suppresses sensory spike rate

Prior work has established that expectation modulates neural activity in sensory and decision-making brain regions, with activity in decision-making regions increasing with predictability and activity in sensory regions decreasing [11]. The reduction in activity in sensory areas during predictable stimuli may reflect the dampened coding of task-irrelevant features, which could be useful for improving acuity [12]. To assess whether cue syllables modulate neural responses to target syllables, we quantified how much the overall spike rate in each unit changed as a function of the predictive cue syllable in trials in which birds behaviorally responded to stimuli. The presence of a cue syllable significantly suppressed the spike rate evoked by the target syllable, when controlling for spike rate variability across units (Figure 5A; ANOVA for Linear Mixed Effects model X^2^(4, N = 857301) = 15196, p <1e-5; see Methods for details). This suppression was consistent across the motif continuum, stronger in active trials than passive playbacks (Figure 5B), and was most prominent early and continues throughout much of the target stimulus playback (Figure 5C). Moreover, the magnitude of the cue-dependent suppression was consistent within each cue condition, and persisted throughout stimulus playback. In passive playback trials any cue-dependent effects quickly diminished (Figure 5D).

Because cue syllables are differentially informative (i.e. establish different priors) for upcoming target syllables, we reasoned that the magnitude of cue-specific response suppression might co-vary with the strength of the predictive information. We therefore measured the impact of the cues predictive probability on spike rates while controlling for differences in each units response between syllable continua (Supplementary Figure 13). We found that as the predictive strength of the cue syllable increased, the associated spike rates decreased (Figure 5E; ANOVA for Linear Mixed Effects model X^2^(1, N = 857301) = 399, p <1e-5; see Methods for details). This effect was abolished during passive playback (Figure 5F) and when cue labels were shuffled (Supplementary Figure 13). Taken together, these results are consistent with previous observations that signal predictability decreases responses to stimuli (i.e. expectation suppression; [11]), which is believed to reflect a dampening of task-irrelevant noise [12].

Expectation drives sensory likelihood modulation

As Figure 5 shows, prior expectations modulate sensory neural responses. This rules out the feed-forward probabilistic integration hypothesis, that sensory populations representing the stimulus are unmodulated by expectation (Figure 1B). The remaining two hypotheses make opposing predictions about how sensory populations should be modulated by prior expectations. The attention and active sampling hypothesis predicts that sensory representations become increasingly discriminable in high-probability regions of stimulus space (Figure 6B,D). In our model of attention and active sampling (Extended Data Figure 3), we assume an increase in sensory acuity in one region of sensory space (here, the cued section of the syllable continuum) comes at the expense of acuity in other sensory dimensions (i.e. acoustic features irrelevant to our categorization task; see Methods for details) [53, 54]. Under this model, if neural responses reflect the stimulus likelihood, their similarity should decrease as a function of predictive cue strength (Figure 6F,I), both because signals become more discriminable along the task-relevant dimension (here, the syllable continuum) and because representational noise along task-irrelevant dimensions increases. Alternatively, Bayesian integration mirrors an effect in categorical phoneme perception called the Perceptual Magnet Effect [4], whereby speech perception is warped around categorical boundaries to reduce discriminability of within-category sounds (Figure 6C,E). In the Bayesian model, this perceptual warping results from the integration of prior distributional information with a noisy representation of the acoustic signal, yielding a shift in the posterior toward higher probability regions of acoustic space [6] (Figure 6C). As a result, similarity within high-probability regions of stimulus space increases, compressing within-category representations together (i.e. perceptual magnetism). In the context of our task, increasing predictive probability toward one side of the syllable continuum (i.e. in the context of a predictive cue) leads to two outcomes: the within-category similarity of the posterior on the predicted side of the continuum will increase, and the within-category similarity of the low-probability side of the continuum will decrease (Figure 6G). Under this model, if neural responses reflect this posterior distribution, their similarity should also increase as a function of predictive cue strength (Figure 6F,I).

To determine which model best aligns with our neural data, we employed two methodologies. First, we directly compared similarity of neural responses to syllables across the different continua as a function of cue condition. Second, we used a decoder model to estimate the accuracy of stimulus and stimulus class predictions from neural data in different cue conditions.

Comparing the trial-to-trial cosine similarity of the spike vector response across syllable continua revealed that in the presence of a predictive cue, the within-category similarity is higher in the non-predicted (cue-invalid) class than the predicted (cue-valid) class (Figure 6H; Linear Mixed Effects, see Methods; β=0.009, SE <0.001, p <0.001). Moreover, the within-category similarity across units and continua decreases significantly as a function of the probability of the cue class (Figure 6J; Linear Mixed Effects, see Methods; β=-0.01, SE <0.001, p <0.001). This effect is abolished when cue labels are shuffled (Supplementary Figure 1), suggesting that perceptual acuity is selectively sharpened over predicted regions of acoustic space, which likely decreases the overall representational noise. To test this directly, we compared the variance in spike rates as a function of cue-validity. On average, spike rate variance in the cue-valid condition was slightly, but significantly lower than that in the cue-invalid condition (std-dev ~ cue-condition + (1 ∣ stimulus : unit); β=−0.515Hz, p < 0.001), consistent with the idea that valid cues reduce representation noise.

We next asked whether prediction accuracy increases and whether stimulus predictions are shifted when stimuli are expected. To this end, for each morph and bird, we trained a logistic regression on PCA projections of population activity fit to passive playbacks and uncued behavioral trials (Figure 6K). We then applied the PCA projection and predicted morph positions of the held-out cued trials. Each decoder model performed well above chance, validating that stimulus identity can be decoded from sensory populations (Figure 6L,M).

Both models make dichotomous predictions about the decoded responses. The attention and active sampling model predicts that the decoder accuracy of cue-valid (e.g. cue-left, morph left) trials will be higher than cue-invalid (e.g. cue-left, morph right) trials (Figure 6N). The Bayesian integration model predicts instead that decoder predictions will shift in the direction of the cue (Figure 6Q). We analyzed our population decoder results on the basis of these predictions, and found that decoding accuracy improves for the validly cued stimulus (Figure 6P; Linear Mixed Effects; correct ~ cue-valid +(1∣population) + (cue-valid ∣ population); cue-valid: β=0.007, SE = 0.002, z = −3.78, p <.001), and that the inflection point fit to model predictions did not shift toward the cue (Figure 6Q; z-test between psychometric model fits; z = −0.0555, p = 0.478). Additionally, we confirmed that the neurometric curve does not shift over single units by computing the neurometric curve directly on the response similarity matrices, as in our initial comparisons between neurometric and psychometric curves (Supplementary Figure 2). We measured the change in the inflection point between cue-valid and cue-invalid trials. Across units, we do not find a significant shift in inflection point between high-probability and low-probability cues (Linear Mixed Effects; neurometric shift ~ 1 + (1∣unit); β=0.13, SE = 0.82, z = 1.64, p = 0.10). These results support the sensory modulation model over the Bayesian integration model, demonstrating that sensory acuity is enhanced at the population level and that sensory representations do not shift towards expected stimulus classes.

Receptive fields remapping underlies sensory modulation

The preceding results show that expectation modulates sensory responses and supports a model of sensory modulation where expectation drives changes in sensory acuity. We observed that, on average, spike rates are suppressed as expectation increases (Figure 5E). One possible mechanism for the spike-rate reduction is that expectation modulates the gain of an otherwise static stimulus-response relationship (i.e. receptive field). Alternatively, expectations may drive a more dynamic remapping of receptive fields. To differentiate between these hypotheses, we fit a Maximum Noise Entropy (MNE) composite receptive field model [55] to stimulus-evoked single-neuron activity on a subset of trials where the cue provided a valid prediction of the upcoming target syllable. If the cue has no effect on the receptive field, then model performance (correlation between model predicted and empirical response) should be similar for the same target syllable presented on held-out cue-valid and cue-invalid trials (Extended Data Figure 4A-B). Across all cue strengths, however, the MNE receptive field models provided significantly better (more accurate) predictions of responses to target syllables on cue-valid trials than on cue-invalid trials (Extended Data Figure 4-C; Linear Mixed Effects, trial-correlation ~ cue-validity + (1 ∣ unit) + (1 ∣ day); β=0.015, SE < 0.001, z = 41.074, p < 0.001). Thus, contextual cues rapidly reorganize receptive fields to better encode predicted stimuli. This reorganization is produced by cue-dependent changes in both the gain and stimulus-feature tuning of linear and non-linear components of the receptive fields (see Methods).

To directly link changes in receptive fields to sensory likelihood modulation, we reproduced the similarity analysis from Figure 6H&J with the output of the MNE encoder model. We fit MNE receptive fields to target syllables for each cue condition separately and passed all syllables through the model to generate predicted spiking probabilities for the duration of each stimulus. Cue-driven information is then encoded in the variability of the neural response to a given stimulus across cue conditions and hence will produce distinct spiking probability vectors for each cue condition. We computed the similarity of the spiking probability vectors across the different continua as a function of cued direction, taking the difference of the resulting similarity matrices derived for left and right cue conditions, as we did for the empirical responses. Paralleling our empirical results in Supplementary Figure 6H-J, we see that within-category similarity is higher in the non-predicted (cue-invalid) class than the predicted (cue-valid) class (Supplementary Figure 3A; Linear Mixed Effects, sim-diff-full ~ validity + (1 ∣ unit) + (1 ∣ day): beta = −0.013, SE < 0.001, z = −154.841, p < 0.001) and that within-category similarity decreases as a function of the probability of the cue class (Supplementary Figure 4; Linear Mixed Effects, class-sim-cue ~ validity + (1 ∣ unit) + (1 ∣ day): beta = −0.07, SE = 0.001, z = −12.970, p < 0.001). These results further support the notion that neuronal responses are dynamically restructured to optimize the differentiation of expected stimuli.

Expectation improves perceptual acuity

The preceding physiological evidence supports a model of expectation-dependent sensory modulation in which sensory representations are flexibly reorganized to improve acuity in expected regions of stimulus space (i.e., the attention and active sampling model). These neural changes should also lead to improved behavioral acuity in expected regions of stimulus space. Testing this prediction in the original behavioral task is not possible, however, because the cued portion of the stimulus space is tied to response class (peck left or peck right), and therefore changes in perceptual acuity cannot be dissociated from the behavioral decision.

To directly test whether expectation modulates sensory acuity, we designed a modified behavioral task in which cues predict the entire syllable continuum rather than a response class (Figure 7A). By presenting the same syllable continua under differing levels of expectation, we can assess how perceptual acuity is modulated by expectation.

In the modified task, we paired each of three syllable continua (A-E, B-F, C-G) with a cue syllable. Each cue preceded its paired syllable continuum with 80% probability (e.g. p(morphAE∣cueAE)=0.8), and the other two syllable continua with 10% probability (e.g. p(morphBF∣cueAE)=0.1). These cued trials accounted for 80% of trials. On the remaining 20% of trials, we presented either an uninformative cue (e.g. p(morphAE∣cueNI)=0.33) or no cue (p(morphAE∣peck)=0.033).

Given our physiological results, we predicted that in trials where syllable continua are more expected, perceptual acuity would increase and the birds would perform better when discriminating the stimulus. We computed psychometric functions for each continuum and cue condition and took the slope of each as an estimate of perceptual sensitivity across the stimulus space (syllable continuum).

As predicted by our model and consistent with our physiological results, sensitivity improved in the presence of predictive cues (Figure 7C; permutation test controlling for subject and morph, see Methods; r = 0.22, p <0.001), coinciding with an improvement in behavioral accuracy (Figure 7B; Linear Mixed Effects; accuracy ~ cue probability + (1∣subject) + (cue probability∣subject); cue β=0.068, SE=0.014, z(37993)=5.02, p <0.001). These behavioral effects corroborate our observations of improved sensory acuity at the neural level.

Discussion

Expectation plays a varied, yet fundamental role in perception. It can facilitate generalization through probabilistic integration, and it can improve acuity through attention. How these diverse outcomes of expectation-driven categorization and acuity are balanced in the course of real-world perception has not been clear. Here, we find that early sensory processing reflects prior information, thereby improving sensory acuity, while relegating probabilistic integration of these expectations to downstream circuits involved in decision-making and behavior.

To disambiguate models for how expectation might influence sensory representations we trained songbirds on a categorical perceptual decision-making task and manipulated the predictive contextual information in sequences of vocal elements. Songbirds exploit this information, biasing the categorical perception of their vocalizations. A Bayesian model of perceptual decision-making captures both qualitative and quantitative aspects of this behavioral bias (Figure 3), reflecting the integration of predictive contextual information with uncertainty over natural stimulus dimensions. This model is similar to that which has been proposed for human context-dependent categorical speech perception [3, 6]. Neural recordings revealed that many sensory neuronal responses throughout the auditory forebrain are broadly responsive across the natural stimulus space dimensions in which our task was embedded, mirroring the animals perceptual behavior (Figure 4). Syllable sequence predictability influenced these sensory representations by suppressing spike rates and modulating syllable encoding and decoding (Figure 5, Figure 6, Extended Data Figure 4). Contrary to the explicit predictions of the Bayesian model, these neural responses do not directly represent the integration of prior information in these sensory regions. Instead, the context-dependent modulation more closely reflects an increase of perceptual acuity in predicted regions of stimulus space (Figure 6), facilitating an increase in behavioral performance (Figure 7). Our results indicate that the coordinated variability of sensory forebrain neuronal populations dynamically shifts in the face of predictions, facilitating optimal encoding along (anticipated) stimulus-relevant dimensions. This restructuring of the stimulus-response mapping is suggestive of a top-down predictive model reshaping the stimulus likelihood within sensory regions in anticipation of upcoming stimuli [12]. This conceptualization has the potential to explain how internal models can reduce spiking variance when predictions are valid, and casts “noise” when predictions are invalid as predictive error.

Current speech research aims to uncover neural systems involved in processing predictive information related to lexical and pre-lexical feedback [3]. Many have proposed that a Bayesian framework provides a mechanistic explanation for speech categorization and comprehension more broadly [6, 3, 8]. However, human studies have methodological limitations, leaving gaps in our understanding of neural representations of speech category information and their modulation under various comprehension-related conditions. Our results support Bayesian integration as a mechanism for categorical perception, but leave open the possibility that biases imposed through probabilistic integration, such as categorical perception, are at least in part the product of task-dependent decision-making, rather than early sensory and perceptual processes. This is reminiscent of behavioral observations in speech, where the degree to which speech is categorically perceived is task-dependent [3]. Our observations suggest a functional segregation of Bayesian integration processes that is adaptive for communication in the sense that it preserves a veridical sensory representation of the stimulus that can be used flexibly in the service of multiple task demands.

Although our findings suggest that sensory populations do not reflect Bayesian integration, it remains possible that some perceptual biases are encoded in sensory systems. In speech, some secondary sensory populations have been found to exhibit categorical responses [56]. Some aspects of categorical speech perception also appear to be less task-dependent. For example, native Japanese speakers often have trouble distinguishing between the English phonemes /r/ and /l/ (as in rake vs lake) because there is no distinction between /r/ and /l/ in Japanese [4, 5], a perceptual bias that can extend for years after exposure to a second language [57]. Reflections of prior expectations have been observed throughout the sensory hierarchy in mice [22]; it may be that the role of expectation in the brain differs along more dimensions than sensory versus decision making systems. For example, language-related sensory systems may adapt to phonetic categories during critical periods of language acquisition imposing immutable biases to perception. However, our observations contextualize the broad qualitative differences observed between sensory and decision making brain regions, where suppression in the sensory brain is linked to dampening noise in task-irrelevant dimensions [12] and increased activity is associated with the integration of expectation and sensory experience [11]. Our results suggest that, even while animals perform Bayesian inference at the behavioral level, sensory populations reflect expectation in a manner wholly unrelated to Bayesian integration. Simultaneous and large-scale recordings across the perceptual and decision making hierarchy will be crucial for understanding how expectation is utilized broadly across the brain. Studying these hierarchies remains a challenge in non-model systems such as the European starling (and more broadly, songbirds) where neural substrates for motor control and cognition outside of song production are not well characterized. For instance, the lateral-most region of the NCL is a promising candidate for probabilistic integration in songbirds, paralleling the premotor and cognitive functions of the primate frontal cortex. However, the role of NCL has been primarily described in visual processing and multisensory integration, with no evidence yet found in auditory cognition [58], highlighting a significant area for future research.

Methods

Ethical note

All procedures were approved by the Institutional Animal Care and Use Committee of the University of California (S05383).

Summary

Experiments consisted of a behavioral component and a chronic physiology component. The experimental protocol for the behavioral component was kept constant by using the same software and hardware in both conditions, with the addition of chronic electrophysiological recording in the physiology component.

Subjects

Behavioral data were collected from 20 wild-caught European starlings of unknown sex. Before beginning experimental training, subjects were housed in a large mixed-sex aviary. Of the 20 starlings used for behavior, 10 individuals were used for chronic physiology.

Datasets

Our final behavioral dataset was composed of 4.8 million behavioral trials from 20 birds. Our final chronic neural dataset was composed of 402,797 behavioral trials, with 365,360 responses, a total of 1,594,257 audio playbacks, occurring over 5,345 hours (222 days) of recording, across 10 birds.

Stimulus generation

Stimuli were syllables of European starling song synthesized from a Variational Autoencoder (VAE) trained on syllables extracted from a library of European starling song [59].

Training dataset

Syllables were segmented from the full songs of starlings with the dynamic thresholding approach outlined in [60] and available in the vocalization segmentation python package (https://github.com/timsainb/vocalization-segmentation). Syllables were zero-padded symmetrically at their beginning and end to be 1 second long. Spectrograms of each syllable were computed with 128 frequency bands spaced between 50 and 22050 Hz, and downsampled to 128 time-bins (128 Hz), resulting in a 128x128 spectrogram of each syllable, used to train the VAE.

Neural network

The neural network architecture we used followed those in our AVGN jhk fmhtcg bn. We used a convolutional VAE architecture with a 16-dimensional latent space. The network was trained on batches of 32 syllables at a time. Artificial neurons used a leaky ReLu non-linearity. The network was trained with the ADAM optimizer in Tensorflow.

Sampling and synthesis

Each syllable stimulus (used for cues and endpoints) was sampled from the original dataset (Supplementary Figure 5) and passed through the VAE. The stimuli were chosen to be diverse, well-reconstructed in the VAE, and roughly equidistant both in spectrogram space (both input and reconstruction) as well as the latent space of the VAE. It is not expected that distances in spectral or neural network latent space would have a 1 relationship with an animal’s perception of similarity. Morph syllables were sampled from 126 evenly spaced points only the linear interpolation between the latent (16D) representations of a pair of endpoint syllables and passed through the decoder of the VAE, producing the final 128 syllable continuum including the two endpoint syllables (Extended Data Figure 6). Waveform stimuli were then generated from the spectrogram output of the decoder of the VAE using the Griffin-Lim algorithm. These waveforms were the stimuli used for playback to the birds.

Behavioral training paradigm

Birds were initially trained to differentiate between the two syllable endpoints for a single continuum. After several days of above-chance accuracy with one pair of syllable endpoints, the number of endpoints was increased until the birds showed above accuracy classification of the endpoints of all 9 continua. After learning the correct response for all endpoints, birds were transferred to the full stimulus set which included all 128 syllables (linearly sampled and equally spaced in latent space) spanning each of the 9 continua (1152 syllables total). After the birds were performing reliably above chance on each full syllable continuum for several days, we added cue syllables preceding the target syllables to provide context-dependent information at p=0.125, p=0.25, p=0.5, p=0.75, and p=0.875.

Training parameters

Several behavioral parameters were used in behavioral training, given here for reproducibility. Trials were reinforced on a variable ratio schedule between 2-4 responses, manually set for each bird to maximize the number of trials each day without the loss of more than 10 grams of weight from baseline when in the restricted feeding condition. Punishment was set at a 5-second lights-off period, during which new behavioral trials could not be initiated. A minimum of 1 second between trials, regardless of response, was imposed. Birds could not respond during stimulus playback. Birds were given a 5-second window to respond after stimulus playback. Lighting conditions were set to match seasonal sunrise and sunset times in the experimental location (San Diego, California).

Cue stimulus

Like the morph syllables, the cue syllables are 1-second long, synthesized by reconstruction from the variational autoencoder. Behavioral trials were presented with one of 6 cue no cue P(L—No Cue)=0.5 (NC), cue with no predictive information P(L—Cue)=0.5 (CN), cue left at p=.875% P(L—Cue)=0.875 (CL1), cue left at p=0.75% P(L—Cue)=0.75 (CL0), cue right at p=.875% P(L—Cue)=0.125 (CR1), cue right at p=0.75% P(L—Cue)=0.25 (CR0). 16% of trials were presented in the no cue condition (NC). 4% of trials were presented with the uninformative cue condition (CN). The remaining 80% of trials were evenly split between the cue right and cue left conditions. Because the CN condition was sampled with a substantially lower probability than the other conditions, resulting in a low number of total trials in comparison to each other cue condition, it was not included in physiological analyses. In passive physiology playback conditions, due to time constraints in playing back the full stimulus set of 128 interpolation points for each of 9 morphs and 6 cue conditions, we played back only the 87.5% predictive cue conditions in the AE and BF morphs.

Psychometric fit

To assess the shift in categorical perception, in each of the birds (n=20) we fit a psychometric (four-parameter logistic) function both to the overall responses to stimuli in the left and right categories of the morph, as well as to each individual morph. The fit psychometric across all morphs is given in Supplementary Figure 6, across all birds is given in Supplementary Figure 7, and broken out across all birds and morphs is given in Supplementary Figure 8 logistic(x)=max.+min.−max.1+(xinflection)slope

Bayesian integration model

To formalize our hypothesis, when a stimulus varies upon a single dimension x, the perceived value of x as a function of the true value of x and contextual cue information can be described by Bayes’ rule: (1)P(xtrue∣xsensed,cue)︸posterior∝P(xsensed∣xtrue,cue)︸likelihoodP(xtrue∣cue)︸prior

By modulating the prior distribution of the categorical stimuli (x) with a cue, we predict that the perception of x will shift.

Preceding each to-be-categorized target stimulus (x), the cue stimulus provides predictive information about the category of the target stimulus. By treating this cue stimulus as a prior probability over x, we predicted that the determined posterior probability of x given sensory information and the cue stimulus would shift the classification of stimuli near the boundary between the two classes in the direction predicted by the cue stimulus.

Explicitly, we treat the likelihood of a target being sensed P(xsensed∣xtrue,cue) as a Gaussian probability distribution around the true target xtrue [61, 6]: (2)P(xsensed∣xtrue)=1σ2πe−12(xtrue−xsensedσsensed)2 and set the prior probability as a function of the cue (3)P(xtrue>categorical boundary∣cue)=cueprob where cueprob represents the predictive probability of the cue stimulus. In our case then, (4)P(xtrue∣cue)={cueprob∕64xtrue>categorical boundary(1−cueprob)∕64xtrue<categorical boundary}

We predict that birds will make a categorical decision on the basis of the posterior, (5)P(right peck∣xsensed,cue)=P(xtrue>categorical boundary∣xsensed,cue)

Under this model, the categorical decision of the bird is modulated by the prior cue information, resulting in a shift in the categorical decision point along the stimulus dimension in the direction predicted by the cue (Figure 2I).

Bayesian fit

In addition to fitting a psychometric function capturing the shape of the behavioral responses, we fit a Bayesian model reflecting our probabilistic hypothesis described above. This model used five the shape of the Gaussian of the likelihood (σsensed), a parameter corresponding to side bias in the apparatus (γ parameters representing inattention to the cue stimulus (δ), the target stimulus (β), and overall inattention to the task (α).

To fit the model, we used the LMFIT Python package [62] (See Supplement for additional information).

Response time

For each behavioral trial, we measured the time between the end of a stimulus presentation and the time that a subject’s beak was detected in a behavioral response port. In Extended Data Figure 1 we found that the response time varied based on stimuli and cue conditions.

We used a Linear Mixed Effects model to statistically test the relationship between response time and cue probability plotted in Extended Data Figure 1B. We predict response time from the stimulus probability (given the cue) controlling for side bias and overall subject differences in response time (response time ~ stimulus probability + (1 + stimulus class ∣ subject)). We observe that as expectations increase, response times decrease (β=-0.151, SE<0.001, z=−188.011, p <0.001).

To parameterize the decay in response time as a function of the distance in the morph from the decision/class boundary, we fit the decay in response time (controlling for side bias) to an exponential decay function (Supplementary Figure 9). To account for side biases in decision making (e.g. the bird having a position preference when engaging with the behavioral apparatus that positions them further toward the left or right peckport), for each analysis we z-scored response time for each bird’s responses to each class (over a 500 trial block, to account for changes in side bias over time). In addition, in the analyses looking at response time relative to the morph interpolation point, we discounted the average bias of the cue for each trial. To account for variability in response time related to correctness, we ran all response time analyses only on correct trials.

We excluded two birds from analysis (B1426, B1170) who we observed did not exhibit the same decay in response time as a function of distance from the decision boundary (Extended Data Figure 1C). For syllable continua where a decay was observed (set at an r^2^ > 0.001 and decay range > 0.1 standard deviations), we found a strong relationship between the exponential decay constant and the psychometric slope (Extended Data Figure 1D; r^2^ = 0.421, p=6e-8, n=153).

Chronic electrophysiology

We used 32 or 64-channel Neuronexus Si-Probes (A4x2-tet-7mm-150-200-121, Buzsaki32, Buzsaki64, A1x32-Edge-5mm-20-177) implanted either unilaterally or bilaterally. Probes were coated with PEDOT using an Intan RHD Electroplating Board no more than one week prior to implant. Probes were mounted on 3D-printed drives (described in “Microdrives and head caps”), which were stereotactically implanted using the procedure outlined in “Electrode implant procedure”. Extracellular voltages were amplified and digitized at 30kHz using an Intan RHD recording headstage, output through an SPI cable through an electrically assisted commutator to an Open Ephys recording system.

Behavioral neural acquisition interfacing with PiOperant

Behavioral and physiology were synced using a custom-designed Raspberry-Pi-based system (PiOperant) for automating our behavioral paradigm and interfacing with the OpenEphys neural acquisition device (Supplementary Figure 10). PiOperant interfaces with our behavioral panel using the Python software pyoperant (https://github.com/gentnerlab/pyoperant). Behavioral states and audio signals were input and synced with OpenEphys over two HDMI inputs (digital and analog) and a ZMQ interface containing additional information about behavioral trials.

Microdrives and head caps

Microdrives and head caps (Supplementary Figure 11) were custom-designed over the course of this experiment and were printed using a FormLabs Form3 3D printer using FormLabs standard grey resin printed at 25-50 micron resolution. Microdrives were comprised of a drive, a shuttle, and a MiniTaps 6/16” 00-90 gold screw, hand-tapped and fastened to the drive with a brass nut. The screw was used to raise and lower the shuttle manually, at a depth of 282 microns per full rotation. Head caps were designed to be removable and enable moving probes further down, as well as easy to explant allowing re-use of probes.

Electrode implant procedure

Subjects were given analgesia by means of a 5mg/kg dose of carprofen (Rimadyl) administered intramuscularly. Animals were then anesthetized with a gaseous mixture of Isoflurane/oxygen (1-2.5%, 0.7 lpm). The scalp and feathers around the scalp were then removed and part of the skull over the y-sinus (the stereotactic reference sinus between the cerebellum and the two hemispheres of the brain) was visible. A craniotomy was opened above the recording site. A second craniotomy for the ground was then performed several millimeters away from the primary craniotomy. A platinum-iridium ground wire was then inserted in the craniotomy above the dura and glued to the skull. The baseplate for the head cap was then cemented (Metabond) to the skull. The durotomy was then performed in the original craniotomy, and the electrode, attached to the microdrive, was stereotactically lowered at a rate of no more than 100 microns per minute. Once the final site was reached, the microdrive was then cemented to the skull, and a silicone base was applied above the craniotomy to prevent infection. The head cap was then screwed into the baseplate, protecting the recording site and probe. The headstage was then attached to the outside of the head cap.

In some individuals, multiple implants were performed in serial when one probe failed by explanting and removing the first probe and microdrive, creating a new craniotomy in the opposite hemisphere and durotomy, and implanting a new probe/microdrive. In one individual, two probes/drives were implanted simultaneously one in each hemisphere.

Recordings and behavior blocks

Recordings were performed 24 hours per day in order to track individual neurons over days. Recordings consisted of (1) behavior blocks, in which subjects freely interacted with the behavioral apparatus, (2) a free feeding period, in which the behavioral apparatus presented food to the bird without requiring the bird to perform trials, (3) a passive playback block, in which lights were turned off and the birds were passively presented with stimuli, and (4) a sleep block, in which the lights were left off and no stimuli were played back.

We recorded from 10 subjects over a total of 222 days (5317 hours) of recordings. Chronically implanted subjects performed over 400,000 behavioral trials during recording. In addition, during the evening after the birds had completed their behavioral trials for the day we turned the lights out in the behavior boxes and passively played back the same morph stimuli to the birds totaling 1.2 million passive playbacks while recording.

Chronic behavior blocks

Chronic behavior blocks were matched to behavior blocks without physiology. The behavioral apparatus was left on throughout the day, allowing subjects to initiate trials through a peck in the central peck port. Trials were intermittently reinforced with a food reward and punished with the lights briefly turning off on incorrect trials. Using this paradigm, subjects performed several thousand trials per day.

Chronic passive playback blocks

At a set time at the end of each day, we turned the lights out in the bird’s operant conditioning block and passively played back the morph stimuli to the bird. The bird’s activity and sleep state during this time was not monitored. The silence interval between stimuli was randomly sampled between 1.1 and 1.5 seconds.

Spikesorting and merging over long-term chronic recordings

Spikesoring was performed over each 12-hour block of recording using Kilosort 2-2.5 [63] and SpikeInterface [64]. LFP was bandpass filtered between 300 and 6000 Hz and further normalized using common median referencing. To retain units across days/sorts, we additionally used an overlapping procedure to merge each neighboring pair of recordings together. To do so, we took the last 30 minutes of the previous recording, and the first 30 minutes of the following recording, and separately sorted that hour-long recording, which overlapped with the two larger recordings. We then computed the overlap between units in the overlapping recording and each of the two full recordings. Units were then considered to be the same unit if their “agreement” score (SpikeInterface; the spike coincidence of the two units) was above a set threshold (set at 0.5). Units from each of the larger recordings that were merged with the same unit in the overlapped recording were then merged, allowing the same unit to be tracked over multiple days (Extended Data Figure 5).

Stimulus alignment

Stimuli playback was aligned to neural data using a 1kHz sine wave sent from the MagPi behavioral control device to the OpenEphys acquisition board collected simultaneously with neural data, alongside a binary switch indicating the onset and offset of playback. An additional message giving information about the specific trial was sent over the local network via ZMQ.

Localizing units

Unit locations were defined as the location of the peak recording channel on which the unit was present. The recording channel was determined from its position within the shank, and the shank’s position relative to the stereotactic implant. Stereotactic implant locations were recorded relative to the Y-sinus between the cerebellum and two hemispheres of the brain, and the depth relative to the surface of the brain. Implant locations relative to nuclei were then determined relative to voxel mapping of the European starling brain atlas [65], as shown in Extended Data Figure 2A,B. We recorded from units in the primary auditory forebrain region Field L, two secondary auditory forebrain regions (the caudal mesopallium [CM] and caudal medial nidopallium [NCM]), and the caudal lateral nidopallium (NCL). Note that while NCL is a higher-order forebrain region implicated in visual and multi-modal working memory [66, 67, 68], our recordings were performed only on the most medial regions of NCL (Extended Data Figure 2B), which have been less well characterized. Sample unit spike trains for each nucleus are shown in Extended Data Figure 2C.

Neural feature representation and response similarity

We represented spike trains as vectors using the methods outlined in Figure 4A-F. In particular, a PSTH of spike trains was computed with 10ms time-bins, which was then smoothed with a Gaussian kernel with a σ of 25ms. Morphs were sampled at a resolution of 128 points. For physiological analyses, we reduced the sampling resolution, binning the 128 interpolation points into 16 points along the morph, thus the neural response vectors and similarity matrices are 100 time-bins by 16 interpolation bins, and 16 interpolation bins by 16 interpolation bins, respectively.

We computed neural response similarity as the cosine similarity of the Gaussian convolved spike vectors, which has been effectively used to find similarity in spike trains in the past [69]. A number of different similarity metrics could have been used in its place, for example, correlation coefficients [70, 71] and Euclidean distance between Gaussian convolved spike trains. We compared the cosine similarity to several other similarity metrics used in neural analyses including the correlation coefficient, Euclidean distance, and Manhattan distance, and found broadly similar results (Supplementary Figure 12).

Estimating a neurometric function from the similarity matrix

The neurometric function is computed on the basis of the similarity matrix and is detailed in Extended Data Figure 7. For each interpolation point, we took the average of the within and between-category similarity (SC1 and SC2) and took the ratio (SC1SC1+SC2) as the categorical similarity ratio. We then fit the same four-parameter logistic function as used in the psychometric function to the categorical similarity ratio as a function of the interpolation point.

Between subject neurometric versus psychometric variability

To determine whether between-subject variability in the slope of the psychometric was reflected in the neurometric, we performed a linear mixed effects model in Python/ neurometric slope ~ morph + psychometric slope + (1∣unit). Controlling for the morph and random effects of the individual unit, the relationship between the individual variance in the psychometric slope and neurometric slope is not significant and slightly negative (t(41171)=-5.75, p > 0.999, β=-0.04) and only explained an additional 0.03% of the variance (r^2^ = 0.778).

Assessing task relevance for units

Task relevance was measured as the categoricality of neural response. Unit categoricality was computed using the similarity matrix (as seen in Figure 4). The similarity matrix used to compute a unit’s categoricality was the mean cosine similarity matrix across interpolation responses, where the cosine similarity matrix was computed over average response vectors for each interpolation point.

Similarity matrices were divided into four quadrants, corresponding to the within-category similarities for each category, and the between-category similarities. Categoricality was computed as the mean similarity in the within-category quadrants of the similarity matrix (i.e. the top left and bottom right), minus the between-category similarities.

Subsetting task-relelvant units

We operationalized task-relevant, categorical, units on the basis of their response characteristics to the morph stimuli. Task-relevant units were determined by a threshold set in the categoricality metric. This threshold was set at a categoricality metric value above 0.1. These thresholds were set based on visual assessment of unit responses (Supplementary Figure 14) and similarity matrices. For reference, figures showing units sorted by categoricality metric are provided for subject (Extended Data Figure 8), morph (Extended Data Figure 9), and region (Supplementary Figure 15).

Comparing spike rate across units, cues, and morphs

The physiological analyses performed in the main text were performed over unit spike rates in response to the morph stimuli, where spike-rate was z-scored over the unit’s spike rates across all stimuli. A figure visualizing the main effect of cue and interactions between cue probability and stimulus class is shown in Supplementary Figure 13. In addition, we shuffled the cue labels to ensure that our results were not due to inherent sampling biases present in the data (e.g. a left cue is more predictive of a left morph point, thus more cue left to left morph point samples exist in the dataset).

Cue suppresses spike rate model comparison

An analysis of variance compared a baseline mixed-effects model with only a random intercept for the unit/stimulus to a model including both a random intercept for unit stimulus and the fixed effect of mode10:spike rate∼(1∣unit_stimulus)mode11:spike rate∼cue+(1∣unit_stimulus) where unit_stimulus is a variable representing a combination of the unit, and stimulus (e.g. neuron 8, stimulus BF, interpolation point 7).

Spike rate suppression increases with cue strength

We next tested the relationship between the cue’s predictive strength and the spike rate, again using an ANOVA between Linear Mixed Effects models. mode11:spike rate∼cue+(1∣unit_stimulus)mode12:spike rate∼cue+cup_p_right:side+(1∣unit_stimulus) where cue_p_right is the cue probability and side is the stimulus class.

Differences in spike rate as a function of time

In the main text, we compared differences in spike rate as a function of their cue (i.e. within versus between cue).

To ensure that the effects of spike rate modulation occur between cue conditions, and not only between cue conditions and the uncued condition (where the main effect of cue on spike rate is greatest) we did not include the uncued condition in the spike rate differences between cue conditions in Figure 5B. We only included units and stimuli where we had active and passive behavioral trials (n units = 4722). We then, for each unit and stimulus, took the average absolute difference in response vectors between trials for trials with the same cue, and trials with different cues. The difference between the average absolute difference between cues, minus within cues, will equal zero when there is no difference between cue conditions. To ensure that no factors exogenous to between-cue differences are causing this effect, in Supplementary Figure 16 we show the same analysis where cue labels have been shuffled within stimuli.

Within cue response similarity

For each unit and cue, we computed the cosine similarity matrix across each morph. Cosine similarity matrices were computed by taking the average cosine similarity across trials for each interpolation point (16) in the morph. Analyses were only performed over active behavioral trials, where the subject provided a response. We then contrasted the cosine similarity matrices across different cue conditions. Supplementary Figure 6H shows the average cosine similarity across left cues subtracted by the average cosine similarity across right cues. Blue in the top left of the plot (the orange bounding box) depicts less similarity in the predicted left class in left-cued trials. The reverse is true for the red in the bottom right. We statistically test this with a Linear Mixed Effects Model (Cosine Similaritycue left - cue right ~ side + (1∣unit) + (side∣unit). β=0.009, SE <0.001, p <0.001). We measured this relationship showing that predicted morph classes are less similar within class in Supplementary Figure 6J. Each point and confidence interval consists of the within-class similarity relative to the same unit’s response to uncued stimuli across trials. The negative relationship shows that higher probability cued trials exhibit less similar responses. We statistically test this with a Linear Mixed Effects Model (Cosine Similarityrel. cue none ~ prob. class + (1∣unit) + (prob. class∣unit); prob. β=-0.01, SE <0.001, p <0.001). In a similar manner as in Supplementary Figure 13 and Supplementary Figure 16, we repeated this analysis over the same data in which cue labels had been shuffled within unit/interpolation. In the shuffled condition, we observe that the effect is removed.

Acuity trade-off model

Central to the acuity trade-off model is the idea that focusing on task-relevant stimulus dimensions improves the precision of their representation, but at the expense of less accurate representations of irrelevant dimensions. Thus, a key feature of this model is that noise in neural measurement and resulting representations decreases for stimuli in expected regions of stimulus space. This decrease in noise yields neural responses that are more easily separable. For example, consider two simple, 1D Gaussian distributions over our signal dimension (i.e. the likelihood for two similar stimuli in our Bayesian model) separated by a fixed distance. Reducing measurement error (σ) decreases the overlap between the two distributions, and thus decreases the similarity (on average) between points sampled from those distributions (Extended Data Figure 3A). Somewhat paradoxically, when the mean of the distributions are sufficiently close in the 1D space, reducing σ results in an increase in similarity between points sampled from those distributions (Extended Data Figure 3B). For example, for a 1D Gaussian distribution when the mean is equal and only the variance is changed, the average difference between two points sampled is 2σ∕sqrt(π). The average difference between two points sampled from a Gaussian distribution with a standard deviation of 1 (N(0,1)) is 1.13, and with a standard deviation of 2 (N(0,2)) is 2.26. This effect is illustrated in the context of a single dimension from our current stimulus set (i.e., along a single syllable continuum) in Extended Data Figure 3C. To generate Extended Data Figure 3C, we sample from Gaussian distributions with standard deviation (σ) along the syllable continuum and take the average difference between samples at each point along the syllable continuum as similarity. Extended Data Figure 3C then shows the difference between two similarity matrices when sigma is varied to mimic a cue that predicts the left or right stimulus class along the continuum. In this scenario, the predicted increase in similarity along the diagonal contrasts with our empirical observations of a decrease in similarity, particularly along the diagonal within the cued stimulus class (Extended Data Figure 3D).

To account for this discrepancy, we can expand the 1-dimensional model to account for stimulus dimensions that are irrelevant to the task immediately at hand (i.e. on a given trial). This includes, for example, acoustic features that might be relevant for other syllable continua, response classes, and estimates of acoustic characteristics exogenous to the task-relevant stimuli such as background acoustics or other characteristics not relevant to classification, but which are also represented by neural responses.

Under the assumption that the capacity to measure all possible characteristics of the sensory environment is resource-limited, improved accuracy in representing a task-relevant stimulus dimension (here the syllable continuum) comes at a cost of decreased representation accuracy on task-irrelevant stimulus dimensions. We refer to this model as the “acuity trade-off model”, referring to the trade-off that exists in measuring and representing all features of our environment, requiring us to selectively attend to and represent only task-relevant features with high fidelity. Evidence for this model comes from both behavior, where limited attentional resources are available to keep track of different feature dimensions simultaneously [53, 54] as well as theories and empirical observations of neural computation, where neural coding has been observed to shift to decrease noise in stimulus relevant dimensions at the expense of increasing noise in stimulus relevant dimensions [72, 73].

By accounting for the task-irrelevant dimensions (Extended Data Figure 3E), we observe as task-irrelevant dimensions are added, the diagonal line (corresponding to the change in the similarity of nearby signals on the morph dimension) increases. This reflects expectation reducing the similarity of neural response along the diagonal in this model. The plots in Extended Data Figure 3E are generated in the same manner as Extended Data Figure 3C, except where additional task-irrelevant dimensions are added and, in contrast to the sharpening of the task-relevant dimension, the noise in measurement of the task-relevant dimension increases. A sample of points sampled from the 1 task-irrelevant dimension version of this model is provided in Extended Data Figure 3F, where noise in sampling in the task-relevant dimension is decreased for the cued signal, whereas noise in sampling is increased for the behaviorally irrelevant dimension when the signal is cued. This decrease in similarity when additional behaviorally irrelevant noise is present (here through the addition of behaviorally irrelevant dimensions) occurs because the most similar signals (near the diagonal) are most susceptible to becoming more distant in representation when task-irrelevant noise is increased.

A self-contained jupyter notebook (Google Colab link) is available to reproduce figure and aid in understanding the model through interaction.

MNE receptive fields

We used the Maximum Noise Entropy (MNE) model to calculate receptive fields for all task-relevant units [74, 55] (Extended Data Figure 4). MNE models were computed separately for each unit and predictive cue condition. Model fitting used a jackknife procedure, averaging estimates from four subsets of the training data to yield the final parameters. Model-predicted spiking probabilities were correlated with empirical spike trains on held-out trials to evaluate receptive field model performance (Extended Data Figure 4). The correlation values were then averaged across trials within a given cue condition.

To assess whether expectation modulates receptive fields, we trained MNEs for each neuron using the target syllable evoked responses from trials with a valid cue, i.e. trials where the cue accurately predicted the correct response to the subsequent target stimulus. We held out a subset of cue-valid trials equal to the number of cue-invalid trials (trials where the cue predicted the incorrect response) collected during the same recording session. We then used the model to predict responses on these held-out trials and used the correlation coefficient between predicted and actual responses (averaged across trials for a given predictive cue) as a measure of model performance (Extended Data Figure 4,C). To ensure consistency, we repeated these prediction tests using four different random samples of held-out cue-valid trials and averaged performance across these four samples. If the expectation does not alter the receptive field, then the models should fit responses in both the valid and in-valid conditions equally well. We used a linear mixed-effects model with a fixed effect for cue-validity and random effects for each unit’s identity and recording day to predict trial correlation values. We used paired t-tests for post-hoc comparisons.

To assess cue-dependent gain and tuning changes in the receptive fields, we first fit an MNE to the responses of a neuron in the no-cue condition, then refit responses for that same neuron to held-out data from the cued and no-cue conditions, using the no-cue refit as the null hypothesis for change across conditions. As proxies for the feature tuning and gain of the receptive field, we examined changes in the orientation and magnitude of the MNE feature vectors (the h and J terms), respectively.

In the context of the MNE model, the orientation of the feature vector in high-dimensional stimulus space defines the features to which the neuron is tuned. If those features change, then the vector orientation changes. To measure this change, we computed the cosine distance between feature vectors before and after refiting. Because the cosine distance is invariant to scaling, a change in gain alone will not alter the orientation of the feature vector. In contrast, the orientation of the full feature vectors does change significantly between the cued and the no-cue conditions (Linear Mixed Effects, cos-diff ~ cue-present + (1 ∣ unit) + (1 ∣ day); beta = 0.048, SE = 0.003, z = 17.838, p < 0.001). We also examined the orientation of the linear and nonlinear components (h and J) separately. In both cases, the feature vector orientation changes significantly more for the cued compared to the no-cue condition (Linear Mixed Effects, cos-diff-h ~ cue-present + (1 ∣ unit) + (1 ∣ day): beta = 0.068, SE = 0.007, z = 10.369, p < 0.001; cos-diff-J ~ cue-present + (1 ∣ unit) + (1 ∣ day): beta = 0.048, SE = 0.003, z = 18.042, p < 0.001). To examine the non-linear changes in more detail, we tried to look at changes in matched sets of non-linear features across conditions. This is difficult to do with strict assurance, but as a proxy to feature similarity, we restricted the analysis to only the closest pairs of eigenvectors (i.e. those with the minimum pairwise cosine distance from initial and refit MNE J matrices). Even here, the minimum change in these nearest eigenvectors is larger for the cued compared to the no-cue condition (Linear Mixed Effects, eig-min-cos-dist ~ cue-present + (1 ∣ unit) + (1 ∣ day): beta = 0.019, SE = 0.002, z = 8.974, p < 0.001). Collectively, these results consistently support our claim that expectation modulates the tuning of receptive fields to explicit stimulus features.

The foregoing feature vector orientation analyses rule out the possibility that the cue-dependent response modulation is explained entirely by changes in the receptive field gain, but changes in gain may still contribute to the modulation. To assay the change in receptive gain directly, we first compared the change in the magnitude of the linear feature vector (h), taken as the change in the L2 norm of the linear feature vector, after refitting to the cue and no-cue conditions. We find that the magnitude of the linear feature vector is significantly greater for the cued compared to the no-cue condition (Linear Mixed Effects, mag-diff ~ cue-present

• (1 ∣ unit) + (1 ∣ day); beta = 0.004, SE = 0.001, z = 4.752, p < 0.001). We used a similar logic to examine the change in magnitude of the nonlinear features across cue conditions by summing the absolute values of all eigenvalues for the J matrix. As with the linear feature, this proxy for the gain of the nonlinear features was significantly greater in the cue compared to the no-cue condition (Linear Mixed Effects, eig-diff ~ cue-present + (1 ∣ unit) + (1 ∣ day); beta = 0.186, SE = 0.031, z = 6.052, p < 0.001). We note that these cue-dependent shifts in the magnitudes of the linear and quadratic MNE terms correspond directly to the width and height of the tuning curves modeled in high-dimensional stimulus space, consistent with our conclusion that expectations enhance the selectivity of receptive fields. These more selective models produce fewer spikes with higher confidence, aligning with the expectation-induced firing rate suppression we observe in our empirical data.

To better understand how changes in the MNE receptive fields are related to the expectation-driven changes in the sharpness of the likelihood function, we attempted to replicate the empirical shifts in similarity (Fig. 6H) using the MNE models. To do this, we fit MNE models with target syllable data for each cue condition, then passed all target syllables through the model to generate predicted spiking probabilities for each stimulus and cue condition. We pooled the produced spiking probability vectors across models and separated them by unit, interpolation, and cued response (left vs. right). We then binned the spiking probability vectors for each unit, interpolation, and cued response into 16 bins spanning the target syllable continuum. We computed the cosine similarity between pairs of probability vectors within and between bins to generate a similarity matrix for each unit, interpolation, and cued response. We subtracted the left and right similarity matrices for each unit and interpolation and averaged the resulting difference matrices across all units and interpolations. We performed the analysis for both the linear (only h term included) and full MNE (h and J terms included; Supplementary Figure 3A&B, Supplementary Figure 4). As a control, we completed the same analyses using a version of each MNE feature vector that was shuffled before producing spiking probability vectors for each stimulus, then subtracted the resulting similarity matrix from that for the unshuffled MNEs. We used a linear mixed-effects model with a fixed effect for cue-validity and random effects for each unit’s identity and recording day to predict similarity values.

Stimulus decoder

For each population and syllable continuum, we trained a logistic regression with L2 regularization and balanced class weighting using scikit-learn. Models were trained to predict the bin in the syllable continuum where the stimulus occurred, where the stimulus continua were split into 16 bins (from the total 128 interpolation points along the syllable continuum). Because individual neurons were tracked longitudinally rather than in sessions, neural population representations are sparse; not all neurons are present during any given trial.

Activity for each neuron was represented as a histogram aligned to the 1-second playback of the target syllable (we used 20 time bins, each corresponding to 50ms). Time-varying spike rates were then z-scored for each neuron and clipped between −4 and 4 standard deviations to weight units equally in the subsequent PCA projection. Thus, at this stage in processing, population activity is represented as a matrix of shape (# trials, # units, 20). Neural populations were projected into PCA space (256 dimensions), fit to the training data (passive and uncued trials). The decoder was trained on all trials that were either uncued (i.e. with no cue, or with the uninformative cue) or passive playback trials where the bird was not performing the task. Analyses were then performed on the held-out cued data, specifically looking at prediction accuracy between cue-valid and cue-valid trials. Any population without at least one cue-invalid sample for each syllable-continuum bin was discarded.

Neurometric across cue conditions

For each unit and syllable continuum, we fit a neurometric model to spike response vectors (as described earlier). To compare changes in the neurometric as a function of the cue probability, we subsetted units/morphs where there existed enough trials for each morph and cue (at least one per stimulus) to compute a similarity matrix. We additionally excluded any model fits where the fit inflection point did not converge. In total, this yielded 1762 units in which we had well-fit neurometrics.

Morph prediction behavior

The primary task uses cue syllables to predict the likely correct response class associated with the target stimulus on the current trial (i.e. the left and right stimulus classes, corresponding to the left or right half of each morph), biasing perceptual decision-making toward a stimulus class. The morph prediction task instead, uses cue syllables to predict the morph, independent of the stimulus class. We used this behavior to assess whether the accuracy increases and the psychometric slope sharpens when a morph is expected.

We used three of the morphs from the original dataset (AE, BF, CG). The trial structure is as follows. At the beginning of each trial, initiated by a peck in the center peckport, one of three things would happen. (1) On 10% of trials a morph would be played without any cue. (2) on another 10% of trials, an uninformative cue would play, which equally predicts all three morphs. (3) On the remaining 80% of trials, an informative cue would play. The informative cue predicted a single morph (i.e. morphAE follows cueAE) 80% of the time, and the remaining 20% of trials following cueAE are divided between morphBF and morphCG. This yields trials with four sets of expectations. On the non-cued trial, the probability that any given morph will play after a center peck is only 1/30 (10% of trials have no cue, and there is a 1 in 3 chance of hearing each morph). After the uninformative cue plays, the probability that any given morph will play is 1/3 (equal changes for each morph). After an informative cue plays, the probability that the predicted morph will play is 0.8, and each of the non-predicted morphs is 0.1. To focus on challenging trials that define the psychometric slope, we sampled only the 32 points surrounding the midpoint of the original 128 points in the morph. In addition, we applied white noise over the stimulus (at 25% of the maximum amplitude) to keep the accuracy around 70% and avoid overtraining.

We retrained two birds on the modified task (B1590 and B1591) over approximately 7 weeks. B1590 engaged in 17069 trials, while B1591 engaged in 21879 trials. We compared the accuracy across cue conditions, as well as the psychometric slope. To compare accuracy, we fit a Linear Mixed Effects model, predicting correctness for each trial by the cue probability, controlling for the subject as a random intercept and slope. To compare psychometric fits, we used a bootstrapping approach. We estimated the psychometric slope by sampling 1000 trials (with replacement) from each morph for each subject and fitting the psychometric function to those trials. We repeated this 1000 times and took the mean psychometric slope parameter. This method accounts for differing numbers of trials across cue conditions, and stochastic error in model fitting. To then statistically test the relationship between cue probability and psychometric slope, we z-scored the psychometric slope within subject and morph. We then computed the Pearson correlation between cue probability and z-scored slope. Finally, we compared our observed correlation between cue probability and slope to a distribution generated by shuffling the cue probabilities (shuffled within subject and morph) 1000 times.

Statistics and Reproducibility

No statistical method was used to predetermine sample size. No data were excluded from the analyses. The experiments were not randomized. The Investigators were not blinded to allocation during experiments and outcome assessment.

Extended Data

Supplementary Material