What our brain hears beyond words
How do we know that a sentence was said by a woman? How do we know that what she told us was a question? Researchers from the University of Zurich and the NCCR Evolving Language described the complex processes happening in our brain, that allow us to understand the information contained in pitch.
By the NCCR Evolving Language
When we listen to someone speak, we hear more than their words! Their pitch also reveals key paralinguistic information, that can tell us more about the speaker’s identity – their age, gender, etc. – but also about the meaning of a sentence, allowing us to differentiate between questions and statements, for example.
To understand what someone is saying, our brain transforms the auditory information into abstract and invariant representations that interface with our conceptual system. But how does that work for pitch? In a new study published in Communications Biology, a group of researchers from the University of Zurich and the NCCR Evolving Language studied how the brain registers this important information. Using magnetoencephalography (MEG), they were able to precisely pinpoint the pathways involved in the processing of pitch. And to their surprise, these were not the same as the ones used for the rest of the auditory information!
Pitch-varying stimuli
To study how pitch is processed in the brain, the researchers had to create speech signals which only contained variations in this paralinguistic cue. For this, they created multiple stimuli, not spoken by a real person, tweaking the pitch height and dynamics of the auditory signal. “This allowed us to carefully control the acoustic parameters, so as to get rid of potential confounds,” says Chantal Oderbolz, first author of the study. The varying height of the signal represented the gender of the fictional speaker, and the dynamics of the signal, whether the stimulus could be understood as a question or a statement.
The synthesized stimuli were then played to 34 participants, while their brain activity was recorded by an imaging machine that measures neuromagnetic signals. After each stimulus, participants classified it as either a question or an affirmation.
The researchers then analyzed the recorded data with computational methods, to determine what exactly was happening in the brain when we hear pitch. “Using Representational Similarity Analysis (RSA), we obtained (dis)similarity maps from our data, i.e. how different stimuli relate to each other, which we compared to models of what these maps would look like if the brain was representing the stimuli in an acoustically detailed way or just as the sentence type,” Oderbolz explains.
A pathway in the right side of the brain
With their in-depth analysis of the participants’ brain activity, the researchers determined that the first processing stages of the signal in early auditory regions on the right side of the brain care strongly about the acoustic details of the stimuli. “This means that at this point, the brain represents information about whether the speaker was a male or a female, and how clearly the speaker signaled the sentence type,” Chantal Oderbolz underlines. In contrast, later stages of processing, happening in right anterior extra-auditory regions, only care about the sentence type category (question or affirmation). “This shows that there is a transformation from an acoustic to an abstract representation during the processing of pitch,” the researcher adds. And the more precise this transformation was, the better listeners were able to classify the stimuli correctly.
After this transformation, the data shows exchanges between the right and left sides of the brain, which could indicate the transmission of the final representations, determining the listener’s classification of the signal.
A hierarchical process
The results show that pitch is processed by the brain in a hierarchical fashion. “This means that the brain builds meaning step by step, from simple to complex, turning sounds into meaningful units,” says Oderbolz. Such an organization is already well established in spoken word comprehension, but also vision and motor control.
And now, this has been extended to pitch processing. For the researcher, this adds to the theory that this may be a general organizational principle the brain uses. “It has also been observed in non-human primates, which has interesting evolutionary implications,” she adds.
In the future, the researchers would like to take their analysis further, in a less controlled environment. “Speech perception doesn’t happen in isolation: people usually speak in full sentences that are very rich in information and context,” Oderbolz says. “A future research avenue would take into context how pitch is processed in a more naturalistic setting, and what the consequences are for perception and behavior,” senior author Martin Meyer completes. In his new Vivid Sound Lab, carrying out these types of studies that investigate all sorts of language comprehension and production in a naturalistic environment will be possible.
Using magnetoencephalography
For this experiment, the researchers used a magnetoencephalography (MEG) machine located at the Human Neuroscience Platform (HNP) in Campus Biotech, Geneva – the only one available to researchers in Switzerland to this day.
MEG has the ability to measure the brain’s activity with a high temporal resolution. It records the very small magnetic fields that are generated by the electrical activity of neurons. “Because of this, MEG is great for studying speech perception since it’s such a fast and dynamic process that involves many different regions in the brain,” says Chantal Oderbolz.
Though the technique is superb for temporal analysis, it is a bit less precise for spatial information. Therefore, the researchers combined their MEG data with magnetic resonance imaging (MRI), to be even more confident about where exactly the signals originate from.
“Though it has a lot of advantages, a lot of factors can make participants incompatible with the machine,” shares Oderbolz. Indeed, the signal is sensitive to anything magnetic or metallic, so people with implanted devices, dental retainers, tattoos, and even recently dyed hair, could not participate in the study. “The list is long, and it was challenging to find people who checked all the criteria,” adds the researcher. “But it was definitely worth the effort, because we now have a great MEG dataset, and really cool results, adding to our understanding of how we transform speech into meaning,” she concludes.
