Experimental design and data analysis scheme. Credit: Communications Biology (2025). DOI: 10.1038/s42003-024-07434-5
When a person’s hearing and vision are uncompromised and function at a relatively high level, the human brain is able to take in various sights and sounds from any environment and seamlessly allow said person to perceive what’s happening around them.
But how does it work? Spoiler alert: There’s more than meets the eye.
A new study led by Western researchers reveals how the brain processes multi-sensory audiovisual information by developing a new 4D imaging technique, with time as the fourth dimension. The researchers targeted the unique interaction that happens when the brain is processing visual and auditory inputs at the same time.
Computer science professor Yalda Mohsenzadeh and Ph.D. student Yu (Brandon) Hu used functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG) on study participants to analyze their cognitive reactions to 60 video clips with corresponding sounds. The results revealed that the primary visual cortex in the brain responds to both visual and low-level auditory inputs, while the primary auditory cortex in the brain only processes auditory information.
Mohsenzadeh, an expert in AI and machine learning, says this asymmetry or unevenness in the brain has implications for building more biologically plausible multi-sensory neural networks. A neural network is an AI method that teaches computers to process data.
“Originally, neural networks were loosely inspired by the human brain, but nowadays, AI doesn’t necessarily try to mimic the human brain. They focus more on optimizing results and related tasks based on cutting-edge AI frameworks,” said Mohsenzadeh, a core faculty member of the Western Institute for Neuroscience. “We are interested in making better AI models by understanding how the brain works because we can still get some new ideas from the brain. And we should use them.”
The findings, published in the journal Communications Biology, will likely impact how artificial intelligence (AI) algorithms process audiovisual data moving forward as many programs currently analyze real-world things like “cats,” “beaches” and “toasters” with images and video, but not sounds.
‘Real’ world still inspires
For the study, Mohsenzadeh and Hu presented a series of one-second natural setting or real-world video clips to participants in an MRI scanner and an EEG session to gather both spatial and temporal information about brain activity.
Combining the two brain imaging techniques (fMRI for spatial information and EEG for temporal resolution) produced a highly detailed, 4D map of neural responses for the human brain.
While this is an important finding for basic science, the new technique should also improve AI models by showing an integration of sensory information with visual information yields far better results.
“Our brain is optimized for processing visual information. A fact well-established in neuroscience,” said Mohsenzadeh. “Many studies have also shown how auditory information is processed in the brain, but vision plays a dominant role in shaping perception. We wanted to see how visual and auditory cortices share information, and it is very clear that the part of our brain processing visual data is doing the heavy lifting and gathering way more information than the part processing the audio.”
With this new understanding and 4D map of the brain, Mohsenzadeh and her team in the Cognitive Neuroscience and Artificial Intelligence Lab will look to improve deep neural network models, especially those designed to do audiovisual related tasks.
More information:
Yu Hu et al, Neural processing of naturalistic audiovisual events in space and time, Communications Biology (2025). DOI: 10.1038/s42003-024-07434-5
Provided by
University of Western Ontario
Citation:
Brain 4D imaging technique drives AI audiovisual analysis (2025, March 18)
retrieved 19 March 2025
from https://medicalxpress.com/news/2025-03-brain-4d-imaging-technique-ai.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
