| News

Cow Goes Moo: Disney’s Artificial Intelligence Project Gets Ears

Artificial intelligence-based system associates images with sounds

Girl and highland cow

Researchers at Disney get up to all sorts of scientifically laudable projects.

It would be naive to imagine that their primary objective is not the pursuit of further profits for the corporation, but if spinoff developments results in better entertainment experiences for the deaf, there is at least a net gain to the ever-belaboured human condition.

Nevertheless, it is tempting to wonder why they aren’t concentrating their considerable resources to rolling Miley Cyrus’s bloody tongue back into her head, getting some clothes on her and bringing back Hannah Montana.


The cow goes “moo.” The pig goes “oink.” A child can learn from a picture book to associate images with sounds, but building a computer vision system that can train itself isn’t as simple. Using artificial intelligence techniques, however, researchers at Disney Research and ETH Zurich have designed a system that can automatically learn the association between images and the sounds they could plausibly make.

Given a picture of a car, for instance, their system can automatically return the sound of a car engine.

A system that knows the sound of a car, a splintering dish, or a slamming door might be used in a number of applications, such as adding sound effects to films, or giving audio feedback to people with visual disabilities, noted Jean-Charles Bazin, associate research scientist at Disney Research.

To solve this challenging task, the research team leveraged data from collections of videos.

“Videos with audio tracks provide us with a natural way to learn correlations between sounds and images,” Bazin said. “Video cameras equipped with microphones capture synchronized audio and visual information. In principle, every video frame is a possible training example.”

One of the key challenges is that videos often contain many sounds that have nothing to do with the visual content. These uncorrelated sounds can include background music, voice-over narration and off-screen noises and sound effects and can confound the learning scheme.Girl and highland cow, artificial intelligence

“Sounds associated with a video image can be highly ambiguous,” explained Markus Gross, vice president for Disney Research. “By figuring out a way to filter out these extraneous sounds, our research team has taken a big step toward an array of new applications for computer vision.”

“If we have a video collection of cars, the videos that contain actual car engine sounds will have audio features that recur across multiple videos” Bazin said. “On the other hand, the uncorrelated sounds that some videos might contain generally won’t share any redundant features with other videos, and thus can be filtered out.”

Once the video frames with uncorrelated sounds are filtered out, a computer algorithm can learn which sounds are associated with an image. Subsequent testing showed that when presented an image, the proposed system often was able to suggest a suitable sound. A user study found that the system consistently returned better results than one trained with the unfiltered original video collection.

Combining creativity and innovation, this research continues Disney’s rich legacy of inventing new ways to tell great stories and leveraging technology required to build the future of entertainment.

Source: Eurekalert/Disney Research
Image: Pixabay/ATDSPHOTO

Comments are closed.

Our weekly newsletter

Sign up to get updates on articles, interviews and events.

Join Trebuchet on PatreonExclusive content and full media memberships available

This is the patreon page for Trebuchet podcast and website. We publish a beautiful printed magazine biannually and release an irregular podcast on contemporary art every month (or so). 

Our website is updated every other day with new art news, art criticism and much more. Become a backer and join us in discovering new forms of art that raise the heart rate and electrify the mind.  

Cookies and GDPR

By visiting this site you agree that you're fine with us using cookies. (read cookie policy)