- | 9:00 am
Here’s what sound looks like, according to artificial intelligence
A new web app transforms music into a wild light show using generative AI.
Unless you’ve experienced it yourself, the phenomenon of synesthesia is nearly impossible to grasp. The neurological condition in which one sense can trigger an experience through another (seeing sounds or feeling a color, for example), is relatively rare—just one in approximately 2,000 people are synesthetes. But a new app from Amsterdam-based studio Modem helps the average person understand how synesthesia works. It’s doing so by plugging a music synthesizer directly into Stable Diffusion to see what the AI comes up with.
The result of this approach doesn’t work or look anything like Lensa, DALL-E, Midjourney, or Stable Diffusion, despite using the same underlying technology. OP-Z Stable Diffusion—as the app is called—translates music into an explosion of animated images. All you need to make it work is Teenage Engineering’s OP–Z media synthesizer connected to the USB port of your computer.
[Image: Modem x Bureau Cool]
[Image: Modem x Bureau Cool]
But no, this is not your parents’ rave animation. While the result may remind you of a more sophisticated version of Apple’s music screen savers, there’s more here than meets the eye.
“It’s not randomly creating images after the music,” says Bas Van De Poel, cofounder and innovation director at Modem. “Otherwise it would just be like a Winamp visualization.” If you weren’t a sentient human being in the ’90s, Van De Poel is referring to the MP3 player that showed surreal, brightly colored animations following the rhythm of any given song. Instead, he tells me, OP-Z Stable Diffusion follows the principles of synesthesia.
[Image: Modem x Bureau Cool]
A TRIP INTO SOMEONE’S BRAIN
Synesthesia itself is wild. For the people who have it, it actually makes them see things that don’t really exist. Some people with synesthesia will see a color flash before their eyes when they hear specific words. Others will listen to music and experience different shapes, colors, and motion right in front of them.
Working with creative developer Bureau Cool, Van De Poel and his team used the synesthesia theories developed by medical experts and others who have documented or experienced the condition—like Dr. Richard E. Cytowic, psychologist Stephen Palmer, and composer Olivier Messiaen—to create the light show. The app transforms pitch, key, notes, rhythm, and BPM into text prompts that are fed into Stable Diffusion.
Featured Videos
Today's Top Stories:
More Top Stories:
FROM OUR PARTNERS
