Experts at the University of Washington have created an artificial intelligence that can produce music with the help of visual prompts, an article from Science Daily said. Applying machine learning, Audeo recreates silent piano performances.
The system can play pieces from watching a top-down, silent video of a pianist playing the instrument. When tested using recognition applications like SoundHound, audio generated by Audeo was detected 86% of the time.
It is important to note that these applications have a 93% success rate when identifying audio from source videos, said the paper, which is co-authorized by Kun Su and Xiulong Liu, who are both doctoral students in electrical and computer engineering.
Senior author and applied mathematics and electrical and computer engineering assistant professor Eli Shizerman said, “To create music that sounds like it could be played in a musical performance was previously believed to be impossible.”
“An algorithm needs to figure out the cures, or ‘features,’ in the video frames that are related to generating music, and it needs to ‘imagine’ the sound that’s happening in between the video frames,” he added.
The key to creating this system, according to Shizerman, is an imaginative and precise AI. The researchers thought it “a surprise” that the system was able to recreate pieces “that sounded pretty good.”
The AI takes different steps to work out the visual cues and convert them into music. It looks at the keys pressed by the pianist and plots them in a diagram that a synthesizer can decipher. The data is cleaned up and polished.
During the finishing process, the developers add in some information that will allow the AI to generate the pieces the way it needs to be. This includes the strength and length of pressing each key.
The second step is necessary for accuracy. According to Shizerman, “If we attempt to synthesize music from the first step is like how a teacher goes over a student composer’s music and helps enhance it.”
The system was created to see how artificial intelligence can recreate musical pieces. Moreover, the researchers are hopeful that the system will be able to enhance the way humans “interact with music,” especially in teaching students how to play the piano.
Worth noting is that the system was only tested on renowned pianist Paul Barton’s videos. Researchers highlighted that more research using other musician’s videos is required to further assess Audeo’s capabilities.