Close
AI Music

AI, with a taste for music, is here to stay

Zijing Fu

posted on December 2, 2022 9:33 pmEditor : Wang Boyuan

In 1787, Mozart invented a game which involved using a dice to compose music. The procedure involved preparing close to 300 measures of music and using a dice roll to determine at random which measures would be used in the final composition. The result would still need to adhere to certain guidelines, such as a predetermined order for rolling the dice; for instance, if he was rolling for a beginning measure, the dice would only select from all beginning measures. As a result, when Mozart finished putting everything together, the piece would flow smoothly.

A Mozart dice game. The score consists of 13 separate sheets containing two tables and 176 bars of music.
A Mozart dice game. The score consists of 13 separate sheets containing two tables and 176 bars of music.

Centuries later, AI developers step up the game by using AI to replace the dice's function and even Mozart's function. For instance, Mubert API, an AI tool that "arranges samples into composition," enables automatic music generation after users enter just a few keywords or a straightforward prompt. According to Mubert, the Text-to-Music tool analyzes and chooses pertinent sounds (which musicians created) and then creates arrangements and compositions from them.

Users have been having fun with the tool on social platforms, creating tracks such as “astronaut riding a horse”, “refrigerator floating in a pond”, etc. The outcomes are surprisingly good, although the tracks might not precisely reflect the prompt. It is understandable given the abstractness of music by nature.

The tags seem to be having a bigger influence on the generation outcome than the prompts. 
The tags seem to be having a bigger influence on the generation outcome than the prompts. 

Mubert isn’t the only service trying to AI generate music. As early as 2017, Aiva, a virtual composer created by Aiva Technologies, was recognized by SACEM, France and Luxembourg authors’ right society, as a composer. Aiva’s works would be used in ads, films and games, with clients including Nvidia, Vodafone, TED, etc. according to Avia’s website. Avia lets users compose with pre-defined styles, or “compose with influence”, which means uploading a sample for Avia to emulate.

Google also has a machine learning project dubbed “Magenta Studio”, which includes a collection of music plugins. There are currently x features in Magenta Studio, making the plugins capable of generating notes, adjusting the timing and velocity of an input drum pattern, creating grooves based on the rhythm of any input or merging submitted samples.

Microsoft’s AI music understanding and generation project, “Muzic”, realizes symbolic music understanding, automatic lyrics transcription, song writing, lyric generation, lyric-to-melody generation, melody generation, accompaniment generation, and singing voice synthesis. A lot of the work has been done by researchers at Microsoft Research Asia’s Beijing office.

Chines tech companies such as NetEase, Baidu, ByteDance and Tencent are also joining the AIGC for music field. Earlier this year, NetEase, an internet giant known for its games and music apps, released “Tianyin”, an AI composer and lyric generator service. According to Tianyin’s website, since there is no separate copyright for music composing in Chinese laws, the works generated by Tianyin can be used by users commercially without any additional authorization.

Tencent’s “Music XR Maker System” targets a more niche scenario — according to Dong Zhi, the Head of Computer Vision at Tencent Music Tianqin Lab, the system is capable of providing songs, dances, lip-syncing, expressions, movements, lighting, cinematography, choreography, etc, for digital humans. Such capabilities have been applied to Xiaoqin, a digital human created by Tencent.

Baidu has been tapping into the AIGC field for several years now, and had once stunned users with its image-to-music AI composer in 2016. The company now focuses on digital humans, and the accompanying AIGC features, which included music and lyrics.

ByteDance, who owns viral social app TikTok, bought its way into the AI music world. The company acquired Jukedeck, a British firm focusing on AI composing, in 2019. Notably, Jukedeck’s co-founder and CEO Ed Newton-Rex, who joined TikTok’s AI Lab at the time recently found a role at Stability AI, the company behind Stable Diffusion. “Generative AI is no longer a moonshot. It works, and it’s here to stay,” said Newton-Rex on LinkedIn.

According to Q-Bit, a tech media, AIGC's startups and commercial rollout plans will continue to increase in the next 2-3 years, and the market size will potentially exceed one trillion RMB by 2030. Let’s hope Newton-Rex is right, that Generative AI, after years of brewing, is finally seeing explosive growth in terms of technological advancement, demand and applications. 

In China, AI composing technology has been brewing for some time, but the market penetration and overall popularity seems to be relatively low, which can be attributed to the rather infantile performance and functions, or low demand from the consumer market. For musicians who have professional creative ability, lyric writing or song writing is a way to express and communicate their personal emotions, so when AI took over, their expressions would lose their meanings. AI music tools could be used to inspire, not replace musicians, especially when they need an efficient and simple way of arranging and producing.  

To put AI music in the context of a more viral AI field, AI composing, along with other AI music capabilities, are more like assisted driving rather than autonomous driving. They assist musicians by helping them automate their work flows, get inspirations, or roll the dice like Mozart. Human creators are still at the center of the whole process. For example, Mubert API buys sounds created by musicians for their AI composer to use. AI music, at least at its current stage, can not be done without the help of human contributors. 

Furthermore, since AI composers fundamentally learn from mainstream music works, it is questionable how far it has to go to generate something truly generic and has true artistic value, instead of just catering to the ears of the audience. Like a user who experienced AI composing pointed out, without creativity, music are just disposable.