Riffusion - Music Made From AI Stable Diffusion

https://www.riffusion.com/about

  Quote

You've heard of Stable Diffusion, the open-source AI model that generates images from text?

photograph of an astronaut riding a horse

Well, we fine-tuned the model to generate images of spectrograms, like this:

funk bassline with a jazzy saxophone solo

The magic is that this spectrogram can then be converted to an audio clip:

Really? Yup.

This is the v1.5 stable diffusion model with no modifications, just fine-tuned on images of spectrograms paired with text. Audio processing happens downstream of the model.

It can generate infinite variations of a prompt by varying the seed. All the same web UIs and techniques like img2img, inpainting, negative prompts, and interpolation work out of the box.

Expand  

 

 

 

 

Edited December 15, 2022 by luke viia

Genius.

The samples on that 'about' page are amazing, see the 'Looping and Interpolation' section, where it interpolates between 'typing' and 'jazz'

https://www.riffusion.com/about/typing_to_jazz.mp3

Another interpolation between 'church bells' and 'electronic beats'

https://www.riffusion.com/about/church_bells_to_electronic_beats.mp3

Edited December 17, 2022 by zazen

                                                     

 This is freaking RIDICULOUS. Some highlights of Experimental electronic noir waltz, into avantgarde atonal string quartet, into rhythmic bucket clattering. The thing just does an effortless DJ set for you. Very inspiring! (my post formatting is super messy) 

Edited December 17, 2022 by chim

Can someone try black metal IDM?

alright alright, how long until I dont have to work or do anything ever again?

  On 12/17/2022 at 10:55 PM, logakght said:

Can someone try black metal IDM?

I tried all evening, with various followup prompts to try and nudge it, but all I got was BOCCY variations on some grunge guitar riffs and rock drums. The dataset is fairly limited to a low-mid tempo, and not a lot of material from black metal or blast beasts. I'm pretty sure they put various constraints to keep the tempo and keys similar across variations.