Jump to content

Avatar

Spleeter - software to generate "stems" from final mix


  • Please log in to reply
12 replies to this topic

#1 redion1992

redion1992
  • 11 posts

Posted November 3, 2019 - 12:43 PM

This came across my timeline today, and I immediately thought it would be useful for this community.

 

https://github.com/deezer/spleeter

 

This is a tool that claims to be able to split sources into "stems" via similar techniques to some of Harmonix's post-RB4 DLC output, by isolating the frequencies of each part. The difference is that it attempts to do so automatically using machine learning, with a dataset already having been created.

 

This may be useful in producing better-sounding customs. Hopefully it will help some of you still out there authoring!



#2 jerrylive365

jerrylive365
  • 85 posts
  • Location
    California

Posted November 3, 2019 - 05:01 PM

Very interesting. Will have to give this a look.



#3 appledoo

appledoo
  • 22 posts
  • Location
    California

Posted November 4, 2019 - 03:35 AM

i didn't know I needed this in my life


am I late to the party?


#4 doa

doa
  • 252 posts
  • Location
    Northern CA

Posted November 4, 2019 - 05:51 AM

I'm interested to see what kind of results people get with this software.



#5 MrPrezident

MrPrezident

    C3U Moderator

  • 1,002 posts
  • Location
    Nanaimo, BC, Canada

Posted November 4, 2019 - 10:49 PM

Very cool!  Thanks for sharing this info.


FORUM THREAD  /  MY CUSTOMS -  "Almost Everything from Al Green to Ween" since 2016

p89ZznR.png KhDyqmU.png
 
 


#6 expertwin

expertwin
  • 149 posts

Posted November 5, 2019 - 06:31 AM

I have trouble getting this to work. Either the audio files aren't produced or they end way too early.

#7 Caserol23

Caserol23
  • 17 posts

Posted November 6, 2019 - 12:40 PM

This came across my timeline today, and I immediately thought it would be useful for this community.

 

https://github.com/deezer/spleeter

 

This is a tool that claims to be able to split sources into "stems" via similar techniques to some of Harmonix's post-RB4 DLC output, by isolating the frequencies of each part. The difference is that it attempts to do so automatically using machine learning, with a dataset already having been created.

 

This may be useful in producing better-sounding customs. Hopefully it will help some of you still out there authoring!


Hmmm... Linux Based?! - this is real challenge too hunt down the bugs, very thanks for this  :smug:  :smug:  :smug:  :rock:  :rock:  :rock:


- Official.Caserol23 @ FB

- Caserol23 @ Twitter


#8 yaniv297

yaniv297
  • 307 posts

Posted November 8, 2019 - 09:40 AM

I've been using RX7 for this and results have generally been good, especially for drums and vocals.


Yaniv's Workshop - including all my work (hopefully) neatly organized

Spotify Playlist of all my releases

 

Notable Projects:

The National Homecoming Festival! Huge project featuring LOADS of tracks from the National, plus a bunch of openers!

11 Songs of Love... and One Song of Disappointment! Featuring Bruce Springsteen, R.E.M, Mac DeMarco and more!

Primavera Sound project: Day One, Day Two. Featuring The Cure, Talking Heads, Pavement and more!

Arcade Fire Project
..

WIP Thread

Hit me up if you wanna collab on any of those!


#9 StackOverflow0x

StackOverflow0x

    Wii Guy

  • 1,421 posts

Posted November 10, 2019 - 05:58 AM

I tried this and I'm just blown away by how easily it outputs great results! Drums and vocals have turned out really well! Bass too. Basically free karaoke versions of songs with this tool for the most part. And for the bass extraction, that should help for charting since it's usually the hardest to hear and comes out fairly well too.



#10 Shroud

Shroud
  • 204 posts
  • Location
    EU

Posted November 12, 2019 - 06:07 PM

Bah, doesn't work for me...

 

I always get "WARNING:spleeter:cannot reshape array of size 0 into shape (0)" which sounds like a low-level python error, but all I am doing is their basic suggested command for separating 2,4 or 5 stems.



#11 StackOverflow0x

StackOverflow0x

    Wii Guy

  • 1,421 posts

Posted November 13, 2019 - 06:02 AM

I know it sounds complicated to set up, but I just followed the instructions exactly as they provided. Download the right version for whatever version of Python you have installed. Download Conda. Use git to get everything. A more common error I've seen is the input path in the arguments to run it being incorrect. You can drag and drop a file into the console window to get the path in there if that's faster.



#12 Shroud

Shroud
  • 204 posts
  • Location
    EU

Posted November 13, 2019 - 07:10 AM

I know it sounds complicated to set up, but I just followed the instructions exactly as they provided. 

 

They don't really have good instructions. I know this is open source, which is great, but often it means the documentation is non-professional.

 

However... eventually I DID get some results, and they are surprisingly good!

 

Their example file worked, so I thought there might be some format requirements, which I didn't notice from the documentation. Comparing the example file with the few mp3 I have and I was using to test the program, I noticed that all my mp3 had variable bit rate, while the example mp3 had fixed bitrate.

 

So I tried a fixed-rate mp3 and it worked! Well, partially... first time it crashed in the middle of processing (but at least it started), perhaps because of lack of RAM. I know I have an old PC with only 4Gb of RAM, and I know that some intensive processing must be required for this task... but really how is it possible that to process a ~4 MEGA byte file the program uses 4 GIGA byte of memory? It actually does, because it my second try, after closing every possible other program in order to free more RAM, I monitored spleeter and it went up to more than 3.5 Gb of usage at some point.

 

This time it didn't crash, but it got stuck at some point, however it did manage to get as far as producing some of the supposed audio files: vocals, drums and piano (I was trying the 5-stem processing version). The third one is empty, but it could be that it got stuck after creating the file but before filling it with data. Vocals and drums are VERY good. I don't know if the models used by the program are for a specific music genre, but I tried it with a pop-synth song that I have no idea if it's supposed to be a good or bad choice for spleeter ("Adamski - Killer"). If you're familiar with the song, the drums are synthesized not real acoustic drums, but they came up pretty well. The voice is also very well isolated, not perfectly but essentially you would only hear a few "beeps" around the vocals, not processing artifacts but small sounds which are in the original audio, which apparently confused the program a bit, but they are not particularly annoying.

 

The processing is slow (more than half an hour but it's hard to tell when it got stuck exactly) because I am using the CPU option and, as I said, I have an old PC. There is an option to use the GPU which is claimed to take a time shorter than the song duration itself, but requires to have an Nvidia graphic card specifically, I have a GeForce 550 Ti but it's unbranded so I am not sure it will work.



#13 Shroud

Shroud
  • 204 posts
  • Location
    EU

Posted November 14, 2019 - 07:53 AM

More tests and more failures... with variable bitrate mp3 it just never works for me, and with fixed bitrate mp3 it always gets stuck between generating stem files. Sometimes no stem files are generated at all, some other times I get 3 or 4 (voice, drums, other and I may get piano or not), while it never generates the bass stem. There is a slim chance it's still processing but I let it go for over 2 hours and it didn't show a sign of life...

 

One interesting thing is why does it generate the "other" stem (basically the background track) before other stems. I would have expected that the algorithm identifies and extracts stems of known instruments first, and then saves whatever remains into other.wav.

 

I focused on testing the 5:stems option but probably I should use the 4:stems option instead, because now I understand that the "piano" stem in 5:stems mode is really looking for piano sounds in the audio, possibly including electric piano but not keyboards in general, so if the song does not include piano specifically, the 4:stems mode should be used instead.

 

And yes, unfortunately there is no standard way to extract a guitar stem, apparently due to guitar sounds being too variable in music. But the whole software is based on machine learning, and with a proper dataset (meaning, if you have plenty of guitar stems from other songs, which I don't) it should be possible to train the algorithm to recognize the guitar as well. Perhaps this would need to be done multiple times for different guitar sounds (i.e. with various combinations of the usual guitar effects such as distortion, reverb, chorus, flanger, etc...), but I would bet that if Spleeter manages to attract enough interest, then people will start to train/create and share new modes for extracting other instruments.

 

For RB3 custom authors, there is great potential here... I have my own issues with running the algorithm but no reason to believe the same happens to others, so if you don't count those issues, extracting the stems literally takes one single command line, and you don't need to mess with any options other than picking the model you want - to recap, the current models are 2:stems (vocals + rest), 4:stems (vocals + drums + bass + rest) and 5:stems (vocals + drums + bass + piano + rest).






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users