Jump to content

Avatar

Spleeter - software to generate "stems" from final mix


  • Please log in to reply
25 replies to this topic

#1 redion1992

redion1992
  • 11 posts

Posted November 3, 2019 - 12:43 PM

This came across my timeline today, and I immediately thought it would be useful for this community.

 

https://github.com/deezer/spleeter

 

This is a tool that claims to be able to split sources into "stems" via similar techniques to some of Harmonix's post-RB4 DLC output, by isolating the frequencies of each part. The difference is that it attempts to do so automatically using machine learning, with a dataset already having been created.

 

This may be useful in producing better-sounding customs. Hopefully it will help some of you still out there authoring!



#2 jerrylive365

jerrylive365
  • 98 posts
  • Location
    California

Posted November 3, 2019 - 05:01 PM

Very interesting. Will have to give this a look.



#3 appledoo

appledoo
  • 22 posts
  • Location
    California

Posted November 4, 2019 - 03:35 AM

i didn't know I needed this in my life


am I late to the party?


#4 doa

doa
  • 262 posts
  • Location
    Northern CA

Posted November 4, 2019 - 05:51 AM

I'm interested to see what kind of results people get with this software.



#5 MrPrezident

MrPrezident

    C3U Moderator

  • 1,062 posts
  • Location
    Nanaimo, BC, Canada

Posted November 4, 2019 - 10:49 PM

Very cool!  Thanks for sharing this info.


FORUM THREAD  /  MY CUSTOMS -  "Almost Everything from Al Green to Ween" since 2016

DONATE AT CHARTATHON.COM
 
 


#6 expertwin

expertwin
  • 157 posts

Posted November 5, 2019 - 06:31 AM

I have trouble getting this to work. Either the audio files aren't produced or they end way too early.

#7 Caserol23

Caserol23
  • 65 posts
  • Location
    Catmon, Malabon Philippines

Posted November 6, 2019 - 12:40 PM

This came across my timeline today, and I immediately thought it would be useful for this community.

 

https://github.com/deezer/spleeter

 

This is a tool that claims to be able to split sources into "stems" via similar techniques to some of Harmonix's post-RB4 DLC output, by isolating the frequencies of each part. The difference is that it attempts to do so automatically using machine learning, with a dataset already having been created.

 

This may be useful in producing better-sounding customs. Hopefully it will help some of you still out there authoring!


Hmmm... Linux Based?! - this is real challenge too hunt down the bugs, very thanks for this  :smug:  :smug:  :smug:  :rock:  :rock:  :rock:


GyyPEOS.gif


#8 yaniv297

yaniv297
  • 374 posts

Posted November 8, 2019 - 09:40 AM

I've been using RX7 for this and results have generally been good, especially for drums and vocals.


Yaniv's Workshop - including all my work (hopefully) neatly organized

Spotify Playlist of all my releases

 

Notable Projects:

Bruce Springsteen Project - 11 classics from the Boss!

Journey Through the Past: A Neil Young 8-pack!

The National Homecoming Festival! Huge project featuring LOADS of tracks from the National, plus a bunch of openers!

11 Songs of Love... and One Song of Disappointment! Featuring Bruce Springsteen, R.E.M, Mac DeMarco and more!

Primavera Sound project: Day One, Day Two. Featuring The Cure, Talking Heads, Pavement and more!

Arcade Fire Project
..

WIP Thread

Hit me up if you wanna collab on any of those!


#9 StackOverflow0x

StackOverflow0x

    Wii Guy

  • 1,422 posts

Posted November 10, 2019 - 05:58 AM

I tried this and I'm just blown away by how easily it outputs great results! Drums and vocals have turned out really well! Bass too. Basically free karaoke versions of songs with this tool for the most part. And for the bass extraction, that should help for charting since it's usually the hardest to hear and comes out fairly well too.



#10 Shroud

Shroud
  • 278 posts
  • Location
    EU

Posted November 12, 2019 - 06:07 PM

Bah, doesn't work for me...

 

I always get "WARNING:spleeter:cannot reshape array of size 0 into shape (0)" which sounds like a low-level python error, but all I am doing is their basic suggested command for separating 2,4 or 5 stems.



#11 StackOverflow0x

StackOverflow0x

    Wii Guy

  • 1,422 posts

Posted November 13, 2019 - 06:02 AM

I know it sounds complicated to set up, but I just followed the instructions exactly as they provided. Download the right version for whatever version of Python you have installed. Download Conda. Use git to get everything. A more common error I've seen is the input path in the arguments to run it being incorrect. You can drag and drop a file into the console window to get the path in there if that's faster.



#12 Shroud

Shroud
  • 278 posts
  • Location
    EU

Posted November 13, 2019 - 07:10 AM

I know it sounds complicated to set up, but I just followed the instructions exactly as they provided. 

 

They don't really have good instructions. I know this is open source, which is great, but often it means the documentation is non-professional.

 

However... eventually I DID get some results, and they are surprisingly good!

 

Their example file worked, so I thought there might be some format requirements, which I didn't notice from the documentation. Comparing the example file with the few mp3 I have and I was using to test the program, I noticed that all my mp3 had variable bit rate, while the example mp3 had fixed bitrate.

 

So I tried a fixed-rate mp3 and it worked! Well, partially... first time it crashed in the middle of processing (but at least it started), perhaps because of lack of RAM. I know I have an old PC with only 4Gb of RAM, and I know that some intensive processing must be required for this task... but really how is it possible that to process a ~4 MEGA byte file the program uses 4 GIGA byte of memory? It actually does, because it my second try, after closing every possible other program in order to free more RAM, I monitored spleeter and it went up to more than 3.5 Gb of usage at some point.

 

This time it didn't crash, but it got stuck at some point, however it did manage to get as far as producing some of the supposed audio files: vocals, drums and piano (I was trying the 5-stem processing version). The third one is empty, but it could be that it got stuck after creating the file but before filling it with data. Vocals and drums are VERY good. I don't know if the models used by the program are for a specific music genre, but I tried it with a pop-synth song that I have no idea if it's supposed to be a good or bad choice for spleeter ("Adamski - Killer"). If you're familiar with the song, the drums are synthesized not real acoustic drums, but they came up pretty well. The voice is also very well isolated, not perfectly but essentially you would only hear a few "beeps" around the vocals, not processing artifacts but small sounds which are in the original audio, which apparently confused the program a bit, but they are not particularly annoying.

 

The processing is slow (more than half an hour but it's hard to tell when it got stuck exactly) because I am using the CPU option and, as I said, I have an old PC. There is an option to use the GPU which is claimed to take a time shorter than the song duration itself, but requires to have an Nvidia graphic card specifically, I have a GeForce 550 Ti but it's unbranded so I am not sure it will work.



#13 Shroud

Shroud
  • 278 posts
  • Location
    EU

Posted November 14, 2019 - 07:53 AM

More tests and more failures... with variable bitrate mp3 it just never works for me, and with fixed bitrate mp3 it always gets stuck between generating stem files. Sometimes no stem files are generated at all, some other times I get 3 or 4 (voice, drums, other and I may get piano or not), while it never generates the bass stem. There is a slim chance it's still processing but I let it go for over 2 hours and it didn't show a sign of life...

 

One interesting thing is why does it generate the "other" stem (basically the background track) before other stems. I would have expected that the algorithm identifies and extracts stems of known instruments first, and then saves whatever remains into other.wav.

 

I focused on testing the 5:stems option but probably I should use the 4:stems option instead, because now I understand that the "piano" stem in 5:stems mode is really looking for piano sounds in the audio, possibly including electric piano but not keyboards in general, so if the song does not include piano specifically, the 4:stems mode should be used instead.

 

And yes, unfortunately there is no standard way to extract a guitar stem, apparently due to guitar sounds being too variable in music. But the whole software is based on machine learning, and with a proper dataset (meaning, if you have plenty of guitar stems from other songs, which I don't) it should be possible to train the algorithm to recognize the guitar as well. Perhaps this would need to be done multiple times for different guitar sounds (i.e. with various combinations of the usual guitar effects such as distortion, reverb, chorus, flanger, etc...), but I would bet that if Spleeter manages to attract enough interest, then people will start to train/create and share new modes for extracting other instruments.

 

For RB3 custom authors, there is great potential here... I have my own issues with running the algorithm but no reason to believe the same happens to others, so if you don't count those issues, extracting the stems literally takes one single command line, and you don't need to mess with any options other than picking the model you want - to recap, the current models are 2:stems (vocals + rest), 4:stems (vocals + drums + bass + rest) and 5:stems (vocals + drums + bass + piano + rest).



#14 Caserol23

Caserol23
  • 65 posts
  • Location
    Catmon, Malabon Philippines

Posted November 18, 2019 - 08:52 PM

This will time consuming but this is the great tool to split up and may be isolation of track, this job well get done


GyyPEOS.gif


#15 Shroud

Shroud
  • 278 posts
  • Location
    EU

Posted November 19, 2019 - 09:04 AM

This will time consuming but this is the great tool to split up and may be isolation of track, this job well get done

 

I can't get the GPU mode working, which supposedly speeds up the whole processing. The CPU mode still crashes often, generally takes about half an hour to produce results, and in most cases it doesn't output all the stems, so I have to repeat the stems generation a few times per song. Overall I expect a couple of hours to process one song, but it is really "time consuming" only for my PC, not for me because all I have to do is write a command line and wait for the stems... so despite all the flaws I can accept the situation  :)

 

Now the real question is whether the stems produced are usable for RB3 customs. When you listen to the separate stems, they sound pretty bad, so I wonder whether players missing notes in-game for a few seconds will make the audio also sound bad. But if you listen to all stems simultaneously, they definitely add up to the original (good) audio.



#16 Alternity

Alternity
  • 1,086 posts

Posted November 19, 2019 - 05:18 PM

I didn't try that program, but I recently installed tensorflow GPU for another project and I had a bit of trouble to make the GPU work too, I just didn't take the required versions as seriously as I needed. When they say 10.0 for CUDA for example I assumed that 10.1 would work just fine, but I was wrong, 10.0 was explicitely needed. When they say that Python 3 would work just fine, I thought that the new Python 3.8 would, but it didn't, I had to use Python 3.7. I don't know if I was the one that can't read, or if they weren't precise enough on the TensorFlow website, but yeah, I hope this is your issue. That's for TF2.0 by the way, so if that software uses TF 1.x, they will most likely require other versions? idk, but triple check that.



#17 fab4

fab4
  • 7 posts

Posted November 28, 2019 - 02:29 PM

Anyone know if this will run on 32 bit Windows 7?



#18 Alternity

Alternity
  • 1,086 posts

Posted November 28, 2019 - 08:43 PM

Anyone know if this will run on 32 bit Windows 7?

 Basically you're asking if TensorFlow will run on win7 32bit, and this answer will be on the TensorFlow website.



#19 Shroud

Shroud
  • 278 posts
  • Location
    EU

Posted November 29, 2019 - 07:26 AM

I didn't try that program, but I recently installed tensorflow GPU for another project and I had a bit of trouble to make the GPU work too, I just didn't take the required versions as seriously as I needed. When they say 10.0 for CUDA for example I assumed that 10.1 would work just fine, but I was wrong, 10.0 was explicitely needed. When they say that Python 3 would work just fine, I thought that the new Python 3.8 would, but it didn't, I had to use Python 3.7. I don't know if I was the one that can't read, or if they weren't precise enough on the TensorFlow website, but yeah, I hope this is your issue. That's for TF2.0 by the way, so if that software uses TF 1.x, they will most likely require other versions? idk, but triple check that.

 

I know it's my fault that I've been rushing into testing spleeter without paying much attention to requirements and setup, but at this time of the year for various reasons my time for hobby is near-zero... but at the same time the spleeter documentation could really be more helpful than it is.

 

In my case, when reading the wiki, I checked my NVIDIA control panel and it seemed to me CUDA drivers were already installed. On CUDA website there seemed to be no drivers to download anyway but rather a "toolkit" for developers. The spleeter wiki didn't make me think I would need to actually write any code, just to have drivers so that the spleeter processing would be redirected to the GPU instead of the CPU, and make everything much faster. But whenever I tried to activate the GPU mode in Conda, spleeter always crashed.

 

Yesterday I tried to install the CUDA toolkit and see if it made any difference (mainly because I seemed to understand that it also updates the drivers themselves). I made sure to download 10.0 and not the newer ones. That's a 1.5 GIGAbyte stuff to download and install... and I am not sure it made a difference. Now GPU mode doesn't crash* but it still takes half an hour to complete.

 

*actually in one case my PC got completely stuck and I had to power off in the hard way, then it took half an hour just to recover

 

Then I noticed from here https://docs.nvidia....ex.html#windows that CUDA installation instructions also tell me to install Visual Studio and build this "nbuild" project with it. But I don't get the point, is this just for testing out CUDA or is this actually required when using CUDA from another program (i.e. spleeter)? I do not expect to be required to do any coding for using spleeter...



#20 Alternity

Alternity
  • 1,086 posts

Posted November 29, 2019 - 08:37 AM

You shouldn't need to build anything for CUDA to work fine with TensorFlow. I'm pretty sure you can start python and try to import tensorflow and see if it loads your GPU as expected. If it does, then the issue may be with spleeter? idk at this point to be honest. Assuming you followed the TensorFlow installation instructions for the version you are trying to run (which I believe is 1.5), it should work. The only thing I'd watch is your Python version, it's not mentionned anywhere on the TensorFlow website but Python 3.8 doesn't work yet with it so gotta install Python 3.7 (unless that was fixed in the last week, which is totally possible).






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users