This thread will detail the steps that I use to chart songs using 5.1 mixes of songs. All of the steps used in this guide are done in REAPER.
Step 1: Separate the Audio
The first step is to separate the channels of your source audio. Add your source audio to a new REAPER track. Once it's there, right-click your audio item and navigate to Item processing > Explode multichannel audio or MIDI items to new one-channel items. This will take each channel of your source audio and put it on a new subtrack of your song audio. The standard order for channels in the FLAC format is Left, Right, Center, LFE, Surround Left, and Surround Right, so label your tracks as such. The original audio file is no longer needed, so you can delete it from your project.
Step 2: Mix the Channels
The second step is to mix the channels. For your song to sound how it should, you'll need to make some simple adjustments to the mix to fit a stereo format.
The Left and Right channels should be at 0 dB and panned 100% left and 100% right, respectively. If you can't see the volume fader or the panning option, expand your tracks until you can see them. The LFE and Center tracks are both panned to the center, but the volume faders should be set to -3.00 dB each. The Surround Left and Surround Right channels should be set to -6.00 dB and panned 100% left and 100% right, respectively. In addition to all of this, make sure both your SONG AUDIO and MASTER tracks are set to 0.00 dB, as they sometimes are not set to this value.
Step 3: Use the Channels to Chart
Not really much of a step, but it's still important. The entire reason to use a 5.1 mix is the additional separation it provides. All of your audio tracks should be subtracks of your SONG AUDIO track, and you can solo/mute each one to hear the song in a different way. Use this to your advantage.
Step 4: Export the Audio
You should export your audio like you would export any other song for use in-game. Solo your count-in and your song audio, and render to a stereo WAV. The point of the mixing step is to get the source audio from a 5.1 mix to a stereo mix with additional channels, and this is the last step of that process.
That's about it for processing 5.1 audio in REAPER for use in Rock Band. File formats other than FLAC should use a similar channel layout for mixes, but if you're unsure, you can just look it up and label your tracks accordingly during Step 1. From there, the process is exactly the same. Hope this thread helps!