Remember back to your Computational Statistics and Data Analytics class from primary school (we all had that class, right?) - correlation is not causation.
There are a lot of moving parts here and various things could be, or it could be something entirely outside of what we're considering.
Before Magma C3, there was the original Magma created by Harmonix. That would create songs without audio encryption, in RBA format. Then we had to use tools to convert from RBA to CON. At this point there was no automated conversion to PS3 and I do not believe there were PS3 customs at all.
Magma C3 first came out and there was no audio encryption. One of the many changes added, which I think may be one of the culprits (but I doubt), is that Magma C3 allowed you to specify the audio quality of the mogg. The default quality chosen by Harmonix for Magma was 3. I think I defaulted Magma C3 to 5 but the user could change it. That would result in better audio quality with less compression, but a higher file size as a result.This definitely causes problems for Wii customs due to the file size limitation they have, but it does not create issues for Xbox and I wouldn't think it would create issues for PS3.
At a certain point we started integrating encryption into C3 customs using Magma C3. I think PS3 customs need encryption for them to work, but I may be misremembering. The encryption process in Magma C3 is the same as in C3 CON Tools. So if you have Song A that comes out of Magma C3 encrypted and you run it through the PS3 converter and it doesn't work, and then you re-do that same Song A in Magma without encryption and then run it through the PS3 converter and it works, you probably narrowed it down to "something that Magma C3 does." But you can't say that it's Magma C3 simply because some songs not created with Magma C3 work and some songs with Magma C3 don't work - unless that applies to ALL Magma C3 and non Magma C3 customs. There could be less channels in the one made with non Magma C3, for example.
Lastly, there's one step going on behind the scenes that many people don't know. I added an obfuscation step to my tools. So if you create a song with Magma C3 and it encrypts the audio, and then you use C3 CON Tools to extract that mogg file, it will not only be encrypted, but C3 CON Tools will mess with the file so you can't even attempt to decrypt it. The file is modified so it's not recognizable by the game or any other known existing tool, as a way to protect C3 files. So if you put that modified mogg file on your PS3 - of course it won't work. That's why I specified you would need to find a different tool to extract the mogg from the CON file. It will still be encrypted that way, but it won't be modified - and it might work if you drop it in your PS3 custom.
There's not much I can think of that would be the obvious culprit. The encryption choices for PS3 are for the MIDI to EDAT conversion, it has nothing to do with the audio. That uses the latest and strongest encryption we had at the time I stopped working on the tools. But that's the same audio encryption used in stereo customs and in multitrack customs and in karaoke customs. Meaning if it works for one, it should work for the other. My understanding is that like 99% of customs convert just fine, so I don't think it's the encryption itself. The audio quality issue makes no sense, since it's still the same audio stream, just sounds a little better and has a bigger file size. Do all the ones that fail have the same size or are all above a certain size? Are there longer/bigger, but not multitrack customs that work (that would defeat the file size theory).
Do all of these customs that fail have 14 channels? (That's what Hotel California has). I'd be interested to find out if that's the case. If there are any multitrack customs with the same amount of channels that do work, what happens if you drop that working mogg into the files for the song that doesn't work? Does it "work" with the wrong audio? That could help eliminate the mogg as the culprit, for example.
I've spent a while thinking about this the last couple of days. So far, that's all I got. I'll wait to get thorough responses to all the different suggestions above. Maybe I can help you guys puzzle it out. It's not something I will ever be able to do by myself, since I don't own a PS3 and can't test all these theories.