Okay, so I managed to get my new drum kit working on my PS3/Rock Band. As it turns out, neither MIDI-OX nor the midi cable I bought were the problem. I was thrown for a loop because the video Fujiskunk posted above about how to get an Alesis drum kit to work on RB3 requires a slight alteration if you want to get a DTX502 drum brain up and running. At the 5:27 mark, the author of the video shows you what you'll need to select in terms of inputs and outputs in the "Midi device" section of MIDI-OX. He tells you to select "USB 2.0-MIDI" and "MIDIOUT 2 (USB 2.0-MIDI)" in the output section. However, when I hooked everything up to my Yamaha kit, only one USB-related option showed up --- the one for my USB-to-MIDI cable. But once I selected "MIDI mapper" --- which the guy in the video explicitly tells you not to select --- I got it working. I'm not sure why this worked, but at this point I don't really care. I'm just happy that I can now use my new kit to play drums on Rock Band.
Anyway, I'm going to provide some instructions here in case anyone else buys a Yamaha kit with a DTX502 brain in the not-too-distant future:
1. Buy a USB cable that can plug into both the DTX502 brain and a computer (one end will be type B, the other will be type A).
2. Buy a USB-to-MIDI cable. There are a bunch of options here, but I went with a Roland cable that cost around $40 Canadian.
3. Buy a Madcatz midi pro adapter. These things are expensive now, but it's well worth buying one if you want to escape from the fragile kits that came with the game.
4. Download MIDI-OX (it's free!) from www.midiox.com.
5. Hook up the DTX502 brain to your computer with the USB cable. Make sure your kit is OFF when you're doing this.
6. Hook up the USB-to-MIDI cable to your computer. Make sure your computer is ON when you're doing this. Depending on what type of cable you buy, you might have to install a driver.
7. Hook up the other end of the USB-to-MIDI cable to the Madcatz midi pro adapter.
8. Plug the Madcatz midi pro adapter into your PS3. I don't think it matters whether your PS3 is on or off at this point, but I had it on.
9. Turn on your drum kit. Your PC should make a noise indicating that it's detected a new device.
10. Install MIDI-OX and fire it up.
11. Click on "Options" and select "Midi devices."
12. Select the name of your drum kit in the Input section.
13. Select the name of your USB-to-midi cable and "Midi mapping" in the output section.
14. Click OK and you should be good to go.
Unfortunately, these instructions are meant for PS3 owners who run a PC with a recent version Windows on it. I have no clue if this will work for Xbox/Mac owners. You guys are on your own.