You can delete everything; even the RB2 songs but you will lose the 8 RB2 songs unless you back them up. (Back up their DTA entries also!)
The encryption type can be selected in the PS3 converter in the options on top of the app before converting. If a song already has a PS3 version it is highly likely that this version works and therefore should download this instead of the xbox version. I assume some people still convert songs to every platform even if they don't own said platform so in theory there could be some PS3 versions that were never tested; so look out for that.
Once you get a single song working fine you can just add them in small ammounts; let's say 10 or 20; and load the game to make sure everything is fine... don't need to test them all you just want to be sure your DTA file is correct. I also suggest every time you have modified your DTA back it up in case it gets fucked up. Adding/removing songs all the time could result in a typo in the file or something else; good example is the guy that posted before you which had a foreign character and any song that was after that character was ignored.
Also I sent you a PM a while ago