I had no idea it was collins singing on "more fool me" until I read about it on the net. Listening to it now I can certainly hear a difference, but for me it's kind of like technical mistakes in films or television, where you usually don't really notice them until they're pointed out to you. Collins really was the perfect replacement for gabriel after he left.
That and For Absent Friends, the only 2 songs where Phil Collins took lead before Gabriel left. And yes, you have no idea how many people react just like you did. Then again, when a song is sung with some presence, you just can't mix the 2 up. Of course, as you point out, once you know you notice, but otherwise it's difficult. Carpet Crawlers is another good example.
I'm surprised nobody started a "guess which songs we'll get that we don't know of yet" thread or discussion.