I can easily see RS4 being rarer. It was only 07/08 in states. I feel that the M3 can be a bit ostentatious and most people will think of you differently when you say you have one. The RS4 is more under the radar and to be honest most people know ///M and AMG but S and especially RS Audi's are pretty uncommon. BMW and Merc have been bringing ///M and AMG to US for over a decade. Audi has only had C5 RS6, B7 RS4, and the new B8 RS5.
I saw a review by Chris Harris and he talked about how great the RS4 was compared to its newer Euro version and C63. Matt Farah said that it has a much lower douche bag factor than the "other German sports sedan". I am not trying to bash M3. I know it is the better car, just maybe not for winter.