I owned an RS4 prior to my M3.
M3 has better steering, better suspension, and has a more dynamic feel overall. RS4 has a torquier engine, better exhaust sound, and the AWD allows you to drive the car harder with more of a safety net.
Both great cars, each having some advantages over the other.
The B7 S4 is not in the same league whatsoever in terms of any performance parameter, and feels very sloppy compared to the M3 or the RS4.
2014 Audi S6 * 2015 Audi S3 * 2015 Porsche Cayman GTS
Former 2011 BMW M3 owner