I owned an RS4 prior to my M3.
M3 has better steering, better suspension, and has a more dynamic feel overall. RS4 has a torquier engine, better exhaust sound, and the AWD allows you to drive the car harder with more of a safety net.
Both great cars, each having some advantages over the other.
The B7 S4 is not in the same league whatsoever in terms of any performance parameter, and feels very sloppy compared to the M3 or the RS4.
Precision tool: 2011 BMW ///M3
Sledgehammer: 2014 Audi S6