I owned an RS4 prior to my M3.
M3 has better steering, better suspension, and has a more dynamic feel overall. RS4 has a torquier engine, better exhaust sound, and the AWD allows you to drive the car harder with more of a safety net.
Both great cars, each having some advantages over the other.
The B7 S4 is not in the same league whatsoever in terms of any performance parameter, and feels very sloppy compared to the M3 or the RS4.
Audi S6 * Audi S3 * Porsche Cayman GTS
--Former BMW M3 owner