I owned a B7 RS4 prior to my M3 as well.
It's hard to answer your question without a little more info like price and options. The B7 RS4 is a great car but has it's issues as well.
Great: Torque and AWD make it an animal out of the gate. It's very easy to drive on the street and will make novice drivers appear like pros. It can be driven hard in any condition. The exhaust is phenomenal in sport mode. Solid build and materials throughout. Interior is on par with many high end luxury vehicles. It's rare and exclusive. I can count on one hand how many I have seen since it's arrival in the US. I have E92 MT and have to say the B7 RS4 shifter is superior. It's precise and smooth. The resale value on the car is very good. In December 2011, I sold my 2007 RS4 with 71,000 for $39,000.
The Not So Great: My car had DRC issues which was covered under warranty but was still annoying. The carbon issue is no joke. The car would run fine however I noticed loss of power every 7500-11000 miles. After a carbon clean (usually $750-1200 depending on the shop) it was noticeably faster. I had two carbon cleans in 50,000 miles. The car constantly consumed oil. I followed the break in period to a "T" but the car consumed a quart every 1000 miles from the first 1000. Although it's an animal out of the gate it gets winded quickly past 60 mph. Not that big of a deal in a city but it can be demotivating on the highway. Lastly, the BOSE radio is completely BROKE. It was an expensive option that never failed to disappoint. The EPS on the M3 is superior in every way.
I prefer the M3. I love its looks. Although the OEM exhaust is a bit week the engine sound is superb. I have had 0 issues and have 10250 miles on the car. It's a more rewarding experience and its a blast in the snow with proper snow tires. My family and friends also comment that the M3 looks better inside in out compared to the RS4. If I had to complain about the M3 it would be the individual color choices at $5000 and the lack of Brembo break options.
Let me know if you want more advice before your purchase I will be happy to help.