I as well, sold my RS4 and eventually purchased a M3. Quite a long thread and very informative. I saw a few pointers on the carbon-build up, it's not a big issue and is a standard maintenance...now. I believe about every 30k it was recommended. Approximately $1k-$1200 to have it done. It is the Lamborghini Gallardo engine, so it's not a bad trade off.
Versus the M3, I'm not used to a rear wheel drive vehicle, but the Audi Quattro can't be touch...dry, wet, snow.
The seats themselves, specifically the side bolsters, wear unevenly vs. the rest of the seat. Many used ones you'll see this, Audi/Recaro production opted to use some inferior leather.
Per the transmission, you'll want the 2007 model. Audi made a change in the transmission from the 07 to 08 models. The 07 feels much more natural/organic/by-wire sort of feel.
Per the engine/exhaust note, there's the "S" button. It stands for super awesome or something like that. When pushed, baffles on the exhaust open, throttle response changes and steering tightens up.
Mods, forget it. There isn't much; which drove me to getting rid of it. I did the AWE downpipes and exhaust. It sounded ungodly, so wicked. Well, there's the supercharger, but you can buy two Kia's for the price of that thing.
The downs: The navigation will drive you bonkers. You're better off getting out a compass and reading star alignments to find your way.
I really don't have a M3 vs. RS4 stance, as I haven't owned the M3 long enough to make a fair claim/assessment. But, the RS4 is very exclusive, only 2012 imported. Half were probably ruined by douchers. It was never touted as a 0-60 car, though respectable, it's the 65mph - 125mph faster than you in 6th gear car.