I also went from a 335 to an M5 (E60) and now I have a 2013 S4. It is the right fit for me for now. Deep down I think I am still a BMW guy, but BMW (IMO) dropped the ball on the new 3 series, the interior (seats specifically) in particular. I really like the new 5 series, but I wanted a smaller car.
The Audi styling grows on me more and more. The view from the back is a little soft.