I came from a Dinan Stage 3 335i to the M3. The M3 feels much slower, though from watching videos I know that it's not actually the case. I very much miss the torque of that Dinan Stage 3. I'm glad I traded it in at 40k miles as I didn't want any warranty issues, but it was an incredible car.
The M3 is an incredible car in a different way. The handling is absolutely incredible, especially compared to the 335, and it, of course, has much more prestige and better looks. The interior is also superior. It's a much better car than the tuned 335, all things considered. I still miss that torque feel, however, and I sort of wish I would have gone for a used Porsche 997 turbo. I really don't think you can go wrong either way, but make sure you test drive a few cars before making your decision.