E46 M3 is a better looking, much more involving driver oriented car. But (for me and my wife, at least) not suitable for more than short (couple of hours) drive. Which is why I bought an E92 M3 with EDC.
You should also consider the E46 issues with cracking in the rear subframe mounting area. These problems are real.