I wouldn't do it... especially coming from an E9x. From your other post, you seem like an enthusiast who takes your M to the track.
The E46 is a great car, but I think you would be disappointed.
Let us know what you decide!
E92 M3 | Jet Black | All OEM Everything
E30 M3 | Alpine White | Mostly OEM