I went from an E46 M3 to an Evo IX MR back to an E46 M3.
Pushing 24psi of boost into a tiny engine has a lot of its own drawbacks. The fact that a good clutch for an Evo costs $2000+ is worth mentioning, too. The Evo owners I know with fast and properly built cars (2.3L strokers, 35R or 37R, fuel system upgrades, AEM EMS, clutch, exhaust) have spent well over $20k.
And then there's that the car is just plain embarrassing to drive if you're over the age of 25. Sorry, but it's true. It's a high school car.
Would I own one as a dedicated track car? yeah. Would I ever own one as a street car or daily driver again? Hell no.