I heard the new ZL1 drives pretty nice
Kidding aside, I traded a supercharged mustang for my M3 and I'm in the same boat. I also had a Camaro Z28 with LS1 a long time back. I don't know if I would be able to go back to a pony car, but it was certainly a lot cheaper to mod, bolt-ons and blowers.
My M3 is still stock now for this reason. I'm thinking of keeping mostly that way, maybe with minor stuff like exhaust. Otherwise it's a well balanced package, ZCP wheels are perfect, suspension I won't touch, and in general I don't want to screw it up.
BTW just learned about the Borla ATAK for the M3. Sounds great and relatively inexpensive compared to other M3 mufflers and I'm thinking of starting with that before going down the path of x-pipe/tune/etc.