That's an interesting question to ask.
Leaving the car stock may be better in the long term. However, in my mind, mods are to enjoy the car now. I think my car is much more enjoyable and performs/sounds better since I modded it. And that's what I care about.
I can see the other point though.
Current: 2009 E90 M3 | SG | FR | 6Speed | Akrapovic Evo | JRZ RS1 | StopTech Trophy | Cantrell Brake Ducts
On order: 2016 F80 M3 SO | AB XT1 | ZCP | etc.