Each person is different, and it really comes down to your skill level and how comfortable you feel in the car. If you are smooth and push the car hard, you'll want mods sooner than later. If you feel like you can do more, just not sure how to advance your skills without help, then you're not there yet. Bottom line is if you do track days more than 3 times a year and you are even moderately aggressive, you'll at least want extra tires and wheels. After you do 2 track days and your $350/ea PS2s are gone, you can easily justify the costs! Get a square setup from apex. You can't go wrong for the price and they can tell you exactly what size tires and wheels you can run with your suspension setup. Plus, they're very inexpensive for a pretty nice wheel. Then camber plates to preserve the precious tires, and the list goes on I'm afraid.
Glad to hear you're enjoying your M3 the way that it was intended, and not drag racing!