I'm in the same exact boat as OP. Ok, almost. I'm considering a loaded 335i RWD Sedan vs a used 08/09 M3 Sedan. Affording it is not the issue, but whether for nearly the same price, a loaded 335i vs not as loaded used M3 for a DD is the question. I know I WANT a M3. Will I track it? No as it'll be my only car and if something happens, well, the wife won't be so happy and I'll never back into another performance car. Maybe some autoX on Sundays with the local SCCA. Having two kids also means no free time. In any case, it'll be a DD. However, I feel if I get the 335i, I'll just be settling for second best.
8 years ago, I wanted a G35 coupe. Even put a deposit down, however financially, it would've cost nearly twice more than my current Mazda6s sedan. So I settle for my second choice. To this day, it was a good and right choice but now without regrets. Every time I saw a G35 coupe, I just dream of owning one. I'm at the same cross road now. However, performance wise for a DD, the 335i and M3 around town is fairly close. The low end torque of the 335i really moves the car. But the M3 was definitely a blast to drive around the country road. Scared my sales rep a bit when I told him out for a ride.
For OP, I also took into consideration maintenance cost after maintenance period ends. The biggest cost driver would be the tires and brakes/rotors. The M3 will run more for the tires while almost twice as much for the brakes/rotors (based on my research). It could be less if you get it done at an independent shop and get the OEM rotors/brakes from some of the sponsors on the forum. But if it's just a DD, and based on your own driving route, you might only be changing it once every 40k or few years. Oil change cost is not that much more between the two vehicle. The only thing that would concern me is the warranty period. But then again I'm only looking at CPO M3's.
Hmm...I think I talked myself into getting the used M3. Just need to talk my wife into it :P