I drive the M3 to work most days unless the weather is unusually bad. I do get some grief from customers. "Nice car I must be paying you too much". It used to bother me a little but I don't care as much anymore. I usually say something just as smart ass back like keep working hard some day you will be successful. These are people with much nicer houses than mine and 2 kids in college.
I think it comes down to the fact that most people don't prioritize their spending on cars. I have found that lots of people don't know what an M3 is.