I considered an e60 m5 before buying my m3. I need a 5 seater, since I have three kids, but I just couldn't live with the size of the M5. Not weight/nimbleness, where the M3 likely has and will keep an advantage, but the phyisical size. I don't like big cars for some reason. I don't even like midsize. The smaller the better.
Just my 2 cents.