Coming from an M3, I think you will love it for the new factor for awhile, then you'll desperately want to come back into the fold.
I went the opposite way - I just leased a Countryman S for my daily DC commute, what a ball of fun that car is. I just cannot seem to bring myself to get something bigger, clumsier, less fuel efficient.
I have decided the M3 is as big as I'll go for now.
Have you considered a used E63? Or looked again at a CPO E60 M5?
2015 F80 ED
2004 Z4 3.0
2014 X1 28i ED