The X5M is the wife's DD; the M3 mine. I get to drive the X5M on the weekends as it also serves as the family car. The M3 definitely feels more raw and in my opinion sounds better. The power delivery on the X5M just feels smooth and effortless. They are different cars but equally enjoyable. They both leave me with a stupid grin on my face after a spirited drive.
If I HAD to sacrifice one, it would probably be the M3.