I'll vote for the e46. Cheaper, more low key, but still timeless design. Plenty of fun as a DD and not over the top. Cheaper parts, mods, and less gas.
You will have to factor in maintenance for the e46 since it will be much older. Though a e9x may also break your wallet if it is out of warranty.