I guess it depends on what you want out of your car. I went from E92 335i to E46 M3. I had a tune on the 335i (325-330 rwhp) which makes it very close to E9x M3 in terms of acceleration, but it just wasn't an M overall.
E46 M3 can be had for quite cheap nowadays, but you gotta set aside $2-3k for some chassis reinforcement and Vanos work. But after that, they'll last well past 100k miles. Just make sure that you know what you should be looking at, or get a PPI done from someone familiar with these cars.
In retrospect, other great candidates would have been 128 sport, E36 M3, and E46 330. You just can't go wrong with the bullet proof S/M52s.
One important thing to note, since you've owned an E46, is that E46 M3 has higher steering ratio than non-Ms, so you may be disappointed in the M3's slower steering feel. Steering precision and accuracy is still on-point, just quite a bit slower.
I can write a lot more on this, but I'll stop here. Let me know if you're interested to talk more.