I bought my E90M3 because my wife would not ride in my turbo E36M3 anymore, which meant we had to take her car, which was an 08 X5 Sport 4.8i at the time and while it was very good for a truck, it was not interesting for me. I kept the turbo E36M3, which I enjoy more than the E90M3 since it I can feel the road and it has serious power (500+ rwhp and 500+ lbs rwtq).
Most cars will be a step down after the E92M3. I don't see you finding anything stock with 4 doors that is comparable without spending that roughly $50k that you do not want to spend. Cadillac CTS-V, Mercedes C63, BME E90M3, Audi S4 or RS4 are what come to mind.
The little AWD cars like the Mitsu and Subi are fun, but not at the same level. For a daily driver, I'd rather have the 2.5L of the Subi than the 2.0L of the Mitsu given the weight, but the Mitsu is the better driver's car in stock form. They are great off the line if worked hard, but can't pull for as long as the M3. If all you do is squirt around town, they may be OK, but there is a reason they do about 103 in the quarter and the M3 does about 113. Hotrodded ones can go pretty well.
What about trading the X5 for an X5M?