Before my M i had an 04 330i ZHP for 5 years and to this day miss that car a lot. maintenance is cheap compared to M and it was fun to drive. Steering was perfect and the 6spd was butter once warmed up. This would be the one i'd go for instead of E46 M if I was looking for a daily driver. You can get one for about 10k and the car still looks so sick (mine was Black Sapphire). If its just for fun and not a DD the E46 M is pretty awesome.
Your not crazy.