I haven't owned the E36 M3, but came pretty close to buying one. I haven't owned the E46 M3 either, but I did have a E46 325 (and have the S54 in my current car).
I bought the E30 M3 track car mainly because I also dreamed about owning one when I was a teenager (I think we are about the same age). There are more cost effective and less "needy" alternatives...
Hey, it puts a large grin on my face every minute on the track. I just wish I had more time to work on it myself and develop it (and the inside needs some cleaning/sorting out).