I've owned all of those. I would only go with the E30 if you plan on doing an M5x/S5x swap.
E36 M3: you will probably find the power lacking but it's a nice middle ground between the E30 and E46. It's still a very small, light car. It inspires huge confidence as it's highly controllable at all speeds. I had to supercharge mine to bring numbers closer to the Euro spec models, which makes the chassis and engine feel a little more in sync. Otherwise it's ALL chassis. This model is the easiest to work on, but a lot tends to go. Anything made of rubber needs to be addressed. VANOS is a typical issue. Cooling system needs to be replaced immediately. The list of fixes on my E36 is endless.
E46 M3: my favourite BMW, and honestly I wish I never sold mine. It seems to blend the best of everything into a great package. The S54 needs absolutely nothing and is an incredible engine. Unfortunately it's hard to find a clean one. Lots of "bros" bagging these cars. Cooling system, bushings, suspension, electronics are going to be an issue. Valve adjustment is recommended when you buy it. But it's a very rewarding car and I'd say the last truly mechanical M car.
For either the E46 or E36, budget an additional $5,000 to put away. Get your Inspection II done right off the bat if it isn't done. This allows you some flexibility when things start to go. Out of my E36, E46, and E92 it's the E46 M3 that I miss the most.