Nobody necessarily said you "have to speed" to do spirited driving BUT spirited driving at the speed limits and if you never track will feel just as fun and feel the same in a 328 as an m3 so again why get an m3 unless its for the badge, which I suspect you did it for. I actually think the 328/330 with NA I6 is more fun at lower speeds to drive the piss out of and redline the engine and it has MORE than enough handling to do anything at street speeds.
So yes any type of spirited driving at the speed limit could be had with a 328 equally well as an m3. The thing is at higher speeds and a track, the m3 provides experience that a 328 could obviously not match.
So seems like you should be in a 328 and maybe paste an m3 badge on the back to satisify your badge desires, or slap a sign on your rear window with your yearly income to show people you are better than the car you drive in a 328
