You really need to drive them, they both drive completely different,I love the E46 M3 its one of my fav cars and while performance power wise is close to the 335, they go about how they get there in completely different ways.
Exhaust is different, but I personally always loved the E46 M3 exhaust.
Since you are using it as a weekend car and not modding power I would go M3 without question. If it was a daily drive I may saw closer to the 335. The M3 will absolutely be more fun to drive, especially if you ever take it to a track where it just loves to rev.