I've driven an M3 and an M5 on the track back to back as well. I agree with most of you said. However, I had no expectations that the M5 would "feel" better than the M3 or be more fun. The M5 is a sports sedan, not a sports car. The M5 has a dual mission of being both luxurious and fast, while the M3 does not. I feel the same as you, I definitely thought the M3 was heaven driving on a track. What's not to love about it? It's 600lbs lighter, has an engine that rev's to 8,300rpm and is extremely perfectly well balanced. I felt like the M3 was almost too easy to drive compared to the M5. I thought the M5 was just as fun on the track though. Insane amounts of lethal torque, and it accelerated so crazy out of apex's. It was definitely more of a challenge to control the torque coming out of turns and although it was more difficult to handle in the corners because of the weight, I thought it braked better than the M3. For a dedicated track car I'd pick the M3 all day everyday, but the M5 was just as capable and in the hands of an experienced driver, an F10 M5 can lap E92 M3's all day everyday. It's just that you can use your seat coolers to keep your butt cool and listen to your Bang & Olufsen in the M5 while racing on the track, whereas in the M3 you have the music of the S65 V8. The M5 is a capable track car no doubt, just not everybody's cup of tea.
Once you go Piano Black, you always go back.