I use S4 at the track. You still get the rev matched down shifts so you aren't wasting your clutch discs but its not nearly as rough as S5. An S4 up/down shift won't upset the car in places where S5 would.
Not sure why you would use S3, that would upset the car more than using S5/6 on a downshift.
2013 F30 328i Melbourne Red
2011 E90 M3 Monte Carlo Blue
2004 E46 M3 Imola Red