Instead of Inertia, think moment forces (lever arm). Take a wheel and cut out a slice of the pie.
You hold a stick with a weight (the rim plane plus a bit of wheel sidewall) on it 9.5" long. In the other hand, you hold another stick with a weight slightly heaver (because the circumfrence difference of the rim is 6.28" longer) and your stick is 1/2" longer. Which one is takes more strength to hold up?
The only real difference is the additional weight of the sidewall on the shorter stick because the tread area of the tires will wash out.
It's this moment force that requires a bit more work to both accelerate and decelerate since you've shifted the heaviest part of the wheel further away from the hub.
Then there is the predictability of tires with short sidewalls as mentioned.
edit: Also keep in mind races are won in 1/10ths at a time. For road work / commutting... it doesn't really matter all that much because it's doubtfull you are looking for extra little edge. So it really comes down to roadmanners and using the additional sidewall as a bit of buffer from the imperfections of the roadway.