I too am under the impression that what ever there is will be small and it would take a very skilled driver to notice the subtle differences, but the fact remains that they are there no matter how small.
I have also driven numerous cars with different wheel sizes and to be honest with you, there is more of a difference from changing tyre brands than you will notice changing rim size. The most notice thing apart from ride comfort when changing up rim size was how the car turns into the corner, with the bigger rim and lower tyre profile the turn in will be quicker, factional yes but it is there all the same. The other thing that can be noticed is the steering, lower profile means more tyre deflection from the surface which is less noticeable with a higher profile tyre.
This assumption that acceleration or topspeed will lessen due to bigger rim size is something I can't work out, only if the width increase can the performance decease but I still think the width would need to be quite big to be really noticed.