There's no good solution short of getting a proper commercial rubber remover. Many of us use Mothers Racing Rubber remover. It works great but on horizontal surfaces it runs off so you need to apply it to your sponge or towel.
The Canadian guys have something called FSW or something like that which is the same stuff but it goes on as a foam and sticks better. I probably have the name wrong as I haven't been able to locate that product anywhere.