Good for you then. Unfortunately, you're the exception rather than the rule and your friend was just playing the odds; as was already mentioned, Obama received better than 95% of the black vote (I believe it was more like 97%). You wouldn't believe how many times I was told I was a racist simply because I'm white and don't support Obama. There is stupid everywhere. No point getting your panties in a twist for it.
"Never attribute to malice that which can be adequately explained by stupidity".