I'm all for some "spirited" driving but that sounds reckless. Driving that fast should be saved for the track. If they make fun of you for not being reckless with them then they are probably not the kind of people you want to associate with. Eventually something bad will happen when you're pushed outside of your comfort zone.
When I lived in Tucson I got together with some local guys for a drive and it was a similar situation. They left me in the dust on a road I had never driven and 10 minutes in a guy crashed into a tree. He jumped in someone else's car (he was not hurt) and they all took off again. When we got to our destination they gave me a hard time for being so slow, even the guy that left his car against the tree. I never went anywhere with them again.
If you're not a member already, consider joining the BMWCCA and go to one of the HPDE events. It's off public roads and you'll have an instructor in the car with you, teaching you about driving techniques. It's probably the safest way to have a great time in your car around other like-minded enthusiasts.