I have gone on a lot of caravan runs over the past few years and I def know what your talking about. I myself do not enjoy driving all out on the HWY or any other non track locations. It is a common theme with all caravans I seem to go on that its all about who can go the fastest and show off their cars speed and exhaust note.
What I usually do is just roll at the speed that I am comfortable at and if that means I show up late or slightly behind the group I am ok with that. So my advice to you would be to do the same. If you are not comfortable driving at the caravan speeds don't. Enjoy the drive and catch everyone at the end point.