The ultra great majority of roads are crowned to evacuate water, PLUS wind (even if you don't feel it) can also push cars a little, so ANY car slightly pulling over one side of the road is not an indication of a problem by any means. And I'm assuming you checked your tire pressures, because uneven ones also contribute to pulling to one side. Even uneven weight distribution (a heavy driver alone) can have an effect.
Only way to find out for sure is to try driving on a deserted (or abandoned) road with a driver and a passenger of roughly the same weight, where you can turn around and drive exactly over the same footprint as before. And tire pressures checked just before the run (I used 35psi all around myself). If car pulls to the same side on BOTH directions, then you might have an issue... but now you'd have to find out where
. Good luck.