Vibration could be due to two reasons. 1) pads melting onto the rotor, and/or 2) pad compound incompatibility. I once used Pagid and Project Mu on the same rotor and I experienced quite a bit of vibration.
OE 18s will fit StopTech 380 mm front when used with a 10 mm spacer (5 mm too I believe)
My first set of StopTech 380 front rotors, when used with StopTech Street Performance pads, lasted over 20 track events. The rears still have more than 50% life left.
I typically recommend my customers to try out the Street Performance pads first on the BBK. This combo should be quite capable. If this is not good enough, you can always order some race pads later.
It is perfectly fine to get a front BBK only. I am currently running Brembo 365 front with stock rears. I also have customers who are very experienced drivers use front StopTech only (355 and 380) without issues. Plus, you can always tune the brake balance by using different pads front/rear.
Hope this helps