I certainly don't think everyone should go to the track. If you aren't interested then you aren't. No need to force yourself to go. But if you are interested in performance driving, you should go. It is definitely a lot safer than driving fast on the street.
No way can you safely drive at track speed on the street. You can however go street speed on track if you don't feel comfortable going fast. I instruct at some track days and I always encourage the driver to drive at a pace they are comfortable with. The first couple of laps I let the driver drive and I observe what he is doing without saying anything. Then I'll comment and give tips and pointers based on what I see. If you are gripping the wheel hard, nervous and sweating, chances are you are driving over the your limit. I had a driver that didn't feel comfortable going faster than 80mph in his Porsche 911. I told him no problem... just remember to point the guys behind you by ASAP.