I don't ride on roads -- partly because I fear them and partly because I have excellent Rails-to-Trails close at hand. Plus, the scenery is better on the trail.
But I do not find riding (or driving) on the weekends any better. Actually I usually find it worse as the weekend warriors are out in full force both on the roads and on the trails...
So, while off-road riding may not be for you, perhaps looking for alternatives would be in order: Perhaps taking a jaunt out into the country or sticking to residential streets would be safer?
I feel strongly that biking is very very good for you -- unless you run into something or something runs into you or you fall off -- then it ain't. I think the best is to ride lots and (find a way) to ride safe....