If you have the luxury of picking the time of day to ride that less traffic must be contended with then pick those times. If not, well, like you stated it will be something you have to live with. Cagers are crazy and will not at times wait for a few seconds to have a clear path. It is just one of the dangers of cycling.