On my long tours I only use rear panniers but have experienced problems with cross winds or tractor trailer rigs passing by me on the road. There is a section on Hwy 1 in Big Sur known as Hurricane Point and going south from this area I had a gust come down a ravine that moved me and my bicycle a couple feet to the right. It happened so fast I did not have time to worry about crashing.
Windiest in the afternoon hours so better to ride in the morning or at night (if practical).