The thing about being larger is that your increased mass and ability to put out power is a greater factor than the increased frontal area in the wind. That's why TT specialists are not little wispy guys like the pure climbers.
Regarding susceptibility to crosswinds, I suspect heavier riders may have an advantage but a LOT of it is the bike itself and your balance on it. Among my bikes there are some that seem to ignore crosswinds and some that are easily pushed around by them. I find my metal bikes tend to be less susceptible, whereas the most susceptible is a CF Bianchi from 2006 which presents some flat surfaces to crosswinds. But I seem to have reduced that by moving my position slightly up and forward, so that there's more weight on the bars. I've found when hit by crosswinds, the normal reaction is to sit up a bit , which unweights the front a bit. If I put weight back on the front, it seems to stabilize the bike.
__________________
"Don't take life so serious-it ain't nohow permanent."
"Everybody's gotta be somewhere." - Eccles