The chamois is a BIG and important difference.
Beyond that, good brands will take into account the different waist/hip/thigh proportions of men and women in their sizing. I bought a pair of women's tights (no chamois, big discount) a few winters ago, figuring that it was all spandex, they'd simply stretch to fit. Two rides later I gave them to a woman I know, because I got tired of them sliding down due to my lack of hips and thin (relative to a woman my height) legs.
You can probably live with jerseys that are a little tight/loose, but bibs and tights really do need to fit right to be comfortable.
BB
__________________
Formerly fastest rider in the grupetto, currently slowest guy in the peloton
