What size t-shirt do you wear? If you get a "club" jersey, get that size.
Unless it's a Hincapie jersey. The 3X Hincapie that was too small when I wore a 2X in everything else is only snug now that I'm into a XL. I might be able to wear it if I make it down to a size L.
It might be worth a visit to your friendly LBS (or REI) to see if they have something that'll fit you.