I experimented with a thin silk LS base layer under a wool jersey for winter riding, because I had a wool LS jersey that was a little scratchy against bare skin. The silk base layer got really wet, and was clammy against my skin. I was much more comfortable with a wool base layer, or a synthetic base layer - both wicking and comfort-when-wet were better with those fabrics. Silk was the worst performer of the three, though better than cotton.
If you want to wear underwear or shorts, for example under regular clothes, IMO underwear/shorts made of a synthetic material would be your best bet - perhaps boxer briefs, or compression shorts (keeping location of seams in mind). I'd stay away from regular briefs, the way they're normally constructed places seams in high-friction areas.