Sadly it seems since Project 96 that US interest in track (at least at a high level) has fallen off, at least for men. I wonder if Phinney would trade his Polkadot jersey from the tour for an Olympic medal on the track? I doubt it. The dream of something like Roubaix seems to eclipse everything else, no matter how unlikely.
I'm not sure about other countries, but here for a domestique to swap to track, they'd probably be taking a big pay cut (not saying they are getting fat paid as is) and most can't afford to do that.