This is the chainring either not being round or the bolt pattern not being exactly aligned around the BB axle centerline. (Or the bolt holes becoming elongated, perhaps due to riding with loose chainring bolts.) It is also possible that the rear cog isn't exactly concentric with the hub axle, either because the cog was drilled and threaded off center or the hub threads being off center.
Sheldon Brown has a description of a technique to better center the chainring if there is any play with the bolts. I'm guessing that perhaps the bike was set up using this technique and that since then the chainring position has shifted. (Are you doing skid stops? That might be enough to shift the chainring position.)
In my experience, the best solution by far is to spend the money for a quality track crankset. A crankset with the issue you describe is completely not OK on the velodrome. (By contrast, your issue doesn't matter at all on a derailleur bike and there is little incentive for the makers of derailleur cranksets to spend the extra money on precision bolt hole drilling.)
Ben