I still can't figure out why it's not more popular.
Maybe it's a marketing or image thing. The overwhelming majority of people who try track riding/racing seem to like it. A many of those that do stick with it.
There's something for everybody in track racing as well. I can't think of another genre of cycling where so many different types of athletes (small, large, sprinter, enduro, fast twitch, slow twitch, etc...) can be competitive in one event or another.