Very cheap bikes can be "complete rubbish" but many years ago I did my longest ever day (200 miles) on the cheapest 5 speed, steel framed, chromed-steel wheeled bike in the shop, and only a few years ago I did my toughest ever off roading (a full day on a mountain bike route in Snowdonia) on a borrowed steel and lead pipe mountain bike that was simply awful — but it was a great day's riding.
I'm a musician and I have long said that "money saved on cheap instruments is money wasted" and the same could be said to some extent about bikes. Buy the cheapest clunky bike from your local department store and either you will not enjoy riding it and give up, or you will not enjoy riding it and spend a lot more on a better one.
However, beyond that basic advice of "don't get the cheapest because it's cheapest" it gets more complicated. A bike is essentially a simple machine, and a single speed or fixed gear especially so. A new rider needs to learn to distinguish between illusory improvements (colour, style, etc.); things that make a real difference to daily riding; and things that are nice but inessential luxuries. The keen rider may be tempted to recommend "just save up another $500" when an extra $100 would be enough.
Whatever you buy, chances are you'll upgrade the seat and pedals and, in due course, the tyres, and you may also tweak the gear ratios. Then, with a few miles under your belt, you'll start dreaming about the exact bike that you want, not the one that someone else says you want.
So, don't get the very cheapest, but get one worth upgrading and develop your own wish list for your next bike.
