First of all, NEVER BE ASHAMED OF YOURSELF IF YOU ARE DOING SOMETHING POSITIVE!
Secondly, I started riding @ 390 something on an old MTB my father-in-law gave me, it was a green Mongoose with a front shock that contained struts and machined aluminum. He had it set for maximum drop on the bars and it killed my back and hands every time I rode it. But I continued to side because I had forgotten that rush and feeling of freedom that I had as a child when I would ride my bike. Riding helped me FINALLY kick the cigarettes for good(been 1 1/2 years now).
On a third note, I am currently helping my in-laws in running a day care facility they had tenants in since they had retired, but have found out it wasn't in the cards. I now work over 12 hours a day and only get to ride when I need to run to the store for some last minute thing or because of something happening. The children here do not make fun of me, or give me any grief. They in fact say exactly the opposite, they find 'cool' and 'awesome' and other such positive remarks that I ride a bike too. They have said they want to bring thier bikes to ride with me also.
So, in closing, Just do what YOU do. If you enjoy it and you aren't breaking the law, have a great time!