I think more people will be thinking "good for you" than anything else. If you get any looks of scorn, it's probably because they are jealous because they know they should be doing more.
When I started, I could not ride up the hill back to my house. So for the first month or so, I had to walk my bike back to my house every ride.
Make a note of your own personal milestones, you will look back in a year and impress yourself.