I'd just keep at it. It takes a while for the body to adapt to cycling and gain the specific strength you need to be efficient.
It can be a little confusing if you see people who look less fit dropping you, but it happens. They've just put in the work.
Are you drafting behind other riders? That saves a lot of energy.
My other tip would be to do some high intensity work in the limited time you have to train. Go up and down a big hill as many times as you can, as hard as you can once a week.