You can make all kinds of arguments for why someone should or shouldn't ride on the road but at the end of the day, if you are not comfortable, it will not be enjoyable, and I think the reason most of us are riding bikes as opposed to other activities is because we enjoy it.
You're enjoying single track so it's not like you're giving up bikes altogether. Enjoy that and don't dwell on whether your decision is justified. After you move, you may start up again or maybe you won't. Do whatever feels right.