"I get 3 weeks in a year, and that needs to be distributed among all the vacations." I'm not meaning to be snarky here, but what you're saying is you could do a three week tour. You choose not do do it. I don't have a lot of money, and I have time irregularly, but getting out and doing things like bike touring and canoeing are things that I HAVE TO DO to stay human.
It can be tough to find good touring partners, but they're out there.
As for money, I can't afford to tour in exotic places, but that's fine, as I've chosen to do door to door tours in order to cut down on my carbon footprint.