You don't need specialized gear to start. When I started bike touring in the mid-70s, I'd never heard the words "pannier" or "chamois," or even "Shimano." I stuffed a few old surplus camping items in my old high school gym duffel bag, tied it between the brake levers of a borrowed Schwinn 10-speed, rolled up some clothing in my old Scouting sleeping bag, tied it and a pup tent to the book rack that came with the bike, and took off on a two week trip. I had $40 in my pocket and came home with change.
Later I gradually spent more money on gear and the bike, but not to start.
Even now, I don't use a handlebar bag or front panniers. I use a set of Arkel Drylite panniers that cost under $100. Sum total of camping gear, including a nice tent and down quilt, is around $500. I use home-made rain gear, thrift-store clothing, cheap running shoes on platform pedals. I've never owned a bike worth $500. (I know cyclists who spend more that that on shoes and pedals!)
And a simple lifestyle at home allows me to tour long and often. I pay $100/year for a Tracfone plan, seldom used, on a salvaged free phone. I share many expenses and barter services with neighbors and friends. I'm working today on my wood pile for cheap/free heat next winter. Most important, daily cycling allows me cheap transportation, keeps health care expenses to a minimum, and keeps me in shape for the best vacation travel I can think of.