Dont fret about a rear rack made for road bikes etc, such as an Axiom streamliner, and using some smaller panniers, as the rear weight wont be much, and will be fine on singletrack etc--again, not much weight in small panniers means they arent bouncing around and will be stable and have more volume than a seat bag.
handlebar bags can work fine also, and added bonus of some being waterproof, especially as some of your wording indicates that you may be in the UK, nice to have waterproof stuff.
Given no camping stuff, its not hard to have a simple set of clothes, off bike clothes can be as simple as a light pair of long quick dry type pants that look fine, arent heavy, a simple rolled up t shirt or button shirt for that matter, again light, 1 or two pairs of undies, a light fleece, a windshell or better yet, a rain jacket, rain pants, and you're set. Toiletries, repair stuff, pump, spare tube, and you pretty much have things covered.
Handlebar bag can hold phone, wallet stuff, some snacks, even some clothes or whatever.
very doable, and even on trails and such, panniers can work fine and be solid, especially with a lighter load.
start looking at options and adding up prices of things.
there is a UK bikepacking bag company, names escapes me, with reasonable prices for a whole kit of bags, you may know the name as I cant remember.
waterproof panniers by ortlieb are cheaper in europe than in here, so searching around can give you an idea of costs---as Im sure as with everyone, you have to balance costs, but getting well made stuff is generally worth it if you are a long time biker and intend to remain that way, as it will get used and last, and of course, work better than cheap stuff.
Alpkit is the company.
again though, dont forego rear panniers, they work fine for what you want to do--and not heavy panniers with a good attachment system will be even less "floppy" than a higher up, quite full seat bag thingee. I have ridden a lot on rough terrain and small, light rear panniers are fine , stay put and dont wag around, and are easy to take off. Bonus, you can strap light stuff to the top of the rack if need be, and handling is not bad at all, certainly in my experience not as dire as you worry about.
Last edited by djb; 02-03-19 at 10:05 PM.