Settling on a bike is half the fun. I'd agree about a second wheelset with narrower tyres if you do end up going on group rides, however that pushes out your budget somewhat. Maybe you could look around for a secondhand wheelset for that after you have the bike.
See how you go this weekend on those test rides again. There's a lot of bikes out there though the bigger brands tend to have the better spec for price.