A 4mi commute? Ride whatever is comfortable. Rack and bag works great, or use a backpack if needed...its maybe 20min on the bike. 20min on the bike though, I would guess most setups can be comfortable for that long.
The bike I have set up for weekend camping, commuting to work, and general family riding feels fine whether or not it has bags weighing it down.
Someone that weighs 215# will flex a bike overall more than someone who is 160# and carrying 15# in a pannier. The whole claim that touring bikes are too stiff when unloaded seems totally arbitrary to me since it needs to account for who is riding the bike.