I use to wonder about that. I too would take my wheel off and have no trouble with the rim brakes. Then putting it back on it wouldn't fit.
Then I realized that I was taking it off with a flat tire and putting it on with an inflated tire. Duh...
So then I knew to not inflate the tire until it was back on the bike. When I got a bike that had a release on the calipers to let them open up for tire changes, then I was in hog heaven and thought I and my bike were up town and spiffy