I think swapping the chain ring is a great idea. I mean you can ride it with a 28, but you are more likely to enjoy it with a 24, so ride the 24 and if it gets to easy switch back. The more you enjoy it the more you ride the faster you get in shape.
I mean I CAN do roofing with a jewelry hammer, but its a lot faster and more enjoyable with a roofing hammer.