I just looked at mine and I'm sure I could pop the inserts out without much trouble using my staple remover (handy tool in my desk). I didn't actually remove them because the insert may break but I don't really think it would. I would use them, and if I go for Tubus or such racks on my new bike I'll pop the inserts out. And save them for when I ride my old bike again.
I do have the front bags and use them on the back, mostly because if I had the back rollers I would fill them up. I'll still get a set one day for touring.