Adding any rack (and the weight it is carrying) will effect the handling. My view of these wonderful bikes is this:
There's always a compromise for practicality. You will lose performance. The tradeoff for being able to carry stuff that's not strapped to your body is you can't ride like Speed Racer. If I want to go zooming, I have a road bike for that. And in a few months I'll have the IRO set up as an alley-killer. My townie is for town, get it? Mountain bikes for mountains, race bikes for racing, touring bikes for touring, etc.