I dont find that a bigger frame degrades handling response for me that much. Wheel size does.
You can always opt for a larger frame that has more aggressive angles and a shorter stem. At.the end of the day, nothing should.compromise frame fit and comfort. Maybe the xl frame just takes some getting used to.