As long as you are somewhat "active" and don't eat/drink too much, the time off should not dramatically decrease your performance. After three to four days in the saddle, you will start to feel like yourself again. The key is not to treat the time off as though it is Christmas.
I have taken 3 weeks off for a beach trip and did not have much issue. That included some swimming, beach walks, and eating somewhat healthy.
On the other hand, I have taken 2 weeks off during the holidays (big meals, desert, and beer!) and when I got back on the bike I felt like a pudge...