With 1.5" slicks, you'll be able to ride fast (enough) on pavement (or, at least the same speed as you're going, with much less effort), and still be able to take on most "gravel" roads and trails.. Rock gardens aren't fun, but that's why they make SPDs.
Source: see my bikes <-- 1993 StumpJumper