As soon as a motor vehicle goes on the road, the risk goes up for everyone, not only for the vulnerable road users but also for other motorists, himself and occupants in buildings. Curb jumpers smashing into department stores and bus stop shelters are more common than you think.
You don't know if that driver is civil, drunk or a road rager out to teach someone a lesson. Over 90% of drivers think they are good drivers but think lowly of other drivers.
There's a big difference between riding your bike on a road or riding your bike on the same road with a moving car on it. If you happen to slip on some road debris, you'll get some scratches but you'll be able to get up and continue - much like riding on a mountain bike trail or a MUP. But if you slip on road debris with a car going by, your risk of death or hospitalization is much higher.