Yeah, drafting can be tricky. The groups I've ridden with that ride my slowish pace of 12-15 are so skittish all over the road that drafting isn't safe. The experienced riders who'd be safe to draft with are too fast for me, at least at my current level of conditioning.
Looking at my videos of recent group rides, it's like watching kids in carnival bumper cars. It's amazing there was only one crash, an unforced spill by the one guy who repeatedly veered all over the place and into my path repeatedly. The loudest sound on my recording was me slamming on the brakes every time he'd swerve into my path.