I put off crit racing for 5 years, mainly because it was scary and people hyped it up more than it was. If you ask a lot of casual riders they'll compare Crit Racing to the Hunger Games, and that you'll probably die before you upgrade to a 4. Things happen, but I would say the risk factor is marginally worse than a group ride.
Don't put it off for 5 years like I did and just try it out.