Been in the bike biz since 1982 and have seen steel, aluminum, titanium, and carbon fiber frames fail. No failure outside of one titanium frame, happened on its own. All of them had come from outside sources, ie crashes, drops, or otherwise. Without a question I have been presented with more carbon frame failures than all of the other materials combined. Frankly, if you ain't sponsored with a sweet bike deal, don't race them, it can get expensive.
Carbon is just fine if you like. So is steel, aluminum, and titanium. Run what you can afford and enjoy every time you can climb aboard and get a few miles in.