A local bike shop owner, very knowledgeable guy, told me if you crash a CF bike and something breaks, an aluminum bike would also break.
Ride before you buy. And remember that fit and feel is
#1 to the rider. Then components.
I'm in the same boat. I'm readying to buy a quality bike. I like the Cannondale Synapse with Shimano 105 components. I also like the Trek Checkpoint ALR4. Both these bikes are aluminum. As for CF I like the Specialized Roubaix and the new 2019 Giant Defy Advanced 3. I'm leaning heavily towards the Trek. It's a do everything sort of bike. Food for thought....
And I agree with popcycle.... CF is not fragile. If it was, no manufacture would make bikes with it. So no, its not fragile.