I have 2 aluminum bikes and a steel bike. I have ridden multiple centuries on each of them. To be honest with you, I don't notice that much of a difference between how they ride, but I suspect people vary in how sensitive they are to certain vibrations, flexing, etc. For me, tires are a much bigger factor in comfort over long rides than frame material. I fully admit that I have never ridden a CF bike for any distance whatsoever, so I have no opinion on how they ride.
With the kind of riding I do, marginal differences in the weight of the frame just really don't make big differences in overall speed--it's probably the least important factor for me, as the equipment I need to make my bike self-sustaining over a long ride probably negates any chance that the percentage difference in gross vehicle weight is even going to be noticeable.
Some of this is just esthetics--steel bikes, in my opinion, look better with the thinner tubes that the material allows. To me, many of the CF and AL forms just look bulbous in a way that I don't appreciate.