Weight is only one advantage, it's definitely not the case that the weight savings are very significant unless you spend a lot more. There's a rumor that S-Works top-of-the line frames are the same as Specialized's lower tier frames but happen to weigh less when out of the factory -- kind of like how Intel would sell identical computer chips at different prices based on how overclocked they could safely get them. One can actually pay thousands less and get a lighter frame if they picked a lighter paint!
Aerodynamics and compliance are two other factors that some riders may discern. If you buy from an LBS, you can also expect better service and support (or you picked the wrong LBS). All those reasons plus cachet plus companies knowing that there are people who are willing to pay at the pointy end of the price spectrum means high-end prices are inflated.
Objectively that added price probably isn't worth it from a measurable standpoint, but a big part of cycling is enjoying your ride and that is pretty subjective.