I get where you are coming from but there are different ways to look at this. The magic number in your eyes is less than 20 lbs but lets for a moment start using the metric system. Make 10 kg or less your magic number. You have that beat at 9.5 kg! Now if you change out some of the stuff mentioned above, you can easily get to less than 9 kg. Carbon wheels will save you another 500 grams, now you are 8.5 kg.
Remember numbers are social constructs, changing your reference may help you see things differently because as you said it is the idea that bothers you, no the real weight.
With all that said if you really want to go lower one of the first things I did with my Trek that weighs in around your bike was change the tires to Rene Herse Tires and swapped out the cassette for Dura Ace. The next thing I did was change out the wheelset and saved another 5-600 grams. When I ride my bike, the thought of the total weight just vanishes.