Unless you're planning on racing, don't worry about the skinny guys. They'll always be faster on the climbs simply due to power to weight ratio. On the flats you can probably keep up with them and on the descents you'll pass them with ease as gravity is your friend in this case.
I'm sort of in the same boat as you but not as worried about it. When I was big (6', 245) I never had big arms or shoulders. Now that I'm down to 165 and can climb really well, I look a little anorexic in the upper body. I need to build some muscle in my core to prevent back problems more so than because I want to have upper body muscle definition. I'm sure I'll never have big arms unless I lift or something, which isn't something I care about.