I am hoping this is just in my head and I will get used to but if not I may try using a longer stem to see if that helps.
Yup, it's mostly in your head. You'll get used to it.
I used to have 100mm stem and changed to 32mm stem with short reach drop bar to experiment. A 32mm stem would actually be too short for my measurements. Steering felt twitchy at first but after nearly a month of riding with it, it doesn't feel twitchy anymore! Totally confident descending at 40 mph.