Also remember that drop bars are positioned out further than flat bars. Perhaps that extra length and having your nose over the stem is making it uncomfortable
I road raced for pretty much most of the 80s, using drop bars and mounted pretty low. I was a svelte 150 lbs, but then as I grew up (or out) the drops became less and less comfortable. I still have a road bike with drops, which I ride once a year to remind me of how much I don't like them.