Hopefully these are solo descents or a very small loose group widely spaced. You just need to do more of them till you get comfortable but hopefully not overconfident.
What about your braking technique? You aren't riding the brakes all the way down are you. You should be using them to slow you down to a slow speed then releasing them and letting them cool while the bike accelerates. Just before you start to get antsy about the speed, brake again to slow back down to a speed you can again release the brakes for a while.
As for changing your saddle position, why? That's where the fitter said it should be for you and apparently it's been comfortable for you. If it's more forward than normal, it's probably because you bought the wrong size bike. Get another bike if you want your CG further aft on the bike. But I see no reason to if you learn proper braking of front and rear and realize the front is what stops you.