I feel your pain.
For construction fasteners, I carry around driver tips for phillips
#1 ,2,&3; square drive
#1 ,2,&3; Torx
#2 0 ,25, &30 (also use T40); combo square/phillips; 1/4", 5/16",& 3/8" hex drivers;
as well as 1/4" and 3/16" flat blade and
#2 phillips screwdrivers. SAE & metric allen keys often needed as well.
It's difficult to tell the difference between a T20 screw head and a T25 one or between T25 and T30 but at least the Torx system works well in use.
My suggestion is don't bother fighting it- it will only get worse.
multi tools commonly come with a T25 these days, & get a Torx Y wrench for home.