Your assumption about the head tube angle is incorrect. A longer axle to crown fork will reduce the head tube angle. reduce the seat tube angle and raise the bottom bracket. Trail will be increased, as will wheel flop. I wouldn't describe it as "messing up" the handling, but it will be different. This is very similar to how I design my gravel frames, so I know the handling won't be bad, but it will have more wheel flop at low speed.
As Andrew noted, your fit on the bike will change and you will have to make some saddle and bar height/reach adjustments to match the position you have now.