I have a
SRAM 11-36T cassette with GRX 46/30T up front and Ultegra derailleurs on one wheelset. This works without issue. I just had to adjust the B-screw (my derailleur officially goes up to 32T). You should be fine. A smaller front (if you could even find it) will negatively impact shifting. I can cross-chain with no problem (but you shouldn't need to do that).
The link I have is to Amazon, but it is better to buy from a company that doesn't make its employees piss in a bottle.